|FHWA > Policy Information > Travel Monitoring > Adus > Quality Control Procedures... > Recommendations for Quality Control Procedures|
Quality Control Procedures for Archived Operations Traffic Data: Synthesis of Practice and Recommendations
Recommendations for Quality Control Procedures
This section provides the following recommendations for quality control procedures as they relate to archived operations data:
Recognize that quality control is one part of quality assurance
"Quality control" and "quality assurance" are often used interchangeably to mean the same thing. The term "quality control" has been used in this report to describe the process of reviewing and manipulating data that have already been collected. However, one should recognize that data quality actions restricted to removing suspect data that have already been collected (referred to as "scrap-and-rework") are ineffective in the long-term because they address the symptom but not the root cause of poor data quality.
This distinction in terms is highlighted here because some data archives may be focused primarily on quality control, such as reviewing and editing traffic data that have already been collected. However, total quality management principles indicate that process efficiency and customer satisfaction are most cost-effectively met when quality is considered in all steps of a product cycle. This means that if data archives are to meet customer's needs, data quality must be considered in all steps of the traffic monitoring system. Quality assurance refers to these various quality considerations that are made throughout the traffic data collection and archiving cycle. In several cities, for example, transportation planners have worked with the traffic operations personnel to designate selected detector locations for priority maintenance. Other examples of quality assurance actions are discussed in AASHTO's Guidelines for Traffic Data Programs update in 2007.
In some cases, users or managers of archived data may have little or no direct control over quality assurance processes other than quality control. For example, planning groups that wish to use archived data may currently have little to no input on procuring and testing operations-based traffic sensors. At the same time, though, users of archived data should recognize that long-term improvements to poor data quality can only be made by moving toward a position that influences how real-time traffic data are collected and archived.
Provide metadata to document quality control procedures and results
This report recommends that ADMS provide adequate metadata to document quality control procedures and results. Metadata are "data about data" and typically describes the content, quality, lineage, organization, availability, and other characteristics of the data. Metadata are typically used to:
An ADMS metadata standard, ASTM E2468-0529 was published in 2005 for the purpose of providing a standard framework for ADMS documentation, including quality control procedures. The framework standardized in ASTM E2468-05 includes qualitative and quantitative data that are associated with an information system or information object for the purposes of description, administration, legal requirements, technical functionality, use and usage, and preservation. As such, this type of metadata can be differentiated from other metadata in that it describes and provides an interpretation of an organized collection of data, not a single data element. For example, ASTM E2468-05 includes a section titled "Data Quality Information," within which there are the following subsections:
This metadata standard was adapted from a geographic information system (GIS) metadata standard developed by the Federal Geographic Data Committee (FGDC); thus, some of the sections (e.g., cloud cover) may be not applicable to archived traffic data.
Another ASTM standard under development (work item WK7604) for ADMS will include metadata for specific traffic monitoring data elements. For example, this standard will provide data definitions for quality control flags within ADMS. This standard will also likely provide data definitions for the number of data samples within summary statistics. For example, traffic operations data are often collected in short time intervals (e.g., 20 seconds to 1 minute) and aggregated to longer time intervals (e.g., 5 to 15 minutes) when loaded into an ADMS. There may be occasional interruptions in the operations data feed, which results in missing data samples. Therefore, it is important to document the number of data samples that have been used to calculate summary statistics. If any adjustments have been made to summary statistics based on missing data samples (i.e., factoring up traffic counts to account for missing data samples), this should be clearly documented or represented in the ADMS.
Provide metadata to document historical traffic sensor status and configuration
In many real-time traffic monitoring systems, the configuration of point-based traffic sensors is relatively dynamic. That is, traffic sensors fail, are damaged, are replaced, are relocated, etc. on a routine basis. Real-time traffic data applications mostly care about where the data is coming from at the current time, not where or how data has been collected in the past five years. However, the historical configuration of the traffic sensor system has critical importance for the analytical uses of archived traffic data.
Data archives should provide metadata to document the historical status and configuration of a traffic sensor system. The most basic metadata would be an inventory of traffic sensors, with a date and time field representing when the shown sensor configuration (e.g., information about sensor location, type, characteristics, etc.) was activated and when (or if) the shown sensor configuration has been deactivated. Depending upon the archived data applications, data managers may find it helpful to add additional metadata to track temporary sensor configurations that may only be used during maintenance or reconstruction activities.
Use database flags or codes to indicate failed validity criteria
This report recommends that database flags or markers be used when traffic detector data does not meet one or more of the established validity criteria. The specific implementation design of database flags or codes is not specified here. However, the implementation design should include the ability to distinguish between the failures of different criteria. That is, a unique database flag or marker should be associated with each validity criterion, so that it is possible to determine how many data records have failed each individual validity criterion.
In most planning-based traffic data systems, flagged or marked data are then manually reviewed by a traffic data analyst who makes an accept/reject decision based on supporting information (e.g., local knowledge, comparative traffic data from other time periods or locations, or other anecdotal information from field data collection notes or logs) or professional judgment. If manual review is used, a second flag could be used to indicate whether data values have passed or failed manual review. Due to the sheer volume of incoming data in many data archives, it may not be feasible to manually review all flagged or marked data from traffic operations systems. However, periodic monitoring of the extent of flagged data is highly recommended, and any deviations or extreme fluctuations should be further investigated.
If the flagged data in question is accepted by the analyst after manual review, it is then typically included in data summarization and reporting procedures. Some traffic data systems provide the flexibility to exclude data that have failed a validity criterion but was subsequently accepted. Other traffic data systems may simply flag all data that fails the validity criteria without discarding the flagged data. Users that query the database then have the ability to exclude flagged data on a criterion-by-criterion basis.
In many data archives, the validity criteria and the resulting flags are applied to original source data before it is aggregated and loaded into the data warehouse. Therefore, nearly all data archives that offer summary statistics have made the decision to exclude data that have failed validity criteria. For example, validity criteria are applied to 30-second data, and the invalid values (i.e., those with "failed" flags) are not included in the calculation of 5-minute data stored and offered through the data archive.
The quality control process may assign a severity level to each validity criterion. In practice, however, most data archives assign a simple pass/fail to each criterion. The severity level is used to indicate the seriousness of the criterion failure as well as the necessity of manually reviewing the failed data. The following severity levels should be considered when flagging or marking data that have not met validation criteria:
At a minimum, implement basic foundation for data validity criteria
From the state-of-the-practice review, there are several univariate range criteria and multivariate logical consistency criteria that are used in most of the data archives surveyed. These criteria that are widely accepted form the basis of recommended validity criteria in this report. Table 11 summarizes the validity criteria that are recommended for implementation in data archives. The table also contains suggested default parameters for these criteria. These default parameters can be adjusted or customized for local or regional differences in traffic data collection systems.
Nearly all of the validity criteria should have parameters that can be customized for individual detectors or groups of detectors. Certain criteria may use parameters that do not vary regardless of the detector location or traffic characteristics. For example, data values less than zero will be used for all detectors regardless of location or functional type. Other criteria, such as the maximum vehicle count, may utilize different parameters for freeway mainlanes than for ramps or arterial streets.
Further develop other validity criteria for ADMS
The validity criteria in Table 11 are basic criteria that build a solid foundation for archived data quality control. However, subtle errors or biases could potentially go undetected with these basic criteria. The available literature contains numerous examples of more sophisticated validity criteria. The traditional (i.e., planning-based) traffic monitoring industry uses validity criteria that compare current values in temporal and spatial context. That is, data values are checked for consistency by comparing to historical data at the same location, or to other current data values at nearby locations.
Researchers and data archive managers should investigate and further develop validity criteria that evaluate current data values for temporal and spatial consistency. These criteria could identify suspect or invalid data in which there are:
In developing these and other additional criteria, they should be tested with local data to verify expected results before implementation. The testing of new or different validity criteria is often a trial-and-error process that involves exploratory data analysis. The following is a suggested approach to testing validity criteria for implementation:
In developing validity criteria, data managers should also pay attention to the data source and whether data values are measured directly or estimated/calculated. For example, many freeway mainline detectors rely on two consecutive loops to collect speed data. In some cases, however, speed is an estimated value derived as a function of occupancy. For such detectors, it may not be obvious when data errors are the result of equipment failure or poor calibration.
Use visual review when feasible
Validity criteria often identify only the upper and lower bounds for valid data. As such, it may be difficult to identify subtle data quality issues (such as slight undercounting of vehicles) using automated validity criteria. When feasible, visual review of summary data should be used to supplement automated validity criteria. There are certain traffic patterns and trends that may be more suitable for manual visual checks. These checks may be difficult to automate, or data analysts may intrinsically value the ability to visualize the data to be used in the performance monitoring analysis. In these cases, a "human-in-the-loop" can be used to visually review a standard chart, graphic, or table to determine whether the represented data portrays a reasonable pattern or trend. These visual checks will require more staff time, but are valuable in building confidence in the quality of the analysis database.
For example, Figure 3 shows the flow rate and speed profiles by time of day for a group of adjacent lane detectors west of downtown Phoenix. At this location, one would expect to see a strong morning peak to coincide with peak traffic driving east to Phoenix. Showing flow rates in conjunction with speed values helps to determine the typical times of congestion, as well as the reduced volumes during congestion.
Figure 3. Flow Rate and Speed Profiles Used to Check Data Quality in Phoenix, Arizona
Figure 4 shows another example of a chart that is used to visually review large amounts of speed and travel time data in the FHWA's Urban Congestion Reporting Program. In this chart, average weekday speeds for an entire month are displayed together as a way to identify odd or suspect days. Depending upon the extent of the suspect data, further "drill down" analyses may be conducted for that day to determine if the speed data are plausible.
Figure 4. Average Weekday Speed Profiles Used to Identify Suspect Data in the FHWA's Urban Congestion Reporting Program
Another visual review approach uses highway capacity theory to identify suspect data.30 Figure 5 shows a chart that plots traffic flow rates versus speeds (a full year of data are shown). This chart enables the data analyst to compare the shape of the field data points to the expected parabolic shape, as indicated in highway capacity theory. Figure 5 shows an example of loop detector data that embodies typical characteristics of a speed-flow curve: a near-parabolic shape with a flattened top that peaks at traffic capacities near 2,000 vehicles per hour per lane. Figure 6 provides an example of data that does not appear to conform to traffic flow theory. In this case, the data at this location should be further examined to determine its validity.
Figure 5. Example of Speed-Flow Curve Identifying Valid Archived Data
Figure 6. Example of Speed-Flow Curve Identifying Suspect Archived Data
29 ASTM E2468-05, Standard Practice for Metadata to Support Archived Data Management Systems, available from ASTM International at http://www.astm.org.