Quality Control Procedures for Archived Operations Traffic Data: Synthesis of Practice and Recommendations

Recommendations for Quality Control Procedures

This section provides the following recommendations for quality control procedures as they relate to archived operations data:

• Recognize that validity criteria (i.e., quality control) are only one part of a comprehensive quality assurance process that does more than just discard suspect data that have already been collected.
• Provide metadata to document quality control procedures and results.
• Provide metadata to document historical traffic sensor status and configuration
• Use database flags or codes to indicate failed validity criteria.
• At a minimum, implement basic foundation for data validity criteria.
• Further develop other spatial and temporal consistency criteria for ADMS.
• When feasible, use visual review to supplement the automated validity criteria.

Recognize that quality control is one part of quality assurance

"Quality control" and "quality assurance" are often used interchangeably to mean the same thing. The term "quality control" has been used in this report to describe the process of reviewing and manipulating data that have already been collected. However, one should recognize that data quality actions restricted to removing suspect data that have already been collected (referred to as "scrap-and-rework") are ineffective in the long-term because they address the symptom but not the root cause of poor data quality.

This distinction in terms is highlighted here because some data archives may be focused primarily on quality control, such as reviewing and editing traffic data that have already been collected. However, total quality management principles indicate that process efficiency and customer satisfaction are most cost-effectively met when quality is considered in all steps of a product cycle. This means that if data archives are to meet customer's needs, data quality must be considered in all steps of the traffic monitoring system. Quality assurance refers to these various quality considerations that are made throughout the traffic data collection and archiving cycle. In several cities, for example, transportation planners have worked with the traffic operations personnel to designate selected detector locations for priority maintenance. Other examples of quality assurance actions are discussed in AASHTO's Guidelines for Traffic Data Programs update in 2007.

In some cases, users or managers of archived data may have little or no direct control over quality assurance processes other than quality control. For example, planning groups that wish to use archived data may currently have little to no input on procuring and testing operations-based traffic sensors. At the same time, though, users of archived data should recognize that long-term improvements to poor data quality can only be made by moving toward a position that influences how real-time traffic data are collected and archived.

Provide metadata to document quality control procedures and results

This report recommends that ADMS provide adequate metadata to document quality control procedures and results. Metadata are "data about data" and typically describes the content, quality, lineage, organization, availability, and other characteristics of the data. Metadata are typically used to:

1. Determine the availability of certain data (for example, through searches of a data catalog or clearinghouse);
2. Determine the fitness of data for an intended use;
3. Determine the means of accessing data; and,
4. Enhance data analysis and interpretation by better understanding the data collection and processing procedures.

An ADMS metadata standard, ASTM E2468-05²⁹ was published in 2005 for the purpose of providing a standard framework for ADMS documentation, including quality control procedures. The framework standardized in ASTM E2468-05 includes qualitative and quantitative data that are associated with an information system or information object for the purposes of description, administration, legal requirements, technical functionality, use and usage, and preservation. As such, this type of metadata can be differentiated from other metadata in that it describes and provides an interpretation of an organized collection of data, not a single data element. For example, ASTM E2468-05 includes a section titled "Data Quality Information," within which there are the following subsections:

• 2.1 Attribute Accuracy - An assessment of the accuracy of attributes within a data archive (contains several subsections).
• 2.2 Logical Consistency - An explanation of the fidelity of relationships in the data set and tests used.
• 2.3 Completeness Report - Information about omissions, selection criteria, generalization, definitions used, and other rules used to derive the data set.
• 2.4 Positional Accuracy - An assessment of the accuracy of the positions of spatial objects (contains several subsections, may not be applicable).
• 2.5 Lineage - Information about the events, parameters, and source data which constructed the data set, and information about the responsible parties (contains several subsections).
• 2.6 Cloud Cover - Area of a data set obstructed by clouds, expressed as a percentage of the spatial extent (may not be applicable).

This metadata standard was adapted from a geographic information system (GIS) metadata standard developed by the Federal Geographic Data Committee (FGDC); thus, some of the sections (e.g., cloud cover) may be not applicable to archived traffic data.

Another ASTM standard under development (work item WK7604) for ADMS will include metadata for specific traffic monitoring data elements. For example, this standard will provide data definitions for quality control flags within ADMS. This standard will also likely provide data definitions for the number of data samples within summary statistics. For example, traffic operations data are often collected in short time intervals (e.g., 20 seconds to 1 minute) and aggregated to longer time intervals (e.g., 5 to 15 minutes) when loaded into an ADMS. There may be occasional interruptions in the operations data feed, which results in missing data samples. Therefore, it is important to document the number of data samples that have been used to calculate summary statistics. If any adjustments have been made to summary statistics based on missing data samples (i.e., factoring up traffic counts to account for missing data samples), this should be clearly documented or represented in the ADMS.

Provide metadata to document historical traffic sensor status and configuration

In many real-time traffic monitoring systems, the configuration of point-based traffic sensors is relatively dynamic. That is, traffic sensors fail, are damaged, are replaced, are relocated, etc. on a routine basis. Real-time traffic data applications mostly care about where the data is coming from at the current time, not where or how data has been collected in the past five years. However, the historical configuration of the traffic sensor system has critical importance for the analytical uses of archived traffic data.

Data archives should provide metadata to document the historical status and configuration of a traffic sensor system. The most basic metadata would be an inventory of traffic sensors, with a date and time field representing when the shown sensor configuration (e.g., information about sensor location, type, characteristics, etc.) was activated and when (or if) the shown sensor configuration has been deactivated. Depending upon the archived data applications, data managers may find it helpful to add additional metadata to track temporary sensor configurations that may only be used during maintenance or reconstruction activities.

Use database flags or codes to indicate failed validity criteria

This report recommends that database flags or markers be used when traffic detector data does not meet one or more of the established validity criteria. The specific implementation design of database flags or codes is not specified here. However, the implementation design should include the ability to distinguish between the failures of different criteria. That is, a unique database flag or marker should be associated with each validity criterion, so that it is possible to determine how many data records have failed each individual validity criterion.

In most planning-based traffic data systems, flagged or marked data are then manually reviewed by a traffic data analyst who makes an accept/reject decision based on supporting information (e.g., local knowledge, comparative traffic data from other time periods or locations, or other anecdotal information from field data collection notes or logs) or professional judgment. If manual review is used, a second flag could be used to indicate whether data values have passed or failed manual review. Due to the sheer volume of incoming data in many data archives, it may not be feasible to manually review all flagged or marked data from traffic operations systems. However, periodic monitoring of the extent of flagged data is highly recommended, and any deviations or extreme fluctuations should be further investigated.

If the flagged data in question is accepted by the analyst after manual review, it is then typically included in data summarization and reporting procedures. Some traffic data systems provide the flexibility to exclude data that have failed a validity criterion but was subsequently accepted. Other traffic data systems may simply flag all data that fails the validity criteria without discarding the flagged data. Users that query the database then have the ability to exclude flagged data on a criterion-by-criterion basis.

In many data archives, the validity criteria and the resulting flags are applied to original source data before it is aggregated and loaded into the data warehouse. Therefore, nearly all data archives that offer summary statistics have made the decision to exclude data that have failed validity criteria. For example, validity criteria are applied to 30-second data, and the invalid values (i.e., those with "failed" flags) are not included in the calculation of 5-minute data stored and offered through the data archive.

The quality control process may assign a severity level to each validity criterion. In practice, however, most data archives assign a simple pass/fail to each criterion. The severity level is used to indicate the seriousness of the criterion failure as well as the necessity of manually reviewing the failed data. The following severity levels should be considered when flagging or marking data that have not met validation criteria:

• Error - An "error" is a condition that is highly likely to be caused by equipment malfunction. An example would be 24 consecutive hours of zeros in a single lane of a four-lane highway. But even here, it is possible that one lane is closed, so it could be marked as "actual." The default action for data marked as containing an error is not to use the data for further processing.
• Caution or Warnings - A "caution" or "warning" indicates that data are outside the acceptable ranges, but not significantly so. These data should be investigated to determine whether to accept or reject them. The default action for a caution or warning condition is to use the data in downstream processing.
• Informational - "Informational" messages indicate data toward the boundaries, but not outside them. Depending on conditions, these data might need to be investigated further. These data will be used in downstream processing unless the operator specifically changes their status.

At a minimum, implement basic foundation for data validity criteria

From the state-of-the-practice review, there are several univariate range criteria and multivariate logical consistency criteria that are used in most of the data archives surveyed. These criteria that are widely accepted form the basis of recommended validity criteria in this report. Table 11 summarizes the validity criteria that are recommended for implementation in data archives. The table also contains suggested default parameters for these criteria. These default parameters can be adjusted or customized for local or regional differences in traffic data collection systems.

Nearly all of the validity criteria should have parameters that can be customized for individual detectors or groups of detectors. Certain criteria may use parameters that do not vary regardless of the detector location or traffic characteristics. For example, data values less than zero will be used for all detectors regardless of location or functional type. Other criteria, such as the maximum vehicle count, may utilize different parameters for freeway mainlanes than for ramps or arterial streets.

Further develop other validity criteria for ADMS

The validity criteria in Table 11 are basic criteria that build a solid foundation for archived data quality control. However, subtle errors or biases could potentially go undetected with these basic criteria. The available literature contains numerous examples of more sophisticated validity criteria. The traditional (i.e., planning-based) traffic monitoring industry uses validity criteria that compare current values in temporal and spatial context. That is, data values are checked for consistency by comparing to historical data at the same location, or to other current data values at nearby locations.

Researchers and data archive managers should investigate and further develop validity criteria that evaluate current data values for temporal and spatial consistency. These criteria could identify suspect or invalid data in which there are:

• Rapid fluctuations in values across successive time periods;
• Detectors in adjacent lanes at the same location reporting significantly different values or trends;
• Detectors in adjacent upstream or downstream locations reporting significantly different values or trends;
• Detectors from multiple locations reporting the same values (indicative of a system problem);
• Reported values that are significantly different from the location's history for similar days of the calendar.

In developing these and other additional criteria, they should be tested with local data to verify expected results before implementation. The testing of new or different validity criteria is often a trial-and-error process that involves exploratory data analysis. The following is a suggested approach to testing validity criteria for implementation:

• Identify data records that do not meet the candidate criterion - Use database queries or filters to identify several dates, times, and locations at which the data do not meet the candidate criterion.
• Look at the suspect data values in their temporal context - Use time-of-day charts to visualize the data trends before and after the failed data value. Does the failed data value represent an abrupt (suspect) or gradual (not as suspect) change? Does the suspect data value seem possible or reasonable given the daily trends for that day, or for other days in that week or weekdays in that month?
• Look at the suspect data values in their spatial context - Graph the data values from all lanes at that location as separate lines on a time-of-day chart. Are the trends throughout the day consistent and reasonable? Does the suspect data value represent abrupt or gradual change?
• Would the suspect data values also be flagged by other existing validity criteria? When considering a new validity criterion, it should identify suspect data that is not otherwise identified by existing criteria. Otherwise, this additional criterion will be redundant and may slow data processing and data warehouse loading.
• Consider how the inclusion or exclusion of data failing this criterion may impact your data applications - A candidate criterion may identify data that in some cases seems reasonable, but in other cases does not seem reasonable. In determining whether to implement this candidate criterion, one should consider the likely impact on the data. Will implementation of the criterion have a large impact, in terms of excluding a significant portion of the data from further analyses? Consideration should also be give to how conservative or aggressive the quality control should be. In other words, would including invalid data be a bigger problem than excluding valid data?

In developing validity criteria, data managers should also pay attention to the data source and whether data values are measured directly or estimated/calculated. For example, many freeway mainline detectors rely on two consecutive loops to collect speed data. In some cases, however, speed is an estimated value derived as a function of occupancy. For such detectors, it may not be obvious when data errors are the result of equipment failure or poor calibration.

Use visual review when feasible

Validity criteria often identify only the upper and lower bounds for valid data. As such, it may be difficult to identify subtle data quality issues (such as slight undercounting of vehicles) using automated validity criteria. When feasible, visual review of summary data should be used to supplement automated validity criteria. There are certain traffic patterns and trends that may be more suitable for manual visual checks. These checks may be difficult to automate, or data analysts may intrinsically value the ability to visualize the data to be used in the performance monitoring analysis. In these cases, a "human-in-the-loop" can be used to visually review a standard chart, graphic, or table to determine whether the represented data portrays a reasonable pattern or trend. These visual checks will require more staff time, but are valuable in building confidence in the quality of the analysis database.

For example, Figure 3 shows the flow rate and speed profiles by time of day for a group of adjacent lane detectors west of downtown Phoenix. At this location, one would expect to see a strong morning peak to coincide with peak traffic driving east to Phoenix. Showing flow rates in conjunction with speed values helps to determine the typical times of congestion, as well as the reduced volumes during congestion.

Figure 3. Flow Rate and Speed Profiles Used to Check Data Quality in Phoenix, Arizona

Figure 4 shows another example of a chart that is used to visually review large amounts of speed and travel time data in the FHWA's Urban Congestion Reporting Program. In this chart, average weekday speeds for an entire month are displayed together as a way to identify odd or suspect days. Depending upon the extent of the suspect data, further "drill down" analyses may be conducted for that day to determine if the speed data are plausible.

Figure 4. Average Weekday Speed Profiles Used to Identify Suspect Data in the FHWA's Urban Congestion Reporting Program

Another visual review approach uses highway capacity theory to identify suspect data.³⁰ Figure 5 shows a chart that plots traffic flow rates versus speeds (a full year of data are shown). This chart enables the data analyst to compare the shape of the field data points to the expected parabolic shape, as indicated in highway capacity theory. Figure 5 shows an example of loop detector data that embodies typical characteristics of a speed-flow curve: a near-parabolic shape with a flattened top that peaks at traffic capacities near 2,000 vehicles per hour per lane. Figure 6 provides an example of data that does not appear to conform to traffic flow theory. In this case, the data at this location should be further examined to determine its validity.

Figure 5. Example of Speed-Flow Curve Identifying Valid Archived Data

Figure 6. Example of Speed-Flow Curve Identifying Suspect Archived Data

²⁹ ASTM E2468-05, Standard Practice for Metadata to Support Archived Data Management Systems, available from ASTM International at http://www.astm.org.
³⁰ Eurek, E., Improving Planning-Level Speed Estimation Models, Undergraduate Fellows Program, Texas A&M University, August 2004.