United States Department of Transportation - Federal Highway Administration FHWA HomeFeedback
OHPI Logo Office of Highway Policy Information

Traffic Data Edit Procedures Pooled Fund Study

Traffic Data Quality "TDQ"

FINAL REPORT

By Mark Flinner, Minnesota Department of Transportation and Henry Horsey, Intelligent Decision Technologies, Inc.

For Transportation Pooled-Fund Study SPR-2 (182)

Excerpt from the full report.


Introduction/Executive Summary

Background and Problem Statement

Challenges facing staff from state and local agencies who are responsible for providing traffic data for safety analysis and highway design include:

Introduction to the Project

The Traffic Data Edit Procedures (TDEP) pooled fund study SPR-2 (182) began in 1995 when fourteen states and the Federal Highway Administration (FHWA) expressed the desire to work together to:

  1. Learn from other experts in the field who are in charge of the operation and data quality control of traffic data collection equipment
  2. Document and qualify existing and potential traffic data screening methods
  3. Refine the list of traffic data screening methods into a logically integrated "rule base"
  4. Test the rule base in the context of actual data and expert analysis
  5. Develop software to assist in the evaluation of the rule base and to put revised software into production

Due in part to Minnesota's earlier experience assisting in the development of the Traffic Data Analysis Expert System and Editor, an "expert system" designed to work with continuously collected volume data from automatic traffic recorders (ATRs), Minnesota was chosen to lead this study. The participating states are Minnesota, Wisconsin, South Dakota, Indiana, New York, Connecticut, North Carolina, South Carolina, Georgia, Florida, New Mexico, California, Idaho, and Montana. Arizona and other states attended various meetings as they could.

In the subsequent four years traffic data analysis staff from the participating states and representatives from the FHWA met to:

Accomplishments and Key Results

Major accomplishments of the study include the following:

Publication of Candidate Traffic Edit Topics, Task A2 Report, that included the following work:

Publication of Expert Knowledge Base, Task A3 Report that included the following work:

With the publication of the Task A3 report, the first three objectives of the study were fully met. A prototype version of software intended to test the effectiveness of the coded portions of the rule base was developed called "TDQ" (Traffic Data Quality). The resulting prototype is described in this report and was successfully run in September 1999 on a Minnesota Department of Transportation data set.

However, the lack of funding for contractor support and other state management priorities resulted in the TDQ prototype not being tested again until the summer of 2000. At that point in time, the prototype was unable to complete an analysis. The lack of funds precluded troubleshooting and resolving the problems encountered in the summer of 2000.

This study was unable to meet its last two objectives. While the rule base developed in this effort represented the consensus expertise of traffic analysis experts from around the country, it has not been validated by rigorously testing against actual data from a spectrum of traffic monitoring sites. Prior experience with knowledge-based systems suggests that such testing would have resulted in further modification of the rule base and a better understanding of the types of traffic monitoring sites where the rule base would be most applicable. Thus the fourth objective of the study was not met.

The fifth objective of the study, to put the TDQ software into production was likewise not met. This objective could only be met after extensive testing and modification of the rule base, as well as subjecting the system to a variety of other testing including user interface testing, data integrity testing, stress testing, and system integration.

This study has elevated the standard for traffic data analysis and has helped to increase the consistency of data analysis methods among the participating states. Such improvements in data analysis will result in a more accurate understanding of roadway usage and traffic trends to support investments in roadway construction and maintenance. We invite others to explore ways to implement what has been learned in this study and to develop a more complete understanding of how rule bases can assist the traffic monitoring community in providing improved data to their clients.

It is appropriate to thank several people for contributions to this project that might otherwise go unnoticed. Mark Flinner of the Minnesota DOT and the Project Technical Liaison, deserves a great deal of credit for the existence of this software. Mark's patient advice and endless willingness to review our work has been essential. Joe Racosky of In Motion, Inc., did a great job of documenting much of the knowledge that is built into TDQ. Much of the technical design and virtually all of the development of the programs are the work of Tom Conlon and Simon Damberger, members of the IDT development team. A special thanks is also due to the support staff at IDT, particularly Barb Johnson and Virginia Lilly. Finally, Fred Newcomer, the TDQ Development Manager at IDT deserves considerable credit for shepherding the project through its trials and tribulations to the release of 0.6.

 


Traffic Data Edit Procedures - TDQ prototype software Rule List

Rule_ID A.3 Rule #
Rule Name
Rule Description
0 V42 Date is Correct and Unique

If the date of the input data is not correct or unique, the record will not be loaded into the database. An input error message will be reported.

1 V43 Lane and Direction are Correct

If the lane or direction fields in the input data do not match the station record, the input data will not be loaded into the database. An input error message will be reported.

2 C49 Number of Axles = Number of Axle Spaces + 1

Any vehicle record where the number of axles does not equal the number of axle spaces plus one will be flagged.

3 W70 Number of Axles = Number of Axle Weights

Any vehicle record where the number of axles does not equal the number of axle weights will be flagged.

4 W35 Sum of Axle Weights Does Not = GVW

Any vehicle record where the sum of the axle weights does not equal the recorded GVW will be flagged.

5 V1 Completeness of Data

If the input data is insufficient or invalid in any way, an error message will be reported.

6 V2 Zero Volume for an Hour

Any hourly volume of zero in any lane will be flagged.

7 V4 Extreme Hourly Volume per Lane

The hourly volume in any lane will be reported as anomalous if exceeds this global extreme maximum:

8 V32 1:00 AM to 2:00 AM Volume vs. 1:00 PM to 2:00 PM Volume If the 1:00 AM to 2:00 AM volume is greater than the 1:00 PM to 2:00 PM volume of the same day, a warning will be reported.
9 C1 No Classification Data If no volumes for any vehicle classes are present in the input data, an error message will be reported.
10 W51 Record Contains Valid Date Any vehicle record containing an invalid or unexpected date will be flagged
11 W52 Record Contains Valid Lane Number Any vehicle record containing a lane that does not match the station record will be flagged.
12 W53 Record Contains Valid Class Number Any vehicle record containing an invalid class number will be flagged.
13 C24 Number of Axles Min/Max Any vehicle having more or less than the number of axles in this range will be flagged:
14 W36 Wheelbase Exceeds Value for Class Any vehicle of this class having a recorded wheelbase greater than this maximum will be flagged:
15 W39 GVW Exceeds Value for Class Any vehicle of this class having a recorded GVW greater than this maximum will be flagged:
16 W28 Front Overhang Out of Range Any vehicle with a front overhang outside of this range will be flagged:
17 W26 Rear Overhang Out of Range Any vehicle with a rear overhang outside of this range will be flagged:
18 W30 Sum of Axle Spaces > or = Recorded Vehicle Length Any vehicle where the sum of the axle spaces is greater than the recorded vehicle length will be flagged.
19 W24 Record Contains Off-Scale Warning Any vehicle record containing a vendor's off-scale warning code will be flagged.
20 W46 Wheelpath Imbalance Exceeds Threshold Any vehicle with the total weight on one side exceeding the total weight on the other side by more than this maximum will be flagged:
21 C35 Vehicle Exceeding Speed Min/Max Any vehicle with a recorded speed outside of this range will be flagged:
22 W25 Extreme Speed Any vehicle with a recorded speed greater than this global extreme maximum will be flagged:
23 W43 Heavy Class 6 Vehicle With Close Follower Any class 6 vehicle with an excessive GVW and followed within 2 seconds by another vehicle will be flagged.
24 C26 Extreme Axle Spacing Any vehicle with any axle space greater than this maximum will be flagged:
25 C27 Minimum First Axle Space Any vehicle with a first axle space (following the steering axle) less than this minimum will be flagged:
26 C28 Minimum Subsequent Axle Space Any vehicle with any axle space less than this minimum will be flagged:
27 C29 Minimum Spacing Between Axle Groups Any vehicle with a tandem or tridem axle space less than this minimum will be flagged:
28 W37 Axle Spacings vs. Min/Max Default Values for Class Any vehicle of this class will be flagged if this particular axle space is greater or less than this range :
29 W40 Axle Weights vs. Min/Max Default Values for Class Any vehicle of this class will be flagged if this particular axle weight is greater or less than this range :
30 C30 3S-2 Drive Tandem Spacing Any 3S-2 tractor with a drive tandem spacing outside of this range will be flagged:
31 W50 Class 9 Front Axle Weight vs. Default Min/Max This rule is implemented by rule W40 in the TDQ Prototype
32 W50 Class 11 Front Axle Weight vs. Default Min/Max This rule is implemented by rule W40 in the TDQ Prototype
33 V3 Consecutive Hourly Zero Volumes The number of consecutive zero-volume hours in any one lane will be reported as anomalous if it exceeds this daily maximum:
34 V7 Consecutive Hours with Same Non-Zero Volume The number of consecutive hours with the same non-zero volume in the same lane will be reported as anomalous if it exceeds this daily maximum:
35 V28 Sunday Hourly Directional Split Sunday's hourly directional split will be reported as anomalous if the leading direction's percentage varies from its historical minimum or maximum by more than these tolerances:
36 V28 Monday Hourly Directional Split Monday's hourly directional split will be reported as anomalous if the leading direction's percentage varies from its historical minimum or maximum by more than these tolerances
37 V28 Tuesday Hourly Directional Split Tuesday's hourly directional split will be reported as anomalous if the leading direction's percentage varies from its historical minimum or maximum by more than these tolerances:
38 V28 Wednesday Hourly Directional Split Wednesday's hourly directional split will be reported as anomalous if the leading direction's percentage varies from its historical minimum or maximum by more than these tolerances:
39 V28 Thursday Hourly Directional Split Thursday's hourly directional split will be reported as anomalous if the leading direction's percentage varies from its historical minimum or maximum by more than these tolerances:
40 V28 Friday Hourly Directional Split Friday's hourly directional split will be reported as anomalous if the leading direction's percentage varies from its historical minimum or maximum by more than these tolerances:
41 V28 Saturday Hourly Directional Split Saturday's hourly directional split will be reported as anomalous if the leading direction's percentage varies from its historical minimum or maximum by more than these tolerances:
42 V9 Hourly Volume vs. Next/Prior Day The total hourly volume will be reported as anomalous if it is greater than or less than the total volume for that hour of the previous or following day by these tolerances:
43 V17a Daily Directional Volume vs. AADT The daily directional volume will be reported as anomalous if it is greater or less than the previous year's adjusted directional AADT by this tolerance:
44 V33 Daily Combined Volume vs. AADT The daily combined volume will be reported as anomalous if it is greater or less than the previous year's adjusted AADT by these tolerances:
45 V5 Sunday Daily Directional Split Sunday's daily directional split will be reported as anomalous if the leading direction's percentage varies from its historical minimum or maximum by more than these tolerances:
46 V5 Monday Daily Directional Split Monday's daily directional split will be reported as anomalous if the leading direction's percentage varies from its historical minimum or maximum by more than these tolerances:
47 V5 Tuesday Daily Directional Split Tuesday's daily directional split will be reported as anomalous if the leading direction's percentage varies from its historical minimum or maximum by more than these tolerances:
48 V5 Wednesday Daily Directional Split Wednesday's daily directional split will be reported as anomalous if the leading direction's percentage varies from its historical minimum or maximum by more than these tolerances:
49 V5 Thursday Daily Directional Split Thursday's daily directional split will be reported as anomalous if the leading direction's percentage varies from its historical minimum or maximum by more than these tolerances:
50 V5 Friday Daily Directional Split Friday's daily directional split will be reported as anomalous if the leading direction's percentage varies from its historical minimum or maximum by more than these tolerances:
51 V5 Saturday Daily Directional Split Saturday's daily directional split will be reported as anomalous if the leading direction's percentage varies from its historical minimum or maximum by more than these tolerances:
52 C48 Full Day of Data Exists If less than 24hours of data is present, a warning will be reported as anomalous.
53 C4 Extreme Daily Percent in Any Class Except 2 The daily percent of vehicles binned to any class except 2 (cars) will be reported as anomalous if it exceeds this maximum:
54 C37 Excessive Daily Percent by Class The daily percent of vehicles binned to any class except 2 or 3 will be reported as anomalous if it exceeds this maximum:
55 C38 Excessive Daily Volume by Class The daily volume of vehicles binned to any class except 2 or 3 will be reported as anomalous if it exceeds this maximum:
56 W16 Unloaded Class 9 GVW Distribution Peak The majority of unloaded class 9 GVWs are expected to fall within this weight range:
57 W16 Unloaded Class 11 GVW Distribution Peak The majority of unloaded class 11 GVWs are expected to fall within this weight range:
58 W17 Loaded Class 9 GVW Distribution Peak The majority of loaded class 9 GVWs are expected to fall within this weight range:
59 W17 Loaded Class 11 GVW Distribution Peak The majority of loaded class 11 GVWs are expected to fall within this weight range:
60 W68 Percent of Vehicles With GVW Out of Range for Class The daily percent of vehicles flagged for excessive GVW will be reported as anomalous if it exceeds this maximum:
61 W67 Percent of Vehicles With Invalid Class The daily percent of vehicles flagged for an invalid class disignation will be reported as anomalous if it exceeds this maximum:
62 W21 Average Class 9 Steering Axle Weight The daily average class 9 front axle weight will be reported as anomalous if it falls outside of this range:
63 W21 Average Class 11 Steering Axle Weight The daily average class 11 front axle weight will be reported as anomalous if it falls outside of this range:
64 W65 Percent of Records With Invalid Dates The daily percent of vehicle records flagged for an invalid date will be reported as anomalous if it exceeds this maximum:
65 W66 Percent of Records With Invalid Lane The daily percent of vehicle records flagged for an invalid lane will be reported as anomalous if it exceeds this maximum:
66 W56 Average Steering Axle Weight for Light-GVW Class 9s The average steering axle weight of all class 9 vehicles with a GVW of less than 32,000 lbs. will be reported as anomalous if it falls outside of this range:
67 W56 Average Steering Axle Weight for Mid-GVW Class 9s The average steering axle weight of all class 9 vehicles with a GVW of between 32,000 lbs. And 70,000 lbs. will be reported as anomalous if it falls outside of this range:
68 W56 Average Steering Axle Weight for Heavy-GVW Class 9s The average steering axle weight of all class 9 vehicles with a GVW of more than 70,000 lbs. will be reported as anomalous if it falls outside of this range:
69 W56 Average Steering Axle Weight for Light-GVW Class 11s The average steering axle weight of all class 11 vehicles with a GVW of less than 32,000 lbs. will be reported as anomalous if it falls outside of this range:
70 W56 Average Steering Axle Weight for Mid-GVW Class 11s The average steering axle weight of all class 11 vehicles with a GVW of between 32,000 lbs. And 70,000 lbs. will be reported as anomalous if it falls outside of this range:
71 W56 Average Steering Axle Weight for Heavy-GVW Class 11s The average steering axle weight of all class 11 vehicles with a GVW of Class 11s more than 70,000 lbs. will be reported as anomalous if it falls outside of this range
72 W58 Percent of Class 9s With Front Axle Weight Flags The daily percent of class 9 vehicles flagged for an out-of-range front axle weight will be reported as anomalous if it exceeds this maximum:
73 W58 Percent of Class 11s With Front Axle Weight Flags The daily percent of class 11 vehicles flagged for an out-of-range front axle weight will be reported as anomalous if it exceeds this maximum:
74 C2 Percent of Records With Vendor Warning Codes The daily percent of vehicle records containing a vendor's warning code will be reported as anomalous if it exceeds this maximum:
75 W62 Percent of Vehicles Where GVW Is Not = Sum of Axle Weights The daily percent of vehicle records where the GVW is not equal (within rounding error) to the sum of the axle weights will be reported as anomalous if it exceeds this maximum:
76 W60 Percent of Vehicles With Overhang Flags The daily percent of vehicles with overhang flags will be reported as anomalous if it exceeds this maximum:
77 W8 Percent of Vehicles Where Length < Wheelbase The daily percentage of vehicles where the sum of the axle spaces is greater than the recorded vehicle length will be reported as anomalous if it exceeds this maximum:
78 W10 Class 9 Average Length Within Range + Average Wheelbase The average class 9 vehicle length and wheelbase relationship will be reported as anomalous if the average length is not within the sum of the average wheelbase and this range:
79 W10 Class 11 Average Length Within Range + Average Wheelbase The average class 11 vehicle length and wheelbase relationship will be reported as anomalous if the average length is not within the sum of the average wheelbase and this range:
80 W45 Percent of Records With Off-Scale Warnings The daily percent of vehicle records containing a vendor's off-scale warning will be reported as anomalous if it exceeds this maximum:
81 W47 Pattern of Vehicles With Wheelpath Imbalance An otherwise anomalous percent of wheelpath imbalances will not be reported as anomalous if opposite wheelpath imbalances are detected in opposite directions (likely due to crosswinds).
82 W54 Percent of Vehicles With Wheelpath Imbalance The daily percent of vehicles with wheelpath imbalance flags will be reported as anomalous if it exceeds this maximum:
83 W59 Percent of Vehicles that Exceed Extreme Max Speed The daily percent of vehicles with globally extreme speed flags will be reported as anomalous if it exceeds this maximum:
84 C40 Percent of Vehicles Slower Than Speed Min The daily percent of vehicles with speeds less than the station minimum will be reported as anomalous if it exceeds this maximum:
85 C40 Percent of Vehicles Faster Than Speed Max The daily percent of vehicles with speeds greater than the station maximum will be reported as anomalous if it exceeds this maximum:
86 W61 Percent of Heavy Class 6 Vehicles With Close Follower The percent of class 6 vehicles flagged for excessive GVW with a closely following vehicle will be reported as anomalous if it exceeds this maximum:
87 C15 Average 3S-2 Drive Tandem Spacing The daily average drive tandem spacing for 3S-2 vehicles will be reported as anomalous if it falls outside of this range:
88 W63 Percent of Vehicles With Wheelbase or Axle Spacing Flags The daily percent of vehicles with wheelbase or axle spacing flags set by the default values for their class will be reported as anomalous if it exceeds this maximum:
89 W64 Percent of Vehicles With an Axle Weight Flag The daily percent of vehicles with an axle weight flag set by the default values for their class will be reported as anomalous if it exceeds this maximum:
90 W55 Average Left Axle Weight vs. Average Right Axle Weight

The average left and right axle weights for all vehicles will be reported as anomalous if they differ by more than this maximum percent:

91 V19 Hourly Directional Volume vs. History An hourly directional volume will be reported as anomalous if it differs from its historical minimum or maximum for that hour by more than these tolerances:
92 V40 Hourly Combined Volume vs. Recent History An hourly combined volume will be reported as anomalous if it differs from its historical minimum or maximum for that hour by more than these tolerances:
93 V39 Daily Combined Volume vs. Recent History A daily combined volume will be reported as anomalous if it differs from its historical minimum or maximum by more than these tolerances:
94 V17b Daily Directional Volume vs. History A daily directional volume will be reported as anomalous if it differs from its historical minimum or maximum by more than these tolerances:
95 V29 Daily Percent Distribution by Lane vs. History The daily lanal distribution will be reported as anomalous if any lane differs from its historical average percent by more than these tolerances:
96 C12 Daily Volume Binned to One Class vs. History The daily volume binned to a single vehicle class except 2 or 3 will be reported as anomalous if it differs from its historical minimum or maximum volume by more than these tolerances:
97 C11 Daily Percent Binned to One Class vs. History The daily percent binned to a single vehicle class will be reported as anomalous if it differs from the historical average percent for that class by more than these tolerances:
98 C23 Daily Volume of Both Class 6 and 1 Exceed History The daily volumes of class 1 and class 6 vehicles will be reported as anomalous if both are greater than their average historical values.
99 C22 Daily Ratio of Class 2 to 3 vs. History The daily ratio of class 2 vehicles to class 3 vehicles will be reported as anomalous if the number of class 2s per one class 3 varies by more than these tolerances:
100 C42 Daily Ratio of Class 9 to 8 by Lane vs. History The daily ratio of class 9 vehicles to class 8 vehicles in a lane will be reported as anomalous if the number of class 9s per one class 8 differs from the historical minimum or maximum ratio by more than these tolerances:
101 C19 Daily Ratio of Class 9 to 8 by Direction vs. History The daily ratio of class 9 vehicles to class 8 vehicles in each direction will be reported as anomalous if the number of class 9s per one class 8 differs from the historical minimum or maximum ratio by more than these tolerances:
102 C41 Daily Sum of Class 8 and 9 vs. History The daily sum of class 8 and class 9 vehicles will be reported as anomalous if it differs from the historical minimum or maximum sum of these two classes by more that these tolerances:
103 C14 Daily Class 8 Directional Split vs. History The daily directional split percentages for class 8 vehicles will be reported as anomalous if the leading direction's percentage varies from its historical minimum or maximum by more than these tolerances:
104 C13 Daily Class 9 Directional Split vs. History The daily directional split percentages for class 9 vehicles will be reported as anomalous if the leading direction's percentage varies from its historical minimum or maximum by more than these tolerances:
105 C43 Daily Sum of Class 8 and 9 Directional Split vs.History The daily directional split percentages for the sum of class 8 and class 9 vehicles will be reported as anomalous if the leading direction's percentage varies from its historical minimum or maximum by more than these tolerances:
106 C46 Daily Directional Split of Any Class (not 8 or 9) vs. History The daily directional split percentages for any vehicle class will be reported as anomalous if if the leading direction's percentage varies from its historical minimum or maximum by more that these tolerances:
107 C17 Daily Directional Split of Sum of Class 4 thru 13 vs. History The daily directional split percentages for the sum of all commerical vehicles will be reported as anomalous if the leading direction's percentage varies from its historical minimum or maximum by more that these tolerances:
108 C47 Daily Directional Split of Class Groups vs. History The daily directional split percentages for any class group (passenger, truck, semi-truck and multi-trailer) will be reported as anomalous if the leading direction's percentage varies from its historical minimum or maximum by more that these tolerances:
109 C16 Monthly Directional Split of Sum of Class 4 thru 13 vs. History The monthly directional split percentages for the sum of all commercial vehicles will be reported as anomalous if the leading direction's percentage varies from its historical minimum or maximum by more that these tolerances:
110 C47 Monthly Directional Split of Class Groups vs. History The monthly directional split percentages for any class group (passenger, truck, semi-truck and multi-trailer) will be reported as anomalous if the leading direction's percentage varies from its historical minimum or maximum by more that these tolerances
111 W18 Unloaded Class 9 GVW Distribution Peak Shift A shift in the unloaded GVWs for class 9 vehicles will be reported if the central tendancy of the input data is not within these percents of the historical central tendancy
112 W19 Loaded Class 9 GVW Distribution Peak Shift A shift in the loaded GVWs for class 9 vehicles will be reported if the central tendancy of the input data is not within these percents of the historical central tendancy
113 W23 Loaded vs. Unloaded Class 9 GVW Distribution Peaks A parallel shift in Class 9 GVWs will be reported if the loaded central tendancy's shift from its historical value minus the unloaded central tendancy's shift from its historical value is not within these percent tolerances:
114 W20 Incidental Class 9 GVW Distribution Peak Shift A shift in the major incidental GVW peak for class 9 vehicles (if there is one) will be reported if the central tendancy of the input data is not within these percents of a matching historical central tendancy
115 W18 Unloaded Class 11 GVW Distribution Peak Shift A shift in the unloaded GVWs for class 11 vehicles will be reported if the central tendancy of the input data is not within these percents of the historical central tendancy
116 W19 Loaded Class 11 GVW Distribution Peak Shift A shift in the loaded GVWs for class 11 vehicles will be reported if the central tendancy of the input data is not within these percents of the historical central tendancy
117 W23 Loaded vs. Unloaded Class 11 GVW Distribution Peaks A parallel shift in Class 11 GVWs will be reported if the loaded central tendancy's shift from its historical value minus the unloaded central tendancy's shift from its historical value is not within these percent tolerances:
118 W20 Incidental Class 11 GVW Distribution Peak Shift A shift in the major incidental GVW peak for class 11 vehicles (if there is one) will be reported if the central tendancy of the input data is not within these percents of a matching historical central tendancy
119 C6 Daily Average Speed per Lane vs. History The average vehicle speed in each lane will be reported as anomalous if it differs from the historical average speed for that lane by more than these tolerances:


FHWA Home | OHPI Home | Products and Publications | Feedback
FHWA
United States Department of Transportation - Federal Highway Administration