The Long-Term Pavement Performance Program

PART II. DEVELOPING THE STUDIES AND THE PAVEMENT PERFORMANCE DATABASE 

© American Concrete Pavement Association, www.acpa.org LTPP data requirements are based on achievement of the program’s goal and objectives and, as a result, include literally thousands of measurement concepts.

CHAPTER 9. LTPP DATA QUALITY EFFORTS

The many quality control/quality assurance processes described in this chapter keep the focus of the people involved in the LTPP program on the quality of the work they do. Achieving a high level of data quality is a program expectation. This expectation created the environment that enabled development of cutting-edge quality control processes more than 10 years in advance of accepted industry standards and will enable continued improvements in quality control processes.

Introduction

Since the program’s start, data quality has been a prime concern in the development and operation of LTPP activities. High-quality data have been critical to the program’s success. Given the program’s data requirements—a large number of data elements of varying complexity—as well as the available data acquisition and source options, achieving high-quality data is not only a major challenge, but also one that requires significant resources.

LTPP data requirements are based on achievement of the program’s goal and objectives and, as a result, include literally thousands of measurement concepts. During the program planning process, experts in each associated engineering discipline participated in developing the data requirements, which were then reviewed and critiqued by other stakeholders. Like the data requirements, the methods used to acquire LTPP data were developed by expert staff working in concert with program stakeholders and other authorities. A comprehensive planning process was used to examine and evaluate data acquisition methods that took into account budget, complexity, and time considerations.

Due to the complexity of many LTPP data elements, an early program decision was made to select qualified regional support contractors who had experience in operating technical data collection equipment and working with highway agencies to collect data. Increased planning was needed for data collection that requires special equipment, such as the falling weight deflectometers (FWDs) used in nondestructive deflection testing and the profilers used to collect profile data. These early program planning efforts to establish data requirements, acquisition methods, and sources provided a solid foundation for achieving high-quality data, but they were just the beginning of the LTPP quality initiatives.

Data quality control/quality assurance (QC/QA) processes have been developed, updated, and implemented throughout the LTPP program. The processes fall into several categories:

• QC processes before data collection.
• QC processes during data collection.
• QC processes after data collection.
• Database checks and reviews.
• Regional quality management plans and Federal Highway Administration (FHWA) QA audits.
• Higher order data checks.
• Peer review.
• Feedback processes.

These processes are detailed over the remainder of this chapter. In addition, products that emerged from LTPP QC/QA development and implementation efforts are summarized in chapter 10. The various documents referenced in this chapter—directives, guidelines, manuals, and studies—are available at LTPP’s InfoPave™ Web site.

Quality Control Processes Before Data Collection

After the selection of data collection equipment and regional support contractors, before data collection began, three elements were identified as critical to providing a solid foundation for high-quality LTPP data. These elements are data collection standards, personnel requirements, and equipment requirements.

Data Collection Standards

To promote uniform, consistent, and high-quality data, the LTPP program has published formal standards in the form of data collection guidelines, guides, and manuals that specify data collection frequencies, types of data collection, standard definitions, measurement procedures, test protocols, operation of electronic data collection devices, and data collection forms. These standards are implemented through the issuance of LTPP program directives, which present program policy to be followed by regional support contractors, materials laboratory contractors, and participating highway agencies. When guidelines are revised and updated, program directives are used to document changes in processes, procedures, equipment, and data collection activities.

A team process has been used to develop data collection guidelines. The process starts with the development of a draft guide by the central technical team. LTPP managers solicit review and comment from stakeholders—related expert task groups (ETGs), highway agencies, regional support contractors—on an initial draft. Feedback from the combined team obtained through written comments, teleconferences, Webinars, regional meetings, and national meetings is used to craft a proposed final version. The proposed guidelines are tested and refined through pilot activities and, when final, are issued through a formal LTPP program directive. Following implementation, data collection personnel can report equipment problems, issues with data collection forms, needed changes to procedures, and other issues using the formal Operational Problem Report (OPR) process. Changes and updates to the guidelines are also issued in a program directive.

FIGURE 9.1. Data collection guidelines published as manuals for profile measurements and processing, distress identification, and falling weight deflectometer measurements.

Some of the data collection guidelines have been compiled and published as manuals containing standards specific to the LTPP program. Examples include the highly requested Distress Identification Manual for the Long-Term Pavement Performance Program,¹ the Long-Term Pavement Performance Manual for Profile Measurements and Processing,^2,3 and the Long-Term Pavement Performance Manual for Falling Weight Deflectometer Measurements⁴ (figure 9.1). These publications, like most of the many other LTPP guides, manuals, and guidelines (some of which date back to 1988), have undergone major revisions and updates over the years. A list of the major LTPP data collection guidelines is presented in table 9.1.

In addition to providing standard definitions, measurement procedures, and data collection forms, guidelines also typically address equipment issues and, in the case of electronically collected data, software and data processing issues. For example, the latest version of the LTPP Manual for Falling Weight Deflectometer Measurements covers all aspects of measurement, from equipment setup through data handling procedures. Similarly, the LTPP Manual for Profile Measurements and Processing includes everything from equipment maintenance to inter-regional comparison tests and processing profile data in the office.

Data collection guidelines typically include and require the completion of data collection forms, whether data are collected using electronic means (which have comprised the bulk of LTPP data) or manual methods. These forms include written instructions on all requested pieces of information and pertinent information needed to complete the form. Also, all forms are numbered and dated for document control, and revisions to forms are issued by formal directives that document the changes. Most of the data collection guidelines within the LTPP program are implemented through formal issuance of directives.

Personnel Requirements

Well-trained and knowledgeable data collection personnel are another critical element to achieving high-quality data. Accordingly, the data collection contractor selection process has included consideration of experience in operating various types of pavement data collection equipment, conducting field pavement data collection activities, or performing laboratory material tests, depending on the subject of the data collection contract. In addition, the LTPP program has established minimum criteria for data collection personnel, including previous experience in data collection, formal training with the data collection procedures using a training plan and syllabus approved by the LTPP program staff, and time spent assisting experienced personnel in performing data collection before doing so independently. Implementation of personnel requirements within the LTPP program is accomplished contractually with the data collection contractors or through formal program directives.^5,6

In the case of pavement distress data, for example, an accreditation program was established in 1992, requiring that all pavement distress data be collected by accredited raters. The regional support contractor staff are required to attend the distress accreditation workshops held by the LTPP program to receive and maintain their accreditation. To be eligible to attend, raters first have to be trained to a high level of competence in the knowledge and procedures contained in the Distress Identification Manual using a combination of classroom and field training activities. The regional support contractors are required to provide formal training based on the FHWA National Highway Institute’s “Pavement Distress Identification Workshop” course. In addition, new raters are required to assist experienced personnel in performing distress data collection for asphalt and jointed concrete pavement (a minimum of two sections each) before becoming eligible to attend an accreditation workshop.⁷

The first distress accreditation workshops were held in 1992 in Reno, Nevada,⁸ and workshops were held at least annually early in the LTPP program. More recently, the workshops have been held less frequently due to budget limitations. The accreditation process consists of two major parts: a written examination to test the general knowledge of the rater and a field data collection or film interpretation examination to measure the rater’s capabilities in observing and recording distress data. Grading and accreditation are accomplished by comparing the rater’s results to the reference values established by a team of experienced raters.

FIGURE 9.2. Distress raters evaluate a pavement surface in an LTPP accreditation workshop, where raters’proficiency is tested.

The distress accreditation workshops are intended for trained, experienced personnel only; they are not intended to train personnel. Their primary purpose is to assure high quality and uniformity in identifying pavement distress. Furthermore, to maintain their accreditation, distress raters have to meet minimum recertification requirements, including performing at least 15 surveys in a 12-month period—ideally evenly distributed throughout that time period—and successfully completing the accreditation workshop annually, or as required by the LTPP program. Figure 9.2 shows a distress accreditation workshop held in Columbus, Ohio, in May 2010.

Training sessions focused on the handling of weather and seasonal data were conducted as well. Personnel learned how to review the weather plots and identify and edit erroneous data collected from the automated weather stations, and they were trained in data processing QC procedures for the Seasonal Monitoring Program.

To promote consistency in data collection across the test sites, national and regional meetings of regional support contractor staff as well as teleconferences involving data collection personnel are conducted on each major data collection topic. These interactions, which vary in frequency according to the need and the maturity of each data collection activity, allow for the exchange of ideas and improvements as well as the discussion and resolution of issues.

In addition, the regional support contractors’QC plans, mandated by the LTPP program, include processes for training new staff and reviewing the performance of existing staff, including some exchanges of equipment operators between the regions to review each other’s efforts. The training requirements within these QC plans address not just the distress data collection, but all collection activities such as FWD, profile, and the Seasonal Monitoring Program. (Regional contractors’QC plans are addressed later in this chapter, along with FHWA QA audits.)

Equipment Requirements

In developing the LTPP data collection plans, it was determined that the program should own and operate specialized data collection equipment to best achieve the program’s objectives. However, some data collection services were engaged due to consideration of the equipment cost, advanced operating requirements, or volume of data measurements to be performed. Contracted services have been used to collect photographic distress data, perform laboratory materials tests, and take ground penetrating radar measurements. The LTPP program has acquired FWD, inertial profiling, manual distress survey, dynamic load response, seasonal monitoring, and traffic data collection equipment.

LTPP data collection activities rely on equipment that is often of a fairly sophisticated nature, which has been built using technologically advanced components to meet the program’s specifications (table 9.2). The basic inertial profiler components, for example, include laser sensors, accelerometers, data collection triggers, a distance-measuring instrument, and a computer system that combines the measurements from each of the components to provide the longitudinal profile data. In 2013, texture measurements, surface temperature, GPSr (Global Positioning System receiver), and photo and video capabilities, were added.

Because properly functioning equipment is critical to achieving high-quality data, the maintenance, repair, checks, and calibration of equipment used in data collection have been an integral part of LTPP QC/QA efforts since the program’s beginning. Most data collection guidelines address equipment maintenance, repair, checks, and calibration procedures, and each regional support contractor is required to establish and follow an equipment preventive maintenance plan that is accepted and monitored by LTPP program staff. Throughout the course of the program, many directives have been issued that address specific maintenance checks and calibration procedures for LTPP data collection equipment.

Calibrations and checks are performed on all data collection equipment to insure proper functioning before they are used in the field or laboratory. In those cases where it is not possible to calibrate a device, equipment checks are used. For example, temperature sensors (thermistor probes) in the Seasonal Monitoring Program were placed in substances of known temperature such as ice and boiling water. If the sensors were found to be outside an established range, they were either returned to the manufacturer for adjustment or replacement.

Although the LTPP program uses existing technology to the extent that is practical, in some cases it has had to develop calibration and check procedures for program equipment. For instance:

• The program developed the first reference calibration procedure for FWDs in the United States. This development process was independent of the equipment supply contractor, and Federal, State, and international highway agencies subsequently adopted the LTPP FWD reference calibration procedure.^9,10,11
  
  
• The program developed cutting-edge laboratory equipment calibration methods for highly sensitive tests for elastic response of pavement materials in 1997,¹² which were later expanded and modified in a 2005 publication.¹³ These tests, informally called startup procedures, involved expert instrumentation engineers who checked the internal functions of advanced laboratory electronic measurement equipment to identify sources of bias and error that could not be reliably detected through inspection of the output data. These checks measured the output from a controlled electrical input into the system to discern if the instrumentation was correctly calibrated to the manufacturer’s specifications.

A number of software programs have also been developed and implemented to help identify potential equipment problems and to check calibrations. Examples include:

• FWDCAL—This program automates analysis of the relative calibration test procedure results for the FWD geophones and includes these functions: check FWD relative calibration data file for compliance with LTPP test setup requirements, calculate new relative gain factors for each geophone, check the ratios between existing and new relative gain factors to determine if they are within established tolerances, perform an analysis of variance on the data to determine the statistical significance of key test factors, provide a statistical summary of the test results, provide guidance to the user on needed gain changes or further testing needs, compute the gain factor for a replacement sensor, process up to three data sets in the same file, and compute an average new relative gain factor from relative calibration tests performed as a part of the LTPP reference calibration procedures.
  
  
• MOBILE—This program was used in the Seasonal Monitoring Program to perform measurements on the time-domain reflectivity calibration reference cable installed in a user-specified port and was intended to be an aid in diagnosing problems with the time-domain reflectivity data collection system whenever an anomaly was encountered.

FIGURE 9.3. Profiler rodeo held at the Minnesota Road Research Project (MnROAD) site in Albertville. From May 14 to 18, 2007, researchers performed a comparison test of the profilers used by the four LTPP regional support contractors.

Another important concept, introduced by the LTPP program in the early 1990s, was the use of equipment comparisons, which are referred to as “FWD thump-offs” and “profiler rodeos.” Such comparisons were used not only as a means of accepting equipment for program use, but also for performing periodic evaluations of the equipment’s operational status and for cross-training exercises. For example, upon delivery of the profilers now in use, a stringent acceptance procedure was conducted in February 2013 on reference test sections in College Station, Texas, and a week-long training was held in April 2013.¹⁴

In the profiler rodeos, the LTPP profilers were compared against each other as well as against a reference device, the Face Dipstick and rod and level, to ensure that the equipment was collecting high-quality profile data (as described in chapter 6). These rodeos required the establishment of multiple test sections. The profile of each test section was then measured by the Face Dipstick and profilers, and the resulting data were analyzed to determine compliance with established elevation bias and precision criteria along with the International Roughness Index (IRI) values.¹⁵ Figure 9.3 shows four profilers prepared for testing at a 2007 rodeo in Minnesota.

Ultimately, it is the responsibility of the regional support contractors to maintain the data collection equipment they use in good-working, calibrated condition. This equipment has included devices such as the levels used for surface elevation surveys and pavement and air temperature probes used in conjunction with FWD testing and profile surveys. Furthermore, as noted earlier, each regional support contractor is required to establish and follow an equipment preventive maintenance plan as accepted and monitored by LTPP program staff.

Quality Control Processes During Data Collection

While the data collection guidelines, personnel requirements, and equipment requirements described in the previous section provide an excellent foundation, additional QC elements are needed to assure that data are of high quality. Key elements to be considered during data collection are the ambient conditions, monitoring of data collection activities in the field by responsible personnel, and the use of QC tools to check equipment and data in the field.

Ambient Conditions

Ambient conditions must not be neglected when planning for or performing data collection because they can significantly affect data quality, as shown in these LTPP-specific examples:

• Hot-mix asphalt pavements in some southern States can show less cracking during the hot summer months than in the winter. Likewise, temperature affects the distresses visible in Portland cement concrete pavement, including crack and joint openings. Thus the temperature of a pavement at the time distress measurements are performed is important to identify potential discrepancies in time-series trends.
  
  
• Inertial profiler laser height sensors can malfunction at very high and very low temperatures. At approximately 0 °C and 50 °C (32 °F and 122 °F), some lasers begin to produce errors, and at approximately -40 °C and 70 °C (-40 °F and 158 °F), a laser sensor can turn off to prevent damage.¹⁶
  
  
• The rolling radius of the tire, which is related to tire pressure and hence temperature, directly affects distance-measuring instrument measurements.
  
  
• The interior vehicle environment can be critical to the operation of on-board computers as many computers have a safe operating range of temperature that should be observed before power is applied.
  
  
• Inclement weather conditions, such as rain, snow, or heavy cross winds, can interfere with the collection of acceptable distress, FWD, and profile data.
  
  
• The ability to see and rate some types of distresses is very sensitive to lighting conditions, angle of the incident light, and light intensity.

These ambient condition considerations led to the issuance of formal LTPP program directives, such as these:

• Cessation of “Night Time” Manual Distress Surveys.¹⁷
  
  
• Special Provisions for Cold Weather FWD Testing.¹⁸
  
  
• FWD Surface Temperature Measurements in Cold Weather.¹⁹
  
  
• Data Collection Guidelines Under Less Than Ideal Conditions.²⁰

Most of these directives have been incorporated into the latest versions of the data collection guidelines discussed in the previous section.

Monitoring and Reviewing Data Collection in the Field

Reducing errors at the time of testing, when it is possible to correct a problem, is a priority within the LTPP program. It is standard procedure for the regional support contractor’s staff carrying out data collection activities to follow program guidelines and to continuously monitor the data being collected to ensure that the equipment is functioning properly and the data appear reasonable. If either questionable data or improperly functioning equipment is suspected during data collection, activities are suspended until the issue in question is resolved.

In the case of field activities, data collection personnel are required to survey the test section area to ensure there are no obstacles or debris on the pavement surface that could affect data collection. They are also required to check and to ensure that the start and end of each test section are well defined so that measurements are taken at the appropriate locations.

Once data collection activities begin, a number of monitoring tools are available to determine whether the data appear reasonable and the equipment is functioning properly. For example, cameras are installed on all LTPP FWD units to aid in establishing the accurate location for joint deflection tests on rigid pavements. Another example is the automated data checks contained in the LTPP FWD data collection software, which includes the following error conditions:

• Roll-off—Deflection of the pavement surface, as recorded by a deflection sensor, does not return to near zero within 60 milliseconds of the trigger activation.
  
  
• Nondecreasing deflections—Deflections measured by the deflection sensors do not decrease with increasing distance from the load plate.
  
  
• Overflow—A measured deflection exceeds the range of the deflection sensor or 2030 µm (80 mils) for the FWDs operated by the program.
  
  
• Load variation—The peak load for repeat drops at the same drop height varies by more than the amount specified.
  
  
• Deflection variation—The load-normalized peak deflections for repeat drops vary by more than the amount specified.

If deflection errors occur, the operator is required to attempt to identify the source of those errors. If the errors are determined to be due to problems with the FWD equipment, the problems have to be fixed before testing continues. If the errors are related to localized pavement conditions, the operator is required to reposition the FWD and comment on the condition. If the errors are determined to be due to pavement conditions that are representative of the test section as a whole or due to factors such as truck traffic that are out of the operator’s control, then the operator is required to accept the error and comment on the condition.

As described, there are many valid reasons why some of these “errors” may actually be an accurate reflection of pavement conditions, such as a transverse crack between sensors leading to a nondecreasing deflection warning. This is another example of why the LTPP program stresses the importance of using trained and experienced field personnel.

Quality Control Tools to Check Equipment and Data

Once data collection has been completed, field personnel are required to review the collected data prior to leaving the site. In the case of longitudinal profile data, for example, profiler operators use the LTPP ProQual software (discussed in chapter 6) to evaluate the acceptability of the profile runs based on established criteria. ProQual uses collected profile data to compute IRI values for the left and right wheelpaths, as well as the average IRI of the two wheelpaths (figure 9.4).

Profiler operators are also required to use ProQual to perform a visual comparison of profile data collected by the left, right, and center sensors for one profile run. As a further check on the data, the operator compares the current profile data with those obtained during the previous site visit. If there are discrepancies between profiles from the current and previous site visits, the operator is required to investigate whether these differences were caused by equipment problems or were due to physical pavement features such as those caused by maintenance or rehabilitation activities.

FIGURE 9.4. ProQual software, used to evaluate the acceptability of profile data.

After the profile visual comparison is completed, an IRI comparison of current versus previous site visit data is also performed. If IRI values from the most recent profiler runs meet the LTPP criteria and the operator finds no other indication of errors, no further testing is needed at that site.

Other examples of automated field data collection QC tools used within the LTPP program include the following:

• AWSSCAN—A program developed to permit field checks on the integrity and quality of the automated weather station data collected as part of the Specific Pavement Study (SPS) -1 (structural factors for flexible pavements), -2 (structural factors for rigid pavements), and -8 (environmental effects in the absence of heavy loads) experiments.
  
  
• ONSFIELD and MOBFIELD—Programs developed to permit field checks on the integrity and quality of the onsite and mobile data collected as part of the LTPP Seasonal Monitoring Program. They were used in the field to identify data anomalies whenever onsite or mobile seasonal data were collected.

There are also instances within the LTPP program where data collection is not automated, as is the case with manually collected distress data. In those instances, efforts were made to develop and implement procedures that ensure consistent and uniform data. In the case of the manual distress surveys, for example, the regional support contractors were required to run the DiVA (Distress Viewer and Analysis) software on each section’s data after the survey as part of the office review to identify inconsistencies among previous surveys. (The DiVA software is described in greater detail later in this chapter.) Furthermore, regional support contractor staff in the field are required to comply with the following policy:

• Raters assigned to perform a manual survey on an LTPP test section must obtain a copy of the distress map made during the prior survey. (This map would have undergone office review and had any discrepancies and questions resolved before it was made available.)
  
  
• The prior survey map is to be used by the rater to (1) become familiar with the types, severity, and quantities of distresses; (2) determine the survey limits; and (3) determine the wheelpath locations.
  
  
• The rater is then to map the test section independently: a new map, not an edited version of the prior map, is to be drawn.

The intent of this policy is to promote consistency, not to perpetuate errors. The rater is required to note disagreements with the prior map information and bring any issues to the attention of the office reviewer for resolution. In addition, inconsistencies with prior surveys are required to be documented by the rater using video or photographs.

Quality Control Processes After Data Collection

After data have been collected, the regional support contractors carry out numerous quality checks before entering the data into the database. These checks range from basic—are profile data files readable and complete, is the manual distress data package complete, are survey date stamps correct—to sophisticated, as explained below. The primary objective of these data checks is to prevent “bad” (i.e., erroneous) data from being entered into the database. Examples of the data check tools developed by the LTPP program to avoid loading bad data into the LTPP database include:

• AWSCHECK—Used for automated weather station data. In addition to range and integrity checks, the program plotted the climate data, allowing for time-history consistency checks to be performed. This preprocessor program allowed deletion of bad measurement data and adjustments to data fields, corrected time stamps for daylight saving time, and prepared the data file for loading into the database.
  
  
• FWDCHECK—Used until about 1996 to analyze deflection data to determine test section homogeneity, the degree to which test pit data are representative of the section, the presence of data outliers within the section, and overall reasonableness from a structural capacity viewpoint.
  
  
• FWDSCAN—Scans electronic FWD data files to identify data collection rule violation issues, data file format integrity, and range values violations.
  
  
• P46CHECK—A preprocessor program that automates checks on the results of laboratory resilient modulus tests on unbound materials. In addition to all routine checks on data values, advanced statistically based checks are performed.
  
  
• P07CHECK—A preprocessor program and a computed parameter generator. While the primary function of this preprocessor program is to check the integrity of the results of resilient modulus measurements in indirect tension on asphalt concrete (AC) cores, it also uses an algorithm to calculate the test results from the raw measurement data to store in the database.
  
  
• ProQual—Both a pre-upload data check processor program and a computed parameter generator (like P07CHECK). ProQual processes, evaluates, and generates computed parameters for both longitudinal and transverse profile measurements. The output files from this program are used as input files to load data into the database.
  
  
• SMPCHECK—A preprocessor program allowing deletion of bad data and time-based adjustments. In addition to automated range and integrity checks on the data, time-history plots of this temporal data are also used to identify data inconsistencies. Output files created by this program were used as input files to the database.
  
  
• LTAS—LTPP Traffic Analysis Software, providing multiple functions (see sidebar).

Electronic data obtained from third parties are also evaluated and subjected to automated data checks before entry into the LTPP database. Two of the largest modules of data from other noncontract agencies are traffic monitoring data and climate data. The following explains the data checks that have been used by the LTPP program for these data:

• Traffic monitoring data are supplied by participating highway agencies in the standard data submittal formats described in FHWA’s Traffic Monitoring Guide,²¹ shown in figure 9.5. Screening and processing of the traffic data submitted by the highway agencies are described in the LTAS sidebar.
  
  
• Until 2013, climate data were obtained from the National Climatic Data Center and Environment Canada. Due to their large volume, these raw data were loaded into an independent set of database tables. Automated checks on completeness, range, and logical statistics were performed on the data, and flags were set in the records to store the results. Records not passing the LTPP QC checks were excluded from use in the analysis process that creates temporal summary climate statistics for each LTPP test site. This summary information was then loaded into the database.

FIGURE 9.5. The Traffic Monitoring Guide, published by FHWA’s Office of Highway Policy to assist States with collecting and submitting traffic data in a uniform format to FHWA.

Database Checks and Reviews

As data are entered into the database by the regional support contractor, they undergo a hierarchy of automated, progressive QC checks. The results of these checks are recorded in the record status field. Every data record in the database starts with a record status of A. When a data record with a status of A passes all level B checks, it is assigned a status of B, and so on. (Chapter 8 provides more detail about the different quality checks performed on the data.) Over time most of the level-B checks were removed, which left three major types of database QC checks:

• Level C checks—Determine if all required fields within a specific table have been populated. In some cases, level C checks were supplanted by non-null restrictions placed on critical fields during the table design that prevented a record from being created if a value for that field was not entered.
  
  
• Level D checks—Determine if the values entered in a field are valid and reasonable. For example, the range check for deflection data from the center sensor on an FWD is 5 to 2032 µm (80 mils).
  
  
• Level E checks—Determine if the value in one field of a table is comparable to the value in another field that may or may not be in the same table (commonly referred to as a “relational check”). For example, a level E check is used to see if pavement layer temperature gradient data exist for each FWD data set.

The QC level checks are performed sequentially. Level D checks are applied only to records passing level C checks, and level E checks are applied only to records passing level D checks. Record statuses of A and B are used for data that either have not undergone QC check processing or have not passed the level C checks. If a record fails a check, its record status remains at the next lower status. For example, records failing a level D check have a status of C. Alternatively, the record status can be manually upgraded if the record has been examined and has been found to be acceptable.

Records with level E status could mean either of the following:

• Records have passed all of the data checks.
  
  
• Records may have failed some data checks but were later manually upgraded after inspection.

In either case, level E records may contain errors that have not been detected by the data review process.

On the other hand, records with a non-level E status could be interpreted as any of the following:

• Records have not completed the QC process.
  
  
• Records have completed the QC process, but have been left at a lower level because they contain an anomaly.
  
  
• Records have not been subjected to the QC process by policy.

It is important to note that data users assume responsibility for conclusions based on their interpretation of the data collected by the LTPP program. Level E data should not be considered more reliable than non-level E data. Likewise, non-level E data should not be considered less reliable than level E data. The record status for non-level E data can be used as a relative indicator of potential issues or pavement anomalies that might exist for these data.

It was not practical, particularly when considering the wide variety of materials, climatic conditions, and local practices, to inspect all of the data for all types of potential anomalies. As the program evolved and improvements were made to the data QC checks, level E data included in previous releases may have been reclassified. It is also important to note that the LTPP program works diligently to identify and address critical data errors.

Likewise, it is important to recognize that the program has established standard and formal methods for the correction of data errors identified from the database checks. It is the LTPP program’s policy to not load known “bad” data into the database, which is why the automated data check processors and screening methods were developed. Although steps are taken to prevent entry of erroneous data, with such vast quantities of data coming from multiple sources, there are instances when data in the database are still found to contain typographical errors.

When data errors are identified, the standard response is to correct the error if possible, to remove the data from the database, or to place the data at a lower data quality status. Error correction procedures are transmitted by LTPP program directives such as “Manual Upgrades to QC Checks,” which describes the steps to take when data fail an automated check.²²

During the QC error resolution process, errors that are not possible to rectify are also identified. Some examples include:

• Equipment measurement errors. When a record failing a QC check can be traced to an identifiable equipment measurement error, manual upgrades cannot be employed to elevate a data element to a higher status.
  
  
• Required data not available. Circumstances can develop where critically required data are not available. There are instances when a required data element was not collected, was collected improperly, or is no longer possible to obtain or measure (e.g., thickness of an AC layer that was subsequently milled). In these exceptional circumstances, a test section may be removed from the LTPP study.
  
  
• Issues requiring investigation. When new tables are added to the database or new QC programs are issued, some records failing a QC check require further investigation to determine the cause. There are instances when it cannot be immediately determined if the error was a result of equipment malfunction, abnormal phenomenon, or program error. Some of these problems are resolved through the Software Performance Report (SPR) process, which is discussed later in this chapter.

The above database checks and review guidelines apply to data after entry into the LTPP database. Error correction guidelines are also contained in the field data collection and processing documents.

After the data have been entered into the LTPP database and have undergone checks by the regional support contractors, the technical support services contractor’s staff reviews the contents of the entire database before releasing data to potential users. See chapter 8 for illustrations of the data flow and QC checks at different points in time. Data problems identified following release are documented using the Data Analysis/Operations Feedback Report (DAOFR) process, which is discussed later in this chapter.

LTPP Quality Assurance Efforts

There are two types of QA procedures the LTPP program follows to maintain a high quality level in the database. These efforts, performed by the program staff, the technical support services contractor, and the regional support contractors, are discussed below.

LTPP Quality Assurance Audits

Since the early days, the LTPP program has performed QA audits of data collection contractor activities, including those of the regional support contractors, materials testing laboratories, central pavement distress data contractor, and FWD calibration centers. However, those audits were not formalized until 1998 through “LTPP Regional Operations Quality Assurance Reviews,” a program directive that enhanced and further standardized the data collection process used by the regional contractors.²³ This directive required, on a routine basis, the conduct of QA reviews of the regional data collection contractors in the areas of automated weather stations, distress, FWD, profile, and the Seasonal Monitoring Program.

After implementation of the directive, multiple QA reviews of regional support operations were conducted each year by teams consisting of four highly qualified individuals, typically including one from the LTPP program, two from the technical support services contractor, and one from a regional support contractor other than the one under review. These teams reviewed field and office operations during the course of their visits, sometimes together and at other times independently. Members of the review team were not allowed to interact with data collection or office personnel during the conduct of their activities; the reviewers could only observe and take notes.

Regional contractors were rated by the review team as being “Fully Compliant,” “Compliant—Minor Changes Required,” or “Non-Compliant—Major Changes Required,” depending on how closely LTPP guidelines and directives were followed. Individual personnel were also given a “Compliant” or “Non-Compliant” rating on the basis of the review results. In the event that one or more individuals were found to be “Non-Compliant,” the contractor was required to submit a remediation plan to the LTPP program office within 15 days of receiving such a rating. A follow-up review of “Non-Compliant” personnel was conducted within 6 months after the initial review and if at that time the individuals were still rated “Non-Compliant,” they were not allowed to collect data for the LTPP program until they were found to be compliant at a later review. Beginning in December 1998, only contractor personnel rated “Compliant” were permitted to collect LTPP data.

QA audits were also performed on FWD reference calibration facilities used by the program and operated under cooperative agreements with selected highway agencies. The facilities used LTPP-provided equipment and followed LTPP test protocols. Annual audits were performed in conformance to test protocols, audit results were documented, and certificates of compliance were issued.

QA audits for materials testing were performed independently by the American Association of State Highway and Transportation Officials (AASHTO). The LTPP program required that the materials testing contractors be certified by AASHTO standards.

In 2001, LTPP’s approach to QA audits was significantly revised with the re-letting of the four regional LTPP data collection contracts. As part of those contracts, FHWA required the regions to formally document their data collection and processing QC systems in accordance with ISO (International Organization for Standardization) 9001 quality management systems accreditation standards. The documents were referred to as “Regional Operations Quality Control and Data Flow Plans,” as described in the following section. Some of the relevant features of this management process include:

• Designation of a Regional Data QC Manager on each regional contractor’s staff, with the following responsibilities:

– Conducting both regularly scheduled and impromptu internal audits of compliance with QC, data collection and data processing guidelines, and procedures.

– Documenting internal audits.

– Documenting corrective actions resulting from internal and external audit findings.

– Conducting annual, or more frequent, reviews and updates of the QC and management procedures.

• The technical support services contractor established a QA audit team and process to assess compliance by the regional support contractors with their data QC and management guidelines and compliance with LTPP program requirements. The QA audits include the following:

– Announced field or office visits to review designated sections of the QC and management plans. The different parts of the plans are rotated so that each part is reviewed on a 2-year cycle, budget permitting. Prior to an audit, a data review is conducted to identify data issues of concern to be investigated during the audit.

– Unannounced audits of field data collection or office personnel. Auditors arrive unannounced to observe activities and compliance with both the contractor’s internal requirements and the LTPP program’s requirements.

– A documented audit report that includes a description of audit activities, items reviewed, positive findings, corrective action requests, and improvement recommendations. All corrective action requests and improvement recommendations are discussed with the regional support contractor in order to reach an agreement on corrective actions to be taken. On each audit visit, all corrective action findings and improvement requests previously agreed to are reviewed.

Regional Quality Control and Data Flow Management Plans

Each region developed a formal Regional Operations Quality Control and Data Flow Plan circa 2001. These QC plans specified data collection, processing activities, data flow, and roles, responsibilities, and qualifications of staff members by position. QC activities were put in place to provide formal checks to identify bad, erroneous, and missing data prior to entry into the database. The regional contractors submitted QC and data flow plans for the following data elements:

• Deflection measurements.
  
  
• Longitudinal profile measurements.
  
  
• Distress surveys.
  
  
• Seasonal Monitoring Program data.
  
  
• Automated weather station data.
  
  
• Traffic data.
  
  
• Maintenance and rehabilitation data.
  
  
• Materials sampling and testing.
  
  
• Information Management System.

Plans varied among the regions; for example, the North Atlantic Regional Support Contractor developed the first LTPP QC plan for the automated weather station. Initially this region prepared a separate document for each data element, but later combined these into one document. In contrast, the Western Regional Support Contractor prepared a single document covering the QC processes for all of the data elements they collected and processed.

In 2004, the regional support contractors developed Working Guides as a means of standardizing regional procedures and ensuring consistency in training, field data collection, maintenance, calibration, and the processing and uploading of data. The purpose of these guides was to ensure compliance with the Data Quality Act.²⁴ In the North Atlantic Region, individual Working Guides were issued for each data element (like the initial QC and Data Flow Plans).

Higher Order Data Checks

Although the QC processes discussed in the previous sections of this chapter can identify a wide array of data issues or problems, they cannot identify all of the potential data problems. Once these initial QC efforts have been completed, higher order data checks and reviews are used to identify other types of data issues. Important higher order review activities within the program include time-series checks, data studies, and engineering data analysis techniques. The term “data study” was introduced into the LTPP lexicon to indicate intensive higher order data reviews that employ a variety of analysis techniques performed against the entire LTPP data set and data subsets. While not in explicit alignment, these data studies tend to serve the same purpose as the level 2 QC checks described in the Strategic Highway Research Program (SHRP) 5-year report on the LTPP Information Management System (IMS).²⁵ The final tier of higher order data checks comes from the results of formal analysis of LTPP data.

Time-Series Checks

An important higher order review activity performed within the LTPP program is the time-series review of pavement section performance data. In the case of distress data, for example, once data have been uploaded to the database, time-series checks are performed using the DiVA program. This software automates extraction, compilation, and time-series review of distress data from the database. The key subset of distresses included in the time-series review by pavement type is summarized below by pavement surface type:

• Asphalt-concrete-surfaced pavements—fatigue crack-ing, longitudinal cracking, transverse cracking, patch/patch deterioration, and block cracking.
  
  
• Jointed concrete pavements—corner breaks, longitudinal cracking, transverse cracking, and patch/patch deterioration.
  
  
• Continuously reinforced concrete pavements—longitudinal cracking, transverse cracking, patch/patch deterioration, and punchouts.

The DiVA software produces a graph of the selected distress versus time for each section and distress type, starting from the time the test section was included in the LTPP program. An example is shown in figure 9.6. The graph includes both photographic and manual distress data for each test section along with error bands around the individual data points and a trend (best-fit) line to aid in the review. The variability bands are applied to the total quantity of distress—the sum of quantities over all severity levels. The upper and lower limits of each distress represent three standard deviations. Ninety-nine percent of the data from a given data set are expected to fall within three standard deviations of the mean of that data set.

Actions resulting from the time-series review vary from data being acceptable (most common outcome), to identification of issues that may be caused by correctable items such as unreported maintenance or rehabilitation activities on the test section (most common issue), to correction of individual distress data classification issues (infrequent occurrence).

FIGURE 9.6. A distress-versus-time graph produced by the DiVA software.

Data Studies

In addition to the time-series checks, another important higher order review activity within the program is the conduct of data studies using data extracted from the database. Data studies are also used to develop, refine, or implement new or advanced data collection systems. Some examples of data studies conducted by the LTPP program include:

• LTPP distress data variability study—Since the methods used to rate pavement distresses rely on subjective interpretation by trained personnel, a study was conducted in 1997 to examine variability between raters and between rating methods.²⁶ Based on a statistical sample of data obtained from nine distress rater accreditation workshops (and later updated and confirmed with data from an additional 13 distress rater accreditation workshops),²⁷ the study was conducted by engineering and statistical experts and peer-reviewed by the Transportation Research Board’s (TRB) ETG on Pavement Distress Monitoring. In addition to documenting probable ranges of uncertainty in distress measurements, the study provided recommendations on improving the rating methods that were implemented into the LTPP distress rater accreditation process, thereby reducing future distress data variability.
  
  
• LTPP distress data consolidation study—The objectives of this study were to produce a comprehensive consolidated distress data set and to reconcile differences between data collected using photographic and manual methodologies. This study led to the implementation of the distress data time-series checks using the DiVA software, which, in turn, led to the identification and correction of numerous discrepancies: 17 percent were attributed to human error, 6 percent to data collection methodology, 36 percent to the strategies used in the study, and 41 percent were unidentifiable.²⁸
  
  
• LTPP AC resilient modulus data study—A data study was performed on the resilient modulus test, developed by the LTPP program, for AC mixes in indirect tension. Due to the severity of the problems found with the data from these tests and the uncertainty associated with test results, which had originally appeared reasonable, significant amounts of data were archived, removed from the database, and not distributed to the public.²⁹
  
  
• LTPP field data collection equipment studies—As a part of the QC process, data studies are routinely performed on the LTPP FWDs and pavement profilers. These data studies typically consist of a statistically designed experiment that uses an analysis of variance approach to evaluate the results of side-by-side equipment comparisons. These procedures are also used to evaluate equipment during the procurement process.
  
  

FWD calibration center pooled-fund study—A State-sponsored pooled-fund study, managed by the LTPP program, was established to investigate and improve the LTPP-developed FWD calibration protocols. The purpose of the study was to evaluate current methods, procedures, and instrumentation and develop an improved system compatible with current computer technology.³⁰

Data Analyses

The last tier of LTPP higher order review activities is associated with the performance of formal data analysis projects. A multiproject approach, by topic area, is used for analysis of LTPP data. The LTPP program, other FHWA program offices, the National Cooperative Highway Research Program, and the States have sponsored many research analysis projects that have used LTPP data. More detailed information concerning LTPP data analysis efforts is provided in chapter 10.

LTPP-sponsored analysis projects are primarily intended to yield insights, benefits, and products from analysis of the data. A fundamental requirement of any research investigation is evaluation of the empirical data used in the investigation. In recognition that the operational structure of the LTPP program serves to provide relevant data of known quality for achievement of program goals through analysis of the collected data, the DAOFR process, discussed below, was established in 1998.³¹ Over time, the DAOFR process has been used to report suspect data issues discovered in data analyses and studies by users both within and outside the LTPP program.

Peer Review

An integral component of LTPP’s data quality approach is extensive peer review, both external and internal, of the various program areas and activities. At the start of the program, an external peer review process was created to monitor the status and progress of the LTPP studies and provide technical assistance to SHRP and later to FHWA, as discussed in chapter 2 and appendix A. The TRB LTPP Committee and its ETG structure incorporated experts and practitioners from the highway pavement community and provided advice regarding the technical and managerial direction of the LTPP program. National outreach stakeholder meetings such as the 1990 SHRP Assessment Meeting in Denver, Colorado; 1996 SHRP LTPP National Meeting in Irvine, California; 2000 LTPP SPS Workshop in Newport, Rhode Island; and 2010 LTPP Pavement Analysis Forum in Irvine, California, provided additional external peer review input on LTPP program activities.

Internal peer review is also a major contributor to high-quality data and the program’s success. Internal peer review touches on every element of the program, including the various QC/QA activities described in this chapter.

A few highlights of LTPP peer review activities, both external and internal, are summarized below:

• The LTPP experiment designs were developed and reviewed by statisticians with pavement research experience. These designs were also reviewed by an ETG consisting of statisticians and pavement research engineers.
  
  
• The LTPP data requirements were developed by experts in each of the engineering disciplines associated with the program. These requirements were reviewed and critiqued by other stakeholders through TRB-facilitated national meetings. This process continues.
  
  
• The LTPP data collection guidelines are developed by experts in each of the engineering disciplines associated with the program. These guidelines are reviewed and critiqued by the appropriate TRB LTPP ETGs.
  
  
• Before the contents of the database are released, they are reviewed by experts and specialists on the technical support services contractor’s staff. By design, these review staff are not involved with data collection or analysis.
  
  
• The LTPP Strategic Plan for Data Analysis was developed and continues to be refined based on input from program staff, highway agency personnel, industry stakeholders, and academicians through an outreach process.³² The plan was developed in concert with the LTPP ETG on Data Analysis, and it is periodically updated using the ETG peer review process. Review of LTPP-sponsored data analyses and reports is based on a formal peer review process. Data analysis documents that contain estimates and projections are required to contain a description of the analysis methodology, and they receive both an internal and an independent external review by subject matter experts before publication.
  
  
• All published LTPP reports use the FHWA publication process, which includes editorial review for conformance with FHWA publication standards and provides a report documentation page. Since the introduction of section 508 of the Rehabilitation Act, LTPP publications have been formatted in conformance with the Act and reviewed by individuals trained in section 508 compliance standards.

Furthermore, to allow input from both external and internal sources, better serve its customers, and determine customer satisfaction, the LTPP program provided customer support services and customer survey processes starting in 1989. In 2006, FHWA moved the formal LTPP Customer Support Service Center to its Turner-Fairbank Highway Research Center, in McLean, VA.

Feedback Processes

Another important data quality element is the feedback process. Initial LTPP feedback efforts were mostly limited to national and regional meetings involving LTPP State Coordinators and regional support contractors. Teleconferences were also held with the data collection staff from the regional support contractor offices. These meetings and conferences were scheduled as needed and allowed problems, ideas, and solutions to be shared among all participants. Later, more formal procedures (OPRs, DAOFRs, and SPRs) were established to ensure that the feedback processes contribute toward the improvement of data quality and to the LTPP program’s success.

Operational Problem Reports

The first LTPP feedback mechanism, formally implemented by policy in 1995, was the OPR form, which enables personnel in the regions to report problem issues—equipment, software, and procedural problems associated with data collection, and, in particular, uncommon circumstances. This feedback mechanism provides a uniform way of reporting, handling, and tracking problems associated with the LTPP automated weather station, distress, FWD, profile, and seasonal monitoring data. Over time, LTPP program directives were issued extending and formalizing the OPR process to different data collection areas, such as materials sampling and testing.

A log of the OPRs is used to track the circumstances of the problems and their resolution. OPRs are reviewed, analyzed, and discussed by the LTPP program and contractor staff during teleconferences, and resolutions are provided by the regional support or technical support services contractors. Quarterly updates are sent to the program office and regions.

The OPR feedback process has resulted in several major benefits: standardizing the process for submitting problems associated with data collection activities; simplifying tracking when a problem was submitted, who was responsible for resolving it, whether or not it had been resolved, and how and when it was resolved; and reducing the probability of issues being forgotten or resolutions not making it back to the problem’s initiator. More than 400 OPRs had been submitted and resolved by 2014.

Data Analysis/Operations Feedback Reports

The second LTPP feedback mechanism formally instituted by policy (in 1998) was the DAOFR process,³³ which enabled users of the database to report issues encountered during data analysis that suggest or demonstrate the need for corrective actions or further investigation (figure 9.7). In an effort to examine all data in the database, the program developed a series of DAOFRs for use by the regional support contractors and data analysts to provide commentary and to load, edit, fix, clean, or remove any suspect data collected by the regions or received from the highway agencies. The approach to completing DAOFRs varies for each data collection element. For example, the Traffic DAOFR process started with the out-of-study sites (sites that were no longer being monitored) as well as the tables that had been completely populated (no new data were expected) or were part of the module closeouts. Data for active SPS and General Pavement Study sites were examined next.

The DAOFR form and DAOFR resolutions are available to LTPP data users online.³⁴ Situations in which use of the DAOFR process is appropriate include the absence of critical data for specific test sections; data that appear to be incorrect, contradictory, or otherwise suspect; data not collected but necessary to fill voids identified in the analysis; and recommendations arising from the analysis regarding data collection procedures that can be improved. By 2014, close to 1,400 DAOFRs had been submitted.

FIGURE 9.7. LTPP Data Analysis/Operations Feedback Report, used to provide feedback to the program regarding data that require investigation or correction.

 

Software Performance Reports

An early program feedback effort was the implementation of the LTPP database SPRs, which the contractors used to report problems, comments, change requests, and document changes made to the database management software. Although suggestions for software diagnosis and correction are often provided via phone calls and emails, actual diagnosis (by re-creating the conditions and modifying the software) occurs via the SPR process. An SPR submitted on a specific version of software includes a concise description of the problem and relevant extractions of test section information.

The SPR process was put in place in the early 1990s, and an SPR form was adopted to report issues with the traffic software in 1996,^35,36 but the SPR process was not formally implemented by policy until 2005.^37,38 In 2011, the SPR form was Web-enabled.³⁹ In all, more than 4,300 pavement performance database and traffic SPRs have been submitted.

Conformance With Data Quality Systems and Standards

Another important approach to data quality that was implemented by the LTPP program in the early years was to adopt existing concepts in other large databases to maintain consistency across the database. Such adoptions include the following:

•   Federal Information Processing Standards for geographic codes.
  
  
•   National Transportation Thesaurus for keywords.
  
  
•   World Geodetic System 1984 for location coordinates.
  
  
•   AASHTO classification system for soils and unbound base material.
  
  
•   AASHTO materials testing standards.
  
  
•   American Society for Testing and Materials test standards.

Still another data quality approach implemented from 2006 to 2008 was the self-assessment of the program relative to its compliance with U.S. Department of Transportation Information Dissemination Quality Guidelines, and the resulting report, as discussed in chapter 8.^40,41

Summary

LTPP QC/QA practices have resulted in a high-quality pavement performance database. The program’s quality expectations have created an environment that promotes advancements in QC processes to help in understanding pavement performance. The program has developed an extensive and robust set of modern pavement engineering QC methods and tools, several of which have been adopted by the pavement engineering community. These and other products are discussed in the next chapter. The next section also includes chapters on lessons learned and plans for the future.

References

1. J. S. Miller and W. Y. Bellinger. Distress Identification Manual for the Long-Term Pavement Performance Program (5th rev. ed.) (FHWA-RD-13-092). Federal Highway Administration, McLean, VA, May 2014. http://www.fhwa.dot.gov/publications/research/infrastructure/pavements/ltpp/13092/13092.pdf.
   
   
2. R. W. Perera, S. D. Kohn, and G. R. Rada. Long-Term Pavement Performance Program Manual for Profile Measurements and Processing (FHWA-HRT-08-056). Federal Highway Administration, McLean, VA, November 2008. http://www.fhwa.dot.gov/publications/research/infrastructure/pavements/ltpp/08056/08056.pdf.
   
   
3. Directive P-49, “LTPP Manual for Profile Measurements and Processing, December 2013—Revised Appendix B.” Mar. 11, 2014. Federal Highway Administration, McLean, VA. http://ltpp.org.phtemp.com/dirs/PRF/P-49.pdf.
   
   
4. P. N. Schmalzer. Long-Term Pavement Performance Manual for Falling Weight Deflectometer Measurements, Version 4.1 (FHWA-HRT-06-132). Federal Highway Administration, McLean, VA, December 2006. http://www.fhwa.dot.gov/publications/research/infrastructure/pavements/ltpp/06132/06132.pdf.
   
   
5. Directive D-10, LTPP Distress Rater Qualifications,” Feb. 7, 1997. Federal Highway Administration, McLean, VA. http://ltpp.org.phtemp.com/dirs/DIS_01-49/D-10.pdf.
   
   
6. Directive D-46, “LTPP Distress Data Quality—Accreditation of Distress Raters,” Oct. 21, 2005. Federal Highway Administration, McLean, VA. http://ltpp.org.phtemp.com/dirs/DIS_01-49/D-46.pdf.
   
   
7. Directive D-10, “LTPP Distress Rater Qualifications,” Feb. 7, 1997. Federal Highway Administration, McLean, VA. http://ltpp.org.phtemp.com/dirs/DIS_01-49/D-10.pdf.
   
   
8. PCS Law. Accreditation for the Long-Term Pavement Performance Studies Pavement Distress Raters (SHRP-P-653). Strategic Highway Research Program, National Research Council, Washington, DC, 1993, p. 13. http://onlinepubs.trb.org/onlinepubs/shrp/SHRP-P-653.pdf.
   
   
9. AASHTO R32-03. “Calibrating the Load Cell and Deflection Sensors for a Falling Weight Deflectometer.” Standard Specifications for Transportation Materials and Methods of Sampling and Testing, Part 1B: Specifications, American Association of State Highway and Transportation Officials, Washington, DC, 2006.
   
   
10. AASHTO R 32-09. “Calibrating the Load Cell and Deflection Sensors for a Falling Weight Deflectometer.” Standard Specifications for Transportation Materials and Methods of Sampling and Testing, Part 1B: Specifications, American Association of State Highway and Transportation Officials, Washington, DC, 2010.
    
    
11. “FWD Calibration Improvements and Plans for the Future.” LTPP Newsletter, vol. 5, no. 4, Fall 2009, pp. 1–3. http://www.fhwa.dot.gov/publications/ltppnews/LTPPFall2009.pdf.
    
    
12. S. Slavi, T. Merport, T. Wilson, J. Groeger, and A. López. LTPP Materials Characterization Program: Resilient Modulus of Unbound Materials (LTPP Protocol P46) Laboratory Startup and Quality Control Procedures (FHWA-RD-96-176). Federal Highway Administration, McLean, VA, January 1997. http://www.fhwa.dot.gov/publications/research/infrastructure/pavements/ltpp/96176/96176.pdf.
    
    
13. J. L. Groeger and A. Bro. Long-Term Pavement Performance Materials Characterization Program: Verification of Dynamic Test Systems With an Emphasis on Resilient Modulus (FHWA-RD-02-034). Federal Highway Administration, McLean, VA, September 2005. http://www.fhwa.dot.gov/publications/research/infrastructure/pavements/ltpp/02034/02034.pdf.
    
    
14. “New Profilers Hit the Road Smoothly.” LTPP Newsletter, vol. 9, no. 2, Spring 2013, pp. 1–2. http://www.fhwa.dot.gov/publications/ltppnews/13074.pdf.
    
    
15. Soil and Material Engineers. 2007 Comparison Testing of LTPP Profilers Final Report. Federal Highway Administration, McLean, VA, October 2007. http://www.infopave.com/InfoPave_Repository/Reports/C20/C20_70/rodeo07.pdf.
    
    
16. R. W. Perera and G. E. Elkins. LTPP Manual for Profile Measurements and Processing. Federal Highway Administration, McLean, VA, December 2013. http://www.ltpp.org/Profiler_Manual_12-19-13.pdf.
    
    
17. Directive D-19, “Cessation of ‘Night Time’Manual Distress Surveys,” May 10, 2000. Federal Highway Administration, McLean, VA. http://ltpp.org.phtemp.com/dirs/DIS_01-49/D-19.pdf.
    
    
18. Directive FWD-5, “Special Provisions for Cold Weather FWD Testing,” Jan. 28, 1994. Federal Highway Administration, McLean, VA. http://ltpp.org.phtemp.com/dirs/FWD/FWD-05.pdf.
    
    
19. Directive FWD-14, “FWD Surface Temperature Measurements in Cold Weather (below 0°F),” Feb. 1, 1995. Federal Highway Administration, McLean, VA. http://ltpp.org.phtemp.com/dirs/FWD/FWD-14.pdf.
    
    
20. Directive SM-07, “Data Collection Guidelines Under Less Than Ideal Conditions,” Mar. 14, 1995. Federal Highway Administration, McLean, VA. http://ltpp.org.phtemp.com/dirs/SMP/SM-07.pdf.
    
    
21. Traffic Monitoring Guide. Federal Highway Administration, Office of Highway Policy Information, Washington, DC, September 2013, ch. 7. http://www.fhwa.dot.gov/policyinformation/tmguide/tmg_fhwa_pl_13_015.pdf.
    
    
22. Directive I-85, “Guidelines for Manual Upgrades to LTPP IMS Quality Control Checks,” Mar. 24, 2003. Federal Highway Administration, McLean, VA. http://ltpp.org.phtemp.com/dirs/IMS_050-099/I-85.pdf.
    
    
23. Directive GO-6, “LTPP Regional Operations Quality Assurance Reviews,” Apr. 1, 1998. Federal Highway Administration, McLean, VA. http://ltpp.org.phtemp.com/dirs/GO_01-49/GO-06.pdf.
    
    
24. U.S. Statutes at Large 114 (2000): 2763A-153. Treasury and General Government Appropriations Act for Fiscal Year 2001. http://www.gpo.gov/fdsys/pkg/PLAW-106publ554/pdf/PLAW-106publ554.pdf.
    
    
25. W.O. Hadley and C. Copeland. Long Term Pavement Performance Information Management System: Five-Year Report (SHRP-P-679). Strategic Highway Research Program, Washington, DC, March 1994. http://onlinepubs.trb.org/onlinepubs/shrp/SHRP-P-679.pdf.
    
    
26. G. R. Rada, C. L. Wu, R. K. Bhandari, A. R. Shekharan, G. E. Elkins, and J. S. Miller. Study of LTPP Distress Data Variability, Volume I (FHWA-RD-99-074). Federal Highway Administration, McLean, VA, September 1999. http://www.fhwa.dot.gov/publications/research/infrastructure/pavements/ltpp/99074/99074.pdf.
    
    
27. G. R. Rada, A. L. Simpson, and G. E. Elkins. “Updated LTPP Manual Distress Data Bias and Precision Estimates” (Report No. 07-0419), in TRB 86th Annual Meeting Compendium of Papers CD-ROM, Transportation Research Board, Washington, DC, 2007. http://pubsindex.trb.org/view.aspx?id=801092.
    
    
28. A. L. Simpson and J. F. Daleiden. Distress Data Consolidation, Final Report (FHWA-RD-01-143). Federal Highway Administration, McLean, VA, October 2003, p. 34. http://www.fhwa.dot.gov/publications/research/infrastructure/pavements/ltpp/reports/01143/01143.pdf.
    
    
29. A. Yau and H. L. Von Quintus. Study of LTPP Laboratory Resilient Modulus Test Data and Response Characteristics: Final Report (FHWA-RD-02-051). Federal Highway Administration, McLean, VA, October 2002. http://www.fhwa.dot.gov/publications/research/infrastructure/pavements/ltpp/reports/02051/02051.pdf.
    
    
30. L. H. Irwin, D. P. Orr, and D. Atkins. FWD Calibration Center and Operational Improvements: Redevelopment of the Calibration Protocol and Equipment (FHWA-HRT-07-040). Federal Highway Administration, McLean, VA, October 2011. http://www.fhwa.dot.gov/publications/research/infrastructure/pavements/ltpp/07040/07040.pdf.
    
    
31. Directive GO-15, “Data Analysis/Operations Feedback Process,” Nov. 9, 1998. Federal Highway Administration, McLean, VA. http://ltpp.org.phtemp.com/dirs/GO_01-49/GO-15.pdf.
    
    
32. “Long-Term Pavement Performance Data Analysis Program: Strategic Plan Objectives, Analysis Outcomes, and Supporting Projects.” Dec. 11, 2013. Web page. http://www.fhwa.dot.gov/research/tfhrc/programs/infrastructure/pavements/ltpp/pdf/tablecloth_complete.pdf.
    
    
33. Directive GO-15, “Data Analysis/Operations Feedback Process,” Nov. 9, 1998. Federal Highway Administration, McLean, VA. http://ltpp.org.phtemp.com/dirs/GO_01-49/GO-15.pdf.
    
    
34. “LTPP Data Analysis/Operations Feedback Report.” http://www.fhwa.dot.gov/research/tfhrc/programs/infrastructure/pavements/ltpp/getdata.cfm.
    
    
35. Directive TDP-05, “Software Performance Report (SPR) Form,” May 28, 1996. Federal Highway Administration, McLean, VA. http://ltpp.org.phtemp.com/dirs/TDP_01-49/TDP-05.pdf.
    
    
36. Directive TDP-40, “Traffic Data Processing Software Performance Reports,” June 10, 2005. http://ltpp.org.phtemp.com/dirs/TDP_01-49/TDP-40.pdf.
    
    
37. Directive I-123, “Pavement Performance Database Software Reports,” June 10, 2005. Federal Highway Administration, McLean, VA. http://ltpp.org.phtemp.com/dirs/IMS_100-149/I-123.pdf.
    
    
38. Directive I-152, “Software Performance Reports,” Aug. 15, 2011. Federal Highway Administration, McLean, VA. http://ltpp.org.phtemp.com/dirs/IMS_150-199/I-152.pdf.
    
    
39. Directive I-155, “Software Performance Reports,” Dec. 19, 2011. Federal Highway Administration, McLean, VA. http://ltpp.org.phtemp.com/dirs/IMS_150-199/I-155.pdf.
    
    
40. DOT Report for Implementing OMB’s Information Dissemination Quality Guidelines. U.S. Department of Transportation, Research and Innovative Technology Administration, Bureau of Transportation Statistics, Washington, DC, August 2002. http://www.rita.dot.gov/bts/sites/rita.dot.gov.bts/files/subject_areas/statistical_policy_and_research/data_quality_ guidelines/index.html.
    
    
41. G. E. Elkins. Long-Term Pavement Performance Compliance With Department of Transportation Information Dissemination Quality Guidelines (FHWA-HRT-08-065). Federal Highway Administration, McLean, VA, November 2008. http://www.fhwa.dot.gov/publications/research/infrastructure/pavements/ltpp/08065/08065.pdf.