• Federal Highway Administration >
• Publications >
• Research Publications >
• 11035 >
• 005.Cfm >
• Pedestrian and Bicyclist Traffic Control Device Evaluation Methods

Pedestrian and Bicyclist Traffic Control Device Evaluation Methods

CHAPTER 4. CONDUCTING THE EVALUATION

DATA COLLECTION PLAN

A detailed data collection plan highlighting data needs, procedures, and schedules is critical to a successful study.

The selected MOEs determine the types of data needed. Chapter 3 of this report provides an overview of different methods for evaluating a treatment. Having established a MOE of interest, the data collection plan should describe the exact type of data needed and the locations where the data should be collected (see figure 15). For example, if the traffic control device is to improve pedestrian compliance, then the action of each pedestrian on each approach leg treated needs to be considered. If the evaluation is to also consider whether the pedestrian is crossing within a crosswalk or prior to the crosswalk, then the data collection plan also needs to indicate whether the pedestrian waited for the appropriate signal and where the pedestrian crossed (e.g., within the crosswalk, 25 ft prior to the crosswalk, 50 ft prior to the crosswalk, etc.).

© Texas Transportation Institute

Figure 15. Photo. Plans specifying locations where data will be collected.

Developing a standardized worksheet for the data collectors will help provide consistency in the data collection and data reduction process. The data collection plan should include the proposed worksheets.

For some evaluations, videotaping the area of interest helps reduce the data after the events (see figure 16). If the data collection is to include videotaping of the site, the data collection plan needs to clearly indicate the field of view needed for each camera.

© Texas Transportation Institute

Figure 16. Photo. Videotaping an area of interest.

To ensure usability, especially if more than one group will be participating in data collection, the draft data collection plan for videotaping should be provided to someone who is not familiar with the study for testing. If agency policy prohibits videotaping, live scoring of surveillance cameras without archival storage of the video can be used instead of replaying an episode to rescore it.⁽⁵⁾

The Guide to Good Statistical Practice in the Transportation Field provides the following advice regarding data collection:⁽⁶⁾

• The design of the data collection plan is one of the most critical phases in developing a data system. The accuracy of the data and the estimates derived from the data are heavily dependent on the data collection design.

• Accuracy is dependent on proper sample design, making use of sampling complexity to minimize variance. The data collection process itself will also determine the accuracy and completeness of the raw data.

• Data collection from 100 percent of the target group is usually the most accurate approach but is not always feasible due to cost, time, and other resource restrictions. It is also often more accurate than the data requirement demand and can be a waste of resources.

• A probability sample is an efficient way to automatically select a data source representative of the target group with the accuracy determined by the size of the sample.

• For most statistical situations, it is usually important to be able to estimate the variance along with the mean or total.

• Sample design should be based on established sampling theory, making use of multistaging, stratification, and clustering to enhance efficiency and accuracy.

SAMPLE SIZE

To satisfy statistical requirements, it is necessary that a sufficient sample size be obtained. The suggested minimum for some typical measurements is shown in table 6.⁽⁷⁾ However, a qualified statistician should be consulted to determine the appropriate sample size for the study. Many research institutes, universities, and statistical consulting firms offer statistical consulting services. These professionals should be consulted before the study is conducted to ensure that the appropriate experimental control and sample size are present for a valid statistical analysis.

Table 6. Suggested minimum sample sizes.⁽⁷⁾

If data are collected in the same manner within different evaluations, it is possible to compare or merge results across regions. The ability to document traffic control devices resulting in improvements in more than one area provides added justification for including the device in future editions of MUTCD.

DATA COLLECTION PERIODS

The time and date of data collection should also be included in the data collection plan. For example, the following may be needed:

• Time of day (e.g., morning or afternoon, or peak versus nonpeak period).

• Day of the week (e.g., some trails and nearby intersections have more activity on Saturdays and Sundays than during the week).

• Time of the year (e.g., when school is in session).

For evaluations using a before-after design, data must be collected before and after the installation of the traffic control device. As a result, data collection must be planned before the countermeasure is installed. The data collection plan should indicate when the before and after data will be collected. For many pedestrian-related traffic control devices, the number of pedestrians at a site may change because of the introduction of a device that favors pedestrians. For those situations, it is critical that before data, including the number of pedestrians, are collected.

The periods of data collection should be the same for the before and after periods. The before and after data should be collected on the same day of the week, time of day, and season to avoid potential confounding from other variables (e.g., volume differences between day and night or between peak and nonpeak traffic). It is also important to allow a sufficient acclimation period after the installation of the countermeasure to allow for any novelty effects. An acclimation period of 1–2 months is usually sufficient for most countermeasures. In some cases, it may be necessary to observe a site immediately for any initial signs of road user confusion. This initial confusion may not be a good indicator of the long-term effects of the treatment.

STATISTICAL ANALYSES

Testing for statistical significance quantitatively determines the likelihood that an observed change was caused by the installation of the countermeasure rather than by chance. Note that testing for statistical significance is only part of the process. It does not demonstrate the practical importance of the difference (see the following section).

In addition, the cost of the device needs to be considered. For example, a new flashing beacon may have a statistically significant effect on traffic speed or on yielding; however, the cost of the beacon versus the benefits of the speed reduction or yielding must still be considered. Appendix B of this report includes additional information concerning statistical analysis.

PRACTICAL DIFFERENCE ANALYSES

In many situations, the difference between two sets of measures may be statistically significant but not of practical difference. The value of the practical difference is dependent on what is measured and how it is measured (technique and equipment). It could also be dependent on the sponsor (e.g., how to value a change with respect to quality of life or feelings of safety). At a minimum, the practical difference needs to be greater than the measuring error of the device used. For example, when speed is measured with tube counters, the accuracy is about ±4 or 5 percent. Because light detection and ranging (LIDAR) equipment is used in enforcement, accuracy is more critical. For LIDAR guns, the guideline is that measurements are accurate within 1 mi/h. Therefore, a difference of at least 1 mi/h (if LIDAR was used) or 4 mi/h (if a tube counter was used) is needed before a speed change should be termed practically different.

EVALUATING THE RESULTS

Cost-Benefit Analysis

A cost-benefit analysis is useful in determining if a particular treatment is cost effective. It is important to be able to justify the cost of a treatment, whether based on a reduction in delay, a reduction in crashes, improvements in services, or another MOE. The dollar amounts used to place a value on reductions in delay or crashes have changed over time. Table 7 lists 2001 crash costs included in the 2010 Highway Safety Manual taken from a 2005 FHWA report.^(8,9)

Table 7. 2001 crash cost.⁽⁸⁾

Another measure of benefit is delay savings. One method used to calculate delay savings, along with the methodology for raising the dollar values to reflect the current year's costs, is found in the Texas Transportation Institute's 2009 Annual Urban Mobility Report.⁽¹⁰⁾ Delay savings calculations require simulation or field study data. Appendix C of this report contains more information about calculating delay savings.

Regardless of the MOE, the perceived benefits of a countermeasure are estimated or forecasted and then compared to the cost of installing and maintaining the treatment. Typically, the expected benefit (savings) is divided by the cost and expressed as a ratio. If the result is better than 1:1, the treatment is deemed to be cost effective.

As with crash evaluations, it is important to consider the MOE when conducting a cost-benefit analysis. For example, the benefit realized by preventing one fatal or serious injury crash could be enough to justify paying for the treatment based on the simple ratio of benefit to cost. Therefore, care must be taken to not overemphasize the effect of one crash at a low-crash location. If a crash surrogate MOE is used, it is not practical to use a cost-benefit analysis.

Life Cycle Costs

Related to the discussion of cost-benefit analysis is the concept of evaluating life cycle costs of a treatment. Using this method, the entire cost of installing, operating, and maintaining a treatment is evaluated over its entire expected service life, which can be compared to the anticipated reduction in crashes, delay, or other cost that would be incurred if no changes were made. When using this method, it is critical that the practitioner has an accurate estimate of the service life of the treatment being considered and a reliable method of obtaining credible estimates of the benefits to be gained and the costs of maintenance over that period of time.