Pedestrian and Bicyclist Traffic Control Device Evaluation Methods
CHAPTER 5. SUGGESTED SECTIONS FOR AN EVALUATION REPORT
INTRODUCTION
A report that records the details of an evaluation needs to describe the treatment, the issues that led to the evaluation (or the problem or question that the evaluation addressed), the evaluation procedure, and the conclusions and recommendations that resulted from the evaluation. This chapter contains an overview of suggested sections for an evaluation document. These sections include the following:
Front material. An introductory section or chapter. Background information on the study, problem statement, or issue to be resolved. A description of the countermeasure or treatment evaluated. A description of the evaluation process. A description of the findings and conclusions. A list of relevant recommendations, comments, and discussions. Any necessary appendices.
The following sections address components of each of these sections. Not all of the components of these sections may be required for every report. The descriptions can be used as guides on how to appropriately describe the tasks and efforts associated with an evaluation. The practitioner can use the components that are relevant, omit those that are unnecessary, and include others that may be needed based on the requirements of the evaluation, the sponsoring agency, and the nature of the project.
FRONT MATERIAL
The elements at the beginning of a document can be collectively called the front material. The necessary items form a function of the funding source for the evaluation. The order of the elements may also vary depending on the requirements of the sponsoring or performing agency. However, each is commonly found in a technical report and helps introduce the reader to the report. Some of these elements are described in further detail in the following sections. Front material includes items such as the following:
Front cover (outside and inside, as needed). Foreword (optional). Notices, disclaimers, or other official statements (as required by the sponsoring agency and/or the agency conducting the evaluation). Technical report documentation page (TRDP) (required for federally funded projects). Abstract or executive summary with conclusions (an abstract is a part of the TRDP). Common unit conversion factors (required for projects funded by certain agencies). List of commonly used abbreviations and symbols (required for projects funded by certain agencies; may be tailored to include certain abbreviations specific to the project and the report). Acknowledgments (optional). Table of contents. List of tables. List of figures.
Cover
How the report is being published, whether as a hardback, softback, or Web-only document, will help determine if outside and inside covers are necessary. The outside cover is generally more elaborate, using cover art or photos along with textual information (see figure 17). The inside cover page has no to few graphics and more text. In general, a cover has several required elements, including the following:
 |
| © Texas Transportation Institute |
Figure 17. Photo. Example of a report cover.
TRDP
The TRDP has a predetermined format in which the authors provide the required information. This page is required in reports on projects that are federally funded. Refer to the front of this document for an example of a completed TRDP.
Abstract or Executive Summary
An abstract or executive summary provides readers with a concise account of the problem, treatment, and evaluation as well as a summary of the conclusions. In many cases, the abstract is 250 words or less, but there is not always a defined limit. Executive summaries tend to be longer, sometimes a full page or more. The TRDP includes an abstract.
Table of Contents
The front material should contain a table of contents to show the document organization and to provide a reference for readers looking for specific information within the report.
List of Tables and List of Figures
A list of tables and a list of figures are supplemental to the table of contents and help readers find key illustrations of findings and conclusions.
INTRODUCTORY SECTION OR CHAPTER
The beginning of the body of the report should contain an introductory section or chapter. This introduction typically includes the following elements:
A basic description of the observed problem that necessitated the evaluation. A list of the objectives of the evaluation (based on MOEs, discussion of alternatives, or other metrics to determine if the evaluation was successful in accomplishing its purpose). A description of the organization of the document (commonly a list of the chapters
and appendices). Detailed background information related to the study (depending on the nature of the project and the length of the report).
BACKGROUND
If the nature of the project, the treatment, and the issue addressed in the evaluation can be summarized briefly, then a background section could be included in the introductory chapter. However, for a detailed literature review, summary of the current state of practice, or other expanded account of related information, it may be necessary to put the background material in its own chapter. Common elements of the background section include the following:
Purpose of the study/problem statement. Description of the sponsor of the project. Sponsor's experimentation number. Location where the study was conducted. Project objectives. Literature review (with references) or references to similar evaluations/reports. Comparison of the study to previous evaluations. Definitions of terms and acronyms (if needed). Identification of critical actions and appropriate MOEs (particularly with respect to pedestrian/bicycle treatments). Discussion of previous evaluations of similar devices (if available).
Possible questions that could be asked in developing the background include the following:
How does the treatment relate to MUTCD? Why was the tested device chosen? Why will existing methods, measures, and devices not work? How will the device improve operations, delay, safety, or cost (or another MOE)?
COUNTERMEASURE
After the need for the project has been established and the treatment(s) or countermeasure(s) has been introduced in the background section, a more detailed description is necessary to acquaint readers with its various features and attributes according to the following outline:
- Description of the device.
Dimensions. Type of retroreflective sheeting (if applicable). Timing of flashing if the device includes active flashing elements. Version number of any operating software.
Pictures (color is preferred over black and white), diagrams, and layouts (see figure 18). - Costs.
Capital costs. Installation costs. Maintenance costs. Life-cycle costs.
Practicality of maintenance. MOEs. Expected benefit. Target user group (e.g., blind, deaf, elderly, bicyclists, etc.).
 |
| © Texas Transportation Institute |
Figure 18. Photo. Example of test devices in the field.
EVALUATION PROCESS
The procedures and methods used in the evaluation must be clearly described in detail for two reasons: (1) to properly document all of the personnel, equipment, and tasks used to conduct the evaluation and (2) to allow others to reproduce the results when installing the treatments at
other locations.
Elements that can be used to describe the evaluation process include the following:
Problem identification (clearly state why the new traffic control device is needed). MOEs used in the study.
Measures must make sense with respect to the problem being addressed and the proposed traffic control device countermeasure. Action of the user of interest (during the time period of interest (e.g., night versus day)). Characteristics of subjects being observed.
Evaluation design used (e.g., before-after, cross sectional, within subjects, or between subjects). - Evaluation method used (e.g., field, lab, survey, etc.).
Findings.
Statistical analysis. Practical differences. Cost-benefit.
Discussions.
Confounding factors. Conflicting measures. Comprehension concerns. Limitations of study.
Recommendations.
FINDINGS AND RESULTS
After describing the evaluation process and analysis methodology, the findings and results must be clearly stated. Some examples of commonly used methods for presenting findings include
the following:
-
A table that summarizes the entire study. Table 1 in the executive summary of Safety Effects of Marked versus Unmarked Crosswalks at Uncontrolled Locations: Executive Summary and Recommended Guidelines is an example of condensing a study's findings into one table.(11)
A worksheet that presents a method for using the findings. Appendix A of the Transit Cooperative Research Program study on pedestrian treatments contains a worksheet that can be used to select a pedestrian treatment for a site.(12) A table that lists all countermeasures and shows the evaluator's opinion on the effectiveness of the countermeasures (e.g., check boxes under a range that goes from high to low).
A discussion of unique qualities of the study location(s) and whether the results are transferable to other locations. A graph that illustrates pedestrian delay for each countermeasure evaluated. A photograph of pedestrian using treatment device (or not using it, depending on the effectiveness of the device).
There are several principles that practitioners should apply when presenting results as follows:
Use the appropriate level of precision (e.g., report a 1.2 percent increase instead of a 1.207 percent increase). Include text to accompany any tables, figures, and photographs to describe the information contained in these graphical representations. Emphasize the reporting of results that are relevant to the MOEs defined in the
evaluation process. Other results may also be reported, but they should not overshadow those that relate to the MOEs and should help answer questions posed in the project's problem statement.
Report results objectively; do not allow a bias toward or against a particular treatment affect the interpretation of results or how they are reported to the reader. Discuss how results are similar to and different from other implementations (both within the United States and abroad).
CONCLUSIONS AND RECOMMENDATIONS
Based on the findings and results, the evaluator makes conclusions and recommendations for further action, revisions to existing policies or guidance documents, additional installations of the treatment, or discontinuation of the treatment. This may be included in the report as a part of the section on findings and results, or it may be its own section or chapter in the report. Some items to consider when developing conclusions and recommendations are as follows:
Conclusions.
Describe what the evaluators determined that the findings and results mean. Provide realistic and common-sense conclusions. Discuss personnel and costs.
What quantities of personnel and costs were used to do the study? How would someone else replicate the study, and what do they need?
Use clear language and confirm that conclusions do not contradict one another (do not confuse the conclusions).
Recommendations.
Applicable types of sites: Where would the treatment work? Any changes to the evaluation method that would improve the results of the treatment. Additional research needed to improve or refine the conclusions.
COMMENTS AND DISCUSSION
Following the listing of relevant conclusions and recommendations, it is often appropriate to document other comments or topics of discussion that arose during the study. Some items for comments and discussion include the following:
Areas for additional research including activities that would advance the knowledge of the profession. Aspects of the study that did not go as planned. Limitations of the study (e.g., where the results are applicable, transferable, etc). Applicability of results to other areas or, conversely, limitations of the device in given areas.
APPENDIX
In some reports, it is appropriate to include one or more appendices to provide additional information that benefits the readers but does not lend itself to inclusion in the main body of the report. Appendices often include large, multipage items that the report authors can refer to for a detailed description, leaving the most pertinent information in the main body. Some examples of items included in appendices are as follows:
Reproduction of the survey questionnaire and listing of answers. Items that would make the body of the report inordinately long. One possible report format is "short report with a big appendix." This entails moving multiple tables into
the appendices and only including key information in the main report. Draft language of revisions to guidelines, documents, or policies when appropriate. A more detailed tabulation and presentation of data.
|
