U.S. Department of Transportation
Federal Highway Administration
1200 New Jersey Avenue, SE
Washington, DC 20590
202-366-4000
Federal Highway Administration Research and Technology
Coordinating, Developing, and Delivering Highway Transportation Innovations
 |
| This report is an archived publication and may contain dated technical, contact, and link information |
|
Publication Number: N/A
Date: April 1997 |
|||
Guide for Developing a State Transportation Research ManualSECTION 10.0 PROJECT FINAL EVALUATIONProjects are evaluated throughout the course of the research effort, as described in Section 6.0. In addition, after the research is completed and project deliverables are implemented, a final project evaluation is performed. Final evaluations are performed to assess the value of the research to the Agency, to learn what was done well, and to determine improvements for future research efforts. Evaluations can provide benefits to the agency that may result in 1) cost savings, 2) improved processes or methods, 3) safety improvements, and 4) improved or increased information for management decision making and policy formulation. Cost savings and safety improvements can be quantified. Improved methods or information can be documented by qualitative means. COMMENTARY: It is important to clarify any misconceptions regarding the types of evaluation that should be performed on research projects. Section 10 shows that the final evaluation is conducted after the major experimental portion of the research project is complete, during implementation of the research results. Section 6.0, Conducting and Monitoring Research Projects, covers the ongoing project evaluation process. Evaluation is the last major procedure performed by the research unit staff. They continue involvement in technology transfer and implementation activities, as necessary. The project final evaluation process can vary in its details, depending on agency policy, time available, and the level of expertise of the staff. Keep the evaluation goals clear in order to assess value and to find ways to improve the research process. Have appropriate staff resources to conduct the evaluation. Furthermore, research unit staff must be trained to deal with the complex methodologies that may be required by the agency. 10.1.2.A Process Internal to Agency The research unit performs final project evaluations. Depending on the complexity of the project, each project is reviewed and research performance and research results are analyzed. Final evaluations are made approximately 6 months to 1 year after the major experimental portion of the research effort is completed. Implementation must be progressing, but may not necessarily be completed. Users of the research results, the researcher, and others involved in the research process are interviewed and surveyed. Standardized forms are used to record the data. The evaluation material is analyzed and reported to the Project Committee and the Research Advisory Committee. Project evaluation reports contain quantitative and qualitative analyses. Agency staff comment on final evaluations and recommend improvements in the innovation and its implementation process (if applicable). Committee members recommend technology transfer opportunities and further research on the same or related topics. The research unit applies these recommendations to future research, technology transfer and innovation, and program evaluation. If additional research is recommended, the research unit may identify the problem for the next annual program or incorporate it into the current year's program. COMMENTARY: Timing and reporting details should be added to the discussion; see the sections below. Specify when the evaluation should be completed, how long the reviewing bodies have to make comments and recommendations, and any specific policies that govern the effort. The research unit should standardize reporting to streamline the paperwork that is part of the evaluation process. Although the research unit is given the lead in the project's final evaluation, the Project Committee can be substituted. The following sections would be modified accordingly. Project evaluation is assigned to a research unit staff member who conducts the evaluation according to the prescribed methods. COMMENTARY: Methodology is prescribed in this section or in a subsection immediately following it. Included are as follows:
Selected agency personnel participate in project evaluations. They are individuals who influence implementation of the project results and recommend improvements in the research program. Members of the Research Management Committee and others in the Agency may be asked to participate in the evaluation process. Research staff members respond to Agency participants' specific suggestions and recommendations. COMMENTARY: The groups or agency personnel giving feedback on the evaluation should be specified. The documentation each group gets also should be detailed (the more senior management may get higher-level material, while project-level staff may have all support documentation). Deadlines for review should be specified. Treatment of comments and recommendations should be discussed, including specifics on how the research unit comes to closure on the feedback provided by others in the agency. The Project Committee is uniquely qualified to make recommendations and suggestions regarding the performance of the research and the relevancy of the results of the research effort. Project Committees regularly supply information for the final project evaluation, either in a final evaluation form to be filled out by each committee member or through a consensus recommendation by the committee formulated during one of its regularly scheduled meetings. COMMENTARY: Providing opportunity for the members of the Project Committee to comment and make recommendations about the project is most important. They are very familiar with the project and have a vested interest in the successful outcome of the research.The means by which the committee or committee members' comments and recommendations are solicited are at the discretion of the research unit. For instance, the final evaluation can be an excellent agenda item for the committee toward the end of the formal committee's functional responsibilities. 10.1.2.B Process External to Agency At the discretion of the research unit, the Project Committee, or the Research Advisory Committee, the project may be evaluated by others outside of the agency. The evaluation may be performed through contract, by organizations affected by the research, or by research peers in other agencies. External project reviews concentrate on research methodology validity, data collection and analysis, and research results. COMMENTARY: Project evaluation by others outside the agency may be used for particularly large, sensitive projects or for difficult research projects. Specify who or which group can request such an external evaluation. Details of this process could be included. 10.1.2.C Economic and Quantitative Process To measure research project performance and benefit, economic and other quantitative analyses of varying sophistication are performed according to project size and available data. These analyses include net savings of dollars, time and lives; cost-to-savings comparisons; cost-benefit; present value; and return on investment. Data must show the conditions within the agency before and after the implementation of the innovation. The research unit requests the following data from the operating unit:
COMMENTARY: Add any data element that may be used for analysis to the list contained in this section. Economic analyses are data driven. If the data do not exist, it is difficult to arrive at defensible statistics. The research unit manager must assess the data availability before committing to perform any analysis. Operations units do not ordinarily keep records of the information listed; unless arrangements can be made at the start of the research and implementation efforts for data collection and retention, other methods of evaluation must be used. There are two schools of thought regarding quantitative analyses for research activity benefits. One group argues there are too many suppositions regarding data and too many unknowns regarding benefits. This group holds that the costs generally are overstated and the benefits are understated. The second group maintains that benefits analyses can be performed and that benefit/cost and net present value determinations are acceptable methods. While complex methodologies may not be appropriate for every project or every research unit, some basic comparisons to costs and anticipated benefits may be possible. If data are not available, qualitative analyses may be appropriate. The research unit determines whether a quantitative or qualitative analysis is used. Is there technical expertise in the research unit to do the analysis? Which methodology properly answers the questions asked regarding the value of the research project. Are the data available from the operating units? Which data are critical to measure? Articles on quantitative evaluations and benefit/cost calculations are included in the bibliography. C.1 Benefit/Cost Analysis Methodology The general types of data required for the benefit/cost analysis are listed in the preceding section. Topics requiring special consideration for each benefit/cost analysis are defined as follows:
COMMENTARY: The specific evaluation methodology used by the research unit or the agency should be inserted here. A discussion of the premises governing the methodology should be given as well as the data required for the analysis. The output of the methodology should also be discussed, as well as format, applications, data limitations, and similar details.Articles on benefit/cost calculations are included in the bibliography. In addition to determining a benefit-to-cost ratio of the results of the research, research benefits are quantified. This value is expressed as the net effectiveness of the research effort. Did the research save the agency money, time, or lives—and if so how much? Quantification of the benefits, less the costs, yields the net value. Appropriate inflation/discounts are factored into the calculations. Data required for the quantitative analysis are listed in Section 10.1.2.C above. COMMENTARY: The use of standard net present value methods are discussed in the Transportation Research Circular 426, "The Scientific Approach to Research," cited in the bibliography.The most important aspects are as follows:
Qualitative evaluation of the performance and the results of the research project provides valuable information. For research project results, qualitative evaluations focus on the following:
Qualitative evaluations are conducted through written or personal interviews that track research results incorporated into the agency's standard plans, specifications, practices, or procedures, and their effect. Qualitative evaluations are also conducted for the research project process, concentrating on the following:
In addition, the administrative processes used by the agency are reviewed to determine if improvements could be made for future research efforts. COMMENTARY: A survey or some other interview instrument is generally used to evaluate the quality of the overall research effort. A checklist can be included in this section showing the basis for a qualitative evaluation. A personal interview can be the best way to clarify ambiguities. Data from implementation evaluations may provide useful information. Some of the questions to be asked are as follows:
Other questions for user interviews include: "Did the research ..."
Qualitative evaluations are used if data are insufficient to perform quantitative evaluations. User input significantly increases the level of credibility of a qualitative evaluation. The research unit, the researcher, and any users involved with the research should be interviewed/surveyed to determine research performance as detailed in the narrative above. Other areas for evaluation should be described in this section. Consistency in data collection (e.g., interviewing) is critical when performing qualitative evaluations because the answers given are subjective. If qualitative evaluations are performed, the research unit should have a written protocol for interviewing/surveying the research project participants. The usefulness of research projects is objectively determined by final project evaluations. The final evaluation products are reports. Draft reports are circulated to project committee members, researchers, users, and any other appropriate individuals who may have input for evaluation documentation. Final evaluation reports are submitted to the Research Advisory Committee. They are used for the research unit program evaluation process. COMMENTARY: A general outline of a project final evaluation report can be included in this section. Reports should contain the following:
|
