U.S. Department of Transportation
Federal Highway Administration
1200 New Jersey Avenue, SE
Washington, DC 20590
202-366-4000
Federal Highway Administration Research and Technology
Coordinating, Developing, and Delivering Highway Transportation Innovations
 |
| This report is an archived publication and may contain dated technical, contact, and link information |
|
Publication Number: N/A
|
|||
Guide for Developing a State Transportation Research ManualSECTION 11.0 PROGRAM EVALUATION11.1 Overall Program Performance Expenditure of public funds is subject to careful scrutiny. Because the profit motive does not exist in the public arena, public programs that receive funds must prove their value in other ways. After careful selection of projects from problem statements and development of the work program, the research effort follows scrupulously defined procedures that ensure unbiased, meaningful results. Projects are evaluated to assess the cumulative effect of the program as a whole. The overall performance of the research program is a summation of all of the components of the program. The success of the individual research projects, technology transfer effort, and implementation of research results are good indicators of overall program performance. The research unit keeps accurate records on all of its activities, costs, and accomplishments. Several parameters are used to evaluate overall program performance: success of implementation and technology transfer activities, milestones, funding and schedule adherence, benchmarking, peer exchange, and satisfaction of customer and management needs. COMMENTARY: This aspect of program evaluation involves considerable record-keeping.As with previous sections of the guide, the research managers are reminded that they should be selective in their choice of subsections and content. The section on implementation results is too important to the research program to omit and FHWA regulations require a tracking effort. The easiest method of tracking may be quarterly tabulations. Actual items to be tracked should be selectively chosen by the manager. Implementation steps should also be documented. Section 11.3 covers the success of implementation in the program. The section on Milestones is actually an activities list. Although not all of the suggestions may be necessary, they show the range of effort for the research unit. The data collected for the Funding and Schedule Adherence subsections are necessary to satisfy the requirements of the FHWA. The cyclical report (quarterly, semiannual, or annual) for each project can be the means of transmitting this information to the FHWA. Tracking program funds involves the accounting unit of the agency. The last section, Benchmarking, is an attempt to put generalized data for the research unit into a performance setting. This is not mandatory, but the research manager should consider some aspect of benchmarking. All suggestions may not be appropriate for an individual research unit. This section on overall program performance lists several project achievement areas of the program. To give the most accurate picture of the research unit's accomplishments, the items should be combined. Forms 7.2 A, B, and C can be used.11.1.2.A Implementation Results Implementation of individual projects is discussed in Section 9.0, Implementation Process. Combining the summaries of the individual projects documents the progress for the entire program. The summaries include the actions at the following times:
In addition to the tabulations listed, partial- or full-project implementations are documented. Although a project may have been completed, subsequent implementation achievements are documented as they occur. Form 7-2 A, in the Appendix, is used for this purpose. The work program provides a the summary of all research activities planned for the year: projects, technology transfer efforts, technical assistance, seminars, and implementation efforts. Milestones for these activities include the following:
Attainment of each of these milestones is documented. COMMENTARY: Achievement of milestones may also result in technical accomplishments, as defined under 7.1, Project Level Reporting. Repetition isn't required in this section. Each funding source in the work program has a fiscal year limitation and each activity has a specific budget. A record of both the funding source and budgeted project expenditures is kept. Allowances are made for overexpending on the individual SPR projects for the year, but the total program funds for SPR or other funding sources cannot be exceeded. Individual project overruns are documented. Projects are the most important scheduled activities. Most other activities can be planned throughout the year. Adherence to the project schedule is contingent on many factors, so the principal investigator and unit manager must be in frequent communication to avert major slippage. Quarterly reports compare the planned to the actual project progress. A tabulation shows the number of projects that are ahead of, on, and behind schedule. Qualifying statements are made for those projects that are behind schedule. Form 11-1 in the Appendix is used for this purpose. The achievements of the research program cannot be easily shown on a total performance basis—the diversity of the activities is too great to permit their summation; however, the quality of the program can be judged by observing, over time, the progress of some of the measurable parameters. Benchmarking demonstrates research progress and shows quality changes. Some of the factors that are benchmarked to show the performance of the program are as follows:
COMMENTARY: Benchmarking results in the comparison of similar information and requires considerable attention to data collection. The factors suggested above can be best compared on an annual basis; other factors may be compared on a semiannual or biennial basis, but the point of benchmarking is to run a time series of the factor to judge its trend. The selection of factors is very important. Qualification of the data is cumbersome in a graph, so the research manager must select factors that can stand on their own. It may be difficult to decide on a factor that does not have to be qualified or that can stand alone. Hence, the concept of benchmarking progress may have to be dropped as a measure of overall program performance. The factors listed above may not necessarily reflect that more is better. Documentation of successful performance of research is important to ensure continued management and financial support. Objective and quantifiable factors should be the basis for this support. Overall program performance is measured by a combination of the achievement of implementation and milestones and a qualified adherence to financial and scheduling limits. Because management reviews the performance measures, the measures should be presented in a concise form. A good research program is measured by its implementation of results and its timely solutions to Agency problems One technique, designed to improve the quality of the program, is a peer exchange of the management process. A team, with knowledge of state research programs, can bring its expertise to provide recommendations to enhance the research unit's performance. COMMENTARY: Peer exchange is meant to assist the research staff with program performance. (Refer to Sections 7.1, Project Level Reporting, and 11.1, Overall Program Performance). FHWA mandates peer exchange in 23 CFR Part 420.207 (refer to the Appendix). The peer exchange is an activity independent of the research process, conducted with staff outside the agency. The agency can accept its reporting as it would any other report designed to improve management processes. The peer exchange process is not a process compliance review—it is intended to form the basis for states to discuss the research process among themselves and with other knowledgeable researchers. The exchange can result in a better understanding of successful practices for future use. 11.2.2.A Team Peer Exchange With Research Unit The team of at least two members may consist of representatives of FHWA, universities, TRB, the private sector, other agencies, and the research units of other states. At least two of the members of the team will be drawn from a preapproved list compiled by FHWA. COMMENTARY: The cost of travel of the peer exchange team may be charged against the SPR program and 100% federal funding. The state can assist the individual members with travel. University representation on the panel is based on the fact that much of the contract work performed for the state research program is accomplished by universities. The peer exchange team will spend at least 2 days with the research unit staff. Although the items on the agenda may vary due to requests of the team, the typical agenda covers
COMMENTARY: FHWA does not have peer exchange procedures; however, the state should include a peer exchange process in its manual. If definitive guidelines are issued for the process, the state can revise its manual. If the state does not have the peer exchange process, the state will not be in compliance with the regulations. Agendas for peer exchange meetings highlight successful research processes within the agency, but all aspects of the program should be discussed. The peer exchange meeting is scheduled regularly to look at the progress and success of each of the instituted procedures. Every effort should be made to distinguish peer exchange from audit review. Prior to the meeting, the members of the team should scan the strategic plans or their substitute, the research manuals, and the current year's work programs. A.3.1 Program Development a. Problem Solicitation and Selection Peer exchange of the research process starts at the beginning of the process. Therefore, copies of the strategic plan, the solicitation letters for problem statements, and the problem statements received in the current fiscal year are made available to the team. Current lists of the members of the Research Advisory Committee and the Research Management Committee as well as the minutes of the most recent meetings, are made available to the team. b. Work Program Process The process for putting together a work program (which is subject to policy, financial, and management considerations) is discussed with the team. c. Contract Research Process Because of the magnitude of contract research, this aspect of the program warrants discussion. The team is given copies of the contract research process, including the proposal review forms, a list of recent contract projects, a list of all proposals received, the results of the proposal review process for each project, and names of all contractors selected. COMMENTARY: The program development portion of the peer exchange provides the team with information needed to understand the research process in each other's agencies. This research guide covers several of these items under Sections 4.1 (Research Problem Solicitation), 4.2 (Project Selection Process), 7.1 (Project Level Reporting), and 11.1 (Overall Program Performance). Information made available to the team depends on the management plans adopted by the participating states. As in previous sections, reference to the strategic plan can be replaced with reference to the state's critical issues or the solicitation process. As suggested in Section A.2, Meeting Agenda, FHWA requirements may not compel the states to include all issues, but a discussion of issues suggested could be beneficial.A.3.2 Project Progress a. Project Monitoring Satisfactory progress and the transfer of information on research projects is essential to a good relationship with the customer. Copies of the most recent reports are made available to the team. The team also is given an agenda and the minutes of the Project Committee meetings for some selected projects. Research staff members provide details of contractor and in-house project meetings and their frequency. Examples of project implementation results for some projects are also given. b. Reporting Discussions between the peer exchange team and project staff depend on the type and distribution of reports generated by a project. COMMENTARY: There is an expected overlap between A.3.1, Program Development, and A.3.2, Project Progress. A discussion of the program cannot be complete without including a discussion of select projects. A.3.3 Technology Transfer All aspects of technology transfer, as described in Section 8.0, and implementation, in Section 9.0, are discussed with the team. Examples are defined in Sections 11.3.2.A, Internal Review, and 11.3.2.B, User Feedback. COMMENTARY: This may be the most important aspect of the RD&T management process. Although only select projects are discussed, the research unit would be well advised, particularly with the agency's administration, to maintain a complete history of technology transfer progress for all projects. Obviously, project activities are only a portion of the technology transfer effort. All customer-oriented activities should be discussed with the team. The team's input for this activity could be most rewarding for the research unit. A.4.1 Administrative a. Committee Structure The type and membership of the committees affects the potential for the research unit to interact with other Agency units and with organizations outside the Agency. The team is given details of the committees, as defined in Section 3.3, Research Committees. The current membership, minutes from the most recent committee meetings, and the research process as it relates to the committees are provided to the team. b. Resources Research unit resources are best defined by staff size andbudget in the work program. The current appropriation, source of funds, allocation of funds, organization chart, and explanation of the use of staff are provided to the exchange team. c. Staff Training The technical abilities of the research team can be defined by education and practical experience. The peer exchange team is given a list of all training programs available to staff, including state- sponsored courses, research-developed courses, FHWA courses, and university programs in transportation. In addition, the team is told how supervisors advise staff of the training courses. A list of all personnel and their degrees, training courses, and years of experience is made available to the team. COMMENTARY: The administrative procedures show the extent of resources available to each of the research units. The team only can use them as guidelines because a research unit can redeploy its resources and reorganize its process as warranted. The administrative factors are not expected to be discussion issues. The RD&T management process is the only issue the team will discuss. For those states that do not make use of a committee structure, item a., above, should be omitted. Any task forces or project groups actively involved in the program or projects should be listed. There may be good reason to ignore the exchange of the administrative information listed. It can allow the team members to relate to the size of the programs, but there is no need to have the data, unless the team intends to examine each other's program successes. A.4.2 Peer Exchange Report The peer exchange team writes a report on the visit that covers all aspects of the agenda items. The report summarizes the discussions, itemizes the findings, and reiterates the successful practices discussed. Copies of the report are filed with the research unit. The research unit forwards the report to the Divisional office of FHWA. COMMENTARY: The team report has more significance when the research process is thoroughly discussed and there is agency support for a further review. A.4.3 Meeting Frequency and Location The research unit will request a peer exchange in the research office at least once every 3 years. A.5.1 Analysis of Issues The peer exchange is a vigorous effort conducted for the benefit of all participants. It is accomplished by qualified peers to improve the RD&T management process. The deliberations of the team are discussed with research staff and agency management. Every effort will be made to incorporate those practices that can improve the quality of the research process. A.5.2 Report to FHWA The peer exchange team provides a report to the agency. In addition, the research unit reports on the outcome of discussions with agency staff and management. The report is forwarded to the FHWA Divisional office for further discussion, at their discretion. COMMENTARY: If changes to the research process are made, the research manual may also have to be revised, but the guidance offered by the exchange team may only affect how procedures are carried out. The exchange process is an important aspect of FHWA regulations and important in its own right. The agency has the potential for considerable gains from an extensive and intensive review of its management system. 11.2.2.B Exchange With External Research Units Research unit staff members program time to serve as peer exchange team members in other states, using the same review procedures described in Section 11.2.2.A, Team Peer Exchange With Research Unit. The research unit allocates a staff member's expenses for an annual peer exchange trip. Travel funds are not programmed to cover the cost of other team members. If a pooled fund project is set up by FHWA to cover the costs of the peer exchange team visiting the State, the Agency will participate in the pooled fund project. The research unit does not expect to serve as a team member more frequently than once every 3 years. However, if other states cannot serve, or if more than one State representative is used on the teams, a research member's time is programmed annually. COMMENTARY: The subsections above discuss the allocation of expenses for research staff participating in peer exchanges with other states. Staff member expenses for a visit to another state are not directly billed to the agency. It is assumed that an agency would have no difficulty with contributing to an SPR pooled fund project to cover the travel costs. The research manager should make revisions to conform to the policy of the agency and the funding arrangements made by the agency. Peer exchange is designed to let the states collaborate formally. Staff learn from and give guidance to other agencies about the research process. This is an excellent opportunity to participate in and benefit from a nonintrusive review of the agency's research process. The process covers the problem solicitation process, work program, contract research effort, project monitoring, project reporting, technology transfer, and implementation efforts. The result of the exchange is a concentrated discussion of the research management process with concrete examples of good practice. 11.3 Success of Technology Transfer and Implementation Efforts An annual evaluation of the research program measures the success of the technology transfer and implementation efforts. The evaluation consists of an internal review and feedback from others in the Agency who participated in or were the beneficiaries of technology transfer and implementation. COMMENTARY: The question to be answered is, "Has the research unit met its technology transfer and implementation effort goals for the year (or for whatever period is designated for the evaluation time frame)?" If no goals were set, the evaluation asks, "What accomplishments in technology transfer and implementation were made this year?" This is important because it demonstrates the research unit's performance. Did the research unit increase agency personnel knowledge regarding the technologies available to solve operational problems? Did the research unit implement research results to solve operational problems in a thorough, well-executed manner? Evaluation helps the research unit manager to assess whether sufficient resources were committed to technology transfer and implementation. Sections 8.0, Technology Transfer; 9.0, Implementation Process; and 11.1, Overall Program Performance contain information related to technology transfer and implementation activities.An internal review of the research unit's technology transfer and implementation activities is performed annually. This internal review is the research unit's assessment of the technology transfer and implementation activities during the past year. Internal review determines successful strategies that can be duplicated and recommends improvements for the others. Internal review asks the following:
COMMENTARY: Internal review enhances research unit effectiveness. The review may be associated with the agency's total quality program initiatives. The review should be a research unit self-assessment geared toward program improvements. Outside feedback should be sought, but the critical part of the exercise is to encourage the research unit staff to seek ways to improve the program. Improvements are cataloged annually. Care should be taken that internal review does not just count items performed—it should assess effectiveness or quality of performance and the difference it made to the agency and its customers. Generally, a report of activities is not expected for such an evaluation, but if significant process improvements were made, those successes should be documented and acknowledged. Because technology users are those most affected by the technology transfer and implementation activities, the research unit solicits their opinions regularly. A written survey is conducted periodically to determine user perception regarding the research unit's technology transfer and implementation activities and to suggest improvements. Users are chosen from the group of agency staff and others associated with technology transfer and implementation activities performed during the period. The user survey asks about the following:
COMMENTARY: While user feedback is important, care should be taken not to make the survey so complex and tedious that it is too much of an effort. A straightforward and relatively short survey will be easier for the users and will facilitate research unit analysis. Careful consideration should be made in the selection of those to be surveyed. The goal of the survey is to improve the program (which should be clearly stated as the survey's purpose) and those who have the potential to make a genuine contribution should be solicited. It is not necessary to survey every person involved with every project. Personal interviews are preferred. The benefit of having a research program is best measured in terms of the successful transfer of technology. The research staff determines if technology is transferred by assessing users of the research. A survey of the beneficiaries of the research should highlight the efforts of the research program. Research program benefits have been viewed, in the foregoing subsections, from the perspective of aggregating the output of the projects, reviewing the research process with peers and implementing the innovations. Program merits can also be expressed qualitatively, that is, through nonquantifiable measures that reflect program responsiveness, management interests, and the strength and stability of the research process. COMMENTARY: Qualitative benefits of the research program are found in many other areas of the transportation industry beyond those listed below. This is not an exhaustive inventory of nonquantifiable benefits of research. It is offered to provoke more thought on the subject and encourage the research manager to discover others and add to this category. There often are many ways that the activities of the research unit, the projects, the unit's involvement with others, and the unit's technology transfer efforts positively affect those outside the unit. 11.4.2.A Meeting Customer Needs The research program focuses primarily on the needs of the agency, as determined by the agency goals. The projects are categorized by goal. Periodically, measures are taken to develop and include problems that address all the agency's goals. Goals that do not have projects associated with them are pursued by contacting the appropriate Agency staff. Pursuing agency needs fosters enhanced professional relationships between Agency units. All citizens and industries in the state have transportation needs. The relevant research committees can only enlist a few customers from among these citizens and industries. A continuous, active role is fostered with other units of the Agency to contact and interact with transportation users in the state, in order to elicit research needs and membership on appropriate research committees. The active roles played by transportation users outside the Agency can enhance the use of research in solving the state's transportation research problems. Whenever upper management requests special research assistance, the research program will adjust ongoing activities to give a quick response. Records of the responses are maintained. These records include the research request, the requester, the approximate level of response, and the total time to respond. Prompt responses to the agency's management can heighten their opinion of the need for research. The research unit will attempt to include all functions of the agency in its research program. Although not all functions may be included each year, an effort will be made to include all functional areas regularly. The objective of this effort is to be responsive to all aspects of the agency's operations, which may not be explicitly stated above in A.1, Agency Goals. Requests for research are formulated and received from all areas, both internal and external to the Agency. 11.4.2.B Administrative Benefits Adequate funding for research is sought annually. Adequate financial resources are an indication of the support and importance assigned to the program and to the research staff. In addition to the federal SPR funds, resources include other federal funds, state and private funding, and non-research staff funded from other parts of the agency or private industry. B.2 Administrative Cost of Program The cost of administering the research program may be higher than other programs within the agency. In addition to the program development and management requirements discussed in the manual, significant technology transfer activities must be included. Unless there are substantial changes to the administrative aspects of the program, this part of the annual budget should remain relatively constant over time. The technology transfer efforts of the research unit and of the operations staff reflect a significant investment of research funds. Costs are tracked over time. The importance of the technology transfer activity is stressed in Section 8.0, Technology Transfer. The agency is the primary beneficiary of staff training and educational programs, outlined in Section 12.1, Staff Training and Education. The operational goals of the agency are enhanced by skilled staff. Training records are maintained for each member of the staff. 11.4.2.C Maintaining Research Process (Excellence) Some of the benefits of a research program are found in the quality of the processes used to develop and report the projects. Superior products from research can be expected when the elements of the process are models of excellence, open to continuous improvement, and scrupulously followed. The problem solicitation, project selection, and project and program reporting aspects of the research process are discussed in earlier sections of the manual. Program development is currently being conducted in the most advantageous manner for the organization and policies of the Agency. As the research unit and process matures and changes occur in the Agency, refinements to the research process will be examined and incorporated to maintain a high level of performance. Problem solicitation is broadened by regularly changing the membership of committees. Project selection regularly incorporates improved techniques in response to Agency-modified policies, to add to the annual research work program. Project results are reported on a regular basis. C.2 Project Development and Conduct The development of the project objectives and activities are achieved with input from customers and technical staff. Improvements are constantly sought. In addition, the customers' desires are always considered. The conduct and monitoring of the research are in accordance with the suggestions described in Section 6.0, Conducting and Monitoring Research Projects. Periodic training of staff and management review of staff performance enhance project development. The value of the research reports is judged by the Project Committee, using the requirements and elements outlined in Section 7.0, Program Reporting. The elements listed are reviewed on a project basis for conformance to special needs of the customer. Every effort is made to modify the reports to address these needs. This section describes the less tangible benefits of the research program. Satisfaction of the customer and management are paramount to the continued success and growth of the program. The improvement of the research process by enhancing the quality of the product and professional ability of the staff is itself a program benefit.
|
