|This summary report is an archived publication and may contain dated technical, contact, and link information|
The Laboratory Assessment Program TFHRC 2003-2004
On behalf of the Federal Highway Administration's (FHWA) Turner-Fairbank Highway Research Center (TFHRC), I am pleased to share with you our first Laboratory Assessment Program Report. This report describes our lab assessment process and briefly discusses the first four assessments conducted.
We have been very fortunate to have had a high caliber of members on the four review panels, and we thank them for their valuable contributions. We are grateful for the time, effort, and expertise that each of them has devoted to us. Review panel members have come from diverse backgrounds—government, academia, industry, and international communities—and we have benefited from their varied perspectives.
We hope this report will encourage you to learn more about the Office of Research, Development and Technology (RD&T)—its people, labs, services, and research. We welcome your feedback through the TFHRC Web site at http://www.tfhrc.gov. A link to the Laboratory Assessment Program Report was added to our home page to facilitate your input.
Dennis C. Judycki
The Laboratory Assessment Program
The Turner Fairbank Highway Research Center (TFHRC) is owned and operated as part of the Federal Highway Administration (FHWA). The research conducted at TFHRC supports the FHWA mission to enhance mobility through innovation, leadership, and public service.
As part of its commitment to continually reevaluate the effectiveness and efficiency of its operations, TFHRC set about in fiscal year 2003 to establish a lab assessment process that would utilize external, independent evaluators from industry, academia, and government. A pilot assessment of the Human Centered Systems Lab was conducted that year. Following evaluation of the pilot, FHWA developed a plan for an ongoing, full-scale Lab Assessment Program and, in fiscal year 2004, conducted three additional lab assessments: Asphalt, Hydraulics, and Traffic Research.
This first report on the Lab Assessment Program outlines the program's goals, criteria, and process. It summarizes the first four lab assessments. Learn more about the panels' recommendations and how we are addressing them by visiting our Web site.
A lab assessment is an onsite, independent investigation by technical and scientific experts whose knowledge and expertise enable them to make credible and unbiased judgments regarding the conduct of research. The review provides a means to determine whether the research activities have high potential value and whether they are achieving their stated objectives. Additionally, the review produces a set of specific observations and recommendations for improvement of operations. FHWA has articulated the following four goals for the Lab Assessment Program:
Review Panel Composition and Selection
Each expert/peer review panel is comprised of three to five experts from outside FHWA who are qualified to perform an independent review of the technical and scientific merit and quality of the research. FHWA strives to have a mix of experts from industry, academia, and government on each review panel. Panel members are chosen for their technical expertise, practical experience in running research programs, and knowledge of customer needs. They include individuals with a variety of backgrounds:
Panel members may be nominated by the lab manager and staff, FHWA managers, Transportation Research Board (TRB) committees, American Association of State Highway and Transportation Officials (AASHTO) committees and working groups, and other technical and professional associations. In addition, nominations are routinely solicited from AASHTO's Research Advisory Committee. TRB's Research and Technology Coordinating Committee (RTCC) reviews the recommendations to help ensure the independence of the review panels.
Panel members may not be under contract to the FHWA lab. Individuals asked to serve on expert/peer review panels are asked to complete a disclosure statement and to identify any possible conflicts of interest.
Assessment Scope and Criteria
The TFHRC lab assessment criteria are based on the Office of Management and Budget's three criteria for Federal investment in research-relevance, quality, and performance. The review covers the administration and operations of the laboratory and its recently completed research, research in progress, and near-term future activities. Although lab assessments are not intended to be program reviews, as the panel members evaluate relevance, they do consider how the work of the lab supports the agency's mission and how well it addresses customer needs.
The expert/peer review agenda is planned with the panel chairman in advance. Effort is made to balance the panel's need for overview information with its need to engage in independent fact finding, investigation, and deliberations. The expert/peer review panel obtains information from many sources, including:
After meeting with the lab staff, FHWA managers, stakeholders, and customers, the review panel prepares a draft report that discusses strengths, key issues, opportunities for improvement, and other observations. The draft is shared with the lab managers and staff. At that time, the panel has the opportunity to receive feedback and to correct any misinterpretations or factual errors. The panel then finalizes its report and prepares a brief presentation for a closeout session with senior management. During this meeting, the review panel presents its recommendations and distributes its written report to the Associate Administrator and other senior managers.
The lab assessment is not really complete until the agency considers and addresses the panel's recommendations.
Within a month following the assessment, the lab manager develops a matrix of actions to address the panel's recommendations. Progress is reviewed by FHWA management every 6 months. This followup process allows the lab to implement improvements in a deliberate and measurable way during the 4-year period between assessments.
HIGHLIGHTS OF THE 2003-2004 LAB ASSESSMENTS
During the 2003 and 2004 fiscal years, expert peer review panels conducted assessments of four labs: the Human Centered Systems, Asphalt, Hydraulics, and Traffic Research labs. The review panels each had five members from industry, academia, and government with at least one member representing each. The panel members were chosen from more than 100 nominations.
A dedicated meeting room and computer resources were provided for the panels, as well as an independent facilitator. The facilitator's tasks were to keep the panels focused on their assignment, assist them in formulating recommendations and writing their report, and serve as a liaison between the panel and FHWA staff during the assessment.
All four panels commented on the cooperation afforded them during the reviews. Staff members communicated openly and provided information freely during the presentations, lab tours, and interviews.
Although the 2003-2004 assessments covered four distinctly different labs, a significant level of commonality appeared in the observations and recommendations of the review panels. Overall, the lab assessment panels were impressed with the competence of the researchers and the quality of the work being done. They found good matches between projects and staff skill sets, great enthusiasm for projects, and generally good morale. TFHRC's research is relevant, high quality, and well executed, meeting State and national needs. Research products from the labs are recognized and used nationally and internationally.
Across the board, panels found that the research projects showed good technical approaches to real problems being faced by industry. In particular, it was noted, TFHRC fulfills a role in fundamental, high stake research that cannot be fulfilled elsewhere.
The advice received from the panels has been invaluable. In addition, the experience of preparing for the assessment has been a useful exercise for FHWA staff. TFHRC has also benefited from the onsite interactions and follow-on exchanges between the review panel members and lab staff.
The assessments have also served as a valuable opportunity to obtain feedback from customers and stakeholders who were interviewed by the panel. At the same time, the assessments have raised the visibility of the work of the labs with these customers and stakeholders and have strengthened ties to other labs and organizations.
The Lab Assessment Program is enhancing the relevance, quality, and performance of FHWA's research program. FHWA plans to go forward with the lab assessment schedule to assess all of the labs within 4 years, and then repeat the cycle. To provide continuity, one or two members of each of the original panels will be invited to participate in the next review cycle. In the interim, the labs are tracking progress on action items and creating a record of changes made in response to advice from assessment panels.
Individual Lab Assessment Highlights
Following are summaries of the four lab assessments conducted in 2003-2004, in order of their occurrence. Each summary identifies the expert/review panel members, describes the lab's functions, and provides a summary of the review panel's observations.
For more detailed information about the recommendations of the panels and how they are being addressed, visit our Web site at www.tfhrc.gov.
APRIL 28–MAY 1, 2003
Human Centered Systems
Thomas B. Sheridan
Richard F. Pain
Some notable strengths include peer reviews and involvement of external subject-matter experts in projects, which helps ensure a certain level of quality in the work and enhances staff development. The assessment panel also endorsed activities to develop and update roadmaps for research.
The panel made the following recommendations:
MARCH 1-4, 2004Asphalt Labs
Expert/Peer Review Panel
Nicholas P. Vitillo
Facilitator: Barbara Harder
The Review Panel observations and recommendations included the following:
Panel recommendations include:
MAY 24-27, 2004TReL
Expert/Peer Review Panel
Facilitator: Barbara Harder
The expert/peer review panel noted several significant strengths of TReL, including:
The panel recommended the development of a long-term strategic plan to guide the future work of the TReL, and emphasized the importance of customers and stakeholders' involvement in the planning process.
AUGUST 9-12, 2004Hydraulics
Expert/Peer Review Panel
D. Max Sheppard
Facilitator: Barbara Harder
The expert/peer review panel noted the following key findings:
However, FHWA staffing levels appear insufficient to continue to provide the high quality research performed to date and to meet new agency goals.
For Additional Information
To learn more regarding a specific laboratory assessment, visit our Web site at www.tfhrc.gov or contact the laboratory manager. For information regarding the laboratory assessment program, please contact:
Office of Corporate, Research, Technology, and Innovation Management