• Federal Highway Administration >
• Publications >
• Research Publications >
• 05032 >
• Nondestructive Evaluation >
• The Laboratory Assessment Program TFHRC 2003-2004

The Laboratory Assessment Program TFHRC 2003-2004

Foreword

On behalf of the Federal Highway Administration's (FHWA) Turner-Fairbank Highway Research Center (TFHRC), I am pleased to share with you our first Laboratory Assessment Program Report. This report describes our lab assessment process and briefly discusses the first four assessments conducted.

We have been very fortunate to have had a high caliber of members on the four review panels, and we thank them for their valuable contributions. We are grateful for the time, effort, and expertise that each of them has devoted to us. Review panel members have come from diverse backgrounds—government, academia, industry, and international communities—and we have benefited from their varied perspectives.

We hope this report will encourage you to learn more about the Office of Research, Development and Technology (RD&T)—its people, labs, services, and research. We welcome your feedback through the TFHRC Web site at http://www.tfhrc.gov. A link to the Laboratory Assessment Program Report was added to our home page to facilitate your input.

Dennis C. Judycki
Associate Administrator for Research, Development and Technology
Director, Turner-Fairbank Highway Research Center

The Laboratory Assessment Program

The Turner Fairbank Highway Research Center (TFHRC) is owned and operated as part of the Federal Highway Administration (FHWA). The research conducted at TFHRC supports the FHWA mission to enhance mobility through innovation, leadership, and public service.

As part of its commitment to continually reevaluate the effectiveness and efficiency of its operations, TFHRC set about in fiscal year 2003 to establish a lab assessment process that would utilize external, independent evaluators from industry, academia, and government. A pilot assessment of the Human Centered Systems Lab was conducted that year. Following evaluation of the pilot, FHWA developed a plan for an ongoing, full-scale Lab Assessment Program and, in fiscal year 2004, conducted three additional lab assessments: Asphalt, Hydraulics, and Traffic Research.

This first report on the Lab Assessment Program outlines the program's goals, criteria, and process. It summarizes the first four lab assessments. Learn more about the panels' recommendations and how we are addressing them by visiting our Web site.

A lab assessment is an onsite, independent investigation by technical and scientific experts whose knowledge and expertise enable them to make credible and unbiased judgments regarding the conduct of research. The review provides a means to determine whether the research activities have high potential value and whether they are achieving their stated objectives. Additionally, the review produces a set of specific observations and recommendations for improvement of operations. FHWA has articulated the following four goals for the Lab Assessment Program:

• Enhance research quality, performance, and relevance by providing lab managers feedback and suggestions for improvements.
• Supply an opportunity for an exchange of views among technical experts.
• Offer increased opportunities for FHWA customers and stakeholders to provide input to research and related activities.
• Conduct a credible, professional, and objective assessment that further improves customer and stakeholder confidence in the conduct of research and the outcomes produced.

Review Panel Composition and Selection

Each expert/peer review panel is comprised of three to five experts from outside FHWA who are qualified to perform an independent review of the technical and scientific merit and quality of the research. FHWA strives to have a mix of experts from industry, academia, and government on each review panel. Panel members are chosen for their technical expertise, practical experience in running research programs, and knowledge of customer needs. They include individuals with a variety of backgrounds:

• Scientists from similar labs.
• Scientists in allied or other disciplines performing similar work.
• Managerial or technical mentors from the private sector, academia, or other public agencies.
• Scientists who have conducted successful and productive lab assessments for other agencies (for example, the National Science Foundation and National Institute of Standards and Technology).
• Recently retired scientists and researchers who, before retirement, would have qualified in one of the above categories.
• Engineers and other professionals in disciplines performing similar work.

Panel members may be nominated by the lab manager and staff, FHWA managers, Transportation Research Board (TRB) committees, American Association of State Highway and Transportation Officials (AASHTO) committees and working groups, and other technical and professional associations. In addition, nominations are routinely solicited from AASHTO's Research Advisory Committee. TRB's Research and Technology Coordinating Committee (RTCC) reviews the recommendations to help ensure the independence of the review panels.

Panel members may not be under contract to the FHWA lab. Individuals asked to serve on expert/peer review panels are asked to complete a disclosure statement and to identify any possible conflicts of interest.

Assessment Scope and Criteria

The TFHRC lab assessment criteria are based on the Office of Management and Budget's three criteria for Federal investment in research-relevance, quality, and performance. The review covers the administration and operations of the laboratory and its recently completed research, research in progress, and near-term future activities. Although lab assessments are not intended to be program reviews, as the panel members evaluate relevance, they do consider how the work of the lab supports the agency's mission and how well it addresses customer needs.

An illustrative list of issues related to each of the three criteria follows. The list is intended as general guidance, not as a formal checklist, for the panel.

Relevance

• Research supports the mission of the agency and is based on direction as expressed in the FHWA multiyear research and technology performance plans.
• Research purpose is clear and addresses a specific interest, problem, or need.
• Research is designed to make a unique contribution to address a specific interest, problem, or need, and is not needlessly redundant of other Federal, State, local, or private efforts.

Quality

• Research maximizes quality through the use of clearly stated defensible methods for awarding contracts, and Federal managers and contractors are held accountable for cost, schedule, and performance results.
• Quality assessment of the research is conducted through comparative methods such as best practices identification, expert/peer reviews, and benchmarking.
• Research is reported in publications that are peer reviewed, not just FHWA reports.
• Methods are in place for maintaining the expertise of research personnel and the capabilities of laboratory facilities.
• Quality guidelines for statistical information are based on structured planning and sound statistical methods.
• Research demonstrates objectivity in presentation, substance, and integrity, (i.e., protecting information from unauthorized access, corruption, or revision).

Performance

• Research activities are managed in a manner that produces high quality, identifiable results, utilizing research procedures and practices that comply with or exceed accepted standards for performance and reproducibility.
• Research activities are guided by an established set of high priority research objectives with performance outputs and milestones that show how the outcomes will be reached.
• Research activities have well-defined metrics that encourage research project performance and promote broader goals such as implementation of research results, including disseminating knowledge, applications, or tools; transitioning technology to the private sector, if appropriate; and encouraging innovation, cooperation, and education.
• For major research projects, appropriate termination points and other decision points are adequately defined.

REVIEW PROCESS

The expert/peer review agenda is planned with the panel chairman in advance. Effort is made to balance the panel's need for overview information with its need to engage in independent fact finding, investigation, and deliberations. The expert/peer review panel obtains information from many sources, including:

• Briefing book sent to panel members before the review.
• Presentations and other materials about the administration and operations of the lab and the conduct of its research.
• Lab tour.
• Interviews—face-to-face and telephone—with lab personnel, customers, and stakeholders.
• Technical reports and other lab products.
• Panel discussions.
• Firsthand observations of lab activities.

After meeting with the lab staff, FHWA managers, stakeholders, and customers, the review panel prepares a draft report that discusses strengths, key issues, opportunities for improvement, and other observations. The draft is shared with the lab managers and staff. At that time, the panel has the opportunity to receive feedback and to correct any misinterpretations or factual errors. The panel then finalizes its report and prepares a brief presentation for a closeout session with senior management. During this meeting, the review panel presents its recommendations and distributes its written report to the Associate Administrator and other senior managers.

ASSESSMENT FOLLOWUP

The lab assessment is not really complete until the agency considers and addresses the panel's recommendations.

Within a month following the assessment, the lab manager develops a matrix of actions to address the panel's recommendations. Progress is reviewed by FHWA management every 6 months. This followup process allows the lab to implement improvements in a deliberate and measurable way during the 4-year period between assessments.

HIGHLIGHTS OF THE 2003-2004 LAB ASSESSMENTS

During the 2003 and 2004 fiscal years, expert peer review panels conducted assessments of four labs: the Human Centered Systems, Asphalt, Hydraulics, and Traffic Research labs. The review panels each had five members from industry, academia, and government with at least one member representing each. The panel members were chosen from more than 100 nominations.

A dedicated meeting room and computer resources were provided for the panels, as well as an independent facilitator. The facilitator's tasks were to keep the panels focused on their assignment, assist them in formulating recommendations and writing their report, and serve as a liaison between the panel and FHWA staff during the assessment.

All four panels commented on the cooperation afforded them during the reviews. Staff members communicated openly and provided information freely during the presentations, lab tours, and interviews.

Although the 2003-2004 assessments covered four distinctly different labs, a significant level of commonality appeared in the observations and recommendations of the review panels. Overall, the lab assessment panels were impressed with the competence of the researchers and the quality of the work being done. They found good matches between projects and staff skill sets, great enthusiasm for projects, and generally good morale. TFHRC's research is relevant, high quality, and well executed, meeting State and national needs. Research products from the labs are recognized and used nationally and internationally.

Across the board, panels found that the research projects showed good technical approaches to real problems being faced by industry. In particular, it was noted, TFHRC fulfills a role in fundamental, high stake research that cannot be fulfilled elsewhere.

The advice received from the panels has been invaluable. In addition, the experience of preparing for the assessment has been a useful exercise for FHWA staff. TFHRC has also benefited from the onsite interactions and follow-on exchanges between the review panel members and lab staff.

The assessments have also served as a valuable opportunity to obtain feedback from customers and stakeholders who were interviewed by the panel. At the same time, the assessments have raised the visibility of the work of the labs with these customers and stakeholders and have strengthened ties to other labs and organizations.

LOOKING AHEAD

The Lab Assessment Program is enhancing the relevance, quality, and performance of FHWA's research program. FHWA plans to go forward with the lab assessment schedule to assess all of the labs within 4 years, and then repeat the cycle. To provide continuity, one or two members of each of the original panels will be invited to participate in the next review cycle. In the interim, the labs are tracking progress on action items and creating a record of changes made in response to advice from assessment panels.

Individual Lab Assessment Highlights

Following are summaries of the four lab assessments conducted in 2003-2004, in order of their occurrence. Each summary identifies the expert/review panel members, describes the lab's functions, and provides a summary of the review panel's observations.

For more detailed information about the recommendations of the panels and how they are being addressed, visit our Web site at www.tfhrc.gov.

APRIL 28–MAY 1, 2003

Human Centered Systems
Expert/Peer Review Panel

Thomas B. Sheridan
Panel Chairman
Professor Emeritus, Massachusetts Institute of Technology

David Huft
Director, Office of Research
South Dakota Department of Transportation

Ian Noy
Chief, Ergonomics Division
Transport Canada
President, Systems Ergonomics

Richard F. Pain
Transportation Safety Coordinator
Transportation Research Board

Louis Tijerina
Senior Technical Specialist
Ford Motor Company
Facilitator: Barbara Harder
B.T. Harder, Inc.

The assessment panel was impressed with the dedication and enthusiasm of the Human Centered Systems Lab Team. The customers interviewed were satisfied with the quality and standard of work and cited increased awareness and growing appreciation for human factors research.

Some notable strengths include peer reviews and involvement of external subject-matter experts in projects, which helps ensure a certain level of quality in the work and enhances staff development. The assessment panel also endorsed activities to develop and update roadmaps for research.

The panel made the following recommendations:

• Include human factors personnel in problem definition, concept exploration, and requirements specification phases of the project planning.
• Solicit critical stakeholders for active participation in the project. For example, original equipment manufacturers have a significant role to play in the implementation of in-vehicle warnings and displays.
• Review and update 5-year strategic plans, spending plans, and tactical plans biennially. This includes professional development of staff and discretionary long-term research.

MARCH 1-4, 2004

David Newcomb
Panel Chairman
Vice President, Research and Technology
National Asphalt Pavement Association

Edward Harrigan
Senior Program Officer
Transportation Research Board

Bruno Marinelli
Manager, Central Labs Materials Engineering Branch
Manitoba Transportation and Government Services

David Spivey
President
DSI, Inc.

Nicholas P. Vitillo
Manager, Bureau of Research
New Jersey Department of Transportation

Facilitator: Barbara Harder
B.T. Harder, Inc.

The Review Panel observations and recommendations included the following:

• Research personnel are highly self-motivated, qualified, and professional. The environment fosters world-class expertise.
• Laboratories are very well equipped. Few facilities in the world have similar capabilities.
• Stakeholders are very positive about the role that TFHRC is taking.
• TFHRC fulfills a role in fundamental, high stakes research that cannot be fulfilled elsewhere.
• The mission needs to be redefined in a post-Superpave^®era and a long-term view of laboratory direction needs to be verbalized.
• Staff needs to be involved in planning for the future.
• Evolving issues in asphalt technology need to be identified.
• New areas of research need long lead times to build expertise and capabilities.
• External communication (reporting results) needs improvement.

Panel recommendations include:

• Developing and articulating a long-term view of laboratory direction.
• Establishing an external advisory mechanism for stakeholder input.
• Establishing a laboratory information management system.
• Improving the system of work plan development, project tracking, and followup.
• Tracking implementation and technology transfer activities.

MAY 24-27, 2004

Edward Seymour
Panel Chairman
Associate Agency Director
Texas Transportation Institute

Paul Jovanis
Panel Vice Chairman
Professor, Civil and Environmental Engineering
Penn State University

Peter Kohl
Vice President, Business Development
McCain Traffic Supply

Joel Meena
Assistant State Traffic Engineer
Wyoming Department of Transportation

Anson Nordby
Principal Transportation Engineer
Department of Transportation City of Los Angeles

Facilitator: Barbara Harder
B.T. Harder, Inc.

The expert/peer review panel noted several significant strengths of TReL, including:

• Excellent ties to the transportation industry.
• Highly skilled, dedicated, and motivated staff.
• Research products have been well received and used by the industry (e.g., QuickZone, Intelligent Transportation Systems Deployment Analysis System, and Corridor Simulation (CORSIM)).
• Adaptive Control Systems Lite is being developed in a nontraditional approach by partnering with industry for more rapid development. This strategy could become a model for future developments of this nature.
• The construction and use of the TReL local intersection for research in safety and mobility is a constructive step in broadening the lab's focus beyond a roadway efficiency emphasis.

The panel recommended the development of a long-term strategic plan to guide the future work of the TReL, and emphasized the importance of customers and stakeholders' involvement in the planning process.

AUGUST 9-12, 2004

E.V. Richardson
Panel Chairman
Professor Emeritus, Colorado State University
Senior Associate, Ayres Associates

Scott Sabol
Professor
Vermont Technical College

Richard Long
Director, Research Center
Florida Department of Transportation

D. Max Sheppard
President
Ocean Engineering Associates, Inc.

Stan Davis
Consultant
Office of Bridge Development
Maryland State Highway Administration

Facilitator: Barbara Harder
B.T. Harder, Inc.

The expert/peer review panel noted the following key findings:

• Research is relevant, high quality, and well executed, meeting State and national needs.
• Research products from the laboratory are recognized and used nationally and internationally.

However, FHWA staffing levels appear insufficient to continue to provide the high quality research performed to date and to meet new agency goals.

For Additional Information

To learn more regarding a specific laboratory assessment, visit our Web site at www.tfhrc.gov or contact the laboratory manager. For information regarding the laboratory assessment program, please contact:

Debra Elston
Acting Director, Office of Corporate, Research, Technology, and Innovation Management

Office of Corporate, Research, Technology, and Innovation Management
Federal Highway Administration
Turner-Fairbank Highway Research Center
6300 Georgetown Pike McLean, VA 22101-2296
202-493-3181
E-mail: debra.elston@fhwa.dot.gov