The purpose of the Handbook for Expert/Peer Reviews is to provide guidance for participants in the Turner-Fairbank Highway Research Center (TFHRC) Laboratory Assessment Process. The primary audience for the Handbook is members of the panels serving to review a laboratory. The Handbook acquaints panel members with the process and expectations associated with their involvement in the review. The Handbook is also a useful source of information about the reviews for laboratory managers and staff as well as the customers and stakeholders of the laboratories being reviewed.

Expert/Peer Reviews

An expert/peer review is an independent assessment by technical and scientific experts whose knowledge and expertise enable them to make credible and unbiased judgments regarding the conduct of the reviewed research. Assessing TFHRC laboratories through expert/peer reviews will ensure that the research performed at the TFHRC is relevant to the mission and customer needs, and meets established quality and performance standards. Such laboratory assessments provide, through independent evaluation, a means to determine whether the research activities have high potential value and whether they have achieved stated objectives. The laboratory assessment process is continuous, with each laboratory having the benefit of an expert/peer review every 4 years.

Assessment Goals

The main goals of the TFHRC Laboratory Assessment Process are to

Enhance laboratory and related research program relevance, quality, and performance by providing feedback and suggestions for improvements to laboratory managers.
Provide an opportunity for exchange of views among technical experts.
Provide increased opportunities for Federal Highway Administration (FHWA) customers and stakeholders to provide input to research and related program activities.
Provide a credible, professional, and objective assessment that further improves customer and stakeholder confidence in the conduct of research and the outcomes produced.

FHWA Vision, Mission, Goals, and Roles

The FHWA Research and Technology (R&T) Program supports the vision, mission, strategic goals, and primary roles of the FHWA, which are defined as follows:

Vision

Improving Transportation for a Strong America

Mission

Enhancing Mobility Through Innovation, Leadership, and Public Service

Goals

Safety
Mobility and Productivity
Global Connectivity
Environment
National Homeland Security
Organizational Excellence

Roles

Leaders for National Mobility
Stewards for National Highway Programs
Innovators for a Better Future

To support the FHWA role of "Innovators for a Better Future," TFHRC leadership is committed to

Invest in and conduct transportation research with and on behalf of partners and stakeholders.
Create an environment for and deliver innovation in the transportation community through the development of tools, technology transfer, training and technical assistance.
Continually evaluate and improve the effectiveness and efficiency of its key business process of technology and innovation deployment.

In light of these guiding principles, TFHRC provides the FHWA, its customers and stakeholders, and the world highway community with research and development related to new highway technologies—focusing on solutions to complex technical problems by developing economical, environmentally sensitive designs; efficient, quality-controlled construction practices; durable materials; and products that result in a safer, more reliable highway transportation system.

Assessment Criteria

The TFHRC Laboratory Assessment Criteria are based on three criteria for Federal investment in research established by the Office of Management and Budget. These three criteria are relevance, quality, and performance. Relevance means both relevance to the mission of the agency and relevance to the customer. Panel members will want to determine that the laboratory staff (both Federal employees and contractors) is aware of how their work supports the mission of the agency and the needs of their customers. The laboratory assessment process at TFHRC particularly highlights the performance and quality criteria, as shown below.

Relevance

Research supports the mission of the agency and is based on direction as expressed in FHWA multiyear R&T program plans.
The research purpose is clear and addresses a specific interest, problem, or need.
The research is designed to make a unique contribution to addressing a specific interest, problem, or need, and is not needlessly redundant of other Federal, State, local, or private efforts.

Quality

Research maximizes quality through the use of clearly stated defensible methods for awarding contracts, and Federal managers and contractors are held accountable for cost, schedule, and performance results.
Quality assessment of the research is conducted through comparative methods such as best practices identification, expert/peer reviews, and benchmarking.
In addition to FHWA reports, research is reported in publications that are peer reviewed.
Methods are in place for maintaining the expertise of research personnel and the capabilities of laboratory facilities.
Quality guidelines for statistical information are based on structured planning and sound statistical methods.
Research demonstrates objectivity in presentation and substance, and integrity, (i.e., protecting information from unauthorized access, corruption, or revision).

Performance

Research activities are managed in a manner that produces high quality, identifiable results, utilizing research procedures and practices that comply with or exceed accepted standards for performance and reproducibility.
Research activities are guided by an established set of high-priority, research objectives with performance outputs and milestones that show how the outcomes will be reached.
Research activities have well-defined metrics that encourage research project performance and promote broader goals such as implementation of research results, including disseminating knowledge, applications, or tools; transitioning technology to the private sector, if appropriate; and encouraging innovation, cooperation, and education.
For major research projects, termination points and other decision points are adequately defined.

To assist the Expert/Peer Review Panel as it examines the laboratory’s activities, questions for consideration based on the assessment criteria are provided in this Handbook.

Role of the Research and Technology Coordinating Committee (RTCC) in the Laboratory Assessment Process

The Transportation Research Board (TRB), with the FHWA support, convened the RTCC to provide continuing guidance and advice on the nation’s highway research program. The RTCC’s mission enables the committee to take a broad view of highway research that is not restricted to a particular program, topic area, or agency. RTCC membership includes top-level administrators, researchers, and practitioners from State governments, academia, and industry.

The RTCC encouraged the FHWA to put a laboratory assessment process in place. Because of the members’ knowledge of transportation research and technology, their extensive personal network of contacts, and their independence from the TFHRC management, the RTCC is well suited to provide an advisory function for the laboratory assessment process. In its advisory role, the RTCC may review and comment on the assessment criteria as well as the proposed composition of expert/peer review panels. Their review helps ensure the high caliber and independence of the panel. The RTCC also receives periodic updates on laboratory assessment-related activities.

Role of the TFHRC Leadership Council in the Laboratory Assessment Process

The Research, Development, and Technology (RD&T) Leadership Council (LC) mission is to continuously improve the RD&T organization and its achievement of FHWA’s strategic goals. The Council works on a wide range of issues related to the effective management of TFHRC including items such as developing research priorities, setting a research agenda that meets customer needs, and the delivery of high-value work relevant to FHWA and USDOT missions. The LC meets periodically throughout the year. It is headed by the Associate Administrator for Research, Development, and Technology and composed of TFHRC senior staff. The LC monitors laboratory assessment activities and outcomes, and makes decisions regarding implementation of process improvements at the TFHRC level.

Role of the TFHRC Office of Program Development and Evaluation in the Laboratory Assessment Process

The Office of Program Development and Evaluation at TFHRC is responsible for the general administration and management of the laboratory assessment program. These functions include:

Facilitating and coordinating the conduct of expert/peer reviews at TFHRC.
Providing counsel to laboratory managers regarding the conduct of a review.
Publishing and distributing the Handbook for Expert/Peer Reviews.
Developing training information for TFHRC personnel for participating in expert/peer reviews.
Providing guidance for proper interpretation and application of assessment criteria.
Collecting recommendations for nomination to expert/peer review panels, and forwarding such recommendations to the RTCC.
Preparing letters of invitation to the panel members.
Providing assistance with travel arrangements for panel members.
Helping to make arrangements for meals, including a welcome dinner, as appropriate.
Helping to compile briefing material for the panel and distributing materials in advance of the review.
Setting up appointments with customers, partners, and/or stakeholders, as necessary.
Coordinating with other offices at TFHRC to provide administrative assistance and computer resources for the panel, when necessary.
Receiving expert/peer review panel reports.
Evaluating the effectiveness of the laboratory assessment process and providing recommendations for improvements and enhancements.
Managing funding for the laboratory assessment process.
Providing current status and program information to the RTCC.
Guiding the laboratory manager in the preparation of an Action Matrix to address the recommendations of the panel.
Posting summaries of the expert/peer reviews and activities taken in response to panel recommendations on the TFHRC website.

Role of Customers, Partners, and Stakeholders

Customers, partners, and stakeholders represent the offices and organizations that rely on the results produced by the TFHRC laboratory or that have a share in the success or failure of the activities the laboratory performs. The expert/peer review panel may interview these important individuals either through telephone interviews and, as available, through a focus group of customers conducted at a lunch during the review. These interviews create a significant opportunity for customers, stakeholders, and partners to give candid feedback and influence the work of the laboratory.

Laboratory Assessment Process

Expert/Peer Review Overview

The expert/peer review is accomplished through the collaborative efforts of the TFHRC senior managers, the Office of Program Development and Evaluation, the expert/peer review panel, the review facilitator, the laboratory manager and other staff, and the laboratory’s customers and stakeholders. Panel members are selected through a formal process that provides for nominations from a broad array of transportation and scientific bodies. The expert/peer review generally takes up to 3 days, although in some special cases, more time may be appropriate. During the first 2 days, the panel gathers information, including touring the laboratory, interviewing staff, stakeholders, and customers, and synthesizing material. On the last day, the panel prepares a report of its observations and conducts a closeout session with the TFHRC senior management.

The expert/peer review panel obtains information to perform its review from

The Briefing Book sent to review panel members before the review.
Presentations and other materials about the administration and operations of the laboratory and the conduct of its research.
Laboratory tour.
Interviews with laboratory personnel, customers, and stakeholders.
Firsthand observations of laboratory activities.
Review of the TFHRC website, technical reports, and other laboratory products.
Panel discussions.

The assessment criteria guide the direction of the expert/peer review. The content of the review deals with the administration and operations of the laboratory and its recently completed research, research in progress, and near-term future activities. The Laboratory Assessment Process is continual with each laboratory being assessed every four years.

The laboratory may request that the panel focus on a specific area of interest while performing the review. Details about this focus area, if identified, are provided in the Briefing Book, which contains materials prepared for the expert/peer review panel by the TFHRC laboratory manager, and is sent to the panel with the Handbook for Expert/Peer Reviews in advance of the review. Members of the review panel, as well as customers and stakeholders, may identify additional topics for discussion during the review.

The agenda for the review is worked out with the panel chairman in advance of the assessment. This generally is done during a telephone conference call, which includes the panel chairperson, the laboratory manager, the laboratory assessment program manager, and the facilitator. The agenda is intended to be flexible and may be adjusted, if required, as the assessment is underway. Effort is made to balance the panel’s need for overview information with the need to provide sufficient time for independent fact-finding, investigation, and panel deliberations. (See the Appendix for a sample agenda.)

The expert/peer review panel chairperson may also choose to have a teleconference with the panel members before the review. During this teleconference, the panel chairperson has the opportunity to discuss the draft agenda with the panel. At this time the panel members can also receive answers to questions about the general conduct of the review.

The review is held onsite at TFHRC, where information is available about the operations and conduct of research at the laboratory(ies) being reviewed. The expert/peer review panel has a meeting room at TFHRC and computer resources available for its use during the entire review process. Time is reserved in the daily schedule to allow the expert/peer review panel to confer among itself, as necessary. Working dinners also may be scheduled for the expert/peers to discuss the day’s topics and to synthesize what they have observed. A facilitator helps keep the panel focused an on track and serves as a liaison between the panel and TFHRC staff.

It is important that the panel has the support of senior management and is able to maintain contact with management throughout the review, from the welcome dinner to the closeout session. The welcome dinner not only serves as an opportunity for the panel members to get to know each other, but also as an opportunity for senior management to show their support of the laboratory assessment process, letting the panel members know that they value their time and welcome their advice. Generally, an opportunity for senior management (the Associate Administrator and Office Director) to touch base with the panel is provided at the end of the second full day of work. That session is an opportunity to provide the management perspective, answer questions from the panel, and reaffirm the agency commitment to obtain any information that the panel needs, as they continue their deliberations. Finally, the close out session is a face-to-face meeting between the panel members and senior management at which the panel presents its observations and recommendations. These meetings signal that the laboratory assessment process has the full support of management.

Following the completion of the expert/peer review, the laboratory manager prepares an Action Matrix that incorporates all observations contained in the panel’s final report. The Action Matrix is approved by the Associate Administrator and is used as a continuing tool to measure the progress of improvements based on the review.

Expert/Peer Review Panel

The expert/peer review panel is comprised of approximately three to five experts/peers who are external to the Agency. The panel members are qualified to perform an independent, competent review of the technical and scientific merit and quality of the research.

Nomination of an expert/peer as a qualified reviewer is a distinguished recognition of the individual’s professional accomplishments in his or her field of endeavor. The expert/peer’s participation in the review process is broadly acknowledged within the transportation and scientific community.

Panel members may be

Scientists from similar laboratories.
Scientists in allied or other disciplines performing similar types of work.
Managers of research programs from the private sector, academia, or other public agencies.
Managerial or technical mentors from the private sector, academia, or other public agencies.
Scientists that have conducted successful/productive laboratory assessments (for example, from the National Institute of Standards and Technology or other such groups).
Recently retired scientists/researchers who, before retirement, would have qualified in one of the above categories.
Engineers and other professionals in disciplines performing similar types of work.

Panel members may come from

Federal laboratories.
Other Federal agencies.
State Departments of Transportation.
Academia.
Private sector.
Customers or stakeholders.

Panel members may be nominated by the laboratory manager and staff, FHWA managers, TRB Committees, American Association of State Highway and Transportation Officials (AASHTO) Committees, and other scientific and professional organizations. In addition to AASHTO Technical Committees and working groups, nominations are routinely solicited from AASHTO’s Research Advisory Committee (RAC). The TRB RTCC reviews recommendations for panel members and alternates, and may provide comments on the composition of the panel. The RTCC review serves as an extra check to help ensure the high caliber and independence of each expert review panel.

Because of the varying perspectives each panel member can bring to panel deliberations, it is advantageous to have diverse backgrounds represented on a review panel. Panel members employed by other Federal and State agencies may possess unique and indispensable expertise.

It is possible for someone to be nominated to serve on a panel for more than one laboratory assessment. However, an effort is made to rotate peer review responsibilities across the available pool of qualified candidates. Repeated use of the same reviewer for multiple assessments is avoided because it can raise serious questions about the objectivity and independence of the review. (If an expert has repeatedly served as a panel member, some may question whether he or she is sufficiently independent from the agency to be selected as a peer reviewer.) Rotating panel membership among qualified experts helps the FHWA to obtain fresh perspectives and reinforces the reality and perception of independence from the agency.

The FHWA strives to have a mix of experts from industry, academia, and government on each review panel to help evaluate the technical merits of the laboratory research, as well as to help determine how well the research is meeting the needs of FHWA’s customers and stakeholders.

Panel Facilitator

At the discretion of the FHWA, a review facilitator accompanies the panel throughout the review. The facilitator is independent and external to the agency, and assists the panel in its duties, knows about the conduct of peer reviews, and acts as a liaison to the FHWA for the panel.

Conflict of Interest

Panel members should not be under contract to the FHWA laboratory in any way. Panel members will be asked to disclose relevant activities and possible conflicts of interest. The panel members discuss these items at an initial meeting of the panel. This disclosure allows the panel members to understand the unique experience-based perspectives contained within the group. Panel members who are Federal employees are subject to Federal requirements governing conflicts of interest (18 U.S.C. Section 208 5 C. F. R. Part 2635). (See the Appendix for a copy of the Conflict of Interest Disclosure Form).

Approval for Use of Photographs

Panel members will be asked to sign a release form to allow use of photographs taken during the conduct of the review for publishing in electronic or printed materials. Signing the release form is at the discretion of the panel member and will not affect their selection for the panel. (See the Appendix for a copy of the Multimedia Model Release Form.)

Logistics

Arrangements for hotel, transportation, meals and other items are coordinated through the TFHRC Office of Program Development and Evaluation. (See the Appendix for a Logistics Fact Sheet).

Honorarium

Panel members are offered an honorarium for their participation. The honorarium is intended as recognition that their time and advice is valuable. Whether a panel member can accept the honorarium may depend on the rules of his/her employing organization.

Panel Chairperson

The expert/peer panel chairperson is invited, in writing, to chair the review. He or she is usually included in planning the review’s technical content and in preparing the agenda for the review. The panel chairperson is in contact with the TFHRC laboratory manager and/or the assessment program manager to prepare for this leadership role. The panel chairperson receives the Handbook for Expert/Peer Reviews, a Briefing Book that provides information about the operations of and research conducted by the TFHRC laboratory being reviewed, and any other items, as needed, to support his or her role as panel chairperson. The panel chairperson may lead any pre-review activities, such as a telephone conference call with the panel.

General Characteristics of the Panel Chairperson

The panel chairperson must be a highly credible individual from outside of the FHWA. He or she may be a laboratory manager or an expert in a discipline of importance to the laboratory being reviewed. He or she must be a good communicator and be able to synthesize information, elicit constructive discussion, and keep discussions on-topic and within reasonable time allowances. Most importantly, he or she must have time to commit to the review, and must be trusted by the laboratory manager and staff.

General Responsibilities of the Panel Chairperson

The main responsibilities of the panel chairperson are to

Help plan the agenda and direct the technical aspects of the review.
Moderate the review sessions with the panel, stakeholders, and customers.
Direct organization and content of the panel’s report.
Incorporate input from all members of the panel into the panel’s deliberations.
Act independently in the external expert/peer role.
Relate personal experience and offer considered opinion on the topics under discussion.
Be available for a minimum amount of follow-up discussions with the laboratory personnel based on the content of the panel’s report.
Meet the items listed under panel member responsibilities.

Panel Members

The expert/peer review panel members are invited, in writing, to be part of the panel. Like the panel chairperson, they are provided with the Handbook for Expert/Peer Reviews and a Briefing Book that provides information about the operations of and research conducted by the TFHRC laboratory being reviewed. All panel members also receive an agenda, logistics fact sheet, and other administrative and technical information necessary for participating in the review. Review panel members attend all of the sessions of the review including the closeout session.

General Characteristics of Panel Members

Review panel members must be able to encourage, criticize constructively, and provide positive feedback on laboratory accomplishments. Panel members must be trustworthy, able to understand the laboratory’s operations and research quickly, contribute cogently to and synthesize discussion, communicate well, have time to commit to the review, and possess expertise that contributes to the review. All panel members must be motivated to participate and contribute their expertise to enhancing the laboratory’s business.

General Responsibilities of Panel Members

Main responsibilities of panel members are to
Act independently in the external expert/peer role.
Relate personal experience and offer considered opinions on the topics under discussion.
Be fully engaged in all of the review sessions.
Conduct interviews with customers and stakeholders.
Synthesize facts gathered during the review and contribute to the expert/peer review panel’s report.
Discuss focus area issues cogently and concisely.
Seek the best courses of action for the laboratory.
Present some portion of the panel’s report during the closeout session with the TFHRC senior managers, as requested.

Panel Meetings

Review panel meetings are held at the TFHRC. General sessions consist of presentations from laboratory staff, interviews with customers and stakeholders, and panel deliberations. In addition, the review panel tours the laboratory’s facilities and may visit other facilities and operations relevant to the review proceedings. The panel also meets in executive or closed sessions. These meetings allow the panel to discuss and synthesize the information gathered from the tours, presentations, literature, and interviews, and to prepare its report.

Panel Activities

Background Materials

The laboratory manager and Laboratory Assessment Program staff compile a Briefing Book that is sent to each panel member prior to the expert/peer review. The Briefing Book is a collection of materials about the laboratory, specifically prepared for the panel members. The Briefing Book is designed to rapidly acquaint the panel with the TFHRC and the resources, operations, and research of the laboratory being reviewed. The material is sent to each panel member on a CD in advance of the laboratory assessment. Through the Briefing Book, panel members can get a general understanding of the laboratory and its work before they arrive for the review. In addition, a notebook with the same materials, a printed version, is provided to each of the expert/peers upon his or her arrival.

The Briefing Book is typically organized to tell the following story about the work of the laboratory: "Within the context of the mission of the agency, we utilize these resources (human resources, financial resources, specialized facilities, and equipment) to conduct this research, to produce these products, for these customers, who report these success stories."

The Briefing Book generally contains

FHWA and TFHRC overview material (vision, mission, roles).
Identification of focus area (if any).
Detailed information about laboratory resources.
- Biographical information on the laboratory staff and contractors.
- Laboratory budget information.
- Descriptions of specialized facilities and equipment.
- Existing laboratory accreditation information.
Highlights from past reviews, if applicable.
Detailed information about the research being conducted by the laboratory.
Research results, products, and services.
Technology transfer and deployment activities.
Customer and stakeholder information.
Success stories.

Other material sent to the panel in advance of the laboratory expert/peer review may include the panel member’s biographical information and contact information, logistical information – hotel and transportation arrangements, and other information regarding the review.

Pre-Review Telephone Conference Call

The expert/peer review chairperson may conduct a pre-review telephone conference call with the panel to discuss the purpose of the review and clarify any issues the panel may have about the review. This call is conducted approximately two weeks before the review and after all of the panel members have received their Briefing Books. The laboratory manager, the Laboratory Assessment Program Manager, and the review facilitator are included in this conference call. This call helps to form a working team from the individual peer experts invited to be panel members.

Welcome Dinner

A dinner to welcome the panel members is held the evening before the initial expert/peer review introductory sessions. The panel members, TFHRC senior managers, the laboratory manager, the Laboratory Assessment Program Manager, and the review facilitator attend. The dinner provides an opportunity for panel members to meet the senior TFHRC staff and meet one another prior to the beginning of the review.

Introductory Sessions

The review panel begins work by conducting a session with only the panel members, often at a panel breakfast. During this executive session, the panel chairperson

Outlines the purpose and objectives of the review.
Discusses his or her vision for the conduct of the review.
Defines assessment criteria and procedures.
Assigns activities for individual panel members, as appropriate.
Answers any questions from the panel.

Panel member assignments may be to

Interview stakeholders or customers.
Lead discussion on a technical or operational topic.
Prepare summaries of key areas of the review for the panel report.

The first general session of the review includes introductions and a welcome by TFHRC management; goals and description of review procedures; an overview presentation about the operations of the laboratory to be reviewed and the research it accomplishes; and a tour of the laboratory and, if appropriate, other associated facilities. As the review continues, the expert/peer review panel gathers more information through interviews with laboratory staff, customers, and stakeholders.

The Remainder of the Review

During the afternoon of the first day of the review, after the introduction to the review, the overview of the laboratory and its work, and the laboratory tour, the review panel takes time to prepare its plan for the remainder of the review. The panel is guided by the assessment criteria developed for the review and uses these criteria in its planning. This session allows the panel to identify the specific interviews to be conducted, to determine the additional fact-finding it must perform, to organize its time for synthesis of information, and to plan the contents of the final report.

Existing Accreditations – The panel is to consider existing laboratory accreditations within the laboratory’s field of science. These accreditations may have similar assessment requirements to those of the expert/peer review. The expert/peer reviews will consider such laboratory accreditation processes and requirements and avoid duplicating assessment activities, when possible.

Focus Area – If a focus area has been identified, the panel incorporates the area into its plan for the review. Focus areas originate most often from the manager of the laboratory being reviewed. The focus area directs the panel’s attention to issues of the laboratory’s particular concern and on which the expert peers may provide unique insight. A focus area may deal with program, technical, or administrative issues. The focus area, along with laboratory operations and research conduct, all should be considered within the context of the established assessment criteria.

Fact-Finding and Investigation

One of the most productive tools for fact-finding is conducting interviews with staff, customers, and stakeholders. Interviews can be conducted in various forums and offer an excellent opportunity to obtain information on an informal basis. They can be used to fill in information gaps and obtain insight not present in the formal records. An interview conducted informally but privately, will allow an employee or contractor to share information more freely. It is important that both parties (interviewer and interviewee), maintain the common goal of seeking information, ideas, and verification throughout the interviewing process. Following the introductory briefings and review of briefing materials, panel members should identify individuals to be interviewed. Panel members are encouraged to interview managers, key laboratory staff, and key laboratory contract employees. Panel members are also encouraged to interview customers and stakeholders; this is generally arranged in a focus group that takes place during a lunch. Panel members may also want to call customers and stakeholders identified in the Briefing Book or call other industry contacts to get unbiased, independent feedback.

Site Visit – A site visit may be arranged as part of, or in conjunction with, the review. This visit allows the panel members to view a research project, demonstrate laboratory work products in development, or show implementation of research results, however, not all reviews can accommodate the time this activity requires.

Synthesis of Information and Report Preparation

After completing all of its fact-finding activities – review of briefing materials, technical reports and other lab products, as well as interviews and discussions with laboratory representatives, stakeholders, and customers, and other activities – the panel will need time to synthesize the material it received in order to write its report. The report includes the observations and recommendations of the entire panel. The report provides review objectives, the process followed by the review panel, and a discussion of the panel’s observations, which might include items such as strengths, key issues, and opportunities. The report summarizes the views of the panel as a whole (including any dissenting views). If necessary, minority opinions are noted in the report. The panel report is written during the review and completed before the panel’s departure.

Report Template – a report template is provided to the panel in advance to facilitate the work of preparing a report. The report template includes background information about the laboratory and participants in the review. The template documents that the panel members conducted the review on the specified dates and based on the given information that is described, have made the listed observations and recommendations. Panel members are encouraged to identify strengths and best practices, as well as areas for improvement. Bullet point lists are strongly encouraged to ensure that key points are captured. The use of bullet points for recommendations also facilitates the creation of a follow-up Action Matrix by the laboratory staff. Of course, the panel may elaborate on any of the bullet points. They are also free to edit any of the background material provided in the report template. The report template is intended to facilitate the preparation of the report and is not intended to constrain or restrict the panel’s observations. (See the Appendix for a sample panel report format).

The panel needs to consider how many recommendations to bring forward and whether and how to prioritize them. In developing recommendations, the panel should aim at solving the identified problems, evaluate ideas carefully, and consider the level at which the ideas can be implemented. The panel should bear in mind that recommendations made at the laboratory level may be most easily implemented.

The panel report is discussed with the laboratory manager and the laboratory Federal and contract staff before it is presented to senior management at the closeout session. At that time, the panel has the opportunity to receive feedback on its observations and to correct any misinterpretations or errors in its assessment conclusions. The session with the laboratory staff also serves as a briefing regarding the contents of the expert/peer review report, as well as an opportunity for fact checking. This briefing provides the staff with a first hand view of the work of the panel – providing an important communications function – the staff is made aware of what information and issues will be brought forward to management. This open communications is an encouragement and support for the staff.

Presentation of the Panel Report to Management at the Closeout Session

The closeout session is an opportunity for the review panel to present its observations and recommendations to FHWA management in a face-to-face session, thus ensuring that the recommendations are heard and understood by senior management so that questions can be answered and misunderstandings avoided. All applicable managers are included – those whose areas were reviewed or will be affected by the recommendations, i.e., the FHWA managers who are in a position to decide upon and champion recommended changes. The group typically includes, in addition to panel members, the Associate Administrator, and selected FHWA managers, including the laboratory manager, his or her Office Director, the Laboratory Assessment Program Manager, and the review facilitator.

The panel chairperson officially transmits the panel’s written report to the Associate Administrator, and leads the report-out and discussion. He or she may wish to invite other panel members to present or respond to questions related to specific recommendations. All attendees of the closeout session receive a copy of the panel's report. The closeout session is the last formal activity of the review.

Action Matrix and Follow-up Activities

Although the closeout session is the last formal activity of the review panel, the laboratory assessment is not really complete until the agency considers and addresses the panel’s observations and recommendations. The laboratory manager prepares a matrix, which includes each of the panel’s observations or recommendations and details the associated offices, people, and resources required for implementation. The draft Action Matrix is prepared in conjunction with the TFHRC office director and headquarters program officials who will be responsible for implementation, and is formally approved by the Director, TFHRC . The Action Matrix is updated every six months and provides a periodic report of progress on the implementation of the action items. As an additional follow-up activity, the Action Matrix is sent to the expert/peer review panel members. This distribution serves as an accounting of the laboratory’s response to the panel’s observations or recommendations. This follow-up process allows the laboratory to implement improvements in a deliberate and measurable way during the four-year period between reviews.

Public Disclosure and Summary of Laboratory Assessments

The FHWA recognizes that the disclosure of the panel members and the substance of their comments can strengthen public confidence in the Laboratory Assessment Process. The FHWA discloses the names and affiliations of the panel members and a summary of their comments but does not attribute specific comments to specific reviewers. Summaries of the reviews are provided on the TFHRC web site, along with information on how the TFHRC is responding to recommendations.

General Information for TFHRC Laboratory Participants

The laboratory assessment process requires specific information distribution and excellent communications among the participants in the expert/peer review and among those who will be affected by its conduct. The review manager is responsible for facilitating proactive communications among all participants in the review, including customers, partners, and stakeholders.

Laboratory Personnel

The TFHRC laboratory manager and staff are responsible for providing information about the laboratory to the expert/peer review panel as requested. In addition, laboratory personnel may be interviewed by the expert/peer review panel. All laboratory participants will receive information regarding their role in the conduct of the review before the actual review, and may be included in pre-review communications with the review panel, such as a telephone conference call.

Associate Administrator for RD&T and TFHRC Leadership

Ongoing, open communication with the Associate Administrator for RD&T and other TFHRC leaders is a critical element of the review. Regular status meetings with these managers are encouraged during review planning. The availability of the Associate Administrator and management is an important aspect for scheduling the review’s closeout session.

TFHRC Training Opportunities

A short training seminar is available to familiarize TFHRC laboratory managers and personnel with the conduct of an expert/peer review. The seminar explains review benefits to the TFHRC laboratories, clarifies review process expectations, educates participants about their roles and contributions to the review, and provides mentors (people who have participated in a past review) for newcomers to the process.

Preparatory Activities for TFHRC Laboratories

The following list identifies preparatory activities that should be considered by the laboratory being reviewed.

Review specific goals of the laboratory assessment process. Consider identifying a focus area for the review panel to address. (Consideration should be given to issues of particular concern on which the expert/peer review panel may be able to provide unique insight).
Prepare the content of the Briefing Book, which will be distributed to the review panel by the Office of Program Development and Evaluation. The material sent to the panel should help them understand the basics of the laboratory’s research and administrative activities. All materials should be clear, consistent, and easy to understand for individuals not familiar with the laboratory.
Arrange/develop an overview presentation that explains research activities at the laboratory.
Arrange a tour of the laboratory.
Define roles of laboratory staff who will be involved with the review.
Recommend stakeholders and customers who should participate in the review.
Identify activities and outcomes that best demonstrate the laboratory’s commitment to quality.
Identify activities and outcomes that best demonstrate the laboratory’s adherence to high research performance standards.

Questions for Consideration by the Expert/Peer Review Panel

The following questions are provided for the panel as it prepares for its fact-finding and investigation activities. These questions are given to stimulate thought and not intended to limit the lines of inquiry. The questions are not a required list, and panel members are not expected to systematically require answers for each question. The questions reflect the items listed in the Assessment Criteria. (See page 3.)

Relevance

Does the research support the agency’s mission and strategic goals?
Is the research based on direction specified in FHWA multi-year program plans?
Are appropriate scientific and technical objectives being posed, taking into consideration stakeholder and customer needs, laboratory strengths, and time horizon for the research being accomplished?
What process exists to incorporate needs of stakeholders and customers – is this effective; are customer needs being met?
Is there an appropriate balance between meeting customer needs and long-term vision?
Are Federal and contract laboratory staff able to articulate relevance of their work to the agency’s goals?

Quality

Contracts and funding
- Does the TFHRC maximize quality through the use of clearly stated defensible methods for awarding the contracts?
- Is there a fully documented process that maintains quality for distribution of funds for noncompetitive research performance?
- Is the funding appropriate for the objectives?
Are Federal staff and contractors held accountable for cost, schedule, and performance results?
Output/Outcomes
- What quality assessment of the research operations, products, and deliverables (such as reports) is conducted? Are comparative methods used such as best practices application, expert/peer reviews, and benchmarking?
- Are quality guidelines or sound statistical methods used in the conduct of the research?
- Is the research protected from unauthorized access, corruption, and revision?
- Is the scientific or engineering quality of the work comparable to similar efforts at similar institutions (e.g. other Federal laboratories and research facilities)?
Staffing
- What are the qualifications and capabilities of the laboratory manager?
- What are the qualifications of the scientific and engineering staff?
- How is the expertise of the staff, Federal and contractor, maintained?
- What is the condition of staff morale?
What is the state of the equipment and facilities, and are these sufficient to conduct research that meets objectives?
Are administrative processes effective and efficient?
Are there quality improvement mechanisms to assure continued excellence?

Performance

Are research activities managed in a manner that produces high quality, identifiable results, utilizing research procedures and practices that comply with or exceed accepted standards for performance and reproducibility?
- How are the projects and researchers selected (is it a merit based, peer-reviewed, competitive process)?
- Does the research build on work already done or being conducted elsewhere?
- Have the investigators leveraged the work of leaders in the field?
- Are partnerships, if any, well-chosen and managed?
Are the research activities guided by established high-priority research objectives with performance outputs and milestones that show how the outcomes will be reached?
- How well crafted are the project plans/designs?
- Is the use of the laboratory experiment, modeling, simulation, and/or field tests appropriate, and how well are these integrated?
- Is the choice and use of the equipment appropriate?
- Are there appropriate output and outcome measures, schedules, and decision points?
- Is data collection and analysis rigorous and robust?
- Are the research conclusions supported by the results?
- Are the ideas for further study reasonable?
Do performance metrics exist that explain how outcomes will be reached?
Are termination procedures and appropriate exit strategies clearly defined for research projects?
Does the work being examined appear to be considered significant?
Has past research produced identifiable results?
Where have the laboratory’s research results been published?
What patents, licensing agreements, and software, have been produced?
What technology transfer activities have been accomplished, and how have the results been deployed?

Glossary

Assessment Criteria: The standards by which laboratories are measured by expert/peers during expert/peer reviews. The assessment criteria are quality, performance, and relevance. Particular emphasis is placed on quality and performance.

Briefing Book: A collection of materials providing information about the laboratory and the research it conducts that is prepared for and distributed to each of the expert/peers before his or her arrival for the review at the TFHRC.

Closeout Session: A meeting of the expert/peer review panel with the Associate Administrator, selected managers of the TFHRC, including the Laboratory Assessment Program Manager, and the review facilitator. The session is an opportunity for the review panel to present its observations and recommendations, discuss them with TFHRC managers, and transmit its written report to the Associate Administrator. This is the last formal session of the review.

Customers, Partners, and Stakeholders: The offices and organizations that rely on the results produced by the laboratory being reviewed or that have a stake in the success or failure of the activities performed by the laboratory.

Existing Laboratory Accreditations: Accreditations within its field of science received by a laboratory; these accreditations may have similar assessment requirements to those of the expert/peer review. Some laboratories at the TFHRC are accredited by respected organizations within their area of science. The expert/peer reviews will consider such laboratory accreditation processes and requirements and avoid duplicating assessment activities, when possible.

Expert/Peers: Technically qualified independent professionals who are invited to review the research activities of a TFHRC laboratory.

Expert/Peer Review: An independent assessment of the quality, performance, and relevance of a TFHRC laboratory. Reviews are conducted through up to three-day meetings of three to five expert/peers. These expert/peers visit the laboratory, discuss the research activities with the scientists and technical personnel, and formulate their opinions, which are documented in a report to TFHRC senior management.

Expert/Peer Review Panel: The individuals that perform the independent review of the laboratory activities. There are three to five visiting expert/peers on a review panel, which includes a panel chairperson. The list of panel members is developed based on suggestions from laboratory managers, customers, and stakeholders, and is vetted by an external advisory source, such as the TRB’s RTCC.

Expert/Peer Review Panel Chairperson: The leader of the expert/peer review panel. This person is an independent expert/peer who is external to the Agency.

Expert/Peer Review Panel Report: The report of observations and recommendations from the expert/peer review panel, which includes the considered thoughts and opinions of the entire panel.

Focus Area: An area that will receive particular attention by the expert/peer review panel during the conduct of the review. A focus area may be identified by the TFHRC laboratory being reviewed, or during the review by the review panel, or others such as customers and stakeholders.

Laboratory Assessment Program Management: The TFHRC Office of Program Development and Evaluation manages the program. In its capacity as program manager, the office coordinates and facilitates all aspects of the program.

Laboratory Assessment Process: The method of expert/peer review that is used to facilitate enhanced performance and quality in the research activities conducted at the TFHRC. Assessments are accomplished through periodic laboratory reviews by expert/peers. The process is continuous with each laboratory being assessed every four years.

Office of Program Development and Evaluation: Develops and executes policy, budget, program management, and administrative mechanisms to enable a nationwide R&T program to be carried out in cooperation with FHWA partners. This includes initiatives in the areas of strategic planning and quality, outreach and consultation, legislative development and interpretation, R&T budget development, and program monitoring and evaluation. This office is responsible for the Laboratory Assessment Program.

Research and Technology Coordinating Committee (RTCC): The RTCC provides continuing guidance and advice on the nation’s highway research program. It is composed of members knowledgeable in transportation research and technology. It provides an advisory function for the laboratory assessment process. In it’s advisory role it reviews and comments on the assessment criteria and the composition of expert/review panels.

Review Facilitator: At the TFHRC’s discretion, review panels may have a review facilitator who is independent and external to the Agency, who will advise the review panel and act a liaison to the FHWA. Such a facilitator knows about the conduct of expert/peer reviews at the TFHRC and assists the review panel in its duties.

Appendix

Supporting Materials

SUPPORTING MATERIALS

Sample Agenda

Conflict of Interest Disclosure Form

Multimedia Model Release Form

Logistics Fact Sheet

Sample Panel Report Format

Program Contacts

Sample Agenda

Subject Laboratory

Date

Monday
6:30 p.m.	Welcome dinner with selected senior managers -- local restaurant
Tuesday
7:00 - 8:00 a.m.	Panel breakfast at hotel – Panel and Facilitator
8:00 a.m.	Depart for TFHRC, Turner Building lobby sign in
8:15 - 8:45 a.m.	Panel check in to Room F-100 (room reserved for panel use)
8:45 - 9:00 a.m.	Welcome and introductions, Room T-104/5
9:00 - 9:30 a.m.	Goals of Review, Assessment Criteria, Process Description
9:30 - 9:45 a.m.	RD&T Overview
9:45 - 10:00 a.m.	Break
10:00 - 12:00 p.m.	Laboratory Overview and Tour
12:00 - 1:00 p.m.	Lunch, Room F-100
1:00 – 2:00 p.m.	Technical discussions with laboratory manager
2:00 – 3:30 p.m.	Panel only session – preparation for interviews, technical discussions, and telephone interviews with stakeholders/customers, Room F-100
3:30 - 4:00 p.m.	Break
4:00 - 5:00 p.m.	Fact-checking with laboratory manager, Room F-100
5:30 p.m.	Panel dinner at local restaurant
Wednesday
8:00 a.m.	Panel arrive at TFHRC, Room F-100
8:00 - 11:30 a.m.	Fact-finding – interviews, discussions, meetings with laboratory manager and staff, and/or telephone interviews with selected customers and stakeholders (Rooms F-100, F-208, T104/5 and laboratory space)
11:30 - 11:45 a.m.	Break
11:45 - 2:00 p.m.	Luncheon discussions with selected customers or stakeholders, Room F-208
2:00 - 3:00 p.m.	Panel interviews with laboratory staff, Room F-208, F-100, T-104/5 and laboratory
3:00 - 3:45 p.m.	Break
3:45 - 4:30 p.m.	Panel only meeting, begin synthesis of material for report, Room, F-100
4:30 - 5:30 p.m.	Question and answer session with Office Director and Associate Administrator, Room F-100
6:00 - 8:00 p.m.	Panel reconvenes at hotel for light dinner and report preparation
Thursday
7:30 a.m.	Arrive at TFHRC, Room F-100
7:30 - 10:00 a.m.	Panel report writing, Room F-100
10:00 - 10:30 a.m.	Review observations with laboratory manager, Room T-104/5
10:30 - 11:00 a.m.	Review observations with staff, T-104/5
11:00 - 11:30 p.m.	Finalize report, Room F-100
11:30 - 12:30 p.m.	Closeout session with Associate Administrator and selected managers
12:30 p.m.	Adjourn

Conflict of Interest Disclosure

TFHRC Laboratory Review Panel

December 2004

NAME: ________________________________________________ TELEPHONE: ___________________________

ADDRESS: _________________________________________________________________________________________

__________________________________________________________________________________________

EMAIL ADDRESS: __________________________________________________________________________________

CURRENT EMPLOYER: _____________________________________________________________________________

LABORATORY BEING REVIEWED: __________________________________________________________________

Please answer the following questions regarding relevant organizational affiliations, if any. Information is "relevant" if it is related, and might reasonably be of interest, to others concerning your knowledge, experience, and personal perspectives regarding the subject matter and issues to be addressed by the panel activity for which this form applies.

1. Do you have any current relevant affiliations with FHWA and/or with any organizations or individuals under contract to FHWA? This includes relevant business relationships, as an employee, owner, officer, director, consultant, etc., and any relevant remunerated or volunteer non-business relationships, e.g., professional organizations, trade associations, public interest or civic groups, etc., with FHWA.

NO ______YES ______

2. Do you have any pending and/or potential affiliations with FHWA and/or with any organizations or individuals under contract to FHWA? This includes relevant articles, testimony, speeches, etc.

NO ______YES ______

3. Is there any relevant aspect of your background or present circumstances not addressed above that might reasonably be construed by others as affecting your judgment in matters within the assigned task of the committee or panel on which you have been invited to serve, and therefore might constitute an actual or potential source of bias. NO ______YES _____

If the answer to any question was "YES", please explain. Additional sheets or a CV may be attached in answering the questions. Please provide a brief description of relevant positions with the organizations or groups with which you may be closely identified or associated. Also provide the date, title, and publication (or relevant representative examples, if numerous) of any articles, speeches, and testimonies.

After completing the form sign, date, and mail to:

Turner-Fairbank Highway Research Center
Attention: Deborah Frank
Laboratory Assessment Program Assistant, HRPD-1
Office of Program Development & Evaluation, Room T-305
6300 Georgetown Pike
McLean, VA 22101-2296

The form will be returned to you at the conclusion of the Panel Review.

SIGNED: _____________________________________________________________ DATE: _____________________

FEDERAL HIGHWAY
ADMINISTRATION

Multimedia Model Release Form

Name: Date:

Location:

I hereby grant to the Federal Highway Administration (FHWA) the absolute and irrevocable right and permission, in respect of the photographs or audio or videotape recording and their transcripts, that it has taken or has had taken of me or in which I may be included with others, to copyright the same, in its own name or otherwise (and assign my rights throughout the world in such photograph and audio and video recordings and their transcripts), to use, reuse, publish, and republish, and otherwise reproduce, modify and display the same, in whole or in part, individually or with other photographs, and with any copyrighted matter, in any and all media now or hereafter known, for illustration, promotion, art, advertising and trade, or any other purpose whatsoever; and to use my name in connection therewith if it so chooses.

I hereby release and discharge FHWA from any and all claims and demands arising out of, or in connection to, the use of the photographs, including without limitation any and all claims for libel or invasion of privacy. FHWA may sell, assign license, or otherwise transfer all rights granted to it hereunder.

This authorization and release shall also inure to the benefit of the specific legal representatives, licensees, and assigns of FHWA, as well as the staff representative(s) (if any) for whom it took the photographs.

I am of full age and have the right to contract in my own name. I have read the foregoing and fully understand the contents thereof. This release shall be binding upon me and my heirs, legal representatives and assigns. I further release FHWA from any responsibility for injury incurred during the photography or audio or videotaping session.

Signed: Date:

Printed Name:

Address:

City, State, Zip:

Phone: Fax:

Please fax the completed form to 202-493-3170.

If necessary, the form may be mailed to D. Frank, Turner-Fairbank Highway Research Center,
6300 Georgetown Pike, Room T-201, McLean, VA 22101.

Logistics Fact Sheet

The following information is provided to the expert/peer review panel members when they agree to participate in the review. It is also included in the Briefing Book or transmittals of information to the panel before the review.

Panel: The names and information about panel members and the panel chairperson of the expert/peer review are provided.

Location and Facilities: Directions to the TFHRC and the security admittance process are detailed. In addition, a meeting room, computer resources, and other materials or tools to facilitate discussions are available.

Lodging: Panel members are responsible for making their own hotel reservations. A block of rooms at a local hotel will be held for panel members. Panel members will be notified by email regarding the local hotel and be reminded to make their reservations. Panel members should arrange to arrive the evening before the review begins, in time for the welcome dinner scheduled with TFHRC senior managers.

Transportation: Panel members also are responsible for making their own air or rail transportation reservations. Assistance will be provided in making these reservations. Panel members will be contacted by email. The TFHRC will reimburse travel costs and/or the use of personal automobiles at the standard Federal rate.

Ground Transportation: Arrangements for local transportation between the airport or train station and the TFHRC will be provided. Expert/peers should not require a car during the review.

Meals: Breakfast is included in the cost of the room. Lunches are provided each day during the review. At the beginning of the review, panel members pay for the lunches and are reimbursed for these costs submitted on an expense voucher. Dinners may be arranged and are designed to allow time for panel deliberations. Special dietary arrangements can be accommodated.

Dress: Dress for the entire review is business casual.

Expenses: The TFHRC will reimburse all meal and lodging expenses. Other expenses, such as meals during travel, also will be reimbursed, subject to limits. Expert/peers must submit vouchers and receipts to the FHWA for reimbursement of all covered expenses. Expense vouchers and directions for submitting them are provided.

Sample Panel Report Format

The report is a cogent description of the expert/peer review panel’s understanding of the laboratory’s activities and the panel’s observations regarding the laboratory. Bullet points, short descriptive paragraphs, and other methods to express information clearly and concisely are encouraged. A template showing a proposed format for the final report is provided to the review panel. Elements included in the template are

Brief Overview of the Laboratory.
Objectives of the Review.
Assessment Criteria and Scope of the Review.
Focus Area, if identified.
Interviews with Customers/Stakeholders.
Common Themes Emerging from the Interviews and Discussions.
Observations.
Recommendations.
Minority Opinion, if necessary.

Program Contact

Debra Elston
Acting Director, Office of Corporate, Research, Technology, and Innovation Management

Office of Corporate, Research, Technology, and Innovation Management
Federal Highway Administration
Turner-Fairbank Highway Research Center
6300 Georgetown Pike McLean, VA 22101-2296
202-493-3181
E-mail: debra.elston@fhwa.dot.gov

Laboratory Assessment Process Handbook for Expert/Peer Reviews at the TFHRC Version 2.1