TMIP Peer Review Program Assessment and Evaluation Report

3.0 Categorization of Past TMIP Peer Reviews

This assessment and evaluation sought to take a wide-angle view of the TMIP Peer Review Program. A great deal can be learned about the peer review program by simply looking back at what the program has achieved so far. This section of the report provides a historical glimpse of the program by categorizing the peer reviews conducted since the program's inauguration based on the following:

1. Peer reviews conducted by calendar year
2. Peer reviews conducted by geography
3. Peer reviews by panel participant affiliation
4. Peer reviews by agency size
5. Peer reviews by agency motivation

3.1 TMIP Peer Reviews by Calendar Year

Since 2003, twenty-eight peer reviews have been conducted.

Table 1 below identifies the peer reviews conducted from 2003 to 2011. The table notes which reviews have been included in a previous synthesis and/or evaluated in past FHWA sponsored reports.

1. TMIP Peer Review Program Synthesis Report^[2] dated November 2004 was prepared by the Volpe National Transportation Center (Volpe, 2004). This synthesis report summarized the first seven peer reviews conducted between 2003 and 2004.
2. TMIP Peer Review Program Synthesis Report 2^[3] dated September 2005 was prepared by the Texas Transportation Institute (TTI, 2005). This synthesis report summarized the next five peer reviews conducted between 2004 and 2005.
3. TMIP Peer Review Program Evaluation Report^[4] dated April 2009 was prepared again by the Volpe National Transportation Center (Volpe, 2009). This evaluation report summarized the four peer reviews conducted between 2005 and 2007. In addition, this report also interviewed four past host agency participants to address the program's overall effectiveness.

Table 1 also illustrates that twelve new peer reviews have been convened between 2008 and 2011 that have not yet been included in a synthesis report. This assessment and evaluation will not summarize these new peer reviews in the same manner as previous synthesis report efforts. The reader is encouraged to review the final reports from each of those agency peer reviews for specific details regarding those individual meetings. In contrast, this assessment will examine all twenty-eight peer reviews, the results of the program in totality, to identify common trends, themes, and challenges.

The number of peer reviews conducted each year has varied considerably. In 2004 for example, ten peer reviews were conducted, while only one peer review was conducted in 2007 and 2010 respectively. The variability by year is a function of host agency interest in the program, timing of application submittals, and scheduling, as opposed to program resource constraints. Since the program began there have been approximately three to four peer reviews per year on average. Figure 1 presents the number of peer reviews conducted during each calendar year.

Figure 1 TMIP Peer Reviews by Calendar Year

3.2 TMIP Peer Reviews by Geography

TMIP peer reviews have thus far been performed in nineteen states across the country, for two State Departments of Transportation (NCDOT, IaDOT) and twenty-six different Metropolitan Planning Organizations (MPOs). There are approximately 385 MPOs in the United States, so there are certainly opportunities to hold many more additional reviews. Some notable regions where peer reviews have not yet been requested include the Pacific Northwest (Portland, Seattle), parts of Texas (Dallas, Houston), the Gulf Coast and Florida, the northern Midwest, as well as portions of the Southwest (Arizona, New Mexico, Nevada). Where reviews have been conducted is of course a function of host agency interest and not determined by TMIP staff. Figure 2 illustrates the agencies that have participated in TMIP peer reviews since 2003.

Figure 2 TMIP Peer Reviews by Geography

3.3 TMIP Peer Reviews by Panel Participant Affiliation

The twenty-eight peer reviews conducted since 2003 utilized eighty-seven different panelists from a variety of backgrounds. The eighty-seven panel members included representatives from MPOs, State DOTs, private consulting firms, educational institutions (colleges/universities), federal government officials (FHWA, FTA), and other groups including environmental advocacy groups. Overall, the program has experienced very good participation and representation on the peer review panels among practitioners from different industry sectors (federal, state, local, private, and academic). Figure 3 illustrates the breakdown of panel representatives by affiliation type.

Figure 3 TMIP Peer Reviews by Panel Participant Affiliation

Because peer review panels are often comprised of nationally recognized practitioners and researchers in the industry, a number of individuals have participated in multiple peer reviews. Eighty-seven different individuals have participated in the twenty-eight peer reviews to date. Figure 4 illustrates panel participation among individuals who have participated in two or more peer reviews. Fifty-one individuals have only participated in a single peer review (not shown in Figure 4). Seventeen individuals have participated in two peer reviews, and nineteen individuals have participated in three or more peer reviews.

Figure 4 TMIP Peer Reviews by Panel Participant Affiliation

3.4 TMIP Peer Reviews by Agency Size

TMIP peer reviews have been convened in some of the largest metropolitan areas in the country (e.g. New York City, Los Angeles), as well as some relatively small planning areas such as Dubuque, Iowa and Logan, Utah. Over the eight-year period during which TMIP peer reviews have been conducted there's been an almost equal distribution of reviews across agencies of differing sizes (large, medium, small).

For this assessment and evaluation, the population ranges used to classify the size of the agency is somewhat subjective. The population ranges do however produce logical breakpoints when all the reviews are considered together. Table 2 below illustrates the population ranges used to determine agency size and the number of reviews performed. Figure 5 plots the host agency population and illustrates the large-size, medium-size, and small-size agency breakpoints. These same large, medium, and small agency size classifications are utilized throughout the remainder of this report. Figure 6 presents a zoomed-in look at the host agency population within each agency size category.

Figure 5 TMIP Peer Reviews by Agency Size

Figure 6 TMIP Peer Reviews by Agency Size Category (Large, Medium, Small)

3.5 TMIP Peer Reviews by Agency Motivation

Each TMIP peer review is unique in that the host agency has the freedom and flexibility to frame the meeting in many different ways. The goal is to develop a peer review meeting agenda and schedule that supports a discussion of the issues, challenges, and questions faced by the host agency and their own particular travel modeling needs. Nonetheless, there are some common stated reasons for why agencies express interest in and subsequently participate in the TMIP peer review program.

In general, there are four primary motivating factors for agencies when requesting a peer review:

1. The most common motivating factor is to have a group of technical experts conduct an independent model assessment of an agency's travel modeling assumptions, tools and procedures.
2. A relatively common motivation expressed by host agencies is to obtain feedback on specific short and long-term model improvements the agency can implement to enhance their existing modeling tools and procedures.
3. Agencies also look to peer review panels to guide model development efforts soon to be initiated at the host agency. An example is the DRCOG (Denver) peer review, which was convened to support development of a roadmap for the agency's migration to an activity-based demand model.
4. Finally, determining the accuracy, reliability, and defensibility of an agency's travel modeling tools is a frequent motivating factor. Agencies want assurances their travel forecasting methods reflect current industry best practices so the outputs of their forecasting tools can be used confidently in transportation policy, air quality, land use and economic development planning efforts.

Figure 7 presents the four primary motivating factors by agency size. The desire to have an independent model assessment and obtain a list of specific and prioritized model improvements are the two most frequently expressed motivating factors. Agencies regardless of size tend to identify the same motivating factors.

Figure 7 TMIP Peer Review Motivating Factors by Agency Size

[2] http://www.fhwa.dot.gov/planning/tmip/resources/peer_review_program/peer_review_program_synthesis_report.cfm

[3] http://www.fhwa.dot.gov/planning/tmip/resources/peer_review_program/peer_review_program_synthesis_report_2.cfm

[4] http://www.fhwa.dot.gov/planning/tmip/resources/peer_review_program/peer_review_program_evaluation.cfm

[5] During this analysis, peer reviews were convened in Detroit (SEMCOG) and Arizona (AZDOT).