Skip to content U.S. Department of Transportation/Federal Highway AdministrationU.S. Department of Transportation/Federal Highway Administration

Office of Planning, Environment, & Realty (HEP)
Planning · Environment · Real Estate

HEP Events Guidance Publications Glossary Awards Contacts

TMIP Peer Review Program Evaluation

(PDF version)

April 27, 2009

Prepared by
Volpe National Transportation Systems Center, USDOT
For the Federal Highway Administration

Table of Contents

Executive Summary

This report summarizes the results of an evaluation of the Travel Model Improvement Program (TMIP) Peer Review Program. The purpose of the evaluation was to understand the value to transportation planning host agencies of a peer review of their travel demand model. The study involved interviewing modeling staff from eight metropolitan planning agencies that have hosted a TMIP-sponsored peer review of their travel demand model.

Major findings are detailed below, grouped into four categories: motivations for conducting a peer review, planning the peer review, implementing technical recommendations, and overall satisfaction with the process. Following the findings, recommendations for enhancing peer reviews and the peer review process are presented.

AGENCY MOTIVATIONS FOR CONDUCTING A PEER REVIEW

One of the major advantages of peer reviews is that each host agency can frame the meeting to address its unique modeling needs. Nonetheless, there were some common reasons for holding peer reviews:

PLANNING THE PEER REVIEW

Interviewees agreed that planning their peer review was very time consuming. However, they also agreed that good planning is a critical factor in a successful peer review. Planning tasks that interviewees found particularly important were:

IMPLEMENTATION OF RECOMMENDATIONS

The most common technical issues addressed in peer reviews were:

Agencies implemented approximately 70 percent of peer panelists' recommendations. The major reasons for not implementing recommendations were:

OVERALL SATISFACTION

Interviewees were very satisfied with their peer review and the peer review process. They felt that panelists were highly skilled and that the recommendations were appropriate and helpful. The post-meeting report was helpful in documenting these recommendations. Interviewees said that the peer review helped build their staff's modeling skills and help improve their travel demand model.

RECOMMENDATIONS FOR IMPROVEMENTS TO THE TMIP PEER REVIEW PROGRAM

The following recommendations are drawn from the interviews as well as from observations of the Volpe Center team.

Abbreviations

BMC
Baltimore Metropolitan Council
COMPASS
Community Planning Association of Southwest Idaho
DRCOG
Denver Regional Council of Governments
EWGCOG
East-West Gateway Council of Governments
NJTPA
North Jersey Transportation Planning Authority
SANDAG
San Diego Association of Governments
SEMCOG
Southeast Michigan Council of Governments
SCAG
Southern California Association of Governments
TMIP
Travel Model Improvement Program

I. Introduction

This report summarizes the results of an evaluation of the Travel Model Improvement Program (TMIP) Peer Review Program. The purpose of the evaluation was to understand the value to transportation planning host agencies of a peer review of their travel demand model. This report presents strengths and weaknesses of peer review meetings and the peer review program as described by modeling staff from agencies that have hosted a peer review. It also provides recommendations for improvements to the TMIP Peer Review Program and to aspects of the overall TMIP Program that could help grow the educational impact of peer reviews.

TMIP is a partnership between the Federal Highway Administration (FHWA) and the Federal Transit Administration (FTA). The program's goals are to:

Improved modeling techniques allow planners to generate better, more robust transportation forecasts that decision-makers can use when making important decisions about transportation services under various scenarios changes in population, economic growth, and land use.

TMIP's peer review component gives transportation planning agencies the opportunity to have their model reviewed by modeling experts from around the country. These experts make recommendations on how to proceed with model enhancements. Peer reviews are particularly valuable because each host agency customizes its peer review to meet its unique modeling concerns.

This study examined peer reviews conducted by eight metropolitan planning organizations (MPO) across the country. Section II of this report describes the methodology used in the study. Section III presents findings drawn from major themes in the interviews. Section IV presents recommendations for peer review program enhancements, followed by the conclusion in Section V.

II. Methodology

This evaluation was conducted by the U.S Department of Transportation (DOT), Research and Innovative Technology Administration (RITA), Volpe National Transportation Systems Center (Volpe Center) at the request of the FHWA Office of Planning, Environment, and Realty. It consisted of eight semi-structured telephone interviews with travel demand modeling staff from eight transportation agencies that have hosted peer reviews. Five of these eight agencies conducted their peer review after 2005 when the most recent peer review evaluation was conducted. These five agencies, therefore, have not been evaluated previously.

FHWA chose three additional agencies to provide a more balanced perspective of the program. Each of these additional agencies conducted its peer review at least 5 years ago, giving them substantially more time to address peer review recommendations than the five agencies that conducted peer reviews since 2005. This evaluation focused more on agencies that are in the process of developing their model rather than refining an existing model. Four agencies studied have conducted more than one TMIP-sponsored peer review. This evaluation focused on the most recent peer review.

Modeling staff from the following agencies were interviewed:

Interviews focused on:

The interview questions are included in Appendix A.

Table 1 provides basic information about these agencies and their peer reviews.

Table 1: Characteristics of Participating Agencies
Agency Metro Area No. of Peer Reviews Dates of Peer Reviews Primary Goal for Peer Review Modeling Platform
Baltimore Metropolitan Council Baltimore, MD 2 Sept. 23-24, 2004
Feb. 28, 2005
 
Evaluate and feedback on the effectiveness of the current model; identify short- and long-term model enhancements. TP+ 3.0; TRNBUILD 3.0.6 (transit)
Community Planning Association of Southwest Idaho Boise, ID 1 June 5- 6, 2007 Feedback on the technical aspects of the travel demand model including recommendations for components needing improvement. Cube Voyager and TP+
Denver Regional Council of Governments Denver, CO 2 Oct. 31, 2003
April 20, 2004
 
Identifying key planning issues and developing a vision for a new modeling system Migrated to TransCAD and Urbansim from MinUTP and DRAM/EMPAL
East-West Gateway Council of Governments St. Louis, MO 1 Dec. 7-8, 2006 Provide guidance on model fixes and enhancements for its compliance review. Cube, TRNBUILD
North Jersey Transportation Planning Authority Northern NJ, metro NYC 2 Oct. 27-28, 2005 Assess the plans for the development of the newly enhanced travel demand model and recommend both near- and long-term model enhancements. 4-step CUBE/Voyager TP+ with FORTRAN routine for mode choice
San Diego Association of Governments San Diego, CA 1 June 23-24, 2005 Provide guidance on near- and medium-term model enhancements to its existing four-step model. 4-step, TransCAD
Southeast Michigan Council of Governments Detroit, MI 1 Dec. 6-7, 2004 Assess the current travel demand model and recommend both near-term and long-term model enhancements. 4-step, TransCAD
Southern California Association of Governments Los Angeles, CA 3 Nov. 3, 2003
April 16, 2004
January 9-10,2006

 
Examine current model and focus on assessing the model updates, including trip distribution, mode choice, trip assignment; assess plans for model improvement and recommend near- and long-term enhancements, review validation targets for all model components. Migrating to TransCAD

III. Findings

This section lays out the common themes expressed by interviewees. They are divided into four categories: motivations for holding a peer review, planning the peer review, implementation of technical recommendations, and overall satisfaction.

A.MOTIVATIONS FOR HOLDING A PEER REVIEW

Agencies were looking to the peer review panelists for expert advice on the best approaches to improve their model's capability in high-priority areas. Common reasons for requesting a peer review included:

B.PLANNING THE PEER REVIEW

Planning a peer review requires significant effort on the part of the host agency. Interviewees made several observations about the process of planning the peer review:

C.IMPLEMENTATION OF RECOMMENDATIONS

Table 2 lists the most common topics for recommendation and their status of implementation. It shows that host agencies implemented approximately 70 percent of the panels' recommendations. In addition, agencies either have implemented or are considering implementing 23 percent of the recommendations. Seven percent were not addressed.

Agencies were most likely to implement recommendations related to basic model elements such as modeling boundaries and network revisions. Other frequently implemented recommendations were related to mode choice, validation, and calibration. Altogether, agencies implemented over 80 percent of the recommendations that address basic model elements.

Table 2: Implementation of TMIP Panel Recommendations
Agency/ Recommendation Topics Model Platform, Land Use Integration Activity-Based Modeling General Model Details/Data Needs Freight Component Mode Choice/ New Starts Validation/ Calibration/ Feedback
BMC         + - + o
COMPASS         + o - o
DRCOG o + + - + +
EWGCOG o     + o + +
NJTPA + o o     + +
SANDAG     o o + + +
SCAG         + + + +
SEMCOG o o + o + +

Key:

+
Implemented
o
Considered/partially implemented
-
Not implemented
Focus of discussion

Several recommendations were partially implemented or delayed because of:

None of the interviewees said they could not implement a particular recommendation because agency decision-makers disagreed with it. This is to be expected since most agencies vetted issues with management before deciding the charge to the peer review panel.

The list below presents details on the most common technical topics addressed in peer reviews:

Interviewees agreed that the technical recommendations were helpful and resulted in significant improvement to their models. However, one agency acknowledged that the comments and advice provided by the panel could be a "double-edged sword." With so many possible model enhancements, panels occasionally presented an overwhelming list of options that the agency could not even consider given staffing constraints and the maturity of its current model.

D.OVERALL SATISFACTION

The evaluation interviews asked about overall satisfaction with the peer review process. Interviewees were generally very satisfied with the process and made the following key points:

IV. Recommendations

The following recommendations are drawn from the interviews as well as from observations of the Volpe Center team.

V. Conclusion

The results of this evaluation indicate that the TMIP Peer Review Program offers an effective process for agencies to improve their travel demand models. Agencies found the program to be very valuable for improving travel modeling in their regions. The process was instrumental in providing a valuable forum for them to identify and address important regional model components. When asked what type of agency would benefit from the TMIP Peer Review Process, interviewees indicated that it would be good for any agency involved in travel modeling, particularly MPOs, transit agencies, and state departments of transportation.

Appendix A: Interview Questions

Evaluation Questions for Evaluation of TMIP Peer Review Program

Appendix B: Description of Peer Review Meetings

The following is a summary of the peer reviews held by the agencies included in this evaluation.

1. BALTIMORE METROPOLITAN COUNCIL (BMC)

Baltimore, Maryland

The Baltimore Metropolitan Council (BMC) held two peer review meetings. The first focused on the status of the BMC travel model improvement process and provided guidance on near-term and long-term model development issues including demographic forecasting, incorporating new model functions such as managed lanes and truck traffic, and proposed new mode choice model. At the second peer review meeting, presentations and discussion focused on BMC's work plan to implement changes to its model based on the panel's recommendations from the first meeting. Both meetings were held at the office of the BMC in Baltimore, Maryland, with the same panel members.

The TMIP peer review was requested by the Baltimore Regional Transportation Board (BRTB), the MPO for the Baltimore metropolitan region. BMC staff provides support to the work activities of the BRTB. The Baltimore metropolitan region has a population of about 2.5 million people and covers an area of approximately 2,260 square miles.

The Baltimore model utilizes the traditional four-step process of travel demand forecasting. TP+ software (version 3.0) is used for running the model, except for the transit pathbuilding mode TRNBUILD. The purpose of the peer review was to provide the decision-makers in the region the opportunity to have the model reviewed by a panel of experts to determine how the model could be improved to meet future policy needs. Specifically, BMC asked the peer review panel to:

Major recommendations from the second peer review meeting included:

In an August 2005 memorandum, BMC addressed its planned next steps to incorporate these action items in a future work plan. Our interview with BMC focused on the extent to which it had made progress in moving these recommendations forward.

2. COMMUNITY PLANNING ASSOCIATION OF SOUTHWEST IDAHO (COMPASS)

Meridian, Idaho
June 5-6, 2007
http://www.fhwa.dot.gov/planning/tmip/resources/peer_review_program/compass/

The Community Planning Association of Southwest Idaho (COMPASS) is the MPO for northern Ada and Canyon counties of Idaho, including the city of Boise. The area has a population of approximately 481 thousand people. At the time of the meeting, COMPASS had recently transitioned from a three-step to a four-step model, including a mode choice component that was borrowed and adapted from Wasatch Front Regional Council (Salt Lake City MPO). The purpose of the peer review was to provide COMPASS with feedback on the technical aspects of their travel demand model, including recommendations for components that needed improvement.

Recommendations included:

3. DENVER REGIONAL COUNCIL OF GOVERNMENTS (DRCOG)

Denver, Colorado

DRCOG is the MPO for the Denver area. The agency's planning area has a population of 2.5 million, with an additional growth of one million people expected over the next 20 years.

DRCOG held two peer review meetings in Denver, Colorado with the same panel members. The first meeting focused on identifying key planning issues and developing a vision for a new modeling system for the Denver region to include comprehensive redevelopment of all transportation and land use modeling elements. At the time of this meeting, DRCOG was in the early phases of replacing its existing land use and travel demand model with a fully integrated modeling system. The purpose of the second peer review was to discuss the elements to be implemented for DRCOG's model update.

The panel was charged with helping the DRCOG project team identify approaches to the development of an integrated model. The model improvement project, named the Integrated Regional Model Project, is essentially complete. As part of this project, DRCOG completed a Travel Behavior Inventory Project, a $1.5 million travel/activity/ demographic survey of the region.

Recommendations from the first meeting included:

DRCOG reconvened the peer review panel in April 2004 to address general model design, potential staging issues, and other practical considerations for model development based on recommendations from the previous meeting. The peer review panel recommendations included:

4. EAST-WEST GATEWAY COUNCIL OF GOVERNMENTS (EWGCG)

St. Louis, Missouri
December 7-8, 2006
http://www.fhwa.dot.gov/planning/tmip/resources/peer_review_program/ewgcg/

The East-West Gateway Council of Governments (EWGCG) is the MPO for the St. Louis, Missouri, metropolitan area. It has a population exceeding 2.4 million people and includes three counties in Illinois and four counties in Missouri.

The primary purpose of the peer review was to provide guidance to the EWGCG on model fixes and enhancements for its compliance review. Specifically, the panel was asked to:

The panel's main recommendations included:

5. NORTH JERSEY TRANSPORTATION PLANNING AUTHORITY (NJTPA)

Newark, New Jersey
October 27-28, 2005
http://www.fhwa.dot.gov/planning/tmip/resources/peer_review_program/njtpa/

The North Jersey Transportation Planning Authority (NJTPA) region encompasses an area of 4,200 square miles with a population of 6.5 million people. NJTPA is the MPO for the 13-county Northern and Central New Jersey region, including the city of Newark, comprised of 384 municipalities. It is the Nation's fifth largest planning region.

NJTPA requested that the panelists assess the current travel demand model and recommend both near- and long-term model enhancements including:

At the time of the peer review, NJTPA had a standard four-step transportation model using TRANPLAN software. However, one of the main purposes of the peer review was to receive comments on implementing a four-step CUBE/Voyager TP+ model with a special FORTRAN routine for mode choice, a process the agency had already begun. The panel recommended that the NJTPA keep its near-term focus on using the validated Phase I CUBE model for its conformity analysis and concentrate on improving basic model elements such as networks and travel times. Other recommendations made by the panel included:

6. SAN DIEGO ASSOCIATION OF GOVERNMENTS (SANDAG)

San Diego, California
June 23-24, 2005
http://www.fhwa.dot.gov/planning/tmip/resources/peer_review_program/sandag/

The San Diego Association of Governments (SANDAG), the MPO for the San Diego, California, metropolitan area, encompasses an area of 4,200 square miles with a population of 3 million people.

SANDAG requested that peer review panelists provide guidance on near- and medium-term model enhancements to its existing four-step TransCAD model including:

The panel felt that SANDAG's current model is consistent with the state of the practice, and the biggest question for SANDAG as it moves forward is how much to invest in revisions to the existing model versus the development of a new activity/tour-based model. The panel's primary recommendations included:

7. SOUTHEAST MICHIGAN COUNCIL OF GOVERNMENTS (SEMCOG)

Detroit, Michigan
December 6-7, 2004
http://www.fhwa.dot.gov/planning/tmip/resources/peer_review_program/semcog/

The Southeast Michigan Council of Governments (SEMCOG) is the MPO for Detroit, Michigan. The region has a population of approximately 4.9 million people.

SEMCOG requested that the panelists assess the current travel demand model and recommend both near-term and long-term model enhancements. SEMCOG has a relatively new TransCAD-based four-step travel demand model. The model includes consideration of external, commercial vehicle, and transit trips. It is based on a variety of data sources, including census information, a SEMCOG household travel survey, traffic counts, and a transit-on-board survey.

The first day of the meeting focused on SEMCOG presentations on the current travel demand model and its plans for model improvements. Overall, the panel felt that the current model represents the state of the practice, addressing time-of-day, commercial vehicle, and external trips well. For future model enhancements, the panel's recommendations included:

For the long term, the panel recommended considering implementing an activity-based model.

8. SOUTHERN CALIFORNIA ASSOCIATION OF GOVERNMENTS (SCAG)

Los Angeles, California

The Southern California Association of Governments (SCAG) hosted a total of three peer review meetings. The first two were held in November 2003 and in April 2004. Both meetings took place in Los Angeles, California, with the same panel members. The evaluation focused on the third meeting, held in January 2006, which had a different set of panelists.

The purpose of the third peer review meeting was for the panelists to examine SCAG's current model and assess model updates, including changes to the trip distribution, mode choice, and trip assignment phases. SCAG also requested that the panelists comment on the agency's plans for model improvement and recommend near- and long-term model enhancements as well as to review validation targets for all model components.

SCAG is the MPO for six Southern California counties in the Los Angeles metropolitan area. The region encompasses an area over 38,000 square miles and includes a population exceeding 18 million people. The current SCAG model follows the traditional four-step modeling structure with three ancillary models feeding into network assignment, including an external trip model, a regional airport demand allocation model, and a heavy duty truck (HDT) model. The panel felt that SCAG 's current four-step model is consistent with the state of the practice, particularly the plan for the freight model, the use of the "strategic work trip" purpose in trip distribution, and the use of four time periods in traffic assignment. It also indicated that SCAG has done well in data collection efforts and recommended that SCAG make its impressive survey datasets available to other agencies.

The TMIP peer panel made a number of recommendations for SCAG to consider, including:

Appendix C: Host Agency and Peer Review Panel Members

H - Host agency contact person
P - Panelist

Last Name First Name Organization BMC COMPASS DRCOG EWGCOG NJTPA SANDAG SEMCOG SCAG
Baber Charles Baltimore Metropolitan Council (BMC) H              
Bhat Chandra University of Texas at Austin       P P P   P
Blain Larry Puget Sound Regional Council           P    
Boyce David Northwestern University       P        
Bradley Mark Mark Bradley Consulting               P
Cervenka Ken North Central Texas COG P     P     P  
Chiao Kuo-Ann New York Metropolitan Transportation Council         P      
Crandall Mick Utah Transit Authority (UTA)   P            
Davidson William A. PBConsult, Inc.             P  
Diogo Robert North Jersey Transportation Planning Authority (NJTPA)         H      
Forinash Chris U.S. Environmental Protection Agency P           P  
Garry Gordon Sacramento Area COG           P    
John Jennifer Tri-County Metropolitan Transportation District of Oregon (TriMet)   P            
Killough Keith KLK Consulting P   P   P      
Lawton Keith Portland METRO     P          
Lee Deng Bang Southern California Association of Governments (SCAG)               H
May Jeffrey Denver Regional COG P              
McFarlane Bill San Diego Association of Governments (SANDAG)           H P  
Miller Eric University of Toronto P   P          
Morris Michael NCTCOG     P          
Outwater Maren Cambridge Systematics           P    
Pihl Eric FTA, U.S. Department of Transportation P         P    
Purvis Chuck Metropolitan Transportation Commission               P
Quackenbush Karl Central Transportation Planning Staff (Boston MPO)   P            
Replogle Michael Environmental Defense     P          
Rossi Thomas Cambridge Systematics P              
Rousseau Guy Atlanta Regional Council       P   P    
Ryan Jim FTA, U.S. Department of Transportation               P
Sabina Erik DRCOG     H         P
Schlappi Mark Maricopa Association of Governments (MAG) - Retired   P         P  
Shoaib Lubna East West Gateway COG       H        
Spear Bruce FHWA, U.S. Department of Transportation P             P
Spielberg Frank BMI-SG / Vanasse Hangen Brustlin, Inc. P P P P P     P
Taylor Stephanie Southeast Michigan Council of Governments (SEMCOG)             H  
Waldinger MaryAnn Community Planning Association (COMPASS)   H            
Updated: 03/28/2014
HEP Home Planning Environment Real Estate
Federal Highway Administration | 1200 New Jersey Avenue, SE | Washington, DC 20590 | 202-366-4000