TMIP Peer Review Program Assessment and Evaluation Report

1.0 Executive Summary

This report provides an in-depth assessment and evaluation of the Travel Model Improvement Program (TMIP) Peer Review Program. The purpose of the assessment and evaluation was to understand the value transportation planning agencies derive from hosting a TMIP sponsored review of their travel modeling tools and procedures. This assessment goes beyond past synthesis and evaluation efforts which only summarized the peer reviews conducted over the preceding one-two year time period.

This assessment and evaluation reviewed all twenty-eight peer reviews conducted since the program's inception in 2003 to identify the common trends, themes, and challenges faced by state departments of transportation (DOTs) and metropolitan planning organizations (MPOs) in the development, application, and improvement of their travel modeling tools and procedures. This report documents the generalized findings culled from each of the peer review final reports^[1]. In addition, this report presents a number of recommendations and proposed improvements to the TMIP Peer Review Program based on the reviewed documents and feedback elicited from recent host agencies.

1.1 TMIP Peer Review Program - Trends and Themes

To summarize the achievements of the program, the reviews were categorized along a number of different dimensions, including year, geography, agency size, peer panel composition and agency motivating factors. This exercise revealed that the program has performed a wide variety of reviews for large, medium, and small-sized agencies and has done a good job attracting peer panelists with diverse backgrounds and varied expertise.

The assessment also examined the peer reviews conducted to date to draw out the salient generalized lessons, observed model limitations, suggested recommendations as well as general policy and modeling trends. Not surprisingly, given the diversity of the host agencies, this report illustrates the broad patterns present in the practice of travel modeling.

The major trends and themes were identified by isolating the specific technical questions posed by the host agency to the peer panel as well as the prioritized model improvement recommendations presented to each host agency by the peer panels. The technical questions and recommendations were grouped into major topic areas in order to understand what topics have been most discussed.

A few key findings emerged from this look at the past TMIP peer review reports:

1. Most of the technical questions are centered on modeling guidelines, data collection/preparation and observed best practices, which is to be expected since the peer reviews are primarily model assessment exercises.
2. In general, the same kinds of questions are asked by host agencies regardless of agency size and the peer panels generally recommend the same kinds of model improvements.
3. Furthermore, the same kinds of questions and recommendations have been discussed over the years the program has been active with no clear chronological trends.
4. Most of the shorter-term recommendations in each review are centered on increasing the detail of the existing travel modeling tools (e.g. geographic, market segmentation, time of day, land use types, mode choice sets, etc.)
5. The inclusion of freight/commercial modeling, transitioning to activity-based demand modeling, dynamic traffic assignment (DTA) and microsimulation, and land-use modeling frequently appear as longer-term panel recommendations.

Finally, the assessment examined past TMIP synthesis reports and elicited feedback from recent host agency participants to evaluate the overall effectiveness of the TMIP peer review program. Participant satisfaction is very high and overwhelmingly agency staff and participants benefited from participation in the program. There is almost universal agreement that participating in the program has helped advance the modeling tools and procedures utilized by the host agency.

1.2 TMIP Peer Review Program - Recommendations

A list of recommendations for improving the TMIP peer review program has been compiled based on a comprehensive review of the twenty-eight peer reviews conducted since 2003, a review of past TMIP synthesis and evaluation reports, as well as 'user-experience' feedback elicited directly from past agency participants. The recommendations below are not in priority order.

1. More actively promote the program during years when interest appears low and continue promoting equal participation among large, medium and small-sized agencies especially in regions where TMIP peer review program participation has not yet occurred.
2. Continue to attract peer panelists with a diverse set of backgrounds and varied expertise without becoming too reliant on particular individuals and/or representatives from particular industry sectors. The peer networking and knowledge sharing offered by the program may be more important than the technical assessments.
3. Consider ways to make peer review meeting materials (e.g. presentation slideshows, model documentation) available to a broader audience beyond just the final report.
4. Consider having the TMIP program play a more active role before, during and after the peer review.
5. Consider having the TMIP program review and comment on published documentation to ensure it is reasonably thorough, prior to agreeing to sponsor a peer review.
6. Consider having the TMIP program review and help create the peer review agenda so it is clear and it is reasonable to expect that the agenda can be covered in the allotted meeting time.
7. Consider incorporating a "Preliminary Model Assessment" phase as the first phase of the TMIP peer review process, which takes place in advance of the meetings with the peer panel. The preliminary assessment would provide the "independent eye" often cited by host agencies and could help shine a light on elements of the travel modeling tools and procedures most deserving time and discussion during the formal in-person review.

[1] http://www.fhwa.dot.gov/planning/tmip/resources/peer_review_program/