Reasons to Monitor and Evaluate
Monitoring, evaluating, and reporting performance is a cornerstone part of PBPP. The purpose of PBPP is to ensure that results of previous investments and policies inform future decision-making so that transportation agencies can better understand approaches that work best given constraints and conditions. If data on performance are simply collected but not analyzed or used to influence future decisions, planning and programming is not performance-based. In order for performance to inform future decisions about investments and priorities, data must be collected, evaluated, and reported on an on-going basis.
Monitoring, evaluation, and performance reporting plays a critical role throughout the PBPP planning process by providing information to inform each step.
In discussing this critical element of performance-based planning and programming, it is important to make distinctions between different types of efforts:
It is especially important to distinguish between monitoring system-level performance and evaluating performance of specific strategies and investments. System level performance is impacted by a variety of factors, and certain factors have a larger impact than others. The distinction is important given the limited control that transportation agencies have over some outcomes. For instance, if greenhouse gas (GHG) emissions are one measure of performance, an agency may find that GHG emissions are increasing due to factors like rapid population and economic growth. However, this does not mean that the region's transportation investments in transit, demand management, and other strategies are not having an effect. It may simply be that economic and societal trends are having a greater impact.
Understanding overall system performance should influence future allocations of resources to improve performance. At the same time, information on the efficacy or cost-effectiveness of investments and strategies should influence choices about future investment priorities. PBPP is a cyclical/iterative process, and the information gained both through monitoring and evaluating performance will inform future cycles of decision-making.
Monitoring provides information on actual conditions on a periodic basis and allows for periodic assessment of whether targets have been or are likely to be attained. Monitoring system performance is an ongoing process, with data being amalgamated on various metrics and performance areas annually, quarterly, or even monthly or more often. Monitoring updates transportation officials with information about progress made toward goals relative to targets and resource allocation efforts.
One challenge to monitoring and evaluating performance is the difficulty of collecting data. Coordination between agencies can be especially helpful, given the vast amount of data being collected by different agencies and the role that operations data can play in providing very detailed information. Determining a monitoring strategy involves evaluation of ways to balance the need for frequent information updates with the need to use resources in the most effective manner. Monitoring plans address issues such as what is being tracked, what data need to be collected, who will collect it, how it will be collected, where it will be stored, and how it will be reported back to the end user. For instance, the CMP involves development of a monitoring plan to define the extent and duration of congestion, which defines what data will be collected and on what elements of the transportation network.
Utah DOT: GIS-based Tool to Enhance Cooperation and Decision-Making
Rather than looking at past data to evaluation how already-chosen investments would affect its performance, Utah DOT started its most recent long range planning cycle by developing a vision based on community values. This vision shaped Utah DOT's strategic goals in key performance areas; the organization then turned to data to fulfill its vision and set targets. In this process, UDOT developed a Geographic Information System (GIS) repository, UPlan, an interactive mapping program that supports the department through visualization of its data, tracking assets, providing stronger analysis, and better collaboration. An immediate benefit of UPlan is the utilization of public data, such as census surveys and FHWA information. Using this system has saved UDOT time and resources as information and data is more easily and efficiently shared. A recent method of using UPlan has been to use it to support making performance-based investment decisions. Five areas of performance measurement are currently available, they include: Safety, Congestion, Economy, Environment, and Asset Management. More non-traditional measures such as greenhouse gas emissions, land use, and transit access are currently being developed. Its use by various departments has resulted in UDOT gaining a clearer vision and an improved understanding of its needs and wants, as well as enhanced communication and cooperation within the DOT and with other state agencies.
For more information, see http://uplan.maps.arcgis.com/home/ and http://www.fhwa.dot.gov/planning/performance_based_planning/case_studies/utah/.
Determining the time horizon over which outcomes are measured is also important. Information collected on a yearly basis may be used in the consideration of adjusting the approach to achieving a particular target, while the target may remain unchanged except for during multi-year planning cycles. Many policies or investments can take years to implement fully and it may be many more years before they effect changes to travel behavior, safety and environmental outcomes, and other important goal areas. For example, it may be possible to improve safety over a few-year period, while it will take decades for land use changes to become widespread and have significant impacts on travel behavior. Thus, it is important to ensure that short-term expectations are not derived in a linear fashion from long-term targets- programs may take several years to implement and their impacts can increase over time. Supplementing data with information from departments on obstacles encountered, and external factors affecting performance, can help ensure that these factors are taken into consideration.
OKI: Monitoring Congestion Levels within the Transportation Network
This is an example of using regional-scale maps using color-coding to display measured speed and congestion data and metrics derived from these data. This figure shows an example from the Ohio-Kentucky-Indiana Regional Council of Governments for the Cincinnati Metropolitan Area. The data is derived from travel-time surveys conducted throughout the region over a period of three years.
Source: "OKI Congestion Management Process Findings and Analysis", Ohio-Kentucky-Indiana Regional Council of Governments, 2007.
Evaluation goes a step beyond monitoring and tracking and attempts to understand whether implemented strategies have been effective in contributing toward positive performance outcomes. Two types of evaluation may be conducted:
One approach to evaluation is for an agency to fund studies to measure the effectiveness of particular strategies or projects by examining conditions before and after, or with and without, a strategy of interest. For instance, a study could be conducted to quantify vehicle-miles-traveled (VMT) reductions or mode shifts of a transportation demand management (TDM) program, to quantify the speed improvements associated with traffic flow improvement projects, to examine the reduction in vehicle delay associated with operational strategies, to assess the lives saved through a safety campaign, or other similar types of impacts.
One example of a program evaluation includes efforts by the National Capital Region Transportation Planning Board (TPB) in the Washington, DC region to quantify the effectiveness of its Commuter Connections TDM program. TPB conducts a regional State of the Commute Survey, along with additional surveys such as a Guaranteed Ride Home Program survey and tracking of participation rates in programs, in order to analyze the vehicle travel reductions and air quality improvements associated with the program. The North Central Texas Council of Governments (NCTCOG), the MPO for the Dallas-Ft. Worth area, has conducted evaluations of its Thoroughfare Assessment Program, which involves retiming traffic signals on major corridors. An extensive data collection and system analysis process occurs for selected thoroughfares by means of assessing operational characteristics; estimating air quality benefits; and using performance measures such as travel time, delay, speed, and number of stops in order to develop and implement improvements. The results demonstrate reductions in travel delay and emissions.
Another approach is for an agency, such as an MPO, to develop guidance for evaluating strategies, and require local project sponsors to conduct evaluations of their projects and programs. Guidance can be provided on when an assessment should be done, what measures should be used, how data should be gathered, what methods should be used to analyze the data, and other aspects of evaluation studies. This approach is appropriate where partner agencies are responsible for implementation of CMP strategies, or where the MPO does not currently have sufficient resources to conduct studies. The East-West Gateway Council of Governments in St. Louis, Missouri provides guidance to localities on when a focused evaluation of strategy effectiveness is warranted, and how to conduct them. For example, if little is known about the actual benefits of the project, effectiveness evaluation can determine whether such strategies should be implemented more broadly (e.g., a trip reduction program that has not previously been used in the region), or if changes are required in the implementation of the strategy to produce the desired benefits.
Evaluation: NJTPA - Guidebook on Project Performance Measurement
The North Jersey Transportation Planning Authority (NJTPA) has created a Guidebook for Project Performance Measurement, which provides guidelines to the agency for evaluating the impacts of strategies chosen to inform future selection of packages of investments. The Guidebook addresses the use of various types of measures, both quantitative and qualitative, to evaluate investment decisions, while also noting the challenges to project-level evaluation. The Guidebook also delves in some cases into detailed instructions about how to evaluate various types of projects based on the mode(s) involved and performance area.
The way in which information about system performance in comparison to targets is communicated to policymakers and the public can have significant implications for support for an agency and its funding. It is important that public reporting of performance be done in a clear and concise manner. State DOTs, MPOs, RTPOs, and transit agencies across the board are feeling the pinch of fewer and fewer resources for quickly expanding needs more than ever. Reporting performance in a way that emphasizes the link to funding levels can be especially important for some agencies. Although this is done in scenario planning for forecasting investment impacts, reporting on funding shortfalls in relation to system performance deficiencies can provide an even more concrete way to demonstrate linkages. Historical performance information (such as the effect of inflation on fixed revenue streams) can also provide context for results.
Transportation organizations communicate performance results to a number of different audiences. First, the organizations collect and analyze data and circulate performance results internally. In addition, they report results both to the general public and to leaders and policymakers. In the case of the public and policymakers, simple graphics, visuals, and dashboards are very helpful in communicating information in ways that the public can understand, rather than simply reporting out data. To the extent possible, visuals should show past performance to provide context for current results. In addition, where appropriate, the organization can also provide counterfactual information, for example, about performance that would have been expected without the investments that were made. This can be particularly relevant for congestion; although in many areas congestion is worsening due to population growth, it is likely that investments in transit and demand management may have slowed the rate at which congestion has gotten worse.
In terms of communicating to policymakers, transportation officials can use reporting data to show the link between funding levels and performance, as well as the long-term cost savings of investing in infrastructure now to prevent costly repairs down the road.
There are a number of ways to clearly communicate performance results to the public, many of which are highlighted in the following discussion. Dashboards, which have an interface similar to an automobile dashboard, and scorecards, are designed to be easy to read. A common element of these methods is a clear display of information that communicates effectively.
The concept of "performance journalism" has been used by some agencies, notably the Washington State DOT, in order to clearly communicate information about performance. Performance journalism is the combination of quantitative reporting using charts, tables, and measurements, along with narrative storytelling. The goal is to share the performance of the agency's complex and diverse programs and projects clearly and concisely in a format that the public can easily understand. Key principles of performance journalism include:
California DOT's Regional Progress Report is part of an ongoing state effort to understand the intersection between land use, mobility, housing, infrastructure and natural resources preservation as they relate to a region's economic vitality, quality of life, and environmental quality. In 2007, the first California Regional Progress Report introduced regional quality of life indicators based on Regional Blueprint Planning goals. The 2010 Report builds on the foundation laid in 2007, but expands upon it to help meet the state's need for coordinated sustainability planning and assessment.
CUUATS Report Cards
The Champaign Urbana Urbanized Area Transportation Study (CUUATS), the transportation division of the Champaign County Regional Planning Commission (CCRPC), the MPO for the Champaign-Urbana metropolitan area, has adopted an objectives-driven approach to its transportation planning. In its recent plan, Choices 2035, CUUATS identified 12 regional goals and specific objectives to support each. The plan identifies measures of effectiveness in tracking progress toward each objective and uses them to determine whether objectives were met in tracking its performance. This has led to increased public engagement, greater accountability, safety improvements, and enhanced bicycle infrastructure in the area. Below is an example from CUUATS' LRTP 2011 Report Card, which identifies each goal, corresponding, SMART objective(s), Measures of Effectiveness (MOEs), and periodic performance updates that CUUATS uses to track its progress in accomplishing its goals.
Source: CUUATS, http://www.ccrpc.org/transportation/.
DVRP Tracking Progress Reports Communicate Clearly
The Delaware Valley Regional Planning Commission (DVRPC)'s "Tracking Progress" report on achievement of targets in LRTP with focus on encouraging public engagement. The report uses engaging graphics such as "dashboard indicators" to convey performance on a variety of measures. The Commission's easy-to-understand indicators are explained, and the site provides a concise explanation of performance along with visuals and links to more information for readers interested in honing in on performance in a particular area. The Commission, whose territory encompasses multiple states, also differentiates performance based on jurisdictional boundaries.
Source: DVRPC, Tracking Progress, http://www.dvrpc.org/LongRangePlan/RegionalIndicators/Transportation.htm.
VDOT Interactive Online Dashboard
The Virginia Department of Transportation (VDOT) has developed an interactive online dashboard to clearly report its performance to the public. The dashboard can be used to navigate to greater levels of detail for each performance area. The dashboard is available and comprehensible to anyone interested and clearly identifies the indicator and results. In addition, it provides links to public participation and survey results.
Minnesota Department of Transportation - Transportation Results Scorecard
MnDOT has been recognized for the presentation of its projected and actual performance on a variety of measures using a combination of colors and symbols that are easily comprehensible to the general public. The Annual Transportation Performance Report provides an overview of system performance and the Scorecard condenses this information into easy-to-understand graphics and assessments.
For more information, see: http://www.dot.state.mn.us/measures/.
WMATA Vital Signs Report
The Washington Metropolitan Area Transit Authority (WMATA) created an Office of Performance in 2010 to ensure the incorporation of performance measures into its decision-making processes. Performance results from the Office's Vital Signs Report are used by the General Manager to inform his Execution Plan and focus staff resources. Tracking performance has also enabled WMATA to present its Board of Directors with various scenarios of predicted performance based on the level of investment in both performance and customer/demand projects. WMATA has also created an interactive Metro Scorecard, which is easily accessible online and provides a broad overview of performance in the key areas of safety, security, reliability, and budget performance. In the Vital Signs Report, results for each indicator are provided along with a discussion about: the reason the indicator is used to track performance; why performance has changed; trend data; and actions the agency is taking to improve performance.
For more information, see: http://www.wmata.com/about_metro/scorecard/index.cfm.