Also in PDF (1.8MB)
Abbreviations
The Southern California Association of Governments hosted this peer review on 13-14 June 2019 in Los Angeles. The discussions that took place during the peer review sessions, and supporting technical documentation provided by RAND Corporation and SACOG, provide the basis for this report.
The views expressed in this report do not represent the opinions of the Federal Highway Administration (FHWA) and do not constitute an endorsement, recommendation, or specification by FHWA.
The FHWA would like to acknowledge the peer review members for volunteering their time to participate in this peer review. Panel members included:
The FHWA would also like to thank Sarah Jepson, Kome Ajise, KiHong Kim, Hsi_Hwa Hu, Mana Sangkapichai and Jisu Lee of SCAG for their assistance in organizing and hosting the peer review
The FHWA's Travel Model Improvement Program (TMIP) sponsored this peer review. TMIP provides technical support, and promotes knowledge and information exchange in the transportation planning and modeling community.
In this 7-hour meeting, split over two half days, representatives from several metropolitan planning organizations (MPOs) and State departments of transportation (State DOTs) met to review SACOG's recent work with Robust Decision Making (RDM) and the potential for expanding this work using the TMIP Exploratory Modeling and Analysis Tool (TMIP-EMAT).
The remainder of this report is organized into the following sections:
Three appendices also are included:
The primary objective of the peer review was to assist agencies in gaining a better understanding of the use of TMIP Exploratory Modeling and Analysis Tool (TMIP-EMAT) and robust decision-making (RDM) to better manage uncertainties in long range planning.
More generally, the TMIP peer review program provides transportation planning agencies the opportunity to network with travel modeling peers from around the country, have their models reviewed by each other, and share their successes, issues, and challenges. Peer reviews are designed to ensure that the techniques being developed or implemented meet the current and future needs of the agency.
The specific objectives of this peer review were to:
The classic modeling paradigm is to predict, then act:
For example, a 20-year plan might predict the future traffic volume on a road, assuming that a nearby parcel is developed. The action would then be to expand the road, so that its capacity exceeds the predicted traffic volume.
The classic paradigm can break down in conditions of deep uncertainty. Deep uncertainty exists "when parties to a decision do not know, or cannot agree on, the system model that relates action to consequences, the probability distributions to place over the inputs to these models, which consequences to consider and their relative importance."1
Under these conditions, the predict-then-act paradigm can break down because the uncertainties are often under-estimated, with assumptions that later turn out to be wrong. Furthermore, the presence of deep uncertainty can empower parties to a decision with different policy preferences to offer competing analysis using differing assumptions and methods, thus contributing to gridlock in the decision-making process.
For the road expansion example considered earlier, imagine how much more difficult the decision would be if:
How can quantitative analysis best be used to inform decisions in such an environment?
RDM turns the predict-then-act paradigm backwards. Rather than insisting that stakeholders agree on the prediction, it focuses on gaining agreement on the decisions that must be made today. RDM uses an "XLRM" framework (Lempert, Popper, et al. 2003) to guide stakeholder engagement, data assembly, and model development. In this framework, the X refers to external factors, or uncertainties; the L refers to possible policy levers; the R refers to relationships between the other elements (relationships that are reflected in the modeling); and the M refers to performance metrics. An RDM analysis (Figure 1) includes the following steps:
RDM brings four key concepts together:
RDM runs a model thousands of times to stress-test proposed decisions against a wide range of futures. Algorithms and interactive visualizations are then employed to identify a small number of policy-relevant scenarios. Decision makers may then use these results to identify strategies that are robust across many scenarios (Lempert, Groves et al. 2006, Lempert 2019).
The TMIP Exploratory Modeling and Analysis Tool (TMIP-EMAT) provides an additional tool to help planning agencies implement exploratory modeling and RDM in transportation planning. It is designed to enable existing transportation modeling tools, such as travel demand models, to be used to perform exploratory modeling. The workflow for TMIP-EMAT includes the following steps:
In 2008, California passed legislation (SB 375), which requires MPOs in California to meet greenhouse gas (GHG) emission reduction targets.3 The Sacramento Area Council of Governments (SACOG) used the RDM methodology to test a large number of scenarios related to its regional transportation plan / sustainable community strategy (RTP/SCS), to examine the plan's performance with respect to mobility, equity, and GHG reduction. The study stress-tested SACOG's 2016 RTP/SCS over many futures to identify key vulnerabilities and potential responses, including replacing gas taxes with mileage based fees, and encouraging more zero-emission electric vehicle (ZEV) use.
SACOG used the XLRM framework described above to organize this study.
Performance Metrics (M)
The RDM study focused on a few goals of SACOG's 2016 RTP/SCS, with corresponding performance metrics and target values (Table 1).
| Goal | Metric | Target value (per day) |
|---|---|---|
| GHG Reduction | Total GHG emissions from all passenger vehicle travel in the SACOG region | -16,400 metric tons CO2 equivalent |
| SB 375 Emissions | SB 375 GHG emissions4 | -13,100 metric tons CO2 equivalent |
| Mobility | Total Person Trips | - 11.8 million person trips |
| Equity | Person trips by low and middle income cohorts | - 3.75 million person trips |
Uncertainties (X)
The analysis considered seven uncertain external, behavioral and technological factors (Table 2). External factors included the price of gasoline, fleet fuel economy, and economic growth. Behavioral factors included millennial behavior, VMT elasticity with respect to the cost of driving, and VMT elasticity with respect to economic. Finally, the analysis considered the technology adoption factor of ZEV adoption.
| Uncertain parameter | Lower Bound | RTP/SCS value for 2036 | Upper Bound |
|---|---|---|---|
| Price of Gasoline (2010$) | $1.00/gal | $4.70/gal | $8/gal |
| Average ICE Fuel Efficiency | 15 mpg | 28.2mpg | 50 mpg |
| Employment Growth | 21% | 49% | 61% |
| Millennial Behavior5 | 0 | 0 | 1 |
| Sensitivity to cost of driving | -0.762% | -0.24% | -0.026% |
| Sensitivity to economic growth | 0.6% | 0.65% | 0.7% |
| ZEV/Plug-in Hybrids | 0% | 13% | 40% |
Policy Levers (L)
As its base case policy this study considered SACOG's 2016 RTP/SCS. The SACOG study also considered two additional policy levers: 1) a VMT fee, and 2) policies to promote high-ZEV penetration.
Relationships (R)
SACOG developed its 2016 RTP/SCS using SACSIM, the agency's travel-demand model. It was not possible to use SACSIM for this RDM study, so a cohort model was used for the analysis. The cohort model organized SACSIM model projections into 450 cohorts, using age, household income, residential density, and transit proximity. Cohorts are characterized by number of people and trips per capita. The cohort model was used to interpolate and extrapolate from SACSIM results.
| Age | Household income (2012 $) | Residential Density (dwelling units / acre) | Household transit proximity6 |
|---|---|---|---|
| 16 and under 17 to 25 26 to 40 41 to 65 66 and over |
Low: less than $25,000 ($25k) Low-Middle: $25k - $50k Middle: $50 - 75k High-Middle: $75-125k High: $125k and above |
Very high: more than 20 High/Medium high: 12-20 Medium: 6–12 Low: 2- 6 Very low or farmhouse: less than 2 Mixed use: n/a |
Less than 1/4 mile 1/4 to 1/2 mile Greater than 1/2 mile |
SACOG provided data on age and income cohorts, with VMT per capita related to density and transit access. The simulation model then shifted these distributions based on policy levers and uncertain futures. The new distributions were then used to estimate desired model outputs.
Ten thousand cases were run, exploring the relevant uncertain parameters. Twelve percent of the cases met the scenario criteria. This does not indicate a 12% probability of meeting the goals, because the analysis has not made any assumptions about the probability of each case.
Dark Grey = case meets scenario criteria
Light Grey = case does not meet scenario criteria
After generating the database of model runs, the study next used "scenario discovery" classification algorithms (Bryant and Lempert 2010, Lempert 2013) to identify a small number policy-relevant scenarios and their key driving forces. Of the seven uncertainty areas, four were found to be important for determining whether the base case scenario met all goals. They included gas prices, fuel efficiency, employment growth, and VMT elasticity with respect to cost of driving. All four goals are met in a future with low gas prices, high fuel efficiency, economic growth that is neither too high or too low, and residents whose travel patterns are sensitive to the cost of driving (VMT elasticity with respect to the cost of driving that is high in magnitude) (Figure 4). In Figure 4, the dark green bars show parameter variation ranges that best differentiate futures within and outside of this scenario. Variables without green bars are not a key driver/differentiator for this scenario.
An ultra-low GHG scenario (total GHG emissions at most 8,200 metric tons per day) was also tested, with 6% of cases meeting scenario criteria.
Figure 6 illustrates the sensitivity to input parameters for three cases: those scenarios that meet SB 375 (but may miss other goals), those that miss SB 375 but meet the other goals, and those that meet all goals including ultra-low GHG.
The study's RDM results demonstrate the importance of economic growth assumptions in the ability to meet SB 375 goals. If employment growth (used as a proxy for economic growth) is too high, the SB 375 goals will not be met. This is not surprising, as the calculated emissions in SB 375 are based on VMT resulting from land use. Goals other than SB 375 are met with high fuel efficiency, and higher employment growth. All goals (including mobility, equity and GHG reductions) are met with moderate to high ZEV penetration, low to moderate gas prices, high fuel efficiency, and economic growth that is neither too low or too high.
The sensitivity analysis found that current plans might be vulnerable to exogenous assumptions, which are often treated as static predictions. Important factors include economic growth, fuel prices and fuel efficiency of vehicles. Depending how these factors play out, meeting the goals of the plan may require additional policy measures. On the other hand, the analysis revealed that variations in ZEV adoption and millennial travel behavior were less important in terms of meeting RTP/SCS goals.
An initial question was on the mechanics of running the model. Was an effort made to find the "corners" of the uncertainty region? It may not be necessary to perform 10,000 model runs. Rather, an adaptive sampling approach might enable fewer runs. For example, one might start with 50 runs, and use those results to choose 50 more. The peer review also discussed the advantages and challenges of the study's cohort model. The reviewers noted the more simple structure of the cohort model allowed for multiple runs but lacked the detailed spatial representations and network effects seen in traditional MPO travel demand modeling. The group discussed the tradeoff between model run times and complexity as a key issue that needs to be addressed if RDM and uncertainty analysis is to be better integrated into practice.
Sacramento is a large region, and it may be necessary to split it into subregions for items like employment growth. There is also the possibility of running with a portion of the base model.
Had TMIP-EMAT been available to SACOG at the time of their RDM study, SACOG might have been able to conduct the RDM study using SACSIM directly, thereby avoiding or augmenting the cohort model and addressing many of the above questions.
More broadly, what will an MPO Board do with this information? Although the Board (and more broadly, elected officials and members of the public) understand that uncertainty exists, the message to them needs to be crisp. They do not need the details of 10,000 model runs. Generally the peer review found the RDM study a positive step for the profession, but also called out the limitations of the demonstration study. A near term application from the work is helping professional planners better think about uncertainty.
Thomas Small, current council member and former mayor of Culver City, spoke about the use of RDM to deal with neighborhood concerns with cut-through traffic. With several projects underway (transit-oriented development near the metro station and development at a studio), the neighborhood was concerned about traffic impacts. Mitigation can be difficult: some actions (e.g., stop signs) must meet warrants, proposed actions might adversely affect other neighborhoods, and local traffic may be affected by what happens outside of the jurisdiction. RAND helped Culver City conduct a shadow process, a structured exercise that runs alongside a city's formal planning, provides a space for exploration and experimentation, and feed promising ideas back in the formal process. In Culver City, this process gathered neighborhood leaders into the same room with city staff to engage in a backcasting exercise,7 scenario development, and stress testing of potential plans. The city is now installing temporary interventions, and is working on gaining neighborhood approval for the overall traffic mitigation plan. The RDM process went beyond the usual public meeting process, and helped to broaden the discussion of what could happen.
Participants believed that these methods have value for planners, to broaden the discussion of what could happen. There is some danger in building regressions on top of regressions. One noted that current econometric models are not effective with drastically changing futures. The largest uncertainties can come from changes in the political environment, changes in the need for transportation (e.g., online shopping vs. retail store shopping), climate change, and changes in land use. Traditional sensitivity analysis is well suited for dealing with smaller, well-defined variations. Consideration of deep uncertainty goes beyond traditional sensitivity analysis, as the uncertainties may be larger in both type (for example, a sudden change in political attitudes) and magnitude. The challenge is to bring these methods into the planning process, where existing regulations (e.g., for air quality conformity) seem to call for plans based on single point forecasts. How can RDM methods be reconciled with policy development while meeting legal requirements?
Participants discussed how to present these results to the MPO Board, elected officials, and the public, recalling Rittel and Weber's 1973 paradox: "The more skilled policy professionals get at analysis, the less people seem to listen to them." Rittel and Webber explained this paradox by contrasting "tame" and "wicked" problems:
The problems faced by MPOs are often "wicked" problems. Fortunately, elected officials understand that uncertainty exists. They will often welcome a candid discussion of uncertainty, noting that they never really believed the previous point forecasts. They will find the idea of reducing risk attractive. However, they also need to make decisions today, and do not have the time to look through a massive analysis.
Backcasting is an exercise that first imagines a hypothetical future. Participants then identify some of the actions that must be taken to realize that future.
Following the presentations and discussion on the afternoon of day one, the group engaged in a backcasting exercise on day two, and then discussed how to move forward over the next 1 to 5 years.
Steven Popper of RAND led a backcasting exercise, imagining a headline from the year 2037: "Autopia comes of age: L.A. Plans its Way to a Workable Urban Future." The imagined future, in Figure 7, depicted narrowed streets; wide sidewalks; transit including a subway station and an automated mini-bus; pedestrian, bicycle, and scooter activity; and small passenger cars of uniform design (presumably automated, electric, and shared).
He posed the following questions:
What does the lead story say about the role played by transportation planning? Participants noted that in order to reach this desired future, the role of planning had to be broader than just transportation planning. Land use is also very important (what happens to housing costs, are commutes shorter, therefore less traffic congestion?). Furthermore, given the absence of cut-through traffic, it appears that the region found a solution to long-distance travel.
Stakeholders include local residents, employers, businesses (concerned with deliveries), other parts of government (e.g., housing authority, school board). There was significant community input, via an inclusive process that included the young, old, and low-income households. The public had interactive tools, so that they could explore the consequences of various futures.
What types of persistent planning problems were overcome? Planners overcame the following problems:
What necessary precursors were required for these solutions to appear? Good leadership provided the vision of this future. The tools and performance metrics existed to monitor progress, and the desires of the community were taken seriously. There might have been some crisis (e.g., an oil shock) that led to new attitude and enabled this change. Given the lead time for major transportation and land use investments, the changes had to start back in 2019.
Furthermore, given the dynamic and uncertain environment, there was a greater emphasis on operations than on construction. Relevant system performance measures had been established and were being monitored on a regular basis. Policy-makers were able to understand how the system was actually being used day-to-day, and were able to make frequent adjustments to ensure that the existing right-of-way and facilities were moving people and goods as safely and efficiently as possible.
What did not happen? To enable this hypothetical future, participants noted, in the context of the backcasting exercise, that the region was able to avoid:
The group believed that a major contribution of the RDM approach is to change how we do planning. Structuring the uncertainties provides a significant benefit. Types of uncertainties mentioned included:
The mindset of RDM is focused on gaining agreement on the actions to take today, and not on developing the perfect forecast. This requires a long-range vision of what the future could be like, as well as a willingness to make frequent updates today's plans. Such an approach is consistent with the MPO's role as a convener of stakeholders, and its responsibility to maintain a comprehensive, cooperative and continuing (3C) planning process. It is also consistent with the documents that MPOs are required to produce, with the Long Range Transportation Plan (LRTP) providing the vision, while the LRTP and other documents, such as the Transportation Improvement Program, providing an opportunity for updates every few years. The Federal role is to encourage this change in mindset and use of these new methods. A challenge is that MPOs may not have the resources to put into a new initiative, as they are too busy with required tasks. Likewise, the travel demand models traditionally employed have long run times and many input parameters, making them less suited for the multiple runs needed for an RDM analysis.
There are several ways to start. The beta tests of TMIP-EMAT with Oregon DOT, SANDAG, and Greater Buffalo-Niagara Regional Transportation Council will provide some examples. One could take a small piece of the problem (e.g., cross-border traffic in San Diego), or run a shadow process, in which an innovative new modeling process runs in parallel with the traditional planning process, for the purpose of evaluation or gaining additional insight. For example, an agency with a new activity-based model might initially run it in parallel with their four-step model, to see how the results compare.
There is value in using these methods both at the beginning of a planning process (to better understand the options), and at the end (as a sanity check).
Several software tools can aid the process. SACOG has a sketch planning tool for single projects. VisionEval has regional components, but no network. TMIP-EMAT provides a workbench, but is not yet mature enough where someone can simply use it as a plug in. A challenge with software is also with the core models. How can core model developers make it easier to pull information out of the models, to facilitate the building of meta-models?
Communication of the results is important, and there is an opportunity to work on improved visualization of results. There is no need to present 10,000 scenarios, but it is often helpful to present a few representative scenarios that illustrate the story you are trying to tell. There is an opportunity to engage with stakeholders, ask them to challenge your assumptions, with the end goal of building a consensus in favor of the plan. The RDM process provides a way to build consensus to make needed decisions today even through the future is uncertain.
This peer review enabled representatives from five MPOs and two State DOTs to learn how one MPO applied TMIP-EMAT and RDM to their long-range transportation plan, to explore the plan's ability to meet desired goals under a variety of futures. The discussions revealed a number of shared challenges. The methods presented during this meeting will be useful for dealing with all types of uncertainty in long range transportation planning.
| Time | Session |
Speaker(s) |
12:30 PM |
Check-in / Arrivals |
|
1:00 – 1:15 |
Opening Remarks and Introductions |
Facilitator, FHWA Office of Planning, Systems Planning and Analysis Team
|
1:15 – 1:30 |
Overview of and Goals for Peer Review |
Facilitator, FHWA Office of Planning, Systems Planning and Analysis Team |
1:30 – 2:45 |
SACOG RDM Pilot Project
|
|
2:45 – 3:00 |
Break |
– |
3:00 – 4:30 |
New capabilities enabled by TMIP-EMAT
|
|
4:30 – 5:00 |
Open Discussion / Q&A - Review of Day Two Agenda/ Wrap-up and Next Steps |
|
5:00 pm |
Adjourn |
– |
| Time | Session |
Speaker(s) |
8:30 – 9:00 am |
Check-in / Arrivals |
— |
9:00 – 9:15 |
Review of Day One / Debrief |
Volpe, US DOT |
9:15 – 10:30 |
Full Group Discussion |
Discussion with all of the participants, considering the following questions:
|
10:30 – 10:45 |
Break |
|
10:45 – 11:45 |
Full Group Discussion continued |
|
11:45 am – 12:00 pm |
Wrap-up and Concluding Remarks |
|
12:00 pm |
Adjourn |
— |
Bryant, B. P. and R. J. Lempert (2010). Thinking inside the box: A Participatory, computer-assisted approach to scenario discovery. Technological Forecasting and Social Change 77: 34-49.
Copperman, R., S. Sun and M. Milkovitz (2019), Exploratory Modeling and Analysis with TMIP-EMAT, presented at the TRB Planning Applications Conference, Portland, Oregon, June 2019.
Kalra, N., S. Hallegatte, R. Lempert, C. Brown, A. Fozzard, S. Gill and A. Shah (2014). Agreeing on Robust Decisions: A New Process for Decision Making Under Deep Uncertainty. WPS-6906, World Bank.
Lempert, R. (2013). Scenarios that illuminate vulnerabilities and robust responses. Climatic Change 117: 627-646.
Lempert, R. (2019). Robust Decision Making (RDM). Decision Making under Deep Uncertainty: From Theory to Practice. V. A. W. J. Marchau, W. E. Walker, P. J. T. M. Bloemen and S. W. E. Popper, Springer: 329.
Lempert, R.J., G. Ballard-Rosa, (2019) Meeting California's Climate Goals Under Wide-Ranging Scenarios: A Demonstration of Robust Decision Making for Long-Term Transportation Planning. Manuscript submitted for publication.
Lempert, R. J., D. G. Groves, S. W. Popper and S. C. Bankes (2006). A General, Analytic Method for Generating Robust Strategies and Narrative Scenarios. Management Science 52(4): 514-528.
Lempert, R. J., S. W. Popper and S. C. Bankes (2003). Shaping the Next One Hundred Years: New Methods for Quantitative, Long-term Policy Analysis. Santa Monica, CA, RAND Corporation.
Rittel, H. W. J.; M. M. Webber (1973). Dilemmas in a General Theory of Planning. Policy Sciences. 4 (2): 155-169.
NOTICE
This document is disseminated under the sponsorship of the U.S. Department of Transportation in the interest of information exchange. The United State Government assumes no liability for its contents or use thereof.
The United States Government does not endorse manufacturers or products. Trade names appear in the document only because they are essential to the content of the report.
The opinions expressed in this report belong to the authors and do not constitute an endorsement or recommendation by FHWA.
This report is being distributed through the Travel Model Improvement Program (TMIP).
U.S. Department of Transportation
Federal Highway Administration
Office of Planning, Environment, and Realty
1200 New Jersey Avenue, SE
Washington, DC 20590
August 2019
FHWA-HEP-19-041
1 Society of Decision Making under Deep Uncertainty: http://www.deepuncertainty.org/. Accessed on 15 July 2019.
2 The term "Red Team" comes from Cold War military exercises of 50 years ago, when the assumed adversaries of the United States were Communist countries (e.g., the Soviet Union, China) with predominately red flags.
3 https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=200720080SB375
4 The calculation of SB 375 emissions is based on the changes in VMT that result from changes in land use. The calculation of SB 375 emissions is not sensitive to fuel prices, fuel efficiency, economic growth, or new technologies.
5 Today's millennials (in the age 26-40 cohort) drive fewer trips per person than in older cohorts. It is debatable whether this behavior will persist as millennials age. The millennial behavior variable is an indication as to whether the lower VMT will persist as millennials age into the 41 - 65 year cohort. (Lempert, Ballard-Rosa, 2019)
6 Distance between the household and nearest rail station or bus stop providing high quality transit service.
7 As described in the next section, a backcasting exercise starts by defining a desirable future, and then identifies the actions that must be taken to reach that future.
