Puget Sound Regional Council (PSRC) Peer Review

6.0 Peer Review Panel Recommendations

On the last half day of the meeting, the peer review panel took about one hour in an executive session, closed to all participants of the meeting except for the panel members. The reason for this was to allow panel members to speak freely and openly among themselves while developing formal recommendations. This section details the recommendations of the panel. Section 6.1 details the panels overall findings. The remaining three sections are organized around three key themes - tasks and activities PSRC must do, tasks and activities PSRC should do if they can, and tasks and activities that the panel is not convinced are priorities for PSRC.

6.1 Overall Findings

The panel made the following findings during the peer review panel meeting:

• SoundCast demonstrably meets or exceeds performance of the trip-based model;
• Model credibility is established by making model inputs and outputs easy to consume, and by educating other agency staff about model capabilities; and
• PSRC is on the leading edge of travel modeling in the USA. PSRC outlined issues and limitations, but the panel sees PSRC as a model for other MPOs across the country to emulate.

6.2 Must Do

6.2.1 Trip-based model retirement and SoundCast implementation

• PSRC has all the evidence needed that SoundCast meets or exceeds the calibration requirements set for the trip-based model. It acceptably replicates observed behavior, within the limits of data at the aggregate total, county, or district levels for which PSRC has made comparisons.
• Retire the trip-based model. Expected key applications include traveler response to pricing and congestion, and their equity implications - something for which microsimulation activity-based models are demonstrably superior to trip-based models. No amount of recalibration of the trip-based model will enable it to inform these issues.
• Looking forward, focus on what PSRC can do at the detailed level that could not be done with the trip-based model. Bring in other datasets to see how SoundCast is doing at smaller, more detailed levels than districts.

6.2.2 Focus on tools that make SoundCast more accessible to local stakeholders

• The web front-end model controller and other visualizations will bring more stakeholders to the table. However, PSRC needs to ensure that visualizations and other periphery software and tools are consistent with a larger strategic plan.
• Evaluate what is important to PSRC's model users and do what they would do first. Be one step ahead. For example, what will the revenue and tolling users do to evaluate PSRC's data? Post that information in PSRC's online documentation.
• Accept that, in the end, it is not possible to track, develop, and support every tool that external stakeholders find interesting.

6.2.3 Estimation and calibration/validation

• Given the August 2016 schedule for having an operational model, re-estimation with 2014 survey is likely not feasible or necessary.
• Instead, focus on re-calibration to the 2014 household survey.
• Calibration should first ensure alignment between synthetic population and expanded household survey population.
• The general process should involve a first pass through all the models to calibrate to un-adjusted survey targets, including network assignment.
• Adjustments to household survey targets will likely be necessary to account for discrepancies between key model outputs (e.g., traffic counts and transit volumes by time of day).
• Adjustments to household survey targets should ideally be based on adjustments to household expansion factors to ensure consistency across all model summaries.
• Calibration should be focused on some components such as activity generation, destination choice, mode choice, and auto ownership.
• Issues observed in initial calibration and validation included:
  • Time of day distributions
  • Activity generation in some areas such as Kitsap County
  • Consistency of synthetic population and expanded survey
• Consider network validation procedures to ensure that estimation / calibration inconsistencies are not related to network inputs.
• Going forward, expand PSRC's estimation dataset to include transit on-board surveys (in lieu of over-sampling transit-oriented households in the next household travel survey).

6.2.4 Utilize other data sources

• Other datasets could be useful to externally validate SoundCast at a level that was not possible with the trip-based model.
• Presenting comparison results online in an interactive way could improve PSRC's credibility. For example:
  • Get more detailed person-level data than ACS/CTPP or the Decennial Census can offer from one of the many marketing databases that exist. A 50% sample, even if biased, will point toward ground truth much better than 1% of a 1% sample will.
  • Look at an OD matrix (GPS-based, triangulated signal, or some combination thereof). Use 20% of the OD pairs to calibrate SoundCast more. Then use the remaining 80% to show that the base year validates well.
  • Use real travel times from directions APIs to validate travel times in SoundCast. Compare these to OD pairs that happen to appear in the household travel survey.

6.2.5 Invest in outreach and wetware

• Open documentation is commendable, both in concept and quality of implementation.
• Other strategies to connect with model user community include:
  • Informal or incremental peer review: periodic external review of modeling program milestones, guiding Innovations in Travel Modeling Conference (ITM16) discussions.
  • Formal SoundCast user training for external users (at executive and user levels).
  • Continue training PSRC staff in scripting, data mining, and visualization of data.
  • Making products accessible online: data, modeling summaries, visualization of model outputs.
  • Stakeholder-defined workshops: bring in outside users, help them construct model runs, run and interpret the outcomes with them.

6.2.6 Focus on improving the run-time performance of the model

• Reliability of model runs is a higher priority than run-time performance.
• Do some basic profiling of the entire model run stream to determine which parts of the model to focus on for performance tuning.
• Look at high-level model flow: reduce the number of feedback loops by warm-starting the model, as well as shadow pricing within it.
• Ensure the adequacy of hardware, and examine whether bottlenecks in run time can be overcome with AWS or other easily scalable solutions.
• Determine if more hardware will help if licensing issues can be escaped (another reason to consider MATSim)

6.3 Should Do (if possible)

6.3.1 UrbanSim

• Until SoundCast model run times are reduced significantly, it will be a challenge to run an integrated land use and transportation forecast through time.
• If PSRC desires a continuously integrated platform, PSRC will have to live with run times of a week or more.
• The most useful implementation of UrbanSim is a framework that allows PSRC to evaluate what policy it would take to meet its land use vision goals.

6.3.2 Workplace Location

• Workplace location should continue to come from SoundCast.
• SoundCast implementation of workplace location is also easier to calibrate and change to reflect alternative land use scenarios.

6.3.3 Transit Submode Placement

• Transit submode choice can occur in either the DaySim demand model or the network supply model.
• Placement of submode choice in DaySim provides users with easy means to calibration of submodal shares.
• Placement of transit submode choice in the network model will facilitate integration with FAST-TrIPs.
• Current implementation validates reasonably well (and significantly better than the trip-based model).

6.3.4 Shadow Pricing

• Shadow pricing setup should be reviewed to ensure:
  • Reasonable run times
  • Convergence and stability
  • Effects on other components such as skims is reasonable

6.3.5 Sensitivity Testing

• Upon completion of 2014 calibration / validation, model system should be subjected to a set of sensitivity tests that address key model capabilities, including:
  • Pricing sensitivities (investigation of changes in activity-making, flows)
  • Capital investments
  • Land use and demographic changes
  • Simulation variation

6.3.6 Health Modeling / Active Transportation Modeling

• This is a commendable goal to implement tools around health and active transportation.
• Be careful to not overstep the bounds of what the data would allow or suggest. Easy first steps are travel model performance measures around walking and biking (e.g., time spent walking) compared to sedentary travel behavior (e.g., time spent traveling in a car).
• Engage experts from the public health community.
• If transportation-related public health data does not exist for Washington or Seattle, consider looking at California Health Interview Survey (CHIS) data and Integrated Transport and Health Impact Modeling (ITHIM) implementations at MTC, Sacramento Area Council of Governments (SACOG), and SANDAG.
• There are a lot of well-established methodologies to measure particulate matter and ozone exposure for sensitive communities. Continue to expand the practice, but do not reinvent the wheel.

6.3.7 Dynamic Network Models

• ABM-DTA integration is a hot research topic, but years away from successful, much less market-ready, implementation.
• It is unquestionably the future of travel forecasting and will be capable of meeting PSRC's desire for finer operational details that cannot be modeled adequately otherwise.
• Substantial conceptual and methodological hurdles remain.
• Consider incremental changes: MATSim and planning-level DTA.
• Explore ways to use network models as sketch tools (e.g., define population affected by a local project, and then replan just those individuals rather than re-simulating entire population).

6.3.8 Consider Open Street Maps for Networks

• Consider investigating tools that utilize Open Street Maps (OSM) for networks and building footprints. Use General Transit Feed Specification (GTFS) for transit networks at the same time.
• Look to New York City; they are doing a lot with this.
• There are scripts that exist that can convert OSM to networks. They would be a good starting point.
• There also may be good data for bike modeling stored on OSM.
• This will also make it easier to publish web-based visualizations (credibility, user interaction, etc.).

6.3.9 Employment Data Restrictions

• Use of Longitudinal Employer-Household Dynamics (LEHD) Origin-Destination Employment Statistics data instead of Quarterly Census of Employment and Wages (QCEW).
• Continue with planned investigation of LODES data before doing this. Others will benefit from PSRC's analysis of the LODES data to date. It has gone through significant changes since the last TRB publications on the dataset.

6.3.10 Benefit-Cost Analysis

• Ongoing refinement of ABM-based BCA methods is occurring in San Francisco, San Diego, and Tampa regions.
• PSRC can look to these agencies for guidance and lessons learned.

6.4 Not Priorities

6.4.1 Truck and Commercial Vehicles

• The Quick Response Freight Manual is widely used, with no success stories of approaches that are small, incrementally better.
• Any improvements will require extensive data collection efforts.
• State of the art is tour-based truck models that incorporate all commercial vehicles - easily as large an effort as person-based ABM development.
• If work is absolutely necessary here, focus on short-term operation strategies that can be completed outside of the SoundCast framework.
• Partner with Portland Metro.

6.4.2 Sketch Planning Tools

• Spend resources making SoundCast / UrbanSim a platform that can answer these questions in a reasonable and accessible manner.
• Think about the ways PSRC can adapt SoundCast process to do sketch planning. (Perhaps topic for ITM16 roundtable or specific research work item). This will likely tie in with PSRC's visualization strategies and dashboard implementation. Aim to make those solutions solve PSRC's sketch planning tool needs.