The first day of the peer review panel meeting included time for PSRC presentations to the panel members as well as discussion of key topics. This section documents the key points that were discussed.
PSRC presented details of their modeling needs to the panel. One of the key points of emphasis for PSRC in the transition from their trip-based model to ABM is consistency in results and maintaining credibility with partner agencies. This might be particularly challenging, since their model is being used in a variety of ways by different groups. In addition, some users do not need detailed model results at all, instead using only population and employment data, transit coverage, or future transportation project information.
Another topic presented to the panel was information related to the survey PSRC conducted groups using the model. One panel member asked about the relationship between PSRC and the Washington DOT. PSRC noted that Washington DOT is the biggest user of the travel models supported by PSRC, but their modeling work is usually performed by consultants, rather than PSRC. Washington DOT does stress that they want consistency in what they do and the work PSRC does, even though the DOT is more focused on corridor type analyses, while PSRC looks at a more regional view typically.
One key point was offering better access to the information and data PSRC has available to them and sharing that information more effectively. One panelist noted the importance of being able to create data products and update those products in a timely manner for a MPO. PSRC has been investing in getting staff to script so that analyses and processes performed routinely or multiple times can be accomplished more quickly. Even so, PSRC noted that at this point, they are more reactive to the needs of the planners at the agency and modeling partners. This may change as their partners get a better idea of what PSRC is capable of. One of the panel members voiced similar struggles in that panel member's region, but agreed that scripting is very important. In that panelist's region, money was invested in training staff in Python and SQL. A couple other panelists suggested R as a good scripting tool, particularly for visualization.
PSRC uses UrbanSim in two ways. The first is called baseline mode where land use forecasts are produced. This represents a market-based forecast, reflecting past trends and current land use. The second is to allocate growth, where growth is determined by a land use visioning process. The visioning process helps to ensure the land use futures used in modeling are consistent with the vision the stakeholders, policy-makers, and citizens have for the region. The perception of the baseline is that it represents a future that nobody wants. The approach is also used as a gap analysis, to evaluate how the forecasts can be affected so they look more like the vision. Several panelists liked this approach, noting that the baseline only has credibility for backcasts, to show the tool works. The baseline does not have credibility for forecasts, because it does not match political vision. On the other hand, the panelists cautioned that judgment was needed, since the visioning approach may not be suitable for all needs and objectives. One panel member noted that investment bankers, for instance, may not care about the vision, and so PSRC should maintain a land use scenario that captures those interests.
PSRC noted that UrbanSim has its own work location choice model, but the ABM also has a work location choice model. One panelist suggested that there could be a compelling case for including the work location model in UrbanSim, since the ABM does not have residential location choice, but UrbanSim does. However, UrbanSim and the travel model are not very well integrated due to difficulty in running them both, which means the accessibility variables fed to UrbanSim may be inadequate inputs for work location choice. Another panel member argued that since PSRC's implementation of UrbanSim is vision-based, then the work location should be in the ABM so that it is sensitive to policy.
Two panelists asked about calibration/validation of UrbanSim. PSRC suggested that the overall model was calibrated well and adequately responded to policy levers. One of the panel members expressed concern that the calibration and validation data sources were truly independent, particularly as it related to work location, and suggested AirSage data might be a good option as a completely independent data source. Another panelist argued that the AirSage and CTPP data were bound to be inconsistent in many ways, and suggested the CTPP may be more reliable due to the much larger sample size.
PSRC's synthetic population for the ABM is created to match certain control totals, as opposed to actively aging the population in forecast years, which a couple of panel members agreed was unnecessary anyway. One panel member cautioned that some of the longer term choice constructs might be changing due to things like the “gig economy” (e.g., people working from home and shifting jobs more often).
PSRC had a number of questions related to the integration of UrbanSim with the travel model. PSRC's current approach runs the models iteratively for every 10 years. There is also questions about which accessibilities to use in UrbanSim. One panel member suggested that they set the integration time frame so that PSRC can run the whole system within several days, and that the accessibilities that are passed to UrbanSim be tested as it may not even make much difference. Another panelist mentioned that buffered or decayed measures are simpler to compute than logsum accessibilities, and could be beneficial. In addition, careful consideration should be given to decide on segmentation variables to use for the accessibility measures, which measures make sense for each model component, and to ensure comparability in the measures used for UrbanSim and the ABM. Another panelist suggested that while land use models are useful for decision making, integrating tightly with travel models is problematic and usually prohibitive.
In general, the ABM specification, as described in Chapter 4, was already set at the time of the peer review meeting, but several points were discussed.
Several comments were made on the long-term choice model components of the ABM. One panelist commented that a toll transponder ownership model could be useful to obtain good toll related results. While it can be difficult to model, another panel member mentioned that pay-to-park at one's workplace is an area of considerable interest in that panel member's region, due to the vast number of parking options. This may be a modeling area to explore for PSRC. Shadow pricing on the work location model (so that the number of workers in a zone better aligns with the number of individuals choosing to work there by the model) was also discussed. PSRC believes it can be difficult to calibrate and time consuming. One panelist liked the idea of using shadow prices, since it should give reasonable results for certain types of policy analyses (like adding jobs to a particular zone). The alternative to shadow pricing (while maintaining equilibration between jobs and work locations) is to take jobs out of the pool during simulation, but the panel noted that this can lead to problems.
Another key area of concern for PSRC is the path type model, which is used for the logsum generation that feeds many of the model components. Path types include toll or no toll choice for auto modes, premium and basic transit mode options, park and ride path type, as well as bike and walk path types. These alternatives appear as alternatives under main travel mode options, but the alternatives are not actually selected in simulation. Instead, they are used to represent the diversity in lower level alternatives across the main modes. The actual choices for these options are made by the assignment models. A couple of the panel members found this to be confusing, and believed it would be better to simulate the choices at the mode level rather than in assignment.
PSRC has had to deal with the toll or no toll choice component issues resulting from the types of toll scenarios they typically examine. The toll scenarios typically have tolls everywhere in the network, so the choice is not really one of toll versus no toll, but one toll versus another toll. PSRC posed the question of whether it would be better to remove the toll choice component from mode choice and consider it only in assignment. One panel member believed it to be problematic to consider the toll choice only in assignment and not mode choice. Another panel member noted that treatment in assignment only would result in losing information about who is using the toll road, which can be important for environmental justice issues.
Several panel members brought up the concern that reliability would be particularly important for the types of toll scenarios PSRC is examining (e.g., tolls everywhere). One of the key motivations of pricing is better reliability, with one panelist noting that one can see people shifting to use managed lanes earlier than expected based on the model due to reliability issues not captured by the model. One issue is that measures of reliability are not easy to extract from network assignment, particularly since they are not additive. Even so, the panel suggested that an ABM has the granularity needed to look at the tradeoffs between travel time and reliability.
PSRC described that new household survey data was collected for the region in the previous year, but this data had yet to be utilized in either the existing trip-based model or the ABM. They are committed to moving forward with using the data for the ABM only, not wanting to continue maintaining two separate travel models, but they will face political pressure if they wait too long. One panel member's region now has an operational ABM and the transition to the ABM was rather simple. When the transition was made, the trip-based model was retired. The panelist conceded that there were problems with the model, as one would expect, but they are working on it and are open about the issues.
PSRC was still debating whether to re-estimate all of the ABM model components and perform model calibration and validation with the new survey data or whether to embark on calibration and validation tasks only. For political reasons, PSRC needs an operational model by August of 2016, and they prefer that model to be the ABM. One panelist strongly believed that re-estimating the ABM model components would be a mistake, and PSRC should only perform calibration and validation (other panelists agreed). There simply would not be enough time, and they can have a 2014 model without explicitly estimating the model on the data, as long as calibration is based on the 2014 data. Moreover, the panel member suggested PSRC carefully consider what targets would need to be hit for the calibration and validation processes to be considered a success. PSRC may also benefit from applying the ABM and getting experience with it to better understand how re-estimating model components would be valuable (e.g., what aspects of the model do not perform as well as PSRC would like).
One of the main concerns of PSRC, as described in Section 5.1, is credibility, and part of that deals with the model consistency and performance. For calibration, one panel member suggested that PSRC make a list of calibration priorities by thinking about where credibility problems currently exist. Another panel member emphasized that people care about aggregate measures, and those deserve particular scrutiny during model calibration.
PSRC has struggled in the past with transit validation, often finding that transit shares from the household travel survey do not match the model. Several of the panel members commented that this was not satisfactory. One panel member suggested PSRC invest in an on-board transit survey, and another suggested backward engineering the number of transit trips from boardings data if possible in order to assert targets. Panelists agreed that transit validation is an issue for many regions.
Panelists debated the impacts of commercial vehicles and how those are treated in models. One panel member offered that certain types of commercial vehicles (e.g., taxis, Ubers, UPS trucks, etc.) may not be represented very well by truck/commercial vehicle models. But panel members agreed that commercial vehicles are important, making up 10-15% of a region's overall VMT. In one panel member's region, ride sharing has replaced taxi to a large extent, with taxi ridership down on the order of 70%. Overall, ride sharing plus taxi in that region has remained relatively stable, with only a small combined increase in trips. Another panel member suggested that taxi and ride sharing are viewed differently, with people using the modes in very different ways. Nonetheless, these different modes should be able to be included in ABMs rather easily, since wait times and costs are easily represented.
One measure of model performance that PSRC has been looking at is link travel times from the model and comparing to observed freeway travel times. They have found the model is not performing well. One panelist pointed to network coding issues as a likely culprit, suggesting this is a common issue.
Another panel member offered that the issue may be the volume-delay function (VDF), which is a mathematical way travel models predict congestion levels for specific demand levels on network links. These VDF functions suggest that travel times monotonically increase with volumes, but that is not how actual traffic responds. As congestion levels reach level-of-service E or F (per the Highway Capacity Manual), the curve will tend to bend back on itself. That is, there is a point where as congestion worsens (i.e., travel times increase), volumes get smaller. There is no way around this issue with a static assignment model, though a more dramatic curve could help.
One panelist suggested that, in addition to link level delay, a better representation of intersection delay could help. Another panel member's region had issues with intersection delay, later finding there was a simple miscoding issue of a parameter. PSRC noted that they do not have information about signals versus stop sign intersection treatments, which could be problematic for modeling intersection delays.
Several panel members agreed that this issue was not a particularly important one in the grand scheme of things. Given that PSRC has only several months to get the model ready, there are plenty of other more important areas of the model to focus on. One panel member concluded, noting that static user equilibrium assignment was designed to distribute vehicles on paths, not get accurate travel times on congested links.
SoundCast has been operational for several months (though not calibrated to the 2014 data). In this time, PSRC has been applying the model simultaneously with the trip-based model whenever that model is run. Results between the two models can then be compared for the following purposes:
One panelist suggested that PSRC not worry about how well the two models match. However, for PSRC, this goes back to an issue of credibility, and if the two models differ too much, it could hurt the agency's credibility.
PSRC described that the aggregate model results have generally shown that the two models are very similar across VMT, average trip lengths, time of day splits, and mode shares. This was surprising to the panel, since other regions' comparisons have not been nearly so consistent.
The panel made a few other recommendations. One panel member suggested making special considerations for military bases, since their travel patterns are different from other households, and PSRC has two military bases, including one that is quite large. Another panel member suggested that the ABM developers may know about certain issues related to the time of day model components of DaySim, and suggested contacting them. However, adjusting time of day constants would not be a good use of time, since they would not change the models' sensitivities to policies. One panelist suggested that for calibration, PSRC should adjust the weights to better reflect different geographies for which PSRC is interested.
In the spring of 2014, PSRC began collection of a new household travel survey, covering a minimum of 4,700 households. The sampling plan included targets for specific segments of the population. The survey was conducted fully online, and included an automated geocoder for selecting travel locations in the diary data. PSRC found that the new data suggested a sizable difference in the transit and non-motorized trip numbers between 2006 and 2014, which was concerning. One panelist suggested that pedestrian (and bicycle) counters could be helpful to verify such results and this type of data is being collected more and more in many regions.
The household survey data was supplemented with an add-on survey of 1,200 households conducted in the spring of 2015, which also included a GPS sample of 250 drawn from the 2014 survey effort. The GPS data was collected using a smartphone app developed for the project. PSRC did note that the app for GPS data had some bugs. For instance, the app would sometimes not recognize the start of trips. They also found that some stops would not be recognized as stops because their durations were too short.
One panelist commented that the survey may not capture certain activity purposes well, and the GPS data can be useful in validating the household survey data. PSRC was not sure that was the case, and also found that the modal shares suggested by the GPS data were similar to those found in the household travel survey. Another panel member was concerned that the GPS data app would ask too many questions (10 in total) of respondents whenever they completed a trip, and this may be a deterrent to answering the questions.
PSRC is planning on shifting to a more continuous data collection strategy, rather than the typical large surveys that are conducted by regions every five to ten years. They are also considering collecting this data mostly through smartphones in order to avoid gaps due to under reporting or misreporting of trips. There are issues PSRC is working through on this front, including how to combine data across years for use in model estimation and ensuring comparability of different methods.
One area that PSRC is investing resources is in some new modeling directions including the following:
Visualization Tools
Tableau is a tool PSRC has explored for developing visualizations of model outputs, but it has some drawbacks including that it is rather rigid in how it is programmed, making it difficult to adapt easily. One panelist commented that another issue is that it is public, and there are certain types of information that an MPO may not want public for certain scenarios, or there could be issues with proprietary data. Another panel member thinks RStudio is a very good visualization tool, noting that it can be scripted and is free. However, it requires a certain amount of overhead cost to get comfortable with coding in JavaScript.
In addition, to Tableau, PSRC is experimenting with iPython Notebook, which they have found to be easy to use, interactive, and a powerful visualization tool. One panel member had different opinions, noting that it would be a challenge to use as an end script with the full model to produce automatic summaries.
Web Mapping
PSRC has experimented with CityPhi 3D visualizations available with the Emme software package. They have found that it can generate some impressive graphics but have not figured out how to tell a coherent story with it or how to make it really useful to them.
Open Street Maps
PSRC has been experimenting with Open Street Maps, which is a free world map created by volunteers with local knowledge. PSRC thinks this could be a way to ensure consistency in the networks used across partner agencies, including a repository where partner agencies could put new projects for forecast year networks. On the other hand, PSRC is concerned about the time and effort required to make such a tool work, as well as data oversight concerns since the tool is publicly accessible. A couple panelists agree that it could be challenging to generate networks directly from Open Street, but noted that there are others working on this very issue, so there could be information or resources to start from. The New York region may be the furthest along in this regard, so PSRC might reach out to someone there. Overall, the panel thinks that resources like Open Street Maps should be embraced, since they have so much of the information that MPOs like PSRC need.
Modeling Dashboard
PSRC has developed a model dashboard, which allows for managing model runs remotely. The tool archives run inputs and outputs, controls runs submitted by external users, and avoids privacy issues for employment data (since it need not be shared for a model user to submit a run). The tool is a GitHub repository, written in Python. Currently, it is not completely finished, and PSRC has only used it internally to this point. PSRC believes the tool will help with reproducibility of model runs, because it takes a snapshot of inputs, which will be helpful when PSRC starts allowing other partner agencies to use the tool. However, there are a number of roadblocks to getting the tool operational, including the web front end to specify and run models, that some of the data used by the model is proprietary, model run times, and licensing issues with Emme software. Despite these obstacles, the panel members were generally positive about the tool and encouraged PSRC to continue its development.
Bicycle Modeling
PSRC has begun doing some of the initial steps that will be needed for modeling bicycle trips. This includes coding perceived travel times on links, which accounts for important bicycle attributes like slope, facility type, and vehicle volumes. Model coefficients have been calibrated, starting with those used in the Portland Metro area. PSRC currently has the ability to run bicycle assignment (using the perceived travel times) and pass skims into the mode choice model. One challenge they have found is that the network does not include many local, minor roadways, which bicycles often utilize more than automobiles do. Panelists suggested that Open Street Maps might be useful to this end.
At present, PSRC has a simple truck model. However, there are stakeholders in the region that would like a more sophisticated model. PSRC is currently collecting more truck count data so that they can better calibrate the existing model, but they are not actively working on advancing model capabilities. PSRC was interested in finding out from the panel whether there were incremental improvements that they could make to improve the model, and in finding out when a more sophisticated model might be valuable from a planning perspective.
While freight is a major part of the region's economy, the questions typically being asked of PSRC related to freight are simple and typically short-term. For instance, they may get questions related to routing of trucks through Seattle or truck movements in and out of ports or major industrial areas. A couple of panel members thought that a behavioral freight model was not needed to answer these types of questions. PSRC only needs a simple tool to answer strategic questions about truck movements. One panelist suggested that PSRC try to get local carriers to submit to the American Transportation Research Institute (ATRI) to get good origin-destination (OD) data on truck movements. Overall, the panel generally agreed that freight modeling was not a particularly important avenue of modeling improvement for PSRC at this time.
Dynamic traffic assignment (DTA) is a procedure that PSRC ultimately wants to incorporate into SoundCast. However, it is a question of when should PSRC do this and when can they do it.
One panel member argued that PSRC should wait on adopting DTA and learn from what other regions have done. At this time, there are not many regions with DTA experience. In the experiences this panelist has had with regional DTA, there have been huge issues, including calibration, run times, and responsiveness of the model (e.g., too much congestion predicted).
Another panel member suggested taking a step back and thinking about the purpose of implementing DTA. Static assignment does not produce reasonable delays with heavy congestion, is not useful for many operational strategies, cannot work with reliability measures well, and does not give enough information back to the demand model. This panel member's experiences have been more positive, though on much smaller scales (e.g., regions with 100,000 population, as opposed to several million). In larger regions, sub-area analysis with DTA is possible, though it does not allow for speed feedbacks with the demand model and supply/demand equilibration. This panelist would like to implement a simplified DTA (e.g., that does not consider signals) in a large region with ABM, thinking that this would provide considerable gains over static assignment.
In the San Diego region, a regional DTA implementation project is already underway. However, this region is smaller, has simpler geography than Seattle (which has mountains and large bodies of water), and has a number of data advantages, like automatic signal timing information from a large portion of signals in the region. Even still, one panel member believes the San Diego region is still at least two years away from having an integrated DTA with the ABM.
One panelist suggested testing INRO's (the makers of Emme transportation modeling software that PSRC is already using) DTA solutions. Multiple panel members suggested MATSim as an alternative. While it is not really a DTA, regions like Toronto have found success with it. In that case, the region settled for a simpler demand model but wanted a DTA, due to a different set of priorities for the region. The panelists suggested that MATSim can get up and running fairly quickly without a great deal of effort (e.g., several days). As MATSim is free, this would also have the added benefit of removing proprietary software from the modeling process, since that software's primary use is static assignment.
PSRC has studied and experimented with a number of other analytical tools. These include the following:
Benefit-Cost Analysis Tool
PSRC's benefit-cost analysis tool translates a variety of transportation measures onto a common scale to evaluate projects and policies. The measures used by the tool include the following:
One panelist really liked the idea of being able to look at a portfolio of projects in an economic framework, comparing them side-by-side. Other regions are doing similar things. In the San Diego region, work has been done with a similar style tool that streamed benefits depending on when projects are forecast to come online, and included things like discount rates.
One issue that can come up with benefit-cost analyses, as pointed out by one panel member, is how to deal with travel time changes. The problem arises when a change actually worsens travel times, but utilities are improved (due to other factors). PSRC noted that they would really like to be using mode-destination logsums as evaluation measures rather than travel times, since those would incorporate any change to the utility functions and can be valued using the marginal rates of substitution between utility and costs. However, PSRC has struggled to operationalize it. One of the panelists suggested that no matter how you compute benefits and costs, something will always be missed. The challenges are what to include, how to value it, and how to reconcile that mobility changes are going to dominate no matter how benefits and costs are computed.
PSRC stressed that their aim with this tool is to tell a coherent story about how a project affects people. There are challenges with this. One panel member suggested that for many people, certain projects will not actually provide benefits and may actually harm them (e.g., a new transit line only helps the people that use it, and could hurt those that do not use it, but pay for it in taxes). Another panelist noted that it can be problematic that the models are always looking at an average day, and on average, a project may only save one minute. On the other hand, if cumulative benefits (over the life of a project) were presented, the numbers would appear more substantial, but would not be as relatable. Another challenge emerges when a project generates revenue, as the impact to individuals may be typically negative.
Health Impacts Modeling
PSRC has been experimenting with the World Health Organization's (WHO) online health impact tool known as Health Economic Assessment Tool (HEAT) for evaluating health impacts of active transport modes. They chose this tool because it is very user friendly and is interactive.
One of the panel member's regions has looked at health with a tool integrated with the ABM. While this tool is new for the region, the panel member's region had a champion for its implementation in the health department. This panel member was skeptical of the value of tools like this, cautioning that the correlation between health and regional travel may not be very significant. In this case, the panel member sought guidance from health experts to make valuations of the impact that additional walking and biking can have. Relying on health experts removes some of the risk for the MPO.
Another panel member cited experience in one region, where valuations of health benefits were performed by health experts, but those numbers were ultimately discounted by a significant amount (e.g., an order of magnitude). Other regions have used valuations as is, without discounting. Overall, this panelist believed that in order to obtain satisfactory benefit valuations for walking and biking, health benefits have to be part of the equation, since these modes will never compete along the dimension of travel times.
Highway Operational Modeling
Operational improvements on the highway side have replaced mega-projects as the way forward, for the simple fact that there is not sufficient money to invest in new infrastructure and maintain what already exists. PSRC would like the ability to capture operational improvements in static traffic assignment model component. Two panelists offered that increasing capacities on network links to approximate operational strategies with static assignment would tell the wrong story, and cautioned against it. In one region, a panel member noted that operational strategies are going to be considered in the model until dynamic traffic assignment (DTA) is implemented. Another panel member suggested that DTA will be easier to implement, the longer PSRC waits, since they will benefit from the experience other regions have with implementing DTA.
Pavement Condition Impacts on Modeling
One issue PSRC is struggling with is that a large portion of the transportation funds available for roadways in the region (e.g., about 60%) is dedicated to maintenance. They would like a way to quantify the tradeoff in the level of operations and maintenance investments. Unfortunately, PSRC does not have pavement data. While Washington DOT does have some data, it is only for state routes. The panel did not have many comments on this topic.
FAST-TrIPs
FAST-TrIPs is a dynamic transit assignment tool that is schedule-based and disaggregate. PSRC would like to use it to account for the effects of crowding, reliability, and taste variations across riders. The work PSRC is currently doing is more on the research side of things, teaming with San Francisco County Transportation Authority (SFCTA) and Metropolitan Transportation Commission (MTC). One panel member commented that PSRC already has all they need to implement FAST-TrIPs with SoundCast, since the ABM is disaggregate and includes all the traveler information that would be used by FAST-TrIPs.
Transit Sketch Planning Tools
PSRC talked briefly about some sketch tools they have developed in-house as well as potentially new ones. They are unsure whether to expend much effort investing in such tools or whether focus would be better spent on the regional model. PSRC has built sketch tools for transit agencies in the past, but those agencies found the tools were not working for them, and ultimately invested in other tools.
Conveyal (conveyal.com) was one tool mentioned by the panelists, which is a data-driven, open source transportation planning tool. This sort of tool could be useful for transit providers that are interested in simple tools that they have access to easily. One panel member suggested that any sketch tools developed by PSRC should be connected to SoundCast. PSRC should avoid situations where internal consistency between SoundCast and sketch tools is not achieved. The panelist also suggested focusing on a limited number of tools, but making sure the tools PSRC maintains really perform well, which will also help with credibility.
