Most of the first day of the peer review panel was spent by DVRPC staff members making presentations on specific items to the peer review panel. During these presentations, many topics came up which provoked discussion among peer review panel members and between panel members and DVRPC. This section documents the key points that arose during these presentations.
One of the presentations made by DVRPC to the peer review panel dealt with the region's household travel survey (conducted in 2012-2013) and an on-board transit survey (conducted in 2010-2011).
DVRPC noted that transit-dependent areas were oversampled in the household travel survey to ensure survey data was collected for this critical segment of the population. Expansion processes included expansion over six or seven dimensions (including income, area type, vehicles, and others). One panel member expressed concern over the expansion process with regard to oversampling and recommended comparisons be made to observed Census data to ensure the data is expanded appropriately. Another panel member noted that often when reporting directly on survey results, unweighted statistics are sufficient, as long as the biases and limitations of the sampling process are adequately described. When expansions are really needed, they should be context specific, and no one set of expansion factors will serve all purposes.
One concern noted by DVRPC was that they obtained different expanded trip totals depending on whether person or household trip rates were used. The panel noted this to be typical of expanded surveys. However, a couple of panel members were concerned by the reported trip rates, and believed them to be too low, and that typically household trip rates are near about 10 trips per household. This could be related to the number of respondents that did not travel (about 20 percent), and additional attention may need to be paid to those records. Other panelists suggested this may be a result of underreporting of short trips, and this could make trip rates low by as much as 15 to 20 percent. This could be supported by DVRPC's finding that when they used the sample of GPS survey results to compute trip rates, the trip rates were generally higher than the household travel survey results.
DVRPC's on-board survey consists of a sample of 22,000 transit riders, of which 40 percent stated having no other vehicle and 40 percent stated having no other way to travel as reasons for using transit. Several panel members commented on the response rates (14 percent return rate, 2 percent of overall riders), which seemed quite low to the panel members considering the survey was administered via paper surveys. Another panelist commented that DVRPC should be careful on expansion of these surveys, because a lot of people will refuse to fill them out, and this could result in missing key population segments.
The panel recommended that DVRPC become more familiar with their survey data and expansion factors. Building a better understanding of the data may answer many of the questions that arose during the peer review meeting. The panel also emphasized that understanding the data prior to model estimation and calibration (of DVRPC's new activity-based model) can be very useful to the model development process.
DVRPC maintains several screenlines and cordons, where they have traffic counts in several locations along a specific line. One key screenline is the Delaware River, for which DVRPC has counts every five years dating back to 1965. Overall, DVRPC's traffic count data is extensive, consisting of over 70,000 stations regionwide, including hourly counts in every case and many with 15-minute counts (over 24 hours). One unique feature of the traffic count collections for DVRPC is that they administer the counts themselves, rather than obtaining traffic counts from the State Department of Transportation (DOT) or individual counties.
One validation measure DVRPC uses is to check VMT versus Highway Performance Monitoring System (HPMS) data by county and roadway functional class. Several panel members were skeptical about the validity of HPMS data for such regionwide validation and calibration. One alternative suggestion was to use vehicle inspection data.
DVRPC also collects transit boarding data by time of day at all rail stops and collects alighting counts for heavy rail. Some permanent bike counts are taken on some of the region's bike trails, and permanent pedestrian counters also have been established in certain locations. In particular, bike and pedestrian counts are maintained for a cordon around the city. The panel generally agreed that these traffic count collection efforts were useful sources of information.
DVRPC has a large amount of freeway speed data coming from HERE sensors. The speed data on arterials comes from INRIX, and while it is less robust it is expanding in coverage. Several panel members suggested the use of HERE data, which many of the panel member agencies are using (DVRPC currently is not using this data). Another panelist noted that each speed dataset has its own strengths and limitations, so it is not necessarily that one is better than the others, but they can be used in different ways.
DVRPC's existing travel model was described earlier in section 3. The model produces a number of reports, including reports related to traffic assignment, comparisons of transit boardings to counts, comparisons to CTPP data for journey-to-work, as well as others. One issue discussed during the meeting was the household segmentation used for the existing travel demand model. The existing model does not segment households by auto ownership. The panel believed that auto ownership is a key input to travel models that allows the modeler to better understand transit dependence.
The panel also discussed validation, and one panelist suggested that reasonableness checking is a long process that takes place over time and includes testing many scenarios and looking at reasonableness of model results. It is important to identify what types of changes are expected under different scenarios and check the model results to see if they match expectations.
Another panel member commented on the absence of intersection control delays in the existing models traffic assignment routines. This may account for as much as 80 to 90 percent of total delay. The panel member believes that incorporating control delays is of great importance in order to get the model calibrated properly. The panel member was surprised that the model validated so well for the downtown area, given this limitation, since the downtown area is dominated by signalized intersections.
DVRPC has spent some time exploring dynamic traffic assignment (DTA) procedures in small-scale applications with their existing model. So far, DVRPC has used DTA for some special applications (e.g., evacuation of the city center, I‑95 reconstruction analysis, and analysis of center city bridge reconstruction). There is a strong interest in the region in evacuation modeling, which DTA would be useful for, and in addition, DVRPC believes it would be useful to evaluate time-of-day tolling and other CMP strategies. Most of the tests and analyses that DVRPC has performed have been done using VisSim DTA software, but they also have explored DTALite and DynusT.
Overall, DVRPC outlined several items that they believe are important for modeling in their region that could be better handled via DTA procedures:
Simulation-based DTA also was a topic brought up by DVRPC. Specifically, DVRPC has wondered whether simulation-based DTA is more accurate than macro- or mesoscopic methods and whether it is even feasible at a regional scale. A couple of panel members suggested that origin-destination matrix estimation (ODME) techniques can be useful for microsimulation efforts. However, one of the panelists was skeptical of the accuracy of ODME techniques, particularly in cases when the project being evaluated could be significant enough to change demand profiles, and thus, the underlying ODME trip table.
One panel member warned against trying to have DTA do too much, contending that some organizations are attempting to use DTA for forecasts as far out as 2040. The panel member argued that DTA is really most useful from an operations and corridor analysis, and less so for long-range forecasts. The primary reason for this deals with the (in)accuracy of the trip tables being used for those analyses.
Another item of discussion was to what extent were adjustments to demand profiles and networks needed in order to calibrate the DTA applications DVRPC already has performed. DVRPC noted that counts were used to make adjustments, but that demand did not need to be adjusted much. One panel member believed this was likely a result of the downtown cordon counts DVRPC had to calibrate the regional model with. This cordon likely resulted in a very well-calibrated model for that area, so the DTA application for that area worked well.
Overall, the panel members had mixed experiences with their own work testing DTA techniques. One panelist found the data needs to be quite extensive and warned that DTA may not be feasible on a regional scale. Another panelist had a very rough experience getting DTA to work on a subarea analysis. The panel member wondered, if DVRPC was able to get DTA working on a small subarea, then what is the difficulty (at least in theory) to move to regionwide implementation. DVRPC suggested that it would not necessarily take a lot of work to get DTA to work at the regional level since it already was implemented at the smaller scale, but it could require a lot of adjustments to get accuracy.
A key question posed by DVRPC was how important DTA is in the context of activity-based modeling. One concern DVRPC had was that they would end up losing a lot of information from an activity-based model run if they used traditional static assignment procedures and not DTA. While the panel agreed that this is a valid concern, one panelist warned that ABMs do not have the ability to accurately inform origin-destination time-dependent flows, and thus, any DTA would be starting from a point with a substantial margin of error. On the other hand, most of the panel thought this was a worthwhile endeavor. And, while the panel was a bit skeptical that DTA could be achieved at the regionwide level, they thought there was value in implementing DTA at a smaller scale and gradually ramping up the extent of the implementation, and the panel recommended that DVRPC continue this effort. While this is ongoing, another panel member suggested that DVRPC continually be improving the network to be as realistic as possible, giving DVRPC the best chance to succeed in DTA implementation.
Another topic that was discussed was land use modeling. While DVRPC currently has UPlan, they do not utilize it much, instead relying on land use plans for long-range forecasts, typically developed by DVRPC's land use planning group. Part of the reason for this is that UPlan's functionality does not match DVRPC's land use modeling needs very well. DVRPC describes the model as a land consumption model, rather than a land use model. It mostly forecasts where new development will occur (at a grid level) using total growth as an input to the model. However, it does not forecast redevelopment of currently developed land.
One panelist felt that UPlan does not work well at a regional level because there is too much freedom at that level for the model to place new development. This panelist's agency uses a multilevel tool for growth, densities, redevelopment, and new development. In addition, it requires good communication with the modelers and land use planners in order to identify locations that are planned for development. While this may seem like a lot, it is doable. The panelist believed this sort of tool is really what is needed for a regional-level land use forecast.
Another panel member commented that land use models are really not very accurate, they are simply tools, so leaning toward simple is probably better. The tool need not be integrated with the travel model, though one panelist thought it was important for the land use model to be informed by accessibilities, which would probably be computed by the travel model. The panelist also felt that an in-house approach is typically better since DVRPC would understand how it works (and thus, the deficiencies, etc.).
The panel recommended that DVRPC use a land use modeling tool that is, at least, loosely coupled with the travel model (e.g., via accessibility measures to inform the land use model). In addition, more land use data should be collected and maintained. A data purchase of employment information may be useful.
Another modeling tool DVRPC would like to pursue is an economic analysis tool, capable of quantifying project benefits and costs, including the broader project impacts at a regional level. DVRPC is pursuing an application of the TREDIS model which can be used for these purposes.
DVRPC suggested that one of the key features of TREDIS is that it can be used right out of the box without much calibration. However, one panelist was highly skeptical of this statement, advising that the model is very complicated and takes a lot of effort to make it work at the regional level. According to DVRPC, TREDIS does a lot of the setup when the tool is first purchased (e.g., setting up county-level indicators), but the panelist was not so sure that it was really that simple.
The panel agreed that tools such as these can be useful to agencies, by producing a single measure of valuation that can be compared across projects. However, there are problems associated with this type of tool. Typically, these tools favor large highway projects, while transit projects and bicycle and pedestrian projects are found to be economically unviable.
Given the potential problems with using TREDIS, DVRPC asked what types of tools the panelists' agencies use for economic analyses. There are basically two types of such tools. One type are proprietary tools, such as TREDIS, that are developed by third parties and must be calibrated. A couple of panelists agreed that if DVRPC is willing to put in the effort to make a proprietary tool like TREDIS work and make sure results are reasonable and make sense, it could be viable. The other option is a tool developed in-house by the agency. An in-house model seemed to be favored by at least a couple of panelists though they noted that it would open the agency up to criticism. In-house models would have several benefits, particularly since DVRPC is concerned that transit and bicycle projects need to be evaluated on equal grounds to highway projects. In light of this concern, one panelist suggested inclusion of more abstract benefits in the tool, such as health benefits.
Another panelist worries that these analysis tools are too process driven, and not results driven. It will be extremely difficult to validate these tools because the true value of a project will never be known. For instance, questions like how many jobs are created are immensely difficult to assess. DVRPC noted that given the noise in most (or all) economic data, the projects that they are interested in valuing are far too small to validate, even if the data was available. This requires then that the tools be process driven. Nonetheless, the panelist felt these tools simply are not very good at assessing impacts to larger economic indicators. Instead, the panelist suggested that the focus should be based on things the travel model does well, and quantify the value of those things. Part of the reason DVRPC would like a separate tool, however, is that it detaches the economic valuation from the deficiencies that are known in the model (any model).
The panel recommended that DVRPC pursue an in-house tool for economic analyses. It should start off as something rather simple, but allow for incremental improvements over time. The TREDIS model, however, should be retained in order that DVRPC staff can better learn how economic models work and use those modeling fundamentals in their own tool.
The presentation and subsequent discussion of DVRPC's activity-based model design covered multiple topics, including value of time (VOT) user classes, toll modeling, time-of-day period definitions, and model estimation, as well as other topics.
One key element of the activity-based model will be the way VOT user classes are defined for transit skimming. DVRPC's preferred approach relies on defining several VOT user classes, and generating different transit skims for each. Then DaySim can use the most appropriate skim for each simulated individual. DVRPC believes this approach will improve the model's ability to forecast the effects of different transit policies though additional computing demands will be needed to accommodate these processes.
One panel member liked the idea because it has the potential to reduce or eliminate completely different transit constants that are typically needed in mode choice models. The idea is that the user classes have the potential to explain most or all of the variation in individual choices not picked up by other parts of the model, thus, reducing the need for such constants. The panel generally seems to agree that the VOT user classes will provide value. The idea that these user classes may be a latent variable also was discussed. One panelist suggested that testing be performed during model development to identify how important VOT user classes would really be to overall model performance. This could save DVRPC time rather than making decisions prior to model development, which may make it more difficult to change in the future.
The key point of discussion related to toll modeling dealt with how to model the path choice of a tolled path versus nontolled path. DVRPC's preferred approach is to handle toll choice in DaySim (probably as part of the mode choice model). This would require generating auto skims for each of three or more VOT user classes, assigning a skim to each individual based on predetermined VOTs for the individual (e.g., using lognormal distribution), and then modeling toll choice for auto travelers. The alternative to this approach is to allow Visum to handle path choice within the traffic assignment module. A key reason for the preferred approach is that it allows that a lot of personal characteristics be included in path choice, whereas traffic assignment would require much more aggregation over key personal attributes.
One key problem with this approach was brought up by one panel member. By having toll choice in the mode choice model, an inconsistency emerges in path choice, where it is difficult or impossible to ensure all those choosing tolled paths actually travel on tolled paths. Nonetheless, the panelist believes that in some regions the choice between toll and transit is very key, and this can really only be treated appropriately if the toll choice is handled in the mode choice model.
Another panelist disagreed, arguing that toll choice should be considered in the traffic assignment model component. The panelist suggested that perhaps household travel surveys are not well suited to model toll choice anyway (toll user surveys are more appropriate), so it would be better to allow the path choice algorithms in traffic assignment to perform this function since the functionality already exists. Moreover, the panel member argued that toll versus transit choice can be handled even if toll choice is handled in assignment. It is simply a matter of feeding tolled auto skims to the mode choice model, rather than free auto skims.
A third panelist did not think there was a clear choice, both having merits and flaws; however, the panelist suggested that this was another area where testing prior to making a final decision would provide some real value. Another panelist was not so sure testing would be very useful here, since the model could be calibrated using either approach.
The model design presented by DVRPC calls for different time-of-day skimming definitions for highway and transit. For highway, 12 time-of-day skims are planned: 5 skims in and around each peak period, 1 midday skim, and 1 overnight skim. The idea is that the detailed skims around the peaks will allow DaySim to better forecast peak spreading and time-of-day pricing. DVRPC also plans to use "warm" starts at the beginning of a model run, which should provide for faster overall model convergence. For transit (at least), 5 time-of-day skims are planned, but only 4 assignment periods are planned.
A comment made by one panelist was that time periods not dip under one hour, as to avoid problematic issues dealing with trip durations that exceed the time period duration itself. The panel believed that DVRPC's approach was probably substantiated by the needs they discussed. For instance, detailed time-of-day modeling would certainly be important in order to evaluate peak tolling policies, as mentioned by DVRPC.
However, other panelists disagreed. One panelist believed the highway time-of-day periods might be more than is necessary, and that simple is probably better here. This panelist's experience suggested that highly detailed time-of-day models could be problematic in practice, allowing for too much model sensitivity. The models allow for too much shifting in time of day, whereas many people have a much more limited choice.
Another panelist had similar concerns, raising doubts that peak spreading is much of an issue for a slow-growing region like the Greater Philadelphia region. There is data, however, that is available that could be used to verify the region's needs (e.g., HERE data). From that, five-minute data could really help to inform what is going on and whether such detail is needed. The panel member also suggested that we should strive for our models to be able to answer the questions that come up the most and for the models to be able to answer those question really well. Building overly complicated models to answer rare questions (that come up only 10 percent of the time for instance) is unnecessary and potentially problematic. The types of questions that detailed time-of-day modeling could answer are the type that are rare. There was some dissent on this issue, however, as another panelist argued that even if those questions are not as common, they can be critical and the only way to get the needed policy sensitivity to answer many such questions is via the detailed modeling DVRPC proposed.
One panel member also commented on the transit skimming approach, and argued that time-of-day transit skimming was really unnecessary. The claim was made that there may be little interest in being able to accurately identify transit riders by time of day. And, even if there is, this could be accommodated via postprocesses.
DVRPC left open the question of how much model estimation would be done in the development of their new activity-based model. Their overall approach is to transfer the existing model from the Puget Sound Region Council, and to limit time spent evaluating different alternative model specifications. Instead, DVRPC would like to reestimate many of the parameters of the existing PSRC model using local data, and set priorities among models and parameter types. The primary alternative to this would be to directly transfer the PSRC model and adjust parameter values only during model calibration.
While one panelist commented that it is really difficult to assert parameter values because it is not clear what values to transfer and what values need calibration, the panel generally agreed that model estimation specifically for the Philadelphia region was not the most important thing. One panel member suggested that this boiled down to the question of whether DVRPC's time and resources could be better spent elsewhere in model development than on model estimation. Another panel member suggested that DVRPC run the PSRC model to see what kinds of results it produces. This could help inform the decision and may not be overly costly, since the model components will be generally very similar regardless of whether model estimation is pursued or not. This overlaps nicely with the notion of getting the model to an assignment as quickly as possible, suggested by multiple panelists. Assignment tests can be incredibly valuable to understanding what the model is doing well and what it is not, and so having this during model development could help to inform many model development decisions.
Another panelist commented that there could be some very important differences between the Philadelphia region and the Seattle region (from which the PSRC model comes), so some model estimation is probably a good idea. Moreover, model calibration is much more time consuming than model estimation, and setting up the model structures is really the most time-consuming part of model estimation, which should already be done for DVRPC since they are transferring the model. Thus, spending some time estimating models could be of value. However, the panelist warned that testing alternative model specifications can be quite time consuming since there is no clear point where all alternative specifications have been tested (there is always more that could be explored). From that perspective, the panelist was pleased to hear that DVRPC is not planning on testing a lot of alternative specifications, and instead, focusing mostly on updating parameter values only.
Another point of discussion among panelists was the tradeoff between incorporating sensitivities into the model versus overfitting the estimation data. One can always achieve models that better fit data, simply by adding more variables; however, one panel member worried that oftentimes sensitivities get added to models where they do not exist or are not measurable. The panelist also was skeptical that these model sensitivities provide much value since it is very difficult to validate some of the models. On the other hand, another panel member argued that these sensitivities can be very important, and it is easier to have it in the model and not report on it than to add it in later when a question arises. If the model is insensitive to a policy question, it can have the effect of suggesting to an outsider that their concern is not significant, and from a political standpoint, that may be unappealing.
Overall, the panel did not make a recommendation as to whether DVRPC should pursue model estimation, but they did recommend that DVRPC develop the in-house capacity to estimate the models. This would ensure that if models needed to be reestimated, DVRPC would have the capability in‑house.
A common theme in discussing the ABM development plan was that DVRPC keep using and maintaining their existing trip-based model for some time, even after the ABM is fully developed. One panel member remarked that the new ABM will need to be tested slowly and over time. The existing trip-based model cannot simply be abandoned. The panelists had mixed experiences transitioning to the ABMs developed in their regions. In one case, the ABM and trip-based model gave some very different results, which makes it hard to use the ABM for any policy analysis, as the discrepancies call into question the whole model. In another case, however, the ABM has been used very successfully for planning purposes because the MPO was able to manage expectations for the model. Moreover, the MPO was able to tell better stories with the model that intuitively made sense.
Another common theme brought up at several points in the discussion was that data inform the decisions DVRPC makes. Several panelists commented on how good the data is that DVRPC has at its disposal. Using the data to help inform the modeling decisions could be valuable.
Another comment made by one panel member was for DVRPC to be comfortable with the modeling code for which the ABM is being written. The ABM is being coded by an outside contractor, but the panel recommended that it is important for staff at DVRPC to be able to understand and make changes to the code in the future. With this in mind, it will be important for DVRPC staff to be knowledgeable with the coding language in which the model is written.
The panel agreed that DVRPC should be doing more with their existing model and the new ABM in terms of multiuser class assignment. There is no reason not to be doing this, and the panel believes it to be important though it does increase run times significantly. In addition, the panel commented that the additional travel components outside the ABM demand component be put on a separate track. This includes models for special generators, trucks, and externals. For the time being, estimating trip tables for these trips will be good enough, and in the future, these trip components can be explored more fully.
The panel made a number of recommendations for the activity-based model development, as detailed in section 6; however, there were a number of key themes throughout the discussion that may deserve emphasis. The panel believed it was important to keep the model simple, to the extent possible. This will ensure the model does not become a black box, which may result in credibility issues for DVRPC. The panel also recommended that the activity-based model be checked and rechecked during the course of model development. The model results can be checked against the existing travel model as well as data sources, but also can be tested via scenario testing to better understand the implied sensitivities of the model. The panel emphasized getting to a traffic assignment as quickly as possible, as that is the stage where issues are often revealed.
One issue DVRPC has identified with their existing travel model is that it does not produce robust transit results. With this in mind, they have considered developing an operations model tool for transit. The tool would be developed and maintained by DVRPC, but primarily used by DVRPC's transit planning partners. The tool would work on a smaller scale than the regional model and would likely require that zones be split in many cases, but it would use information from the travel demand model (e.g., skims and trip tables). At this point, DVRPC was unsure whether it would be better for the tool to directly measure tradeoffs between auto and transit modes or whether it would work more as a diversion model between transit modes (or potentially both).
DVRPC noted that the tool would provide more detailed analysis of transit for small-scale transit projects that may be insignificant to the regional model. The analyses that it would be used for include stop-level analysis, service type analysis, diversion of certain lines, collapsing lines, etc. It would provide a means to evaluate such projects.
A couple of the panel members were initially opposed to the idea of a separate transit operations model. DVRPC already has a model that they must maintain, and developing and maintaining another tool could be too costly and has the potential to create inconsistency between the models. One panel member suggested relying on the regional model, and using Visum to create realistic transit profiles via assignment of transit trips.
Another panel member thought this type of tool could be useful, and the panelist's region does something similar in long-range planning. Their key need was a tool that put smaller projects (like transit) on a more even playing field with highway projects, from an evaluation standpoint. However, the panelist warned against calling the tool a model, as that could cause confusion with stakeholders. It also was noted that there are tools like this that may be available for purchase.
A number of suggestions were made to DVRPC if they decided to pursue this tool. One suggestion was that it use fixed trip tables and use an incremental approach to forecasting the effects of changes in the systems. Another possibility once DVRPC's ABM is developed would be to pivot from the transit trips coming from the ABM, for which a great deal of information is known, like personal characteristics, but also trip information like precisely what an individual is paying. The finer-grained zone system being used for the ABM also will be of particular value for a tool like this, but some sort of GIS tool that generates a zone system specifically for a scenario would be of great value as well. Most importantly, the panel agreed that if DVRPC pursued this tool, they should keep it simple but scope out a very specific plan for what the tool will be.
Another concern of DVRPC's was that they have very few tools capable for evaluating bicycle and pedestrian projects. They would like something for these purposes, and make it be an off-model tool similar to the transit operations modeling tool described in the previous section.
Several comments were made by the panel related to this. First, one panelist mentioned that TRB recently released a book on bicycle demand estimation that could be referenced. Second, it was noted that ABMs do not necessarily forecast walk trips very well due to how short they are and how much chance there is for spatial error associated with them. Nonetheless, one option is for DVRPC to look at tours generated by the ABM (once it is developed) that have trips that could plausibly use walk and bike modes. If a tool could be developed to generate bike and walk networks (for a small area) on the fly, the effects of changes to those networks could be evaluated fairly easily. The idea of using the set of plausible bike and walk trips from the ABM was shared by multiple panel members. Regardless of how it is implemented, the panel recommended that the tool be an extension of the model.
