The peer review discussion consisted of several parts. First, a representative from the OIPI described their goals for the statewide model as a planning partner to VDOT. Second, VDOT presented some of the specifics of the statewide modeling plan and goals from their perspective. The details of OIPI's and VDOT's goals are described in the previous section. Finally, the panel members commented on and answered many of the questions VDOT posed to the panel, as described in the previous section.
As described in the previous section, the OIPI would like the statewide model to be used in the project prioritization process. The model should be adaptable in the sense that it could be updated or modified as new approaches become available. OIPI would also like to use information from the model for the multimodal, long range plan for Virginia.
As one panel member mentioned and confirmed by VDOT and OIPI, VDOT is looking for objective measures that one could compare across projects. The legislation passed in Virginia proscribes that the same measures be used everywhere in the state. Other tools are being developed to actually prioritize projects, partly on the basis of those measures identified by the model. The panelist cautioned that there are always uncertainties in model forecasts, which necessitates a subjective layer of evaluation in addition to those objective measures. This is critical to understand that uncertainties exist in the forecasts, and to understand what actually drives the results. Another panel member warned that developing objective tools can be very difficult, and perhaps impossible. There will always be cases where certain projects or types of projects perform poorly with respect to certain metrics, and different metrics may be needed for those projects.
A couple of panelists commented that the statewide model should focus on addressing things that other tools cannot address at present. VDOT and OIPI have a myriad of tools at their disposal already, so identifying what those tools do well, but more importantly, what those tools do not do so well is important. Another panel member noted the importance of different tools being able to talk to one another in some way, or for there to be some internal consistency among the different tools that are used, so that conflicting results do not arise.
VDOT's overall modeling plan and goals for the model were described in Chapter 3. These were presented to the peer review panel, and several specific areas of the statewide model were discussed in more detail.
Values of time (VOTs) used in the model were one key area of concern for several panelists. One panel member noted that while passenger VOTs tend to be rather stable, truck VOTs vary considerably, and the literature on truck VOTs may not be applicable everywhere. And, it will be important to determine truck VOTs, since trucks may be very sensitive to tolls. The panel member also commented that passenger VOTs should not be assumed to be equal across modes (e.g., passenger car vs. train), since, for instance, one can use time productively on a train and not while driving. Other panelists suggested that VDOT may need to make assumptions about things like VOT, and test those assumptions using the model. Then, examine the results and decide whether they make sense. If not, perhaps the assumptions should be changed. The panelists also suggested that VDOT be very transparent about these assumptions and what those assumptions mean for the model.
Another panel comment dealt with using information taken from the literature (e.g., asserted parameters), which can be very valuable. If other studies have found one answer, and VDOT's analysis suggests something contrary, then results like this should be carefully examined to determine why there is a discrepancy. Another panelist was in general agreement, adding that these discrepancies do not necessarily mean something is wrong, but they should be noted, and, if possible, should be tested and paid attention to over time.
Several panel members commented on how models should be used. One panelist worried that people tend to rely too much on their models. Sometimes other more detailed tools are needed to evaluate certain projects or project types. This may be a small scale modification to the model or adding additional information. Using the model results out of the box is a mistake, however, and the analyst should always be interpreting results and making sure they make sense.
Another panel member suggested that some scenarios that may be of interest simply are not capable of being modeled, which can happen for a variety of reasons. In such cases, it may be possible to use other tools for the analysis, but use the statewide model as a visualization tool and to get metrics that are of interest. In that sense, the model would be a post processor of sorts. One panel member's agency is doing something very similar to this. This may be what a lot of statewide models are doing. Another panelist disagreed in concept, and that typically models are run, just like an MPO model, but results and numbers are often "massaged" in certain ways to match expectations. Operationally, these ideas may be very similar, however, where the model is run, but results must still be interpreted, rather than being used without adjustment.
Questions posed to the panel are documented in Section 3.3. This section summarizes the panel members' responses to those questions. Due the number of questions posed by VDOT to the panelists and the limited time for the peer review, the questions were prioritized, and some questions were not discussed.
While the panel member responses to this question were often similar in some ways, there was a fair amount of disparity across panel member responses as well. Therefore, each panelist's response is detailed individually below.
One panelist remarked that their model is used to plan for accessibility, efficiency, and environmental quality. The model is used to address policy questions, such as truck corridor analyses. Future traffic conditions, congestion levels, and air quality are forecast using projected housing and employment. There are no tolls in the panelist's state so the model does not consider VOT explicitly, but will be moving in that direction in the future.
Another panel member noted their state had two primary needs from the statewide model. First, it is used for statewide planning for rural areas, between the states' many MPOs. Second, the model is used to examine impacts of the North American Free Trade Agreement (NAFTA). So the state's goals are varied, evaluation of small projects outside the jurisdiction of MPOs, but also planning issues dealing with how the state fits in on a broader, national and international scale. One of the key challenges of the model is linking up the local level needs with the statewide needs. The first version of the model was developed in 1998, and the state is now on version 3. The panelist noted that with each revision to the model, new uses for the model are identified. Lastly, the panelist remarked that the state is very aware of MAP-21 (especially related to the freight advisory committee), and this will likely become a key policy driver in the future.
A third panelist noted that the need for a statewide model in their state arose in the 1990's, when a lot of new policies and mandates came into effect. The panelist's agency was being left out of important planning discussions because the tools they had were antiquated, particularly relative to what was going in the key metropolitan areas. The model was originally intended to inform MPO models and to incorporate state economy and land use (particularly in response to the new policies which were primarily focused on land use). After using the model, they found that it was very applicable for economic scenario testing. The model is often used to provide bounds on growth, in order to help decision-makers deal with uncertainty in the future. It has also been used to examine specific scenarios, like a large earthquake, and it is used to aid in advancing the state's freight plans. The panelist commented that the agency is continually looking at how the model works, what it does well, and where it can be improved, and taking action to improve it when feasible. This state typically uses other tools for project prioritization.
Another panel member agreed with the general uses the other panel members described, emphasizing that freight was a big driver in the panelist's state. In addition, the model is used to drive project planning in rural areas without MPO coverage and as a scenario testing tool to inform policy makers. Overall, the model is used to evaluate a number of policies, including MAP-21 performance measures, smart growth strategies, the state's freight implementation plan, greenhouse gas emissions, and the state's transit plan.
A topic that arose from this question was the need to model transit in statewide models. One panel member was not aware of any statewide models implemented for transit planning, mostly because the models are not robust enough to get it right. Another panelist noted that in the panelist's state, transit was included in the model, but the detail came from the MPOs and it was not a priority. For intercity and long-distance passenger modes, they were mostly used to ensure good validation of highway measures.
Of the panel member responses to this question, common themes emerged in terms of key challenges. A number of panelists noted data challenges. Accurate socioeconomic and employment data can be difficult to obtain and surveys are typically too expensive to conduct frequently. Another key data concern is keeping all the data (especially input data) up-to-date and consistent with the state's MPO models. Dealing with rail and operations data can be challenging, particularly because it often requires extra time to clean the data. Other data items were also noted as concerns, including validation/calibration data (e.g., traffic counts) and network data,. One panelist commented that limitations with data can create limitations for the model.
Another common theme among the panelists was information technology (IT) and hardware and software issues. In one case IT issues originated as a legislative issue in the state. In several cases, obtaining the necessary hardware to run the model proved difficult, and getting the necessary space to store model run results can also be challenging.
Several panelists commented on staffing needs here as well. It is important to have in-house expertise on the statewide model, so that as issues arise, there are individuals capable of identifying sources of such issues and making modifications if necessary. Getting in-house expertise can be particularly challenging when consultants are retained to actually develop the model, as is the case in Virginia. In addition, it was noted that matching model sophistication to staff expertise can be beneficial. Overstepping the expertise of the in-house staff can create issues.
Keeping the model up-to-date can be an important challenge. For instance, it is desirable for the statewide model to be consistent with MPO models. This goes beyond the consistency of data sources, as described above, and includes consistency between the models themselves, and in some cases, integration of the statewide model with the MPO models. It is clear that maintaining consistency across these different tools is important, but it provides a key challenge to development of the statewide model. In some cases if a good relationship is maintained between MPOs and the state agency, these issues can be mitigated to some extent. But there is also always the challenge of keeping rural areas and areas external to the state up-to-date.
Another challenge that was discussed dealt with modeling rural areas of the state, which Virginia will also have. In one case, trip rates in certain rural areas were much lower than other areas, primarily due to standard of living considerations. In this case, it has been a key challenge to account for these differences in the model in a satisfactory way.
Other challenges were also mentioned. One panel member's state went through a lot of trial and error in development, and the panelist emphasized that keeping the model design simple can be important so that the model is ultimately utilized. Another panelist remarked that there is often too little time in the schedule to do everything that is desired from a modeling perspective. The one-year schedule that VDOT has proposed seemed very short to this panelist, and suggested that VDOT will need to be highly coordinated to accomplish this (e.g., by working in parallel, etc.). A third panel member commented on the accuracy of long-distance person trip flows, noting that there is a great deal of uncertainty associated with such flows. Another panel member felt that AirSage data (which VDOT is obtaining) could be useful to inform those flows, but such data may be biased in several ways, and thus, not the whole answer.
The panel responses to this question were, again, quite varied. Each panel member's response is detailed individually below.
One panelist noted that their statewide model is used for long range planning studies, like freight plans and statewide transportation plans. However, the model is also used for other purposes. For instance, the state's highway system is deteriorating, and so the model has been used to test what happens if different levels of maintenance and preservation are pursued, including a relatively small budget with little preservation and minimal maintenance, or a higher budget to maintain current conditions. The model has been used to evaluate impacts of a major earthquake and how the state's economy and transportation services operate in the event of prior mitigation compared to no preparation in advance. In addition, the model was used to examine the economic impacts of cracking bridges requiring weight restrictions over time on the state transportation system. The impact to commodity flows and the economy ultimately led to a plan for repairing many bridges. Visible studies such as these will lead to intense scrutiny of the modeling tools and methods, which should be planned for in the model development and application program. Thus, good documentation and a peer review process are very important. It is also important to report results in the form appropriate to the tool. For example, this statewide model reports relative changes and growth rates, not quantities for which it was not designed to predict.
Another panel member's agency uses the statewide model a lot for freight applications. For instance, that state's ports group uses the model extensively, and they have used the model for port-to-airport studies and to crosscheck freight commodity flow studies, usually conducted at the MPO level. The model has also been used to answer questions regarding commuter rail, interregional analyses, and corridor studies.
Like the agencies above, a third panel member's agency uses the model for corridor studies and for the state transportation plan. This agency also uses the model for a lot of scenario analysis, including evaluation of smart growth strategies, managed lanes (e.g., tolling and HOV), and accessibility studies. This model includes a time-of-day component, which they use to get efficiency measures from the model. In addition, this state recently started to integrate the model with SHRP 2 - C11 products for estimating economic benefits.
Another panelist's agency is using their statewide model for a variety of applications. One model application that has proven very beneficial for the state is in providing external flows to the MPO models. They have also used the model a great deal for estimating diversions to new roads, and diversions resulting from road closures or work zones, which is captured well by the model. The model is used to obtain point-to-point travel times and provides estimates of ADT, though it does not capture congestion so well, since the model does not have a time-of-day component. The panelist suggested that one nice feature of the model is that it runs in only 10 minutes, which allows the agency to be quite responsive to requests.
One panelist remarked that the process for statewide model development at their agency started with project justification, since there was some opposition to it. Many believed that the MPO models should be enough and that a statewide model was unnecessary. In this case, a lot of work went into the networks and freight models. Instead of worrying about freight mode choice, they focused on enhancing truck trip tables. In addition, there was a desire to make the statewide model complimentary to and consistent with the MPO models. For instance, there was a need to model intercity mode choice for intercity buses and commuter rail, but for more local modal questions, they would defer to the MPO models. The focus was on areas where the MPO models were weak, and less focus was given to areas where the MPO models were strong. In addition, the panelist believed the modular approach to model enhancements has worked well for the state.
A second panelist commented that temporal and spatial resolution is a very important consideration, and this dictates the model structure. In the panelist's case, there were only a few needs for the model, however, which may be different from VDOT. The panelist emphasized that the needs should drive the outputs of the model, and the outputs will drive the model structure. So focusing on the needs prior to setting the model structure is important.
Another panel member's agency has a statewide model users group. In the agency's last round of model updates, they obtained feedback from this user group to help inform where investments were made. They also obtained feedback and recommendations on what was possible with respect to modeling techniques and data availability. The deciding factors upon which updates were made, however, were the primary uses of the model and how much time was available for the updates. The agency for the most part abandoned attempts for operational and economic analysis, but focused on the best possible integration with MPO models in the state.
In another case, the statewide model started getting overly complex and taking too much time to develop. That panelist noted that a lot of the models were simplified (e.g., with look up tables) in order to get the model up and running. For instance, look up tables replaced models in some instances. In addition, freight commodity flows were replaced with truck flows, using modal hubs to identify reasonable trip ends. Two truck types are modeled, including service-related and freight long-distance. These simplifications have resulted in huge gains in model run times, from over two hours to just 15 minutes, which has been very useful to the agency. The panelist mentioned that a time-of-day component was explored, but ultimately it did not work well and was abandoned.
One panel member remarked that model development at their agency began by having a stakeholder meeting with peer reviewers, which helped to get everyone on the same page (including MPOs and other partners) with respect to needs and expectations for the model. This collaboration was important and helped them avoid "dueling" models and identified common objectives. The initial statewide model was in one complete software package, which suffered from the "black box" syndrome. This made improving features of the model very difficult, as well as understanding the internal assumptions and subtle details within the model. This experience led to following a modular approach for the next generation of the model. Resources are focused on specific areas of model performance for upgrades and enhancements. In hindsight, we believe the best approach to developing new models is to keep the design as simple a possible to meet the needs of the agency. Once it is used in practice, improvements can be prioritized to meet the analytical goals of the agency.
VDOT was particularly interested in getting panel feedback on the importance of time-of-day modeling component. The panel had a brief discussion related to time-of-day modeling, which is described in this section.
The primary reason VDOT would like to have a time-of-day model component is that they are particularly interested in the congestion on the I-95 and I-81 corridors, and in managed lane policies. One panelist remarked that, at least for the I-95 corridor, the MPO model may be better equipped to answer questions there, since it is mostly contained within a single MPO.
In general, the panel agreed that incorporating a time-of-day component was a good idea, but they differed some in what they believed was necessary. One panel member questioned whether VDOT planned on asking the model how peak spreading actually occurs, or if they planned on extrapolating existing time-of-day data. If the latter, then a model may not be necessary, or a very simple model may suffice. Other panel members suggested starting with something fairly simple during initial model development, then being able to add more sophistication later on as VDOT better understand how the simpler model is working. In fact, something simple may be all that is needed. One panelist made the comment using multihour peak periods can cause issues for model assignment because multi-hour capacity measures do not work well for traffic assignment. The panelist suggested using a peak hour, rather than multi-hour peak periods. In the off-peak periods, it is less critical that capacity measures are multi-hour, since there is less congestion in off-peak periods.
Several panel members had additional thoughts on the networks VDOT is using. One panel member commented that having a master network will make things easier to maintain. Another panelist noted that starting with the HERE network (as VDOT plans) will make things more difficult in terms of compatibility with the MPO networks. Network maintenance can be a big issue with different networks. One suggestion was to perhaps let the vendor of HERE data do the maintenance, and buy new data every few years. This suggestion was seconded by another panel member. Another panelist warned that VDOT would need to be able to add all of the necessary attributes to the network seamlessly to make this work, and developing linkages between the master network (e.g., HERE) and other data sources (e.g., HPMS) is critical.
