Skip to content U.S. Department of Transportation/Federal Highway AdministrationU.S. Department of Transportation/Federal Highway Administration
Office of Planning, Environment, & Realty (HEP)

San Francisco County Transportation Authority (SFCTA) Travel Model Peer Review Report

5.0 Peer Review Panel Responses to Technical Questions

Based on the topics of interest described in the previous section, SFCTA staff formulated a set of specific questions which were sent to the panel members prior to the peer review meeting. These questions formed the basis of the peers' discussion and are listed below with the panel's responses. The major headings in this section match those in Section 5 above for easy cross-reference.

5.1 General Strategies

5.1.1 Are the strategies that we've already employed appropriate?

The panel thought that the overall approach to this project has been great. The data driven approach to modeling traffic flow and signal timing makes a lot of sense. The open source code-base makes the whole process very transparent for other agencies and the modeling community in general. This is a good example of a true collaborative team effort.

5.1.2 Are there any tricks or strategies that we should be trying that we haven't yet?

The peer review panel offered a number of suggestions related to the simulation time period. It was suggested that the overall simulation period could be extended and potentially incorporate demand from the midday time period immediately preceding the PM peak. It was also suggested that the simulation validation period could be extended from 2 hours (4:30-6:30 PM) to 3 hours (3:30-6:30 PM) in order to provide consistency with and facilitate comparisons to the current 3-hour period used in SF-CHAMP demand assignment components. Finally, depending on the performance measures of greatest interest to the SFCTA, it was suggested that the actual demand associated with the cool down period be incorporated in the model. It may not be very important to impose the condition that all of the demand should clear the network at the end of simulation.

The panel felt that the current process of loading demand could be improved by creating a temporal profile for external gateways and also internal zones. The current flat demand profile or uniform loading of demand over the simulation period is not realistic. The panel suggested that traffic counts data on the bridges could potentially be used as a source for creating a temporal profile for external demand. For temporal profile of internal zones, it was suggested that either link traffic count on internal links or distribution of departure times from the household survey could be used. Alternatively, to improve the loading of external demand, the panel suggested that geographic information associated with external zones could be preserved. This could result in a more realistic representation of the temporal distribution of demand from different parts of the region. For example, if the model is run for simulating demand between 4:30 PM and 6:30 PM, it might contain trips that have started much earlier from an external TAZ to reach the model gateway at 4:30 PM.

Regarding the modeling of bus-only lanes in CBD, the panel felt that the current approach of splitting links into a bus-only part and a right-turn only bay was promising. A similar approach had been used in the DTA model developed for Manhattan. The panel noted that it might be better to code the bus stops on the bus-only parts of the split links. If not, during simulation, vehicles might be stuck behind a bus which has made a stop on the right-turn only part of the link. Another suggestion provided by the panel was that reaction time could be adjusted to be lower so that there is more throughput in the general purpose lanes. The panel felt that in the longer term, taxis should be accounted for separately in the model. Even though taxis are a separate mode in the model, the demand has not been validated. The panel suggested that a placeholder could be created to incorporate taxis in the nearer term.

The panel suggested that including information on the availability and restrictions on commercial vehicle or truck traffic in the model may be important. In the peak period, trucks do not appear on main streets but for the off-peak period inclusion of these restrictions may be important.

5.1.3 How should we prioritize our effort for calibration moving forward?

As the model team had previously done, the panel suggested that an overall bias factor (e.g., average simulated link volume / observed link volume) be looked at first. This might indicate possible systematic errors in a simulated scenario. The next step could be to try and identify possible measurement bias in the sensor data, and use the mean absolute error (MAE) to avoid the influence of bad data. It was indicated that RMSE is too sensitive to measurement errors and bad data. The panel noted that MAE has been widely recognized as a more robust statistic.

The panel also suggested that more sensitivity tests could be designed around future and alternative policy scenarios. It might help to perform targeted policy tests. The panel encouraged the SFCTA modeling team to continue analyzing the test results qualitatively at first. An example that the panel gave was to delete a link from the network and check if the changes in traffic volumes around that link are reasonable and check if no drastic changes in volume have occurred in other parts of the network. The panel noted that it might not be very straightforward to quantitatively validate such sensitivity tests without before and after data. It was also noted that constructing the traffic flow and signal timing settings for the future-year conditions might be very challenging.

In addition, the panel recommended conducting sensitivity tests around traffic flow parameters such as jam density and also flow averaging parameters. Since sensitivity is contextual, the panel suggested analyzing if the ranking of investments might change due to certain changes in these parameters. Based on which parameters affect the ranking of investments to what extent, the modeling team may be able to focus more on those parameters during calibration and validation.

Specifically, try adjustments to reaction time and jam density (150,180 etc.) for surface streets.

5.1.4 Is there a way to deal with movement-specific yellow time in signal phases where there is another movement that has continuing green time?

Due to the discussion of other issues that took priority and time, the panel could not specifically respond to this question.

5.2 Model Formulation

5.2.1 What are the best strategies for estimating or calibrating a generalized cost function? In the long term, we will have observed route choice data available to us soon from 2012 California Household Travel Survey.

It was suggested that inclusion of distance term in the generalized cost formulation for routing behavior for trucks may be a good approach to make the path finding in the DTA model more realistic. It may not have a big impact for autos though.

5.2.2 Do the DTA settings that we have chosen make sense?

The DTA settings currently used appeared to make sense to the panel. It was suggested that further changes to these settings may be guided by the calibration and sensitivity tests conducted.

5.2.3 How sensitive should the model be to these settings?

The panel thought that there are no standards as to how sensitive a model is to the DTA settings since the sensitivity might be dependent on the DTA package being used and modeling assumptions made in them.

5.2.4 What others should we test out and why?

The panel suggested trying incremental loading strategies which can provide a better starting point for path building. For example, 20% of the total demand may first be loaded to obtain a set of reasonable paths which could then be used as a starting point for the full run.

5.3 Traffic Flow Parameters

5.3.1 Does our data collection methodology and associated results make sense?

The panel felt that the current data driven approach for estimating traffic flow parameters is quite logical and the modeling team should continue to refine the approach in this direction. It also felt that SFCTA should try to re-evaluate traffic flow parameters for local streets, to ensure that the data sample includes low volume local streets locations.

The jam density resulting from the effective length (EL) used was more than typical, which may be because narrow streets promote more spacing between vehicles. EL affects the jam density and the panel suggested that SFCTA consider using different jam densities instead of a fixed value. Jam density may vary by facility type. The panel also observed that reaction times in the range of 1.3 to 1.5 were reasonable. Finally, the panel encouraged SFCTA to make sure that the free flow speed assumptions are consistent in both static and DTA models. It was indicated that free flow speed should reflect the average travel time over a link.

SFCTA used curve-fitting to the observed traffic flow data to obtain the triangular fundamental diagram depicting the relationship between density and flow. The panel encouraged SFCTA to view the fundamental diagram as an envelope around the flow data rather than something fitting lines through them.

During the presentation, SFCTA noted that traffic volumes on local streets were being overestimated and the modeling team doubled the free flow time on those streets during calibration. The panel suggested confirming the effect of stop signs in local streets to rule out coding errors. Once that has been confirmed, it was recommended that a perception penalty be introduced in the form of a reaction time factor since it is possible that reaction times are longer on such streets.

SFCTA expressed an interest in accounting for friction in traffic flow due to the presence of pedestrians in some areas. The panel suggested that adjusting the average reaction time might help accounting for pedestrian friction. Since this only occurs in specific areas, the panel recommended developing link/node specific adjustments which are informed by pedestrian demand (the demand could be aggregated within a buffer distance of the link/node).

Finally, the panel indicated that SFCTA could also look at meso-scopic models such as DYNASMART, Dynus-T or DTALite which requires less calibration, as they are based on spatial queuing models for arterial streets, and density-speed based flow models for freeway links.

5.3.2 Are there other observed data that we should be trying to collect?

The panel felt that obtaining free flow speeds from spot speeds may not be ideal since actual free flow speeds tend to be lower. The free flow speeds should reflect travel time over a segment rather than at a location. Nevertheless, they should not include signal delay, in contrast with travel demand models, since signals are explicitly coded in meso and micro models. In the long term, it was suggested that using Bluetooth devices to obtain experienced travel times and corresponding speeds be considered.

In the current model, effective length (EL) is calculated from data collected on arterials and local streets. For calculating EL on freeways, the panel recommended using the aerial photo technique.

5.4 Validation Targets

5.4.1 Are there any validation standards for a large-scale DTA? When is 'enough'?

In the collective experience of the panel members, it was felt that there may not be any validation standards that are broadly accepted and also there may be no national benchmark for root mean squared error (RMSE) of flows and speeds. The panel thought the reason for this may be the limited number of studies currently existing in the DTA arena and different limitations being associated with various DTA packages used in these studies.

The panel members recommended that regular Caltrans static validation standards could be used as a starting point and then extended for more refined time periods. It was noted here again that a 3-hour validation period would facilitate a more direct comparison with the static model. The panel members also stressed the consistency of reporting structure that needs to be maintained for such a comparison.

The panel felt that validation standards for large-scale DTA should not be as stringent as microsimulation models (such as VISSIM, Paramics, etc.). It also indicated that speeds may then be more difficult to match. It was thought that calibration and validation to the turn movement level would probably be overkill for application purposes except in targeted corridors.

5.4.2 How should we measure the "stability" of the results? What should we be looking at other than relative travel time gaps?

The panel noted that specific value of relative travel time gap in may not be as important as the stability of the relative travel time gap.

Maximums and minimums of traffic characteristics such as speeds may be checked to see if those have stabilized over iterations.

The panel suggested that it might also be helpful to look at variation in vehicle miles traveled (VMT) and vehicle hours traveled (VHT) as additional measures of stability.

5.4.3 Similarly, how should we test the model's sensitivity to changes in: network geometry; signal operations; other?

The panel offered various methods to check the model's sensitivity to network changes. At first, it was suggested that progression of traffic on major arterials be confirmed. Another basic test would be to visually inspect the relevant paths for reasonableness. Finally, the panel recommended examining areas in the network that are specifically affected by bottlenecks and queues. These are the areas where static model would be significantly inaccurate in predicting the traffic flow patterns.

The panel mentioned that rounding of fractional trips may also be a source of issues in traffic prediction. Even if there are no trips lost in total, there may be significant loss of trips in specific zones. The panel recommended that bucket rounding be used over arithmetic rounding.

5.4.4 How do you validate to conflicting data?

The general response of the panel to this was to obtain more data so that more cross-checking can be done. The panel felt more observed traffic data on local streets would be useful for validation given that there appears to be considerable overestimation of traffic. Expanding the number of observed traffic count locations was also offered as a long term consideration. The panel felt that the current number of 200 locations may not be sufficient for a city the size of San Francisco. The panel noted that there are 400 count locations (or 800 directional counts) in the SACOG area per one million population and that counts should be distributed geographically and across functional classes of roadways, and not correlated. Because SFCTA models all streets, and because of the suspected over-assignment to local streets, the panel suggested that more counts be taken to ensure that a sufficient cross section of local street locations are included in validation.

5.5 Integration Strategies

5.5.1 How should we prioritize the strategies listed in the integration memo both for "Demand Information for DTA" and "DTA Information for Demand?

Demand model information for DTA

The panel suggested that refining the market segments of the demand being input into the DTA model may be the next logical step of improvement in this area. It was recommended that segmenting by value of time (VOT) may be tried first considering that SFCTA would like to use the model for evaluating pricing policies. The panel noted that this segmentation may not be critical for initial testing but when more sophisticated scenarios are required to be analyzed this might become necessary. Another suggestion that was offered for the longer term was the separation of parking and activity locations which would also be important in analyzing pricing scenarios.

DTA information for demand model

The panel recommended that SFCTA explore ways to hybridize skims from the DTA model (at San Francisco city level) and static model (at the regional level). The panel felt that in this way temporal expansion of the DTA model could be achieved more gradually and systematically. However, the panel noted that a full day DTA model may need to be run for his approach and there may be model run time implications to consider in advance.

The panel suggested that the next step after the development of hybrid approach to obtain skims for the whole region would be adding temporal detail. This would also require changes in temporal resolution to be made to models in SF-CHAMP (tour time model, trip departure time model etc.).

The panel recommended that SFCTA may consider developing a full-scale regional level DTA only after some of the above improvements mentioned improvements to the model have been made. It was suggested incorporating reliability may also be something to be considered in the long term only.

5.5.2 What thoughts or cautions would you give to simultaneously pursuing person-based dynamic transit assignment a la Fast-TrIPs?

The panel suggested that it may be more appropriate to finish developing a full day DTA model before pursuing person-based dynamic transit assignment or integrating with Fast-TrIPs.

5.6 Technology Transfer

5.6.1 What (if anything) about this process/project would be useful for other agencies to learn and hear about? In what format?

The panel felt that the overall approach of minimizing manual intervention by way of coding and automating the various tasks involved is something that other agencies would benefit from greatly. Automation may not be important in smaller sized networks but becomes key when large and dense networks are involved.

Apart from the approach, the panel thought that the process and scripts for converting a static assignment network to a DTA network could be adapted by other agencies and help them gain efficiency in their model development process.

Finally, the panel suggested that the methodology of estimating traffic flow parameters and in some cases the parameters themselves may be transferable to other regions around the country.

5.6.2 What parts of the code base that we have developed (if any) would be useful to operationalize for others?

As mentioned previously, the panel recommended that the code-base for automatic conversion from a static network to a DTA network be made as general as possible. The panel commended SFCTA for developing the code-base in an open-source environment.

5.7 Research

5.7.1 Are there any research or application questions that it seems like we should be able to answer with this project?

The panel noted that the processes and strategies for integrating activity-based demand and DTA supply models were probably of most research value in this project.

Updated: 1/23/2014
HEP Home Planning Environment Real Estate
Federal Highway Administration | 1200 New Jersey Avenue, SE | Washington, DC 20590 | 202-366-4000