Also available as Adobe PDF (5.4 MB)
Notice
This document is disseminated under the sponsorship of the U.S. Department of Transportation in the interest of information exchange. The U.S. Government assumes no liability for the use of the information contained in this document.
The U.S. Government does not endorse products or manufacturers. Trademarks or manufacturers' names appear in this report only because they are considered essential to the objective of the document.
Quality Assurance Statement
The Federal Highway Administration (FHWA) provides high-quality information to serve Government, industry, and the public in a manner that promotes public understanding. Standards and policies are used to ensure and maximize the quality, objectivity, utility, and integrity of its information. The FHWA periodically reviews quality issues and adjusts its programs and processes to ensure continuous quality improvement.
1. Report No. FHWA-HEP-18-026 |
2. Government Accession No. |
3. Recipient’s Catalog No. |
||||
|---|---|---|---|---|---|---|
4. Title and Subtitle Atlanta Planning Commission (APC) Peer Review |
5. Report Date November 28th, 2016 |
|||||
6. Performing Organization Code |
||||||
7. Authors Jason Lemp, Ph.D. |
8. Performing Organization Report No. |
|||||
9. Performing Organization Name and Address Cambridge Systematics, Inc. 100 Cambridge Park Drive, Suite 400 Cambridge, MA 02140 |
10. Work Unit No. (TRAIS) Contract or Grant No. DTFH61-10-D-00005 |
|||||
12. Sponsoring Agency Name and Address United States Department of Transportation Federal Highway Administration 1200 New Jersey Ave. SE Washington, DC 20590 |
13. Type of Report and Period Covered Final Report September 2017 to October 2017 |
|||||
14. Sponsoring Agency Code HEPP-30 |
||||||
15. Supplementary Notes The project was managed by Task Manager for Federal Highway Administration, Sarah Sun, who provided technical directions. |
||||||
16. Abstract This report details the proceedings of a peer review of the Atlanta Regional Commission's (ARC) transportation model. The peer review was intended to help guide the modeling activities of ARC and to aid ARC in prioritizing model improvements. |
||||||
17. Key Words Peer review, MPO, APC, travel modeling, trip-based model |
18. Distribution Statement No restrictions. |
|||||
19. Security Classif. (of this report) Unclassified |
20. Security Classif. (of this page) Unclassified |
21. No. of Pages 56 |
22. Price N/A |
|||
The views expressed in this document do not represent the opinions of FHWA and do not constitute an endorsement, recommendation or specification by FHWA. The document is based solely on the discussions that took place during the peer review sessions and supporting technical documentation provided by Atlanta Regional Commission (ARC).
The FHWA would like to acknowledge the peer review members for volunteering their time to participate in this peer review. Panel members included:
Additional biographical information of each peer review panel member is located in Appendix C.
This peer review was supported by the Travel Model Improvement Program (TMIP), sponsored by FHWA. TMIP sponsors peer reviews in order that planning agencies can receive guidance from and ask questions of officials from other planning agencies across the nation. The peer review process is specifically aimed at providing feedback to agencies on travel modeling endeavors.
The primary objective of the ARC peer review was for ARC to receive guidance on their travel modeling activities. Specifically, ARC wanted to confirm their activity-based model (ABM) was consistent with state-of-practice in regional travel modeling and was capable of supporting the analysis related to relevant policy questions in transportation planning. Further, ARC received feedback on area where the model could be improved.
The peer review panel convened for one full day and one three-quarter day (September 28, 2017 to September 29, 2017). During that time, ARC presented background information and asked for guidance in specific areas of their modeling practices, and the panel discussed these items and offered a series of formal recommendations to ARC.
The remainder of this report is organized into the following sections:
Four appendices also are included:
ARC functions as the federally designated metropolitan planning organization (MPO) for the Atlanta region. The primary responsibilities of the MPO's transportation modeling group include the following:
ARC is the planning organization of the Greater Atlanta region in Georgia. The region's urban core is located in Atlanta and represents the region's largest activity center. The region has 10 counties, though different regional boundaries are used depending on the planning purpose. For some planning purposes, a 20+ county region is considered. Figure 1 shows the geography of the region and the various boundary definitions that are used for different analyses.
The region has a population of roughly 5.7 million people and employment of roughly 2.2 million jobs. By 2040, the regional population is expected to grow to 8.1 million (a 41 percent increase) and regional employment is expected to grow to 4.0 million (an 80 percent increase). Like many regions across the country, the demographics of the population are also expected to change as people live longer. Figure 2 shows how the age distribution of the region is expected to change from 2015 to 2040. The 65 and older cohort is expected to nearly double in size, growing from 11 percent of the population in 2015 to 19 percent of the population in 2040.
The region's largest employment sectors include retail, health care, accommodation and food services, administrative and support, and professional and scientific. While retail is currently the largest employment sector, it is expected to grow at very slow rates to 2040. The sectors expected to grow the most include construction, education, health care, and professional and scientific.
The region is a maintenance area for the Environmental Protection Agency's (EPA) air quality attainment standards for ozone and particulate matter, though the region's air quality measures have improved considerably in the last 10-15 years. Part of the reason for the multiple regional boundaries is to include counties that historically have not met EPA's air quality standards.
This section describes the overall stages in the 15-year development process of ARC's activity-based model (ABM), and then outlines the overall model structure and components of the ABM.
ARC first began the design of their ABM in 2000. In 2003, they completed and debuted their population synthesizer for generating a synthetic regional population, which is a critical data input to the ABM. In 2004, EPA designated 22 counties of the region in non-attainment. This ultimately delayed work on the ABM development, which was completed in 2009, at which point ARC began the process of doing extensive comparisons of the ABM to their existing 4-step model. By 2013, the data used to estimate and calibrate the ABM model components was becoming dated, and ARC initiated the process to revisit estimation of certain model components with more recent survey data collected in 2011 and to recalibration and revalidate the model. The ABM was fully deployed in 2016. At that time, ARC invested in extensive training for partner agencies that use the model and retired the 4-step model completely. More recently, the ABM was integrated with PECAS land use model and the integrated model is used as the production model for the region.
ARC's ABM serves as the official travel model for ARC's regional transportation plan and is used for a variety of analyses, including highway pricing, peak spreading, demographic shifts, work schedules, non-motorized travel, among others. Like all modern ABMs, it is tour-based and it is disaggregate. The ABM is a member of the CT-RAMP family of ABMs. Figure 2 illustrates how the CT-RAMP model interacts with various other model elements of the travel modeling process. The primary inputs to CT-RAMP are the socioeconomic inputs and the congested travel skims by mode. In addition, the model system includes several non-ABM model components, including an external trip model, a truck model, and an air passenger model. Once these model components are run (including CT-RAMP), the trip generated by the model components are aggregated into trip tables by time period and fed into network assignment models. Not shown in Figure 2 is the link between the ABM, the population synthesizer (PopSyn), and the land use model (PECAS). PECAS land use model is responsible for generating land use forecasts, from which the PopSyn generates a synthetic population that is required as input to the ABM.
Compared with a traditional 4-step model, CT-RAMP basically replaces the first three steps of the traditional model: trip generation, trip distribution, and mode choice. Figure 3 illustrates the model components of a traditional 4-step model and the models the types of models that replace them in CT-RAMP. Time of day models are also a key component of CT-RAMP, which are sometimes included as part of 4-step models.
Unlike a traditional 4-step model, which functions only at the trip-level, CT-RAMP functions at four levels of travel behavior including long-term choices, day pattern choices, tour-level choices, and trip-level choices. Within each level, multiple choice dimensions are modeled, as shown in Figure 4. In addition, travel choices are simulated for each household and individual in the population, rather than groups of households defined across one or two household variables.
In Figure 5 the red highlighted boxes indicated where enhancements were made to model components in 2013. These enhancements were made to population synthesis procedures to use both person level and household level controls, usual school and workplace location models, tour level destination, time-of-day, and mode choices, and trip mode choice.
The ABM is integrated with PECAS land use model. Linkages between the models are formed in two ways: PECAS land use model generates the population and employment forecasts that are used by PopSyn and CT-RAMP and mode choice and accessibility logsums from CT-RAMP are fed into PECAS to inform the land use model of the relative congestion levels. These linkages drive the integration of the land use and transportation forecasts, ensuring they inform one another.
ARC's ABM has undergone several rounds of calibration and validation as new data has become available. Model validation typically involves comparing model results to observed data and verifying they match to a reasonable level of accuracy. Model calibration is the process making adjustments to model parameters in order to improve how well the model validates. ARC's model calibration process consisted of step-wise calibration of individual model components of CT-RAMP as well as complimentary models (e.g., air passenger and truck models). Overall the calibration process was guided by a 4-pronged strategy:
In practice, most adjustments that were made were made to alternative-specific constants. In cases of a particularly poor match between model results and observed data, ARC revisited model specifications and/or inputs rather than making large adjustments to model parameters and constants.
The primary objective of the ARC peer review was for ARC to gain important peer feedback on the strengths and weaknesses of their ABM. In particular, they were interested in obtaining panel member thoughts on the model's ability to support transportation planning policy decisions and on areas of the modeling process that could be improved with future model development efforts.
In addition to these general objectives, ARC distributed a list of 10 questions to the panelists for consideration during the peer review process. This list of questions is shown below in no particular order of relevance or importance:
The first day of the peer review panel meeting included time for ARC presentations to the panel members as well as discussion of key topics. The second day began with some additional presentation time as well as discussion time prior to the time set aside for peer review panel deliberation. This section documents the key points that were discussed during the meeting.
At various points during the meeting, different elements of ARC's land use forecasting model, land use forecast results, and implications of the land use forecasts were discussed.
Forecasting Models
The land use forecasting procedures used by ARC were discussed. ARC uses a combination of the REMI (which works at the county level) and PECAS (which works at the level of about 80 zones) for land use modeling. One panel member asked about the consistency between information in the REMI model and the PECAS land use model. ARC noted that the consistency is good, and PECAS uses outputs from the REMI model in its forecasting procedure. Overall, the panel found ARC's land use modeling activities to be impressive.
Workers and Employment
One topic related to land use that was discussed extensively during the meeting was the relationship between employment location and workers' residential locations. One panelist noted that it is often difficult to obtain reliable data on occupation of workers by residential location. It was noted that the Public Use Microdata Sample (PUMS) has this information, though at an aggregate level of spatial detail.
Several panel members brought up the issue of matching workers to jobs. For instance, how does the model reconcile that some workers have two jobs? In one region, a panelist commented that the most recent employment estimates were impacted by the recession, which lowered the estimate of jobs in the region (relative to earlier estimates). Ultimately this led to the transportation plan for the region having the assumption of an approximately one-to-one relationship between jobs and workers. Another panelist noted that ARC was forecasting much higher employment growth to 2040 than population growth and wondered how that was reconciled. ARC noted that PopSyn currently has some built in constraints to ensure that there is a one-to-one relationship between the overall employment forecast by the land use model and number of workers in the region, which ultimately reconciles any disparity between employment and population growth. ARC, however, is considering changing this relationship and using something like 0.95 workers per job in the future. A couple of panelists were skeptical that even 0.95 was adequate. For instance, in one region, a panel member noted that ratio of workers to jobs was about 0.75. Another related issue is that household travel surveys often have a certain portion of workers that do not report having a regular workplace at all (as much as 20 percent in some regions). One panel member opined that there may be a lot of informal jobs (e.g., Uber/Lyft drivers, online work, etc.) that are not captured well in an employment forecast. Further, the prevalence of these jobs is growing and this will be a major challenge for the profession in the future.
University Students
Another area of discussion was around dealing with forecasts of university students. One panelist noted that a common issue with population synthesis is university student residential location. In one region, it was noted that special procedures were needed to cluster university students appropriately around the major universities, and that without this measure, the population synthesis did a poor job of matching the residential location of university students. The procedures involved identifying university student households in PUMS data and setting reasonable controls in the population synthesis. ARC agreed that a more sophisticated placement for university student households would be ideal. ARC also noted that they attempted to conduct a survey of university students a couple years earlier, but the pilot was not very successful due to poor response rates, and ultimately the survey was not completed. In addition to university student households, another panelist suggested a more robust forecasting procedure for group quarters housing as well.
Aging Population
It was pointed out that one area of growing concern in many MPOs around the country is the aging population, and ARC is no exception. ARC expects that the 65 and over cohort to grow from making up about 10 percent of the population to nearly 20 percent by 2040. This is reflected in the residential and population forecasts. One panelist was curious whether PopSyn was able to reflect simultaneously the aging population as well as the changes that would consequently result to household composition. ARC noted that PopSyn reconciles this based on the priority given to the various controls. Ultimately the priority of the various controls is a user input, so the user has a say in how this is reconciled. Another question was posed on how the model accounts for people working to older ages than they did in years past. ARC commented that this is not explicitly accounted for in the model; however, the number of workers is a control to PopSyn, and so to some extent, this is accounted for. It was noted that the growth in older population has implications for transit service, since fixed route, fixed service transit is often not a good option for those with mobility issues.
Long-Term Choice Models
A couple of panel members brought up the idea of that some workers have multiple jobs and/or workplaces and wondered how the long-term choice models in CT-RAMP dealt with this. The response was that the model currently assigns a single workplace to each worker and this is serves as the activity location for all work tours for a worker. The model uses a shadow pricing technique to ensure workers are aligned on a one to one basis with jobs. One panel member noted seeing more people having multiple work locations, including home working, and in one particular region, they are struggling with how to deal with this in the future. The panelist and ARC agreed that it would be interesting to find out what the share is of truly traditional workers-those that have one job with one workplace. It was noted that the model does not explicitly model people working at home, which is largely because the explanatory variables currently available in the model would be inadequate to explain this choice phenomenon, so it would simply rely on setting of a constant. In another region, a panel member noted that the model includes work-from-home as a binary choice model in the set of long-term choice models, and suggests this is an important policy variable since policy makers in the region are very interested in how changing workplace dynamics could impact travel patterns in the future. Admittedly, the ability to forecast work-from-home with exogenous variables is not there, but a couple panel members agreed that having an explicit binary choice model is a useful tool to perform policy analysis and scenario testing.
Mode Choice Model
The mode choice alternative and nesting structure was another topic of conversation. The CT-RAMP mode choice model considers access mode for transit in a higher level nest (of a nested logit model framework) than transit mode. One panelist applauded ARC for this structure, indicating that while the CT-RAMP structure was one that made sense to the panelist, some models use a reverse nesting structure with transit mode in the higher level nest, which seems less reasonable. ARC noted that this nesting structure has worked very well for modeling drive access trips in Atlanta, and adding the kiss-and-ride (KNR) access mode in addition to park-and-ride (PNR) improved the model's validation statistics considerably. ARC also noted validation statistic improvement by redefining the transit modes to two: premium transit and other transit.
One notable feature of the mode choice models was the presence of area type variables in the model's utility functions. One panel member suggested that continuous density variables may be favorable to area type because they avoid cliff effects, while another panel member noted that network and land use density have been found to be very useful for variables in mode choice, particularly for modeling non-motorized modes. ARC noted that CT-RAMP does not account for parking scarcity at the region's PNR lots. One panel member noted that shadow pricing can be used to account for parking scarcity, but few regional models are currently doing this. Another panelist indicated that in one region they do include some effects to discourage the use of certain stations in the model to account for parking scarcity in some way, but it is not prohibitive. ARC and MARTA staff asked the panel how transit-oriented development (TOD) is typically handled in regional models. Panel members indicated that proximity of service and density of service are good indicators of TOD. In addition, mixed use development and employment and intersection density are important indicators of transit, and area type indicators may not tell the model much.
One element of the model that was presented to the panel was the mode choice model estimation. One panel member noted that the coefficient associated with in-vehicle time for transit modes in the mode choice model was lower in absolute value than the coefficient associated with in-vehicle time for non-transit modes, and asked whether this was a cause for concern, since it has implications for valuation of transit scenarios (while noting that model estimation results in other regions found a similar relationship between auto and transit in-vehicle time coefficients). While this difference existed for model estimation, ARC noted that during model calibration, the coefficients associated with in-vehicle time for all modes were adjusted to be identical to one another, to avoid any concerns of key planning partners. Another panelist noted that in-vehicle time coefficients are often varied as part of scenario tests of autonomous and connected vehicles, but similar concerns do not seem to be voiced for such scenarios. Multiple panelists agreed that the disparities likely to exist across modes in terms of travel time sensitivity and it is important to be thinking about this in model development, but ultimately, no agencies seem to be including this difference in their travel models.
Time of Day Choice
The topic of modeling time of day in CT-RAMP was presented and discussed among the panel. The CT-RAMP model is really quite static in the overall scheme of the model, and congestion has only a small impact on time of day choices. One panel member noted similar issues in other models and commented on another region's ABM that includes a shift effect to uncongested time periods when congestion is present in the peak. Another panel member noted that this is a confounding effect, that congestion exists in the peak is a result of the peak being a desirable time of day to travel (primarily because of convention for typical work hours). In terms of validation of the CT-RAMP time of day models, no specific statistics were used to match time of day patterns with observed data, only visual inspection of time of day profiles. One panelist indicated that there may be a possibility to generate more aggregate time periods for the purposes of model calibration to ensure the trips are allocated to assignment periods in appropriate amounts.
Other Topics
One panel member asked about where in the model stream was the choice of using toll vs. free lanes and whether this was carefully considered prior to making the decision. ARC commented that this is included in the mode choice model (as opposed to assignment). This was mostly a holdover from the earliest version of the ABM though careful consideration was given to this decision. The primary reason for handling toll vs. free choice in mode choice was that handling it in mode choice gives more control to the analyst, including the VOT that is used for the choice. In the future, ARC is considering introducing distributed values of time. One panel member noted that in that region, they are reconsidering the placement of the toll vs. free choice, moving it from mode choice to assignment. In another region, a panelist noted that they are currently considering whether to place in mode choice or assignment since the region wants to begin examining managed lane alternatives with their model. A couple panel members noted that inconsistencies can arise when toll vs. free choice appears in the mode choice model. This is because in assignment, those choosing the tolled option can only be considered toll eligible, rather than forced to use the toll road. This could potentially add noise to the model stream. Another panelist mentioned that it seems the industry is trending toward simpler mode choice structures over the recent past, including toll vs. free choice, and therefore, it may be worth a second look for ARC in the future. A couple panelists noted some regions using value of time segmentation in the traffic assignment modules, and that this may be a way to assure getting higher value of time travelers using the priced roadways.
Another question that came up in the discussion was how to deal with and analyze emerging employment centers in employment centers. One panel member pointed out that using density measures (as opposed to area type constants) in model specifications can help in these analyses, since these variables would capture the impact of small or medium changes in employment. Scenario testing can also be an important tool. Another panelist offered an example from another region, where a dense downtown coexists with two other major employment centers located more suburban areas, both with poor transit service and both less dense than the downtown. In this case, the model does a good job of forecasting that those two other job centers have low transit use. Instances like this can serve as good sanity checks for the model.
Highway Networks
Highway networks used in the ARC model were discussed briefly. ARC has historically used lookup tables for free flow speeds, though most recently, NPMRDS speed data was used to adjust free flow speeds in sections where data was available. Data for the period from 3 am to 6 am was used in this process. Ultimately, an average of the lookup table value and observed data was used in the model. One panel member asked about procedures ARC uses in cases when observed free flow speeds exceeded speed limits. ARC noted this was an issue they were still working out due to political challenges involved with having modeled speeds higher than speed limits. Another panelist agreed with ARC's approach of averaging the observed data with lookup tables. The panelist noted that due to political challenges in another region, that region ended up abandoning use of speed data for free flow times in that region's model. One panel member discussed issues of dealing with facility type changes over time, specifically rural arterials to urban expressways. In these projects, the model may see a speed reduction, whereas that result is really not expected or desired from a practical perspective, and this can lead to credibility issues with the model.
Highway Assignment Convergence Criteria
One important discussion point that came up in regards to highway assignment procedures in the ARC travel model was convergence criteria. ARC mentioned that they are considering tightening their convergence criteria from 0.0005 to something smaller. They are doing so because it was found that small differences existed in trip tables at the current convergence level. In other regions, a couple panelists noted using similar convergence levels (0.0005 in one region and 0.0002 in another), but tighter convergence is a concern in those regions also. Another panel member suggested that a tighter convergence might suggest a state of equilibrium where a state of disequilibrium actually exists (e.g., variability of travel times from day to day). While this is true, it was noted that for project evaluation purposes, achieving tight levels of convergence in highway assignment is important, otherwise the model forecasts may suggest erroneous changes in evaluation measures based in model noise rather than project differences. From this perspective, investing in ways to measure reliability may be a better option to reflect natural disequilibrium of the system.
Fuel Economy and Auto Operating Costs
The topic of auto operating costs and fuel economy in forecasts was also discussed. A couple panelists noted issues in dealing with gas prices, since a single auto operating cost is often used. One challenge can be balancing fuel efficiency forecasts with their effects on travelers' decisions to drive. If fuel efficiency is predicted to rise dramatically, then the impact on the model will be to reduce auto operating costs, and thus, increase automobile usage. One panel member suggested that ARC look at some research of the national forecasts of motor vehicle fuel costs and fuel efficiency and compare with what is assumed in ARC's air quality model. There are a lot of areas of disruption in the future, including shared vehicles, autonomous vehicles, and fuel price, and all of these areas will impact the vehicle fleet. As such, ARC should be doing some sensitivity analyses around this. Another panel member suggested that validation tests could be done based upon year to year changes in gas prices to see how the model responds and whether the model's response is reasonable in magnitude compared to year to year changes in traffic counts. One panelist mentioned that such a study was done several years ago that compared vehicle-miles of travel (VMT) with transit shares and gas prices. Ultimately, however, these comparisons are difficult because there is lag in how changes in gas prices affect travelers' choices, not to changes in other variables that may not be controlled for. For instance, a few years back when gas prices were high, hybrid fuel vehicles grew in popularity, partly because there was an expectation that gas prices would stay high. Nonetheless, results of fuel price studies do show that a correlation exists between gas prices and VMT. A couple of panel members opined on how gas prices actually end up impacting traveler decisions. On the one hand, travelers likely do not have a very good sense of how much any particular trip actually costs. On the other hand, travelers certainly perceive the impacts of an increase in gas prices through having a reduced monetary budget for other expenses. Because the effects are not perceived directly (e.g., the exact price of a trip is unknown), the perceived costs of travel are likely less than actual costs of travel.
Value of Time
One panel member noted the discrepancy between the VOT assumed in the mode choice models and the VOT used for highway assignment (of $25 per hour). It was noted that the nesting structure of the various model components plays a role here, and once that nesting is accounted for, there is generally consistency in the VOTs that are used across models. The models are nested through use of logsum variables that appear in various model components, and represent the expected utility a traveler would get from all possible options lower in the choice hierarchy. Highway assignment is at the lowest level of the choice hierarchy.
HOT Lane Toll Levels
In order to generate appropriate toll levels for ARC's high-occupancy toll (HOT) lanes, ARC's model uses a toll optimization process, which uses outputs of highway assignment and updates toll levels for feedback to highway assignment. The algorithm uses a VOT assumption of $25 per hour (consistent with the highway assignment routine), and the algorithm seeks to achieve generalized costs in the general purpose lanes and in the HOT lanes that are equal. Ultimately, convergence of the algorithm is based on user judgment. The feedback loops between toll optimization and highway assignment represent an inner feedback looping mechanism. There is also the more standard outer feedback loop between highway assignment and the demand model (e.g., the ABM). In the first outer feedback loop, the inner feedback loop tends to run on the order of 20 times and in subsequent iterations of the outer feedback loop, it tends to run fewer times. One key question that arose from the panel was how ARC ensures consistency between two alternatives. ARC noted that the optimization procedure is only performed one time. When new scenarios or alternatives are tested in the model, tolls are not adjusted typically. Special procedures would be needed to ensure consistency if scenarios specific to toll levels were to be investigated.
One panelist asked about how the toll optimization procedure reconciles that the high-occupancy vehicles (HOVs) do not pay the toll, while the single-occupancy vehicles (SOVs) do pay the toll. The amount of capacity being sold is really only what is left over after accounting for the capacity used by the HOVs. ARC recognized that it is a bit of a balancing act, particularly in the context of the regional model. Ultimately, what the model does produces reasonable traffic volumes, and that is all ARC thinks is reasonable to ask from the model. To do anything more, a detailed corridor analysis would likely be needed. This prompted another discussion about whether the capacity assumed for a HOT lane should be the same as the capacity assumed for general purpose lanes. The latest version of the Highway Capacity Manual (HCM) shows there is a difference between capacities of these two types of facilities though there is some question of the validity, since HOT lanes are designed to never actually reach capacity. One panel member suggested that models typically use the same capacity for both, but some models incorporate operational settings to ensure conditions on HOT lanes are always better than conditions on general purpose lanes.
ARC's model calibration and validation practices were a key topic that ARC wanted panel input on. ARC presented a variety of information on validation results and statistics for which the panel discussed.
ABM Model Components
ARC noted that work location distances were calibrated to household survey data as well as American Community Survey (ACS) county-to-county worker flows (trip length frequency distribution comparisons to household survey data are shown in Figure 6). A couple panel members suggested that ARC also look at the Census Transportation Planning Package (CTPP) to validate work locations.
For school location, ARC has less validation data. Moreover, data that typically is used by school location models (e.g., school enrollment data) is lacking in the Atlanta region. One panel member suggested that ARC look at data from the state Department of Education for school employment and/or enrollment data. Another panel member noted that the model predicts too few short distance school trips (see Figure 7), and this could be related to the model's treatment of non-motorized modes. For instance, the TAZ sizes may be too large to accurately predict non-motorized trips, and micro-analysis zones (MAZs) could be useful though they do add significantly to model complexity.
Several panel members suggested that ARC take a closer look at income's impact in mode choice. These comments derived from validation statistics related to the cross-tabulation of income and mode that were not as good as the panel would normally like to see (see Figure 8). One panel member made the observation that while the ARC model captures auto ownership's impact on mode choice well and that is more important than income's impact, income is still important. For instance, while a low income household may not be able to afford a car, for a high income household, not owning a car is a choice. These are two very different things. ARC generally agreed that the validation of mode choice by income could be improved.
One panel member observed that the walk share forecast by the ARC model was quite low (only 2 to 3 percent). Recent surveys in other regions have indicated walk shares in the neighborhood of 6 to 7 percent in Sacramento and Baltimore and 10 percent in San Diego. ARC did look at the difference between travel diaries and GPS travel logs and found under reporting of trips, and these trips tend to be of the short variety. However, they did not find many tours that were missed completely, these tended to be missed stops. Ultimately upon further consideration, the panel agreed that the number of walk tours reported for Atlanta (of 2 to 3 percent) might actually be accurate for this region, even though it was lower than several other regions.
Traffic Assignment
One key element for traffic assignment validation was that ARC has both traffic counts and observed speed data to which to validate to. Generally speaking, it is not possible to match both exactly, so it becomes a balancing act between matching speeds or volumes. One panelist noted that observed speeds may be particularly difficult to match in congested areas due to the effects of non-recurring congestion. As a result, the panel member suggested that validation focus on speeds only in areas with small amounts of congestion, and overall, matching traffic volumes is more important than matching travel speeds. Another panel member agreed that traffic volumes were more important.
In terms of traffic volume validation to counts, ARC examined both volume-to-capacity (V/C) ratios and VMT. Like speeds versus volumes, it is generally not possible to match both. One panelist supported the idea of comparing along both dimensions (even if they cannot both be matched) because each is imperfect for certain reasons. Overall the panel members agreed that the validation results compared favorably with expectations, including the Travel Model Validation and Reasonableness Checking Manual. One panel member suggested that it can be useful to consider other, more location specific, validation measures. This can be useful for achieving credibility with policy makers, for instance. Another panel member noted that the root mean squared error (RMSE) statistics reported by ARC (see Figure 9) were very similar to the RMSE statistics that the panel member has seen in other locations.
| Volume Group | Observations | RMSE | %RMSE | Total Volume | Total Counts | Volume / Count Ratio |
|---|---|---|---|---|---|---|
| < 2500 | 926 | 1,391 | 99.0% | 1,744,808 | 1,297,157 | 1.35 |
| 2500 - 4999 | 1,148 | 1,840 | 50.0% | 4,507,075 | 4,249,983 | 1.06 |
| 5000 - 9999 | 1,439 | 2,696 | 38.0% | 9,651,756 | 10,117,288 | 0.95 |
| 10000 - 24999 | 1,238 | 4,442 | 30.0% | 16,971,619 | 18,486,962 | 0.92 |
| 25000 - 49999 | 182 | 6,501 | 18.0% | 6,115,838 | 6,429,569 | 0.95 |
| 50000 - 74999 | 111 | 12,216 | 19.0% | 6,282,256 | 7,082,456 | 0.89 |
| 75000 - 99999 | 108 | 13,811 | 16.0% | 8,132,042 | 9,191,869 | 0.88 |
| >= 100000 | 58 | 15,453 | 13.0% | 6,380,446 | 7,094,626 | 0.90 |
| Total | 5,210 | 4,365 | 36.0% | 59,785,840 | 63,949,910 | 0.93 |
A key deficiency of static traffic assignment models was brought up, in that the volume-delay functions (VDFs) are monotonic, meaning that higher volumes lead to lower speeds. However, a key element of traffic flow is that when demand exceeds capacity, traffic volumes actually break down and low speeds are observed in conjunction with relatively low traffic volumes. The question is then how to obtain estimates of toll revenues under such conditions. The panel members did not have a specific solution, though one noted that static assignment models are simply not going to be able to handle some questions. For instance, the decision to use a toll lane is partly a response to the existence of a queue, so if the bottleneck is the only component of congestion that is modeled (like in static assignment), the model may not adequately reflect the advantage a toll lane has over general purpose lanes. To adequately capture such effects, Dynamic Traffic Assignment (DTA) is really needed.
ARC noted that they were able to achieve significant improvements in model speeds from the previous model by changing the free flow speeds using the NPMRDS speed data and revalidating the model (see Figure 10). One panel member remarked that the modeled and observed speeds matched quite well and better than expected. ARC also presented a comparison of the area accessible in 60 minutes from downtown Atlanta (see Figure 11). The figure compared the results from a Washington Post article for the region to the modeled results. One panel member commented that this sort of comparison is very useful as a presentation tool. One criticism of models is that they do not necessarily show conditions to be as bad as people think it should be, and this type of comparison could be useful in that regard.
Transit Assignment
A number of validation results of transit assignment were presented to the panel. One panel member asked whether ARC looked at various traveler segments in the transit assignment validation process, commenting that in the panelist's experience, looking at very detailed traveler segments in transit assignment validation can often point to where issues exist.
There was some discussion of shuttle buses in the region. ARC noted that they do not have ridership data for the shuttle buses in the region, which mostly serve universities. The transit agency, MARTA, indicated that they have found the ridership on shuttle buses predicted by the model to be too low. MARTA indicated that they have shuttle bus data that can be used for better validation of this transit segment. One panel member suggested that shuttle buses may be better considered as a separate mode due to the price differences to other buses (shuttle buses are free). Another panelist noted similar services in other regions and there is often difficulty in matching boardings data on those routes.
A couple of panel members were quite impressed with the validation results ARC obtained for boardings at rail stations (see Figure 12). Nonetheless, ARC noted that there are a couple of problem areas when it comes to rail station boardings that ARC is currently working on, particularly in the north-south corridor through the urban core of Atlanta.
Several panel members remarked that sensitivity testing and scenario analysis are integral components of model validation. One panel member suggested that temporal validation (e.g., backcasting) efforts can be useful for testing whether model sensitivities are reasonable, since it offers another data point of observed data by which to compare against model results. ARC has not done anything formally in this regard due to lack of data for an earlier year. However, the original model was validated to 2010 while the most recent effort revalidated the model to 2015. In transferring the model from 2010 to 2015, ARC did note that validation results held up fairly well. Another panel member suggested that systematic sensitivity tests (e.g., changing a single variable at a time, like transit fares or gas prices) are useful validation tools. ARC noted that they plan to do such sensitivity tests in the near future.
Earlier in 2017, a bridge along interstate highway 85 collapsed in Atlanta. When this occurred, ARC completed several scenario tests of how this would impact travel patterns using the travel model. The section of highway that collapsed carried an annual average traffic volume of about 230,000 vehicles. The scenario tests that were performed fixed the origin-destination patterns of travelers, while allowing shifts in time of day, number and type of stops on tours, mode choice, and vehicle routing. A couple of panelists thought that day patterns of individuals might actually change in response to the bridge collapse, with people deciding to work from home, for instance. Overall, trip making in the scenario test was reduced as a result of lower stop generation rates. The model predicted an increase in transit ridership of 4 percent, regionwide, though MARTA (the transit agency) noted that ridership actually increased 12 to 15 percent on certain transit lines. MARTA noted that upon reopening of the highway segment, the increase in transit ridership did not persist. A couple of panel members suggested that ARC use the bridge collapse as an opportunity for model validation. The observed data obtained during the bridge closure is data that could never be obtained under normal circumstances, and ARC could take advantage of this. Another panelist suggested using the experience to devise plans for how similar types of incidents could be evaluated in the future.
ARC is very interested in better understanding the impacts that Transportation Network Companies (TNCs) like Uber and Lyft are having on traveler choices and also understating the potential impacts of autonomous vehicles (AVs) in the future. These separate, but related, topics were discussed in detail during the meeting.
On AVs, ARC has already spent some time pursuing scenario tests to examine potential impacts of this technology. The results of the scenario tests were presented to the panel. Several assumptions were made about AVs in the context of the scenario tests: auto operating costs were reduced, the sensitivity to in-vehicle travel time was reduced, parking costs were reduced, road capacities were increased, and AV market penetration was assumed to be 100 percent. ARC did several tests that organized the assumptions into different groups of assumptions that were used for scenario testing. While results varied a bit across the scenarios, the results generally suggested that AVs will increase the number of daily vehicle trips and trip lengths. VMT would increase under all combinations of the assumptions, while vehicle-hours of travel (VHT) would decrease with the capacity assumption, but increase with other assumptions. Transit ridership was also forecasted to reduce under AV assumptions. Several panel members commented on assumptions that were not considered, including the impacts of zero-occupant vehicles on congestion, changes to terminal times (since AVs could have curbside drop off), AVs as an egress mode for transit, and changes to land use patterns, where additional urban sprawl may be a likely impact of AVs, which could further increase trip lengths.
One panelist asked how ARC's board has responded to the analysis. ARC noted that their board is not particularly interested in specific questions like AVs, but ARC believes it is important to understand the potential impacts of emerging technologies. At another MPO, a panel member noted that the board is skeptical of forecasts where a lot of uncertainty exists. ARC noted that public acceptance of AV technology will likely take much longer than the delivery of the technology itself, and this contributes to the uncertainty over when AVs will really start to have an impact on travel patterns. One panelist noted that there will likely be a long period of mixed traffic, where AVs share the highways with traditional user-operated vehicles, and the interaction between the two is associated with additional uncertainty. A couple panel members commented that the era of offering a single point forecast is likely over, and there is a real need to be explicit about the uncertainty that exists in a lot of areas of travel modeling. One suggested providing bounds on the uncertainty is one way to achieve the goal of representing uncertainty. Another suggested that uncertainty can be dealt with by doing more testing and developing a range of possibilities.
One area of impact of AVs will likely be in replacement of the TNC fleet and possibly the replacement of personal vehicles altogether. A couple panelists thought that some people will likely prefer to own their own car for any number of reasons (e.g., keeping personal belongings in it, perceived cleanliness, etc.) and more research into this area is really needed in order to make statements about the transition of the personal vehicle ownership model. The transit agency in Atlanta, MARTA, is thinking about how TNCs may transform the demand for transit services. For instance, forty foot buses may not be tenable transportation alternatives in certain corridors.
A couple of panel members were concerned that travel models are becoming less credible in the face of changing technologies, like TNCs, and remarked on the importance of building in the ability to forecast TNC travel within the models. To do this, it was discussed how new sources of data may be needed, including cell phone and GPS data and new travel surveys. One panel member commented on a recent household survey in San Diego where TNCs appeared in the trip database. They made up only a small percentage of trips (about 0.4 percent), but the panel generally agreed that the share of trips being made using TNCs may be changing rapidly. Another panel member was concerned that TNC trips may get underreported using traditional survey techniques due to the correlation between hard to reach populations and use of TNCs. One panel member remarked on work being done in the San Francisco region (at the San Francisco County Transportation Authority) currently to estimate the size of the TNC market there and encouraged ARC to learn more about that work. One problem noted by several panelists was that the primary TNCs, Uber and Lyft, have little interest in teaming with MPOs. In total, the panel was generally in agreement that incorporating TNCs in the ARC travel model should be an area of focus for ARC.
External Trips
The external travel forecasts for ARC's travel model pivot off of the Georgia statewide model forecasts. The external forecasts were temporally validated, where the external model was calibrated to 2010 conditions and validation was conducted for 2015 conditions (see Figure 13). Panel members remarked that using the statewide model as the basis for external model forecasts and the temporal validation were both solid approaches to handling external travel. Another panel member asked how the ARC model dealt with travelers living in the region that travel outside the region, from the perspective of CT-RAMP. The basic approach used was to remove this sample of travel from the sample used in CT-RAMP model development, since these trips are predicted by the external model. Similar approaches have been used in other regions, as pointed out by other panelists.
A couple panel members commented on procedures for dealing with workplace location choice in CT-RAMP and its relationship to the external model, since some workers come from areas external to the region. One way of doing this is to run the external model first, and then subtract the worker totals entering the region, by zone, from the model's estimates of workers in each zone. The employment used by the ABM is then the difference of the total and the number predicted by the external model. It was recognized that there is no perfect way of doing this and ensuring consistency. Shadow prices are one way of doing this.
Airport Trips
The air passenger model predicts the number of air passengers to and from Atlanta's airport. It was noted that the survey that was used to calibrate the model is rather old, having been conducted in 2009. One panelist suggested that ARC get started on the planning process for a new survey since the amount of time it takes from the planning stage of a survey to implementation and ultimately having data in hand can take years. A couple panel members suggested that in other regions, the airports are doing regular surveys themselves, which can often be used to at least update the air passenger model component of a regional model. A couple panelists also commented on the importance of TNCs as a mode for drop off and pick up of air passengers. Another panel member asked more generally about how meeters and greeters (i.e., people picking up and dropping off air passengers) are forecast by the travel model system. ARC indicated these are currently handled in the ABM, though in the future, these forecasts might be improved by special handling of these trips.
One relatively new feature ARC rolled out was a visualization feature called ABMVIZ. The ABM generates large volumes of data and ARC created ABMVIZ, which is still evolving, to make use of those outputs and make the ABM outputs accessible to the model user. The main objectives of the tool were that it be interactive, dynamic, easy to use, and fast. The latest version of ABMVIZ is a web-based application that is open for planning partners and the public to use, to view outputs of model runs performed by ARC. The tool is continually evolving as ARC adds new visualization features and new modeling scenarios. It is also responsive to feedback from users. The tool itself uses a flat data file output from a SQL preprocessing script in order to ensure the tool can provide fast response for the user (rather than generating queries on the fly). One panel member commented that such tools exist in other regions, but the ARC tool is very sleek. The same panel member suggested that ARC track the tool's usage on their website to find out how much use it receives. A couple of panel members remarked that ABMVIZ would be a good way of communicating model results to the public and showing the capabilities and value of the ABM to non-modelers.
Another feature ARC has worked on is a cloud computing option for running the model. They primarily looked into this as an option to support running the model by partner agencies that did not want to invest in the necessary hardware to run the model themselves. ARC has done some tests and found diminishing returns in the amount of computing power to use to run the model, which is primarily a result that certain model components cannot be multi-threaded. Several panelists were interested in whether ARC had a recommendation. ARC, however, simply wants to provide partner agencies with another option for running the model other than running in house. So far, ARC has not received requests to use the cloud computing option.
New Starts
ARC noted that in order to use their ABM for evaluation of New Starts projects, the model needs to be approved by the Federal Transit Administration (FTA). ARC was interested in hearing panelists' views on whether this was worth the investment in time and effort. The other option to evaluate New Starts projects is to use FTA's STOPS model, but ARC and MARTA were concerned that the STOPS model is too conservative, essentially making Atlanta non-competitive in pursuing New Starts funding. One panelist noted that the San Diego region decided not to initiate a model review from FTA for New Starts funding. Another panel member suggested that there is risk involved in the process if the FTA reviews the ARC model but does not think it meets the standards for New Starts projects.
Transportation Improvement Program (TIP)
ARC described the process they used for their last transportation plan. In the latest TIP, ARC evaluated more than 20 projects using VISSIM for assignment and network analysis. Because of the amount of time it takes to run the ABM, they bundled projects into just a few ABM scenarios, whose outputs were used in the more detailed network analyses. One panel member applauded this approach. In other regions, sometimes the full travel model is run for each project individually, and there may not be a lot to gain from doing this, not to mention it is very time consuming. ARC noted that there process was determined based on outreach initiatives and getting partner agencies and stakeholders involved in the process. Ultimately, agreement was reached that project bundles was the appropriate approach for running the ABM.
Other Modeling Activities
ARC is very interested in training partner agencies in how to run the model and how to interpret model output results. When ARC rolled out the ABM officially (and retired the trip-based model), they invested in extensive training with partner agencies. ARC is also interested in feedback from planning partners to make the model more responsive to all model user needs. Multiple panel members commented on the difficulty that MPOs have in maintaining staff that are knowledgeable in running the model. This is becoming a bigger issues as agencies move toward ABMs, compared with trip-based models. Many agencies are contracting modeling work to consultants. With that being said, the panel commended ARC for their efforts in coordinating with partner agencies.
On the last half day of the meeting, the peer review spent about two hours in an executive session, closed to all participants of the meeting except for the panel members. This was so the panel members could speak freely and openly among themselves while developing formal recommendations for ARC. This section of the report details the recommendations of the panel.
The panel made the following observations about ARC's modeling activities.
The panel made a series of recommendations related to various aspects of ARC's modeling activities, as documented in this section.
This section lists all individuals who attended the meetings, including panel members, ARC staff, and peer review support staff.
| Panel Member | Affiliation |
Bruce Griesenbeck |
Sacramento Area Council of Governments (SACOG) |
Habte Kassa |
Georgia Department of Transportation (GDOT) |
Ram Pendyala |
Arizona State University (ASU) |
Thomas Rossi |
Cambridge Systematics, Inc. |
Wu Sun |
San Diego Association of Governments (SANDAG) |
Panel Member |
Affiliation |
Claudette Dillard |
Atlanta Regional Commission (ARC) |
Brian Gardner |
Federal Highway Administration (FHWA) |
Rob Goodwin |
State Road and Tollway Authority (SRTA) |
Bryan Hobbs |
Metropolitan Atlanta Rapid Transit Authority (MARTA) |
Doug Hooker |
Atlanta Regional Commission (ARC) |
Kyeil Kim |
Atlanta Regional Commission (ARC) |
Kyung-Hwa Kim |
Atlanta Regional Commission (ARC) |
Steve Lewandowski |
Atlanta Regional Commission (ARC) |
Vidya Mysore |
Federal Highway Administration (FHWA) |
John Orr |
Atlanta Regional Commission (ARC) |
Guy Rousseau |
Atlanta Regional Commission (ARC) |
Christopher Silveira |
Metropolitan Atlanta Rapid Transit Authority (MARTA) |
Don Williams |
Metropolitan Atlanta Rapid Transit Authority (MARTA) |
| Panel Member | Affiliation |
Jason Lemp |
Cambridge Systematics, Inc. |
Sarah Sun |
FHWA/TMIP |
| Panel Member | Affiliation |
Jonathan Nicholson |
Atkins |
Rosella Picado |
WSP |
| Time | Description |
9:00 a.m. to 9:15 a.m. |
Introduction |
9:15 a.m. to 9:20 a.m. |
Goal and Objectives of Peer Review |
9:20 a.m. to 9:30 a.m. |
History of ARC ABM Development |
9:30 a.m. to 9:40 a.m. |
Profile of the Atlanta Region |
9:40 a.m. to 10:00 a.m. |
Overview of ARC ABM |
10:00 a.m. to 10:20 a.m. |
Model Structure |
10:20 a.m. to 10:30 a.m. |
Break |
10:30 a.m. to 11:30 a.m. |
Model Calibration / Validation Data Review |
11:30 a.m. to 12:30 p.m. |
Lunch |
12:30 p.m. to 2:30 p.m. |
Model Calibration / Validation (Part 1) |
2:30 p.m. to 2:45 p.m. |
Break |
2:45 p.m. to 4:30 p.m. |
Model Calibration / Validation (Part 2) |
4:30 p.m. to 5:00 p.m. |
Overview of ABMVIZ and ABM Cloud |
| Time | Description |
9:00 a.m. to 10:00 a.m. |
Model Applications |
10:00 a.m. to 10:15 a.m. |
Current Modeling Activities |
10:15 a.m. to 10:30 a.m. |
Future Modeling Plan |
10:30 a.m. to 11:00 a.m. |
Questions and Answers |
11:00 a.m. to 2:00 p.m. |
Peer's Review Time and Lunch |
2:00 p.m. to 3:00 p.m. |
Discussion on Peer's Review |
3:00 p.m. |
Adjourn |
Bruce Griesenbeck is the Principal Transportation Analyst and manages the Data and Analysis group at the Sacramento Area Council of Governments (SACOG). His specialty is travel demand forecasting, and he serves on the Transportation Research Board Travel Forecasting Committee. He received his bachelor's degree in sociology from Swarthmore College, and master's degree in City and Regional Planning and in Civil Engineering from University of California at Berkeley.
Habte Kassa has over 14 years of transportation planning experience. He has been working with the Georgia Department of Transportation (GDOT) as a transportation specialist for more than 10 years. In his current position, he is responsible for managing the development and maintenance of the Georgia statewide model as well as 14 regional MPO models. He is also a technical implementation manager for various research projects. He was the data manager for the 2009 NHTS add-on as well as the ongoing 2016 NHTS add-on. Prior to joining GDOT, he was a transportation planner with the Meadowlink commuter services and operations/safety manager for its parent company, TransWare.
Ram Pendyala is a Professor of Transportation Systems and Director of a USDOT-sponsored University Transportation Center in the School of Sustainable Engineering and the Built Environment at Arizona State University, where he has served on the faculty since 2006. Between 2014 and 2016, he served as the Frederick R. Dickerson Endowed Chair and Professor of Transportation in the School of Civil and Environmental Engineering at Georgia Tech. He is an expert in the analysis of traveler behavior and values, and studies the impacts of new and emerging technologies on cities and people using state-of-the-art microsimulation models of activity-travel demand and mobility patterns. He has published more than 150 papers and serves as the Chair of the Planning and Environment Group of the Transportation Research Board (TRB). He previously served as the Chair of the TRB Traveler Behavior and Values Committee and the Travel Analysis Methods Section. He is an Associate Editor of Transportation Research Part D. He has his PhD and Master's degrees in Civil Engineering with a specialization in Transportation from the University of California at Davis.
Thomas Rossi is a Principal of Cambridge Systematics with 35 years of management experience, specializing in the areas of travel demand modeling and forecasting and transportation planning. He has developed and applied travel demand models throughout the United States, including many activity based and trip based models in large metropolitan areas. Since 1993, he has been working with the U.S. Department of Transportation to conduct research and develop and teach training courses in travel demand modeling as part of the Travel Model Improvement Program (TMIP). He manages the TMIP Peer Review Program for travel models. He served as Principal Investigator on the project that authored National Cooperative Highway Research Program Report 716, Travel Demand Forecasting: Parameters and Techniques and the Strategic Highway Research Program project C10, "Partnership to Develop an Integrated, Advanced Travel Demand Model and a Fine-Grained, Time Sensitive Network." He is the past chairman of the Transportation Research Board's (TRB) Transportation Demand Forecasting Committee and has served as a member of several other TRB committees. He received a Master's degree in Transportation and Bachelor's degrees in Mathematics and Civil Engineering from the Massachusetts Institute of Technology.
Wu Sun is a senior transportation modeler at the San Diego Association of Governments (SANDAG). Wu specializes in the development, application, and project management of travel demand models, especially activity-based models, as well as software design for transportation model applications. Prior to joining SANDAG, Wu worked as a consultant in New York, and then a software engineer in Oakland California. Wu holds a PhD in transportation engineering and a M.S. in computer science.
ARC ABM Specification Report
http://atlantaregional.org/wp-content/uploads/abm-specification-report-2017.pdf
ARC ABM Calibration Report
http://atlantaregional.org/wp-content/uploads/abm-calibration-report-2017.pdf
ARC ABM User Guide
http://atlantaregional.org/wp-content/uploads/abm-user-guide-2017.pdf
ARC ABM Data Dictionary
http://atlantaregional.org/wp-content/uploads/abm-data-dictionary-2017.pdf
ARC ABM Frequently Asked Questions
http://atlantaregional.org/wp-content/uploads/abm-frequently-asked-questions-2017.pdf
