Baltimore Metropolitan Council (BMC) Peer Review
4.0 Peer Review Panel Recommendations
This section summarizes the recommendations of the panel generated during the afternoon panel work session. The recommendations are organized by the following major topic headings:
- Model Structure
- Data Issues
- Validation
- Sensitivity Testing
- Application Advice
The panel agreed to formulate one over-arching conclusion to which all could agree under each heading followed by specific observations to support the statement. The statements are directed toward BMC staff in recognition of the limited time remaining on the consultant's contract, the objective being to encourage agency staff to become familiar with executing the applications while the consultant is still available to assist.
Erik Sabina was unable to attend the afternoon session. The panelists were joined by Ron Milone from the Metropolitan Washington Council of Governments (MWCOG). The panel agreed to identify the panel member who originated the comment to permit staff to seek clarification as needed. Panelists are identified by their initials, as follows:
- Kermit Wies (KW)
- Ron Milone (RM)
- Ken Cervenka (KC)
- Ram Pendyala (RP)
- Lei Zhang (LZ)
- Peter Vovsha (PV)
The panel agreed that the overall model structure is reasonable state-of-practice and is pleased that intra-household interaction has been included.
4.1.1 Panelist specific advice
- PV, LZ, RP: Tour-formation structure should include mode choice sensitive to stop location, preferably through joint choice modeling frameworks. Consider shorter (or continuous) time resolution in view of integration with DTA.
- KC: Best-of-practice modeling approaches will separate the auto access to transit into drive-and-park and passenger drop-off (commonly referred to as kiss-and-ride), and BMC should consider making this model update at a future point in time.
Overall method of using survey data to estimate is acceptable. The next generation of surveys conducted by BMC should inventory lessons learned from model development to improve overall survey usefulness.
4.2.1 Panelist specific advice
- KC: The Census Bureau's ACS (American Community Survey) Part 3 flow data should be useful in preparation of modeled versus observed district-to-district checks of home-to-work flows for different socio-economic groups (e.g., the work trips separated into the cars available to the worker's household). But some cautions will be needed in the interpretation of the findings since there are some documented local concerns with the accuracy of the Census Bureau's place-of-work geocoding in the ACS.
- PV: Establishment surveys will augment the information available on the attraction end. The surveys should be conducted across-the-board with oversampling of special generators.
- KC: As BMC considers an investment in a new household survey, it will be useful to examine the white paper about household survey design and technology that was written a few years ago by Peter Vovsha.
- KC: The last region-wide transit rider survey was conducted in 2008, and it is time for another data collection effort that will benefit from the significant improvements that have been made since 2008 in data collection methodologies that focus on personal interviews conducted with the aid of well-designed computer tablets.
The panel defined “Validation” as comparing model outputs against empirical data. The current work demonstrates that the model estimation is consistent with Household Travel Survey. The original Model Validation Plan prepared May 2014 is useful. BMC staff should use this as a testing guide during initial application.
4.3.1 Panelist specific advice
- All: Complete the tests identified in the Model Validation Plan.
- PV: The model was estimated using the household survey. So far the validation results include only household survey tabs. Need to compare with a completely independent source such as traffic volumes and transit ridership.
- KC: The Model Validation Plan was very thorough, the biggest concern is that the planned tests were not done at the time of this peer review. Whether a trip-based or activity-based model, some areas worth emphasizing include modeled versus observed checks of: screen line and cordon line traffic volumes, by time-of-day; district-level checks of vehicle miles traveled (VMT); travel time contours, by time-of-day, to/from different points in the region; route-level and district-level checks of weekday transit passenger boardings; and checks of district-to-district flows of linked transit trips by mode of access, purpose, and socio-economic group.
- RM: Use on-board survey as an independent comparison. Don't expect a perfect match.
- PV: Recommend cross-comparisons of behavior across age, income, car-ownership, geography using just model results to check for internal consistency and intuitive outcomes.
- RP: Check skimmed travel times against empirical speeds at the corridor level.
- RP: Concerned about variance between expanded survey and synthetic population. While this may be expected to some degree due to different controls used for the weighting process, the model validation process needs to consider this variance. Explain deviations between model predictions and expanded survey distributions/statistics based on the differences between the synthetic population and the expanded survey sample.
- KC: In addition to the typical validation checks, it would be good to compare how the predicted time spent in weekday travel and non-travel activities compare to the observed.
The panel recognizes that “Sensitivity Testing” guidance is covered in validation plan, but has not accomplished to date.
New kinds of sensitivity testing:
- Highway/VMT Pricing: Sensitivity of mode choice, time-of-day by different travel markets including trip purposes and income.
- Land Use: Sensitivity of trip lengths, VMT and mode choice
- Work schedule flexibility: Sensitivity of work tour timing, congestions.
4.4.1 Panelist specific advice
- KC: it would be useful to see how the predicted time spent in weekday travel and non-travel changes in response to changes in the coded road/transit networks (a “change between alternatives” test) and to changes in the zonal demographics (a “change between years” test).
The panel suggested that BMC staff begin executing the model code in-house as soon as possible. This permits intuitive application testing and troubleshooting with the benefit of local knowledge while the consultant is still under contract. In addition to the aggregate measures of model performance that are produced with each model run, staff should devise new queries that explore the specific performance advantages of the ABM (e.g. equity distribution of policies).
4.5.1 Panelist specific advice
- RP: Fully disaggregate Household Activity Pattern analysis. Look at disaggregate records to discover inconsistencies.
- LZ: Examine a real program (e.g. Baltimore Link). Examine neighborhood level transit usage by access to transit and compare with the on-board survey. Tabulate equity and distribution results and compare against program objectives.
