Case Study: The Ohio-Kentucky-Indiana Regional Council of Governments Experience
Setting the Stage
What Does OKI Have?
As noted in the Introduction, OKI's Congestion Management Process includes capital investment applications as well as Transportation System Management (TSM) techniques to guide the development of its long-range and short-range transportation planning, including its Transportation Improvement Plan (TIP). The TIP implements portions of OKI's 2030 Regional Transportation Plan by programming funding for improvements over the next four-year period. It is a compilation of all publicly assisted transportation programs, including highway and transit projects, constrained to available funding levels and prioritized by need. Upon adoption by the OKI policy board, the TIP becomes a programming document that directs the flow of transportation improvements in the region. Under Federal law, an MPO must prepare a Long Range Plan and a TIP; moreover, a project must be included in both the Plan and the TIP as a prerequisite for Federal funding assistance.
Potential TIP projects eligible for Federal Congestion Mitigation/Air Quality (CMAQ) or Surface Transportation Program (STP) are reviewed by OKI's Prioritization Subcommittee. A key component of the Committee's Prioritization Process is the Project Scoring Process.
Under the Project Scoring Process, highway and transit projects are first scored separately using Transportation Factors. Transportation factors take into account items to be examined during the construction/acquisition phase of a project. A subtotal of 45 points is available from the transportation factors. All projects are next scored on Planning Factors, which are factors that should have been considered during the planning, or development phase, of the project. A subtotal of 50 points is available from the planning factors. Finally, all applications are subjected to a Benefit/ Cost ratio evaluation that provides up to ten additional points, resulting in a total possible 105 points.
Transportation factors applied to roadway projects, usually with a 1 to 5 score for each factor, include the following: existing safety; impact on safety; existing level of service (LOS); impact on LOS; average daily traffic; status as a freight corridor; roadway classification; conformance with existing design standards; and status of the project (how soon construction can begin).
Transportation factors applied to transit projects include: project impact on safety and security; useful life of the project; service improvements; system impact; type; time to implementation; ridership impact; capital utilization of the item being replaced; and planning/forecasting (higher points are given for transit projects that have been in an adopted planning document).
Planning factors applied to all projects, also using scores typically ranging from 1 to 5 for each factor, are as follows: replacement/expansion factor (this factor gives preference to projects that invest in replacement and preservation rather than new facilities, in accordance with the OKI TSM approach); environmental justice; land use conformance; air quality/energy improvements (up to 10 points if vehicle hours traveled, vehicle miles traveled, and air emissions are reduced); local funding share (up to 10 points if the local sponsor overmatches its 20 percent share by 30 percent); number of travel modes improved; intermodal connectivity; and the existing condition of the project area.
The final scoring section for all projects makes use of a hybrid benefit/cost ratio analysis. Cost is readily available in dollar terms, while the benefit side is represented by a surrogate that is valued according to the score awarded up to this point for the transportation and planning factors (the points, in effect, represent the intrinsic "benefit" to the region). The factor point subtotal (maximum 95) is divided by the cost of the proposal (in millions of dollars). The subsequent value (which can have a very wide numerical range) is then scored from two to ten points via the scale shown in Table 1. When added to the previous subtotal, a maximum of 105 points is possible. OKI staff initially selected this hybrid method due to the extensive variability in project types and the amount of time it would take to do a true dollar-based benefit-cost analysis with then-available tools.
|Benefit/Cost (Project Transportation and Planning Points Divided by Project Cost, in $Million)||Points Available|
|Greater than 1,000||10|
|Greater than 100||8|
|Greater than 10||6|
|Greater than 5||4|
|Greater than 1||2|
The Prioritization Subcommittee can also accommodate projects that do not fit the highway or transit definition. In these cases, it examines each application and subjectively ranks the application in comparison to the highway and transit applications received.
Once ranked, projects then compete against each other based on the funds available in their respective Federal funding categories-either CMAQ or STP. After the ICC (the parent committee of the Prioritization Subcommittee) develops a final ranking of CMAQ and STP projects, this recommended list is presented to the OKI Executive Committee or Board of Directors for review and approval.
What Does OKI Want?
OKI is generally satisfied that its Project Scoring Process addresses a comprehensive range of critical transportation and planning factors, including the priority OKI assigns to the reconstruction, preservation, and greater efficiency of existing infrastructure. Nonetheless, the OKI Prioritization Subcommittee continually seeks to improve the scoring process, reviewing and revising it on an "as needed" basis. The ICC adopted the last revision to the scoring process on January 20, 2006.
One area of the scoring process that OKI has targeted for future improvement pertains to reducing the reliance of the method on qualitative scoring methods. For instance, the "Project Impact on LOS" factor used for roadway projects assigns a score of 5 to projects with "High Impacts," 3 to projects with "Medium Impacts," and 1 to projects with "Low Impacts." The specific criteria for gauging what are high versus medium or low impacts are not specified but rather left to expert judgment. Similarly, the scoring of a project with an average daily traffic (ADT) flow of more than 25,000 at 5 points, versus a project at over 20,000 ADT at 4 points, can lead to an artificial distinction being drawn between a project that serves 24,900 ADT and another that serves 25,100 ADT.
The hybrid benefit/cost ratio approach used in the Project Scoring Process mixes dollar-based costs with a point-based benefit measure. In this system, benefits are capped at no higher than 105 points, and a total benefit score may not change significantly even if one project saves many times more hours of travel time than another project. For instance, a capacity project on the same congested roadway that has a very large impact on LOS would score only 2 points more on the 105-point scale than a project with a medium impact. Even if the high-LOS-impact project costs only marginally more, it is possible that it would not compete effectively against the medium-impact project. The selection bias in the hybrid benefit/cost ratio approach would be most pronounced for larger projects, where high benefit streams would not be weighted proportionately to the higher dollar costs of the projects.
OKI has long recognized that one solution to reducing the subjectivity of assigning point values to the transportation and planning factors would be to put some factors into dollar terms and apply economic analysis tools to compare the dollar values of benefits to costs directly. For instance, 6 of the 8 transportation factors and the air quality/energy improvements planning factor in the Project Scoring System are directly or indirectly linked to safety and capacity impacts that can be measured monetarily. These factors could be measured in terms of crashes avoided; hours of travel time reduced; and lower vehicle miles and air emissions over the project life cycle. Dollar values could then be assigned to these benefits and the totals compared directly to the dollar value of costs.
Fort Washington Way (Interstate I-71), Cincinnati, OH.
How Has OKI Attempted to Enhance Its Benefit-Cost Analysis Procedure?
The significant impediment that OKI has encountered to date in the application of dollar-based benefit-cost analysis (BCA), and the reason it developed the existing qualitative rankings of project benefits, was the extensive variability in project types and the amount of time it would take to do a dollar-based BCA. OKI has used dollar-based BCA for some very large projects in recent years, but due to the specialized skills and data needed, it relied on outside experts to accomplish the analyses at considerable expense to OKI.
Still, OKI has continued to search for potential tools that would facilitate more applications of dollar-based BCA to projects. Of particular importance to OKI in this search is that the tools should be usable by OKI staff, affordable to acquire and operate, and make use of existing data resources and staff skills to the extent possible. OKI has an experienced professional staff that includes specialists in planning, transportation modeling, environmental analysis, demography and other specialties, but does not have a staff economist. Therefore, a useful economic analysis tool would need to come supplied with economic data and supporting guidance. Other objectives of importance to OKI in selecting a BCA tool are that the model be recognized by Federal and State partners and able to accommodate transit as well as highway projects.
In 2004, following a review of publicly available BCA models, the OKI staff decided to explore the use of the Surface Transportation Efficiency Analysis Model (STEAM) developed by the FHWA in the 1990's.
In the late 1990's, in response to guidance in the Intermodal Surface Transportation Efficiency Act of 1991 and, subsequently, the Transportation Equity Act for the 21st Century, the FHWA undertook the development of tools to assess the efficacy of multimodal transportation alternatives and demand management strategies. As part of this effort, the FHWA developed a computer-based procedure for linking the outputs of regional travel demand model outputs with computerized procedures to evaluate system, corridor, or project alternatives from an economic analysis standpoint.
STEAM accepts input directly from the four-step travel demand modeling process that is used by most MPOs in their planning exercises. It post-processes the traffic assignment outputs from the travel demand model, and its speed models account for the build up and dissipation of vehicle queuing, to estimate highway travel speeds under congested conditions. It also uses vehicle hours traveled (VHT) and vehicle miles traveled (VMT) data generated by the travel demand model to calculate travel time, vehicle operating and safety impacts, as well as impact measures pertaining to air emissions and energy consumption. STEAM then monetizes the post-processed impact measures of the travel demand model, using dollar values for time, vehicle operating costs, safety, and emissions that can be localized for specific analyses. It also allows the analyst to perform risk analysis, thereby minimizing the potential for controversy about the selected unit monetary values for travel benefits or impact estimates. STEAM produces estimates of system-wide impact; i.e., impact estimation is not limited to the improvement corridor or project.
OKI downloaded the STEAM software and user documentation from the FHWA STEAM website. Following a review of the documentation and inquiries to other organizations familiar with STEAM, OKI ascertained that its in-house travel demand model and modeling expertise would be compatible with the STEAM software requirements. OKI then began the process of implementing the STEAM analysis.
Initial Configuration of STEAM
The OKI Travel Demand Model is a key tool in many of OKI's work elements, including corridor studies, air quality analysis, and long-range transportation planning. The current version (6.3) of this model is composed of TRANPLAN programs and a series of FORTRAN programs written by OKI. It is a state-of-the-practice model that uses the standard four-step sequential modeling approach of trip generation, distribution, modal choice, and assignment. The model uses demographic and land use data and capacity and free-flow speed characteristics for each roadway segment in the network to produce a "loaded" highway network with forecasted traffic volumes with revised speeds based on specified speed/volume relationships. The model also accounts for travel demand among multiple modes, including transit and auto, through a mode choice component, also a standard feature of state-of-the-practice models.
OKI's first task prior to running the STEAM model was to establish the interface between its travel demand model and STEAM. OKI attained initial success in this task in October 2004. To create data from the travel demand model in the format needed for the STEAM program applications, OKI added programs to its travel demand model stream. It eventually documented the specifications of the required batch job control file, procedures, and input/output files in a draft document entitled "Preparation of Travel Demand Model Data for STEAM" (March 10, 2006). The documentation covers the generation of the following data files for STEAM: highway network speeds; zonal socioeconomic data; highway and transit person trips; and transit travel times, mode shares, and other data.
The BCA module in STEAM includes many economic variables whose values can be set by the user. If the user lacks data, STEAM contains default values-although these values are not regionally specific and typically should be updated to adjust for inflation. OKI undertook a thorough review of the default economic and other values provided in STEAM to make sure they were appropriate for the OKI region. This process was documented by OKI in the draft document "Preparation of Values for Variables in STEAM" (March 10, 2006). Among the many values researched and updated by OKI were the value of travel time by mode; the cost per gallon of fuel; cost per crash; and project capital costs. OKI made use of industry data reported in public documents, and updated cost data using a variety of national and regional price indices.
FHWA Support to OKI
Early in its effort to implement STEAM, OKI contacted the FHWA's Office of Asset Management for technical support on the implementation, operation, and interpretation of STEAM model results. OKI initially contacted the FHWA in June 2004 as part of its survey of available economic models. Representatives of OKI also attended one of FHWA's "Economic Analysis for Highway Decision-Makers" workshops, held in Frankfort, KY, in August 2004.
In October 2004, OKI notified the FHWA that it had successfully run STEAM but had encountered constraints in the model that needed to be corrected before the model could accommodate the analysis required by OKI. Among these constraints was an inability for the user to change STEAM's default discount rate (by which the model captures the opportunity cost of money) and an unrealistically low cap on the maximum allowed capital costs of transportation projects.
The FHWA agreed that the constraints identified by OKI imposed significant restrictions on the utility of STEAM results for OKI and other potential STEAM users. In February 2005, the FHWA initiated a small contract with the consultant that originally developed STEAM to modify the source code and to provide other programming and technical support to STEAM users. The contractor completed the initial fixes to the STEAM model in March 2005 and implemented improvements to the STEAM website.
Thereafter, OKI and the FHWA interacted regularly on a range of issues associated with the operation and interpretation of STEAM and appropriate economic values to be used in the model. In April 2006, OKI invited the FHWA to Cincinnati to 1) review its data and documentation; 2) evaluate a STEAM assessment of a trial group of six projects; and 3) provide a presentation to the OKI Board of Directors on the role of economic analysis and STEAM in transportation program planning.
Refining the STEAM Specifications
The FHWA review team met with OKI in Cincinnati in April 2006. The review team suggested a few minor changes to the input parameters for the STEAM BCA. Overall, the FHWA team found that the documentation provided in the OKI briefing papers was exemplary in its thoroughness and clarity. The FHWA team also cautioned OKI on the need to coordinate its STEAM analysis with the environmental process and to use particular care in the valuation of environmental parameters in the STEAM model.
The FHWA review team and OKI spent a substantial amount of time discussing the six projects being evaluated with STEAM. OKI had selected the projects for the trial STEAM evaluation because their impacts were already well understood based on the completed "Southwest Warren County Transportation Study." The projects consisted of three lane additions, a road extension, a new interchange, and an interchange improvement.
Earlier OKI testing on other similar projects revealed that STEAM was not appropriate for evaluation of each of the six projects separately due to the limited size of each individual project compared to the size of the region and the fact that some projects would not be implemented for more than 20 years. The FHWA team made the following recommendations to obtain more robust results from the STEAM model:
- The projects should be evaluated as a group rather than individually. This approach was consistent with the MPO's intent to review the projects as a group in the prioritization process and was consistent with observations from the prior study of the complementary nature of the individual projects. The combined effects of the projects in southwest Warren County would provide an unambiguous indication, in the travel demand model, of impacts to a clearly identifiable segment of the travel market, and consequently, would be revealed in the STEAM results.
- The interchange improvement project that primarily addressed safety and queuing should be excluded from the analysis. The impacts of these types of projects are not generally well represented in travel demand models and thus would not show up well in STEAM results.
- Although treated as a group, the remaining five projects would be modeled as being implemented at separate discrete points in time over a 20-year analysis period to reflect their different planned implementation dates. This approach involved several runs of the travel demand model and STEAM for different years, with and without specified projects. Based on these multiple runs, the benefits and costs for all the projects could be interpolated and summed up to yield a collective net benefits value.
To facilitate this consolidated project analysis, the FHWA team developed a new spreadsheet tool named "STEAMStream". The spreadsheet used the net benefit totals generated by STEAM for discrete analysis years (i.e., 2015, 2020, and 2030) and interpolated them to intermediate years of the analysis period (e.g., 2016, 2017, etc.), and then summed them to their present values. This tool not only enhanced the ability of STEAM to handle project groups but provided a more robust estimate of the net benefits of surface transportation projects than was provided by STEAM before the OKI application.
Interpretation of STEAM Results
Following the development of the STEAMStream spreadsheet tool, OKI developed the results of the STEAM BCA of the five Southwest Warren projects (see Table 2). The FHWA review team helped OKI review the STEAM results for reasonableness and consistency.
Collectively, the results show the overall package is cost-beneficial, with the present value of transportation benefits exceeding the present value of costs. The findings of the analysis are generally supportive of those of the "Southwest Warren County Transportation Study," with one exception. The STEAMStream results indicated that one project in the group, an intersection project to be implemented in 2030, caused some erosion in net benefits of the projects as a group. The reason for this erosion of net benefits is unclear. However, a contributing factor may be that the project falls in the last year of the analysis period covered by the travel demand model runs, before it can begin to generate its full benefit stream based on future traffic growth.
OKI intended that the application of STEAM to the Southwest Warren County projects serve only as a test of the STEAM process. In a real planning situation where STEAM was being used to evaluate the best project mix, OKI would have treated the adverse results for the one intersection project as guidance to review the assumptions about the project's overall cost and performance. If possible, OKI would re-test the intersection project by implementing it as an earlier date (e.g., 2025) or by conducting an additional STEAM analysis for the year 2040. Finally, even if the STEAM analysis results still did not support the intersection project, it would be valid for OKI to decide for the project based on other information (e.g., safety benefits not captured by STEAM). An economic analysis process is intended only to assist decision-makers in evaluating a project—it does not make decisions for them.