- Briefing Room
U.S. Department of Transportation
Federal Highway Administration
1200 New Jersey Avenue, SE
Washington, DC 20590
This document is disseminated under the sponsorship of the U.S. Department of Transportation in the interest of information exchange. The U.S. Government assumes no liability for the use of the information contained in this document.
The U.S. Government does not endorse products or manufacturers. Trademarks or manufacturers' names appear in this report only because they are considered essential to the objective of the document.
The Federal Highway Administration (FHWA) provides high-quality information to serve Government, industry, and the public in a manner that promotes public understanding. Standards and policies are used to ensure and maximize the quality, objectivity, utility, and integrity of its information. The FHWA periodically reviews quality issues and adjusts its programs and processes to ensure continuous quality improvement.
This study responds to Senate Report 113-182, which requires the U. S. Department of Transportation to “evaluate the use of benefit-cost analysis by State departments of transportation (State DOTs), and to issue a report to the House and Senate Committees on Appropriations.” This report addresses the four topic areas listed therein:
This study uses a qualitative approach, combining a literature review of existing research on this topic with case studies of nine State DOTs that provide more in-depth information about how benefit cost analysis (BCA) is used at the State level. In addition, a questionnaire was sent to all FHWA Division Offices to gather information about the use of BCA among each office's State DOT counterparts. A Request for Information (RFI) was also posted to solicit input from consultants and others who may be familiar with State practices. Findings from each of these sources are presented in the report in turn. In the final section, the findings are synthesized to offer overall conclusions, which are also summarized here.
There continues to be significant variation in the extent to which State DOTs use BCA, both across States and across different project types and planning stages. Based on questionnaire results from 46 FHWA Division Offices and information from the literature, roughly five to six State DOTs systematically employ BCA to inform decisionmaking; but use of BCA continues to be the exception, not the rule. Many States use BCA only for certain project types or for situations where it is required for funding. Safety projects and large or significant projects (as defined by the state) are most likely to be subjected to BCA, while bicycle and pedestrian1 and roadway rehabilitation projects are the least likely. State DOTs also use several alternatives to BCA, including life-cycle cost analysis and multifactor scoring systems. In comparing current questionnaire results against those from a 2005 study, there appears to be an increase in the number of States that conduct BCA at least occasionally, but otherwise no widespread changes.
Most State DOTs that conduct BCA focus on impact areas with relatively straightforward data, methodologies, and monetization factors. Safety impacts are among the most commonly quantified. More complex areas such as emissions and freight are often excluded, while impacts on equity and the human environment are generally regarded as very challenging to quantify. Quality concerns with State DOTs' BCAs are documented in the literature. Key issues include improper baselines, speculative benefits, including transfers as benefits, and a general lack of transparency and reproducibility. Several case studies highlighted that the “societal” framework of conventional BCA, which focuses on benefits accruing to all users and non-users and costs borne by society at large, may run counter to State governments' tendency to focus on their own constituents and expenditures, which can bias the results.
The accuracy of the traffic demand and other forecasts that underpin BCA has been a source of concern, and the literature suggests that accuracy has not improved over time. With limited exceptions, State DOTs generally do not review prior forecasts and estimates to assess their accuracy against actual conditions.
State DOTs face a variety of challenges when attempting to use BCA for decisionmaking, reflecting institutional, resource, and technical issues. Fundamentally, there is little institutional support within some State DOTs for conducting BCA in the first place, as it is not a requirement for most Federal-aid programs and existing prioritization methods may be perceived as working well. Indeed, some interviewees stated that their State DOT would be unlikely to move toward greater BCA use absent strong evidence that it works better than their current approach.
States moving toward enhanced data-driven prioritization seem equally likely to adopt a multicriteria scoring approach rather than BCA. In the case studies, State DOTs using a non-BCA scoring approach emphasized how transparent and objective it is. This is consistent with findings from the literature that BCA results may not be readily understood due to their use of monetization and discounting factors and other economic techniques.
In at least a few States, the State DOT is required by law or regulation to make decisions and prioritize projects according to specific criteria, and there are often other procedural requirements with the force of law, such as environmental reviews. Moreover, while Federal-aid funding has many fewer category-based restrictions than in the past, there is still somewhat of a focus on prioritization within rather than across categories. Strictly speaking, none of these factors preclude the use of BCA. However, they may greatly reduce the State DOT's interest in pursuing BCA, since the BCA results could not necessarily be used to make prioritization or other decisions.
The literature notes resource constraints with BCA, which the questionnaire and case studies confirmed. BCA and its associated data and modeling needs can strain agency budgets, staff time, and other resources; it also requires specialized expertise that may not be present within the organization.
In addition to these institutional and resource issues, technical challenges remain. In fact, respondents most frequently cited difficulties quantifying and monetizing benefits as challenges for State DOTs. According to published literature and case study interviews, this is especially true for freight, multimodal, and non-motorized projects due to lack of established methodologies and valuation methods.
Finally, a key challenge to the broader use of BCA among State DOTs is simply that factors other than net benefits continue to be important for decisionmakers at the State level. These include local economic impacts and economic development; equity (by region, mode of travel and/or program area); the degree of support from the public and other stakeholders; the ability to leverage external funding sources; and the likelihood of completing a project without delays.
State DOTs have indicated that additional training could help them address some of the technical issues and expertise gaps that were cited. Specific suggestions included the development of a “clearinghouse” of BCA-related resources and overall guidance document on the use of BCA, particularly with regard to the application of BCA across different project types. State DOTs would also welcome technical assistance on specific methodological issues such as travel time reliability, and on incorporating new research findings into BCA practices, such as research aimed at developing estimates and methodologies for quantifying the benefits associated with project impacts that have not traditionally been included in BCA.
At the institutional level, many agencies noted the overall movement toward performance-based, data-driven planning as required by recent legislation. Some agencies would like to see more assistance with this transition, which goes beyond BCA but could include a BCA component.
On the broader question of improving the overall use of BCA and its acceptance in decisionmaking, interviewees had fewer concrete suggestions. The literature suggests that BCA may gain traction when decisionmakers are more informed about how it works and have more confidence in its findings. This could include outreach and communication generally, or specific approaches such as conducting retrospective studies to improve modeling accuracy and thereby foster greater stakeholder confidence in BCA. Other suggestions from the literature include producing more streamlined, readable BCA summaries to aid policymakers, and applying BCA tools to estimate impacts in other areas that influence decisionmaking, such as equity.
This study responds to requirements in Senate Report 113-182, which directs the U.S. Department of Transportation (DOT) to “evaluate the use of benefit cost analysis by State departments of transportation, and to issue a report to the House and Senate Committees on Appropriations.” This report addresses the four topic areas listed in Senate Report 113-182, namely “the extent to which State departments of transportation use benefit cost analysis when making decisions and setting priorities, the quality of such analysis, challenges that State departments of transportation face when trying to use benefit cost analysis, and strategies for addressing those challenges.” The full text of the Senate Report is included here as Appendix D.
For the purposes of this report, research focused primarily on State departments of transportation (State DOT) and on projects under the Federal-aid highway program, as distinct from other State DOT responsibilities such as aviation and rail programs. The study is also focused on benefit cost analysis as defined below, though related concepts such as life-cycle cost analysis are addressed to some extent where relevant.
Benefit cost analysis (BCA) was defined in this study as a systematic process by which the impacts of a project (or other action) are forecast and quantified, so that societal benefits can be compared to costs for the project or a range of alternatives. A BCA typically converts estimated impacts into monetary equivalents, and converts future values to present values using a discounting formula. By contrast, cost-effectiveness analysis and life-cycle cost analysis are tools to estimate the most cost-effective option to achieve a given level of benefits or outcomes. These analyses are related to BCA but do not estimate benefits. Economic impact analysis and similar approaches differ from BCA because they estimate the impacts that a project will have on employment and the regional economy, rather than overall societal benefits and costs.
A BCA is a powerful analytical tool that has been applied to many areas of policymaking and public works. As noted in Senate Report 113-182, “Benefit-cost analysis is an important economic tool that can help State and local governments target their transportation funding to the most effective investments. Using benefit cost analysis, a State or local government would compare the monetary value of all benefits and costs that accrue during the life of a project. This process forces the government to evaluate the value of all of the project's benefits, recognize the full cost of the project, and acknowledge whether or not the benefits outweigh the costs.”
This study has some important limitations and is intended as an overview of the topic. In particular, it uses a qualitative, case study approach rather than a statistically valid sample of State DOTs. Furthermore, analysis of the “quality” of State DOTs' BCA work was based not on direct reviews of their products, but instead on information from the published literature and information from case study interviews. Additional methodology information appears below.
This study uses a qualitative approach, combining a literature review of existing research on this topic with case studies of nine State DOTs that provide more in-depth information about how BCA is used at the State level. In addition, a questionnaire was sent to all FHWA Division Offices to gather information about the use of BCA among each office's State DOT counterparts. A RFI was also posted to solicit input from consultants and others who may be familiar with State practices. Findings from each of these sources are presented in the report, and are synthesized in the final section to offer overall conclusions.
For the literature review, the study team used keyword searches to identify articles from academic journals, the broader transportation industry literature, and existing government reports that provided information on State-level use of BCA, the quality of the analysis, challenges, and potential strategies. The team reviewed articles for their relevance to this study and summarized key points.
The RFI, a copy of which is included here as Appendix C, was posted on a Federal procurement Web site for roughly 5 weeks (June 11 to July 15, 2015) and received responses from three organizations, all of which were research or consulting firms involved in BCA and related work. Information from the RFI responses is folded into the literature review summary below given the similarity in material.
To gather further information on each State DOT's current use of BCA, the study team prepared a short, internet-based questionnaire for completion by all FHWA Division Offices. Each Division Office was asked about the BCA-related practices of their State DOT counterpart. (FHWA Divisions work closely with State DOTs and have a good understanding of their respective States' practices. This approach was chosen to reduce burdens on State DOT staff and expedite the overall timeframe of the study. A direct survey of States would have required an Information Collection Request under the Paperwork Reduction Act.)
The survey was available online from August 18 to September 14, 2015. Of the 52 Division Offices (50 States plus the District of Columbia and Puerto Rico), 46 completed the questionnaire. Survey questions focused on the extent to which BCA is used overall by the State DOT; the types of projects and phases of the planning process for which it is used; the influence of BCA results on decisionmaking; elements included in the BCA; and barriers, challenges, and strategies. Two of the questions – those on overall use and influence – were closely modeled on the questions used in a 2005 GAO (GAO) study of States' practices,2 to allow for limited comparisons across time. Questionnaire responses are summarized below, and a copy of the full questionnaire is included as Appendix B.
This section summarizes overall findings from the literature review. These findings were complemented by information received in the RFI responses in a few areas, where noted.
Although there is an extensive literature on BCA generally, and on its use within the transportation field, there is relatively little that addresses specific questions of whether and how BCA is used by State DOTs. One of the few sources in this area is a 2005 report from the GAO, which looked at BCA for highway and transit projects, in the context of understanding how Federal-aid funds are invested.3 A study on engineering economic analysis from the National Highway Cooperative Research Program4 provided additional detail on State-level practices and challenges. A report from the Pew-MacArthur Foundation in 2013 examined the use of BCA at the State level across all program areas, helping to put the highway-specific findings in a broader context, particularly with regard to challenges and strategies.5 Many of the findings below are drawn from this group of key resources, though other studies shed light on specific issues such as forecasting uncertainty and institutional challenges. An annotated bibliography of other key studies reviewed is included as Appendix A.
The study that most directly addresses the extent of BCA use by State DOTs was performed by the GAO in 2005.6 The GAO's review found that while many State DOTs consider benefits and costs when evaluating alternatives, this is usually not done in a formal, quantitative way through BCA. Of 40 State DOTs responding to GAO's survey, only 12 reported conducting BCA more than half the time for highway expansion projects.
The GAO also found that even when a BCA was completed, the benefit-cost results were not the most important factor considered in decisionmaking and project selection, but were just one of many. According to the State DOT respondents in the GAO study, other factors such as political support, public opinion, and availability of funding all have a much greater influence on project selection than BCA results. A number of States have taken the lead and are using BCA more widely, including California, Minnesota, and Washington.7 However, the actual influence of BCA may be limited even in these States. For example, GAO found through regression analysis of project data provided by California that BCA results were a relatively weak predictor of the likelihood of project selection. Overall, it appears that the use of BCA by State DOTs has not yet matured or become an essential component of their planning process. Similarly, the proceedings of an FHWA-sponsored conference in 1995 noted that while there was longstanding and growing use of BCA at the Federal level, there were few examples of successful use of BCA by state DOTs at that time.8
Although the use of BCA itself is not widespread, State DOTs are increasingly using quantitative methods to understand the impacts of projects and set priorities. A recent NCHRP study on engineering economic analysis9 noted that life-cycle cost analysis (LCCA) has become common practice for prioritizing certain categories of investment, such as pavement management, asset management, and bridge preservation. A 2013 GAO study in this area found that of 16 State DOTs studied, 13 were using LCCA for pavement management.10 The LCCA appears to be more prevalent than BCA because it does not require quantification of user benefits.
One of the advantages of a BCA approach to prioritization is that project benefits and costs are rendered into a consistent unit of measurement, usually dollars. In the absence of this BCA framework, State DOTs have typically prioritized projects within categories and programs rather than attempt a more comprehensive prioritization. A recent report sponsored by the American Association of State Highway and Transportation Officials (AASHTO) on cross-mode project prioritization11 suggests that a multifactor scoring and weighting approach, rather than BCA, is growing in popularity among State DOTs seeking to prioritize across modes and programs. Some form of BCA is commonly included with these approaches, but it is not necessarily a formal, quantitative BCA, nor is it the most heavily weighted factor.
According to GAO, Federal requirements do have a significant effect on BCA adoption among State DOTs. Transit projects are more likely than highway projects to have an associated BCA because of the requirements of some Federal transit funding programs. There is no such requirement for highway projects, aside from discretionary grants such as TIGER.12 According to information received in the RFI and DOT records, almost all States have used BCA as part of a TIGER grant submission; this again highlights the potential influence of Federal requirements. At the same time, several States have laws that require the consideration of specific criteria in highway project selection, such as safety and environmental impact;13 there are also Federal requirements for environmental review. While these factors can be incorporated into a BCA, at least in principle, these specific statutory or policy requirements may reduce the perceived need or value for BCA and/or increase the complexity of implementing a BCA-based approach. It is also worth noting that State DOTs' limited use of BCA appears to be typical of State programs in general, rather than something that is specific to transportation. A 2013 Pew-MacArthur Results First Initiative study on the use of BCA in State government showed that while BCA use is increasing somewhat, it is not yet being mainstreamed into States’ decisionmaking processes. The study also found that other factors are more important than BCA results in making investment decisions.14
Although “quality” is to some extent a subjective term, the BCA literature and transportation economics community generally regard high-quality BCA as having as many of the following characteristics as possible:
The FHWA does not conduct assessments of State DOTs' BCA products or processes, except in connection to discretionary grant programs such as TIGER grants, which are discussed below. Some information on the quality of State DOTs' benefit-cost analyses was also available in the literature, as well as in responses to the RFI.
Overall, while the quality of BCAs varies from State to State and project to project, the literature describes a number of deficiencies commonly found in States' analyses.16 According to GAO, issues that are frequently encountered in States' BCAs include erroneously including economic development impacts or construction costs as benefits; double-counting benefits; omitting certain categories of impacts; not discounting future values correctly; using unrealistic base cases; and failing to include reference to other viable alternatives.
Past TIGER grant submissions have provided some insight into States' capabilities to conduct BCA. Note, however, that the BCAs produced for TIGER grants may not be representative of State DOTs' typical practices, in part because a number of State DOTs only produce full-fledged BCAs where required for TIGER and similar programs, and in part because State DOTs are instructed to use DOT's TIGER program guidance rather than their own methodologies and policies. Homan (2014) also notes that the competitive nature of TIGER grants may create a bias toward showing high net benefits,17 a factor which may not be present for in-house BCAs. Reviews of TIGER BCAs conducted by GAO and FHWA have found that these BCAs raise some of the same concerns as noted above.18 At least in the earlier rounds of TIGER funding, it was not uncommon for the BCAs supporting TIGER applications to include relatively basic errors such as double-counting both local economic impacts (e.g., employment) and direct impacts to users as benefits, as well as incorrect handling of inflation and discounting. Other common errors in submissions are inappropriate baselines, lack of consideration for other project alternatives, unsupported assertions about project impacts, and a general lack of transparency and documentation.19
A separate GAO study on the use of life-cycle cost analysis noted that most State DOTs using LCCA for pavement management do not include any sort of probabilistic analysis or treatment of uncertainty.20 Practices also varied on several key dimensions; for example, several States do not include user costs in their calculations, only direct agency costs, making the analysis less than comprehensive. In addition, a wide range of analytical time periods and discount rates are used, though GAO noted that some of the variation may be appropriate given the range of State experiences and conditions.
Responses to the RFI all indicated that the quality of BCA varies from State to State and project to project. One submitter noted that these BCAs are generally not comprehensive, as they only consider one or two categories of impacts that can be more readily modeled and monetized, such as travel time savings. Another noted that most BCAs do not provide formal treatment of uncertainty.
State DOTs face significant challenges in using BCA to make highway investment decisions; some are specific to transportation and some apply more broadly to State government programs. The Pew-MacArthur study found that resource issues were often a key impediment for States, as BCA can require significant staff time, expertise, and budget. The BCA studies may also require more preparation time than the decisionmaking cycle affords.
Furthermore, there are widespread misunderstandings of what BCA is and how it can be used, such that State-level decisionmakers often do not understand the methodology or findings of a BCA, or do not trust that the results are impartial. Even when a BCA is conducted, it may have little bearing on policy and program decisions, as many of these decisions are made for ideological or other reasons.21 To some extent this may be due to a lack of familiarity with BCA, yet similar observations have been made in countries where BCA is widely used as part of the public investment process, such as the Netherlands.22
The BCA results may also be ignored because the analysis of overall long-term outcomes is at variance with decisionmakers' focus on near-term impacts.23 Similarly, Boardman et al. observe that practitioners may avoid BCA precisely because those results may be viewed skeptically, both by proponents of a project who are concerned that potentially negative results could undermine its support, and by advocates for limiting public expenditures, who are concerned that consistently positive BCA results could lead to such projects being overfunded. In these situations, BCA may be avoided, or may be subject to claims of bias and partiality that affect the results.24
Moving from institutional to technical issues, limited data availability and the limitations of forecasting models continue to present challenges for BCA. This issue is not limited to transportation—the Pew-MacArthur study also cited data and modeling limitations more broadly across State government—but there are some specific issues that have received attention in the literature, particularly with respect to travel demand models and cost estimation.
Among these technical challenges, one key issue is the ability to forecast future costs and benefits at a project level. Most BCAs that include mobility benefits use projections from four-step regional travel demand models to measure impacts against a no-build case or similar baseline. The GAO found that forecasting future highway usage and other aspects of transportation demand is subject to significant error.25 The specific shortcomings of these models are described in more detail in the transportation literature; in particular, these models have limited capability to forecast driver response to changes in capacity, congestion and tolls with facility- or project-level precision. Another modeling challenge relates to incorporating long-term impacts such as land use changes, which have strong influences on travel demand but are difficult to predict. (Land use planning and zoning are also beyond the purview of transportation agencies in most regions.) One should note, however, that such technical challenges are not unique to BCA, and extend to any analysis that relies on projections of future system usage and operational performance, such as for environmental analyses and for prioritization schemes.
Regional models are designed to inform broad policy and investment decisions, based on analyses at an aggregate, regional level. While regional travel models may be one of the best tools currently available to estimate impacts associated with individual projects, they are typically not designed for this purpose and the results are therefore subject to significant limitations, including the level of error that is inherent to the model. A review of the accuracy of traffic forecasts for 210 transportation projects across 14 countries found that one-half of road projects had actual traffic levels that ranged from 20 percent higher to 20 percent lower than originally forecast.26 That study did not find a systematic bias in highway projects, but did note a general pattern of optimism bias, in particular for demand forecasts conducted as part of project funding requests; this is consistent with other studies that have found that project costs tend to be underestimated and demand overestimated.27 Overall, modeling limitations continue to present an important technical challenge for the use of BCA.
Another commonly cited technical issue is the difficulty in estimating and monetizing certain impacts, particularly environmental impacts and non-user benefits. An AASHTO report supported these findings and added that projects involving freight and non-motorized transportation are viewed as especially challenging. The GAO likewise noted that freight impacts can be difficult to estimate and monetize because of the importance of travel time reliability and the existence of broader supply-chain impacts.28
Perhaps as a result of these challenges, States seeking a data-driven approach for optimizing across categories have tended to opt for a scoring and weighting approach instead of BCA.29 One RFI respondent noted that States prefer this approach because of the challenge of applying BCA to project impacts that do not have well-established monetary values or methodologies.
Finally, it is important to recognize that restrictions on the uses of transportation funding can also serve as a disincentive to use BCA. The AASHTO and other sources noted that because certain forms of Federal and State funding have (at least until recently) been restricted to their designated category, State DOTs have tended to use a “programmatic category” approach to selecting projects, rather than using BCA to optimize their spending across categories. While BCA could still be used to optimize within categories, DOTs have typically preferred to use engineering measures to rank projects within categories, such as bridge ratings and pavement ratings, perhaps since these are better understood and more analytically consistent with traditional practice within the engineering community.
The literature notes many strategies to increase State DOTs' use of BCA in their decisionmaking. To address the limited interest in and understanding of BCA, one approach is to demystify the BCA framework and demonstrate its value to decisionmakers. The Pew-MacArthur study, for example, suggested that improved outreach to decisionmakers (e.g. through briefings for agency officials and legislators) could help ensure that they understand BCA, and that, in turn, greater confidence in the process could increase its acceptance and use.30 Boardman similarly cites the need to better explain the conceptual framework behind BCA to decisionmakers who may otherwise apply alternative lenses.31
Another strategy involves increasing the perceived relevance of BCA to decisionmakers. For instance, several sources note that conventional BCA typically has little or no analysis of distributional or equity impacts, even though these are key considerations for decision makers.32 Expanding the scope of BCA to address and potentially estimate these impacts could improve its relevance for decision-makers.
The GAO cited the need to present BCA information in a way that is more useful to decisionmakers, particularly with regard to documentation and discussion of project risks. Pew-MacArthur also suggested that providing concise summaries of BCA reports would make the findings more relevant and useful to decisionmakers. They also noted that adopting a replicable BCA model, rather than conducting each analysis from scratch, can improve the timeliness and thus the value of the BCA results.
Other studies recommended advancing the technical rigor of BCA and the models that provide data for BCA to yield greater confidence in the results. A technical area most commonly cited is the need to improve the travel demand models that underpin many BCAs. In particular, the models have limited ability to forecast land-use changes, and would benefit from refinement in predicting mode choice and other aspects of travel patterns. This may be particularly true in today's context of shifting demographics, volatile fuel prices, and to some extent new transportation offerings (e.g., HOT lanes, bike-sharing, car-sharing, telecommuting, etc.)
To address both technical and institutional issues, GAO, NCHRP, and Pew-MacArthur all cite the need for more studies of forecast-versus-actual project impacts and costs. These studies could provide more data on the specific projects in question, while also refining forecasting and BCA tools for future projects. This, in turn, could engender greater confidence in BCA and increase overall acceptance of the method.
A few sources suggested that the Federal Government offer training and outreach to State DOTs to improve their technical capabilities in BCA.33 The NCHRP noted that a large number of guidance documents and online resources already exist;34 however, this information is either not reaching staff at the State DOTs, or the States are using their own practices, perhaps tailored to their circumstances. State DOTs could potentially benefit from outreach materials that further define BCA and explain its pros and cons relative to other forms of analysis and decision-support tools.
More specific technical assistance on BCA-related issues, such as recommended monetary values and technical approaches, could also be beneficial. As an example, DOT's recent TIGER grant guidance provides not only recommended monetary values for injuries at different severity levels, but also a conversion matrix between the Abbreviated Injury Scale (AIS) and the KABCO scale.35 This is a small but important piece of practical information, as it allows applicants to apply DOT's monetary values, which use the AIS scale, to State-level crash data, which is typically collected using the KABCO scale.
As a step beyond outreach and technical assistance, GAO suggested that States could be encouraged to broaden their use of BCA and to improve their technical rigor through Federal incentives or mandates. The GAO emphasized, however, that any such approach would have to be carefully considered and managed, as some potential approaches would require substantial Federal resources for oversight and/or significant policy or legislative changes.
Several survey questions dealt with State DOTs' use of BCA. In the first of these, respondents were asked about the overall frequency of use. The question phrasing and answer choices were designed to align as much as possible with a similar question used in GAO's 2005 study, with modifications to reflect the scope of the current study. All 46 respondents answered this question and none replied “don't know.” Overall results for this question are presented in Table 1.
Thinking in general about your State DOT's Federal-aid highway program, how often does the State DOT use Benefit-Cost Analysis (BCA) in making decisions and setting priorities?
|Answer Options||Response Percent||Response Count|
|Never (or almost never)||19.6%||9|
|More than never, but less than half the time||60.9%||28|
|About half the time||4.3%||2|
|More than half the time, but not always||4.3%||2|
|Always (or almost always)||10.9%||5|
In general, results from this question indicate that usage of BCA varies significantly across States, but that in general it is not a systematic practice for decisionmaking at the State DOT level. Only seven respondents indicated that their corresponding State DOT used BCA more than about half the time.
For comparison, results from the 2005 GAO study are shown below in Table 2. That question was posed to State DOT staff directly and asked about “analyses of costs and benefits conducted when comparing alternatives in planning and developing highway projects.” Although the two surveys are not strictly comparable due to slight differences in phrasing and the different set of respondents, they do provide some indication of the changes that have taken place over the past decade.
How often does your agency complete a Benefit Cost Analysis when evaluating alternatives for proposed highway/transit capacity-adding projects?
|Answer Options||Response Percent||Response Count|
|Never (or almost never)||30.0%||12|
|More than never, but less than half the time||32.5%||13|
|About half the time||7.5%||3|
|More than half the time, but not always||12.5%||5|
|Always (or almost always)||17.5%||7|
The most salient finding from a comparison of the two surveys is that the percentage of respondents answering that BCA is never used has fallen by 10 percentage points. These BCA new users appear to have moved primarily into the next category of “More than never, but less than half the time,” which grew by 28 percentage points between the two surveys. These results indicate that while more State DOTs are using BCA to some extent for decisionmaking and project prioritization in 2015 than 2004, the difference lies mainly in infrequent use. Several of the 2015 survey respondents mentioned the use of BCA in connection with applications for Federal TIGER grants, for which a BCA has been required, so it is possible that the advent of that program in 2009 is responsible for some of the reported changes. However, other factors may have also played a role, including a more general emphasis on performance-based management.
The next question in the 2015 survey was a follow-up that asked about the specific stages of the decisionmaking process where BCA may be used or not used. (There are 37 respondents here because those that selected “Never” for the previous question were instructed to skip certain follow-up questions.) Table 3 presents these responses. Looking at the overall pattern of responses, project prioritization and programming/State Transportation Improvement Program (STIP) formulation stand out as the stages for which BCA is more typically used. By contrast, BCA appears to be less common as part of higher-level decisionmaking, such as overall resource allocation and long-range plans. This pattern suggests that State DOTs prefer to use BCA for planning stages that involve discrete near-term choices, rather than broader program and policy decisions.
For each of the following stages, please indicate how often your State DOT uses BCA as part of its decisionmaking process.
|Answer Options||Never (or almost never)||More than never, but less than half the time||About half the time||More than half the time, but not always||Always (or almost always)||Don't know||Response Count|
|Policy/program level or overall resource allocation||16||9||3||4||2||3||37|
|Long-range transportation plans||18||8||1||3||4||3||37|
|Operations and maintenance||14||8||4||4||3||4||37|
The next follow-up question addressed the frequency with which State DOTs use BCA for different types of projects; responses are summarized in Table 4. Among State DOTs at least occasionally using BCA, the most common project types for which BCA is applied are safety projects and large or significant projects. Projects requiring an environmental impact statement (EIS) were also listed by a majority of respondents. Very few respondents indicated that their State DOT counterparts use BCA for asset preservation projects or TAP projects, which are often pedestrian and bicycle facilities. This pattern suggests that BCA may be viewed as more useful for larger projects, for which more stakeholder scrutiny may be expected, but also that analytical challenges may play a role, since safety projects tend to be more readily quantifiable in their impacts than asset preservation or bike-pedestrian projects. These issues were addressed to some extent in the follow-up interviews with State DOTs, which are discussed in the Case Studies section below.
For each of the following specific types of projects, please indicate whether or not your State DOT typically (i.e., more often than not) uses BCA:
|Answer Options||Yes||No||Don't Know||Response Count|
|Large or significant projects (as defined by the State DOT)||26||10||2||38|
|Projects requiring an environmental impact statement||19||16||3||38|
|Discretionary or competition-based projects||10||21||6||37|
|Projects expanding capacity||14||17||6||37|
|Asset preservation projects (3R)||9||25||3||37|
An additional follow-up question asked about the impact of BCA on the project recommendations that the State DOTs eventually make; the question and its answer choices were modeled on a similar question from the GAO study. Responses are shown in Table 5 and generally indicate that when BCA is conducted, the results have some weight in overall decisionmaking, but only in a few States are they a highly influential factor.
In your opinion, how much importance do BCA results typically have in your State DOT's decisions to recommend a project from among its various alternatives?
|Answer Options||Response Percent||Response Count|
|Very little to no importance (or BCA not conducted)||8.1%||3|
|Very great importance||10.8%||4|
Table 6: Responses from GAO (2005)
Typically, how much importance would you say that the ratio of benefits to costs has in your decision to recommend a project from among its various alternatives?
|Answer Options||Response Percent||Response Count|
|Very little to no importance (or BCA not conducted)||4.7%||2|
|Very great importance||0.0%||0|
Results on this question from GAO's 2005 study are reproduced in Table 6. Comparing responses across the two surveys, there were many fewer responses of “moderate” influence in 2015 compared to 2005, and relatively more at both extremes. It is unclear whether these differences reflect changes since 2005, or are simply artifacts of the differences in survey administration and respondent population.
The next survey question moved beyond BCA to ask about other types of quantitative analysis that State DOTs may use for decisionmaking and setting priorities. Responses are summarized in Table 7; note that responses exceed 100 percent since some many State DOTs are using multiple types of analysis.
Other than BCA, or in addition to BCA, what forms of quantitative analysis does your State DOT typically use for making decisions and setting priorities?
|Answer Options||Response Percent||Response Count|
|Cost-effectiveness analysis or life-cycle cost analysis||85.7%||36|
|Economic impact analysis||40.5%||17|
|Multi-criteria scoring and weighting||42.9%||18|
|Other prioritization/quantitative methods||21.4%||9|
These responses indicate that a majority of the State DOTs are using cost-effectiveness analysis or life-cycle cost analysis, which is consistent with the literature on asset management systems. A significant minority are also using scoring and weighting systems, a trend that was explored further in some of the case study interviews below. Of the respondents selecting the “Other” category, many cited specific tools or software packages.
The questionnaire did not ask directly about the quality of the BCA work conducted by their State DOT counterparts due to limitations of the questionnaire format and concerns that this information would not be available to the respondent. (The study team addressed issues related to BCA quality primarily through the case study interviews.) However, one question asked about the elements that the State DOT typically includes in its BCAs, which shines at least some light on elements of quality that relate to comprehensiveness. Responses are summarized in Table 8 and, not surprisingly, indicate that impacts such as safety improvements, and to a lesser extent travel time savings, are more frequently included than difficult-to-quantify impacts such as community inclusion, supply-chain impacts, or environmental justice. On the cost side, responses indicate that nearly all States include capital costs and most also include ongoing operations and maintenance costs; however, final disposition costs or residual asset value are generally not included.
Please identify whether your State DOT's BCAs typically (i.e., more often than not) include each of the following elements.
|Answer Options||Yes||No||Don't Know||Response Count|
|Estimated project capital costs||36||1||1||38|
|Estimated project operations and maintenance costs||25||9||3||37|
|Estimated asset disposition costs or residual value||9||20||8||37|
|Safety improvements – fatalities and injuries avoided||34||3||1||38|
|Travel time savings and/or reliability improvements||21||11||5||37|
|Savings in vehicle operating costs||13||17||7||37|
|Reduced noise, vehicle emissions and/or reduced greenhouse gases||15||15||7||37|
|Improved human environment or community inclusion||6||22||9||37|
|Improved access to markets||6||21||10||37|
|Improved intermodal connectivity||9||15||13||37|
|Freight or supply-chain benefits||6||19||12||37|
|Equity and/or environmental justice impacts||7||18||11||36|
|Other factors (please describe):||5|
The last question with descriptive statistics asked about the challenges that State DOTs face in using BCA for project selection and decisionmaking. Respondents were free to select as many responses as they felt applicable. Results are listed in Table 9.
In your opinion, what are the challenges that your State DOT faces in using BCA for project selection and decisionmaking?
|Answer Options||Response Percent||Response Count|
|Modeling limitations, e.g. limitations of travel demand models||44.4%||20|
|Difficulties with quantifying or monetizing benefits, in general or for certain types of projects||84.4%||38|
|Lack of funding for BCA studies||37.8%||17|
|Lack of staff expertise||64.4%||29|
|Lack of institutional support within State DOT||55.6%||25|
|Lack of political support||22.2%||10|
|Lack of support from general public||6.7%||3|
|Insufficient time in process to conduct BCA studies||40.0%||18|
|Preference for another prioritization process (e.g. multi-criteria scoring)||48.9%||22|
|Preference for having BCA performed at local/regional level, rather than at the State level||6.7%||3|
|Other (please describe below)||4.4%||2|
The top response was “difficulties with quantifying or monetizing benefits, in general or for certain types of projects,” which was selected by about 84 percent of respondents. Reports of this challenge appear consistent with the patterns seen on several earlier questions, including the widespread use of cost-effectiveness analysis, for which benefits are not monetized; the relatively low rates of BCA use for TAP and preservation projects; and the relatively infrequent inclusion of difficult-to-monetize benefits such as improved community inclusion.
The next group of responses included “lack of staff expertise,” “data limitations,” and “lack of institutional support within State DOT,” each of which was cited by over half of the respondents. Other common responses, though not quite garnering a majority, included lack of time and funding, modeling limitations, and a preference for an alternative prioritization process. Taken as a whole, the responses suggest that there are multiple impediments to broader use of BCA, both in institutional support for the process and in technical capabilities and resources. However, lack of political and public support was cited less frequently as a challenge, which could indicate that the use of BCA would be greater if the technical and resource issues could be addressed, for example through additional funding support or sharing of expertise.
Following the question about challenges, respondents were asked for strategies to help the State DOT address them. This question was structured as an open-ended written response. Although respondents' submissions varied, they primarily focused on the need for better education and resources. Several respondents recommended general education on the benefits of using BCA, to convince State DOTs and other stakeholders that BCA analysis is worthwhile. Others noted the need for education and training for staff on the skills and knowledge necessary to conduct BCA. Some respondents also suggested the development of standardized modeling tools that would support the use of BCA. Finally, a few respondents stated their belief that only a specific Federal requirement could institutionalize the use of BCA.
The final question on the survey asked for any additional information or feedback on the topic or on the questionnaire itself. Responses included a mix of additional State-specific information and clarification of earlier responses. One respondent noted that their State DOT counterparts are concerned about risk of litigation with BCA, but this was not explained further. Several respondents reiterated the need for outreach on BCA methods (e.g., webinars and case studies) to build institutional knowledge. Others emphasized that even where BCA is conducted, it serves as just one of many factors used for prioritization.
The case studies below are intended to complement the findings from the literature review and questionnaire with additional details and insight. In keeping with the qualitative approach, the case studies are not a representative sample, but instead were selected on the basis of their potential to highlight specific BCA-related issues, such as different approaches to BCA, some alternatives to BCA that are being employed, and relevant challenges that State DOTs face. The case studies are based largely on interviews with State DOT staff, plus limited review of documentation provided by the States. The corresponding FHWA Division Office provided the initial interview contact at each State DOT, and in many cases that contact invited other State DOT staff to join the discussion. (It is nonetheless possible that the interviews may not have covered all relevant BCA-related activities at the State DOT, for example if those activities are occurring in disparate parts of the organization.)
The Connecticut and Nevada case studies highlight State DOTs that are using BCA only for specific project categories—safety projects and large projects, respectively—while Minnesota DOT is a State that has systematically incorporated BCA into its decisionmaking process. In North Carolina, Utah, and Vermont, some form of weighting and scoring system, developed in response to State legislation, is the primary means of project prioritization. However, each of these States has a slightly different approach and outlook for their future use of BCA. Florida and Arkansas have hybrid approaches, using BCA to some extent but relying on other quantitative methods as their primary methods of project selection. Oregon DOT has a different sort of hybrid method, using asset management tools for preservation projects and a qualitative approach to prioritizing expansion projects, with a strong role for stakeholder input.
Connecticut DOT uses BCA extensively as part of its decisionmaking process, but only within one program area: safety-related projects. Connecticut DOT (CTDOT) conducts BCA for most of its planned projects under the Highway Safety Improvement Program. Although some updates have been made over time, CTDOT's basic methodology for safety BCAs has been in place for roughly 40 years, and is based on a research report originally produced for the Bureau of Public Roads.36 The overall approach is to gather historical crash data and then apply engineering-based estimates of the “crash reduction factor” associated with the project or countermeasure being analyzed. The resulting estimate of avoided crashes is then converted to monetary terms using standardized values for fatal, injury, and non-injury crashes. These monetary values are based on estimates from the National Safety Council on the costs of unintentional injury and are periodically updated. The resulting benefit estimates are summed across the project lifetime, with no discounting of future values, and compared against project costs (also undiscounted) to produce a benefit-cost ratio for each project. This represents a fairly limited application of BCA, since it includes only one category of benefits (safety) and does not include discounting.
The CTDOT performs most of the BCA work in-house using an Excel-based tool. The results inform the overall safety project selection process and ensure that projects being pursued do indeed have net benefits. However, the calculated benefit-cost ratio is not the only criterion; other potential impacts, such as congestion relief or quality-of-life benefits, are assessed qualitatively and are taken in consideration in overall project selection. The CTDOT also no longer conducts BCA for some projects included in FHWA's list of nine “proven safety countermeasures,”37 as their cost-effectiveness is considered as sufficiently proven by experience and does not need to be assessed on a project-by-project basis.
Aside from safety projects, CTDOT does not typically use BCA for project prioritization or other forms of decisionmaking. The current approach to project selection is described as a more informal process that balances funding availability, public support, and professional judgment. The CTDOT has been examining other potential approaches that would help to assess cost-effectiveness and compare alternatives in a more rigorous way, but also recognizes that there would be significant challenges in collecting the required data and developing new benefit estimation models in areas such as mobility and emissions, especially with limited available staff time. There would also need to be internal support for changing processes that have been in place for many years. At this stage, therefore, CTDOT has no plans to expand its BCA work into these areas.
The CTDOT reviews a sample of past projects each year, comparing the forecast safety benefits against observed changes in crash rates post-implementation. These reviews have generally found that actual crash reductions tend to be lower than forecast. The CTDOT is interested in using these findings to update their crash reduction factors and forecasts. Other priorities include updating their monetization factors and revising their methodology to more closely reflect guidance in AASHTO's Highway Safety Manual.38 The CTDOT staff noted that FHWA could play a role in this area by facilitating peer-to-peer exchanges across State DOTs, so that States can learn from each other’s practices in this area. In addition, FHWA could assist by promoting greater consistency of practices across the States.
Nevada DOT (NDOT) is required by State law (NRS 408.3195) to prepare a BCA for all projects that expand highway capacity and are expected to cost more than $25 million. This analysis must be prepared prior to final approval for funding from the NDOT Board of Directors. The statute requires the use of a present-value framework and lists factors that must be reflected in the BCA, including capital and operating costs, safety impacts, environmental impacts, and changes in vehicle operating costs and travel times.
To implement these requirements, NDOT has established a Benefit-Cost Analysis Policy (TP 1-11-1) that defines the BCA process in more detail and assigns roles and responsibilities. Under the policy, preparation of project-level BCAs is conducted as part of the statewide planning process and is designed to assist with prioritization of projects in the 20-year Long Range Plan, the 8-10 year Mid Range Plan, and the STIP. However, BCA results are only one factor in the prioritization. Other factors such as un-modeled environmental impacts; qualitative community, historical, or cultural impacts; national security requirements; and potential economic development and freight impacts are also considered. The degree of public support or opposition to the project can also be a factor. The NDOT must also ensure that selected projects are legally eligible for their intended funding source or category.
The NDOT's Performance Analysis division coordinates with other NDOT divisions on the projects that will require BCA, and develops an Annual BCA Plan to manage workflow. The annual plan will sometimes include projects under the $25 million threshold in cases where a BCA is viewed as useful for decisionmaking purposes.
Most of the BCA work is conducted under contracts with State universities due to staffing limitations at NDOT, though some work is done in-house. NDOT has established standardized procedures and monetary values to be used, including values for travel time savings, vehicle operating costs (fuel and other costs), crashes, and emissions. Values for these parameters were derived from a variety of sources, including the AASHTO Highway Safety Manual, the California Air Resources Board, and State-specific crash and VMT data. For injury prevention, NDOT uses the KABCO scale and a value per fatality of approximately $5.3 million, which is lower than current DOT guidance. A real discount rate of 7 percent is used, based on guidance from OMB Circular A-94, with sensitivity analysis of other rates.
Most of the analysis is conducted in a software tool that has been adapted from Caltrans' Cal-B/C model, with some modifications of default values to reflect Nevada conditions and data. The model produces outputs in four areas: travel times, safety, vehicle operating costs, and emissions. Inputs include forecasts of traffic volumes and speeds/congestion, which are in turn derived from the regional travel demand model of the relevant Regional Planning Commission or MPO, and safety-related data such as crash rates. The NDOT staff noted that the model cannot readily handle certain types of projects, such as bridge projects, pedestrian/bicycle projects, and “3R” projects (resurfacing, restoration, and rehabilitation). The BCAs for these types of projects may be calculated manually, though many are beyond the scope of the formal BCA requirement because they do not expand capacity.
Economic development impacts are recognized as a potential qualitative factor in NDOT's BCA policy, but are not included in the BCA calculations. The NDOT staff observed that estimating the impacts of a particular transportation project on overall employment and economic output is very challenging and goes beyond current modeling tools.
The NDOT has not reviewed past projects to assess the accuracy of its earlier forecasts or BCA estimates, in part due to staffing limitations, and in part because that is outside the scope of their responsibilities as defined in the BCA Policy document. They would like to incorporate this into the BCA process in the future.
The current BCA policy has been in place since at least 2011, and NDOT staff report that it has received little or no external criticism. Internally, one key concern is that their BCA model cannot estimate benefits for certain types of projects, making it less useful for decisionmaking. For example, as noted above, the model cannot estimate benefits for 3R projects, so it cannot be used to compare the relative merits of, say, a highway expansion project versus a pavement restoration project. Another issue is the time lag between the BCA and actual project implementation. When this time lag is substantial, a judgment call is needed on whether conditions have changed enough to warrant an updated analysis. Overall, NDOT is seeking to make their BCA approach more comprehensive through improved software, so that a greater range of project types and impact areas can be assessed.
Minnesota DOT (MnDOT) regularly uses BCA and is often cited as a national leader in the use of BCA. The MnDOT's most visible and institutionalized use of BCA is governed by a Cost-Effectiveness Policy in place since 2003. Under that policy, projects meeting certain thresholds—including all projects for which environmental review is required—must undergo a cost-effectiveness assessment during the Alternatives Analysis phase of project development. Other projects, such as maintenance and preservation, are evaluated using decision-support tools that also incorporate measures of cost-effectiveness, including life-cycle cost analysis for bridges and pavement.
As stated in the policy, the goal of this approach is to “make the best overall investment decisions based upon a balanced consideration of both the quantitative cost-effectiveness goals and qualitative goals.”39 The cost-effectiveness analysis is made up of three parts, the first of which is a conventional BCA using guidelines from AASHTO's User and Non-User Benefit Analysis for Highways40 and State-specific monetization and discount factors. The second step is a “best value assessment” of project alternatives, and the third step is a review of other impacts that may be more difficult to quantify.
During the first step, the BCA itself is usually prepared by a consultant as part of the broader environmental documentation (EA/EIS), with information from regional travel models, engineering factors and/or professional judgment. The MnDOT establishes the approach and reviews the consultants' work. The MnDOT also provides recommended values for travel time savings, vehicle operating costs, crashes avoided (by injury severity), and discount rates, which are updated annually. Values for travel time savings and crashes avoided are based on DOT guidance, with a current value of about $10.6 million per fatal crash avoided. The recommended discount rate is 1.7 percent, which is based on the real (inflation-adjusted) return on 30-year Treasury bonds.
If a project with a benefit-cost ratio of less than 1.0 is advanced, the Policy mandates a second step to establish further justification. This step includes looking at whether the alternative selected has the best benefit-cost ratio, ways to change the scope to improve the ratio and whether the project is part of a larger project with a better cost-benefit ratio. Decisions to advance projects with benefit-cost ratios less than one, which are quite rare, must be documented and require varying levels of managerial approval based on the actual score. The third step in the cost-effectiveness analysis looks at social, environmental and community goals and business impacts, addressing factors that are more difficult to assign a dollar value and incorporate into a benefit cost analysis.
The MnDOT staff noted that BCA is not used to allocate funding across program areas, nor is it the sole (or even primary) factor used to prioritize projects within each area. However, the BCA does serve as a procedural check to ensure that selected projects do provide net benefits, and that within each project, there is consideration of the most cost-effective options or alternatives.
The MnDOT continuously evaluates its use of BCA to determine whether and how to expand or change its application. As a result, MnDOT recently incorporated BCA into the application process for two competitive funding solicitations: the Corridor Investment Management Strategy Pilot Solicitation and the Transportation Economic Development Program. In these cases, the application asks for the data needed for MnDOT to complete a basic BCA, with just enough precision to allow the ranking of the project applications. For these programs, BCA thus has a greater role in ranking and prioritization of projects than with mainstream projects governed by the Cost-Effectiveness Policy.
The MnDOT has not been able to review past projects to assess the accuracy of its earlier forecasts or BCA estimates. The MnDOT staff noted that this is due to limited resources, and also that it may not be clear how to use such information to improve future analyses, as there are so many confounding factors for any given project's outcomes.
The Cost-Effectiveness Policy has been modified over time to exclude some smaller projects from the BCA requirement. Overall, it appears to be a well-established approach at MnDOT and is generally accepted. However, some State legislators and other stakeholders would like to see a greater role for BCA, while others feel that BCA does not adequately capture certain impacts, such as economic development.
North Carolina DOT (NCDOT) uses a multicriteria scoring approach rather than BCA to prioritize its capital projects and develop its STIP, though there are some BCA elements within the scoring system. The basics of this approach have been in place since 2009, with several rounds of refinements along the way.
The scoring system was developed as a means of moving toward a more transparent, data-driven decisionmaking process that better aligned project selection with NCDOT's long-term goals. Initially, it was used only for highway projects, with three criteria: congestion, safety, and pavement conditions. In 2011, the prioritization system was expanded to include bicycle and pedestrian projects, and to include economic competitiveness and BCA as additional criteria. At this stage, the scoring system was non-binding, and other factors such as regional balance were also considered in developing the final plan.
In 2013, the North Carolina legislature passed the Strategic Transportation Investment (STI) law, which required NCDOT to prioritize projects according to a quantitative scoring system, for all six of the agency's modes: aviation, highway, pedestrian/bicycle, ferry, rail, and transit. (Certain project categories, such as CMAQ projects and discretionary grants, were excluded from this requirement. Interstate maintenance, bridge replacement, and highway safety improvement projects were also treated separately in the legislation and are allocated according to an alternative set of criteria.)
According to the STI law, “statewide mobility” projects receive 40 percent of available funds, with prioritization decisions made strictly according to the scoring system. Projects in the “regional impact” and “division needs” categories, each of which receives 30 percent of available funds, are prioritized based on a mixture of scoring and local input.
A stakeholder working group, including representatives from MPOs, RPOs, and local governments, was established to develop recommendations for implementing the STI requirements, including the development of criteria for the other modes and the overall scoring and weighting system. The working group's conclusion was that there was no practical means to compare projects across modes using a quantitative approach, so the group instead recommended that projects be prioritized within their modes. At present, 90 percent of funding is reserved for highway projects and 4 percent for non-highway projects, with the remaining 6 percent eligible for either category.
Data collection and scoring are done largely in-house by NCDOT, with IT contractor support. Based on a consultant's review, NCDOT is also implementing some refinements to the process, including the normalization of raw scores to limit the influence of outliers and ensure that each category has its intended weight.
The scoring system for highway projects in the “statewide mobility” category considers congestion, benefit-cost, economic competitiveness, safety, and multimodal/freight/military impacts. The benefit-cost component is not a conventional BCA, but rather a monetized measure of the estimated travel time savings and safety improvements over a 10-year period, divided by NCDOT's share of project costs. (Costs to be contributed by local governments, private sector partners, or other sources are not included in NCDOT's project costs; projects with external funding thus have a higher score on this metric.) Travel time savings are estimated by comparing the build versus no-build outcomes in the statewide travel demand model, with travel time savings valued at roughly $12.75 per hour for automobiles and $75 per hour for trucks. Other impacts such as changes in vehicle operating costs and emissions are not estimated. Safety improvements are estimated by applying safety benefit factors (akin to crash reduction factors) to recent crash statistics for the relevant location or corridor and multiplying the annual total by 10. Injuries are assessed on the KABCO scale and are valued at roughly $4.5 million for K/A injuries (fatal/serious), $117,000 for B/C (moderate/minor), and $6,700 for property damage only crashes. The 10-year stream of travel time savings and safety benefits is simply summed and is not discounted to present value. There is also no adjustment for residual asset value at the end of the 10-year period.
Although NCDOT's treatment of the benefit-cost component diverges from standard BCA practice in several ways (e.g., non-comprehensive benefits and costs, lack of discounting) it should be noted that this element is not being used as a BCA per se, but rather as a ratio that is then scored and entered into the overall multifactor process. The NCDOT noted a goal of scoring projects consistently and expressed interest in improving the sophistication of its analysis over time, but is still working out the details of this relatively new approach. Indeed, this is the first year that safety benefits have been considered alongside travel time savings in the BCA component. There are no current plans to expand the BCA component to include other impacts.
The NCDOT does not typically re-visit past projects to examine the accuracy of its forecasts and associated project scores, though that is a possibility that has been discussed as a future enhancement once more years of data are available. In addition to the inherent challenges of forecasting future demand in a fast-growing State like North Carolina, there is also a long time lag (5 to 9 years) between project prioritization and the opening of a facility.
Overall, NCDOT feels that their prioritization system is working well; it has received positive feedback and little resistance or pushback. The NCDOT noted that including stakeholders early in the process to help develop the criteria and scoring system has increased acceptance of the approach. Several refinements have been made over time and others are planned, such as purchasing third-party traffic data to improve modeling precision, using a microsimulation tool to estimate impacts for intersection improvement projects, and potentially developing a methodology to prioritize across modes. However, in the view of NCDOT staff, it is important to start with a relatively simple, transparent system that builds stakeholder acceptance before attempting to build in more sophistication.
The NCDOT staff noted the theoretical possibility of using a conventional BCA approach to prioritize across all modes based on monetized net benefits, but argued that benefit-cost considerations are only one factor among many. In particular, they are concerned that, unless the BCA is truly comprehensive, particular project types could dominate the selection process at the expense of other projects with widespread support. Moreover, NCDOT is specifically required by the STI law to consider other quantitative factors, at least for highway projects, which could be difficult to reconcile with an exclusively BCA-based approach.
A recent study by the Pew-MacArthur Results First Initiative identified Utah as a State that is “leading the way” in the use of BCA to inform policy and program decisions at State agencies. However, Utah DOT (UDOT) relies primarily on a multicriteria scoring system rather than BCA in its prioritization of highway capital projects.
At UDOT, routine maintenance projects are prioritized using an asset management tool that forecasts pavement condition and incorporates life-cycle cost analysis. The UDOT headquarters staff use this tool to make recommendations to the regional offices.
For expansion projects, State law requires project prioritization to be conducted using a quantitative scoring system. The UDOT developed the specific metrics and weights to be used within the guidelines of the legislation. Although this process does incorporate some BCA-type metrics, the focus is primarily on traffic and safety factors.
Expansion projects are divided into six categories for the purposes of the scoring system: widening, new facilities, intersection upgrades, new interchanges, upgraded interchanges, and passing lanes. Projects are scored and ranked within each of these categories, using criteria that vary somewhat across each of the six. As an example, projects to widen an existing highway are assessed using quantitative scores on traffic volumes, truck volumes, functional class, a congestion index (volume-to-capacity), a safety index, and forecast traffic growth.
For three of the six project types—intersection upgrades, new interchanges, and upgraded interchanges—a calculation of benefit-cost is also included in the scoring, and comprises 25 percent, 35 percent, and 25 percent (respectively) of the final project prioritization score. This metric, while labeled a benefit-cost ratio for purposes of the prioritization process, is conceptually more akin to an estimate of cost-effectiveness or a congestion relief score, as it is based on a comparison of non-monetized travel time savings to project costs.
The rankings produced by the prioritization system are not strictly binding, and UDOT will sometimes exercise discretion in selecting lower-scored projects based on engineering judgment, financial considerations, modal balance, or other factors that may not have been fully captured. The resulting project lists as developed by UDOT are given strong weight by the Utah Transportation Commission in its approval of projects for the STIP. The UDOT staff believe that this illustrates the confidence that the Commission has in the prioritization process.
The UDOT noted that one drawback of the current prioritization system is that it does not permit comparison across project types. For this purpose they are exploring the use of Decision Lens, which is a cross-asset allocation model that allows for project prioritization between different categories. Another area of potential future work is the incorporation of measures of economic impacts into project prioritization. The UDOT has two current projects in that area under the second Strategic Highway Research Program (SHRP2), which is a federally sponsored cooperative research program.
In their interview, UDOT staff cited a lack of precedent within the department as well as a preference for alternative methods of prioritization as reasons for their relatively limited use of BCA. The UDOT does, however, conduct BCA for TIGER grants and other situations where it is required. The UDOT staff can also envision using BCA more in the future as funding decisions potentially become more difficult and additional sophistication is required.
In Vermont, State law requires the use of a standardized scoring system for transportation projects, but leaves some leeway for the Vermont Agency of Transportation (VTrans) to develop specific measures and weights. Under the current approach, funding is initially allocated across categories (e.g., pavement, bridges) and then projects are ranked within each category. The scoring criteria vary by project type but typically include a mix of engineering metrics, such as asset condition and expected usage, as well as a factor for input from the State's Regional Planning Commissions.
For pavement projects specifically, VTrans also uses an asset management system, supported by Deighton's DTIMS software, that helps identify the most cost-effective projects. That system has a “benefit-cost” component that considers traffic volumes on the affected roads, but does not go as far as estimating or monetizing highway user benefits. The VTrans would like to expand this approach to include bridges.
The VTrans does not conduct retrospective studies on the results of its scoring exercises to assess whether project outcomes aligned with predictions. However, it does regularly examine its pavement deterioration model to assess its accuracy. If discrepancies are found, the model is adjusted with the new data to improve its accuracy.
The VTrans is planning to update its prioritization systems in late 2015 or early 2016, with the goal of becoming more data-driven and aligned with the agency's strategic plan. The BCA could become part of the new approach, though community engagement and external stakeholder input are also important factors. A performance-based planning process with optimized budgets would also require internal coordination of agency priorities and goals with respect to safety, mobility, and other outcomes. The VTrans has also considered including measurements of economic development potential as part of its project assessment, as this can be an important area of project impacts, though staff noted that those kinds of analyses can be challenging and could require additional data and modeling assumptions.
According to VTrans staff, although the statutory requirement to use a scoring system does not strictly preclude the use of BCA, it has made BCA somewhat less of a priority, at least to date. Moreover, BCA would require additional staff expertise, and potentially new software tools and other resources. The VTrans further noted that BCA may be difficult to apply to smaller projects, non-motorized projects, and pavement restoration due to limitations of existing methods and tools. This makes benefits more difficult to estimate and reduces BCA's overall usefulness for comparing projects.
The VTrans staff were interested in FHWA guidance in defining and estimating highway user costs. They would be particularly interested in concise, easy-to-understand documents on how and when to conduct BCA and how to monetize benefits, along with practical tools for cost-effectiveness analysis for different pavement treatment options.
Florida DOT (FDOT) is expanding its use of BCA alongside other decision-support tools. The FDOT currently uses BCA in a number of settings, including the development of its STIP, review of major projects that involve private-sector participation, and preparation of reports to the State legislature.
The FDOT's work in this area stems from roughly 2002, when the department responded to a State legislative requirement calling for an analysis of the macroeconomic impacts of FDOT's investments. The FDOT spent more than a year compiling the report, entitled Macroeconomic Analysis of FDOT's Five-Year Work Program. The analysis was prepared using the REMI model along with FHWA guidance, and covered projects in highways, transit, and rail. Since then, although the statute did not specify a timetable, the study was updated in 2006, 2009, and 2015.41 The analysis has also been expanded to include other modes, such as seaport projects, and greater treatment of highway safety benefits. The report includes impacts on Florida employment and incomes, and thus comprises economic impact analysis as distinct from BCA. That said, it also includes conventional BCA elements such as travel time improvements and user cost savings, and was the original impetus for introducing BCA into decisionmaking.
More recently, FDOT expanded their BCA work in response to the requirements of the Federal TIGER grant program. They adopted DOT guidance for monetization of benefits and other areas in order to make their applications as competitive as possible. The FDOT continues to rely primarily on DOT and other Federal guidance in its BCA work, for example in its values of injury reduction, travel time savings, and avoided emissions. (One difference is that FDOT prefers a 4 percent real discount rate, rather than the 7 percent in OMB guidance; FDOT also focuses primarily on impacts on Florida residents, visitors, and businesses, rather than the United States as a whole.) The FDOT also emphasizes the transparency of the analysis.
At present, FDOT uses a hybrid approach for project prioritization and the development of its 5-year work plan and STIP. A scoring system called the Strategic Investment Tool (SIT) is used for most projects and funding categories. Scores are calculated primarily using engineering metrics (e.g., crash ratio, volume/capacity ratio), but also with qualitative assessments of quality of life impacts and the alignment with strategic plan goals. An overall score is then generated using predetermined weights for each metric. Some projects—usually large projects with capital expenditures over $100 million—are also subjected to BCA. The FDOT staff stated that the role of BCA in the overall process is still evolving. In general, BCA results can be an important factor in prioritization, but they are not the sole or primary factor, and the overall process includes human judgment rather than mechanistic adherence to BCA results. The primary tool used for internal, in-depth BCA is the MET (Metropolitan Economic Tool).
According to FDOT staff, the major impediment to expanding the use of BCA is resources: staff time and budget. It is simply not possible to do a “gold standard” analysis on every project. Another key challenge is ensuring coordination with colleagues at the district level and the Florida Turnpike, and overcoming some confusion about the nature of BCA and what it includes.
The FDOT submitted a proposal to NCHRP to re-visit earlier forecasts and BCA estimates to assess their accuracy. Although that proposal was not accepted, they continue to view this as a potentially valuable research area, and it has gained the attention of upper management at FDOT.
Overall, BCA has a definite and growing role in decisionmaking at FDOT, alongside scoring-based project prioritization and economic impact analysis. The FDOT staff believes that the role of BCA within the organization will continue to evolve in the coming years, with the establishment of formal procedures being one possibility.
The Arkansas State Highway and Transportation Department (AHTD) uses BCA for certain project types and areas of responsibility, while relying on a multifactor assessment process and professional judgment for other areas. The AHTD staff described BCA as being incorporated to some degree into three main areas of activity: safety, planning, and project prioritization.
For safety improvements, BCA is done at the project level and takes place during project development. The general approach is to select potential project locations based on crash data and professional judgment, then to use BCA to compare the benefits and costs of alternative potential safety countermeasures at those locations. Forecasts of potential crashes avoided are based on historical crash data and standardized Crash Modification Factors (CMF) for each type of countermeasure, though with a downward revision to account for possible bias in the CMFs.
Avoided injuries are monetized using the KABCO scale, with a current value of roughly $5.5 million per fatality. These monetized values are updated periodically based on inflation and earnings data. Costs are reckoned based on installation costs and annual maintenance costs. Future values are annualized using a present-value formula, with a discount rate set according to the current real yield on U.S. Treasury bonds (as listed in OMB A-94, Appendix C). The BCA results are then compared across all potential countermeasures to assess their cost-effectiveness.
For corridor studies and planning studies, AHTD uses a BCA approach that they described as drawing on methodologies from FHWA and AASHTO.42 The emphasis here has been on travel time impacts for road users. The AHTD does not typically include emissions impacts, both because they are not the lead agency in the State for emissions issues, and because past modeling work showed that the monetized value of emissions changes was typically very small in comparison to the other impacts. Other factors such as environmental or economic impacts are considered qualitatively in the analysis.
For project prioritization, AHTD uses an approach very similar to North Carolina DOT, an agency with which they have actively engaged in discussions. It is a weighting and scoring system in which AHTD gauges each proposed project against a multicriteria list. This is a new process that was brought in for the current round of STIP formulation. Several impact categories are estimated using travel demand models. The process also incorporates project costs and calculates a measure of “return on investment” (ROI) and benefit-cost ratio (BCR), though these are products of the weighted scoring system rather than ROI or BCR measures in the strict senses of those terms.
It is worth noting that AHTD's prioritization approach facilitates “cross-asset” comparison, so maintenance and preservation projects compete with new capacity and expansion projects. The AHTD staff observed that this process tends to bring out the benefits of a project much more clearly. This was especially true for projects that, in a separated system, would have been considered only in the maintenance and preservation category and received a lower score, but score higher in a combined system due to safety and other benefits.
Currently, AHTD conducts most of this BCA work in-house, with consultant support on the long-range plan and the State freight plan. The AHTD is looking to improve the data used for the project prioritization process, especially for travel time reliability, and to expand their BCA to become more comprehensive. In addition to BCA, they are also looking to conduct analysis of local economic impacts, such as job creation, property values, and impacts on economic competitiveness. They have been exploring opportunities to use the EconWorks economic analysis tools,43 as well as the REMI (Regional Economic Models, Inc.) and IMPLAN (Impact Analysis for Planning) models. The AHTD staff stated that they would welcome additional FHWA support and guidance for obtaining and using these tools.
The AHTD has not met any criticisms of their prioritization processes yet and sees little downside to the use of BCA, as long as the methodology is sound and the process is vetted so that outside influences cannot corrupt the calculations. In Arkansas, the multifactor scoring process is viewed as objective and is used to support the State Highway Commission's ultimate decisions on project selection. The Commission's selections have not differed greatly from AHTD's proposals. Overall, the Arkansas highway planning and programming system is moving from a subjective process to a more merit-based project selection process, reflecting how the agency is moving into performance-based management.
To date, AHTD has not compared measured outcomes of their completed projects to their modeled forecasts. They do not believe they have enough years of data to do so at this time.
Aside from the issue of enhanced access to and guidance on analysis tools that include economic development, AHTD would like assistance from FHWA in strengthening their BCA capabilities. This includes technical assistance and workforce development for items such as travel time reliability, system analysis, and business plan construction. Additionally, there are numerous methodologies presented across various official sources, with little guidance on how and when to use them. AHTD stated that they would find a clearinghouse to help agencies sort through the methodologies beneficial.
Oregon DOT (ODOT) uses a combination of asset management systems, a qualitative multifactor assessment process, and stakeholder consultation for project prioritization. In developing its STIP, Oregon prioritizes potential projects within two broad categories: “Fix-It” projects, which are those that preserve current assets, and “Enhance” projects, which expand capacity or create new connections. This represents a change from earlier practice, when ODOT divided funding across a larger number of more specialized categories. In recent years, roughly 75 percent of available funding has been assigned to Fix-It projects, and roughly 25 percent for Enhance projects, but the share allocated to Fix-It projects may increase to around 88 percent in the near term as the State addresses a maintenance backlog.
Prioritization for Fix-It projects is conducted by ODOT staff, with the method of prioritization varying somewhat across asset types. For pavement and bridges, asset management systems are used to track conditions, conduct initial identification of projects, and assess cost-effectiveness. A new tool being developed will also allow ODOT to compare the cost-effectiveness of projects across the two categories, rather than looking at pavement and bridges separately. (This tool employs a form of life-cycle cost analysis, but is focused on direct agency costs and does not include quantified benefits or costs for highway users.) For smaller assets such as culverts and traffic signals, optimization is conducted by ODOT staff using professional judgment and input from their counterparts at the regional level.
Projects in the Enhance category are prioritized using a stakeholder engagement process that works with 11 regional Advisory Committees on Transportation (ACT) around the State. Each ACT includes representatives from local government, the private sector, and other stakeholder groups. Working with the ACTs, ODOT staff qualitatively assess proposed Enhance projects against published criteria in four areas: benefit to the State transportation system, consistency with State and local plans, mode-specific criteria, and cross-modal criteria. The mode-specific criteria vary but are focused on connectivity, safety, accessibility and mobility. The six cross-modal criteria are economic development, social benefits (e.g., health and assisting disadvantaged communities), environmental stewardship, safety, project readiness, and leverage.
The ODOT noted that this qualitative assessment process seems well-suited to their situation, in which only a limited share of funding is available for Enhance projects. They prefer to use a process that incorporates substantive stakeholder involvement rather than one centered on BCA. For these reasons, BCA is not typically employed at all for decisionmaking or prioritization, except where required as a condition for external funding (such as for TIGER grants).
In some cases, however, ODOT has prepared special analyses on particular projects or issues to help policymakers understand the economic impacts of transportation investments. A recent example was the “Rough Roads Ahead” report (2014), which analyzed the actual costs of keeping State roads in current good condition versus actual planned expenditures, and also estimated the impacts on the Oregon economy if these investments are not made. The report used the SWIM2 model (Oregon Statewide Integrated Model), an integrated statewide model of land use, transportation, and economic activity. This approach is closer to an economic impact analysis than a BCA, though it does include user impacts such as changes in vehicle operating costs. Impacts on safety and congestion were not estimated but were discussed qualitatively in the report. Similar reports were also produced to help Oregon legislators and policymakers understand the potential economic impacts of bridge load limits and the need for seismic retrofit of roadways. The ODOT also maintains its HERS-ST model (Highway Economic Requirements System – State Version) and uses the model to forecast the safety and emissions impacts of proposed projects. However, these estimates are not necessarily always used as part of the qualitative assessment of Enhance projects.
|State DOT||Use of BCA and Other Findings|
|Connecticut DOT||BCA is limited to safety projects; conducts retrospective review of estimates|
|Nevada DOT||Conducts BCA for capacity expansion projects over $25 million, as required under State law|
|Minnesota DOT||Longstanding and widespread use of BCA under a Cost-Effectiveness Policy, but other factors are also considered|
|North Carolina DOT||Developed a multi-factor scoring system BCA for project prioritization, rather than BCA|
|Utah DOT||Uses asset management tools for maintenance projects and a multi-factor scoring system for capital projects|
|Vermont Agency of Transportation||Using a criteria-based scoring system for prioritization, but with plans to revise approach in coming year|
|Florida DOT||Growing use of BCA for several project categories and decision-support functions|
|Arkansas State Highway and Transportation Department||Uses BCA for some project categories and functions, but uses a criteria-based scoring system for project prioritization|
|Oregon DOT||Qualitative prioritization process for expansion projects, with a large role for stakeholder input; little to no use of BCA|
This section synthesizes information from the literature review, RFI, questionnaire, and case studies to present overall findings on this study's four key questions: the extent BCA use among State DOTs, the quality of the analysis, challenges faced, and potential strategies to address those challenges.
There continues to be significant variation in the extent to which State DOTs use BCA, both across States and across different project types. Based on the questionnaire results, a small group of roughly five to six State DOTs are systematically employing BCA to inform decisionmaking, but this continues to be the exception, not the rule. Many State use BCA only for certain project types or for situations where a BCA is required for external funding.
Although some individual States have implemented significant changes since GAO's 2005 study of State-level practices, the overall findings on the use of BCA and its role in the decisionmaking process are largely unchanged since that time. One exception is that there appears to have been a reduction in the number of States that “never” conduct BCA. The influence of BCA on decisionmaking also continues to vary significantly. However, even among the State DOTs for whom BCA is a routine and well-established part of their process, such as Minnesota, the BCA results are almost always described as just one of several factors considered in the overall assessment.
Among States that conduct BCA only for certain project types, safety projects and large or significant projects were reported as most likely to be subjected to BCA, while bicycle/pedestrian and roadway rehabilitation projects were the least likely. This is largely consistent with the case study interviews, although some States are attempting to include more project types in their BCA work and facilitate cross-category prioritization. The emphasis on safety projects may reflect a requirement to use some form of quantitative prioritization in creating Strategic Highway Safety Plans under the Highway Safety Improvement Program. Some State DOT staff also noted that safety-related analyses benefitted from greater availability of data and well-established quantification and monetization methods.
State DOTs also use several alternatives to BCA. Notably, asset management systems with a life-cycle cost component are in widespread use across State DOTs, most commonly for pavement and bridges. These systems can generate optimized packages of maintenance spending, but do not always consider benefits and costs to highway users as distinct from the DOT's own costs. More recently, there is an apparent trend toward the use of multifactor scoring systems to facilitate prioritization, either within or across project categories. These approaches, some of which are summarized in the case studies above, facilitate some degree of quantitative decisionmaking, but unlike BCA they do not attempt to monetize impacts or assess the change in societal welfare. In some cases, a criteria-based approach may be used, but with a more qualitative assessment and a formalized program of stakeholder input. The questionnaire indicated that 18 State DOTs are using some form of scoring system in place of, or in addition to, BCA. While direct comparisons to prior years are not available, the literature on this topic and the case study interviews both suggest growing interest in this approach.
Although the study team did not conduct a firsthand review of State DOTs' BCA products, the published literature in this area notes a number of quality concerns. Some of the key issues that have been identified are improper baselines, speculative benefits, the inclusion of transfers as benefits, and a general lack of transparency and reproducibility.
Several of the case studies further highlighted the fact that the “societal” framework of conventional BCA, which focuses on benefits accruing to all users and non-users and costs borne by society at large, may run counter to State governments' tendency to focus on their own constituents and expenditures, which can bias the results. External project funding sources such as grants, local matching funds, and tolls are often excluded altogether from cost calculations; indeed in some cases projects receive bonus points for leveraging external funding. This has the effect of overstating the net benefits of a project. States may also be more prone to double-count new jobs and economic activity within the State as a benefit, in addition to the direct benefits accruing to users and non-users of the improved infrastructure.
Based on information from the case studies and the questionnaire, a recurring challenge with BCA is ensuring the comprehensiveness of the analysis and including an appropriate range of alternatives. Many State DOTs that conduct BCA focus their efforts on impact areas for which relatively straightforward data, methodologies, and monetization factors exist. Safety impacts are among the most commonly quantified; a typical approach is to combine historical data on crash rates and severity with engineering-based crash reduction factors. More complex areas such as emissions and freight impacts are often excluded, though methodologies do exist in these areas, and some models include these impacts. Impacts on equity and the human environment are generally regarded as very challenging to quantify. State DOTs also typically have limited capability to conduct BCAs across program areas and travel modes, making it especially difficult to include a wider range of alternatives in BCAs for highway projects, such as investments in public transit or non-motorized transportation.
Another continuing quality concern is the ability of travel demand models and other forecasts, which are designed to support broad policy and investment decisions at the regional level, to support project-level BCA. With limited exceptions, State DOTs generally do not return to prior forecasts and estimates to assess their accuracy against actual conditions, generally due to resource limitations. This contributes to continuing forecasting inaccuracy and (at least as argued by some in the literature) to continuing distrust of BCA among decisionmakers. This could change over time with the increased emphasis on performance-based planning and related Federal rulemakings. The FHWA Travel Model Improvement Program (TMIP) has been working with State and local transportation planning agencies since 1994 to support effective use of analytic methods and tools in transportation decisionmaking. The program provides multiple resources, including webinars, trainings, peer reviews, and research into emerging and more advanced modeling approaches.
Among the State DOTs in the case studies, the discount rates used varied significantly. In some cases they are tied to the real cost of borrowing (i.e., the inflation-adjusted yield on U.S. Treasury bonds, which is currently 1-2 percent) while in other cases they are set by State policy or reflect the 7 percent real rate recommended by OMB for most infrastructure projects. Monetization factors for injuries also varied to some extent across the case studies, but were generally in the range of $5 million to $10 million per fatality avoided. Some State DOT staff noted an intent to use Federal guidance in these areas, but that it is sometimes unclear which guidance document prevails or how to update figures over time.
State DOTs face a variety of challenges when attempting to use BCA for decisionmaking, reflecting a mixture of institutional, resource, and technical issues. Fundamentally, there is little institutional support within some State DOTs for conducting BCA in the first place, as it is not a requirement for most Federal-aid programs and existing prioritization methods may seem to be working well. Indeed, some of the interviewees stated that their State DOT would be unlikely to move in the direction of greater BCA use absent strong evidence that it works better than their current approach.
The non-BCA methods currently being used often employ engineering metrics (e.g., volume/capacity ratio) that are better understood within the agency, both at the staff level and by decisionmakers. The BCA results may be viewed as more difficult to interpret and explain to stakeholders. Perhaps as a result, States that are moving in the direction of more data-driven prioritization seem just as likely to move to a multicriteria scoring approach, rather than use BCA. In the case studies, State DOTs using this kind of scoring approach emphasized that it was transparent and objective. From an analytical standpoint, however, these scoring approaches also present issues because the impact categories are not necessarily comprehensive or mutually exclusive. Also, unlike the monetary values used in BCA, the weights given to each category in the scoring approaches do not have an empirical foundation.
In at least a few States, the State DOT is required by law or regulation to make decisions and prioritize projects according to specific criteria, and there are often other procedural requirements with the force of law, such as environmental reviews. Moreover, while Federal-aid funding has many fewer category-based restrictions than in the past, there is still somewhat of a focus on prioritization within rather than across categories. Strictly speaking, none of these factors preclude the use of BCA, but their presence may greatly reduce the State DOT's ability to use BCA to inform decisionmaking.
Resource constraints are noted in the literature and were confirmed by the case study interviewees and FHWA questionnaire respondents. Simply put, BCA and its associated data and modeling needs can strain agency budgets, staff time, and other resources. It also requires specialized expertise that the organization may lack.
In addition to these institutional and resource issues, technical challenges also remain. In fact, difficulties quantifying and monetizing benefits were the most frequently cited challenge for State DOTs in the FHWA questionnaire. According to published literature and case study interviews, this is especially true for freight, multimodal, and non-motorized projects due to a relative lack of established methodologies and valuation methods.
Finally, a key challenge to the broader use of BCA among State DOTs is simply that factors other than net benefits continue to be important for decisionmakers at the State level. The BCA typically does not address—at least not directly—several issues that have great influence on project prioritization and other decisionmaking. These include local economic impacts and economic development; equity (by region, mode of travel and/or program area); the degree of support from the public and other stakeholders; the ability to leverage external funding sources; and the likelihood of completing a project without delays.
Several interviewees mentioned strategies that would address technical challenges with pursuing BCA. Outreach and training to State DOTs would help to address the gaps in expertise, and more specifically, there were suggestions for a “clearinghouse” of BCA-related resources and guidance.44 Several State DOTs noted that they would like to apply Federal guidance or other best practices, but that it can be difficult to sort through the various resources available from OMB, the Office of the Secretary of Transportation, the TIGER program,45 as well as non-Federal sources such as AASHTO. Particular needs cited by the States included enhanced information on how to apply BCA across different project types, and more specifics on how monetization factors should be updated over time.
State DOTs would also welcome additional technical assistance from FHWA on BCA methodologies generally and on specific issues such as travel time reliability and the application of BCA to analytically challenging program areas such as operations and maintenance. While the existing products available from the SHRP2 program were mentioned as valuable, State DOTs may need additional assistance in applying these findings and incorporating them into their BCA practices. The FHWA Office of Operations provides resources on how to use BCA in operations and systems management, including an online desk reference46 and a sketch-level decision support tool (TOPS-BC, Tool for Operations Benefit Cost Analysis). In this context, BCA can assist in screening, identifying, and prioritizing management and operations projects or strategies that meet operations objectives.
Additional data collection programs or other data resources could also be helpful in addressing the identified challenges in BCA-related data. As State DOTs move toward implementing legislated performance management requirements, FHWA will develop additional resources related to making investment decisions and tradeoffs, which may also address the use of BCA. The FHWA also continues to make available software tools and models that support BCA, including HERS-ST, the National Bridge Investment Analysis System (NBIAS), and a web-based tool called BCA.net.47
At the institutional level, many agencies noted the overall movement toward performance-based, data-driven planning as required by the MAP-21 legislation. Some agencies would like to see more assistance with this transition. Although the move toward performance-based planning includes many elements beyond BCA, the overall outreach program could include a component on BCA, as well as on alternative methods for project prioritization and their relative merits. Pending and upcoming rulemakings in this area may present a useful opportunity to engage with State DOTs on this topic. The FHWA programs on topics such as Innovative Finance and Performance-Based Practical Design may also provide useful resources to State DOTs, supporting analyses that help with prioritizing planning and project-level decisions, and tools to analyze impacts at both the project level and system-wide. In addition, FHWA and other U.S. DOT program offices will continue to support research and capacity building in important emerging human and natural environment topic areas such as health, climate resiliency, neighborhood connectivity, and non-motorized transportation, for which benefits can be difficult to quantify and which have thus typically been excluded from traditional BCA.48
On the broader question of improving the overall use of BCA and its acceptance as part of the decisionmaking process, interviewees tended to provide less in the way of concrete suggestions. The literature in this area suggests that BCA may gain traction when decisionmakers are more informed about how BCA works and are more confident in its findings. This could include outreach and communication more generally, and specific tasks such as conducting retrospective studies to assess the accuracy of forecasts and estimates. These retrospective studies have been suggested as a means of fostering greater confidence in BCA as a decisionmaking tool. Other suggestions from the literature include producing more streamlined, readable BCA summaries to aid policymakers, and applying BCA tools to estimate impacts in other areas that influence decision making, such as equity.
Boardman, Anthony, Aidan Vining, W. G. Waters, II, “Costs and Benefits through Bureaucratic Lenses: Example of a Highway Project.” Journal of Policy Analysis and Management, Vol. 12, No. 3, (Summer, 1993), pp. 532-555.
Based on their experience working with the Canadian government, the authors identify three archetypical roles in the bureaucracy that have divergent views of BCA: “Spenders,” who are focused on delivering services to their constituents; “Guardians,” who emphasize the need to control public expenditures; and a smaller minority of “Analysts,” who apply conventional BCA. Among other differences, Spenders typically identify project-related expenditures as benefits of a project rather than costs, while Guardians limit their analysis to the public sector balance sheet, viewing government revenue as “benefits” and public expenditures as “costs.” The groups also differ on issues such as discount rates and treatment of sunk costs. The authors conclude that understanding these varying perspectives can help Analysts explain what BCA actually is and perhaps to improve its acceptance within the bureaucracy.
Flyvbjerg, Bent, Mette K. Skamris Holm, and Søren L. Buhl. “How (In)accurate Are Demand Forecasts in Public Works Projects?” American Planning Association 71.2 (2005): 131-46.
The authors present the results of a study they conducted of traffic forecasts for a sample of rail and road infrastructure projects and use them to highlight an important issue with BCA use for transportation. Based on their analysis, the authors conclude that traffic forecasts are overwhelmingly inaccurate to the point of intentional manipulation of data by planners. They call for increased accountability and transparency in the BCA process to avoid this. However, they also blame improper and outdated techniques for forecasting as well as political pressures, and suggest an improved method – reference class forecasting – to alleviate some of the inaccuracy and unreliability of BCA.
Hahn, Robert W., and Patrick Dudley. How Well Does the Government Do Cost-Benefit Analysis? Working paper no. 04-01. AEI-Brookings Joint Center for Regulatory Studies, 2006.
The authors examine the quality of cost-benefit analysis used by the Federal Government during the Reagan, Bush, and Clinton administrations. They outline different points of view on BCA, its importance in government policy, and offer suggestions as to how the BCA process and use within the government could be improved with increased oversight and standardization. Their assessment finds that there is an overall lack of quality in BCA – many of the analyses did not include basic economic information – and that there was no trend in the quality over time or signs of improvement. They cite a lack of punishments or incentives as the reason for this inefficient us of BCA and indicate that increased transparency and accountability would lead to improvements.
Homan, Anthony C. “Role of BCA in TIGER Grant Reviews: Common errors and influence on the selection process,” Journal of Benefit-Cost Analysis, Vol. 5, No. 1 (2014), pp. 111-135.
This paper sheds light on both the quality of BCAs conducted by State and local governments and the overall influence of BCA in the selection process for competitive grants under the Federal TIGER program. The author notes several methodological problems with the BCAs received from applicants, including issues in selecting appropriate baselines and alternatives, improper treatment of transfers and property value changes, and an overall lack of transparency and reproducibility. Drawing on an analysis of past award decisions relative to project-level BCA characteristics (BCA quality and likelihood that benefits exceed costs), the author finds evidence that BCA results did influence the selection process, particularly in earlier rounds.
Misuraca, Pamela. The Effectiveness of a Costs and Benefits Analysis in Making Federal Government Decisions: A Literature Review. Center for National Security, The MITRE Corporation, 2014.
This literature review defines BCA, describes how it can best be used, examines different points of view on its use, and analyzes its importance in Federal Government decisions. It looks at the measures and methodology of BCA and addresses its general position in the decisionmaking process. The author acknowledges that BCA alone is often not suitable as the sole basis for decisionmaking, but in the right circumstances and when done properly, it can be a valuable tool for quantifying the consequences of proposed programs. Some criticisms include the frequent misuse of BCA, and its limitations with respect to health and environmental factors as well as its potential inaccuracy and unreliability. However, if BCA is performed correctly, it enables government to request funding, develop budgets, identify risks, and manage programs' performance to quantify risks or investment decisions.
Theory, Science, and Statistics in the Use of Benefit-Cost Analysis. Conference held by University of Washington Benefit-Cost Analysis Center, October 2011, Washington, DC. Summary of conference proceedings online: http://evans.uw.edu/sites/default/files/public/Conference%20Summary_2011-FINAL.pdf
Conference discussions focused primarily on BCA in a Federal regulatory setting rather than with State and local governments. In that setting, many of the issues are methodological, such as treatment of uncertainty and setting appropriate monetary values for fatalities and injuries avoided. However, the keynote speaker, Professor Robert Haveman of the University of Wisconsin, also noted that one key challenge for BCA is narrow “accounting stances” and the tendency for decisionmakers to focus on budgetary impacts rather than broader societal impacts.
Thank you for your time. This questionnaire will be valuable to us in preparing a Report to Congress on State DOT use of benefit-cost analysis.
1. Contact Information:
|Your Job Title:|
|Your Division Office:|
DEFINITIONS. For the purposes of this questionnaire, consider “benefit-cost analysis” as a systematic process by which the impacts of a project (or other action) are forecast and quantified, so that societal benefits and costs can be compared for that project or a range of alternatives. BCA typically converts impacts to monetary equivalents and converts future impacts to present values using a discounting formula.
For the purposes of this questionnaire, benefit-cost analysis does NOT include other forms of economic analysis that might be used, such as:
2. Thinking in general about your State DOT's Federal-aid highway program, how often does the State DOT use Benefit-Cost Analysis (BCA) in making decisions and setting priorities? Please refer to the Definitions section if you are not sure whether a particular type of analysis should be included here.
The definition of BCA as used for this questionnaire is repeated below for your reference. Throughout the survey, you may also use the “Previous” button to return to this definition.
3. For each of the following stages, please indicate how often your State DOT uses BCA as part of its decision-making process.
never, but less
than half the time
|About half the
|More than half
the time, but
|Policy/program level or overall resource allocation||◯||◯||◯||◯||◯||◯|
|Long-range transportation plans||◯||◯||◯||◯||◯||◯|
|Operations and maintenance||◯||◯||◯||◯||◯||◯|
|Other (please describe below):||◯||◯||◯||◯||◯||◯|
Benefit-cost analysis is a systematic process by which the impacts of a project (or other action) are forecast and quantified, so that societal benefits and costs can be compared for that project or a range of alternatives. BCA typically converts impacts to monetary equivalents and converts future impacts to present values using a discounting formula.
For the purposes of this questionnaire, benefit-cost analysis does NOT include other forms of economic analysis that might be used, such as:
4. For each of the following specific types of projects, please indicate whether or not your State DOT typically (i.e., more often than not) uses BCA:
|Large or significant projects (as defined by the State DOT)||◯||◯||◯|
|Projects requiring an Environmental Impact Statement||◯||◯||◯|
|Discretionary or competition-based projects||◯||◯||◯|
|Projects expanding capacity||◯||◯||◯|
|Asset preservation projects (3R)||◯||◯||◯|
|Other project types (please describe)||◯||◯||◯|
5. Please identify whether your State DOT's BCAs typically (i.e., more often than not) include each of the following elements.
|Estimated project capital costs||◯||◯||◯|
|Estimated project operations and maintenance costs||◯||◯||◯|
|Estimated asset disposition costs or residual value||◯||◯||◯|
|Safety improvements – fatalities and injuries avoided||◯||◯||◯|
|Travel time savings and/or reliability improvements||◯||◯||◯|
|Savings in vehicle operating costs||◯||◯||◯|
|Reduced noise, vehicle emissions and/or reduced greenhouse gases||◯||◯||◯|
|Improved human environment or community inclusion||◯||◯||◯|
|Improved access to markets||◯||◯||◯|
|Improved intermodal connectivity||◯||◯||◯|
|Freight or supply-chain benefits||◯||◯||◯|
|Equity and/or environmental justice impacts||◯||◯||◯|
Other factors (please describe):
6. In your opinion, how much importance do BCA results typically have in your State DOT's decisions to recommend a project from among its various alternatives?
7. Other than BCA, or in addition to BCA, what forms of quantitative analysis does your State DOT typically use for making decisions and setting priorities? (Please select all that apply)
8. In your opinion, what are the challenges that your State DOT faces in using BCA for project selection and decision making? (Please select all that apply)
9. In your opinion, what strategies would best help your State DOT address these challenges in using BCA?
10. Do you have any other information or feedback to share regarding your State DOT's use of BCA, or feedback on this questionnaire?
Thanks again for your assistance with this effort. Please be sure to hit the DONE button below to submit your answers.
Issue Date: June 11, 2015
Response Due Date: July 15, 2015
Submit responses electronically to: Matthew.Carr@dot.gov
Email Subject Line must include: “Benefit Cost Analysis Study: Request for Information”
This is a Request for Information (RFI) notice. It is NOT a solicitation for proposals or proposal abstracts. The objectives of this RFI are to:
This RFI is intended for:
Information gathered will be considered in informing the Federal Highway Administration (FHWA) on evaluating the use of benefit cost analysis by State Departments of Transportation and how they are using benefit cost analysis to make decisions, set priorities, assess the quality of the analysis, and identify the challenges and strategies faced when using benefit cost analysis. Respondents are invited to comment on any of the items described herein; in particular, respondents are encouraged to address the specific questions featured at the end of this RFI.
This RFI is not to be construed as a commitment on the part of the Government to award a contract. Information submitted shall be provided on a voluntary basis only. The Government will not pay for any information or responses submitted as the result of this RFI.
Submissions whether public or private will not be regarded by the Government as tantamount to a proposal. The Government reserves all rights to utilize any of the information supplied in future requirements and procurement opportunities. Therefore, by submission of a response, respondents waive their rights to intellectual property and/or proprietary information with the exception of information that is protected under patent, copyright, and/or trademark and clearly identified as subject to protection.
Benefit cost analysis (BCA) is an important economic tool that can help State and local governments target their transportation funding to the most effective investments. Using benefit cost analysis, a State or local government would compare the monetary value of all benefits and costs that accrue during the life of a project. This process forces State and local government to evaluate the value of all of the project's benefits, recognize the full cost of the project, and acknowledge whether or not the benefits outweigh the costs.
BCA estimates the benefits and costs to society of transportation investments. When discounted future benefits equal or exceed the discounted life-cycle costs, a project is considered economically efficient. Applied correctly, by comparing projects' net discounted benefits to net discounted costs, BCA can support the selection and prioritization of projects that have the best potential to achieve long-term performance objectives of transportation investments. BCA explicitly identifies the benefits to the users of the system and the costs to the agency, providing transparency and accountability for investment decisions.
BCA assigns a monetary value to the stream of benefits to society generated by the transportation investment. Transportation agencies can use BCA to measure the efficiency of spending from the viewpoint of benefits and costs to society. Efficiency, in the context of managing transportation assets, ensures that taxpayer dollars receive the best return on investment.
Please respond to those questions that are relevant to your business. References to supporting literature and web resources are welcome.
1. Please describe your experiences with States and local governments in applying benefit cost analysis.
2. Please describe your experiences with applying benefit cost analysis for making decisions and setting priorities.
3. Please comment on aspects of FHWA encouraging State and local governments to evaluate projects costs and benefits using an appropriate analytical framework.
4. Would you be willing to participate in a follow-up interview with a FHWA representative?
5. Please provide any information you feel is relevant to this topic that is missing from the questions above.
A firm's response to this RFI will not influence the evaluation of any proposals requested in the future by the Federal Highway Administration (FHWA) for the requisite services.
ANY INFORMATION PROVIDED IN RESPONSE TO THIS REQUST FOR INFORMATION MAY BE USED BY THE GOVERNMENT AND SHALL NOT BE MARKED PROPRIETARY.
The FHWA anticipates using information obtained from this RFI in developing requirements which may be issued in one or more solicitations on FedBizOps.gov.
Thank you for taking the time to review and respond to this request for information.
Benefit Cost Analysis - The Federal-aid Highways program represents an important partnership between the Federal Government and each State department of transportation. The Federal role has primarily been to set standards, ensure compatibility among State systems, provide capital assistance, and oversee highway construction. State governments operate the highway system and set local priorities for constructing and repairing roads and bridges.
While remaining sensitive to the role of State governments in setting priorities among highway projects, the Committee believes that the Department of Transportation plays an important role in ensuring that Federal resources are not spent on wasteful projects. Benefit cost analysis is an important economic tool that can help State and local governments target their transportation funding to the most effective investments. Using benefit cost analysis, a State or local government would compare the monetary value of all benefits and costs that accrue during the life of a project. This process forces the government to evaluate the value of all of the project's benefits, recognize the full cost of the project, and acknowledge whether or not the benefits outweigh the costs.
The Committee is aware of FHWA's efforts to support State and local governments in their use of benefit cost analysis. FHWA offers technical assistance to State and local governments that are already engaged in benefit cost analysis, and looks for ways to improve the estimates and models used in the analysis.
The Committee urges the Department to take a more active role in advancing the use of benefit cost analysis. The Committee recommends the Department encourage State and local governments to evaluate project costs and benefits using an appropriate analytical framework, either through strict benefit cost analysis or through a less formal structure if a project size does not warrant a more rigorous approach. The Department should ensure that FHWA division offices reach out to State departments of transportation in order to determine if the State could more effectively utilize benefit cost analysis as it sets its priorities.
The Committee also directs the Department to evaluate the use of benefit cost analysis by State departments of transportation, and to issue a report to the House and Senate Committees on Appropriations no later than 180 days after enactment on the extent to which State departments of transportation use benefit cost analysis when making decisions and setting priorities, the quality of such analysis, challenges that State departments of transportation face when trying to use benefit cost analysis, and strategies for addressing those challenges.