FHWA-HEP-09-026
There is no single National standard or set of regulations dictating how a CMAQ program should be structured and operated at the State or MPO level, and the CMAQ authorizing legislation does not require the issuance of such regulations. FHWA offers CMAQ program guidance, but intentionally leaves discretion to MPOs and State DOTs to develop a process that best responds to needs at the regional or local level. Accordingly, the Federal CMAQ Team found differences in structure and operations at every one of the sites visited. Processes for selecting, programming, and evaluating CMAQ projects are highly diverse from region to region and may even vary within a single State. Agencies also varied in their strengths as well as in the challenges they faced to effective CMAQ implementation.
This section of the report documents effective CMAQ implementation practices found in the seven field locations where interviews were conducted. The Federal CMAQ Team identified effective practices across a diverse set of agencies, where each agency is working to implement a CMAQ program that best responds to local needs and conditions. Effectiveness was assessed in relation to FHWA's CMAQ program guidance and the practices identified during the Phase I analysis. Generally, effective CMAQ programs are those that exhibit some or all of the following characteristics:
The exploration of effective practices and noteworthy examples described below may be useful and instructive to other MPOs around the country as they build their own experience with the CMAQ program and seek to improve their own implementation of CMAQ funded projects. Readers are encouraged to delve more deeply by reading the case study of each site visit (included in Section 6) to identify which practices may be applicable to their own organizations and to contact the organizations for information-sharing.
Federal guidance for the CMAQ program states the need for a selection process that is "transparent, in writing, and publicly available."9 How MPOs or States structure their CMAQ process may vary widely, but in every case it is important that there be clearly understood, documentation and guidance on:
In line with Federal guidance and the findings from Phase I, information gathered during the Phase II study confirmed that adopting transparent project solicitation, prioritization, and selection processes is a critical success factor for effectively implementing a good CMAQ program. Common attributes of open, understandable CMAQ program processes were:
The above characteristics are important to gaining and maintaining trust by policy-makers and the public. Greater understanding of, familiarity with, and trust in MPO processes by potential project sponsors (i.e., applicants to the CMAQ program) translates into more project submittals, better competition, and ideally, the selection of better projects for funding. When project sponsors do not understand how projects rise to the top of the selection process, they may feel that decisions are made for nonmerit reasons, thereby undermining community trust in the MPO planning process.
The results from the site interviews suggest that the public process used to solicit and program CMAQ projects is closely tied to an MPO's overall approach to programming Federal transportation funds. Approaches typically fall in one of two major categories:
Both approaches can support effective CMAQ implementation. Of the seven agencies interviewed for this report, smaller MPOs, with smaller budgets, fewer staff, and relatively less severe air quality challenges, tended to run CMAQ as a stand-alone program. The larger MPOs, with longer-standing programs and more complex congestion mitigation and air quality improvement challenges, tended to integrate CMAQ into the overall planning process. However, whether an MPO runs CMAQ as a stand-alone program or treats it as just one funding source among many, every State and MPO must integrate projects programmed through CMAQ into its regional Transportation Improvement Program (TIP) and then its State Transportation Improvement Program (STIP). As such, CMAQ project solicitation typically follows the same schedule as TIP updates.
In Fort Collins, NFRMPO has produced and put on its Web site a 41-page document that describes the purpose of the CMAQ program, enumerates the agency's priorities for using CMAQ funds, lists project eligibility, and details the project selection process10. This clear outline of the CMAQ program goals, operating procedures, and selection process allows an interested citizen or organization with a project idea to read and understand how to get involved. In addition, the Web site has a consolidated calendar for CMAQ projects under consideration for the FY 2010–2011 funding cycle.
In order to improve the openness and transparency of the CMAQ selection process for project sponsors, planning partners, and the public, the SPC board and staff worked to reformulate the Pittsburgh region's CMAQ process. In 2007, SPC hired an independent facilitation team to assist with the CMAQ review to help ensure a fair and unbiased process. As a result, SPC updated its CMAQ evaluation and selection procedures and established a new CMAQ Evaluation Committee (CEC) to help review applications, prioritize projects, and make recommendations on which projects to fund. The CEC is composed of more than 20 members representing SPC's member counties, transportation management associations (TMAs), the Pennsylvania Department of Transportation (PennDOT), local environmental agencies, and SPC modal committees (freight forum, pedestrian/bicycle, transit operators). The revised process was endorsed by the SPC board and used to develop the recommended CMAQ Program for the 2009–2012 TIP.
SPC developed a revised CMAQ application that project sponsors can download from the Web, complete, and return to SPC electronically. The application is an interactive PDF file that allows project sponsors to attach documents or maps. It includes detailed instructions for applicants and provides a description of the project evaluation criteria used in the scoring, ranking, and selection process. SPC spent considerable effort in updating the application questions to ensure that all pertinent information is supplied for each project and to provide background information about the CMAQ program and guidance to help applicants understand the criteria that will be used to evaluate their projects.
When the RVMPO Policy Committee decided that diesel retrofit projects should be a priority for how to spend CMAQ funding, RVCOG staff organized a public open house specifically to reach out to local businesses operating diesel fleets in the area. The open house received media coverage from two television stations and a newspaper. The open house informed companies that Federal funding was available to support retrofitting their fleets' engines to burn cleaner, provided educational materials to show that retrofits do not adversely affect vehicle operations, and gave tips on how to apply for a CMAQ grant. A product of this outreach led to a successful public-private partnership (PPP) with a private company in the area, Rogue Disposal & Recycling, to begin retrofitting its fleet.
The CMAQ program's emphasis on innovative and nontraditional projects provides an opportunity for MPOs to foster partnerships with organizations that have not typically been involved in the metropolitan transportation planning process, such as air quality agencies, community organizations, and private firms. In Birmingham, RPCGB dedicates about $1.4 million of CMAQ funding annually to support the work of the Alabama Partners for Clean Air (APCA), a consortium of 14 public, private, and nonprofit organizations working to implement projects and programs that improve air quality in the RPCGB area. As such, the CMAQ program helps RPCGB to broaden public awareness of the metropolitan planning process by reaching out to and partnering with a wider range of organizations than it might have otherwise.
One of the distinguishing features of the CMAQ program is that project sponsors must estimate their proposed project's air quality benefits in order to be eligible for Federal funding. As described in the CMAQ Program Final Guidance issued in November 2008, agencies are expected to give priority consideration to cost-effective projects11. This requires that MPOs and State DOTs have the capacity to develop multiple methodologies to quantify outcomes and then apply cost-benefit analyses across a wide range of project categories and types. One of the most effective ways that agencies can accomplish these objectives is to standardize approaches. Developing standardized calculations also helps to clarify the methodologies by which CMAQ project proposals are ranked and selected, thereby increasing the transparency and openness of the programming process.
Site-visit interviews revealed that, while each agency had developed some type of standardized calculations, no two methodologies were the same. Several common components were identified, however, that helped to make the development and use of standardized methodologies more effective. These components were:
Developing quantitative and qualitative measures: Agencies use both quantitative and qualitative measures to estimate project costs and benefits and to evaluate CMAQ proposals. The development and choice of which measures to use is typically an iterative process, with the MPO board providing policy guidance and the staff developing the analytical formulas on which quantitative measures are based. Each MPO interviewed had its own unique set of quantitative and qualitative measures that it used to evaluate project submissions.
Quantitative measures include assumptions and standardized formulas that are tailored to regional characteristics, needs, and goals and used to calculate project benefits. Common quantitative measures among agencies interviewed were:
In some cases, State DOTs provide technical support to MPOs to develop standardized calculation methodologies. For example, while CMAQ projects are programmed at the MPO level in Pennsylvania, PennDOT has a statewide consultant who creates spreadsheets that can be used to calculate estimated air quality benefits for a wide range of CMAQ-type projects. The consultant remains on call to help MPOs with new or nonstandard project proposals (i.e., those for which a methodology has not yet been created). In other areas, the MPO relies on its own staff or an outside consultant to help formulate the quantitative measures. Table 1 provides a summary of how each of the seven MPOs interviewed developed its existing quantitative calculation measures. (More detailed information is included in Section 6: Site Visit Case Studies.)
| Boston MPO | Standardized spreadsheet developed by State DOT |
|---|---|
| Birmingham (RPCGB) | Standardized formulas developed by consultant |
| Denver (DRCOG) | Standardized formulas developed by MPO |
| Fort Collins (NFRMPO) | Standardized formulas developed and refined by consultant |
| Medford (RVCOG) | Standardized calculations performed by MPO |
| Pittsburgh (SPC) | Standardized formulas developed by Statewide consultant |
| San Francisco (MTC) | Supplements State calculations with its own MPO work |
Qualitative measures are more subjective and difficult to standardize. They are typically criteria that respond to MPO policy board goals by awarding points to proposals during the project evaluation process to encourage:
Qualitative measures can be built into the evaluation and ranking process by giving bonus points either upfront in the analysis of benefits or toward the end. They usually are developed on the basis of policy goals and popular support for certain types of projects or programs that support the region's long-range transportation vision but whose quantitative benefits are harder to estimate. For example, a number of MPOs noted that they use CMAQ funds for bicycle and pedestrian projects because they enjoy broad public and political support within the MPO even though they may not compete as successfully as other project types on a strictly quantitative basis, such as those focused on traffic signalization or intersection improvements.
Estimating air quality or congestion benefits: Once an MPO agrees on which quantitative and qualitative measures and methods to use, project benefits can be quantified. Most MPOs require that project sponsors provide the assumptions and related information needed to calculate a project's projected benefits; staff then provide a review to verify the assumptions and results. In Pittsburgh, for example, a project sponsor is able to download an electronic application, use the online tools to calculate benefits, and then submit the completed application to the MPO for review. Other MPOs perform all calculations in-house, using information provided by project sponsors on their applications.
In interviews, agencies emphasized that the use of quantitative analysis to summarize the benefits of projects helps to validate the selection process with the public and project sponsors by allowing everyone to see the calculated benefits. However, several agencies cautioned that the quality of benefits calculations is only as good as the assumptions built into the formulas. While a formula for a certain type of project may be applicable at the State or National level, many of the default variables need to be changed to reflect conditions that exist at the local or regional level.
In addition to the question of how to calculate benefits is the question of which benefits to calculate from the standpoint of improving air quality regionwide and what timeframe to use. Most MPOs follow FHWA guidance that only requires the calculation of benefits in Year 1 of project implementation. The timeframe that is used impacts the types of projects that rise to the top in evaluation. For example, an intersection improvement may show a relatively high benefit in Year 1, but this benefit may erode over time. A bike lane, new bus service, or outreach and educational program, on the other hand, may not show any quantifiable benefits in Year 1 but may have benefits that accrue and build over time. The timeframe of analysis will impact whether a project appears good or not, while the underlying reality may be more subtle or context-dependent. In the State of Colorado, the DOT and MPOs are addressing this challenge by calculating life-cycle benefits for projects for inclusion in the State CMAQ Reporter in addition to the Year 1 calculations reported to FHWA for inclusion in the CMAQ database.
Determining cost-effectiveness: SAFETEA-LU directs States and MPOs to give priority to (1) diesel retrofits and other cost-effective emissions reduction activities and (2) cost-effective congestion mitigation activities that provide air quality benefits. The determination of cost-effectiveness requires calculation of both the benefits and costs for each project. This is usually expressed as the amount in kilograms of air pollutant reduced divided by the cost of the project. In Medford, Oregon, project cost-effectiveness accounts for 16 percent of the total possible score during the project evaluation phase. The next section, on effective practices, provides greater detail on how the RVCOG in Medford awards points in determining which projects to fund.
A separate cost-effectiveness measurement is usually calculated for each of the identified pollutants within the air quality district. Different categories of projects are typically better at reducing different types of pollutants. For example, the CMAQ Evaluation and Assessment Phase I Final Report found that diesel retrofits can be some of the most cost-effective projects in reducing both ozone precursors and particulate matter (PM)12. Traffic flow improvements can be cost-effective in reducing carbon monoxide (CO) and ozone precursors. In the Pittsburgh area, SPC utilizes a set of standardized models to compare benefits and costs in order to develop a cost-effectiveness rating for each CMAQ candidate.
Evaluation and ranking of projects: Once the quantitative and qualitative measures for each project are documented, the evaluation and ranking of projects can be accomplished. Most MPOs incorporate qualitative measures into this step by giving bonus points for certain types of projects. The stated and publicized decision to favor certain types of projects over others gives project sponsors important information about what types of projects are likely to be funded. Each of the seven locations studied develops its own evaluation methodology, with different combinations of points and mixes of qualitative and quantitative measures. This flexibility enables tailoring of the program to meet local needs.
Once projects have been evaluated and ranked, the MPO policy board or its designated committee will typically review the list of eligible CMAQ projects to ensure a balanced funding program. This review will often look at the geographic distribution of project proposals and also will check to ensure that regional policy goals are met by the mix of recommended proposals. MPOs with many project submissions will often organize applications within project-type categories and then reevaluate and rank proposals within these categories to ensure fair and logical comparisons. These agencies have developed multiple evaluation methodologies tailored to each project-type category since, for example, a bicycle project has different assumptions and characteristics than an intersection improvement or diesel retrofit project.
Several of the MPOs have revised the types of project categories that are emphasized over the years to better serve changing needs. In Medford, for example, paving projects were highly ranked with use of quantitative measures because they led to quantifiable reductions in PM, which was the MPO's major air quality challenge. However, when diesel retrofits became a CMAQ funding priority in SAFETEA-LU, the board decided to emphasize these types of projects and revised its ranking process by awarding points explicitly for diesel retrofits.
Staff work with members of the Technical Advisory Committee (TAC) to review project proposals and to score and rank proposals with use of a single set of criteria. Each project can earn up to a total of 115 points on the basis of both quantitative and qualitative factors organized within 10 criteria categories that reflect the Policy Committee's CMAQ priorities.
| Project Evaluation Criteria | Maximum Points |
Points Awarded |
|---|---|---|
| Project results in CO/PM 10 reduction (contribution to overall reduction) | 20 | |
| Project is cost-effective (benefit/cost = kg reduction/ $ spent) | 20 | |
| Project results in long-term air quality improvement (effectiveness in 5 years, 10 years, etc.) | 15 | |
| Project demonstrates potential to reduce reliance on automobiles | 15 | |
| Project demonstrates potential to mitigate congestion | 15 | |
| Project helps to complete a multimodal transportation system | 10 | |
| Diesel retrofit project | 5 | |
| Project is in city limits or inside Urban Containment Boundary | 5 | |
| Project results in reduction of nitrogen oxide (NOx) and/or volatile organic compounds (VOCs) | 5 | |
| Innovation (e.g., new to the area, new technology) | 5 |
The Massachusetts EOT developed a series of air quality analysis worksheets to be used by each of the State's MPOs to evaluate CMAQ-eligible projects. The worksheets incorporate the emissions factors for each of the two air quality regions within the State and allow for changes to input variables, such as average speed and trip length, depending on regional conditions. Outputs of the worksheets include net emissions change and cost-effectiveness calculations for each type of emission. This allows for a comparison of project cost-effectiveness on a statewide basis.
At the Pittsburgh MPO, each CEC member is asked to evaluate the assumptions, calculations, and qualitative benefits submitted with applications for each project. The review imposes a reality check on initial project submissions to ensure greater accuracy and a better understanding of each project by the CEC members. SPC staff also support project review by quantifying each proposed project's impact on air quality, using a standardized set of analysis tools developed by PennDOT. Evaluation criteria include:
The CMAQ evaluation process also considers a set of ancillary factors to assess each project's deliverability, local commitment, and consistency with the policies and goals in the region's long-range plan, county and local comprehensive plans, and the region's congestion management process.
In 2000, the Colorado Transportation Commission expressed concern about the effectiveness of the CMAQ program in improving air quality and adopted a resolution (TC-807) to increase accountability for how CMAQ funds are spent. This led to the development of the CMAQ Reporter, a database maintained by CDOT that requires fund recipients to report annually on the effectiveness of their CMAQ expenditures. As a result, MPOs within Colorado are required to calculate the air quality benefits that will accrue over the lifetime of a CMAQ project.
NFRMPO has clearly established CMAQ evaluation criteria and an easy-to-understand, three-tiered scoring system to select projects. This includes short-term air quality benefits in Year 1 (50 percent of score), long-term air quality benefits in Years 2 to 5 (20 percent of score), and regional planning achievement (30 percent of score). Because the process of demonstrating air quality benefits can be contentious between transportation and environmental agencies, NFRMPO set up a CMAQ Evaluation Committee that includes representatives of both EPA and the Colorado Department of Public Health and Environment (CDPHE) as voting members. Using NFRMPO's established CMAQ evaluation criteria, the 14-member committee scores and then ranks proposed CMAQ projects for funding.
Another important success factor in developing a good CMAQ program is an organization's willingness and ability to reflect on, and change in response to, shifting local needs and conditions, new Federal guidance/legislation, or the evaluation results of project outcomes post-implementation.
Since the CMAQ program was first created by ISTEA13 in 1991, each of the subsequent Federal transportation bills, TEA-2114 and SAFETEA-LU15, has resulted in a shift of priorities within the program at the Federal, State, and regional levels. In order to effectively implement CMAQ, agencies had to be able to adapt to these changes. Several of the MPOs interviewed also discussed the value of conducting a broad-level assessment of the CMAQ program overall on a periodic basis to ensure that it supports shifting regional conditions and evolving regional policy goals.
Unlike other Federal transportation funding categories, projects using CMAQ funding are required to demonstrate air quality benefits in order to be eligible. Establishing CMAQ eligibility for proposed projects requires that agencies make assumptions upfront about air quality and congestion impacts as well as project costs and benefits. Given that CMAQ supports projects that are nontraditional or completely new to an MPO, however, it can be difficult for agencies to define the correct assumptions and to accurately predict impacts. To better understand actual project benefits, some of the MPOs interviewed have introduced post-project analysis and evaluation of CMAQ projects. Evaluation results help MPOs to refine assumptions and estimates for future projects and provide policymakers with better information for future decisions. Post-project evaluation allows historical benchmarks to be set and tracked for actual, rather than projected, congestion mitigation or air quality benefits and costs. These benchmarks make it easier for new sponsors to make assumptions and develop estimates for related nontraditional transportation projects. This practice also allows the MPO to show members of the public that they are receiving a demonstrable benefit from their tax dollars.
Several MPOs noted that post-project evaluation of CMAQ projects is being driven by project sponsors. In both Pittsburgh and Birmingham, it was noted that if CMAQ project sponsors want to be able to compete successfully in the next round of CMAQ funding, the sponsor must show that its previous projects achieved the projected benefits. In Boston, it is up to the sponsor of operations-related projects to undertake usage studies and report on the benefits. The project's future level of funding depends on demonstrating benefits commensurate with the usage projections. In the long run, this may favor projects for which benefits are easier to calculate upfront (e.g., diesel retrofits) over those in which benefits build over time (e.g., bicycle and pedestrian facilities, transit, and public outreach).
Post-project evaluation is expensive both in staff time and money, so it is sometimes seen as a luxury or as being done at the expense of other important work conducted at the MPO. While it is clear that post-project evaluation can help States and MPOs to better understand the costs and benefits of various project types, the costs associated with post-project evaluation are sometimes perceived as outweighing the benefits. However, CMAQ programs that are willing invest the resources needed to evaluate projects post-implementation and use the results to restructure and refocus their programs should be better able to meet their objectives moving forward.
In 2008, DRCOG staff developed a 12-page public document that lays out, in an accessible, plain-English narrative, the evolution of the MPO's 16-year history of using CMAQ funds. It includes how the agency's approach and priorities for using these funds have changed over the years in response to the changes in planning, air quality, and financial contexts in the Denver metropolitan area. The document highlights examples of the types of programs and projects funded by CMAQ in the past and identifies current DRCOG priorities. The guide is an excellent resource for helping new or potential project sponsors understand the regional CMAQ process.
After it delivers a recommended suite of CMAQ projects to the SPC board for inclusion in the TIP, the CEC holds an "after-action" debriefing to document both positive and negative aspects of the recently completed 2-year CMAQ project selection process. At the end of the debriefing, the CEC is dissolved. At the beginning of each future CMAQ cycle, the membership of the CEC is revisited, and new members and organizations have the opportunity to participate. The results of previous debriefings are used to improve processes for the next CMAQ cycle and to educate new committee members about CEC goals and functions.
In evaluating its CMAQ program, RPCGB found that its CMAQ projects were taking too much time to be designed and built. The MPO felt that its smaller CMAQ projects were being subjected to the same design standards as larger projects. Working with the Alabama Department of Transportation (ALDOT), RPCGB developed procedures to speed up the project review process. This included shifting responsibility for planning, design, and the bid process to the project sponsor. This has resulted in a shorter design and bidding period and has reduced project costs for these smaller-scale projects.
In San Francisco, the CMAQ is not operated as a stand-alone program but is viewed instead as just one funding source among many. As such, CMAQ funds are pooled with other funding sources in support of multimodal program areas supporting a broad range of project types. MPO staff observed the significance of CMAQ and its historical impact on regional policy development. The CMAQ program supported the agency's philosophical shift, over the past 2 decades, to consider improved air quality and quality of life as important factors in evaluating all transportation projects, not just those that are CMAQ-eligible. As a result, one staff member noted, MTC now conducts all programming with a "CMAQ mentality."
