TCSP Grant Workshop Denver, Colorado
Track C: Evaluation - Building the Knowledge Base
Steve Grimm, Policy Analyst, Federal Railroad Administration, Washington, D.C.
William Lyons; U.S. Department of Transportation, Volpe Center, Cambridge, Massachusetts
John Suhrbier, Cambridge Systematics, Inc.
Christopher Porter, Cambridge Systematics, Inc.
Introduction to TCSP Evaluation
Evaluation is fundamental to the success of the TCSP Program, as expressed in the legislative intent of TEA-21. TCSP will support many worthwhile local experiments and new ideas combining transportation and land use. The greatest impact of TCSP on promoting livable communities, however, will be through building the knowledge base of state and local agencies, the private sector and the public about how transportation decisions can promote community preservation and improve quality of life. Through evaluation, the limited TCSP funds available for grants can leveraged to produce far-reaching national impacts. Ideally, insights from the 35 grantees in 1999 may benefit 100 or more communities, and similar compounding will be replicated throughout the five years of TCSP.
TCSP grantees have a major role to play in evaluation. Because TCSP funds are not adequate to allow independent evaluations of projects, primary responsibility for evaluation rests with individual grantees. FHWA will provide limited technical assistance to the greatest extent possible. To expand learning from TCSP, FHWA will complement project evaluations with related research and annual evaluations of the overall TCSP program. Program evaluation will be drawn primarily from insights gained through the project evaluations, for example, to examine trends among similar projects.
TCSP evaluations will foster learning about how innovative local alternative transportation investments and strategies can be integrated with land use to foster livable communities that balance economic growth with environmental quality.
The panel presented an overview of TCSP evaluation, fundamentals of evaluation methodologies, some case study examples, and joined workshop participants in an exchange of ideas and insights on the TCSP program and the role of evaluation.
TCSP Evaluation Plans
TCSP applications are selected in large part because they include a strong evaluation plan, indicating a commitment to undertake meaningful evaluations that will benefit a national audience. Grantees are responsible for conducting a systematic evaluation of their project. Undertaking a systematic evaluation of project results will benefit grantees as well as a broad national audience. Most TCSP projects involve planning or developing new transportation approaches and require involvement of new partners and broad community support. By demonstrating the strengths, weaknesses, and feasibility of the new concepts, evaluation can help secure necessary local support.
There is no single standard approach to be employed for TCSP evaluation. Evaluation plans will be tailored to the unique local conditions and objectives of each project. FHWA can offer broad guidance, but the details of evaluation, including specification of goals, outcomes, measures, data collection, and means of describing results are the responsibility of grantees. Because TCSP is a learning process, evaluation plans should focus on identifying what works and what does not. Participants discussed how can lessons be generalized to other situations, other parts of the community, and other regions.
Key aspects of an evaluation plan are described below.
Objectives of an evaluation: Plans should begin with a clear statement of the specific goals for the project. Evaluation should be designed to answer fundamental questions that might be asked by other communities interested in similar strategies.
What is different or innovative? What is the magnitude and nature of change? What factors were important? What worked and what did not?
What performance measures will be used to measure progress? Are they practical measures?
Who is affected? What is the distribution of impacts and costs? What is different, and what are the magnitude and nature of changes?
How has the project been developed and supported through a metropolitan transportation or other planning process?
What to evaluate: At a basic level evaluation should be of Process, Products, and Outcomes.
How does planning process as adapted for this project compare to the traditional process? How many partners were actually involved in substantive roles, and how were they involved? Is the project that emerged from the changed process different from projects from the traditional process?
Products can include major reports produced by a local transportation planning process. For example, twenty year long range plans, Transportation Improvement Programs, or land use plans can be examined to determine whether the project has demonstrable effects on actual decisions.
Outcomes can be the most important but most difficult aspect of evaluations because they are so dependent on time horizons of projects. Because of the small scale of many projects it may be difficult to isolate changes in travel and other direct outcomes. Also, because outcomes from transportation and land use interactions can be very long term, some may be outside the horizon of TCSP evaluations. It is important to start collecting base line data from the beginning of the project to allow later evaluation of longer term outcomes.
Examples of measures of outcomes include changes in land consumption per unit of development, percent of new dwelling units with walk access to transit, changes in mode split, or accessibility of business to intermodal transfer points. It might also be possible to use modeling techniques to forecast and estimate expected changes from projects, or control groups to analyze changes in behavior or perception over-time. Measurement of outcomes will need to be statistically rigorous.
Approaches to Evaluation: Evaluations are not easy and will require some investment of staff and budgets. There will be a broad range in evaluation budgets of proposals. Because TCSP applications had strict length limits, some proposals began with an initial evaluation plan. All projects should develop detailed plans as an initial step. Early planning for evaluation will help focus the project and ensure collection of baseline data.
Scope of Evaluation: The plan should identify factors to be considered. Typically this might include consideration of travel patterns, modes used, vehicle miles traveled, and other changes in activity patterns. Other factors include economic and community impacts, costs, and benefits. Institutional factors may involve how barriers to change were overcome.
Focused versus Comprehensive Evaluation: Projects that expect to evaluate an overwhelming number of factors are unlikely to succeed. It is more practical, realistic, and credible to focus on specific objectives and formulate and evaluate modest but interesting basic hypotheses.
Roles and Responsibilities: Plans must specify who does what among the participants involved in the proposal. Who has lead responsibility for evaluation? What are working relationships among participants in terms of providing budgets, staff, technical tools, data bases, or data collection? Who will produce evaluation products, including reports, manuals, web-sites, or conference papers?
There is no formula for who should conduct evaluations. In some cases, it may be a cooperative effort, with multi-agency coordination, data supplied or collected by one or more agencies, and dedicated staff or contractors performing analysis. Technical expertise might be available in-house, or provided by a university, private contractor, or other public agency.
FHWA Division staff are the major points of contact for grantees for information on evaluation as well as other aspects of the TCSP program. FHWA will continue to provide technical assistance to grantees and applicants for future years through publication of guidance, and conducting workshops such as this one. TCSP evaluation guidance, including: "Guidance for the Preparation of TCSP Evaluation Plans," "TCSP Case Study Examples," "Selected References Evaluating the Relationships Between Transportation and Land Use," and slides from this track are available on the TCSP web-site (http://www.fhwa.dot.gov/tcsp/). In addition, the USDOT/Volpe Center, which is assisting FHWA with TCSP evaluation, will be available for a limited amount of technical assistance directly to grantees. Complementing the project evaluations, FHWA will also conduct a research program and an annual evaluation of the overall TCSP program.
Evaluation Products: Evaluation plans should demonstrate commitments by grantees to produce useful products for a broad national audience that might include public agencies ranging from state Departments of Transportation to Metropolitan Planning Organizations, cities, counties, transportation providers, as well as environmental, education, or health agencies with a transportation interest; elected officials; developers and other private sector; advocacy groups; and the public. Written reports, tools, guidebooks, web-sites, or other media should be produced for dissemination to increase the knowledge base and allow others to replicate strategies.
Project Milestones: Plans will provide realistic schedules and commitments to complete evaluation products. To demonstrate early results, it may be necessary to provide interim reports or other products. Ideally, evaluation will be integrated into projects throughout their life, including design of measurable goals at the beginning, periodic data collection and analysis of results.
Data Sources: If plans did not provide specifics on data sources, this should be described in later more complete plans. It may be possible to make extensive use of existing sources, particularly if broad partnerships are formed for the project.
Assessment Techniques: Depending on local circumstance and project objectives, evaluations could be quantitative or qualitative. Focus groups or interviews may be particularly useful for process evaluations.
Importance of Objectivity: Objectivity is critical if evaluations are to be successful. In addition to successes, it may be equally valuable to document problems and barriers encountered and overcome so that others can learn from all experiences.
Importance of Different Perspectives: Innovative evaluations may come from different and non-traditional perspectives, for example, considering impacts of transportation strategies on housing, public health, air quality, or energy/greenhouse gas perspectives.
Evaluation Case Studies
The panel and participants discussed the "Our Town" and the following three case study examples (materials available at http://www.fhwa.dot.gov/tcsp/).
1) Planning for the upgrade of a rural highway corridor. Goals were to maintain the economic viability of rural communities and minimize disruption of agricultural lands either through upgrading existing roadways or expanding on a new right of way. Evaluation included use of business inventories, economic indicators, and economic modeling.
2) Transit Oriented Design strategies to encourage development around transit stations. The project considered zoning regulations and incentives, involvement of non-tradition partners, including developers. Evaluation looked at actual impacts measured in changes in development patterns and travel activity, with data collected through surveys.
3) Implementation of urban design, land use and Transportation Demand strategies for an urban office complex that is currently oriented toward automobile use and parking. The project considered changes to improve transit and pedestrian friendliness. The session used this application to examine the relationships between travel and land use at employment sites, examining changes in commute modes and mid-day travel. Data were collected through "before" and "after" surveys of employees and visitors.
Discussion with Participants
Workshop participants contributed a broad range of comments and questions on evaluation.
Q: How frequently should grantees submit evaluation reports?
A: It depends on the evaluation plan - early results would be very helpful.
Comment: Because formal evaluation is not a routine activity, the process is in itself a learning experience. Some applicants planned to involve partners in the evaluation process, which helps develop better evaluation plans, is innovative in itself, and will be a valuable local learning process and result.
Q: Were there any applicants who identified various community outcomes (such as providing better housing for all income levels) and indicators for making progress? Are there larger community objectives not necessarily linked to the project being evaluated?
A: It is key to consider the degree to which projects help achieve various regional objectives. Some applicants are proposing regional visioning and are at early stage; others are moving beyond visioning and move towards implementing an agreed-upon vision, perhaps focusing on difficult implementation and institutional steps, and how to move forward.
Q: The "Envision Utah" process expects to evaluate how public perceptions evolve. How can we do a benchmark study? How do we do a viable evaluation of public perceptions of the process?
A: You will not necessarily need quantitative measures. For example, focus groups would be one technique to examine changes in public perceptions, perhaps with a before and after meeting of the same focus groups. Another quantitative technique might involve a simple survey.
Q: How realistic is it for DOT to emphasize evaluations?
What happens to evaluations once completed? Will it be worth grantees' time?
A: Evaluations will be disseminated in various forms. The TCSP program will assist with dissemination by having workshops, publishing reports, or posting results on web-sites.
Q: Is it a problem if grantees cannot identify outcomes?
A: Evaluation plans should present what grantees expect to accomplish. When nothing is expected in the short term, that will be a less attractive project. Applicants must consider the local situation. Long term projects would not be expected to show short term outcomes. It might still be possible to provide interesting insights on process outcomes.
In looking at impacts or outcomes some applicants will have specific analytical products to determine whether expected outcomes are realistic. As part of evaluations, grantees may gain information for their own use in addition to producing insights for a national audience.
Comment: TCSP projects can add value to an existing process, accelerating otherwise expected outcomes. It is necessary to evaluate the value added component.
A: TCSP funds may support strengthening of existing processes. The value added to the larger project may be worthwhile to evaluate and is an appropriate part of an evaluation plan.
Comment: A viable outcome could be that some non-traditional partner is now participating. This could occur in a short time frame.
Comment: Consider being a spokesperson to disseminate successful results of projects and the TCSP program. Evaluation will need to actually document those results.
Q: How do you separate project impacts from those of other impacts? Could you use control groups, for example, analyzing two transit stations, one with TOD, one without?
A. This could be done with before and after data.
Q: What is involved with evaluating institutional factors?
A: Social science relationships, including how changes in attitudes and behaviors are managed at the individual, political, and ultimately, institutional or organizational level. Institution building can be part of the project as well of the evaluation.
Q: It is helpful to differentiate between process, product and outcome. Saginaw will have a design charrette, which will be a process, and use three different sets of measures for the evaluation.
A: Documenting how you deal with the big box redevelopment will provide valuable insights for other communities dealing with similar issues.
Q: If your plan targets a mode share, develop information (such as bike/pedestrian) to measure.
Would changes in family auto ownership be an appropriate measure over 5 years or 10 years?
A: Yes; one variant, if people are moving into an area, is the rate of continuing auto ownership. If they arrive with the average 1.6 vehicles, are they divesting themselves of one or more vehicles.
Q: It is difficult to get local people to come to meetings to discuss issues such as up-zoning. Is it relevant to quantify the distribution and diversity of the population in your project area, and of the involvement of the population?
A: Yes, but it may only be a beginning. You also should consider impacts reflected in key documents to measure the depth of interaction and buy-in from the participants.