Interim Report to the U.S. Congress on the Nonmotorized Transportation Pilot Program SAFETEA-LU Section 1807

Chapter 4: Evaluation Approach and Issues

The enabling legislation requires the collection of data on changes in motor vehicle, nonmotorized, and public transit usage in the pilot communities. The Working Group developed the following documents to use in coordinating the collection of this information:

• Program Evaluation Plan (PEP)
• Statement of Work for Phases 1 and 3 community-wide data collection
• Framework for Phase 2 project level data collection

This chapter summarizes the approach to data collection, as reflected in these documents, describes the four phases of data collection, and summarizes issues and challenges encountered to date. As with the communications activities discussed in Chapter 4, pilot communities opted to pool a share of their authorizations to fund coordinated data collection and evaluation. While the legislation did not expressly fund this activity, each pilot community has undertaken the task of tracking and monitoring outcomes related to NTPP projects and nonmotorized activities.

Program Evaluation Plan

The PEP is an evolving plan that guides coordination and management of all aspects of evaluation. The Volpe Center and FHWA developed the PEP with input from and on behalf of the Working Group. The PEP provides a road map for efficient and comprehensive data collection and evaluation through the duration of the NTPP and identifies key technical aspects of evaluation to consider during specific phases of evaluation.

The Working Group will refine the PEP as the NTPP progresses. The goals of the PEP are to:

• Establish how all aspects of the NTPP fit together, including the relationship between project evaluations and evaluation of the overall program.

• Provide a framework for how evaluation reports to Congress will be organized, including expected content, topics, and themes.

• Coordinate key elements of evaluation, including roles and responsibilities of the Working Group, FHWA, the Volpe Center, and contractors.

• Provide a management document or "blueprint" for evaluation, encouraging efficient, consistent, and coordinated evaluation that results in objective and insightful reports.

• Provide a dynamic and evolving plan to be updated as the FHWA and Working Group make program decisions.

Themes

Prior to the development of the framework for Phase 2, the Working Group identified themes that are related to the NTPP goals. The themes, listed below, represent topics of significant importance that the communities identified as they developed their projects. As applicable, the communities will evaluate results related to the following topics:

• Improving safe access;

• Improving public health;

• Working with land use policy and transportation planning processes;

• Leveraging resources;

• Improving connections to other transportation modes as part of an overall transportation system, with an emphasis on links to public transit; and

• Raising public awareness.

Some of these themes involve measures of direct impacts of projects while others involve types of projects. To the extent possible, the framework identifies information or specific data items that will support conclusions related to these themes.

Four Phases of Date Collection

To manage the collection of data throughout the duration of the NTPP, the Working Group developed four phases of data collection, analysis, and reporting over the life of the NTPP:

• Phase 1: development and administration of a baseline community-wide travel behavior survey to be executed prior to project implementation.

• Phase 2: collection of "before" and "after" data for projects within each community.

• Phase 3: application of the same community-wide travel behavior survey used in Phase 1, to be performed in 2010 to capture travel changes after projects are implemented.

• Phase 4: synthesis and analysis of the data collected and results.

Figure 4.1 illustrates the relationship among the phases of evaluation.

Figure 4.1: Phases of Evaluation

PHASE 1

Phase 1 involves the development and execution of a plan for collecting data at the community level before any projects were implemented. The data collected in Phase 1 will provide a "baseline" of travel behavior and attitudes prior to implementation of the projects to be funded in the four pilot communities. The baseline data can then be used to compare data collected in Phase 3 after the projects are implemented to identify changes. The Working Group developed a statement of work for Phases 1 and 3 to guide applicants' proposals for performing this work.

The pilot communities selected the University of Minnesota to perform this work. The University of Minnesota performed a baseline community-wide travel behavior survey between September and December 2006. The University of Minnesota selected Spokane, Washington, as a control community to help assess whether changes in the pilot communities from 2006 to 2010 might be affected by external factors unrelated to NTPP.

The research team developed a data collection plan. In line with this plan, the research team first mailed a short survey to a randomly chosen set of households in each region. The short survey contained a few questions and asked the respondent to agree to participate in the full survey. Based on their response to a particular question on the short survey, respondents were assigned one of four mode categories for being a potential respondent to the full survey. Those who agreed to participate in the full survey were telephoned or e-mailed a link to the full survey at a later time. The full survey took approximately 18 minutes to complete by telephone.

The Centers for Disease Control and Prevention (CDC) worked with the University of Minnesota to develop survey questions to gauge respondents' physical activity levels and attitudes toward physical activity. Responses to these questions will be used in evaluating how the NTPP improves health, one of the goals of the pilot program, through increased physical activity.

PHASE 2

Phase 2 involves the planning for and collection, analysis, and reporting of project specific "before" and "after" data in each community. To ensure consistent data collection and analysis in the four communities, The Volpe Center worked with the Working Group and its Evaluation Subgroup to develop a framework and set of protocols to guide the development of community-specific project evaluation plans. The framework had to be modest in scope, to reflect the limited resources available for evaluation, and focused to provide information on all projects as well as capture impacts of some of the most significant projects in each community.

The framework:

• Ensures that project data collected and evaluated by each community are consistent with overall program goals and evaluation, as reflected in the PEP;

• Supports qualitative and quantitative assessments of projects;

• Improves the quality, consistency, and relevance of community-level project evaluations as key inputs to program reports;

• Assists the communities in contracting for local services; and

• Provides a sequence of activities, timeline, and process for coordination to promote efficient data collection and ensure that balanced and helpful information is available for the final report.

The framework provides a consistent approach to evaluation of infrastructure and non-infrastructure projects (e.g., training and marketing). Since the communities are making significant investments in both categories of projects, the evaluation effort must develop information on both.

Each community will develop a data collection plan in line with the framework, collect data on the basis of its plan, and conduct an analysis. The communities will decide how much of the Phase 2 work to perform themselves and how much to contract out to universities or consultants. The plans must be developed prior to project implementation to promote early thinking about how selected projects will support program goals and to ensure that baseline data are collected before projects are implemented. The plans should accommodate unique characteristics of each community's projects while adhering to overall program goals and schedules. To the extent possible, the four plans should be consistent and coordinated to support conclusions for the overall program.

Because it is impractical to collect quantitative data on impacts of all projects implemented, the Volpe Center and the Working Group devised three methods of measurement in the framework that can accommodate all project types. The methods, which apply equally to infrastructure and non-infrastructure projects, are aligned with the level of effort required to collect the data.

Each level builds on and incorporates the lower levels. While the communities will collect Level 1 data on all projects, each will collect Level 2 and 3 data only for at least five selected projects. The framework provides flexible criteria for the communities to use in screening projects to determine which are the most promising for thorough data collection and evaluation.

Level 1 is the simplest conceptually and least costly, and requires each community to develop descriptions of all projects, individually or in project type categories, including:

• The length, scope, or number of projects being implemented.
• The location or geographic extent of project.
• The design of the project.
• The estimated cost of project per unit.
• The purpose of the project.
• The estimated or expected level of use of the project.
• The period of performance.
• Whether the activity is new or a continuation of an existing activity.

Level 2 requires counts of facility users (e.g., bicyclists and pedestrians) for at least five projects in each community. For non-infrastructure projects, such as promotional campaigns, training, and similar activities, counts refer to the number of participants and replace the estimated or expected level of use of the project in Level 1. The counts will be performed -- at a minimum -- before and after each selected project is implemented. The communities are encouraged to follow the count methodology developed for the National Bicycle and Pedestrian Documentation Project (NDP) and to contribute count data to the national database on nonmotorized transportation usage being developed for the NDP (more information on the NDP can be found at www.altaplanning.com).⁹

Level 3 -- The counts will measure changes in the number of users over time, but will not provide mode split information and other travel measures required to answer major questions of the enabling legislation. Accordingly, under Level 3 the communities will complete intercept or targeted surveys based on data and measures in Table 4.1. The surveys will also focus on individuals using a nonmotorized facility, or participating in a target group for a significant non-infrastructure intervention.

PHASE 3

Phase 3 is the follow up collection, analysis, and reporting of the community-wide "after" travel survey data (to be conducted using the same data collection plan used in Phase 1). The University of Minnesota will conduct the survey in 2010, after the communities implement their projects. This data will be collected in the four pilot communities, and in the control community (Spokane, WA).

PHASE 4

Phase 4 involves the compilation of the results from the phases of work described above. Work in this phase will be coordinated with other data collection, analysis, and synthesis, including consideration of information generated through communications activities (see Chapter 3), and other project and program evaluation aspects.

EVALUATION ISSUES AND CHALLENGES

Phases 1-3 deal with important challenges and issues that are common to similar efforts involving survey design, data collection, and program evaluation. The University of Minnesota identified the following challenges and issues pertaining to the collection of Phase 1 "before" community-wide survey data that were collected in fall 2006:

• Response rates were lower than anticipated. The statement of work for Phases 1 and 3 set a goal of 100 survey respondents for each mode (car, transit, walk, and bicycle) in each community to provide a sample resulting in a 95 percent confidence interval and a margin of error of ± 10 percent. However, the University of Minnesota was unable to find 100 respondents for all modes in each community for a variety of reasons, including that in some cases, there was low usage of some modes in some communities (for example, transit usage in Sheboygan County). Accordingly, the statistical level of confidence for some observations is lower than planned.

• Self-reporting bicycling and walking travel can have two problems that can impact the recorded rates of bicycling and walking:

  1. The definition of walking trips. Travel surveys can undercount walking because some respondents do not think of walking as a legitimate mode of travel, and omit walk trips in reports of daily travel. Also, it is not always clear what constitutes a walking trip -- surveys may not capture walking from one store in the mall to another, walking five blocks from home to the bus, walking the dog, or stopping at a store during a dog walk. For the purposes of physical activity, all of these walk trips are important. Survey instruments must be carefully designed to capture all relevant walking trips. Discrepancies in definition make it difficult to compare the results of different surveys. The research team addressed this issue by developing some questions that parallel those used in other surveys, such as the CDC's Behavioral Risk Factor Surveillance System (BRFSS).

  2. Survey respondents can consider walking and bicycling as "virtuous behaviors," leading to a degree of overestimation or "halo effect."

The Working Group anticipates that the above issues will also apply to the Phase 2 project-level data collection. In addition, other Phase 2 challenges are likely to include:

• Count accuracy. For infrastructure projects, no one location, date, or time will capture all of the use of the project for which data is to be collected. The NDP recommends times and methods that should capture a large number of project users. For projects that extend significant distances, it will be possible to conduct counts and distribute surveys only at one point along the project's path. The pilot communities must select the best location for each infrastructure project to perform surveys and counts.

• Weather implications. Weather will influence the number of people walking or bicycling on any given day. Weather conditions should be carefully recorded to determine possible effects on nonmotorized travel, and to make any necessary adjustments later.

• Control areas. If possible, each pilot community should select a control area that is likely to be unaffected by projects implemented by the NTPP, to identify possible area-wide factors that affect travel choice, including gas prices, concurrent community-wide physical activity initiatives, and even extreme weather.

Methods For Collecting Statistics Above

1. Surveying a Sample of General Population
2. Surveying Users of the Facility
3. Travel Diary
4. Level of Service (volume and delay)
5. Spatial Analyses/GIS
6. Manual Counts
7. Automated Counts
8. Data Conversion/Modeling
9. Policy/Plan Evaluations
10. Planning Process Evaluation
11. Awareness Surveys or Other Tools