Skip to content U.S. Department of Transportation/Federal Highway AdministrationU.S. Department of Transportation/Federal Highway Administration

Office of Planning, Environment, & Realty (HEP)
PlanningEnvironmentReal Estate

HEP Events Guidance Publications Awards Contacts

Interim Report to the U.S. Congress on the Nonmotorized Transportation Pilot Program SAFETEA-LU Section 1807

Chapter 4: Evaluation Approach and Issues

Previous Return to TOC Next

The enabling legislation requires the collection of data on changes in motor vehicle, nonmotorized, and public transit usage in the pilot communities. The Working Group developed the following documents to use in coordinating the collection of this information:

This chapter summarizes the approach to data collection, as reflected in these documents, describes the four phases of data collection, and summarizes issues and challenges encountered to date. As with the communications activities discussed in Chapter 4, pilot communities opted to pool a share of their authorizations to fund coordinated data collection and evaluation. While the legislation did not expressly fund this activity, each pilot community has undertaken the task of tracking and monitoring outcomes related to NTPP projects and nonmotorized activities.

Program Evaluation Plan

The PEP is an evolving plan that guides coordination and management of all aspects of evaluation. The Volpe Center and FHWA developed the PEP with input from and on behalf of the Working Group. The PEP provides a road map for efficient and comprehensive data collection and evaluation through the duration of the NTPP and identifies key technical aspects of evaluation to consider during specific phases of evaluation.

The Working Group will refine the PEP as the NTPP progresses. The goals of the PEP are to:

Themes

Prior to the development of the framework for Phase 2, the Working Group identified themes that are related to the NTPP goals. The themes, listed below, represent topics of significant importance that the communities identified as they developed their projects. As applicable, the communities will evaluate results related to the following topics:

Some of these themes involve measures of direct impacts of projects while others involve types of projects. To the extent possible, the framework identifies information or specific data items that will support conclusions related to these themes.

Four Phases of Date Collection

To manage the collection of data throughout the duration of the NTPP, the Working Group developed four phases of data collection, analysis, and reporting over the life of the NTPP:

Figure 4.1 illustrates the relationship among the phases of evaluation.

Figure 4.1: Phases of Evaluation

This figure displays the program evaluation stages. Phase 1 involves a baseline community wide survey. Phase 2.1 establishes a common framework to ensure data collection consistency, Phase 2.2 involves the data collection, Phase 3 involves 'after' data collection with a community-wide survey, and Phase 4 synthesizes these data. Other inputs include those from FHWA, the working group, communications, and results from other areas. These phases will help produce an Interim Report to Congress, a Final Report, and Other reports as necessary.

PHASE 1

Phase 1 involves the development and execution of a plan for collecting data at the community level before any projects were implemented. The data collected in Phase 1 will provide a "baseline" of travel behavior and attitudes prior to implementation of the projects to be funded in the four pilot communities. The baseline data can then be used to compare data collected in Phase 3 after the projects are implemented to identify changes. The Working Group developed a statement of work for Phases 1 and 3 to guide applicants' proposals for performing this work.

The pilot communities selected the University of Minnesota to perform this work. The University of Minnesota performed a baseline community-wide travel behavior survey between September and December 2006. The University of Minnesota selected Spokane, Washington, as a control community to help assess whether changes in the pilot communities from 2006 to 2010 might be affected by external factors unrelated to NTPP.

The research team developed a data collection plan. In line with this plan, the research team first mailed a short survey to a randomly chosen set of households in each region. The short survey contained a few questions and asked the respondent to agree to participate in the full survey. Based on their response to a particular question on the short survey, respondents were assigned one of four mode categories for being a potential respondent to the full survey. Those who agreed to participate in the full survey were telephoned or e-mailed a link to the full survey at a later time. The full survey took approximately 18 minutes to complete by telephone.

The Centers for Disease Control and Prevention (CDC) worked with the University of Minnesota to develop survey questions to gauge respondents' physical activity levels and attitudes toward physical activity. Responses to these questions will be used in evaluating how the NTPP improves health, one of the goals of the pilot program, through increased physical activity.

PHASE 2

Phase 2 involves the planning for and collection, analysis, and reporting of project specific "before" and "after" data in each community. To ensure consistent data collection and analysis in the four communities, The Volpe Center worked with the Working Group and its Evaluation Subgroup to develop a framework and set of protocols to guide the development of community-specific project evaluation plans. The framework had to be modest in scope, to reflect the limited resources available for evaluation, and focused to provide information on all projects as well as capture impacts of some of the most significant projects in each community.

The framework:

The framework provides a consistent approach to evaluation of infrastructure and non-infrastructure projects (e.g., training and marketing). Since the communities are making significant investments in both categories of projects, the evaluation effort must develop information on both.

Each community will develop a data collection plan in line with the framework, collect data on the basis of its plan, and conduct an analysis. The communities will decide how much of the Phase 2 work to perform themselves and how much to contract out to universities or consultants. The plans must be developed prior to project implementation to promote early thinking about how selected projects will support program goals and to ensure that baseline data are collected before projects are implemented. The plans should accommodate unique characteristics of each community's projects while adhering to overall program goals and schedules. To the extent possible, the four plans should be consistent and coordinated to support conclusions for the overall program.

Because it is impractical to collect quantitative data on impacts of all projects implemented, the Volpe Center and the Working Group devised three methods of measurement in the framework that can accommodate all project types. The methods, which apply equally to infrastructure and non-infrastructure projects, are aligned with the level of effort required to collect the data.

Each level builds on and incorporates the lower levels. While the communities will collect Level 1 data on all projects, each will collect Level 2 and 3 data only for at least five selected projects. The framework provides flexible criteria for the communities to use in screening projects to determine which are the most promising for thorough data collection and evaluation.

Level 1 is the simplest conceptually and least costly, and requires each community to develop descriptions of all projects, individually or in project type categories, including:

Level 2 requires counts of facility users (e.g., bicyclists and pedestrians) for at least five projects in each community. For non-infrastructure projects, such as promotional campaigns, training, and similar activities, counts refer to the number of participants and replace the estimated or expected level of use of the project in Level 1. The counts will be performed -- at a minimum -- before and after each selected project is implemented. The communities are encouraged to follow the count methodology developed for the National Bicycle and Pedestrian Documentation Project (NDP) and to contribute count data to the national database on nonmotorized transportation usage being developed for the NDP (more information on the NDP can be found at www.altaplanning.com).9

Level 3 -- The counts will measure changes in the number of users over time, but will not provide mode split information and other travel measures required to answer major questions of the enabling legislation. Accordingly, under Level 3 the communities will complete intercept or targeted surveys based on data and measures in Table 4.1. The surveys will also focus on individuals using a nonmotorized facility, or participating in a target group for a significant non-infrastructure intervention.

PHASE 3

Phase 3 is the follow up collection, analysis, and reporting of the community-wide "after" travel survey data (to be conducted using the same data collection plan used in Phase 1). The University of Minnesota will conduct the survey in 2010, after the communities implement their projects. This data will be collected in the four pilot communities, and in the control community (Spokane, WA).

PHASE 4

Phase 4 involves the compilation of the results from the phases of work described above. Work in this phase will be coordinated with other data collection, analysis, and synthesis, including consideration of information generated through communications activities (see Chapter 3), and other project and program evaluation aspects.

EVALUATION ISSUES AND CHALLENGES

Phases 1-3 deal with important challenges and issues that are common to similar efforts involving survey design, data collection, and program evaluation. The University of Minnesota identified the following challenges and issues pertaining to the collection of Phase 1 "before" community-wide survey data that were collected in fall 2006:

The Working Group anticipates that the above issues will also apply to the Phase 2 project-level data collection. In addition, other Phase 2 challenges are likely to include:

Table 4.1: Evaluation Methods and Measures for the NonMotorized Transportation Pilot Program

  G O A L S
  Decrease motor
vehicle usage
Increase nonmotorized transportation usage Increase public transportation usage Decrease congestion Connect to community activity centers Promote better health Decrease energy usage Promote a cleaner environment
M

E

A

S

U

R

E

S
Collect statistics on...
trip purpose1,2,3 trip purpose1,2,3 trip purpose1,2,3 volume4 population connected to activity centers5 ped/bike crashes and geographic dispersion5,6    
number of trips per day by mode1,2,3,6,7 number of trips per day by mode1,2,3,6,7 number of trips per day by mode1,2,3,6,7 delay4 inventory of facilities connected to activity centers5 number of newly active users1,2,3    
VMT1,2,3 miles biked1,2,3 public transit usage (miles)1,2,3          
# of people in vehicle1,2,3 miles walked1,2,3 proximity to nearest transit stop1,2,5          
vehicle ownership1,2 bicycle ownership1,2 bike on bus usage          
Use already collected data on...
          miles walked and biked1,2,3 VMT1,2,3 VMT1,2,3
          trip purpose1,2,3 bike/ped and transit usage1,2,3 bike/ped and transit usage1,2,3
Non- infrastructure projects10,11
Depending on timing, the evaluation of non-infrastructure projects can include the travel behavior measures above          
Note: these measures can be collected at two levels - communitywide and project area specific Note: using models and other tools,8 convert travel data above to calculate physical activity impacts, energy usage and savings, and air pollution

Methods For Collecting Statistics Above

  1. Surveying a Sample of General Population
  2. Surveying Users of the Facility
  3. Travel Diary
  4. Level of Service (volume and delay)
  5. Spatial Analyses/GIS
  6. Manual Counts
  7. Automated Counts
  8. Data Conversion/Modeling
  9. Policy/Plan Evaluations
  10. Planning Process Evaluation
  11. Awareness Surveys or Other Tools
Previous Return to TOC Next
Updated: 12/10/2014
HEP Home Planning Environment Real Estate
Federal Highway Administration | 1200 New Jersey Avenue, SE | Washington, DC 20590 | 202-366-4000