U.S. Department of Transportation
Federal Highway Administration
1200 New Jersey Avenue, SE
Washington, DC 20590

Skip to content
Facebook iconYouTube iconTwitter iconFlickr iconLinkedInInstagram

Federal Highway Administration Research and Technology
Coordinating, Developing, and Delivering Highway Transportation Innovations

This technical report is an archived publication and may contain dated technical, contact, and link information.
Back to Publication List        
Publication Number:  FHWA-HRT-16-082    Date:  August 2017
Publication Number: FHWA-HRT-16-082
Date: August 2017


Federal Highway Administration Research and Technology Evaluation: National Household Travel Survey Program Final Report

2. Evaluation Design

Circle Intersection ImageTo assist the team in developing an evaluation strategy and plan, the logic model shown in Figure 1 was constructed. A logic model is a tool used to visualize the relationship between program components (i.e. inputs, activities, outputs, outcomes, and impacts). It is not intended to be a comprehensive or linear description of all program processes and activities but is designed to make explicit how program stakeholders expect program activities to affect change. While the evaluation questions seek to uncover the effect of program inputs and activities, the logic model shows how each step in the process plays a role in the creation of outcomes and impacts.

The NHTS logic model describes how FHWA and State Add-on funding is used to support the NHTS staff and its team of contractors as they leverage materials, experiences, and lessons learned from previous travel surveys to develop, execute, and report on the next NHTS. These inputs and activities produce fundamental outputs such as datasets, tools, and user guidance documents as well as indepth summary publications and topical briefs that are released to the user community. This user community creates the short-term usage outcomes and outputs as it actively employs the NHTS data and findings, while the NHTS team supports and responds to their needs. The NHTS data, along with outputs developed by the user community, feed into the decisionmaking system that yields projects, policies, and regulations, creating long-term impact in transportation and beyond.

This evaluation demonstrates the value of the NHTS Program through the examination of select areas of the logic model. These areas are described by the key outcome and impact areas (Table 1).

Figure 1. Diagram. NHTS Logic Model. Logic model with five columns: inputs, activities, outputs, short-term outcomes, and medium/long-term outcomes. The inputs column lists the following: FHWA R & T funding, FHWA Headquarters staff, State transportation department Add-on funds, feedback from users, survey research community expertise, contractor support, and lessons learned from previous surveys. The activities column lists the following: work planning, OMB approval process, stakeholder consultation, survey planning, survey fielding, data cleaning/prep, tool development, publicizing data and tool availability, method review for adequacy and improvement, FHWA contractor oversight, and addressing user input. The outputs column lists the following: survey data; website tools; documents detailing stakeholder opinions; list of users of NHTS data, methods, and citations; website statistics; guidance and awareness publications; and new methods and techniques. The short-term outcomes column includes the following: use of data by FHWA Office of Policy; use of data by researchers in transportation, public health, etc.; local/regional modeling; quantification of developing trends; lessons learned; and responsiveness to user community. The medium/Long-term impacts column lists the following: changes in funding for transportation initiatives/policy, changes in project selection, changes in land use or related regulation, and unspecified outcome based on health researchers’ activity.

Figure 1. NHTS Logic Model.


2.1   Evaluation Questions and Key Performance Measures

The Volpe team identified the following four impact areas that mapped to the last two sections of the logic model:

For each impact area, evaluation questions were developed along with key performance measures needed to inform these questions.

The first two outcomes pertain to how the NHTS data are used and their impact as follows:

Breadth and depth of NHTS usage (short-term outcome) seeks to uncover who the NHTS users are (e.g., Government users, academics, consultants, media, industry, etc.) and the range of research questions and topics they are addressing with the NHTS data—both within transportation and in connected fields. In addition, this evaluation topic explores the type and form of NHTS data being used (e.g., datasets, online tools, and publications).

NHTS impact on policy, project and regulatory decisions (long-term impact) demonstrates how NHTS data and outputs inform decisionmaking with respect to policies, legislation, and projects. This evaluation area provides a qualitative assessment of the role of NHTS data, highlighting key examples or cases. The impacts of NHTS data on policy, legislation, and project outcomes is explored at the Federal, State, and local levels; moreover, impacts are assessed not only in the field of transportation, but in other connected fields, such as health, energy, and the environment.

The other outcomes pertain to the NHTS support of its user community and the lessons learned from past NHTS efforts.

NHTS Responsiveness to its user community seeks to understand how responsive the NHTS Program has been to its user community. The measures assess multiple factors that represent responsiveness, including the number of ways in which NHTS solicits feedback and the frequency of its outreach. In addition, this evaluation area provides a qualitative assessment of the changes that the NHTS Program has made in response to user feedback, including changes to the 2016 survey, as well as changes in other outputs or tools. These evaluation questions will deliver an assessment that can guide the NHTS, as well as other programs, in their feedback systems.

The final outcome area, challenges and lessons learned, is a compilation of the challenges and lessons learned in conducting the NHTS surveys, as identified by current and previous Program Managers and the Chief of Travel Monitoring and Surveys Division.[5] The list is meant to be instructive, highlighting issues or topics that may serve as obstacles in the planning, administration, and outreach of the NHTS surveys. By addressing these challenges, FHWA can improve the survey process and outputs, with the long-term goal of increasing the impact of the data in policy and program development. In addition, this evaluation area also presents challenges and lessons learned related to the Add-on program, based on feedback received from a limited set of the Add-on partners.

Table 2 summarizes the key evaluation questions and performance metrics for each impact area of the evaluation. There were four main outcome/impact areas: breadth and depth; policy, project, or regulatory decisionmaking; responsiveness to community; and challenges and lessons learned. The evaluation questions are based on determining how well NHTS serves its user community and whom that community includes.

Table 2. Evaluation questions and performance measures by impact area.

Outcome/Impact Area Evaluation Question Key Performance Measure
Breadth and Depth: Website Analysis

To what extent do researchers visit the NHTS website to access information, including the following:

  • Datasets.
  • User Guidance Documents.
  • Online Analysis Tools.
  • Statistics on visitors, visits, and total page views per month.
  • Number of page views (data center, publications, and online tools).
  • Downloads of data sets, user guidance documents, and publications.
  • Online tool activations.
Breadth and Depth: Compendium and Presentation Analysis

To what extent does the NHTS do the following:

  • Serve a range of user types, including Federal staff, States/MPOs, and academics.
  • Inform a range of fields.
  • Inform range of transportation related topics.
  • Number of citations by user type.
  • Number of NHTS citations by field.
  • Number of citations by various transportation topics.
Policy, Project or Regulatory Decisionmaking: Transportation Focused

To what extent and in what way does NHTS data do the following:

  • Inform transportation policy decisions and rulemaking.
  • Inform transportation project selection and/or project modification.
  • Examples of policy decisions informed by NHTS.
  • Examples of regulatory actions informed by NHTS.
  • Qualitative measure of the role.
  • Role NHTS played in policy and regulatory actions.
Policy, Project or Regulatory Decisionmaking: Non-Transportation Focused

To what extent does NHTS impact non-transportation project selection and/or lead to project modification?
  • Examples of projects informed by the NHTS.
  • Qualitative measure of role NHTS played in project decisionmaking.
NHTS Responsiveness to its User Community

To what extent does NHTS use multiple methods to solicit feedback from its user community?
In what ways has NHTS made changes based on user feedback to the following:

  • Its 2016 survey based on feedback from its user community.
  • Its outreach (e.g., tools, publications, etc.) based on feedback from its user community.
  • Number of different feedback methods used.
  • Frequency with which NHTS reaches out to user community.
  • Qualitative assessment of the following:
    • Effort to collect, process, and act on user feedback.
    • Changes made to 2016 survey.
    • Changes made to outreach.
Challenges & Lessons Learned

What challenges and lessons learned has NHTS identified regarding the following:

  • Survey planning.
  • Survey administration.
  • Effective outreach methods.

What challenges have Add-on partners identified?

  • Compilation of challenges and lessons learned on the following:
    • Survey planning process.
    • Survey administration.
    • Survey outreach.
  • Compilation of Add-on partner challenges.


2.2   Evaluation Methodology

The data collection that Volpe conducted was primarily qualitative in nature, focusing on a literature search and document review, indepth interviews, and email contacts to provide information for the evaluation. Data from these sources are used in the form of examples and cases to support all of the outcome and impact areas.

Two additional sources of information were provided by the NHTS. The first was the NHTS Compendium of Uses NTHS Compendium of Uses*. Volpe accessed this categorized collection of publication abstracts using the NHTS website. The second source was the monthly NHTS Website Usage Report, provided by Oak Ridge National Laboratory for the period from July 2013–May 2015.[6] Each source was analyzed separately to inform the breadth and depth of use outcome.

*Revised 4/18/2018

This section describes the data collection and evaluation methodologies in detail and links them to specific outcome and impact areas and evaluation questions.

2.2.1 Literature Search and Document Review

Volpe conducted a literature search to gain an initial understanding of the NHTS Program, its stakeholders and users, the survey development process, outreach activities, and outputs. This information came from multiple sources, including the NHTS website,(16) which provided detailed information on NHTS history, goals, and topics covered. The website was also a primary source for downloading NHTS publications, presentations, conference agendas, and external publications, and for viewing user guidance documents, datasets, and online tools. Additional documents were found using web search engines, including Google®,(17) the TRB database,(18) and Volpe library resources. Findings from the initial literature review were used to guide question and topic development for indepth interview guides. These findings were also used as a source of general NHTS information for all of the outcome/impact areas.

A more substantial document review was used to provide detailed information for several of the outcome/impact areas. Documents obtained during the literature search were used along with documents identified during indepth interviews and email contacts. Many of these additional reports, presentations, and models developed with NHTS data were not available online or using library resources. These included materials from Federal and State governments, lead users of the NHTS, and members of the NHTS task force. The documents collected from all sources provided examples and cases to inform the breadth and depth of use; user responsiveness; and policy, project, and regulatory Impact outcomes. The process used to extract this information is discussed in the case categorization section.

2.2.2 Indepth Interviews and Email Contacts

Semi-structured, indepth interviews and email contacts provided much of the information used to inform evaluation questions under the four outcome/impact areas. Volpe worked with the NHTS Program Manager to develop an initial interviewee list and gather contact information. Initial interviews with NHTS staff and lead users yielded additional interviewee contacts via snowball sampling. Because of budgetary constraints and a wealth of interview candidates, primary data collection was split into two groups: indepth interviews (longer) and email contacts (brief). Volpe completed 23 interviews and 18 email contacts (41 total). See Appendix A for a complete list of interviews and email contacts. Indepth Interviews

Interviews were conducted with 23 respondents representing four NHTS stakeholder groups: NHTS Program Managers (current/former) and Office of Policy staff, members of the NHTS task force, lead NHTS users (transportation and other fields), and Add-on partners. The interviews were conducted either in person or by phone and generally lasted 60–90 minutes. Volpe developed a detailed interview guide for each group consisting of 15–20 open-ended questions (with detailed probes). The interview guides included questions that were relevant to each stakeholder group, typically covering a minimum of three of the four outcome/impact areas. Question topics asked for each impact area are shown below. Email Contacts

Volpe contacted 18 respondents with a small subset of the questions from the indepth interviews. These respondents generally represented NHTS user sub-groups including DOT staff, academics, and NHTS task force members. The questions were distributed by email and collected by email or phone, as needed. The four to five open-ended questions covered only two outcome/impact Areas: breadth and depth of NHTS use and policy, project, and regulatory decisionmaking. An example of the email sent to interviewees is included in Appendix C.

Both the indepth interviews and email contacts requested supporting documents, when available, to provide specific examples and cases of NHTS use. All interviewees were asked questions related to breadth and depth of NHTS use; policy, project, or regulatory decisionmaking; and NHTS responsiveness to its user community. However, questions related to challenges and lessons learned were asked of only a subset of the interviewees, including NHTS Program Managers (current and previous) and other staff from FHWA’s Office of Policy.

Table 3. Interview topics by interview audience and type.

Indepth Interview topics Program Manager/Office of Policy Task Force Members Lead Users – Transportation & Non-transportation Add-on Partners Email Contacts
Background/General X X X X X
Breadth and Depth of Data Use X X X X X
User Responsiveness X X X X  
Challenges & Lessons Learned X     X  


2.2.3 Interview Analysis

The interviews and email contacts yielded detailed notes with examples and key quotes for each outcome/impact area covered in the interview. These detailed examples and quotes were entered into a spreadsheet under the appropriate outcome/impact area. Additional information describing each example or quote, such as user type, field, topic, or level of government was also coded to provide the context for the example or quote. These examples and quotes then served as raw materials to be used to inform the evaluation questions.

2.2.4 Case Categorization

In addition to the anecdotal examples and quotes pulled from the indepth interviews and email contacts, the Volpe team assembled a more formal case database used to provide concrete examples of how NHTS was used to inform policy, project, and regulatory decisions. The documents obtained through the literature search, indepth interviews, and email contacts provided such cases.

The cases were organized by the impact area they supported, and included additional information to allow sorting by topic, purpose, type of decision supported and applicable level of government. In building its database Volpe sought to understand both what NHTS data were used to support decisions and in what form the data were fed into the decisionmaking system. The cases can be described as providing decisionmaking support either directly or indirectly.

Direct Support. This category describes cases where the NHTS is used in the form of descriptive data or trends. NHTS data and/or findings are delivered to decisionmakers in the form of statistics, trends, or travel topics by way of reports, presentations, and testimony.

Indirect Support. The second category includes cases where the NHTS supports decisions indirectly, as an input (along with other data) into models, calculations, and research that are used in the decisionmaking processes. An example of indirect support includes Highway Statistics Vehicle Miles (VM-1), which uses NHTS vehicle occupancy data.

2.2.5 Analysis of NHTS Website Statistics

The analysis of the Website usage reports from July 2013 to May 2015 is a descriptive analysis of the monthly usage numbers. ORNL compiles this data for the NHTS Program and provided Volpe with a copy of its monthly reports.

This analysis assessed monthly usage in multiple ways to get a sense of the overall magnitude of NHTS usage. It examined the website outputs individually and in relation to one another. The outputs include the following:

The analysis also attempted to discern any trends or usage patterns from the data by plotting the monthly usage statistics during the 23-month period. Unfortunately, data prior to July 2013 were not available, so it is not possible to discern changes in website usage because of the release of NHTS data. Volpe can provide only a very limited snapshot of trends.

2.2.6 Analysis of NHTS Compendium of Uses NTHS Compendium of Uses*

To inform the assessment of the Breadth and Depth of NHTS data use, Volpe analyzed the 2014 NHTS Compendium of Uses NTHS Compendium of Uses*,(1) a document developed by the NHTS staff that provides a comprehensive catalogue of publications citing the NHTS in that year. Volpe based its analysis on the existing classification system of 11 categories but tailored it after conducting a pilot analysis indicating that additional classification information was needed to answer the evaluation questions. It should be noted that while the Compendium is the best available source on published NHTS documents, it does not provide an exhaustive compilation of its uses. Through document reviews and interviews, Volpe learned that NHTS users also develop documents and reports, models and statistics, and other outputs that are not published or publicly distributed, and hence are not included in the Compendium.

*Revised 4/18/2018 Field and Topic Analysis

After completing a pilot analysis consisting of categorizing a sample of publications from each original subject area in the 11 Compendium categories, Volpe moved from the original classification system to a two-tiered system made up of fields and transportation topics. Some original categories became fields, others remained as topics, and new fields and topics were added to acknowledge emerging NHTS uses. Table 4 shows how the original Compendium categories fit into the two-tiered structure.

Table 4. New classification system for 2014 compendium analysis.

Original Categories Fields Transportation Topic Description of Field/Topic
N/A - Field Added F1. Health - Measuring, protecting, or improving overall well-being of communities of people.
N/A - Field Added F2. Transportation - Act or means of moving goods or people.
1. Energy Consumption F3. Energy - Measuring resources, mechanisms, and impacts for storing and using power supplies.
2. Environment F4. Environment - Measuring the impact on natural surroundings, including land use and air quality.
3. Survey, Data Synthesis, and Other Applications F5. Survey Methods, Administration, and Data Analysis - Using, evaluating, or comparing data collection and analysis methodologies.
4. Demographic Trends - T1. Demographic and Travel Trends Research on travel behavior (including among subgroups) during time.
5. Trend Analysis and Market Segmentation - T1. Demographic and Travel Trends Research on travel behavior (including among subgroups) during time.
6. Travel Behavior - T2. Travel Characteristics and Behavior Environments or triggers that influence traveler actions.
7. Transit Planning - T3. Transit Planning Transit related research and operations.
8. Traffic Safety - T4. Traffic Safety Infrastructure and design of transportation systems impacting level of traveler safety (including bike and pedestrian).
9. Policy and Mobility - T5. Policy and Mobility Planning, programming, and related policies (e.g., gas tax, freight).
10. Special Population Groups - T6. Special Populations Research surrounding specific groups of people (e.g., elderly, children).
11. Bicycle and Pedestrian Studies - T7. Bike and Pedestrian Studies All nonmotorized transportation issues.
N/A - Topic Added - T8. Innovative Technologies Research advancing transportation systems (e.g., electric vehicles, mobile devices, etc.).


A total of 313 publications were analyzed from the 2014 Compendium.(1) Each publication included an abstract and citation used to complete this analysis. For each publication, the categorization was completed in two parts. First, based on a review of the abstract, Volpe assigned the publication a primary field. Second, Volpe determined the transportation topic(s) represented in each publication regardless of field.

To help give focus to the NHTS use in a publication, Volpe assigned only one primary field to each publication. The fields show how the NHTS provides information to a variety of disciplines beyond transportation. While only one field was selected, multiple transportation topics could be selected for each publication. Within the transportation field, Volpe further designated the topics as primary and secondary when multiple topics were identified.

Another important component of use includes identifying the range of users developing research, reports, and presentations using these data. Looking at these users allows us to better understand how the NHTS influences decisionmaking at the local, regional, and national level. For this section, Volpe conducted two types of analysis: first, analyzing the lead authors of the publications listed in the 2014 Compendium,(1) and second, analyzing the lead presenters at major NHTS conferences,(5) workshops, and presentations (2004–2015).

5 While other interviewees were not directly asked a question about challenges and lessons learned, they did provide comments on these topics that are included in the findings.

6 Website statistics are available through an internal reporting system. 



Federal Highway Administration | 1200 New Jersey Avenue, SE | Washington, DC 20590 | 202-366-4000
Turner-Fairbank Highway Research Center | 6300 Georgetown Pike | McLean, VA | 22101