Federal Highway Administration Research and Technology Evaluation: National Household Travel Survey Program Final Report

2. Evaluation Design

To assist the team in developing an evaluation strategy and plan, the logic model shown in Figure 1 was constructed. A logic model is a tool used to visualize the relationship between program components (i.e. inputs, activities, outputs, outcomes, and impacts). It is not intended to be a comprehensive or linear description of all program processes and activities but is designed to make explicit how program stakeholders expect program activities to affect change. While the evaluation questions seek to uncover the effect of program inputs and activities, the logic model shows how each step in the process plays a role in the creation of outcomes and impacts.

The NHTS logic model describes how FHWA and State Add-on funding is used to support the NHTS staff and its team of contractors as they leverage materials, experiences, and lessons learned from previous travel surveys to develop, execute, and report on the next NHTS. These inputs and activities produce fundamental outputs such as datasets, tools, and user guidance documents as well as indepth summary publications and topical briefs that are released to the user community. This user community creates the short-term usage outcomes and outputs as it actively employs the NHTS data and findings, while the NHTS team supports and responds to their needs. The NHTS data, along with outputs developed by the user community, feed into the decisionmaking system that yields projects, policies, and regulations, creating long-term impact in transportation and beyond.

This evaluation demonstrates the value of the NHTS Program through the examination of select areas of the logic model. These areas are described by the key outcome and impact areas (Table 1).

Figure 1. NHTS Logic Model.

2.1   Evaluation Questions and Key Performance Measures

The Volpe team identified the following four impact areas that mapped to the last two sections of the logic model:

• Breadth and depth of NHTS use (short-term outcome).
  
  
• Impacts on policy, project, or regulatory decisionmaking (long-term impact).
  
  
• Responsiveness to user community (short-term outcome).
  
  
• Challenges and lessons learned (short-term outcomes).

For each impact area, evaluation questions were developed along with key performance measures needed to inform these questions.

The first two outcomes pertain to how the NHTS data are used and their impact as follows:

Breadth and depth of NHTS usage (short-term outcome) seeks to uncover who the NHTS users are (e.g., Government users, academics, consultants, media, industry, etc.) and the range of research questions and topics they are addressing with the NHTS data—both within transportation and in connected fields. In addition, this evaluation topic explores the type and form of NHTS data being used (e.g., datasets, online tools, and publications).

NHTS impact on policy, project and regulatory decisions (long-term impact) demonstrates how NHTS data and outputs inform decisionmaking with respect to policies, legislation, and projects. This evaluation area provides a qualitative assessment of the role of NHTS data, highlighting key examples or cases. The impacts of NHTS data on policy, legislation, and project outcomes is explored at the Federal, State, and local levels; moreover, impacts are assessed not only in the field of transportation, but in other connected fields, such as health, energy, and the environment.

The other outcomes pertain to the NHTS support of its user community and the lessons learned from past NHTS efforts.

NHTS Responsiveness to its user community seeks to understand how responsive the NHTS Program has been to its user community. The measures assess multiple factors that represent responsiveness, including the number of ways in which NHTS solicits feedback and the frequency of its outreach. In addition, this evaluation area provides a qualitative assessment of the changes that the NHTS Program has made in response to user feedback, including changes to the 2016 survey, as well as changes in other outputs or tools. These evaluation questions will deliver an assessment that can guide the NHTS, as well as other programs, in their feedback systems.

The final outcome area, challenges and lessons learned, is a compilation of the challenges and lessons learned in conducting the NHTS surveys, as identified by current and previous Program Managers and the Chief of Travel Monitoring and Surveys Division.^[5] The list is meant to be instructive, highlighting issues or topics that may serve as obstacles in the planning, administration, and outreach of the NHTS surveys. By addressing these challenges, FHWA can improve the survey process and outputs, with the long-term goal of increasing the impact of the data in policy and program development. In addition, this evaluation area also presents challenges and lessons learned related to the Add-on program, based on feedback received from a limited set of the Add-on partners.

Table 2 summarizes the key evaluation questions and performance metrics for each impact area of the evaluation. There were four main outcome/impact areas: breadth and depth; policy, project, or regulatory decisionmaking; responsiveness to community; and challenges and lessons learned. The evaluation questions are based on determining how well NHTS serves its user community and whom that community includes.

Table 2. Evaluation questions and performance measures by impact area.

2.2   Evaluation Methodology

The data collection that Volpe conducted was primarily qualitative in nature, focusing on a literature search and document review, indepth interviews, and email contacts to provide information for the evaluation. Data from these sources are used in the form of examples and cases to support all of the outcome and impact areas.

Two additional sources of information were provided by the NHTS. The first was the NHTS Compendium of Uses NTHS Compendium of Uses*. Volpe accessed this categorized collection of publication abstracts using the NHTS website. The second source was the monthly NHTS Website Usage Report, provided by Oak Ridge National Laboratory for the period from July 2013–May 2015.^[6] Each source was analyzed separately to inform the breadth and depth of use outcome.

*Revised 4/18/2018

This section describes the data collection and evaluation methodologies in detail and links them to specific outcome and impact areas and evaluation questions.

2.2.1 Literature Search and Document Review

Volpe conducted a literature search to gain an initial understanding of the NHTS Program, its stakeholders and users, the survey development process, outreach activities, and outputs. This information came from multiple sources, including the NHTS website,⁽¹⁶⁾ which provided detailed information on NHTS history, goals, and topics covered. The website was also a primary source for downloading NHTS publications, presentations, conference agendas, and external publications, and for viewing user guidance documents, datasets, and online tools. Additional documents were found using web search engines, including Google®,⁽¹⁷⁾ the TRB database,⁽¹⁸⁾ and Volpe library resources. Findings from the initial literature review were used to guide question and topic development for indepth interview guides. These findings were also used as a source of general NHTS information for all of the outcome/impact areas.

A more substantial document review was used to provide detailed information for several of the outcome/impact areas. Documents obtained during the literature search were used along with documents identified during indepth interviews and email contacts. Many of these additional reports, presentations, and models developed with NHTS data were not available online or using library resources. These included materials from Federal and State governments, lead users of the NHTS, and members of the NHTS task force. The documents collected from all sources provided examples and cases to inform the breadth and depth of use; user responsiveness; and policy, project, and regulatory Impact outcomes. The process used to extract this information is discussed in the case categorization section.

2.2.2 Indepth Interviews and Email Contacts

Semi-structured, indepth interviews and email contacts provided much of the information used to inform evaluation questions under the four outcome/impact areas. Volpe worked with the NHTS Program Manager to develop an initial interviewee list and gather contact information. Initial interviews with NHTS staff and lead users yielded additional interviewee contacts via snowball sampling. Because of budgetary constraints and a wealth of interview candidates, primary data collection was split into two groups: indepth interviews (longer) and email contacts (brief). Volpe completed 23 interviews and 18 email contacts (41 total). See Appendix A for a complete list of interviews and email contacts.

2.2.2.1 Indepth Interviews

Interviews were conducted with 23 respondents representing four NHTS stakeholder groups: NHTS Program Managers (current/former) and Office of Policy staff, members of the NHTS task force, lead NHTS users (transportation and other fields), and Add-on partners. The interviews were conducted either in person or by phone and generally lasted 60–90 minutes. Volpe developed a detailed interview guide for each group consisting of 15–20 open-ended questions (with detailed probes). The interview guides included questions that were relevant to each stakeholder group, typically covering a minimum of three of the four outcome/impact areas. Question topics asked for each impact area are shown below.

• Breadth and depth of NHTS use: These questions focused on how users were employing the NHTS data: What research questions were being addressed? Which elements of the NHTS were used (e.g., trip rates, demographics, vehicle type, etc.)? What form of the data were used (e.g., raw data, structured findings, online tool outputs)? In what form the output was presented (e.g., models, reports, trends, etc.)?
  
  
• Policy, project, and regulatory decisionmaking: These questions focused on how NHTS data and findings were used to inform Federal, State, and local planning, policy, and project decisions.
  
  
• User responsiveness: These questions focused on how NHTS was interacting with the NHTS user community to address current user needs and solicit feedback on issues and future data needs.
  
  
• Challenges and lessons learned: These questions focused on lessons learned in the areas of survey planning, oversight, and outreach. Add-on partners were also asked about their lessons learned from participating as a partner in the NHTS Program. (This was asked only to a subset of the interviewees.)

2.2.2.2 Email Contacts

Volpe contacted 18 respondents with a small subset of the questions from the indepth interviews. These respondents generally represented NHTS user sub-groups including DOT staff, academics, and NHTS task force members. The questions were distributed by email and collected by email or phone, as needed. The four to five open-ended questions covered only two outcome/impact Areas: breadth and depth of NHTS use and policy, project, and regulatory decisionmaking. An example of the email sent to interviewees is included in Appendix C.

Both the indepth interviews and email contacts requested supporting documents, when available, to provide specific examples and cases of NHTS use. All interviewees were asked questions related to breadth and depth of NHTS use; policy, project, or regulatory decisionmaking; and NHTS responsiveness to its user community. However, questions related to challenges and lessons learned were asked of only a subset of the interviewees, including NHTS Program Managers (current and previous) and other staff from FHWA’s Office of Policy.

Table 3. Interview topics by interview audience and type.

2.2.3 Interview Analysis

The interviews and email contacts yielded detailed notes with examples and key quotes for each outcome/impact area covered in the interview. These detailed examples and quotes were entered into a spreadsheet under the appropriate outcome/impact area. Additional information describing each example or quote, such as user type, field, topic, or level of government was also coded to provide the context for the example or quote. These examples and quotes then served as raw materials to be used to inform the evaluation questions.

2.2.4 Case Categorization

In addition to the anecdotal examples and quotes pulled from the indepth interviews and email contacts, the Volpe team assembled a more formal case database used to provide concrete examples of how NHTS was used to inform policy, project, and regulatory decisions. The documents obtained through the literature search, indepth interviews, and email contacts provided such cases.

The cases were organized by the impact area they supported, and included additional information to allow sorting by topic, purpose, type of decision supported and applicable level of government. In building its database Volpe sought to understand both what NHTS data were used to support decisions and in what form the data were fed into the decisionmaking system. The cases can be described as providing decisionmaking support either directly or indirectly.

Direct Support. This category describes cases where the NHTS is used in the form of descriptive data or trends. NHTS data and/or findings are delivered to decisionmakers in the form of statistics, trends, or travel topics by way of reports, presentations, and testimony.

Indirect Support. The second category includes cases where the NHTS supports decisions indirectly, as an input (along with other data) into models, calculations, and research that are used in the decisionmaking processes. An example of indirect support includes Highway Statistics Vehicle Miles (VM-1), which uses NHTS vehicle occupancy data.

2.2.5 Analysis of NHTS Website Statistics

The analysis of the Website usage reports from July 2013 to May 2015 is a descriptive analysis of the monthly usage numbers. ORNL compiles this data for the NHTS Program and provided Volpe with a copy of its monthly reports.

This analysis assessed monthly usage in multiple ways to get a sense of the overall magnitude of NHTS usage. It examined the website outputs individually and in relation to one another. The outputs include the following:

• Visitor statistics (e.g., monthly visitors, monthly visits, and average visits per visitor).
  
  
• Page view statistics.
  
  
• Dataset downloads (e.g., total monthly downloads and number of datasets downloaded).
  
  
• User Guidance document downloads.
  
  
• Online tool activations.
  
  
• Publication downloads.

The analysis also attempted to discern any trends or usage patterns from the data by plotting the monthly usage statistics during the 23-month period. Unfortunately, data prior to July 2013 were not available, so it is not possible to discern changes in website usage because of the release of NHTS data. Volpe can provide only a very limited snapshot of trends.

2.2.6 Analysis of NHTS Compendium of Uses NTHS Compendium of Uses*

To inform the assessment of the Breadth and Depth of NHTS data use, Volpe analyzed the 2014 NHTS Compendium of Uses NTHS Compendium of Uses*,⁽¹⁾ a document developed by the NHTS staff that provides a comprehensive catalogue of publications citing the NHTS in that year. Volpe based its analysis on the existing classification system of 11 categories but tailored it after conducting a pilot analysis indicating that additional classification information was needed to answer the evaluation questions. It should be noted that while the Compendium is the best available source on published NHTS documents, it does not provide an exhaustive compilation of its uses. Through document reviews and interviews, Volpe learned that NHTS users also develop documents and reports, models and statistics, and other outputs that are not published or publicly distributed, and hence are not included in the Compendium.

*Revised 4/18/2018

2.2.6.1 Field and Topic Analysis

After completing a pilot analysis consisting of categorizing a sample of publications from each original subject area in the 11 Compendium categories, Volpe moved from the original classification system to a two-tiered system made up of fields and transportation topics. Some original categories became fields, others remained as topics, and new fields and topics were added to acknowledge emerging NHTS uses. Table 4 shows how the original Compendium categories fit into the two-tiered structure.

Table 4. New classification system for 2014 compendium analysis.

A total of 313 publications were analyzed from the 2014 Compendium.⁽¹⁾ Each publication included an abstract and citation used to complete this analysis. For each publication, the categorization was completed in two parts. First, based on a review of the abstract, Volpe assigned the publication a primary field. Second, Volpe determined the transportation topic(s) represented in each publication regardless of field.

To help give focus to the NHTS use in a publication, Volpe assigned only one primary field to each publication. The fields show how the NHTS provides information to a variety of disciplines beyond transportation. While only one field was selected, multiple transportation topics could be selected for each publication. Within the transportation field, Volpe further designated the topics as primary and secondary when multiple topics were identified.

Another important component of use includes identifying the range of users developing research, reports, and presentations using these data. Looking at these users allows us to better understand how the NHTS influences decisionmaking at the local, regional, and national level. For this section, Volpe conducted two types of analysis: first, analyzing the lead authors of the publications listed in the 2014 Compendium,⁽¹⁾ and second, analyzing the lead presenters at major NHTS conferences,⁽⁵⁾ workshops, and presentations (2004–2015).

⁵ While other interviewees were not directly asked a question about challenges and lessons learned, they did provide comments on these topics that are included in the findings.

⁶ Website statistics are available through an internal reporting system.