U.S. Department of Transportation
Federal Highway Administration
1200 New Jersey Avenue, SE
Washington, DC 20590

Skip to content
Facebook iconYouTube iconTwitter iconFlickr iconLinkedInInstagram

Federal Highway Administration Research and Technology
Coordinating, Developing, and Delivering Highway Transportation Innovations

This report is an archived publication and may contain dated technical, contact, and link information
Back to Publication List        
Publication Number:  FHWA-HRT-17-036    Date:  March 2018
Publication Number: FHWA-HRT-17-036
Date: March 2018


Federal Highway Administration Research and Technology Evaluation Final Report: Eco-Logical

2. Evaluation Design

This evaluation sought to understand the effects of FHWA R&T activities on the implementation of the Eco-Logical approach to transportation project delivery by State transportation departments and MPOs. This chapter includes a logic model to describe the Eco-Logical Program, the evaluation hypothesis and questions, and the methodology used to collect and analyze data to answer the evaluation questions.

The evaluation sought to determine how effective FHWA’s efforts were in disseminating information to facilitate adoption, determine to what extent stakeholders have adopted the Eco-Logical approach, and identify business and environmental impacts obtained by agencies implementing the Eco-Logical approach.

2.1 Logic Model

A logic model is defined by the authors of this report as a series of statements linking program components (i.e., inputs, activities, outputs, outcomes, and impacts) in a causal chain. It describes the relationship between program resources, planned activities, and expected results. It is not intended to be a comprehensive or linear description of all program processes and activities but rather to make explicit how program stakeholders expect program activities to affect change. The logic model helps to explain the theories of change that drive the design of a program that can be tested in an evaluation (see figure 1).

Click for description

Source: FHWA.
Figure 1. Illustration. Eco-Logical evaluation logic model.

Inputs, Activities, and Outputs

The inputs, activities, and outputs in the logic model are described in table 1 in chapter 1. Several of the activities and outputs involved FHWA in collaboration with other agencies and organizations.

State Transportation Department/MPO Activities

FHWA stakeholders, including State transportation departments and MPOs, may choose to adopt any or all nine steps of the Eco-Logical approach (see subsequent section, State Transportation Department/MPO Outcomes, for a description of the steps). These nine implementation steps are activities that agencies can pursue to realize the outcomes and impacts described in the following two sections.

State Transportation Department/MPO Outcomes

The Eco-Logical Program provides strategies to help agencies meet transportation infrastructure needs without compromising the environment. The evaluation team identified three outcomes for the Eco-Logical Program and related them to the nine-step process for implementing the Eco-Logical approach. The outcomes and their accompanying steps are as follows:[2]

The first outcome describes relationships with partners. The second and third outcomes combine the five types of mitigation into two groups. The Council on Environmental Quality regulations (40 CFR 1508.20) defines mitigation as follows:(44)

The second outcome includes the two types of mitigation that can be accomplished in planning, and the third outcome includes the remaining three types of mitigation, which tend to occur in project development.


The Eco-Logical Program is intended to achieve two overarching impacts: (1) improved environmental mitigation and (2) improved project delivery processes (see figure 2). The Eco-Logical Program encourages early identification and consideration of mitigation opportunities at the planning level (i.e., to identify and consider issues earlier) and at an ecosystem scale (i.e., to consider effects in a broader geographic context than at the project level).

Click for description

Source: FHWA.
Figure 2. Illustration. Impacts of improved environmental mitigation and project delivery process.(3)

2.2 Evaluation Hypothesis and Questions

The evaluation team identified the following hypothesis for the program evaluation:

Hypothesis: The Eco-Logical Program and approach have contributed to improved project delivery processes and environmental mitigation.

Table 2 lists the key questions that link the evaluation research to the program logic model. The table also includes measures of effectiveness for each question.

Table 2. Eco-Logical Program evaluation research questions and key performance measures.
Evaluation Component Evaluation Question Key Performance Measures
FHWA activities and outputs How has FHWA enabled State transportation departments and MPO stakeholders to adopt the Eco-Logical approach? Key performance measures include training, technical assistance, and funding disseminated to State transportation departments and MPOs.
State transportation department/MPO activities and outcomes How are State transportation departments and MPO stakeholders incorporating the Eco-Logical approach into their business practices? Key performance measures include changing partnering; sharing data; analyzing effects on conservation program objectives; identifying key sites and actions; documenting, implementing, and evaluating ecological information; and ensuring increased transparency and predictability as well as better relationships.
Impacts to FHWA, State transportation departments, and MPOs How have the Eco-Logical Program and approach contributed to improved project delivery processes and environmental mitigation? Key performance measures include improved environmental mitigation and improved project delivery processes.

2.3 Evaluation Methodology

The evaluation team used the following five data collection and analysis methodologies to inform this evaluation:

The following subsections provide additional detail about these five data collection and analysis methodologies.

Literature and Document Review

The evaluation team conducted a literature review to gain an understanding of the Eco-Logical Program, its stakeholders and users, and FHWA outreach activities and outputs. The evaluation team also conducted a qualitative analysis.

Findings from the initial literature review are detailed in table 1 (see chapter 1) and used to answer evaluation question 1, how has FHWA enabled State transportation departments and MPO stakeholders to adopt the Eco-Logical approach?

The evaluation team determined it would be most appropriate to focus the evaluation on the effectiveness of implementation by recipients of FHWA funding through the 2007 Eco-Logical Grants Program and the 2013 SHRP2 IAP rather than attempt to identify and assess general awareness of the program and approach by hundreds of agencies around the United States.

A more extensive document review was used to provide detailed information for the qualitative coding analysis and as well as for the analysis of the Eco-Logical approach’s steps completed by recipients of FHWA funding. Documents reviewed included the Eco-Logical Program’s annual reports from 2008 to 2015 as well as notes from interviews that were conducted with recipients of FHWA funding that informed those annual reports. (See references 13, 15, 19, 25, 34, and 45.)

Appendix A includes a list of recipients and their project descriptions, appendix B shows interviewee data available from 2008 to 2016, appendix C provides lists of interview questions from each Eco-Logical annual report from 2008 to 2015, and appendix D includes questions from a program-sponsored (FHWA and AASHTO) 2014 questionnaire conducted for the 2007 grantee recipients.(25)

Participation in a Program-Sponsored Peer Exchange

On October 14–15, 2015, FHWA and AASHTO jointly held an Implementing Eco-Logical IAP Peer Exchange for SHRP2 IAP recipient agencies to share accomplishments and lessons learned from their projects with each other and with other transportation and resource agencies.(42) Participants included the following (asterisks denote IAP funding recipients):

A member of the evaluation team participated in the peer exchange to collect data from attendees through targeted listening sessions and general observations. Findings from the peer exchange helped the evaluation team to determine next steps for data collection, including conducting stakeholder interviews. Appendix E lists questions asked by the evaluation team to peer exchange participants and a summary of their responses.

Stakeholder Interviews

In 2016, the evaluation team interviewed recipients from both the 2007 Eco-Logical Grant Program and 2013 SHRP2 IAP. The purpose was to follow up on program-sponsored interviews and to ask targeted questions to inform the evaluation hypothesis and questions. The respondents included 10 MPOs, 6 State transportation departments, and 4 recipients categorized as “other” (see table 14 for a list of all interviewees). All respondents spoke with the evaluation team via a conference call except for one agency, which sent in written responses to the interview questions. Appendix F provides a list of 2016 evaluation interview questions.

To address the intricacies in R&T evaluation, the evaluation team interviewed many stakeholders. The team ensured all interviewees that their identities were to remain confidential to achieve more unbiased answers to questions that were asked. Throughout the document, when interviewees are quoted, the month and year of interview are noted as well as the name of the interviewer, but the interviewee names have been redacted. However, to maintain continuity and comparability between interviewee responses, a generic title is attributed to each interviewee. The aforementioned information is placed in a footnote for each interview.

The evaluation team also collected input from FHWA Eco-Logical Program’s points of contact through interviews and participation in a program visioning session to understand FHWA’s perspective on benefits and challenges, FHWA’s role in encouraging stakeholder implementation, and FHWA’s goals for the program’s past, present, and future.

Analysis of Eco-Logical Steps Completed by Funding Recipients

Based on a review of program annual reports and follow-up evaluation interviews, the evaluation team determined which Eco-Logical approach steps recipients completed, both during and following their periods of performance. (See references 13, 15, 19, 25, 34, and 45.) The program has generally not tracked recipient implementation of the Eco-Logical approach by the nine-step approach except for one annual report.(45)

The purpose of this analysis was to gain insight into the areas where agencies tend to focus their efforts in order to inform evaluation question 2, how are State transportation departments and MPO stakeholders incorporating the Eco-Logical approach into their business practices? Section 3.2 in chapter 3 of this report provides results of the steps analysis. The analysis also helped to identify potential areas where the program may choose to focus attention in the future, as described in chapter 4 of this report. It should be noted that each agency did not necessarily have the goal of completing all nine Eco-Logical steps, and the evaluation did not seek to appraise recipients based on the number of steps they completed.

Qualitative Coding Analysis of Stakeholder Comments

The evaluation team developed a database of stakeholder-identified benefits, challenges, and recommendations of the Eco-Logical Program. Information was gathered from interview notes that informed program annual reports from 2008 to 2015, responses from an Eco-Logical Grant Program questionnaire conducted by the program in 2014, input from the program-sponsored IAP peer exchange held in 2015, and interview notes from follow-up interviews conducted by the evaluation team in 2016. (See references 13, 15, 19, 25, 34, and 45.) Relevant statement(s) directly from interview notes were copied into the database with information on the data source, including agency, agency type, and year.[3] Each statement was then coded into a benefit, challenge, recommendation to FHWA, or recommendation to peer agencies. Each statement was also coded into a theme and category (or sub-theme). Coding the data makes it easier to search, make comparisons, and identify patterns that require further investigation.

Each theme and category was associated with an evaluation question, as provided in table 3. Themes and their associated categories are defined following the table. See appendix I for the number of comments assigned to the qualitative coding themes and categories.

Table 3. Qualitative coding themes and categories.
Theme Category Evaluation Question
Communication Knowledge 1
Communication Outreach 1
Resources Funding 1 and 2[4]
Resources Eco-Logical Grant Program or IAP 1
Resources Staff (includes staff time) 2
Relationships Credence 1
Relationships Data sharing 2
Relationships External stakeholders (includes buy-in) 2
Relationships Internal stakeholders (includes buy-in) 2
Operations Champion/leader 2
Operations Data use 2
Operations Environmental impacts 3
Operations External factors 2
Operations Process or process change 2
Operations Process impacts 3
Operations Staff turnover 2


The communication theme includes benefits, challenges, and recommendations related to the interchange or transmission of information about the Eco-Logical approach. Categories within this theme include the following:


The resources theme includes benefits, challenges, and recommendations related to the critical assets for the Eco-Logical approach to function effectively in agencies. Categories within this theme include the following:


The relationships theme includes benefits, challenges, and recommendations related to making connections between recipients and their stakeholders. Categories within this theme include the following:


The operations theme includes benefits, challenges, and recommendations related to the essential assets, functions, and considerations required to managing an organization or program effectively. Categories within this theme include the following:

Evaluation Limitations and How They Were Addressed

This section describes considerations the evaluation team identified in the evaluation plan developed internally for this report that could impact data collection and analysis. The evaluation team identified several techniques to address each challenge, as described in table 4.

Table 4. Evaluation limitations and how they were addressed.
Consideration Identified in Evaluation Plan How Challenge Was Addressed in Evaluation
Program impacts were difficult to measure because performance measures were limited, agencies may not have been collecting baseline data or information on time/cost savings, or those systems were not yet mature/robust. In addition, many organizations were controlling the process, and it was possible that no singular organization was tracking all elements of the process. Quantitative data may have been limited or may not exist to analyze project impacts. The evaluation mostly relied on qualitative data and proximal measures from synthesis of program materials and stakeholder interviews to infer impacts, where appropriate.
Long-range planning to project development could take many years, so it was challenging to measure the impact of a program on such a long-term scale. Findings from the evaluation confirmed that few agencies track impacts in terms of time/cost savings or improved environmental mitigation (see section 3.3 for more information on impacts).
There was a potential lack of stakeholder participation; the same organizations were contacted frequently in regard to this program. The evaluation team pursued opportunities to leverage data collection in coordination with anticipated program activities to maximize responses and participation. In addition, many recipients were willing to share their successes and challenges through evaluation interviews in 2016 even if they were years beyond project completion (see appendix B for the responses).
Outcomes and impacts from the Eco-Logical Program can affect many different levels of an agency, and implementation can occur in different ways for agencies adopting the approach as follows:
  • Program/process level: Organizing staff, trainings, and procedures.
  • Planning/project level: Implementing procedures in plans and projects.
  • Individual steps or entire approach: Taking on entire approach or elements of approach.
For this evaluation, impacts were strictly related to better environmental mitigation (e.g., avoiding impacts or compensating for impacts at an ecosystem scale) and improved project delivery processes as indicated by time and cost savings, and only for 2007 grant and 2013 IAP recipients. Findings on impacts were based on interviews from recipients (see section 3.3). In addition, the evaluation reported on implementation of procedures and how agencies used the Eco-Logical approach through an analysis of steps, benefits, challenges, and recommendations (evaluation question 2).
The evaluation team needed to determine whether change will be measured based on an agency compared to itself in the past or across the Nation over time. The evaluation team determined that it was most appropriate to compare an agency to itself. The recipient agencies have a variety of Eco-Logical projects and goals, making them difficult to compare to each other or infer nationwide trends.
It may be difficult to attribute FHWA activity and outputs as the direct cause of State transportation department/MPO outcomes and impacts because these activities are diffuse in nature, extend over long time periods, vary widely across applications, and have expected outcomes that are long term. The evaluation hypothesis and questions sought to understand program contributions, recognizing the variety of potential factors that may affect outcomes and impacts. Where appropriate, the evaluation team made inferences regarding effect of the program’s contribution.

[2] While step 9 appears under outcome 2, it is not out of order. The outcomes simply group related steps.

[3] The evaluation team sought to remove duplicate statements where, for example, the same recipient repeated the same benefit during the same interview. A single occurrence of a statement is one agency (including one or more individuals) on an interview call in 1 yr within an evaluation theme-category combination.

[4] The evaluation team recognized that comments related to funding could describe either evaluation question 1 (how FHWA enabled agencies) or question 2 (factors that supported or were challenges to agencies’ implementation of the Eco-Logical approach).

Federal Highway Administration | 1200 New Jersey Avenue, SE | Washington, DC 20590 | 202-366-4000
Turner-Fairbank Highway Research Center | 6300 Georgetown Pike | McLean, VA | 22101