Evaluation of Promoting Roadside Revegetation: An Integrated Approach to Establishing Native Plants

2. Evaluation Design

A logic model is a series of statements that links program components in a chain of causality. Program components include inputs, activities, outputs, outcomes, and impacts. Logic models describe the relationships among program resources, planned activities, and expected results; ultimately, they make explicit how program stakeholders expect program activities to affect change. The project team evaluating Roadside Revegetation traced the desired effects of the guide using the logic model in table 1.⁽¹⁾

AASHTO = American Association of State and Highway Transportation Officials.

The project team sought to answer the following two primary research questions regarding outcomes of the Roadside Revegetation and related material outputs:⁽¹⁾

• To what extent have stakeholders adopted the practices described in Roadside Revegetation and related materials?
• Have Roadside Revegetation and related materials improved the establishment of native plants and resulted in other positive outcomes?

The first research question was intended to find whether Roadside Revegetation’s recommended practices had been effective in meeting end users’ needs.⁽¹⁾ In other words, did Roadside Revegetation materials encourage and help stakeholders change or supplement previous practices and with Roadside Revegetation techniques? The second research question was intended to examine the relationship between activities; outputs; and short-, medium-, and long-term outcomes. In other words, has the development and promotion of Roadside Revegetation, awareness of the materials, and adoption of the promoted techniques resulted in positive outcomes for the environment, safety, visitor experience, and maintenance costs (see table 2)?

2.1 Evaluation Methodology

The project team sought to answer the research questions by collecting both quantitative and qualitative information. The project team first conducted a review of relevant literature on roadside revegetation to begin the evaluation. Statistics from Roadside Revegetation’s website were then analyzed to gain insight on how interest in the guide may have changed over time.^(4,1) The project team also solicited information from FHWA and USFS regarding past requests for copies of Roadside Revegetation or training on the guide’s principles. These background-gathering activities were followed by an online survey of revegetation practitioners and telephone interviews with select survey respondents to gain additional insights not possible in the survey.

Literature Review

Literature was collected primarily from screening relevant websites and through library scans on keywords such as “roadside revegetation” and “native revegetation.”The project team researched various public institutions, State transportation departments, the Bureau of Land Management (BLM), and NPS for relevant manuals and policy and guidance documents. Information, including manuals, policy, reports, and case studies from these and other agencies, was also collected. The project team also recorded other research reports that have cited Roadside Revegetation (see section entitled End Users’ Adoption of Roadside Revegetation Practices).⁽¹⁾

Website Visitation Analysis

The website for Roadside Revegetation is www.nativerevegetation.org.⁽⁴⁾ FHWA’s Western Federal Lands (WFL) has been using Google® Analytics™ to track website visitation statistics since January 2010. WFL provided the project team with historical Web statistics for all dates between January 1, 2010, and February 17, 2015. The visitation statistics summary report for the Roadside Revegetation website includes the following five sections:⁽⁴⁾

• Audience Overview provides data on the numbers of sessions, users, and page views the website has had. It also reports the average number of pages viewed per website session, the average duration of a session, and the percentage of sessions that are started by new visitors. Visitation by country (as defined by “language”) is also noted.
• Content Drilldown sorts the website’s various pages by frequency of visitation, average page view time, and number of unique views.
• Landing Pages shows where visitors are entering the website.
• Exit Pages indicates from where visitors typically leave.
• Pages Report provides basic information for each page viewed on websites, including the number of times a page was viewed, the average time a user stayed on the page, the page’s bounce rate, and what percentage of visitors left through a specific page. ¹

WFL/USFS Correspondence

The staff from WFL and USFS who drafted Roadside Revegetation and content for the associated website noted to the project team that much of the feedback regarding the guidance materials had come in the form of personal anecdotal interaction or correspondence from end users.^{(1, 4)} Recent examples of emails from Roadside Revegetation end users were provided to the evaluation team. These emails offer insights about the perceived quality and effectiveness of the guide and Website. ²

Online Survey

From April 1, 2015, to approximately July 31, 2015, the project team administered an online survey of Federal agencies (appendix A) and offices or units that may have implemented Roadside Revegetation practices.⁽¹⁾ The project team re-opened the survey between March 1 and March 15, 2016, to encourage additional responses from FHWA. End users were expected to be staff from FLMAs, State and local transportation departments, tribal governments, FHWA division offices, and other Federal agencies and administrations that may have contracted out revegetation projects.

The survey asked questions regarding the level of awareness of Roadside Revegetation and its website, the extent to which stakeholders have adopted the practices described in the guide, and how effective the changed practices have been in achieving the establishment of native plants and other positive outcomes along roadsides (see appendix A).^(1,4)

Telephone Interviews

The project team selected respondents for interviews based on responses to the online survey. The project team contacted 10 respondents to invite them to participate in a follow-up telephone interview. Four individuals participated in follow-up calls. The interviews built on the online survey, asking in-depth questions related to the respondents’ experiences using native plants and their end results. Calls were conducted using a standard interview guide that provided consistency across interviews (see appendix B), and interviewees shared a wide range of thoughts on the subject material and Roadside Revegetation itself.⁽¹⁾ The project team documented the interviews in detailed notes.

2.2 Data Limitations

There are no data limitation concerns regarding the literature review and the website visitation statistics.⁽⁴⁾ Regarding email correspondence to FHWA and USFS, a minor limitation relates to feedback collected. FHWA and USFS may not have not tracked all of the email, telephone, and in-person feedback on Roadside Revegetation that they have received.⁽¹⁾ Of the feedback tracked, all example anecdotes provided to the evaluation team reflect positively on Roadside Revegetation. This does not mean that FHWA and USFS selectively tracked feedback, but there may have been end users who had unfavorable views of the guide but who did not provided their input to FHWA and USFS.

In terms of the survey, its results present the following three limitations:

• Given that the size of the target population is unknown, it was not possible to conclusively assess the quality of the response rate.
• Federal regulation limits the scope of information requests to Federal agencies only without Office of Management and Budget approval. FHWA and USFS speculated that a significant cohort of Roadside Revegetation end users included staff from State transportation departments.⁽¹⁾ It was not possible for the evaluation team to send the survey directly to State transportation departments. Instead, the team relied on FHWA division offices to respond as proxies for their counterpart State transportation departments and/or retrieve and report their respective States’ views on behalf of the State transportation departments. Some FHWA division offices chose not to answer any survey questions, noting that the questions would be best directed toward State transportation departments. Other FHWA division offices forwarded the survey to their State transportation department counterparts on their own volition.
• The evaluation team intended to use survey results to screen candidates for more in-depth telephone interviews. The survey elicited 71 responses. However, 17 responses were incomplete in that the potential respondents only provided contact information or a response to the first question. Out of the other respondents, 27 were not aware of Roadside Revegetation.⁽¹⁾ Seven of the top interview candidates indicated that they did not want to participate in follow-up interviews. These factors reduced the potential pool of interviewees to 18. Of the remaining 18, 4 were staff members from the same organization who separately responded to the survey with similar input. (One was contacted and participated in an interview.) Of the remaining candidates, four participated in phone conversations, while others did not answer requests for interviews or declined interviews when the evaluation team tried to schedule convenient times.

1 Bounce rate is defined as the percentage of single-page sessions. It represents the percentage of visitors who enter and leave a site after viewing one page versus continuing on to view additional pages within the same site.

2 The project team received a series of internal emails from FHWA and USFS from Roadside Revegetation end users. The project team used these emails for the analysis.⁽¹⁾