FHWA Research And Technology Evaluation: Public-Private Partnership Capacity Building Program

2. Evaluation Design

2.1 Logic Model

A logic model is a tool used to visualize the relationship between program components (inputs, activities, outputs, outcomes, and impacts). It is not intended to be a comprehensive or linear description of all program processes and activities but rather a tool to make explicit how program stakeholders expect program activities to effect change. While the hypotheses and evaluation questions seek to uncover the effect of program inputs and activities, the logic model shows how each step in the process plays a role in the creation of outcomes and impacts.

An iterative series of discussions with members of the R&T Evaluation Team and P3 Program staff led to the development of the P3 Program logic model (see figure 1). This logic model shows how inputs, including congressional direction, R&T funding, and the mission of the Center for Innovative Finance Support, gave way to activities focused on developing information for the P3 Program. This information became the publications, analytical tools, and materials for P3 outreach and training activities. These P3 outputs and activities are the vehicles by which P3 knowledge is transferred to legislative and executive staff; transportation practitioners at the Federal, State, and local levels; and other stakeholders. Lastly, the building of expertise within these stakeholder groups yields the longer-term impacts of the P3 Program on decisions and actions related to the consideration and use of P3s.

2.2 Evaluation Hypotheses

The evaluation team developed the following four hypotheses to show how the activities and outputs of the Center for Innovative Finance Support’s P3 Program led to the short and medium/long-term impacts:

1. The FHWA P3 Program provides the most complete set of information resources to assist transportation practitioners in all phases of the P3 implementation process (referred to as Complete P3 Resource).
2. The FHWA P3 Program helps increase legislative and policy support for transportation P3s in State and local governments (referred to as Legislative and Policy Support).
3. The FHWA P3 Program has led to more informed decisions on consideration and use (approval) of P3s for appropriate transportation projects (referred to as P3 Consideration and Use).
4. The FHWA P3 Program improves the decisionmaking capabilities of transportation practitioners in the areas of P3 development, procurement, and oversight (referred to as Practitioner Decisionmaking).

Through these hypotheses, the evaluation team investigated how P3 Program outputs in the form of information resources, training, and direct support improve State and local agencies’ knowledge of P3s. This knowledge may impact legislative and policy actions related to P3s as well as initial decisions to consider and use P3s for major transportation projects. The knowledge is also put to use throughout the evaluation process that feeds into the final approval for P3s and the resulting vendor procurement. As P3 projects progress, the resources can also play a role in informing the P3 monitoring and oversight process. A list of evaluation questions that address each hypothesis is shown in table 2. The key performance measures that inform the evaluation questions are also shown. Section 2.3, Evaluation Methodology, provides detailed information on the data collection methods used to inform the key performance measures.

2.3 Evaluation Methodology

This evaluation has both summative and formative elements. Where possible, the evaluation is focused on measuring the success of the P3 Program in transferring knowledge and building expertise on P3s in the transportation community. The evaluation then attempts to link this knowledge to impacts related to P3 actions and decisions. Where information to provide summative findings is lacking, because of the relative newness of the P3 Program, the evaluation takes on a more formative nature, providing recommendations on how the program can best meet its objectives as it moves forward.

The evaluation team used data from four main sources to inform this evaluation:

1. Literature and document review.
2. Interviews with P3 Program users (FHWA employees from headquarters as well as division offices, State, and local non-FHWA P3 conducting agencies, and P3 advisors/consultants).
3. Online survey with FHWA Division Office staff.
4. Program usage data (website usage, outreach activity and webinar attendance, and document downloads).

The literature search, document reviews, and in-depth interviews provide qualitative information for the evaluation. Data from these sources provide examples of and context for P3 Program use to support all of the outcome and impact areas. Additional sources provide more quantitative data for analysis. An online survey conducted with FHWA Division Office staff provides information on program use as well as information on specific P3 information needs. Program usage data, including outreach event attendance, website usage, and document downloads, provide additional information on the level of P3 Program resource use.^[1]

The next section describes the data collection and evaluation methodologies in detail and links them to specific hypotheses.

Literature Search and Document Review

The evaluation team conducted a literature search to gain an initial understanding of the P3 Program, its users, and program outputs such as publications, tools, and trainings. This information came from multiple sources, including the P3 Program website, which provides detailed information on the program’s goals and is the source of all available information resources.⁽²⁾ The website is home to the P3 Toolkit, a source for “analytical tools and guidance documents to assist in educating public sector policymakers, legislative and executive staff, and transportation professionals in implementation of P3 projects.”⁽⁴⁾ The website also provides current information on the state of P3s in the United States, including links to P3 legislation, P3 project profiles, and other procedural documents. Additional background documents on the P3 Program and P3s in general were found using web search engines, the Transportation Research Board (TRB) database, and Volpe library resources.⁽⁴²⁾ Findings from the initial literature review were used to develop questions/topics for in-depth interview guides and the online survey.

The literature search and document reviews support all hypotheses.

Interviews with P3 Program Users

The evaluation team conducted 31 in-depth, semi-structured interviews with a range of P3 Program users, including the following:

• FHWA Division Office staff (eight interviews).
• Leaders of State-level P3 Programs (five interviews).
• State and local transportation department/P3 office specialists in finance and project delivery positions (eight interviews).
• P3 advisors/consultants (seven interviews).
• State-level executive and legislative staff (three interviews).

The interviews were used to better understand the distinct information needs of those involved at different phases of the P3 implementation process, from passing State- or local-level P3-enabling legislation to the implementation and oversight of a P3 project. Once the information needs were identified, the discussion focused on the sources of information, support, and training used to fill the information needs. Those using resources from the P3 Program were probed on their level of satisfaction with the P3 information resources and tools used. Respondents were also asked to identify any gaps in the information currently available from the P3 Program, name other information sources used, and identify future information needs. Interview guides were tailored to each group based on their role in the P3 implementation process. A summary of interview questions as well as a list of interviewees is provided in appendix A.

To address the intricacies in R&T evaluation, the evaluation team interviews many stakeholders. The team assures all interviewees that their identities will remain confidential to achieve more unbiased answers to questions they may be asked. Throughout the document, when interviewees are quoted we note the month and year of interview as well as the interviewer, but the interviewee name is redacted. However, to maintain continuity and comparability between interviewee responses, a generic title is attributed to each interviewee. The aforementioned information is placed in a footnote for each interview.

Information from the interviews informs all hypotheses and findings.

Online Survey with FHWA Division Office Staff

Information from the interviews with FHWA staff was used to develop an online survey. The survey collected information on State, regional, and local P3 activity, P3 information needs, and P3 Program usage. Invitations to the survey were sent via email to 620 FHWA Division Office staff from four disciplines: project delivery/major projects, planning/environmental finance, and technical services. The 10-minute survey was completed by 259 respondents who provided information about their involvement in State or local P3 projects. If involved, the respondents were asked about the type of support provided, the information sought by P3 teams, and the P3 Program resources used to obtain the information. The respondents were also asked to rate their satisfaction with the P3 Program resources used and to identify any aspects of the program that could be improved. Combined with the more contextual information from the in-depth interviews, this information presents a solid picture of how the P3 Program is used and perceived within the FHWA Division Offices.

Data from the survey primarily inform Hypothesis 1, Complete P3 Resource, and Hypothesis 4, Practitioner Decisionmaking.

P3 Program Usage Data

Data on P3 Program usage was gathered from multiple sources to provide an idea of whom the P3 Program serves and to identify the type of information accessed by users. Contact lists from the P3 Program’s outreach efforts, training sessions, and webinar registrations were combined in a P3 Activity Database and broken out by user type: State or local government, Federal government, P3 advisors/consultants, organizations/interest groups, and academics. While this database may not be representative of all P3 Program users, as information was captured opportunistically on multiple dates from multiple sources, analysis of the database provides insight into the type of users who make up the P3 Program’s audience. The P3 Activity Database is supplemented by recently collected P3 Toolkit website download data (January 2016–June 2016) showing what documents were downloaded from the P3 Toolkit website and the type of users downloading them (e.g., international versus domestic, academic versus government).

Information on P3 Program users was complemented by usage information from the P3 Toolkit website. This information was collected internally by the P3 Program using Google® Analytics™.⁽⁴³⁾ Quarterly usage sessions from the website and views of homepages that house P3 content, publications, and analytic tools were collected for the period January 2013–June 2016. These aggregate data were analyzed to understand the frequency and nature of usage.

The program usage data described above informs all hypotheses and findings.

^[1] Program usage data provided to the evaluation team by the P3 Program staff were pulled from program records and reporting systems.