FHWA Research And Technology Evaluation: Public-Private Partnership Capacity Building Program

PDF Version (1.37 MB)

FOREWORD

The Federal Highway Administration (FHWA) has initiated an effort to evaluate the Research and Technology (R&T) development program and communicate the full range of benefits of their program. The R&T evaluation program helps FHWA assess how effectively it is meeting its goals and objectives and provides useful data to inform future project selections.

This report examines how FHWA’s investment in the Public-Private Partnership (P3) Capacity Building Program helps build knowledge and technical expertise and improve decisionmaking for this innovative project delivery method.

This report should be of interest to researchers, practitioners, and decisionmakers involved in the P3 implementation process.

Monique Evans,
Acting Associate Administrator
Research, Development, and Technology

Technical Report Documentation Page

SI* (Modern Metric) Conversion Factors

Table of Contents

Executive Summary

1. Introduction

• 1.1 Evaluation Purpose
• 1.2 Report Structure
• 1.3 P3 Program Background

2. Evaluation Design

• 2.1 Logic Model
• 2.2 Evaluation Hypotheses
• 2.3 Evaluation Methodology

3. Findings on P3 Program Usage

• 3.1 Summary Findings
• 3.2 Detailed Findings

4. Evaluation Findings

• 4.1 P3 Program Influence on Legislation and Policy
• 4.2 P3 Program Influence on P3 Consideration and Use Decisions
• 4.3 P3 Program Influence on Practitioner Decision Making Capabilities
• 4.4 Complete Source of P3 Information

5. Recommendations

• 5.1 Evaluation Recommendations
• 5.2 P3 Program Update

6. Conclusions

Appendix A. In-Depth Interviews

Appendix B. Online Survey—FHWA Division Office Staff

Acknowledgments

References

List of Figures

• Figure 1. Flow chart. P3 Program logic model.
• Figure 2. Graph. P3 Toolkit website users.⁽⁴⁾
• Figure 3. Graph. Landing homepage views.⁽⁴⁾
• Figure 4. Graph. Other homepage views.
• Figure 5. Chart. P3 Program activity profile.
• Figure 6. Chart. P3 Toolkit download profile.
• Figure 7. Chart. P3 Program users in FHWA Division Offices.
• Figure 8. Chart. FHWA Division Office P3 Program resource use.
• Figure 9. Chart. P3-enabling legislation.
• Figure 10. Chart. P3 Projects pre- and post-P3-Program launch.
• Figure 11. Chart. P3 Program use by decision stage.
• Figure 12. Chart. P3 consideration by P3 usage groups.
• Figure 13. Chart. P3 approval by P3 Program usage.
• Figure 14. Chart. Satisfaction with P3 Program resources.

List of Tables

• Table 1. Hypotheses by impact type.
• Table 2. Evaluation hypotheses and key performance measures.
• Table 3. P3 Program activity breakdown.
• Table 4. Google® Analytics^TM top P3 Toolkit document downloads.
• Table 5. Recent P3 Program download capture.
• Table 6. P3 legislation considered/adopted.⁽⁴⁵⁾
• Table 7. Project delivery/major projects top activities.
• Table 8. Project delivery/major projects top P3 resource use.
• Table 9. Top activities supported by finance positions.
• Table 10. Top P3 resources by those in finance positions.
• Table 11. P3 Program leader interviews.
• Table 12. State transportation department practitioners’ interviews.
• Table 13. FHWA Division Office staff interviews.
• Table 14. Legislative and policy interviews.
• Table 15. P3 private organization advisor interviews.
• Table 16. Question 1: Which of the following best describes your specialty at the FHWA Division Office?
• Table 17. Question 2: Has your State considered, explored, or carried out a transportation-related, P3?
• Table 18. Question 3: In your current role at the FHWA Division Office, have you supported State or local teams as they considered, explored, or carried out a transportation-related P3 project?
• Table 19. Question 4: How have you supported P3s in your current role?
• Table 20. Question 5: Have you attended IPD P3-VALUE webinar sessions to learn about P3s?
• Table 21. Question 6: Have you visited the IPD P3 Toolkit website to access information on P3s?
• Table 22. Question 7: Did you access the P3 Program via its website or by webinar? (Summary)
• Table 23. Question 8: P3 Program resources accessed.
• Table 24. Question 9: What type of information were you seeking on the P3 website?
• Table 25. Question 10: How satisfied were you with the information you found on the P3 Toolkit?

List of Abbreviations

Executive Summary

Purpose of the Evaluation

This report is part of the larger compilation of the Federal Highway Administration’s (FHWA) effort to evaluate their Research and Technology (R&T) Development Program. For each evaluation, the FHWA’s R&T Evaluation Program evaluation team (referred to as the “evaluation team” throughout this report) is made up of non-FHWA, third-party evaluators not involved in the research programs and projects being evaluated. This specific report is the evaluation of the Office of Innovative Program Delivery’s (IPD) Public-Private Partnership (P3) Capacity Building Program (P3 Program). The purpose of the evaluation is to understand how the program helps transportation practitioners build knowledge and technical expertise on P3s to inform decisions related to the consideration, development, and implementation of P3 projects.

Program Description

The P3 Program is part of the newly formed Center for Innovative Finance Support (formerly the Strategic Delivery Team in IPD). The center provides tools, expertise, and support for financing to help the transportation community explore and implement innovative financing strategies to deliver costly and complex infrastructure projects.⁽²⁾The main component of the P3 Program is the P3 Toolkit. The P3 Toolkit is an educational resource consisting of analytical tools and guidance documents that assist those exploring, developing, and implementing P3s. The P3 Toolkit addresses Federal requirements related to P3s and covers four key phases in P3 development and implementation: legislation and policy, planning and evaluation, procurement, and monitoring and oversight.⁽³⁾ The Toolkit forms the base of the P3 Program, which also includes a curriculum of training courses and webinars.

Methodology

The evaluation team developed four hypotheses to assess how the activities and outputs of the P3 Program support short-term outcomes and long-term impacts. For each hypothesis, evaluation questions and key performance measures were developed:

1. The FHWA P3 Program provides the most complete set of information resources to assist transportation practitioners in all phases of the P3 implementation process.
2. The FHWA P3 Program helps increase legislative and policy support for transportation P3s in State and local governments.
3. The FHWA P3 Program has led to more informed decisions on consideration and use (approval) of P3s for appropriate transportation projects.
4. The FHWA P3 Program improves the decisionmaking capabilities of transportation practitioners in the areas of P3 development, procurement, and oversight.

Both qualitative and quantitative methods were used to inform the evaluation. The evaluation team used a literature search and document reviews to gain an initial understanding of the P3 Program, its users, and its outputs. In-depth interviews with FHWA staff, State and local transportation agencies, legislators and policymakers, and P3 advisors provided a deeper understanding of P3 information needs and provided examples of program use. Information from the qualitative research was used to develop an online survey that captured data on P3 information needs and program usage by FHWA Division Office staff. P3 Program resource usage data, including outreach event attendance, website usage, and document downloads, provided additional quantitative data for the evaluation.^[1]

Findings

The findings section presents an overview of P3 Program usage and then highlights findings for the four evaluation hypotheses.

P3 Program Users and Resource Use

An analysis of website usage shows that the P3 Program serves hundreds of diverse stakeholders in the transportation community. The P3 Toolkit website reaches 1,500 to 2,500 users per quarter, spiking when new documents, tools, or webinars are introduced. Analysis of other usage data collected from P3 Program outreach registrations and document download forms indicates that there are multiple user groups accessing the program. The most active users fall into three groups: State and local transportation agencies; FHWA Division Office staff; and P3 advisory firms who provide legal, financial, or engineering support. Smaller groups of academics, interest groups, and international users round out the audience.

Aggregate statistics were available to assess usage of the P3 Toolkit website. Since its inception, the website has had two main landing pages, the P3 Toolkit homepage and the P3-VALUE Analytical Tool homepage.^(3,4) Although both pages received more than 1,000 views per quarter in early years, the P3-VALUE Analytical Tool homepage has recently become more popular, while P3 Toolkit homepage views have declined. The homepages for other popular resources have also seen views decline, although the fact sheet and publications’ homepages are still widely used.⁽³⁾

P3 Program usage data that could be linked to groups show that each has a distinct resource use as follows:^[2]

• P3 advisory firms gather P3 information from webinars and outreach activities as well as through P3 Toolkit website downloads such as the P3-VALUE tool and related documents.
• State and local agencies use webinars and outreach sessions more than other sources.
• Academics and interest groups are involved in P3 Program development and outreach activities.

An online survey along with program usage data provide additional detail on the resource usage of FHWA Division Office staff. These users tend to attend webinars and also view fact sheets, primers, and other documents on the P3 Toolkit website. They are also likely to attend P3 outreach events.

Despite increases in P3 legislation and activity in recent years, P3 Program usage data and in-depth interviews suggest that there is not a strong connection between the P3 Program, legislators, and policymakers. Few involved in legislation or policy directly accessed P3 Program resources. This does not mean, however, that information from the P3 Program does not reach this audience. In many cases, the link between the P3 Program and those in legislative or policymaking positions is indirect. That is, the program provides knowledge to Federal, State, and local transportation agencies and P3 advisors who bring the information to decisionmakers. The P3 Program also provides content to other organizations that develop P3 materials specifically for legislators and policymakers.

P3 Consideration Decisions

Neither P3 Program usage data nor in-depth interviews with P3 teams provided solid evidence that program usage impacts P3 project consideration decisions. States using the P3 Program were shown to be slightly less likely to announce consideration of a P3 project than those who did not, and the few interviews conducted did not connect consideration with use of P3 Program resources. To better understand the relationship between P3 Program resources and P3 project consideration, a more thorough identification and exploration of project-screening decisionmakers is needed.

P3 Approval Decisions

There is more evidence demonstrating that P3 Program resources help State and local agencies make informed P3 project approval decisions. An analysis of the Public Works Financing (PWF) Major Projects Database shows that roughly 30 percent of P3 projects announced in the years 2013–2015 were approved. P3 Program use was associated with the vast majority of States who approved transportation P3 projects in this period. Interviews with State and local agencies indicated that knowledge gained from the P3 Program was often used to support the case in favor of P3s.

Research on State and local agencies led to mixed conclusions about the value of the P3 Program for project decisionmaking. Less experienced States who took advantage of P3 Toolkit resources and FHWA staff found program resources to be valuable during project decisionmaking. States who had little or no FHWA staff support struggled to find helpful resources using the P3 Toolkit website. In addition, very experienced P3 States, most of whom have their own P3 education programs, had only limited use for the current materials, using them mainly for introductory information.

Interviews with a limited set of P3 advisors led to conclusions similar to those seen in State and local agencies. Experienced practitioners at established firms see value in the P3 Program through its use as an introductory resource but rarely use or recommend advanced analytic tools. Newer employees at P3 advisory firms, however, report attending webinars and using documents from the website. Although the evaluation interviews indicate limited P3 Program use among advisors, usage data on P3 Toolkit website downloads contradicts this, showing that advisors are a dominant user group. Additional research is recommended to understand more about this large user group, their use of the P3 Program, and their information needs.

P3 Program resources are increasingly valued by FHWA Division Office staff as more States are considering P3s. According to a survey of FHWA Division Office staff, more than one-third of those in roles that support P3s have accessed the P3 Toolkit website or attended webinars. They look to the program for materials to support teams at each stage of the P3 implementation process. P3 Program documents and webinars provide information that FHWA staff share with State and local agencies. Fact sheets, primers, and webinars provide general P3 information used during planning and evaluation, while model contract guides, project agreements, and oversight documents provide specific examples used when developing project documents during the procurement and implementation phases.

To address whether the P3 Program offers a complete set of P3 resources, program satisfaction and the use of other resources were assessed. The results indicate that the P3 Program is a valued resource among some user groups but not used by all. In addition, even among satisfied users, other resources are necessary to supplement the information provided by the P3 Program.

Satisfaction

Depending on their level of experience, satisfaction varied between users:

• Those newer to P3s from State and local agencies who have a relationship with the P3 Program are most satisfied; newer employees at P3 advisory firms also expressed satisfaction.
• Inexperienced practitioners from State and local agencies who tried to navigate the P3 Program website without support were less satisfied.
• Experienced practitioners from agencies and P3 advisory firms were less satisfied, noting that most P3 Program resources were not geared to their level.
• FHWA Division Office staff tended to be moderately to very satisfied with materials but saw room for improvement.

Completeness (Other Resources Used)

Depending on their background, users accessed a variety of resources for P3 information:

• Those developing P3 legislation and policy looked to other States’ legislation and materials for examples and guidance. Legal advisors were also described as critical in conducting a P3.
• State and local agencies noted that P3 advisory firms continue to play a major role in P3 implementation. Some States developed their own peer exchanges to bring real-life P3 examples and issues to the table. Other resources include P3 conferences, training, and other organizations.
• FHWA Division Office staff noted that they use information from more experienced FHWA offices. They also learned from experienced State and local P3 teams and from P3 advisors.
• Other than the use of in-house materials, P3 advisors mentioned gathering information from organizations such as AASHTO and the Design-Build Institute of America.

Recommendations

The P3 Program is already recognized as a valuable resource, but the evaluation process has identified the following five actions that could improve the reach and usefulness of the program:

1. The P3 Program should recognize the distinct user groups that access program resources and identify their information needs. Content development should be focused on a few key target groups and communication plans developed to reach them.
2. Marketing efforts should focus on target groups who are currently underusing P3 Program resources. The program should monitor resource use and seek to improve awareness among target groups not attending events, accessing webinars, or accessing guidance documents.
3. The P3 Program should segment its offerings, aligning documents and tools with the needs of practitioners at different experience levels to better serve those in all user groups.
4. The P3 Toolkit website should be organized in a way that allows distinct user groups to easily identify and access the content needed. The P3 Program team should look to its user groups for guidance on how to organize the website to best meet their information needs.
5. Future resources should include more “real-world” P3 information, examples, and experiences to help teams find information applicable to their unique P3 projects. Examples include actual P3 legislation, peer exchanges, and a P3 project database.

^[1]Internal information was received from P3 Program staff and from Google® Analytics™.⁽⁴²⁾

^[2]Resource usage tied to individuals/groups came from P3 Program event registrations and a form on the P3 Toolkit website download page (https://www.fhwa.dot.gov/ipd/p3/toolkit/download). Document use/views through the website are not included.