FHWA Research And Technology Evaluation: Public-Private Partnership Capacity Building Program

3. Findings on P3 Program Usage

This section presents findings related to P3 Program usage. Although the evaluation questions listed below are linked to Hypothesis 1, the usage findings are presented separately from the hypothesis to allow for a more holistic assessment of usage prior to the presentation of other findings. Hypothesis 1 will be more thoroughly covered in section 4.4.

This section covers two evaluation questions:

1. Who are the P3 Program users?
2. What resources are P3 Program users accessing?

The following four data sources assembled for the P3 Program evaluation were used to identify P3 Program users and resource usage:

• P3 Toolkit statistics (aggregate usage).
• P3 Program activity database (specific usage).
• P3 Toolkit website downloads (aggregate and specific usage).
• FHWA district office online survey (specific usage).

Although these sources may not capture all usage, looking at the usage data and user profiles (where available) provides information on the range of users who access the P3 Program.

3.1 Summary Findings

Evaluation Question 1: Who are the P3 Program users?

An analysis of P3 Program usage shows that the P3 Program serves hundreds, if not thousands, of diverse stakeholders in the transportation community. P3 Toolkit website usage statistics, which are not linked to individual users, provide aggregate information on website use. The P3 Toolkit website reaches 1,500 to 2,500 users per quarter, spiking when new documents, tools, or webinars are introduced.

Other usage data collected from P3 Program activities and document downloads provide information on distinct user groups. Although the available data may not represent all users, they indicate that there are multiple user groups with different information needs. The most active users fall into three groups: State and local transportation agencies; FHWA Division Office staff; and consultants who advise P3 teams on the legal, financial, and engineering aspects of P3 projects. Smaller groups of academics, interest groups, and international users round out the audience for the P3 Program.

Evaluation Question 2: What resources are P3 Program users accessing?

Based on P3 Program usage data that could be tied back to individual users, we see that those from P3 advisory firms gather P3 information from webinars and outreach activities as well as through P3 Toolkit website downloads.^[1] The most popular downloads include the P3-VALUE Tool and related documents. State and local agency users are less likely to download P3 evaluation related documents but look to webinars and other outreach sessions for information. Academics and interest groups are less likely to download documents or attend webinars, but they are involved in outreach activities.

More usage information was available on FHWA Division Office staff who provide direct support to State and local P3 teams. An online survey of these employees shows that although P3 projects have been implemented in only a small number of States, representatives from FHWA Division Offices in most States have looked to the P3 Program as a resource for general knowledge on P3s. Those who use the program tend to attend webinars and view fact sheets, primers, and other documents on the P3 Toolkit website. They are less likely than other groups to view or download P3-VALUE-related materials.

3.2 Detailed Findings

Evaluation Question: Who are P3 Program users?

P3 Web Statistics—Overall Usage

When the P3 Toolkit website was released in the beginning of 2013, it brought in approximately 750 users during the first quarter. Usage quickly increased after P3 evaluation-based webinars began in March of 2013 (second quarter). After the initial increase, usage ranged from roughly 1,500 to 2,500 users per quarter. Usage peaked on three separate occasions that fell within the first quarter of each year. The most recent spike coincides with the start of the latest round of webinars, supporting the updated P3-VALUE 2.0 Tool. This trend is shown in figure 2.

Although usage numbers are strong, Google® Analytics™ reports a bounce rate of 63 percent. A user is considered to have “bounced” if he or she left the landing page without browsing the website further. Therefore, a majority of users either found what they were looking for on the landing page or left without searching the website. It also indicates that 37 percent (approximately 500 to 800 users per quarter) search further through the website after landing.

P3 Web Statistics—Landing Page Views

The P3 Toolkit website appears to have two main landing pages. Not surprisingly, the first is the website homepage.⁽⁴⁾ This page attracted close to 1,800 views as the website was starting up, but views soon leveled off to about 1,000 page views per quarter and began to decline throughout 2015. The Analytical Tools homepage has usage that tends to increase in the first quarter and then decline through the fourth quarter.⁽³²⁾ This homepage has seen its usage increase starting in the fourth quarter of 2014, with page views ranging from approximately 1,200 to 1,400 through the end of 2015. This webpage has become a second homepage for the P3 Program website. This is likely because of support activities such as webinars and trainings held by the P3 Program staff focusing on the P3-VALUE Analytical Tool and its supporting documents. Usage for both homepages increased markedly in early 2016 as a new webinar series started and the new P3-VALUE tool was released. A trend of these homepage views over time is shown in figure 3.

P3 Web Statistics—Other Content Page Views

The homepages for other P3 Program resources generally see lower usage numbers. While fact sheet and publication homepage views spiked upon introduction (approximately 1,500 and 1,000 views, respectively), views leveled out and declined to as low as 500 views in some quarters. The webinar homepage saw a similar pattern, with a spike in usage shortly after introduction and then a rapid decline, ending up below 500 views per quarter. Checklists have a smaller number of views, typically under 200 per quarter and declining. Although there is still decent viewership per quarter for the fact sheets, publications, and webinars, these resources could benefit from some publicity to reenergize usage. Views did increase in the first quarter of 2016 for fact sheets and webinars. A trend of these views is portrayed in figure 4.

P3 Program Activity Database—User Groups

Using information available from the P3 Program activity database (see section 2.3), a profile of P3 Program users was developed. The database includes contact data from attendees of P3 Program outreach activities, webinars, and training programs. It does not include data on users of the P3 Toolkit website or other resources. Figure 5 shows the breakdown of contacts from State or local agencies, Federal agencies, P3 advisory firms, interest groups/organizations, and academic institutions.

Employees from government agencies at the Federal, State, and local levels constitute the majority of attendees at the P3 Program events. Contacts from P3 advisory firms, including those providing legal, finance, and engineering support, are also well represented, contributing almost a third of attendees. A smaller set of interest group/organization members and academics round out the user groups.

P3 Toolkit Website Downloads—Email Contact Scan (User Groups)

Another source of information on P3 Program users comes from the P3 Toolkit website. Starting in January 2016, the P3 Toolkit website required users to provide information such as email address, title, and agency when downloading select documents from the website. An analysis of the contacts helped identify basic user groups. Because of incomplete fields and unidentifiable organizations, the email field was used for analysis. Where possible, the email domain information was supplemented using the job title or agency field. A breakdown of user type is shown in figure 6.

The email contact scan showed that the majority of download requests came from business or personal email addresses, most with the “.com” suffix. Roughly 20 percent of these were identified as international. Many of the non-international addresses were recognized as advisory firms that provide finance, legal, or engineering consulting services to U.S. agencies implementing P3s. Other groups that could be identified include academics and students (.edu), Federal employees (.gov), and those from State and local government agencies (various). A few domestic and international organizations and representatives from foreign government agencies were also seen.

FHWA Division Office Online Survey—FHWA Users

An online survey of FHWA Division Office staff provided additional detail on P3 Program usage among Federal employees. Employees from 44 of the 52 States/territories responding to the survey indicated that P3s had been considered, evaluated, or implemented in their States.^[4] Thirty-four percent of respondents (88 total) from 42 States indicated that they had used P3 Program resources to learn about this project delivery method. FHWA Division Office employees in project delivery/major projects and finance-related positions made up the majority of users within the FHWA Division Offices. Interestingly, only half of those gaining knowledge through use of P3 Program resources had previously supported a P3 project in their State. Many seem to be preparing for possible future P3s in their States. A bar chart of this information is shown in figure 7.

Evaluation Question: What resources are P3 users accessing?

P3 Program Activity Database—Activity and Event Attendance

The outreach and training activities of 690 P3 Program activity database contacts were grouped into four categories for analysis: P3 Program development activities (e.g., listening sessions, roundtables, beta tests), P3-VALUE webinars, individualized State training sessions, and other P3 outreach activities. Table 3 shows the activity breakdown for the distinct user groups. For this analysis, State and local agencies were broken out into two groups, those who attended individualized trainings for their State and those who did not. The numbers in bold text highlight areas used the most by each user group.

Note: Bold text indicates areas used the most by each user group.

MD = Maryland; DC = District of Columbia; LA = Louisiana; NC = North Carolina.

The P3 Program activity database shows differences in the types of activities attended by each group. Webinars are primarily attended by FHWA and USDOT staff, State and local agencies, and advisory firms. Contacts from States newer to P3 who attended customized State P3 trainings have generally not attended other events or activities. USDOT headquarters staff, advisors, academics, and interest groups have been the most involved in the P3 Program development activities. Other P3 outreach activities, including P3 TRB subcommittee events and poster sessions, attract academics more than other groups.

P3 Toolkit Website Downloads—Google® Analytics^TM Downloads and P3 Program Download Capture

There were two sources of the P3 Toolkit website downloads available for analysis. The first shows downloads recorded by Google® Analytics™.⁽⁴³⁾ These downloads cannot be linked to users or groups. They are provided to show what documents were most accessed from January 2013–March 2016. The second source of data comes from a P3 Program document download platform, which does capture user information but shows only downloads from a limited set of documents from January through June 2016.

P3 Toolkit website download statistics pulled from Google® Analytics^TM (January 2013–March 2016) shows the top 10 documents downloaded during a 3-year period. These documents were located throughout the P3 Toolkit website and could be viewed as hypertext markup language, portable document format (PDF), or Microsoft® Excel files. The download statistic shows how often the files were saved once opened. These statistics represent only a fraction of all usage, because document views are not included. Even though downloads were infrequent, they do provide some data on the P3 Program resources that were important enough to download. Table 4 shows download numbers for the most downloaded documents.

The most downloaded documents include the four original P3-VALUE (1.0) Tool modules and their supporting manuals on risk assessment, financial assessment, shadow bids, and PSC. Webinar presentations on related topics, including P3 Evaluation, Risk Assessment, and Value for Money, were also saved by several users. Only one publication, an introductory fact sheet, made the list.

Another look at document downloads comes from more recent statistics. Beginning in January 2016, the P3 Program began capturing information on select documents downloaded from the P3 Toolkit website. These documents represent 16 of the latest publications, analytical tools, and screening tools available through the website. This file-download platform represents a change in how documents were made available on the website, making it easier to find and download multiple documents. The change in the download format resulted in significantly more downloads in the 6-month period (January 2016–June 2016) compared to the previous 3 years (see table 4). In total, 292 users downloaded an average of 9 documents each during the 6-month period. A breakdown of these downloads is shown in table 5.

Despite the fact that these downloads are not directly comparable to those from previous periods, they still provide information on the use of P3 Program resources. The most popular downloads are evaluation related: financial structuring and assessment, risk assessment, value for money (VFM) assessment, and the new P3-VALUE 2.0 materials. These tools and documents have similar themes to the documents downloaded in previous periods. It is also interesting to note that the P3 screening tools were frequently downloaded from the new platform, giving new purpose to these documents.

FHWA Division Office Survey–P3 Resource Use

FHWA Division Office Survey respondents (n = 259) were asked if they attended webinars or used P3 Program resources through the P3 Toolkit website. Thirty-four percent of those surveyed indicated use (n = 88). Figure 8 shows which resources were used by FHWA Division Office staff.

FHWA staff tend to attend webinars and look at introductory materials (fact sheets and primers). Few look at P3-VALUE tools and related documents. Of FHWA employees, financial managers tend to use the most materials, an average of four per user. They skew higher on webinars and previous training materials (79 and 31 percent, respectively). They are also slightly more likely to use the P3-VALUE Analytical Tools (10 percent). Project delivery/major projects staff use an average of three resources, skewing higher on oversight documents (30 percent) and contract development guides (24 percent). Planning and technical positions use few resources (on average two per user) but do skew higher on use of case studies (28 and 3 percent, respectively).

^[1] The only document-specific download data that could be linked to user groups were the most recent download data available, provided for the period January 2016–June 2016.

^[2] Survey information can be located in appendix B.

^[3] Survey information can be found in appendix B.

^[4] Not all FHWA staff surveyed in these 44 States/territories were equally aware of interest or actions taken on P3s. In 20 States/territories, FHWA staff gave conflicting answers (both “yes” and “no” answers recorded).

^[5] Survey information can be found in appendix B.

^[6] Internal information provided by FHWA.

^[7] Internal information provided by FHWA.

^[8] Internal information provided by FHWA.