Publication #: FHWA-HEP-09-015 | JANUARY 2009
Prepared for the
U.S. Department of Transportation
Federal Highway Administration
Prepared by
Wilbur Smith Associates
and S.R. Kale Consulting LLC
Question. What measures should be used to evaluate whether efforts to engage the private sector are successful?
Answer. Measuring success may differ in the public sector from in the private sector. In the public sector, success may be associated with whether input on freight-related initiatives has been obtained from a broad cross-section of freight stakeholders. In the private sector, success may be associated with whether transportation plans and improvement programs include freight needs and projects and whether projects are built or otherwise implemented to address stakeholder concerns. Success may vary depending on time frame of the interested party. For example, private sector participants may feel that their efforts have been less than successful when public sector processes take too long to address critical needs. Or they may feel that success is only partial if funding constraints results in projects being partly completed or phased with long intervals between phases. Through continuing and informed public involvement activities, public and private sector participants may over time re-define how they measure success.
This guidebook for engaging the private sector presents ideas and techniques for broadening the well developed practice of "public involvement" to include private sector business stakeholders as well. One key aspect of stakeholder involvement that must also be addressed is: "How do we measure whether the steps we are taking to involve the private sector are successful?"
While public transportation agencies use public involvement practices for an increasing variety of decision-making processes, the existing literature suggests that assessing the effectiveness of these programs remains a developing practice. In a State of the Practice: White Paper on Public Involvement, the TRB Committee on Public Involvement in Transportation made the following observation:
A challenge to the profession is to develop commonly accepted methods for evaluation of public involvement programs. Agency heads and managers consistently and correctly question the benefits of costly public involvement programs, which sometimes drive up the cost of planning and design. Typically, public involvement practitioners argue that public involvement expenditures are justified in that they prevent delays, lawsuits, and costly reassessment of policies. But such a response has been insufficiently quantified.
Among the resources identified during the development of this guide was the Transportation Planning Capacity Building (TPCB) website sponsored by FHWA: http://www.planning.dot.gov/default.asp.
The TPCB website offers advice and tools for a wide variety of transportation planning topics. The Resource Index page includes the topic area "public involvement." One of the resources provided on the public involvement page includes Public Involvement Techniques for Transportation Decision-Making. The guide offers five steps to systematically establish and implement a public involvement program for a specific plan, program, or project, including step 5:
Assure that proposed strategies and techniques aid decision-making to close the loop. Ask agency staff the following questions:
Are many people participating with good ideas?
Are key groups participating?
Is the public getting enough information as a basis for meaningful input?
Are decision-makers getting adequate public information when it is needed?
How can missing participants be attracted?
Do participants think discussion is full and complete?
Do they think the agency is responsive?
The TPCB website also provides several case studies, including experiences from Florida and Minnesota, in developing formal public involvement programs (PIP). The Florida case study offers several lessons for gathering feedback regarding public involvement activities, including:
Ask Staff What Works and What Doesn't. The Florida Department of Transportation (FDOT) queried employees about techniques that worked and those that didn't. They conducted telephone interviews with individuals directly involved in the public involvement activities associated with the 2020 Florida Transportation Plan (FTP). FDOT ultimately hired a consultant to design a training program and PIP materials. FDOT's PIP guide-Public Involvement Handbook - contains an entire chapter on evaluating effectiveness of public involvement programs.
Design Resource Materials to Promote Periodic Evaluation. As part of the training materials developed for its PIP, FDOT established standards and provided practical guidance to staff on how to initiate and sustain effective partnerships with the public.
The Minnesota Department of Transportation (Mn/DOT) case study offers four lessons from their PIP development process. Two of these lessons focus directly on performance and assessment activities:
Lesson 3: Identify performance measures. To begin evaluating the success of Mn/DOT's PIP, a family of measures for public involvement, complete with desired outcomes and measures, was included in the program. It was intended that these measures would continue to be evaluated and refined to help Mn/DOT achieve its vision for public involvement.
Lesson 4: Design resource materials to support periodic evaluation. As agencies struggle to balance limited resources against the need to provide citizens with ample access to the transportation decision-making process, it is important to assess what works and what doesn't. Mn/DOT was diligent in defining and documenting a set of objectives, methods, and techniques to guide their outreach efforts. To be effective, applicability of the tools/techniques identified will need to continue to be examined to determine the appropriate context for their use.
Materials supporting these two case studies provide a variety of techniques for evaluating the effectiveness of public involvement/stakeholder outreach activities.
Surveys: These typically consist of short specific questions regarding stakeholder involvement tools or techniques that were used for a specific program or project. Surveys can be conducted in-person, by phone, mail, or email. In the case of Freight Stakeholder Groups, feedback surveys can be administered at the conclusion of each meeting. Figure 7 provides a sample of possible questions.
Figure 7: Example Freight Stakeholder Meeting Feedback Questions
Statistical Analysis: Data and statistics about private sector involvement programs can also be used to measure and track the level of engagement. Figure 8 below shows metrics used by the Delaware Valley Regional Planning Commission (DVRPC) to track attendance at meetings of their Goods Movement Task Force. When attendance began to wane in the late 1990s, the DVRPC undertook a series of steps to determine how to make meetings more useful to participants. Those steps resulted in significantly increasing meeting attendance.
Figure 8: Average Meeting Attendance of the DVRPC Goods Movement Task Force
Source: http://www.dvrpc.org/Freight/pdf/2004-03_FinalFHWA.pdf;
DVRPC Goods Movement Task Force web page http://www.dvrpc.org/Freight/DVGMTF.htm
Self-Assessment: Another way to gauge and measure the effectiveness of engaging the private sector is to conduct internal reviews with staff and managers involved in the process. Self assessments can take several forms such as internal staff debriefings, internal surveys, and score cards based on decision-making and cost effectiveness metrics. Figure 9 provides an example of a self-assessment instrument for examining and scoring the current level private sector engagement in the planning process.
Figure 9: Example Self-Assessment Score Card for Engaging the Private Sector
