Skip to contentUnited States Department of Transportation - Federal Highway Administration FHWA Home
Research Home
PRESENTATION
This presentation is an archived publication and may contain dated technical, contact, and link information
Publication Number: N/A
Date: May 2011

 

Federal Highway Administration Exploratory Advanced Research

Measuring The Impact and Performance of Transport Research Programs Poster

4th International Transportation Systems Performance Measurement Conference

PDF Version (337 KB)

PDF files can be viewed with the Acrobat® Reader®

FHWA's Exploratory Advanced Research (EAR) Program focuses on long–term, high–risk research with a high payoff potential. The program addresses underlying gaps faced by applied highway research programs, anticipates emerging issues with national implications, and reflects broad transportation industry goals and objectives.

Effective research is critical for meeting emerging transport challenges.

Research, however, is difficult to measure.  There can be significant time lags between the conduct of research and the return on the investment.  Impacts can be diffuse, accruing to unexpected parties who build on the work of others.

The FHWA Exploratory Advanced Research (EAR) Program has been developing a suite of measures to monitor and improve overall Program portfolio performance and predict the potential impact of research. 

The EAR Program uses different measurements of performance to provide a balanced scorecard for day-to-day program management and communication of results to internal and public stakeholders. 

This poster provides information on EAR Program measures and background on the search for appropriate measurements that could suggest approaches for other transport research programs.

Financial Perspective
How efficient does the program appear to congress, leadership?

FY 2010 Baseline

Target

Comment

94 percent in FY 2010

92 percent of cumulative Program funding for research projects

At least 80 percent of funding for research projects

Internal and public measure of efficiency

For FY 2010 (funds obligated), 27 percent match

Amount of matching non-federal funds by sector

Internal indicator of leverage, partnership

Percent by solicitation rather than fiscal year more meaningful

Measurement being refined

Funds committed, obligated, expended

Internal indicator of efficient management process

Annual amount appears misleading using fiscal year

Under development

Time from budget delivery to awards

Internal and public measure of efficiency

Consider median time and max time. 

 

Customer Perspective
How does the program appear to internal and external stakeholders?

FY 2010 Baseline

Target

Comment

For most recent round 14 percent (but awards not complete yet)

For program overall, 17 percent

Proposals submitted and awards made (yield rate)

Public indicator

Cumulative percent assumes only full proposals (not FY 2007 BAA pre-proposals) and completion of five awards in process

Under development

Number of institutions and external experts involved (and the number who are involved for the first time)

Internal and public indicator of stakeholder engagement and outreach

Under development

Full-time equivalent of researchers, graduate students supported

Internal and public indicator of building capacity but difficult to measure

Requires common definition for FTE across proposals

Under development

Percent of initial stage topics open over one year

Internal indicator of efficient process or complex topics

On hold till new project management system adopted

Number of projects or funding contributing to Agency Strategic Plan by goal area

Public indicator of potential impact.  Difficult to measure without adoption of  new program management support system (PMSS)

Measurement by EAR Program focus areas also could be meaningful

 

Internal Business Process
In which process should the program excel?

FY 2010 Baseline

Target

Comment

Integrated highway system concepts:  4 in FY 2010 and 10 cumulatively (since FY 2006)

Nanoscale research: 3 and 6

Human behavior and travel choices: 4 and 15

New technology and advanced policies for energy and resource conservation: 4 and 12

Information sciences: 4 and 7

Breakthrough Concepts in material science: 5 and 8

Technology for assessing performance: 5 and 15

Number of initial stage investigations in each  EAR Program focus area

Internal and public indicator of program balance and breadth

Based on all active projects in FY 2010 including continuing and new topics

There may be some duplication with linked topics that are assigned to different focus areas

Cumulatively about 20 percent

Percent initial stage topics that lead to solicitations or intramural research projects

Internal and public indicator of program risk

Does not reflect some topics leading to multiple awards

Difficult to assess for FY 2010 since topics still active

Under development

Percent of projects that demonstrate use of effective research approaches

Retrospective indicator as determined by independent reviewers

Under development

Percent of awardees who gain follow-on funding from other sources as a result of EAR Program funded work

Retrospective indicator of building capacity; a possible start towards return on investment

Under development

Percent of projects that lead to continued, applied research with the potential for breakthrough advances

Retrospective indicator of risk and impact

 

Innovation and Learning
Where should the program improve and change?

FY 2010 Baseline

Target

Comment

5 in FY 2010

32 cumulatively

Number of new FHWA personnel involved in initial stage research

Internal indicator of building capacity

10 percent in FY 2010

10 percent cumulatively

Percent of initial stage research involving multiple offices or results that have been picked up by a different office

Internal indicator of program breadth

Under development

Percent of projects that lead to adoption of new research processes or approaches

Retrospective indicator of impact

Under development

Percent of projects that close persistent knowledge gaps, result in new fundamental data, or significantly change current understanding

Retrospective indicator of impact

Under development

Use of merit review in each stage of process

Internal but difficult to measure

Under development

Publications and presentations of project results

Internal and public indicator but difficult to monitor

David Kuehn

Federal Highway Administration
6300 Georgetown Pike
F-220 (HRTM-30)
McLean, Virginia 22101
Phone 202 493 3414
david.kuehn@dot.gov

Poster Presentation for the 4th International Transportation Systems Performance Measurement Conference on May 18, 2011, in Irvine, CA

 

ResearchFHWA
FHWA
United States Department of Transportation - Federal Highway Administration