U.S. Department of Transportation
Federal Highway Administration
1200 New Jersey Avenue, SE
Washington, DC 20590

Skip to content
Facebook iconYouTube iconTwitter iconFlickr iconLinkedInInstagram

Federal Highway Administration Research and Technology
Coordinating, Developing, and Delivering Highway Transportation Innovations

This fact sheet is an archived publication and may contain dated technical, contact, and link information
Back to Publication List        
Publication Number:  FHWA-HRT-15-070     Date:  August 2015
Publication Number: FHWA-HRT-15-070
Date: August 2015


Automated Video Analysis

Analyzing Large Quantities of Transportation Research Data

Logo. The Exploratory Advanced Research Program’s logo of a camera over a street intersection.

The Exploratory Advanced Research Program Fact Sheet

PDF Version (363 KB)

PDF files can be viewed with the Acrobat® Reader®

Exploratory Advanced Research . . . Next Generation Transportation Solutions


The second Strategic Highway Research Program (SHRP 2) Safety research area has produced unprecedented data for highway researchers in the form of the naturalistic driving study (NDS). With approximately 2 petabytes of data available to researchers, most of which is video data, new automated tools for data extraction and analysis are required. Developing a prototype tool was the goal of the Exploratory Advanced Research (EAR) Program project, “Machine Learning for Automated Analysis of Large Volumes of Highway Video.” The National Robotics Engineering Center, at Carnegie Mellon University (CMU), conducted this research, which was funded by the Federal Highway Administration (FHWA) in 2012.

Graphic. Two images captured with two different object-detection tools of the same urban scene of cars parallel parked on a street. The first image is a photo of the scene, where four of the detected parked cars have each been highlighted in a different color (red, green, purple, and blue). The second image is a black-and-white graphic depiction of the same parked cars, this time with the detected cars highlighted in white against a black background. Only the detected parked cars are shown in the black-and-white graphic; the other objects in the scene are not visible. However, the images of the two cars on the right are obstructed by the presence of street lamp poles, which break up the images of those cars and make them more difficult to detect.
A typical challenging situation for identifying discreet
objects in a cluttered urban environment.

Naturalistic Data

The SHRP 2 NDS involved almost 3,000 volunteers going about their regular activities with extensive instrumentation, including video and still cameras, a Global Positioning System, and many other sensors attached to their personal vehicles for up to 2 years. The NDS data will provide unparalleled insight into the connections between driver behaviors, road design, traffic, and many other topics that have previously been difficult, if not impossible, to directly investigate, or were limited in scale by the intimidating logistics of collecting and analyzing this much data. The major challenge is decoding the video data. For example, one researcher might be interested in how drivers react to particular elements of the overall roadway environment whereas another researcher might be focused on the video from the camera pointed directly at the driver’s face and upper torso.

Overcoming a Data Bottleneck

The traditional way to extract features of interest from video is for technicians to physically sit in front of a monitor and manually scroll through frame by frame, logging the location where each feature is found. With approximately 1.2-million hours of video in the NDS dataset, it is estimated that it would take almost 600 technicians a full year, working 40 hours per week, to go through all the video. Automation of these feature extraction tasks would dramatically reduce the time and cost involved.

In accordance, the EAR Program is funding a total of six projects, including the CMU research project, that are directly or indirectly focused on making the SHRP 2 NDS data more accessible. The objective of these research projects is to advance efficient and cost-effective methods and tools to analyze large amounts of safety-related video data. Such tools will reduce the time and costs required to pull needed features from these rich datasets, and thus expand the pool of researchers able and willing to use such data in their research. The CMU research project aims to address the challenge by automating the feature extraction so that the details that researchers need can be quickly and accurately extracted.

Machine-Learning Approach

The research team for this CMU study implemented a machine-learning–based approach. This approach feeds a multitude of features into learning algorithms that are trained on very large datasets and capable of learning with high accuracy exactly which features are important, which provide less useful data, and how features can be used in combination to accurately recognize and classify the desired targets. This means that the great diversity provided by large and complex datasets adds to the power and robust performance of the system.

Contextual Cues

To extract useful information from petabytes of data generated by moving or static video cameras, the research team integrated state-of-the-art algorithms to exploit contextual clues, reduce the amount of expert effort required to guide learning, and manage the computational effort required. When humans interpret sensor data, they rely on contextual cues, such as inferring from a patch of image showing a car that it is likely to lie above an image of a roadway. Roads and vehicles offer many other similar contextual cues that can be exploited within a scene. Researchers traditionally encode these contextual relationships in a graphical model; however, predictions may be inaccurate in these models. This study focuses on a novel set of methods to automate the interpretation of ambiguous video data by building a sequence of context dependent predictions. Instead of a traditional graphical model, this approach defines an inference procedure (e.g., wheels are found near the corners of a car), which can then be used to classify features moving forward.

Data Challenges

The research team implemented powerful labeling tools to enable fast human analysis and labeling of data, anomaly detection to prioritize data access and focus labeler time on significant events, and semi-supervised learning to build predictive models and to minimize the amount of labeled data required. In addition, with datasets running into petabyte sizes, there is a critical need for computational systems that can be scaled to match. The researchers considered two different approaches for managing such computation- and data-intensive machine learning. One approach is parallelization, involving the distribution of work across multiple cores in a network. The other approach is a new class of techniques, known as anytime predictors, used to provide a sequence of results that improves as more time is provided to analyze test data.

Figure 4. Logo. The logo of the Exploratory Advanced Research Program.

What Is the Exploratory Advanced Research Program?

FHWA’s Exploratory Advanced Research (EAR) Program focuses on longer term, higher risk research with a high payoff potential. The program addresses underlying gaps faced by applied highway research programs, anticipates emerging issues with national implications, and reflects broad transportation industry goals and objectives.

To learn more about the EAR Program, visit the Exploratory Advanced Research Web site at www.fhwa.dot.gov/advancedresearch. The site features information on research solicitations, updates on ongoing research, links to published materials, summaries of past EAR Program events, and details on upcoming events.

Future Applications

The primary outcome of this project is a software framework to support efficient application of machine-learning-based feature extraction techniques for very large sets of transportation video data, as well as specific target detectors to support various transportation analysis tasks. An additional benefit will be to expand industry understanding of a highly effective data-processing approach that can be widely used to process large volumes of future data. The goal is to be able to apply this approach to both moving targets (e.g., cars and pedestrians) and static targets (e.g., traffic signs).

Learn More

For more information on this project, contact Lincoln Cobb, FHWA Office of Safety Research and Development, at 202-493-3313 (email: lincoln.cobb@dot.gov).

Publication No.



Federal Highway Administration | 1200 New Jersey Avenue, SE | Washington, DC 20590 | 202-366-4000
Turner-Fairbank Highway Research Center | 6300 Georgetown Pike | McLean, VA | 22101