Skip to contentUnited States Department of Transportation - Federal Highway Administration FHWA Home
Research Home
This report is an archived publication and may contain dated technical, contact, and link information
Publication Number: FHWA-HRT-11-056
Date: October 2012


Layered Object Recognition System for Pedestrian Sensing



There is a significant need to develop innovative technologies to detect pedestrians or other vulnerable road users at locations where they are experiencing increased exposure to dangerous traffic. The motivation of this research was triggered by the Fatality Analysis Reporting System data that indicated that there were 4,882 pedestrian fatalities and 64,000 pedestrian injuries in the United States in 2005.(1) Pedestrian-related traffic crashes accounted for 11 percent of the total fatalities and 2 percent of the total injuries in 2005. Europe and Japan have adopted standards to improve pedestrian protection, including passive energy absorption structure modification, active energy absorption systems installed on selected vehicle models from Jaguar and Citroën, and brake assist systems to reduce impact severity.

In 2007, there was a proposal to develop a real-time in-vehicle vision-based system that would detect pedestrians from a moving vehicle and estimate their position and distance relative to the vehicle at distances that would allow actionable warning time.

In April 2008, there was a panel meeting in Princeton, NJ, for the research team to demonstrate the technical capability in machine vision technology. The meeting also allowed the technical panel members to share the latest pedestrian fatality and injury statistics and collectively define the specifications of the desired research product.

According to information provided by the Federal Highway Administration (FHWA) technical panel, the following pedestrian fatalities occurred in 2007:(2)

2007 pedestrian deaths by road type are as follows:

2007 pedestrian deaths by speed limit in an urban environment are as follows:

The panel members also described scenarios such as pedestrian crossing patterns at intersections, pedestrians unexpectedly darting across the street from behind motor vehicles at midblock locations, etc. The performance requirements of the pedestrian detection system were formed based on the above inputs and the technology limitations known to the research team.

While several researchers have developed pedestrian detection systems, most of these systems suffer from a high false positive (FP) rate of detection (e.g., objects incorrectly detected as pedestrians). The proposed approach in this report attempts to alleviate this problem by following a layered approach (i.e., different layers of processing to gradually reduce FPs by using multiple cues (shape, appearance, and depth)) and classifying objects.


The objectives of this study were to develop an in-vehicle pedestrian detection system capable of simultaneously recognizing pedestrians and other roadside infrastructures such as lamp posts, traffic signs, lane markings, pavements, buildings etc. The detection system should also be capable of distinguishing pedestrians and motor vehicles in and out of danger and determining danger levels. The researchers wanted to produce a system with a high detection rate and a low FP rate, coupled with inherently low-cost sensing technology for potential widespread adoption. The project addresses the Highway Safety Focus Area of the Broad Agency Announcement.


The primary goal of this project was to develop a real-time in-vehicle system that uses stereo vision and advanced computer vision techniques to detect pedestrians under typical driving conditions and meet the metrics in table 1.

Table 1. Pedestrian detection performance specifications.
Pedestrian Detection Rate (percent) FP Rate Range of Detection (meters) Conditions
98 (in path) 0.00001 (in path) 40 (day) Benign mostly urban scenes
90–93 (out of path) 0.003 (out of path) 25 (night)
1 ft = 0.305 m

These metrics were measured on selected video sequences by an offline implementation of the developed system. The true positive rate is defined as the percentage of pedestrians detected compared to the number of actual pedestrians in every frame of the sequence. It was measured on collected video sequences that contained a significant number of pedestrians. The FP rate is defined as the number of nonpedestrians incorrectly identified as pedestrians per hour of driving. It was measured by executing the real-time system in the test vehicle while driving under typical U.S. highway and urban conditions, collecting images every time the system detected a pedestrian, and manually checking the images against the actual presence or absence of pedestrians.


This report describes the work performed during the course of this project, which was partially funded by FHWA. It assumes a basic understanding of linear algebra, probability theory, and prior exposure to computer vision. Section 2 of this report provides a brief review of computer vision and probability theory topics that were used as prior art for the development of algorithms and software for a stereo-based pedestrian recognition system. Sections 3 and 4 describe the developed system configuration and the key innovations. The technical approach used to solve the pedestrian recognition problem is described in section 5. Section 6 describes the experiments conducted, results from collected data, and a comparison between state-of-the-art approaches published in literature. Finally, section 7 provides suggestions for future work and conclusions from this research work.


United States Department of Transportation - Federal Highway Administration