Layered Object Recognition System for Pedestrian Sensing

1. INTRODUCTION

1.1 BACKGROUND

There is a significant need to develop innovative technologies to detect pedestrians or other vulnerable road users at locations where they are experiencing increased exposure to dangerous traffic. The motivation of this research was triggered by the Fatality Analysis Reporting System data that indicated that there were 4,882 pedestrian fatalities and 64,000 pedestrian injuries in the United States in 2005.⁽¹⁾ Pedestrian-related traffic crashes accounted for 11 percent of the total fatalities and 2 percent of the total injuries in 2005. Europe and Japan have adopted standards to improve pedestrian protection, including passive energy absorption structure modification, active energy absorption systems installed on selected vehicle models from Jaguar and Citroën, and brake assist systems to reduce impact severity.

In 2007, there was a proposal to develop a real-time in-vehicle vision-based system that would detect pedestrians from a moving vehicle and estimate their position and distance relative to the vehicle at distances that would allow actionable warning time.

In April 2008, there was a panel meeting in Princeton, NJ, for the research team to demonstrate the technical capability in machine vision technology. The meeting also allowed the technical panel members to share the latest pedestrian fatality and injury statistics and collectively define the specifications of the desired research product.

According to information provided by the Federal Highway Administration (FHWA) technical panel, the following pedestrian fatalities occurred in 2007:⁽²⁾

• 73 percent occurred in an urban area.
• 76 percent occurred at non-intersection areas.
• 90 percent occurred under normal weather conditions.
• 67 percent occurred at night.

2007 pedestrian deaths by road type are as follows:

• 16 percent occurred on interstates or freeways.
• 55 percent occurred on other major roads.
• 27 percent occurred on minor roads.

2007 pedestrian deaths by speed limit in an urban environment are as follows:

• 24 percent occurred on roads with a speed limit less than 35 mi/h (56.35 km/h).
• 31 percent occurred on roads with a speed limit between 35 and 40 mi/h (56.35 and 64.4 km/h).
• 20 percent occurred on roads with a speed limit between 45 and 50 mi/h (72.45 and 80.5 km/h).
• 7 percent occurred on roads with a speed limit of 55 mi/h (88.55 km/h) or greater.
• 7 percent occurred on roads with an unknown speed limit.

The panel members also described scenarios such as pedestrian crossing patterns at intersections, pedestrians unexpectedly darting across the street from behind motor vehicles at midblock locations, etc. The performance requirements of the pedestrian detection system were formed based on the above inputs and the technology limitations known to the research team.

While several researchers have developed pedestrian detection systems, most of these systems suffer from a high false positive (FP) rate of detection (e.g., objects incorrectly detected as pedestrians). The proposed approach in this report attempts to alleviate this problem by following a layered approach (i.e., different layers of processing to gradually reduce FPs by using multiple cues (shape, appearance, and depth)) and classifying objects.

1.2 STUDY OBJECTIVES

The objectives of this study were to develop an in-vehicle pedestrian detection system capable of simultaneously recognizing pedestrians and other roadside infrastructures such as lamp posts, traffic signs, lane markings, pavements, buildings etc. The detection system should also be capable of distinguishing pedestrians and motor vehicles in and out of danger and determining danger levels. The researchers wanted to produce a system with a high detection rate and a low FP rate, coupled with inherently low-cost sensing technology for potential widespread adoption. The project addresses the Highway Safety Focus Area of the Broad Agency Announcement.

1.3 RESEARCH GOALS

The primary goal of this project was to develop a real-time in-vehicle system that uses stereo vision and advanced computer vision techniques to detect pedestrians under typical driving conditions and meet the metrics in table 1.

These metrics were measured on selected video sequences by an offline implementation of the developed system. The true positive rate is defined as the percentage of pedestrians detected compared to the number of actual pedestrians in every frame of the sequence. It was measured on collected video sequences that contained a significant number of pedestrians. The FP rate is defined as the number of nonpedestrians incorrectly identified as pedestrians per hour of driving. It was measured by executing the real-time system in the test vehicle while driving under typical U.S. highway and urban conditions, collecting images every time the system detected a pedestrian, and manually checking the images against the actual presence or absence of pedestrians.

1.4 SCOPE OF REPORT

This report describes the work performed during the course of this project, which was partially funded by FHWA. It assumes a basic understanding of linear algebra, probability theory, and prior exposure to computer vision. Section 2 of this report provides a brief review of computer vision and probability theory topics that were used as prior art for the development of algorithms and software for a stereo-based pedestrian recognition system. Sections 3 and 4 describe the developed system configuration and the key innovations. The technical approach used to solve the pedestrian recognition problem is described in section 5. Section 6 describes the experiments conducted, results from collected data, and a comparison between state-of-the-art approaches published in literature. Finally, section 7 provides suggestions for future work and conclusions from this research work.