U.S. Department of Transportation
Federal Highway Administration
1200 New Jersey Avenue, SE
Washington, DC 20590
202-366-4000


Skip to content
Facebook iconYouTube iconTwitter iconFlickr iconLinkedInInstagram

Federal Highway Administration Research and Technology
Coordinating, Developing, and Delivering Highway Transportation Innovations

 
REPORT
This report is an archived publication and may contain dated technical, contact, and link information
Back to Publication List        
Publication Number:  FHWA-HRT-11-056    Date:  October 2012
Publication Number: FHWA-HRT-11-056
Date: October 2012

 

Layered Object Recognition System for Pedestrian Sensing

PDF Version (9.56 MB)

PDF files can be viewed with the Acrobat® Reader®

FOREWORD

The purpose of this report is to describe the work performed and the results obtained during the Layered Object Recognition System for Pedestrian Sensing Project sponsored by the Federal Highway Administration. The goal of this project was to use stereo vision to detect, classify, and track pedestrians in cameras’ field of views and demonstrate the system’s performance in real time in a test vehicle.

This report will be of interest to researchers, developers, and technologists in the area of highway safety, pedestrian collision warning systems, intelligent transportation systems, and driver assistance systems. It provides information about state-of-the-art practices and directions for future work.

Monique R. Evans
Director, Office of Safety
Research and Development

Notice

This document is disseminated under the sponsorship of the U.S. Department of Transportation in the interest of information exchange. The U.S. Government assumes no liability for the use of the information contained in this document. This report does not constitute a standard, specification, or regulation.

The U.S. Government does not endorse products or manufacturers. Trademarks or manufacturers’ names appear in this report only because they are considered essential to the objective of the document.

Quality Assurance Statement

The Federal Highway Administration (FHWA) provides high-quality information to serve Government, industry, and the public in a manner that promotes public understanding. Standards and policies are used to ensure and maximize the quality, objectivity, utility, and integrity of its information. FHWA periodically reviews quality issues and adjusts its programs and processes to ensure continuous quality improvement.

 

Technical Report Documentation Page

1. Report No.

FHWA-HRT-11-056

2. Government Accession No. 3 Recipient's Catalog No.
4. Title and Subtitle

Layered Object Recognition System for Pedestrian Sensing

5. Report Date

October 2012

6. Performing Organization Code
7. Author(s)

Jayan Eledath, Bogdan Matei, Mayank Bansal, Sang-Hack Jung, and
Harpreet Sawhney

8. Performing Organization Report No.

 

9. Performing Organization Name and Address

Sarnoff Corporation
201 Washington Road
Princeton, NJ 08543

10. Work Unit No. (TRAIS)

11. Contract or Grant No.

Contract No. DTFH61-07-H-00039

12. Sponsoring Agency Name and Address

Exploratory Advanced Research Program
Federal Highway Administration
6300 Georgetown Pike
McLean, VA 22101-2296

13. Type of Report and Period Covered

Final Report

14. Sponsoring Agency Code

 

15. Supplementary Notes

Wei Zhang was the Agreement Officer’s Technical Representative (AOTR) for the Federal Highway Administration. Technical panel members included Ann Do, David Gibson, John Harding, and Jennifer Percer.

16. Abstract

There is a significant need to develop innovative technologies to detect pedestrians or other vulnerable road users at designated crossing locations and midblock/unexpected areas and to determine potential collisions with pedestrians. An in-vehicle pedestrian sensing system was developed to address this specific problem. The research team used stereo vision cameras and developed three key innovations, namely, the detection and recognition of multiple roadway objects; the use of multiple cues (depth, motion, shape, and appearance) to detect, track, and classify pedestrians; and the use of contextual information to reject a majority of the typical false positives that plague vision-based pedestrian detection systems. This report describes the approach and tabulates representative results of experiments conducted on multiple video sequences captured over the course of the project. The conclusion derived from these results is that the developed system is state of the art when compared to the best approaches published in literature. The false positive rates are still higher than desired for the system to be ready for commercialization. This report also provides steps that can be taken to improve the performance in this regard. A real-time system was developed and demonstrated in a test vehicle.

17. Key Words

Pedestrian Safety, Pedestrian detection, Stereo vision, Disparity map, Histogram of oriented gradients (HOG), Contour-based classifier

18. Distribution Statement

No restrictions. This document is available to the public through the National Technical Information Service, Springfield, Virginia 22161

19. Security Classification
(of this report)

Unclassified

20. Security Classification
(of this page)

Unclassified

21. No. of Pages

113

22. Price
Form DOT F 1700.7 Reproduction of completed page authorized

SI* (Modern Metric) Conversion Factors

Table of Contents

1. INTRODUCTION

2. REVIEW OF RELATED TECHNOLOGIES AND RELATED WORK

3. SYSTEM CONFIGURATION

4. KEY INNOVATIONS

5. TECHNICAL APPROACH

6. EXPERIMENTS AND RESULTS

7. CONCLUSIONS AND RECOMMENDATIONS FOR FUTURE WORK

APPENDIX

ACKNOWLEDGEMENTS

REFERENCES

LIST OF FIGURES

Figure 1. Equation. SAD
Figure 2. Equation. The range estimate of an image pixel
Figure 3. Illustration. HOG computed at an image location
Figure 4. Illustration. HOG computation using integral images
Figure 5. Equation. Computation of integral histogram
Figure 6. Equation. Computation of HOG for a specific image patch
Figure 7. Photo. In-vehicle camera sensor and a portable briefcase processing system
Figure 8. Photo. NTSC camera used in the developed pedestrian detection system
Figure 9. Photo. Acadia I™ vision accelerator board
Figure 10. Illustration. Diagram of the developed system
Figure 11. Photo. Example 1 of pedestrian detection
Figure 12. Photo. Example 2 of pedestrian detection
Figure 13. Photo. Example 3 of pedestrian detection
Figure 14. Photo. Example 4 of pedestrian detection
Figure 15. Illustration. VSH
Figure 16. Equation. Bayesian rule
Figure 17. Equation. VSH for a given grid cell
Figure 18. Equation. Maximum number of pixels in each image row
Figure 19. Equation. Maximum number of image rows in a specific height band
Figure 20. Equation. Normalization factor for each cell in which VSH is calculated
Figure 21. Equation. Feature vector extracted from each image patch
Figure 22. Illustration. Two views of the feature space showing the distribution of vectors from which the class conditional likelihoods are estimated
Figure 23. Photo. Likelihood density estimation of original (left) and labeled structures (right) showing buildings and candidate objects
Figure 24. Illustration. Top view of VSH components: hlow (left), hmid (center), and hhi (right)
Figure 25. Illustration. VSH projected on to the image (left three images) and the height of each pixel (right)
Figure 26. Illustration. Likelihoods conditioned on the four labels: candidate objects, vertical structures, ground, and overhanging structures
Figure 27. Equation. Kernel density estimation on the feature vector extracted from each image patch
Figure 28. Equation. Bi-weight kernel used in the kernel density estimation function
Figure 29. Equation. Gibbs distribution used to model the prior probability
Figure 30. Equation. Binary variable used to test the depth neighborhood of image patches
Figure 31. Equation. Smoothness cost associated with each image patch pair
Figure 32. Photo. Process of contour and HOG classification for (a) fixed sub-ROI, (b) local ROI, (c) foreground mask from contour matching, and (d) filtered HOG directions underlying masked regions
Figure 33. Illustration. Example of local contour models
Figure 34. Equation. Foreground mask for the contour template
Figure 35. Photo. Foreground mask examples
Figure 36. Photo. Overview of the pedestrian tracker
Figure 37. Equation. Image correlation tracker
Figure 38. Illustration. Pedestrian tracker data flow
Figure 39. Photo. Pedestrians crossing at an intersection during the day under good lighting conditions
Figure 40. Photo. Pedestrians crossing at an intersection during the day while a vehicle turns right
Figure 41. Photo. Pedestrian crossing an intersection at night
Figure 42. Photo. Pedestrians crossing a road at midblock during the evening
Figure 43. Photo. Pedestrians crossing a road at midblock during the early evening
Figure 44. Photo. Pedestrians crossing a road at an intersection at night
Figure 45. Photo. Vehicle driving on the highway
Figure 46. Photo. Second view of vehicles driving on the highway with tall vertical poles and overhang bridge in the field of view
Figure 47. Photo. Pedestrians crossing midblock in a multilane urban street with overhang bridge as overlapping background
Figure 48. Photo. Pedestrian crossing the street and right-turning vehicle in winter
Figure 49. Photo. Pedestrians on the sidewalk in an urban environment during winter
Figure 50. Photo. Pedestrians walking in the roadway near parked vehicles in an urban environment
Figure 51. Photo. Pedestrians at a crosswalk in front of a vehicle in bright conditions with saturated areas
Figure 52. Graph. ROC curves for Seq00
Figure 53. Graph. ROC curves for Seq01
Figure 54. Graph. ROC curves for Seq02
Figure 55. Graph. ROC curves for Seq03
Figure 56. Photo. Sample output from SC in an alleyway
Figure 57. Photo. Sample output from SC in a dense urban scene with pedestrians in the vehicle path
Figure 58. Photo. Sample output from SC in an urban scene with pedestrians at varying distances from the vehicle
Figure 59. Photo. Sample output from SC in an urban scene with pedestrians entering a building and others in the distance ahead of the vehicle
Figure 60. Photo. SC rejecting poles
Figure 61. Photo. Appearance classifier recognizing a pedestrian
Figure 62. Photo. Appearance classifier output recognizing pedestrians crossing in front of vehicles
Figure 63. Photo. Appearance classifier output recognizing pedestrians while making a left turn
Figure 64. Photo. Appearance classifier recognizing pedestrians in front of a vehicle in a busy urban street
Figure 65. Photo. Appearance classifier recognizing pedestrians 98.4 ft (30 m) ahead of a vehicle in a busy street
Figure 66. Screenshot. Main screen of the GUI for PD and classification
Figure 67. Screenshot. PD interface—display all detected pedestrian candidates
Figure 68. Screenshot. PD interface—PCS-Ped tab with option selected to display detected pedestrians
Figure 69. Screenshot. PD interface—PCS-Ped tab with option selected to display horizon line estimated by the system
Figure 70. Screenshot. PD interface—PCS-Ped tab with option selected to display the SC output
Figure 71. Screenshot. PD interface—PCS-Ped tab with option selected to display an intermediate VSH output of SC
Figure 72. Screenshot. PD interface—PCS-Ped tab with option selected to display depth/disparity map
Figure 73. Screenshot. PD interface—PCS-Ped tab with option selected to capture stereo data for temporary storage
Figure 74. Screenshot. PD interface—PCS-Ped tab with option selected to cancel saving of stereo data and clear temporary store
Figure 75. Screenshot. PD interface—PCS-Ped tab with option selected to stop capture and store captured stereo data to permanent storage
Figure 76. Screenshot. PD interface—PCS-Ped tab with option to automatically divert data to a file whenever a pedestrian is detected
Figure 77. Screenshot. PD interface—PCS-Ped tab with option selected to define maximum number of frames maintained in temporary storage during automatic divert of data
Figure 78. Screenshot. PD interface—PCS-Ped tab with option selected that specifies number of additional video frames saved to disk
Figure 79. Screenshot. PD interface—PCS-Ped tab with option selected that specifies whether PD algorithms should operate while data being stored
Figure 80. Screenshot. PD interface—PCS-Ped tab with option selected that enables PD algorithm to run in live system
Figure 81. Screenshot. PD interface—PCS-Ped tab with option selected that enables PD algorithm to split wide object detections into multiple pedestrian candidates
Figure 82. Screenshot. PD interface—PCS-Ped tab with option selected that enables PD algorithm to refine horizontal placement of initial detection box
Figure 83. Screenshot. PD interface—PCS-Ped tab with option selected that enables PD algorithm to refine vertical placement of initial detection box
Figure 84. Screenshot. PD interface—PCS-Ped tab with option selected that enables PD algorithm to use ground plane estimate to better locate pedestrians
Figure 85. Screenshot. PD interface—PCS-Ped tab with option selected that enables PD algorithm to maintain a fixed aspect ratio when detection boxes are refined
Figure 86. Screenshot. PD interface—PCS-Ped tab with option selected that enables PD algorithm to use image edge information to reject FPs
Figure 87. Screenshot. PD interface—PCS-Ped tab with option selected that enables PD algorithm to use image depth information to reject FPs
Figure 88. Screenshot. PD interface—PCS-Ped tab with option selected that enables PD algorithm to use SC algorithm to detect tall vertical structures
Figure 89. Screenshot. PD interface—PCS-Ped tab with option selected that enables PD algorithm to reject FPs as indicated by SC algorithm
Figure 90. Screenshot. PD interface—PCS-Ped tab with option selected that enables PD algorithm to use ground plane and horizon information to reject FPs
Figure 91. Screenshot. PD interface—PCS-Ped tab with option selected that enables PD algorithm to use image saliency information to reject FPs
Figure 92. Screenshot. PD interface—PCS-Ped tab with option selected that enables PD algorithm to reject FPs detected by three previous rejection algorithms
Figure 93. Screenshot. PD interface—PCS-Ped tab with option selected that enables PD algorithm to compute the ground plane in the scene
Figure 94. Screenshot. PC interface specifying search range for each ROI in the X and Y directions
Figure 95. Screenshot. PC interface showing scale evaluation parameters
Figure 96. Screenshot. PC interface showing specifications at ROI classification
Figure 97. Screenshot. PC interface indicating legacy parameters for debugging
Figure 98. Screenshot. PC interface specifying size of padding around a detection box
Figure 99. Screenshot. PC interface showing filter options
Figure 100. Screenshot. PC interface showing selection options for image enhancement prior to classification
Figure 101. Screenshot. PC interface showing options for classifier output display
Figure 102. Screenshot. PC interface showing selection options to run PC and a post-processing SVM classifier for bush rejection
Figure 103. Screenshot. PC interface showing options to select a HOG AdaBoost classifier or contour plus HOG AdaBoost classifier
Figure 104. Screenshot. PC interface showing selection options to decide distance ranges for three classifiers
Figure 105. Screenshot. PC interface showing classifier debugging options
Figure 106. Screenshot. PC interface showing tracker options
Figure 107. Screenshot. PC interface showing classifier threshold options
Figure 108. Screenshot. Pedestrian tracker interface showing options to set tracker search range
Figure 109. Screenshot. Pedestrian tracker interface showing tracker options

LIST OF TABLES

Table 1. Pedestrian detection performance specifications
Table 2. Parameter settings for SC
Table 3. In-path detection results for sequence 080613111722_BM-SHJ_cross-in-front (parking lot)
Table 4. Full field-of-view detection results for sequence 080613111722_BM-SHJ_ cross-in-front (parking lot)
Table 5. Full field-of-view detection results for sequence 80613112933_ SHJ_walk_BM_stand_on-side (parking lot)
Table 6. Full field-of-view detection results for sequence EuropeTour_ Innsbruck.0_20070128_42_SVS_Data
Table 7. Full field-of-view detection results for sequence EuropeTour_ Wurzburg.0_20070126_19_SVS_Data
Table 8. In-path detection results for Sequence seq00_rerun (Ess sequence)
Table 9. Full field-of-view detection results for sequence seq00_rerun (Ess sequence)

 

Federal Highway Administration | 1200 New Jersey Avenue, SE | Washington, DC 20590 | 202-366-4000
Turner-Fairbank Highway Research Center | 6300 Georgetown Pike | McLean, VA | 22101