Layered Object Recognition System for Pedestrian Sensing

7. CONCLUSIONS AND RECOMMENDATIONS FOR FUTURE WORK

The research team developed a real-time in-vehicle vision-based stereo system that detects and recognizes pedestrians in the camera’s field of view. The system uses a layered or hierarchical approach that progressively operates on all or part of the input image data, with each step increasing in computational complexity and reducing the image area that needs to be processed by subsequent steps. The system integrates multiple cues including depth, appearance, and motion. The key steps are as follows:

1. Large-scale object extraction using stereo depth templates.
   
   
2. SC recognition of multiple classes including ground, buildings, trees, and poles and separation of those classes from vehicles and pedestrians.
   
   
3. Appearance classification using a cascade of classifiers that explicitly recognizes pedestrians and discriminates against other objects such as vehicles and bushes.
   
   
4. Pedestrian tracking using shape and appearance matching.

Based on offline and live experiments using a Toyota® Highlander with a stereo camera head and a personal computer processing unit, the following conclusions were made:

• The system achieves state-of-the-art performance for detection rate and FP rate when compared to other published results.
  
  
• The FP rate achieved by the system is not low enough for deployment as a stand-alone system. This performance argues for either the use of an additional sensor (e.g., using radar/light detection and ranging or reducing the horizontal field of view of the stereo camera to achieve production-level performance).
  
  
• The system needs further optimization to improve its performance to 15 Hz or higher. A higher frame rate is needed so that the system can be used on vehicles traveling at speeds higher than 30 mi/h (48.3 km/h).

If successful, the following recommendations for future work could lead to a commercially viable system:

• Enforce optimizations to increase throughput and reduce system latency.

Implement the developed system on an embedded platform. Potential candidates include the Acadia II™ application-specific integrated circuit in combination with a field-programmable gate array (FPGA) or the automotive-grade multiple digital signal processor and FPGA system jointly developed by Autoliv Electronics and Sarnoff Corporation.

• Improve the classification performance to further reduce FPs. Most of the FPs are from specific objects. While some progress has been made to remove these types of consistent FPs, there is a need for additional development to categorize these detections using supervised or unsupervised learning techniques and to build more focused classifier cascades that can reject them.
  
  
• Test and create enhancements to enable the system to operate in off-road conditions and construction zones.