U.S. Department of Transportation
Federal Highway Administration
1200 New Jersey Avenue, SE
Washington, DC 20590
Federal Highway Administration Research and Technology
Coordinating, Developing, and Delivering Highway Transportation Innovations
This magazine is an archived publication and may contain dated technical, contact, and link information.
|Publication Number: FHWA-HRT-10-004 Date: May/June 2010|
Publication Number: FHWA-HRT-10-004
Issue No: Vol. 73 No. 6
Date: May/June 2010
A new multisensor system differentiates vehicles on the road, which could save lives.
|.Shown here is a prototype motorcycle classifier, a multi-instrument sensor designed to detect cyclists at intersections. The sensor includes an infraredvisible light stereo camera (top left), an infrared thermal camera (right), and an acoustic sensor (below left)|
According to the National Highway Traffic Safety Administration (NHTSA), the number of U.S. traffic fatalities in 2008 reached its lowest level since 1961. Despite an overall improvement in safety, however, motorcyclist deaths continued their 11-year increase, reaching 5,290 in 2008, accounting for 14 percent of all highway fatalities. Bicyclists, too, face disproportionate dangers on the Nation's roadways. Although bicycle trips accounted for less than 1 percent of all trips in 2008, bicycle riders represented 2 percent of all traffic fatalities.
Statistics for intersection crashes are similarly disproportionate. Based on 2008 data from NHTSA's Fatality Analysis Reporting System, 270 (or 3.5 percent) of the 7,772 intersection fatalities were bicyclists, while another 1,441 (or 18.5 percent) were motorcyclists. Another way of looking at motorcyclist and bicyclist fatalities at intersections is to compare the proportion that occurred at intersections with the proportion that occurred elsewhere. For example, in 2006, 2007, and 2008, a total of 4,283 motorcyclists were killed in intersections, representing 16.9 percent of all fatalities at intersections. Over the same period, of all motorcyclist fatalities on U.S. roads, a total of 28 percent occurred at intersections. Similarly, bicyclists accounted for 786 intersection fatalities during those 3 years, representing 3.1 percent of all intersection fatalities. Intersections, the data reveal, were the site of 36 percent of all bicyclist fatalities.
Further still, per vehicle miles traveled (VMT), NHTSA estimates that motorcyclists are now about 34 times more likely than passenger car occupants to die in a motor vehicle crash. But these estimates are highly dependent on the accuracy of VMT estimates for motorcycles, which is challenging to determine using existing traffic sensors. Intelligent transportation system (ITS) technologies typically use inductive loop and magnetometer sensors to detect cars and trucks, but the sensors are not as effective at detecting and classifying motorcycles and bicycles. These sensors detect the effects that electrically conductive materials have on electromagnetic fields, but both motorcycles and bicycles have low conductive masses. Loop detectors require that bicyclists be near the pavement markings for the sensors to trigger, and the detectors' sensitivities are difficult to set to avoid false detections. With regard to motorcycles, loop detectors only can detect their presence, not their types. In fact, participants at a 2007 Motorcycle Travel Symposium sponsored by NHTSA and the Federal Highway Administration (FHWA) cited transportation infrastructure and how it detects and counts two- and three-wheeled vehicles as a key problem area for traffic engineers.
According to Dan Stewart, manager of the Maine Department of Transportation's (MaineDOT) Bicycle and Pedestrian Program, accurate detection is critical for intersection control devices to ensure that red and green traffic signals respond appropriately to motorcyclists and bicyclists and keep all modes of traffic flowing safely. "This problem is only increasing for motorcycles as they are made with less steel, making them harder for conventional systems to detect," he says. "It's frustrating for riders to obey the rules of the road when a traffic signal will not recognize a person on a bike or motorcycle."
Spurred by the 2007 symposium, FHWA and a traffic engineering firm embarked on a multiphase research project to develop a system to detect and classify two- and three-wheeled vehicles more effectively. Phase I of the project, now complete, focused on developing an accurate detector. The result is a multi-instrument device for detecting these vehicles at intersections and ultimately for improving safety for riders.
Developing improved sensors to detect cyclists at intersections could produce a number of benefits. Detection and classification technologies improve safety by enabling signal control systems to lengthen signal times for slower moving traffic, such as pedestrians and bicycles, and shorten signal times for faster moving motorcycles and automobiles. More accurate detection systems would enable ITS technologies to sense motorcyclists and bicyclists traveling alone -- that is, unaccompanied by larger motor vehicles nearby to trigger a detection -- and provide green lights, safety messages, and other cues to riders.
Sensors also can play an important role in traffic monitoring. ITS technologies installed at high-traffic locations can supply useful, continuous monitoring data on traffic demand and vehicle classification, while accomplishing their primary safety and operations objectives. Improved detection systems could benefit transportation planners by collecting more accurate data on demand levels to inform present traffic control needs and planning for future improvements. Improved vehicle classification also could help States meet FHWA requirements for reporting on two- and three-wheeled vehicles as described in FHWA's Travel Monitoring Guide (FHWA-PL-01-021).
"The city of Cambridge [MA] supports bicycles, mopeds, scooters, and motorcycles as part of our comprehensive climate protection plans," says Adam Shulman, a transportation planner with the city's Traffic, Parking & Transportation Department. "In just the past 6 years, bicycling has increased by 100 percent in Cambridge. If a new tool is created that can accurately and inexpensively count bicycles, we could better monitor our successes and help plan and prioritize additional ways to encourage these smaller, health-promoting, and less gas-dependent vehicles."
The researchers knew that the ideal sensor, which they termed "motorcycle classifier," would have to detect and classify motorcycles and bicycles in real-world roadway environments. The sensor would need to work in a variety of weather, lighting, and time-of-day conditions, including sunrise, sunset, noon, night, sun glare in the spring and fall, fog, drizzle, rain, and snow. This need for an over-roadway sensor that works in a variety of conditions led the researchers to select a multiple-technology rather than single-technology sensor.
Because of the inaccuracy of traditional ITS technologies for detecting two- and three-wheeled vehicles, the researchers focused on using infrared (IR) and visible light cameras. Specifically, the researchers chose an IR-visible light stereo camera to identify the riders on two- and three-wheeled vehicles, an IR thermal camera to distinguish cars and motorcycles from bicycles, and an acoustic sensor to distinguish classes of cycles such as large motorcycles versus mopeds.
The researchers designed their new motorcycle classifier based on a multisensor device previously developed as a pedestrian detector. (See "Detecting Pedestrians" in the September/October 2009 issue of Public Roads.) Using an IR-visible light stereo camera, the pedestrian detector focuses on the unique shapes of people, as distinct from other three-dimensional (3-D) objects in the roadway environment, to detect pedestrians crossing intersections. Similarly, two- and three-wheeled vehicles, with human riders, have distinctly different shapes than cars and trucks. So, the researchers applied the same technology to detect riders on two- and three-wheeled vehicles.
|The IR-visible light stereo camera took these photos of a motorcyclist, the above image by the left lens and the below image by the right lens.|
|Algorithms rendered the motorcyclist photos into these disparity maps. The revised algorithm produced the above image, which has a more completely filled-in disparity map.|
The logic flow (underlying programming in the detection system) of the motorcycle classifier uses the different sensors in sequence to detect a vehicle and progressively zero in on its classification. The sequential sensors also provide some overlap for redundancy, facilitating error checking.
|At left are two consecutive images of pedestrians crossing a street as photographed by a stereo camera. To the right are the corresponding disparity maps rendered by the algorithm. The shapes of the pedestrians are clearly visible popping out from the gray background in the disparity maps.|
The logic flow is as follows. (1) Using the IR thermal camera, ask, "Is it a bicycle?" If yes, classify it as a bicycle. (2) If it is not a bicycle, use the IR-visible light stereo camera and ask, "Is it a car or a motorcycle?" If a car (note that in this context a car might mean car, truck, or bus), classify it as such. (3) If the vehicle is not a bicycle or a car, it must be some kind of two- or three-wheeled motorized vehicle. (4) Using the acoustic sensor, ask, "What kind of two- or three-wheeled vehicle is it?" Then classify it as a heavy motorcycle, light motorcycle, moped, scooter, or motor tricycle.
The key to this approach is using the stereo camera (the pedestrian detector from the earlier research) to identify people as riders and therefore distinguishing the two- and three-wheeled vehicles from cars, trucks, and buses. The sensor uses a computer algorithm to build a 3-D disparity map (which provides depth information on objects in an image) and identify the profile of a human body. The term "disparity" comes from describing the 2-D vector between the positions of corresponding features seen by the left and right lenses of a stereo camera. The equipment can compute a map from the disparity coordinates (x, y, d) to a 3-D position, where x is the horizontal axis, y the vertical axis, and d the depth. Instead of detecting pedestrians, in this application the sensor detects cyclists through their 3-D body features in the disparity map.
When applying the pedestrian disparity technique to detecting cyclists, the researchers needed to address several issues. The first was the difference in speed between pedestrians and cyclists, and the need to capture two consecutive frames of the same object in relatively the same location. The speed of motorcyclists and even bicyclists ranges from 20 miles per hour, mi/h (32 kilometers per hour, km/h) to more than 60 mi/h (97 km/h), whereas pedestrians travel at the leisurely rate of 1-4 mi/h (1.6-6.4 km/h), or perhaps faster if running. At a speed of 20 mi/h (32 km/h), or 30 feet per second (9 meters per second), a 5-foot (1.5-meter)-long motorcycle will not occupy the same frame location long enough for the camera to shoot consecutive photos for comparison in the disparity map. To compensate, the researchers increased the sampling rate to 5 shots per second, far above the rate used in pedestrian detection.
The second issue is that although distinguishing a walking pedestrian alone is relatively simple, imaging a body hunched over riding a motorcycle is more complex. However, because the peak point (vertically) of a motorcycle detection is almost always a human body -- the rider's head -- the researchers could window in on the motorcyclist and positively detect him or her. (The same holds true for two riders on a single motorcycle.) The researchers had to revise the original pedestrian detection algorithms significantly to allow for the increased complexity of the disparity maps that would be analyzed. The revised algorithms yield a more completely filled-in disparity map, facilitating a more accurate detection and classification.
|These photos show the distinguishing thermal shape of a motorcycle as captured by the IR thermal camera, with the hot spots outlined in red. Note that the hot spots for motorcycles appear within the vehicle image. The motorcycle on the left is closer in range, and its hot spot associated with the engine and exhaust pipe is relatively isolated. The motorcycle on the right is farther away from the camera, and its engine and exhaust pipe show a relatively larger hot spot compared to the motorcycle's body.|
|These photos depict the IR images of cars, with the morphological shapes of hot spots outlined in green. Note that the hot spots of cars appear beneath the vehicle images, unlike those of motorcycles, which appear within the body of the motorcycle's profile. The researchers found that hot spots underneath cars can have different thermal shapes, which makes the classification more challenging, so they are developing algorithms to overcome this problem.|
As noted earlier, the first step in the logic flow is using the IR thermal camera to distinguish bicycles from other vehicles. To detect a bicycle, the camera is keyed to the unique thermal signature of bicycle wheels: They are much clearer and more readily distinguished than those of motorized vehicles. Unlike motorcycle wheels, which usually move more rapidly, bicycle wheels are better articulated and relatively sharp because they are not partially surrounded by a motorcycle or automobile body.
Similarly, the shapes and characteristics of the thermal signatures of motorcycles and cars help to distinguish between those vehicles. The camera renders the IR images in gray scale, with white being the hottest areas and black being the coolest. The bright IR regions of motorcycles and cars are located in different parts of the vehicle image (such as the engine and exhaust pipes) and are of somewhat different shapes. Researchers can select a region of the image and look at its thermal threshold and shape to determine with some certainty if the image shows a motorcycle. In addition, a closer look for the shapes of car undercarriages can help prevent misclassification of cars as motorcycles.
Because the IR camera technique might confuse cars with motorcycles, another sensor, the visible light stereo camera, then comes into play. The motorcycle classifier examines the red, green, blue (RGB) digital image from the camera and windows it to locate the vehicle wheels.
The key for the classifier to distinguish motorcycles from cars is the shape of the area around the front wheel. A motorcycle's front wheel typically protrudes alone, unlike its rear wheel and the front and rear wheels of a car or truck, which are at least partially enclosed within the vehicle's body. The front wheel therefore helps create a distinct image signature for a motorcycle, partially surrounded by a background of pavement instead of the nearby sheet metal of a vehicle body. This element of the logic flow completes the redundant error checking in the classification of a vehicle as a two- or three-wheeled motorcycle versus a car or truck.
At this point in the logic flow, only vehicles classified as "motorcycles" remain (which includes scooters and mopeds). All motorcycles are shaped about the same from the point of view of IR and visible light images. The researchers could have performed the subclassifications using a digital single lens reflex-type camera, but this would have been prohibitively expensive. Instead, they chose a simpler approach based on acoustics.
Motorcycles, scooters, and mopeds have distinct sound signatures due to differences in their engines and typical operating speeds. The researchers used digital signal processing with phase analysis of the sound to measure the spectrum features for each of the motorized two- and three-wheeled vehicle classes.
Classifying vehicles by acoustic or sound signature is straightforward logically but computationally intensive in its internal algorithms. First, the motorcycle classifier determines whether the sound is that of a motorcycle engine or scooter-moped engine. If the sound is coming from a heavier, more powerful engine, the system employs a second logic set that determines whether the source is a heavy or light motorcycle. If the acoustic pattern is that of a light engine, the system further examines whether the engine has the characteristics of a scooter engine or moped engine.
The researchers selected an outdoor test site near a major highway in Walpole, MA. They pointed both the stereo and IR cameras toward the highway, which is frequently traveled by vehicles and motorcycles. They placed a microphone near the cameras to record the sound signals. The researchers recorded data, including stereo images, IR images, and sounds, continuously from 9 a.m. to 5 p.m. To collect more motorcycle data, researchers deployed the system at the test site for more than 1 week under different weather conditions.
For the system test, the researchers manually scanned through all of the datasets and selected all the motorcycles and randomly selected vehicles. Because there were no bicyclists on the highway at the site during the test period, the researchers rode bicycles themselves and recorded the data.
During phase I, the researchers collected a total of 45 vehicle samples and found the performance of the multisensor motorcycle classifier promising, even though it misclassified vehicles on several occasions. Out of 12 cars, the system classified 1 as a heavy motorcycle and 1 as a light motorcycle. Out of 14 heavy motorcycles, the system classified 3 as cars. Out of 4 light motorcycles, the system classified 1 as a car. Out of 9 bicycles, the system classified 1 as a car. The system classified all 3 mopeds correctly, but classified 1 of 3 scooters as a light motorcycle.
The researchers conclude that the main reasons for the misclassifications are motion blur in the IR images and less-than-ideal acoustic quality. Development is underway to reduce the motion blur and improve the acoustic features using a microphone array.
MaineDOT's Stewart is optimistic about the motorcycle classifier's early showing. "This research has the potential to save lives and dramatically improve the transportation system," he says.
|The IR-visible light stereo camera captured these images of the front wheels of a motorcycle (top) and car (bottom). A motorcycle's front wheel, as photographed, is typically surrounded mostly by pavement, while a car's front wheel is surrounded mostly by sheet metal. These differences help the researchers differentiate motorcycles from cars.|
Next up: The researchers will continue to refine the tool in the second phase of the project, now underway with a 24-month project period. Phase II involves two multi-instrument sensors, one for bicycle detection at intersections and the other for detection and classification of motorcycles at intersections. The researchers will place the IR camera and stereo camera in one enclosure for better image alignment. The stereo camera will use high-resolution lenses. A microphone array will be embedded in the enclosure at strategic locations to reduce the impact of background acoustic noises such as wind. The algorithms will be further improved to enhance the image quality and explore new thermal and disparity features. The researchers also plan to separate the mixed acoustic signals to better classify different motorcycles traveling in a group. The sensors will be able to work as stand-alone detectors or be integrated into existing ITS installations. Phase II also will gauge the level of detections in inclement weather.
|Classified as Dataset||Bicycle||Heavy Motorcycle||Light Motorcycle||Car||Moped||Scooter|
Source: Migma Systems, Inc.
David R.P. Gibson, P.E., is a highway research engineer on the Enabling Technologies Team in FHWA's Office of Operations Research and Development. He has a master's degree in transportation from Virginia Tech. His expertise includes traffic sensor technology, traffic control hardware, modeling, and traffic engineering education.
Bo Ling earned his M.S. in applied mathematics in 1990 and Ph.D. in electrical engineering in 1993 from Michigan State University. He has served as principal investigator for numerous government-funded research projects. Dr. Ling is cofounder, president, and chief executive officer of Migma Systems, Inc. He became a senior member of IEEE in 1998 and is a part-time faculty member of the Department of Electrical and Computer Engineering at Northeastern University in Boston.
Spandan Tiwari received his M.S. in 2003 and Ph.D. in 2007 in electrical engineering from the Missouri University of Science and Technology. He has more than 7 years of experience in computer vision, pattern recognition, image processing, automatic target recognition, signal processing, machine learning, and intelligent processing. He joined Migma Systems in 2007. Dr. Tiwari is a member of IEEE and the International Society of Artificial Life.