Skip to content U.S. Department of Transportation/Federal Highway AdministrationU.S. Department of Transportation/Federal Highway Administration

Office of Planning, Environment, & Realty (HEP)
PlanningEnvironmentReal Estate

HEP Events Guidance Publications Awards Contacts

The Possible Effects of Commercial Electronic Variable Message Signs (CEVMS) on Driver Attention and Distraction

Appendix B - Detailed Description of Studies

B.1 On-road Instrumented Vehicle Approach

The most effective research strategy to emerge from the analysis undertaken in section 6.0 is the on-road instrumented vehicle method. The following describes one possible study which might be conducted using this method.

B.1.1 Method

The on-road instrumented vehicle method employs an instrumented vehicle which is brought to the study site, along with a crew of about two or three researchers. The study site is a location where there is at least one CEVMS installation along a public access roadway. Preferably, there would be several CEVMS installations at the location so that a single test driving scenario might pass a few different CEVMS in the course of about half an hour of driving. The investigation should include at least two or three study sites which already have CEVMS in place. At each study site, approximately 20 to 30 research participants would be recruited from the local area.

Each research participant would drive the instrumented vehicle along a prescribed route, which includes CEVMS installations, standard (non-digital) billboards, human-constructed objects of casual visual interest (houses, barns, etc.), and natural background control scenery (trees, fields, etc.). Each drive takes less than 1 hour (preferably about 30 minutes), and each participant would return for several drives on different days. Other aspects would vary as well, such as the time of day, traffic density, and CEVMS conditions (e.g., CEVMS turned on versus CEVMS turned off). Each participant would complete between three and six such drives. The instrumented vehicle and crew would usually remain at a given study site for about 1 to 2 months. The crew would consist of an experimenter and a safety observer, who would both be present in the instrumented vehicle. The safety observer would also serve as a research assistant or technician. The instrumented vehicle is capable of measuring vehicle speed, vehicle lane position, longitudinal acceleration, lateral acceleration, GPS time and position, and driver eye glance direction and duration. The instrumented vehicle is also equipped with accurate vehicle-mounted or head-mounted eye-tracking equipment, video cameras (forward and cab views) and a voice recorder.

Top of Page

B.1.2 Factors and Measures

The major factors or independent variables in the study are the presence or absence of CEVMS and other comparison visual stimuli (standard billboards, buildings, etc.) along the driving path. If possible, the CEVMS should be capable of being turned off and on or changed along some other dimension like luminance or change rate, according to a prearranged experimental design. The period of time that the CEVMS is off or changed could be kept relatively brief and carefully controlled since the study will follow a strict protocol. Other important independent variables are the time of day (day/night), traffic conditions (peak and nonpeak), and driver variables (age, gender, and route familiarity). One or more of the primary CEVMS variables of interest to the community concerned with outdoor advertising control should be represented by varying levels along the driving route (e.g., different degrees of luminance, change rate, or display spacing) as much as possible. Direct experimental control would be preferable to site selection in this regard.

The primary measure or dependent variable in this study is the frequency, direction, and duration of driver eye glances, which serves as an indication of visual attention and distraction. The fundamental hypothesis is that drivers have limited attention; they self-regulate their attention to perform demanding tasks. In the case of the driving task, a certain proportion of their attention needs to be concentrated on the roadway scene ahead. To the degree that eye glance behavior can serve as a measure of visual attention, eye glances need to be concentrated on the roadway ahead. If the frequency and duration of eye glances away from the roadway ahead exceed accepted norms or criteria for keeping a driver's eyes on the road, then driver safety may be compromised. Thus, eye glance behavior is the primary dependent variable in the study. Eye glance behavior has an intuitive connection to visual attention and is sensitive to subtle visual search strategies, including those which are below the level of conscious awareness (see section 2.7.2). Depending upon the type of eye glance measuring instrumentation selected, the act of measuring eye glance behavior may prove to be a more or less significant distraction to the driver in itself. This experimentally-induced artifact can be controlled by selecting a minimally intrusive measurement method or by ensuring adequate adaptation to the instrumentation on the part of the research participant.

This study includes another class of secondary dependent variables. These are safety surrogate measures associated with driver errors and other measures of driver performance, such as speed changes, headway, lane deviation, and traffic conflicts. These secondary variables can be measured by instrumentation in the vehicle in terms of speed, acceleration, and lane position. These secondary variables can also be directly observed and noted by the experimenter and/or safety observer in the instrumented vehicle for later analysis in terms of sudden braking, inadequate headway, swerving, and conflicts. Thus, events indicative of possible driver error or other maladaptive behavior can be flagged by human observers. Also, for these events, only objective vehicle performance data needs to be analyzed, saving considerable effort and expense by eliminating the need to analyze large amounts of continuous vehicle performance data.

Top of Page

B.1.3 Advantages/Disadvantages

One advantage of this method is its ability to implement accurate eye-tracking measurements which afford the opportunity to observe subtle and often unconscious eye movements. This ability to measure unconscious eye movements correlates with unconscious distraction facilitates incorporation of the notion of self-regulated attention into the experimental paradigm. When a driver is attempting to concentrate on the roadway ahead, a distractor, which unconsciously diverts attention away from the roadway against the driver's will, may have a more severe safety consequence than a distractor which can be maintained under conscious and voluntary control. Thus, in addition to being able to measure distraction which is both conscious and voluntary, accurate eye-tracking determinations have the potential to probe other phenomena, such as unconscious and involuntary distraction as they relate to CEVMS exposure.

Another advantage of this method is the ability to structure driving scenarios to have an appropriate number of CEVMS, standard billboard, and other visual stimuli all located on a controlled course, which all research participants drive in a consistent manner. The ability to choose and structure the test drive assures adequate and uniform exposure to CEVMS and other relevant visual stimuli. The ability to exert experimental control is a valuable asset to this method. It facilitates a clean and robust statistical analysis of the data because all of the participants are exposed to all of the experimental conditions the same number of times in a relatively controlled manner. Experimental control ensures a high level of CEVMS exposure, thereby contributing to the productivity and cost effectiveness of this technique.

However, examined from a different perspective, such a degree of experimental control may also be regarded as a disadvantage. A certain amount of artificiality is introduced into the driving situation thereby. Research participants are definitely aware that they are participating in a controlled experiment, driving someone else's car on a contrived route which does not serve a personal purpose related to daily life. In addition, with the experimenter riding along with the participants in the vehicle, there may be a tendency for the participants to try to please the experimenter and to drive in some unnatural way. The introduction of eye-tracking equipment adds to the artificiality of the situation. Wearing head-mounted eye-tracking gear definitely represents unnatural driving attire. However, most research participants rapidly adapt to the gear with time, and they often report that they are unaware of its presence after a short drive. Vehicle-mounted eye-tracking equipment can be far less intrusive, although the tedious calibration procedures and the presence of the cameras in the car remind participants that their head and eye movements are constantly being monitored. These are all valid experimental concerns; however, none of these interventions is likely to profoundly alter the driving behavior, much less the eye glance movements, of the research participants, as long as they are not informed of the purpose of the study. The enhanced experimental efficiency that this approach has to offer far outweighs its artificiality drawbacks.

Top of Page

B.1.4 Budgetary Cost

A rough budgetary estimate for conducting such an on-road instrumented vehicle study is between $400,000 and $800,000. The main cost drivers for this method are the eye glance measuring technology and the crew needed to implement the experiment at the study sites. The range in this estimate relates to the number of study sites, adequacy of the sites, length of the experimental drive, number of experimental drives, number of research participants, difficulty in obtaining research participants, ability to turn the CEVMS off and on, and numerous other factors which cannot be determined without further planning.

Top of Page

B.2 Naturalistic Driving Approach

The naturalistic driving method is similar to the on-road instrumented vehicle method. The major difference is that the participants drive their own vehicles (or loaned vehicles) for their own personal purposes. The method typically employs a large number of such vehicles. The following describes one possible study which might be conducted using this method.

Top of Page

B.2.1 Method

The naturalistic driving method employs a standardized instrument package which is installed in the participant's own private vehicle or in a vehicle loaned to the participant. The installation is made as unobtrusive as possible so that the participant's vehicle appears and performs as it normally would. The instrument package is capable of measuring many of the same variables as the on-road instrumented vehicle, such as speed, lane position, acceleration, GPS time and position, driver eye glance frequency, direction, and duration. The instrument package is also connected to the vehicle data bus so that additional vehicle-related measures of engine, braking, and steering performance are also recorded. However, because of the unobtrusive nature of the experimental technique, this method cannot support the use of extremely accurate head-mounted or vehicle-mounted eye-tracking equipment. In the present state of technology, these accurate eye movement instruments involve careful calibration procedures with the driver. With this method, the eye-tracking system is mounted in the dashboard in a manner which involves little or no driver interaction. Once the participant's vehicle has been instrumented, data are collected by means of automatic wireless downloads without participant awareness or involvement. The instrumentation is left in the vehicle for a period of 3 to 6 months, during which time the participant drives the vehicle for normal personal or business use.

The fact that participants drive their own vehicles for their own use reduces control and adds uncertainty to the study. It is difficult to control where the participants are going to drive and when. The study site must be selected carefully so that participants are likely to drive by at least some of the target CEVMS installations. The participants must be selected carefully so that they are likely to take the selected roadway with some reasonable frequency. As a result of this increased uncertainty, the number of study sites must be increased to 4 and 5, the number of research participants selected at each site must be increased to 50 and 75, and the duration of measurement for each participant must be increased to 3 and 6. In this study, it is even more important that there are several CEVMS installations at each study site. As was the case for the on-road instrumented vehicle study, each study site needs to include CEVMS installations, standard (non-digital) billboards, objects of casual visual interest (houses, barns, etc.), and natural background control scenery (trees, fields, etc.).

Top of Page

B.2.2 Factors and Measures

As with the on-road instrumented vehicle study, the major factors or independent variables are the presence or absence of CEVMS and other comparison visual stimuli (standard billboards, buildings, control settings, etc.) along the driven path. If possible, the CEVMS should be turned off and on or changed in some other way, according to a prearranged experimental design. However, in this instance, the CEVMS would have to be turned off or changed for longer periods of time because it is not certain when the instrumented test vehicles might pass. These are the primary independent variables. Secondary independent variables could include the type of vehicle (sedan, pickup, or SUV) and driver characteristics (age, gender, and route familiarity). In addition, as much as possible, one or more of the primary CEVMS variables of interest to the community concerned with outdoor advertising control should be represented by varying levels in the selection of CEVMS stimuli.

As in the on-road instrumented vehicle study, the primary measure or dependent variable is the frequency, direction, and duration of driver eye glances. The fundamental hypothesis of self-regulated attention which needs to be concentrated on the roadway scene ahead remains the same. As before, if the frequency and duration of eye glances away from the roadway ahead exceed accepted norms or criteria, then driver safety is assumed be compromised. Thus, eye glance behavior is the primary dependent variable in this study, as well. However, the particular unobtrusive and disengaged dashboard-mounted eye-tracking device may not be capable of making as accurate measurements of eye-movements as can other more delicate vehicle-mounted or head-mounted devices which require periodic participant calibration. Consequently, this study method depends more heavily on secondary dependent variables. Safety surrogate measures associated with driver errors and other measures of driver performance (headway, lane deviation, conflicts, and erratic maneuvers) become increasingly important in this method. Since the participants will be driving according to their own personal schedules, additional dependent variables may include the time of day (day/night), traffic conditions (peak and nonpeak), in-vehicle distractions (eating and/or cell phone use), and state of fatigue.

Top of Page

B.2.3 Advantages/Disadvantages

The naturalistic driving method possesses one major advantage over the on-road instrumented vehicle method: the driving scenario, driving task, and driving purpose are all completely natural. The research participants drive their own vehicles (or ones loaned to them) on their own personal schedules along personally selected routes to meaningful destinations. Although to a lesser degree, the naturalistic driving method shares another advantage with the on-road instrumented vehicle method: its ability to implement eye-tracking measurements. In fact, the dashboard-mounted eye-tracking device is far less intrusive to the driver than the head-mounted eye-tracking device sometimes employed in the on-road instrumented vehicle method.

Unfortunately, some dashboard-mounted eye-tracking devices may not be as sensitive and accurate as a head-mounted device. Also, they may not be able to track extensive head movements or measure subtle eye glances indicative of unconscious distraction. The useful field of view can also be an issue with certain unobtrusive vehicle-mounted eye-tracking equipment. Consequently, this experimental method may be less effective in its ability to probe the subtle phenomena of unconscious and involuntary distraction as they relate to CEVMS exposure.

Another disadvantage of this method is its inherent lack of structured driving scenarios. Since participants drive whenever and wherever they want, it is difficult to ensure adequate and uniform exposure to CEVMS and other relevant visual stimuli. This lack of experimental control and higher degree of uncertainty necessitate an increase in the number of study sites, research participants, and duration of the study, which negatively impacts the productivity and cost effectiveness of the technique. For example, this method typically requires the instrumentation of a relatively large number of vehicles at any given study site instead of the instrumentation of just one vehicle which is shared by many research participants. Another minor disadvantage is that research participants are aware that they are participating in an experiment, even if the study is minimally intrusive in terms of daily life routine.

Top of Page

B.2.4 Budgetary Cost

A rough budgetary estimate for conducting such a naturalistic driving study is between
$2 million and $4 million. The main cost drivers for this method include increasing the number of study sites, installing instruments in a large number of vehicles at a single site, and collecting and analyzing data covering a long period of time. The range in this budgetary estimate relates to the number of study sites, adequacy of the sites, number of vehicles which need to be instrumented at one time, number of research participants, difficulty in obtaining research participants, driving patterns of the research participants, length of the study at any given site, ability to turn the CEVMS off and on, and numerous other factors which cannot be determined without further planning.

Top of Page

B.3 Unobtrusive Observation Approach

The unobtrusive observation method is different from the on-road instrumented vehicle method and the naturalistic driving method. The major distinction is that no study participants are selected, and all data are obtained from the natural flow of traffic past the CEVMS and other comparison stimuli. The following describes one possible study which might be conducted using this method.

Top of Page

B.3.1 Method

The unobtrusive observation method employs an array of static cameras or other sensors mounted near the locations of the CEVMS and other comparison stimuli. The other sensors may include loops, tubes, or radar to measure vehicle passes and driving parameters. The present report will focus on video recording of traffic. The cameras are capable of recording the behavior of vehicles passing the various relevant visual stimuli as a part of the natural flow of traffic. The drivers are usually completely unaware that their vehicles are being observed. Post-hoc analysis of the video recordings from these cameras can yield data similar to some of that obtained by the on-road instrumented vehicle and naturalistic driving methods, which include vehicle speed, lane position, acceleration, and time. However, the data from distal video cameras are usually far less accurate than what can be collected by instruments onboard the vehicle. Moreover, with present measurement technology, such video recordings cannot yield any data concerning driver eye glance frequency, direction, and duration. The camera arrays are usually left in place for a period of several months to 1 year at each study site. There would typically be three to four such sites in the study. At each study site, separate camera arrays would need to be installed at the locations of all selected CEVMS displays, standard (non-digital) billboards, objects of casual visual interest (houses, barns, etc.), and natural background control scenery (trees, fields, etc.).

Top of Page

B.3.2 Factors and Measures

As in the on-road instrumented vehicle and naturalist driving studies, the major independent variables are the presence or absence of CEVMS and other comparison visual stimuli (standard billboards, buildings, etc.) along the driving path. If possible, the CEVMS should be controlled according to a prearranged experimental protocol. However, in this instance, the CEVMS would have to be changed for longer durations because it is possible to predict when vehicles might pass. In addition, one or more of the primary CEVMS variables of interest to the community concerned with outdoor advertising control should be represented by varying levels in the selection of CEVMS stimuli. These constitute the primary independent variables. Since continuous video recording will be employed, the experimenter can decide to select different times of data collection for further analysis. This capability can provide insight into some secondary independent variables such as time of day (day/night) and traffic conditions (peak, nonpeak).

Top of Page

In contrast to the on-road instrumented vehicle and naturalistic driving studies, the primary dependent variable is not driver eye glance behavior. Instead, this study method depends completely on safety surrogate measures associated with driver errors and other measures of driver performance (headway, lane deviation, and erratic maneuvers). These are subtle driving behaviors to measure by means of distal cameras mounted along the roadway. Unless the cameras are mounted very high, multiple vehicle images may occlude each other. For a long stretch of roadway, such as might required for CEVMS exposure, a relatively large array of cameras may be needed. Thus, a large amount of data needs to be collected and analyzed in such a study. Automatic machine vision video analysis algorithms can help in the data analysis process, but such algorithms are not yet sufficiently sensitive and robust to reliably identify all of the subtle indicators of driver errors, conflicts, or maladaptive performance which might accompany CEVMS exposure. The use of other sensors instead of or in addition to cameras may mitigate some of these data analysis problems to a certain extent.

Top of Page

B.3.3 Advantages/Disadvantages

The unobtrusive observation method possesses one major advantage over the other two methods: the data are derived from the natural flow of traffic. Other than erecting camouflaged camera arrays at various locations along the roadway, the experimenter does not disturb the natural flow of human driving. As opposed to the other two methods, the vast majority of drivers are completely unaware that they are part of a study depending on how well the camera camouflage works. Other sensors used for this application can also be hidden and made extremely hard to detect. This is the major advantage of the unobtrusive observation method. Another strong advantage is the large number of vehicles which pass by the CEVMS and other comparison stimuli every day. Sample sizes can be relatively large.

Like the other techniques, the unobtrusive observation method has disadvantages as well. First, with present technology, it is not possible to implement eye-tracking measurements in such a study. The inability to measure eye glance behavior makes it difficult to investigate important constructs, like self-regulated attention and unconscious distraction as they relate to CEVMS exposure. The method is left to rely on safety surrogate measures, such as driver errors and maladaptive maneuvers. These relatively subtle pre-crash and near-crash driving behaviors are difficult to measure by means of distal video cameras. Such driving behaviors also occur very seldom and need to be observed over great distances, leading to the necessity to collect large amounts of video data from extended camera arrays over long periods of time. The collection, reduction and analysis of such large amounts of data tend to make this method time-consuming and expensive.

Top of Page

B.3.4 Budgetary Cost

A rough budgetary estimate for conducting such an unobtrusive observation study is between $1 million and $3 million. The main cost drivers for this method include designing camera arrays which can measure subtle vehicle maneuvers, installing camera arrays to record a large extent of roadway for all CEVMS and comparison stimuli, and collecting and analyzing data covering a long period of time. The range in this budgetary estimate relates to the number of study sites, adequacy of the sites, number and location of cameras in an array, method of recognizing safety surrogate measures, length of the study at any given site, ability to turn the CEVMS off and on, and numerous other factors which cannot be determined without further planning.

Updated: 09/05/2014
HEP Home Planning Environment Real Estate
Federal Highway Administration | 1200 New Jersey Avenue, SE | Washington, DC 20590 | 202-366-4000