U.S. Department of Transportation
Federal Highway Administration
1200 New Jersey Avenue, SE
Washington, DC 20590
Federal Highway Administration Research and Technology
Coordinating, Developing, and Delivering Highway Transportation Innovations
|This report is an archived publication and may contain dated technical, contact, and link information|
Publication Number: FHWA-RD-96-147
Date: October 1997
Development of Human Factors Guidelines for Advanced Traveler Information Systems and Commerical Vehicle Operations: Components of the Intelligent Transportation Systems: Designs Alternatives for In-Vehicle Information Displays
CHAPTER 3. RESULTS
The results will be described in four sections: (1) warning compliance, (2) driving safety, (3) trust in ATIS and self–confidence, and (4) situational awareness, workload, and message acknowledgment. The first two sections address how ATIS design characteristics influence compliance with warning messages and driving safety and the final two sections investigate the intervening variables and cognitive processes that govern compliance and safety.
Three repeated measures analysis of variance were used to evaluate the factors influencing the measures of compliance (time to 10 percent compliance, rise time, and integrated compliance). The analyses included age, gender, information availability, message style, message mode, and message location as independent variables. The significant results will be considered first in terms of the hypotheses and then in terms of the more general framework. The specific hypotheses include:
The first hypothesis states that ATIS messages will promote a higher level of compliance compared to the roadway signs. Table 5 shows the three measures of compliance (see figure 3) for each of the four conditions of information availability. The units for 10 percent compliance and rise time are seconds, and integrated compliance is measured as a percentage. The effect of information availability is not significant for time to 10 percent compliance, F(2.6, 42.2) = 0.90, p>0.05.(1) The effect is significant for rise time and integrated compliance, F(2.8, 46.2) = 3.69, p<0.05, and F(2.8, 45.1) = 46.24, p<0.0001, respectively. Since rise time reflects the time to progress from 10 to 90 percent of the maximum compliance, lower values correspond to a faster increase in compliance. The longer rise time suggests that drivers who receive both roadway and ATIS information are able to act on this information gradually over time, while drivers who receive neither roadway nor ATIS information are forced to react rapidly to avoid approaching hazards. Integrated compliance follows the hypothesized pattern showing higher levels of compliance when the ATIS information is present.
Table 5. Measures of compliance for the four levels of information availability with standard deviations in parentheses.
The second hypothesis states that the command message style will promote greater compliance compared to the notification message style. Message style refers only to the ATIS messages, the roadway sign style was not manipulated. The analysis does not uniformly support this hypothesis. Only the integrated compliance measure shows significance. The effect of message style on integrated compliance is consistent with the hypothesis. Integrated compliance with the command message is 52.0 percent compared to 47.7 percent for the notification message, F(1, 16) = 3.06, p<0.10. Considering only those cases when ATIS messages are presented shows a much stronger effect of message style, F(1, 16) = 19.51, p<0.001. When only ATIS messages are present, compliance with the command–style messages is 64.8 percent and 53.2 percent for notification–style messages.
The third hypothesis states that auditory messages will promote a higher compliance compared to text messages. This hypothesis is not supported by the analysis. None of the three measures showed a significant effect of message modality.
The final hypothesis states that the effects of message mode and style will interact, with drivers complying most fully with auditory command messages. This hypothesis is not supported.
Beyond the effects associated with specific hypotheses, several effects associated with driver age are also significant. A two–way interaction between display location and age indicates that a distributed display promotes greater integrated compliance with older drivers (54.4 percent versus 44.8 percent), but a centralized display promotes greater compliance for younger drivers (50.1 percent versus 48.5 percent), F(1, 16) = 5.81, p<0.05. Older drivers also showed longer rise time compared to younger drivers, F(1, 16) = 5.18, p<0.05. Older driver compliance rose from 10 percent to 90 percent in 10.5 s, compared to 8.9 s with younger drivers.
The surface similarity of the different measures of compliance suggests a high correlation between each of the three measures; however, the correlations between the measures are all relatively weak. Table 6 shows that the highest correlation is less than 0.2.
Table 6. Correlation matrix of measures of compliance.
The higher correlation between the rise time and integrated compliance may reflect a greater stability in these measures. Both the rise time and the integrated compliance draw upon a greater sample of driver behavior and so are likely to be more stable measures. The number of significant effects for each of the measures may also reflect this stability and resulting sensitivity. The low correlation between the measures may also reflect the fact that the three measures reflect different characteristics of compliance. The time to 10 percent compliance and the rise time both reflect the response time of the driver, while integrated compliance is a more holistic measure of compliance that reflects speed, magnitude, and duration of the compliance.
The effect of age is consistent with this interpretation of the differences between these measures. Rise time shows that younger drivers respond more quickly than older drivers. Time to 10 percent compliance shows a similar pattern with older drivers reaching 10 percent of their maximum compliance in 4.9 s, compared to 4.5 s for younger drivers; however, the difference is not significant, F(1, 16) = 1.20, p>0.05. Integrated compliance does not reflect response time and it suggests an opposite effect for older drivers. The integrated compliance for older drivers is 52.0 percent compared to 47.7 percent for younger drivers; however, this effect is not significant, F(1, 16) = 2.50, p>0.05. Although not significant, the direction of these effects suggests that integrated compliance reflects the conservative nature of drivers and the rise time and time to 10 percent compliance reflect the speed of a driver's reaction.
The effects of ATIS and driver characteristics on driver compliance are summarized in table 7.
Table 7. The effects of the independent variables on three measures of driver compliance.
Five repeated measures analysis of variance were used to evaluate the factors influencing five measures of driving safety. Each analysis included age, gender, information availability, message style, message mode, and message location as independent variables. These measures included:
Perceived driving performance is a subjective measure collected at the end of each scenario, with a maximum of 100 and a minimum of zero. The average number of crashes per hour is calculated by normalizing the number of crashes in each scenario by the scenario duration. Turn-signal use is calculated as a percentage and is based on whether or not drivers used the turn signal as they changed lanes. The RMS measures of lane position and velocity were calculated for the period immediately after the message was presented until the point where the drivers acknowledged the message. To enhance the sensitivity of the RMS measures, the analysis uses the RMS for the period just prior to the message presentation as a covariate.
The first hypothesis states that ATIS messages will reduce attention to the roadway and the second hypothesis states that ATIS messages will overload the driver with too much information. These hypotheses will be evaluated simultaneously by examining the effect of information availability on the measures of safety. Table 8 shows that information availability significantly affects three out of five measures of driving safety.
Table 8. The effect of information availability on driving safety.
The first two variables in this table suggest that driving safety is lower when the ATIS is the only information source. The effect of information availability on RMS velocity shows a different pattern of effects that is not consistent with either hypothesis. The lower level of safety when only ATIS information is available, compared to when ATIS and roadway information are both available, supports the hypothesis that ATIS information draws attention from the roadway into the vehicle. The effects of messages on lane position and velocity could be consistent with the hypothesis that ATIS information, particularly when paired with redundant roadway signs, will overload the driver. However, the pattern of results for compliance, situational awareness, and workload data suggests that information overload is not the primary contributor to the degraded safety associated with ATIS messages.
The third hypothesis states that the command messages will draw attention away from the roadway, leading to lower levels of driving safety. The main effect of message style for several variables shows modest support for this hypothesis. Table 9 shows that perceived driving performance and RMS velocity indicate that command messages tend to erode driving safety. To enhance the sensitivity of the RMS data, the RMS data from the time period just prior to the message onset was used as a covariate. This reduces the degrees of freedom from 16 to 15. Because command messages are generally shorter, they transmit less information and should impose less mental workload on the drivers. This suggests that the negative effect of command messages stems from their tendency to encourage overreliance on the ATIS. This overreliance draws drivers' attention from the roadway, leading to unsafe maneuvers and collisions with other vehicles.
Table 9. The effect of message style on driving safety.
The fourth hypothesis states that auditory messages will draw attention away from the roadway and text messages will tend to overload the driver. None of the safety–related variables shows a significant effect for message mode.
The final hypothesis states that message style and mode will interact to induce the lowest level of safety for auditory command messages or text notification messages. Only perceived driving performance showed a significant effect related to this hypothesis. A three–way interaction between gender, style, and mode suggests that mode and style may interact, but that this interaction depends on gender, F(1, 16) = 5.64, p<0.05. Figure 5 shows this interaction, indicating that females perceived their driving performance to be better with notification messages compared to command messages, but only when they were presented through the auditory mode. Male drivers thought they drove more safely when they received notification messages compared to command messages, but only when the messages were presented as text.
This supports the hypothesis that ATIS messages might undermine safety by promoting overreliance rather than by overwhelming drivers with information. If the ATIS messages were to undermine safety by overloading drivers, then performance should be worse for the visual notification messages.
Beyond the specific hypotheses, all the measures of driving safety, except for perceived driving performance, showed a significant effect for age. Older drivers performed more poorly than younger drivers.
Table 10. The effect of age on the five measures of driving safety.
Table 11 summarizes the effects of the independent variables on the measures of driving safety.
Table 11. The effects of the independent variables on the five measures of driving safety.
TRUST IN THE ATIS AND SELF–CONFIDENCE
Past research has shown trust and self–confidence to be important intervening variables that moderate reliance on automation. In this experiment, understanding variations in trust and self–confidence could illuminate the process that governs compliance with ATIS messages. Two repeated measures analysis of variance were used to evaluate the factors influencing trust in the ATIS and self–confidence. These analyses included age, gender, information availability, message style, message mode, and message location as independent variables. This analysis identified several significant effects that will be considered in terms of specific hypotheses and then in terms of how they might influence compliance with warning messages. The specific hypotheses regarding trust and self–confidence include:
The first hypothesis states that information availability will affect subjective ratings of trust and self–confidence. Trust will decline when ATIS information is unavailable and self–confidence will decline when roadway information is not available. The analysis shows that trust and self–confidence are affected by information availability in an orderly manner that is consistent with this hypothesis, F(2.6, 40.92) = 36.0, p<0.001 for trust and F(2.1, 32.8) = 6.6, p<0.005 for self–confidence. Figure 6 shows trust dropping when ATIS information is not available and self–confidence dropping when roadway information is not available. Thus, trust in the ATIS and self–confidence respond to gross changes in the system as hypothesized.
The second hypothesis states that trust in the ATIS with a command message will be more fragile than with a notification message. A two–way interaction between information availability and message style partially supports this hypothesis, F(1.97, 31.82) = 5.55, p<0.005. The interaction, shown in figure 7, between message style and information availability suggests that the command message style does not instill a very high level of trust, compared to notification messages. This is particularly pronounced when drivers are forced to rely on the ATIS when no roadway information is available. However, trust in both message styles declines to approximately the same level when ATIS information is not available. Thus, when command messages are not paired with redundant roadway information, drivers tend to distrust the ATIS, compared to notification messages. This finding is consistent with the more general hypothesis that command messages will lead to a more fragile sense of trust in the ATIS.
The third hypothesis states that self–confidence will be greatest with notification–style messages. The main effect of message style does not support this hypothesis, F(1, 16) = 1.17, p>0.05. However, the interaction between style and information availability supports this hypothesis, F(2.4, 39.0) = 3.42, p<0.05. Figure 8 shows that command messages diminish self–confidence when drivers have only ATIS information to rely upon. This figure also addresses the fourth hypothesis that states that self–confidence will be more robust with notification–style messages. This partially supports the hypothesis because self–confidence does not decline for drivers receiving notification messages in the ATIS–only condition. However, self–confidence for both message styles was approximately equal when neither roadway nor ATIS information is available.
Like message style, message mode interacts with message availability to affect trust, F(1.99, 31.82) = 5.55, p<0.01. Figure 9 shows that drivers' trust is initially higher with auditory messages, but that it is also more brittle, declining more when ATIS messages do not appear on a regular basis.
Beyond the specific hypotheses, trust also shows a main effect and interaction linked to age differences, F(1, 16) = 7.71, p<0.05, F(2.56, 40.92) = 7.08, p<0.005. Figure 10 shows that the older drivers' trust in the system is not as fragile as that of the younger drivers. The trust of younger drivers drops dramatically when ATIS information is not available, while the trust of older drivers declines only a moderate amount. This result suggests an apparent contrast with other studies of older drivers and their acceptance of technology. For example, Kantowitz, Hanowski, and Kantowitz (1997) and Kantowitz, et al. (1996) show that older drivers are less likely to increase their use of a traffic information device, compared to younger drivers. Older drivers' use of the device remains constant, while younger drivers increase their use. Figure 10 shows a possible underlying similarity in the behavior. With the onset of less redundant ATIS, the trust of older drivers remains relatively constant, while younger drivers' trust changes. The underlying cause may not be an inherent reluctance to use technology, but a greater inertia. The greater inertia of older drivers may be reflected in older drivers' continued higher level of trust in this experiment. This parallels the reluctance of older drivers to use the traffic information device in Kantowitz, et al. (1997), where this inertia was reflected in a reluctance to adopt the technology. These results suggest that older drivers' trust will have greater inertia and will be more constant than that of younger drivers; their trust will decline less and increase less than that of younger drivers.
Trust also depends on an interaction between message style and mode, F(1, 16) = 15.14, p<0.005. This interaction shows that the combination of text notification messages engenders the greatest level of trust (80.8) compared to the rated trust for the text command (74.7), auditory notification (75.0), or auditory command (76.8). This interaction is complicated by a four–way interaction between message style, mode, age, and gender, F(1, 16) = 13.3, p<0.01. Figure 11 shows this interaction. This interaction shows that older females tend to distrust ATIS information when it is presented as an auditory notification. This pattern is similar for younger females, who trust auditory command messages more than auditory notification messages. While men trust the auditory command message more than the text command message, they tend to distrust command messages in general.
The independent variables influence trust in several complicated ways. A three–way interaction between age, gender, and location is significant, F(1, 16) = 4.89, p<0.05. A related four–way interaction between age, gender, location, and information availability is also significant, F(2.56, 40.92) = p<0.005. Figure 12 shows that information availability differentially affects trust depending on age, gender, and the location of the display. Table 12 explains these differences.
Table 12. Comments explaining the effects shown in figure 12.
The strong interactions between the driver characteristics of age and gender with message characteristics suggest that drivers' attitudes do not depend only on the characteristics of the ATIS. These interactions suggest that individual differences play a complex, but important role in shaping drivers' attitudes. Table 13 summarizes all the effects for trust in the ATIS and self–confidence.
Table 13. Summary of all significant effects for trust and self–confidence.
ANALYSIS OF SITUATIONAL AWARENESS, WORKLOAD, AND MESSAGE ACKNOWLEDGMENT
Situational awareness, workload, and message acknowledgment are important intervening variables that may illuminate the influence of ATIS messages on driving safety. Six repeated measures analysis of variance examined the dependence of six measures on ATIS and driver characteristics. The analyses of situational awareness, workload, and message acknowledgment included age, gender, information availability, message style, message mode, and message location as independent variables. The analysis of message acknowledgment examined only those cases where drivers received both ATIS and roadway information and so information availability was replaced by acknowledgment type (acknowledgment of ATIS messages versus roadway information). The significant results will be considered in terms of the specific hypotheses and then in terms of how these intervening variables may illuminate the factors affecting driving safety. The specific hypotheses include:
The first hypothesis suggests that situational awareness will decline as drivers' attention is drawn into the vehicle when they receive ATIS information without redundant road–sign information. Figure 13 shows the main effect of information availability that supports this hypothesis, F(2.59, 41.50) = 27.41, p<0.0001. In addition, this figure includes a two–way interaction between age and information availability that suggests that younger drivers are more prone to having their attention drawn into the vehicle when only ATIS information is available, F(2.59, 41.5) = 4.64, p<0.01. Figure 13 shows that the difference in situational awareness between older and younger drivers is smallest when older drivers have access to ATIS information. Situational awareness for older drivers is approximately the same as that for younger drivers when the ATIS information is present. When ATIS information is not available, younger drivers have greater situational awareness. These results do not support the second hypothesis that the ATIS will overwhelm drivers with information because the level of situational awareness is higher when the potential for information overload is greatest (when roadway and ATIS information are both present). These results show that ATIS messages might undermine driving safety by drawing drivers' attention into the vehicle, rather than by overloading them with information. These results are particularly striking because several situational questions could be answered based on the ATIS information. It seems that older drivers are able to use the ATIS more effectively than younger drivers and that information overload is not the cause of the decline in safety associated with ATIS messages.
The third hypothesis states that presenting drivers with text messages will result in higher mental effort because driving and reading ATIS messages draw upon the same visual resources. The results do not support this hypothesis. No main effects involve message modality. The fourth hypothesis states that redundant sign information may lead to a higher workload as drivers process both the roadway and ATIS information. Again, the results fail to support this hypothesis. Message mode and different levels of information availability show no effect on drivers' rated mental effort.
The fifth hypothesis suggests that auditory messages will capture driver attention, resulting in shorter latencies for ATIS message acknowledgment. The analysis showed that the latencies for the auditory ATIS messages do not differ from the latencies for the text messages or the roadway messages. The sixth hypothesis states that centralized displays will support more efficient information assimilation, resulting in shorter latencies. The analysis does not support this hypothesis. Whether information was presented in a centralized or distributed location does not have a significant effect on acknowledgment latencies. In fact, no ATIS design characteristics seem to affect response time to the ATIS messages as compared to the roadway messages.
The specific hypotheses help clarify the cognitive processes that interact with certain ATIS message characteristics to affect driving safety. Examining the specific hypotheses together suggests that the concept of information overload does not explain how the ATIS might undermine driving safety. Instead, it seems that overreliance on the ATIS compromises safety. The sensitivity of subjective ratings of mental effort and physical effort to the effect of age supports this argument. As expected, older drivers show a much higher level of mental effort. Younger drivers showed a mean subjective effort of 35.2 compared to a mean of 64.5 for older drivers, F(1, 16) = 20.21, p<0.0005. Like mental effort, physical effort also shows a strong effect due to age, F(1, 16) = 41.68, p<0.0001. Interestingly, the magnitude of the difference between older and younger drivers is approximately 30 percent greater for physical effort, compared to mental effort. If information overload compromises safety, the sensitivity of these measures should enable them to support the appropriate hypotheses. These findings suggest that an ATIS can undermine safety by drawing drivers' attention into the vehicle, causing them to give less consideration to the roadway. This interpretation is consistent with the effect of ATIS information on situational awareness. Situational awareness is lowest when ATIS information is presented without redundant roadway information.
A complex interaction affecting mental effort reveals how ATIS message characteristics influence drivers' safety and compliance. The main effect of age interacts with message style so that younger drivers experience lower workload with notification messages and older drivers experience lower workload with command messages, F(1, 16) = 9.29, p<0.01. This effect is further complicated by a three–way interaction involving age, information availability, and message style, F(2.60, 41.66) = 4.87, p<0.01, and a four–way interaction involving age, gender, message style, and information availability, F(2.60, 41.66) = 5.10, p<0.01. Figure 14 shows how mental effort is moderated by age, gender, and message style across the different levels of information availability. In general, older drivers experience more effort compared to younger drivers. Furthermore, younger females and older females experience a different level of effort in response to command and notification messages. Younger females experience a higher level of effort assimilating ATIS information presented as commands compared to notifications. In contrast, older females experience a lower level of effort with command messages compared to notification messages. As one might expect, this effect is most pronounced when the ATIS information is present. This effect is particularly difficult to interpret as an effect of information overload; however, a sociological perspective may clarify the issue. Accepting instructions may be consistent with the experience of older women, but collecting information and making autonomous decisions is more consistent with the expectations of younger women. Notification messages support this more autonomous decision–making style, while command messages are more compatible with the role of accepting pre–defined instructions.
For each situation awareness query, drivers rated their level of confidence in the accuracy of their response. Confidence might prove to be a more sensitive measure of situation awareness than the accuracy of their response because the graded scale provides more information than the binary coding associated with correct and incorrect responses. Surprisingly, drivers' confidence in their accuracy did not parallel their accuracy. SA confidence correlates with SA accuracy only slightly (r=0.16). Unlike SA accuracy, drivers' confidence in their accuracy was not affected by information availability. Confidence in the SA accuracy seems to be sensitive to factors other than those that affect SA accuracy. The divergence between the accuracy and drivers' confidence provides a measure of meta–SA. A poor correlation between accuracy and confidence indicates a situation where the driver does not recognize how much he or she does not know. This correlation was lower for older drivers (r=0.11) compared to younger drivers (r=0.21). Figure 15 shows the relationship between SA accuracy and perceived SA accuracy. Not only are older drivers' perceptions poorly correlated with their accuracy, they consistently overestimate their accuracy. Figure 15 shows that even though older drivers' SA is lower than that of younger drivers, their confidence is greater.
Figure 16 shows a three–way interaction between gender, age, and style, F(1, 16) = 6.24, p<0.05, and a two–way interaction between age and style, F(1, 16) = 7.30, p<0.05. These interactions show that older females are particularly confident in their SA query responses when they receive command–style messages, while younger females are more confident when they receive notification–style messages. Message style does not seem to influence the confidence of male drivers. This effect is similar to that for mental demand and trust. Younger females react more positively to notification–style messages compared to older females, who react more positively to command–style messages.
The message acknowledgment latency shows no significant effects, but the percentage of correct responses reflect driver age, F(1, 16) = 13.18, p<0.005. Younger drivers respond more often than older drivers (95 percent compared to 81 percent). In addition, drivers acknowledge text messages (92 percent) more than auditory messages (85 percent), F(1, 16) = 6.38, p<0.05.
Information theory provides a useful framework to further analyze the attention devoted to road signs and ATIS messages. Signal–detection theory describes the ability to detect signals in terms of sensitivity (d') and the response criterion (). d' and should reflect differences in the relative salience of the ATIS compared to roadway messages. This analysis shows whether or not drivers are more sensitive to in–vehicle messages.
Tables 14 and 15 show the drivers' responses to ATIS messages and roadway signs. A', a non–parametric measure of d', has been calculated because it is less dependent on assumptions compared to d' (Wickens, 1984). Drivers appear slightly more sensitive to ATIS messages, but they respond in a more conservative manner compared to their responses to road signs. This probably reflects the greater diversity of cues from the roadway compared to the in–vehicle information sources, which may have prompted drivers to acknowledge in–vehicle messages as if they were roadway messages. As the similarity of the tables suggests, drivers detect and acknowledge ATIS and roadway messages in very similar ways. These similarities, together with the traditional analysis of variance, suggest that the acknowledgment of messages does not differ for ATIS or road signs.
Table 14. Responses to ATIS messages (A' = 0.94, = 0.97).
Table 15. Responses to road signs (A' = 0.91, = 1.10).
Tables 16 and 17 summarize the statistical analyses for situation awareness and workload, and message acknowledgment, respectively.
Table 16. Summary of significant effects for situation awareness and effort.
Table 17. Summary of significant effects for message acknowledgment.