• Federal Highway Administration >
• Publications >
• Research Publications >
• 95176 >
• Section 1 >
• Development of Human Factors Guidelines for Advanced Traveler Information Systems and Commercial Vehicle Operations: Task Analysis of ATIS/CVO Functions

Development of Human Factors Guidelines for Advanced Traveler Information Systems and Commercial Vehicle Operations: Task Analysis of ATIS/CVO Functions

CHAPTER 1. TASK ANALYSIS OF ATIS/CVO SYSTEMS

INTRODUCTION

DATA/INFORMATION COLLECTION

LITERATURE REVIEW

SITE VISITS

FOCUS GROUPS

PROSPECTIVE VERBAL PROTOCOL ANALYSIS

INTRODUCTION

Prior tasks in the Advanced Traveler Information Systems/Commercial Vehicle Operations (ATIS/CVO) project have concentrated on obtaining a description of what ATIS is likely to do; what role it is likely to play in terms of economics, safety, quality of life, and environmental concerns; and what functions it is likely to perform for private and commercial vehicle operations. These previous activities provided a description of ATIS and what people are likely to use it for without examining the way it will actually be accomplished. In addition, prior tasks concentrated on the ATIS without paying much attention to the driving context within which the systems will have to be used. Task E is intended to provide information related to how drivers and other users are going to interact with ATIS/CVO systems in the driving environment.

Goals of the Task Analysis of ATIS/CVO Systems

Task E has two primary goals:

• Develop an understanding of what users of ATIS/CVO systems are going to be required to do to use the system safely and effectively.

• Develop an understanding of the relationship between what ATIS/CVO system users are going to be required to do to use the system and what they are likely to be able to do.

Usefulness of a Task Analysis

A task analysis can be used to perform a number of useful functions depending on the needs of the user, the development stage of the system, and the type of system being described. The major goal of the ATIS project is to develop human factors design guidelines for ATIS/CVO systems. In this context, a task analysis provides an evaluation of the relationship between the way the user will need to interact with the system and the physiological and cognitive characteristics the user is likely to bring to the task.

Appendix A summarizes several papers that document important issues that a task analysis should address. In addition, this appendix reviews several papers that describe potential task analysis methods and thus provides the basis for selecting the methods used in this report.

The task analysis has several useful outcomes for both later tasks in the project and the direct development of human factors design guidelines. These include:

• Identification of "basic" human tasks that will be required of ATIS regardless of the specific design adopted for a particular system.

• Identification of areas where an understanding of the impact of human limitations on a particular type of task is incomplete and will require further research.

• Development of a sequential description of the actions users of ATIS/CVO systems will need to perform to achieve functional goals.

• Identification of potential conflicts between ATIS/CVO tasks and driving tasks.

• Early identification of task demands that may exceed user characteristics for some portions of the population.

• Identification of general areas that will need to be addressed by the human factors design guidelines.

Constraints in the Task Analysis

The task analysis of ATIS/CVO systems was constrained by several conditions. The first condition is that the systems are not well developed. Although there are some reasonable prototypes of ATIS/CVO systems that perform some of the ATIS functions, common approaches to the design of such systems are a result of technological capability rather than any form of standardization or general agreement on how the systems should be designed. In the next 5 to 10 years there is reason to believe that technological limitations that presently constrain design considerations will be lifted and that there will be a multitude of possible approaches to deal with controls and display issues––the primary human factors issues of ATIS. The lack of a mature or well–developed technology provides the task analyst with two possible alternatives. The analyst can use a system–specific approach limited to the existing technology, thus enabling a very detailed look at the tasks involved, but with the attendant risk of providing information that will be outdated when operational systems are released.

The analyst can also develop the task analysis using a function–related approach that concentrates on the tasks that will need to be performed, regardless of design, to achieve the goals of the function. This type of approach is obviously less sensitive to specific design issues, but does have the advantage of being more applicable to developing technologies.

A second condition in a task analysis of ATIS/CVO systems is that these systems are being developed for use by a wide range of drivers. As a consequence, it is difficult to determine, with any certainty, what effect user characteristics will have on task performance. One approach to solving this problem would be to examine the relationship between task demands and user characteristics for several different user populations (e.g., commercial drivers, younger drivers, and older drivers). Such an approach would have significantly increased the complexity of the analysis without adding a great deal to its usefulness, particularly given the limited understanding of specific ATIS/CVO designs.

A third (and perhaps the most important) condition is the lack of information on the effect of secondary tasks, such as those represented by ATIS, on the primary task of driving. While secondary tasks are not new to driving (e.g., radios have been in cars since the 1920s), ATIS is probably the first major system that will represent a secondary task so closely integrated with the driving task. This integration is particularly noticeable with an ATIS that provides instructions to the driver, where the driver must not only comply with the instructions given by the system, but must maintain primary control of the vehicle as well.

Top

DATA/INFORMATION COLLECTION

Task E was designed to provide a systematic, top-down analysis of the tasks performed by users of ATIS/CVO systems in order to meet the required functions for each system as identified in Task C (Battelle Research Center, 1992). The analysis was to be based on a combination of information gathered in earlier tasks and information specifically obtained in connection with the task analysis. Table 1 shows the contributions made by the earlier tasks.

These previous tasks provided a starting point or foundation from which it was possible to identify additional activities that would be necessary to perform the task analysis. These additional activities included:

• Conducting a short literature review aimed specifically at significant issues associated with the task analysis.

• Conducting focus groups to identify how ATIS comparable tasks are presently done and how drivers might use ATIS/CVO systems when they become available.

• Conducting site visits to obtain experiential and observational information on the use of prototype ATIS/CVO systems in "real–world" situations.

• Conducting Prospective Verbal Protocol Analyses (PVPA) with representative drivers on ATIS functions that do not have readily available prototypes.

The following sections of this report provide details of how these data/information collection activities were conducted.

Table 1. Contributions of previous tasks to the task analysis.

Top

LITERATURE REVIEW

In order to perform a task analysis of ATIS/CVO systems that would reflect the goals of these systems and show accurately how they could be used, it was necessary to complete a literature review. The literature review had several functions. First, it served to identify existing techniques that have been developed to analyze systems that are still at the conceptualization phase. Second, the literature review helped to locate existing terminologies to describe driving behaviors and their associated tasks. In addition, the literature review also helped in identifying: (1) the cognitive demands imposed on drivers when using ATIS/CVO subsystems while driving, as well as (2) the human constraints that users bring with them when performing a task. Finally, the literature review examined drivers' behaviors when making use of similar techniques or when taking part in experiments that study some of the functional aspects/characteristics of these s subsystems.

The literature review had an initial goal to identify and summarize task analysis methods or other similar techniques that have been used in the past to describe the user's task sequence for systems that exist only at the conceptualization stage. When systems are being conceptualized, rather than already having been built, it is difficult for future users to adequately describe the potential tasks that they will have to perform. Because ATIS/CVO subsystems are still at that particular stage of conceptualization, it was necessary to perform such a review. In fact, although In–Vehicle Routing and Navigation Systems (IRANS) and In–Vehicle Motorist Services Information Systems (IMSIS) have the privilege of having numerous comparable systems to illustrate their capabilities, In–Vehicle Signing Information Systems (ISIS) and In–Vehicle Safety Advisory and Warning System (IVSAWS) have very few examples from which to choose.

In order to perform a task analysis of IRANS and IMSIS, one could examine how present users of comparable systems (such as TRAVTEK and NAVMATE) accomplish the tasks associated with an ATIS. To describe how these drivers would use IRANS and IMSIS, it is necessary to extrapolate from their actual use of comparable systems. Such an extrapolation of the tasks to be performed on IRANS and IMSIS can be obtained through the use of Prospective Verbal Protocol Analysis (PVPA), the use of focus groups of drivers, and by observation of users accomplishing tasks on similar systems. As a consequence, the first goal of the literature review was to search for alternatives to the traditional task analysis methods.

Some alternatives were identified and they include PVPA (Tolbert & Bittner, 1991); multi–dimensional scaling (Coury, Weiland, & Cuqlock–Knopp, 1992); thinking–aloud protocols (Denning, Hoiem, Simpson, & Sullivan, 1990); and cognitive task analysis (Drury et al., 1987; Redding, 1990).

The second goal for the literature review was to assist the task analysis breakdown in identifying terminologies associated with driving behaviors as well as ATIS–related tasks. In this regard, it was worth noting that most task analyses of driving behaviors are done at a level much finer than the one intended in this task. However, some of these task descriptions were considered useful. For example, the classic task analysis description by Miller (1953) served as a basis by providing the terminology that is relevant to any task domain. Some of the cognitive processes were derived from Miller's (1974) decision–making elements. Finally, most of the driving terminology was obtained from Moe, Kelly, and Farlow (1973) and MacAdam (1992).

In addition to these terminology listings, a description of drivers' tasks was obtained by reviewing their behavior with similar technologies. Studies such as simulation of driver route diversion and alternate route selection (Allen et al., 1991); pilot studies of IVSAWS driver–alert warning system design (Erlichman, 1992); and surveys of driver attitude concerning aspects of highway navigation (King, 1986), as well as the influence of car navigation map displays on drivers' performance, have contributed to a better understanding of drivers' future task demands and have helped to provide the terminology necessary to describe these future tasks.

The literature review also had a goal to identify cognitive demands and human limitations in using these ATIS/CVO systems. Some papers helped this identification by breaking down the driving task into various components and determining the drivers' information needs (Allen, Lunenfeld, & Alexander, 1971; Senders et al., 1967). Others focused on human factors considerations that dealt with driving and navigation tasks as well as with users and display characteristics (Petchenik, 1989; Wierville, Hulse, Fisher, & Dingus, 1988). Finally, others provided this information as well as cognitive/attentional demand requirements by looking at advanced systems in general (Roth, Bennett, & Woods, 1988; Smiley, 1989).

Appendix A provides a detailed summary of each citation included in the literature review. Each summary identifies the topic, type of article, and subject population used in empirical studies. The summaries also include the abstract, a description of the methodology used in the study, and a brief review that documents the utility of the article and the critical findings. The details of these summaries helped to identify appropriate task analysis methods and provided the descriptions of driving tasks that are included in appendix C and appendix D. In summary, such a review was necessary in order to produce a task analysis that would accurately reflect the nature of the future systems and would achieve the goal of describing users' tasks.

Top

SITE VISITS

Three site visits were conducted in connection with the task analysis. The site visits allowed the analysts to participate in or observe the use of prototype ATIS/CVO systems. Thus, these visits provided the analysts with an opportunity to observe the performance of ATIS/CVO tasks within the context of driving or dispatch.

Experiential Observations of IRANS and IMSIS

The Avis Rent–a–Car agency currently has five automobiles equipped with Zexel's NAVMATE system that are available for rent from their San Jose (CA) International Airport office. The NAVMATE system is an autonomous system that provides some of the primary functions of the IRANS and IMSIS subsystems described in the Task C functional description working paper. The available NAVMATE system incorporates the IRANS functions of trip planning, pre–drive route and destination selection, route guidance, route navigation, and the IMSIS function of the services/attractions directory. The IRANS functions are based upon a geographic data base that covers the greater San Francisco Bay area. Vehicle location is determined on the basis of both the global positioning system (GPS) and inertial guidance systems. The IMSIS directory provides a relatively complete listing of commercial establishments, government offices, schools and universities, and recreation areas. There is integration between the IMSIS and IRANS, providing the capability of selecting a destination for IRANS route planning using the IMSIS directory. A more thorough description of the NAVMATE system is in the Task D comparable systems analysis working paper.

Two members of the Task E project team traveled to San Jose and rented a NAVMATE–equipped vehicle from Avis for use during a 3–day period to familiarize themselves with the capabilities and operation of such a system. In preparation for this site visit, selected IRANS and IMSIS scenarios developed during Task B (system objectives and performance requirements) were formatted for use in guiding operational exercises conducted with the NAVMATE system. Upon arriving at the San Jose airport and renting the NAVMATE–equipped automobile, project staff installed a video camera in the back seat of the vehicle and wired the driver with a microphone to obtain a record of the operational exercises. The video camera's field of view included most of the windshield scene and the NAVMATE displays and controls. While driving the car, project staff verbalized their intentions, interpretations, and reactions to the NAVMATE system. The staff member who was the passenger controlled the video camera and maintained a written timeline record of events during each operational exercise.

Operational exercises, based upon a subset of five private vehicle ATIS scenarios, provided the framework within which project staff gained experience operating the IRANS and IMSIS. At the beginning of each exercise, scenarios were selected and reviewed by the driver and passenger, then the driver operated the NAVMATE system and vehicle with minimal assistance from the passenger. Scenarios that were used to guide these exercises included the following from the Task B working paper:

P1: Driver goes directly to the hotel located in the city X miles from the airport.

P2: Driver goes to multiple destinations (street addresses) all located within the city. (Modified by selecting multiple destinations and storing them in temporary system memory for retrieval during sequential portions of the trip.)

P3: Driver goes directly to a destination located in the city.

P4: Driver wants to go to a nearby restaurant (point of interest). Driver obtains two alternatives using IMSIS, compares travel times for the two alternatives using IRANS, and drives to one of the restaurants. (Modified by comparing distance only.)

P5: Driver has an appointment in a large suburban area. However, before the appointment, the driver wants to go to a restaurant (point of interest) and to a service station. The driver uses IMSIS to select a restaurant near the present location, enters the restaurant and the next client's location on the IRANS, and requests the location of a service station on this route. (Modified by selecting multiple destinations and storing them in temporary system memory for retrieval during sequential portions of the trip.)

Exercises were conducted in both daylight and nighttime conditions for a total period of approximately 24 h. The exercises were conducted over much of the greater San Francisco Bay area, providing opportunities to travel in large and small cities, as well as suburban and rural settings. Following the site visit, videotapes were reviewed and edited, resulting in a 6–h set of edited videotapes that provided representative examples of different scenarios and activities illustrating particular issues in system operation. These videotapes were reviewed by appropriate project staff to help familiarize them with IRANS and IMSIS operation. Following the editing of the videotapes, the audio portion of the edited tapes was transcribed. No additional formal analyses or records of this site visit were made, although the site visit provided general experience that was drawn upon during later stages of the task analysis.

Observations of CVO, AVL, and Satellite Communications Systems

Two human factors specialists spent 1 day observing and interviewing drivers of combination vehicles from Tri–State Motor Freight as they moved hazardous materials from a shipping port to a port where the cargo would be loaded on ships for shipment overseas. Each vehicle was equipped with Qualcomm systems for automatic vehicle location (AVL) and communications with the dispatcher. In addition to the observations of the use of the AVL and communications systems, the observations allowed the specialists to view CVO interactions with State regulators and intermodal networks.

The observations started at the company terminal where the trucks and trailers were inspected and made ready for the trip. Prior to departing, the drivers initiated automatic status messages via the Qualcomm system to indicate to central dispatchers (located in another State) that they were in service and beginning the trip. The trip to the port was made during the morning rush hour; thus, the trip included delays due to traffic, and the passing of information from one vehicle to another concerning traffic and road conditions ahead. The inter–truck communications were made using on–board citizens band (CB) radios.

Once at the port, the drivers needed to find out which gate they should enter. This was not clear from signs in the vicinity nor from the briefings they had been given by their dispatchers. Eventually, they were able to learn where they needed to be from other drivers. At the port, they were inspected and weighed. Their shipping papers were also checked by both the port authorities and customs officials. All of this was in order; however, the drivers expressed frequent concern that individual inspectors or others would require something that they did not have or that something would not exactly be as the inspector wanted it.

While at the port, the drivers received different instructions concerning how they were to handle the tarps covering the cargo (i.e., leave them with the cargo or take them back to the terminal). The Qualcomm system was used to obtain instructions from the dispatcher concerning this issue. In this case, the message was exchanged using free text and was entered using the keyboard. Both the message and the answer were received by the system and stored until the driver could view them. After the delivery was made, the drivers also received instructions concerning their next assignment over the Qualcomm system, although most also called the dispatcher from a truck stop to negotiate or verify these instructions.

Observations of Computer–Aided Dispatch, AVL, and IRANS for Emergency Vehicles

Two human factors specialists spent half a day observing and interviewing dispatchers and drivers of department vehicles in the Seattle Fire Department. The Seattle Fire Department has computer–aided dispatch (CAD), AVL, and text communication links to their vehicles. Vehicles are equipped with Travelpilot navigation systems.

During this visit, several managers were interviewed concerning the use of the system. Because the system affected dispatchers and drivers, the human factors specialists spoke with drivers and dispatchers and observed them as they operated the system. Most of these observations were made in the dispatch center, where four dispatchers were observed as they handled incoming calls and coordinated the activities of response vehicles.

Dispatchers use the CAD features of the system to identify the location and status of vehicles within a limited radius of the fire or aid request. If no vehicles are available to respond within that radius, the system initiates a search for appropriate equipment from among the nearest stations to the scene. In this function, the system keeps a comprehensive record of the location, status, and availability of all of the fire and aid equipment in the Seattle Fire Department. When a call is received and its location is entered into the system by a dispatcher, the system identifies equipment closest to the scene for the dispatcher, who then initiates the alarm notification at the firehouse or in the vehicle. Simultaneously, the dispatcher assigns the equipment to the incident within the CAD system, and text notification and a scene location icon are sent to the vehicle IRANS display. As the response develops, the system is updated both by the responses from the AVL and the status messages generated from the vehicle, and by the entries the dispatcher made as a result of either phone or radio messages. This information can be displayed on the dispatcher's screen along with a map that includes the vehicle location and incident scene. The map scale can be changed to provide an overall view of the city or a detailed view of the area around the scene. Both the map display and the information used by the system are limited to map orientation only and provide neither routing information nor traffic information to the dispatcher.

The contemporaneous record–keeping function performed by the CAD system provides a means for positioning the city's fire department assets when responding to a major fire or aid incident involving several different fire companies. This ensures that backup assets are available at the scene and that the city is covered as much as possible with the remaining assets. The CAD system also provides a consolidated record of the fire department response to a particular incident, including the locations of the equipment, their travel times, and their status throughout the incident.

The equipment used on the vehicles provides a small map display that includes the incident scene and the vehicle location. The scale can be adjusted, but does not include traffic or routing information. In addition to the map display, text information can be presented on the screen and "quick keys" are provided for the user to enter changes in status (e.g., arrival at the scene, free for assignment, in the station house, or out of service). Operation of the system is the responsibility of the co–driver, and, in most of the equipment, the screen is placed where it cannot be observed by the driver.

Top

FOCUS GROUPS

Three focus groups were conducted in connection with the function and task analyses. Table 2 gives the composition and primary focus of each of the groups. Each of the focus groups consisted of between 15 and 25 participants. The focus groups in Seattle and Denver involved a full day of activities. The focus group in Bar Harbor, ME, was limited to half a day. Each of the focus group sessions was preceded by a description of ATIS/CVO to acquaint participants with the general characteristics and functions that each system would provide. To minimize the influence that specific design approaches might have on the way in which participants visualized the system, descriptions were based primarily on IVHS America's planning documents and published concepts. (IVHS stands for Intelligent Vehicle-Highway System.) The presentations were followed by sessions that concentrated on specific systems (i.e., IRANS, IMSIS, ISIS, IVSAWS, and CVO).

Table 2. Focus groups to gather information on ATIS functions and tasks.

Top

PROSPECTIVE VERBAL PROTOCOL ANALYSIS

Background

A cognitive task analysis was performed using a prospective verbal protocol analysis (PVPA) approach that was delineated by Tolbert and Bittner (1991). Essentially, this PVPA was an extension of the classical Verbal Protocol Analysis (VPA) (Ericsson & Simon, 1984) that required drivers to "role play" through task steps of a selected scenario in the PVPA and to verbalize the strategy they would use to perform a task with a conceptually described system (described only in outline form). The resulting verbalizations can subsequently help identify human skills required to perform steps effectively and verify the essential correctness of task analysis results developed using expert judgment (e.g., Wheeler & Toquam, 1991).

Described in the following sections are elements of the method used for the PVPA. These include subjects, scenarios, and procedures.

Subjects

Three subjects participated in the PVPA (N=3). Two females (ages 31 and 43 years) and one male (age 32 years) participated. One subject participated as part of her job, while the other two volunteered their time. A description of the general nature of the study was presented, after which an informed consent was obtained from each subject.

Scenarios

The five scenarios used in the PVPA were extensions of scenarios developed as part of Task B (P1, P8, and P12) or were created as part of the task analysis requirements (P16 and P22). These scenarios were modified slightly to capture all of the IVSAWS and ISIS subsystem functions. Descriptions of these scenarios are given in tables 3 through 7. The IVSAWS and ISIS functions were the focus of the PVPA because they were the two ATIS subsystems for which the data collection gathered the least amount of information. In fact, most of the existing systems (e.g., TravTek, NAVMATE) have capabilities that reflect some of the functional characteristics associated with IRANS and IMSIS, but have no or very limited ISIS or IVSAWS capabilities. The purpose, summary, system, and functional characteristics for the data scenarios are delineated below.

Table 3. Scenario P1 as used in the Prospective Verbal Protocol Analysis.

Table 4. Scenario P8 as used in the Prospective Verbal Protocol Analysis.

Table 5. Scenario P12 as used in the Prospective Verbal Protocol Analysis.

Table 6. Scenario P16 as used in the Prospective Verbal Protocol Analysis.

Table 7. Scenario P22 as used in the Prospective Verbal Protocol Analysis.

As can be seen above, ISIS and IVSAWS functions were added or emphasized in the modifications. Table 8 summarizes the breakdown of the functional characteristics across the four data scenarios, showing that all facets were covered for ISIS and IVSAWS.

Procedure

The PVPA data collection sessions were conducted in two phases. During the "preliminary" phase, each subject was initially introduced to the nature of the project and to the PVPA technique. Subjects were then asked to fill out the informed consent form and the demographic survey. In order to become more familiar with ATIS concepts, the experimenter then had the subjects read about the following: (1) ATIS in the context of the overall project, (2) ATIS systems/subsystems, and (3) potential ATIS features/functions. The experimenteranswered any questions the subjects may have had. Next, TravTek and NAVMATE/Zexel video examples (5 min illustrating actual use) were shown to subjects to provide a broad operational context for ATIS. The experimenter stated that the systems were being used for example purposes and that subjects might have other ideas on how the systems should operate. Subjects were told to express their ideas/conceptions of how the systems should operate.

Following the introductory material, a practice scenario (P1) was used to familiarize subjects with the PVPA procedure. During this step (phase), subjects were given the general instruction "to imagine themselves in the context of the scenario and to relate everything they could think of to the experimenter." The practice scenario incorporated IRANS and IMSIS functions and paralleled the NAVMATE video example. A portion of the practice scenario was read, then the experimenter asked the following questions:

• What would you do/be doing at this point?
• What would you expect the system to do?

Table 8. Functional characteristics used for the Prospective Verbal Protocol Analysis.

The subjects' responses were recorded via microcassette (and in writing). Each portion of the scenario was read through, with the same set of questions asked after each portion. The experimenter did not try to ask additional questions at that time. After reading through the whole scenario one time and collecting the subjects' responses, the experimenter went back to the first portion of the scenario and asked the subjects to expand on their responses. This typically involved the experimenter asking for additional information as to how the subjects would "request information" and how the system would "provide information." This basic procedure was used for the remaining responses for each portion of the scenario. When the practice scenario protocol was finished, the subjects had an opportunity to ask questions of the experimenter. Each of the remaining scenarios (P8, P12, P16, and P22) were then read in separate portions with the same set of initial questions being asked for each portion of each scenario.

Limitations of Using Prospective Verbal Protocol Analysis

Suitability of the technique. The technique used appeared suitable for a general idea of (1) what drivers would expect the system to do, and (2) what they might have to do to interact with it. However, subjects reported having some difficulties imagining how the systems would look, feel, and work, even with the introductory materials and videotapes. In this regard, it is pertinent to note, most subjects had strong initial opinions about how information should be presented (auditorially or visually). However, they were less sure of when the information should be presented or how they would specifically interact with the system (at the button–pressing level). Additionally, subjects mentioned very little about regular driving tasks that they would be doing while using ATIS. When asked about the lack of this information in their responses, their comments were, "I just assumed that I would be doing normal driving tasks," and "It's difficult to imagine what I would be doing while driving without actually doing it." Another subject alluded to the possibility that driving tasks are so automated that they aren't thought about and are not easily verbalized (expert knowledge difficult to verbalize). This latter comment (and to some extent earlier comments), it is noteworthy, is consistent with historical critiques of PVPA (cf., Ericsson & Simon, 1984). However, they are also inconsistent with earlier research using PVPA for assessing strategies used in performing a rapid, complex motor task (Triggs et al., 1990) and early success using PVPA (e.g. Zachary, Zaklad, & Davis, 1987; Zacklad, Deimler, Iavecchia, & Stokes, 1982). Significantly, subjects in the Triggs et al. study had recent extensive experience doing their task, and those in the earlier Zachary et al. and Zacklad et al. studies had hundreds of hours working with systems and scenarios similar to those being evaluated. This suggests that PVPA may be most appropriate when subjects have more intimacy with the systems evaluated (e.g., ATIS) than could be achieved in the present study (albeit using video and other materials).

Problems concerning PVPA execution. Some concerns about the execution of the PVPA arose during the administration process. First, the materials used to familiarize the subjects with ATIS, both written and videotaped, may have biased subjects' responses toward what they learned from the materials (even though they were told these were examples and they might have other ideas regarding how systems could work). Second, the initial questions used to elicit subjects' responses were very general, which may have resulted in largely more general responses. More specific questions, however, might also have further biased subjects' responses. Third, greater bias was introduced by asking second–level protocol questions specifically related to each subject's first–level responses. By doing this, though, the subjects' responses stayed their own. This too suggests that PVPA may have been more appropriate when subjects are more familiar with the systems being evaluated (e.g., ATIS).

Top

FHWA-RD-95-176