A cognitive task analysis was performed using a prospective verbal protocol analysis (PVPA) approach that was delineated by Tolbert and Bittner (1991). Essentially, this PVPA was an extension of the classical Verbal Protocol Analysis (VPA) (Ericsson & Simon, 1984) that required drivers to "role play" through task steps of a selected scenario in the PVPA and to verbalize the strategy they would use to perform a task with a conceptually described system (described only in outline form). The resulting verbalizations can subsequently help identify human skills required to perform steps effectively and verify the essential correctness of task analysis results developed using expert judgment (e.g., Wheeler & Toquam, 1991).

Described in the following sections are elements of the method used for the PVPA. These include subjects, scenarios, and procedures.

Three subjects participated in the PVPA (N=3). Two females (ages 31 and 43 years) and one male (age 32 years) participated. One subject participated as part of her job, while the other two volunteered their time. A description of the general nature of the study was presented, after which an informed consent was obtained from each subject.

The five scenarios used in the PVPA were extensions of scenarios developed as part of Task B (P1, P8, and P12) or were created as part of the task analysis requirements (P16 and P22). These scenarios were modified slightly to capture all of the IVSAWS and ISIS subsystem functions. Descriptions of these scenarios are given in tables 3 through 7. The IVSAWS and ISIS functions were the focus of the PVPA because they were the two ATIS subsystems for which the data collection gathered the least amount of information. In fact, most of the existing systems (e.g., TravTek, NAVMATE) have capabilities that reflect some of the functional characteristics associated with IRANS and IMSIS, but have no or very limited ISIS or IVSAWS capabilities. The purpose, summary, system, and functional characteristics for the data scenarios are delineated below.

Table 3. Scenario P1 as used in the Prospective Verbal Protocol Analysis.

As can be seen above, ISIS and IVSAWS functions were added or emphasized in the modifications. Table 8 summarizes the breakdown of the functional characteristics across the four data scenarios, showing that all facets were covered for ISIS and IVSAWS.

The PVPA data collection sessions were conducted in two phases. During the "preliminary" phase, each subject was initially introduced to the nature of the project and to the PVPA technique. Subjects were then asked to fill out the informed consent form and the demographic survey. In order to become more familiar with ATIS concepts, the experimenter then had the subjects read about the following: (1) ATIS in the context of the overall project, (2) ATIS systems/subsystems, and (3) potential ATIS features/functions. The experimenteranswered any questions the subjects may have had. Next, TravTek and NAVMATE/Zexel video examples (5 min illustrating actual use) were shown to subjects to provide a broad operational context for ATIS. The experimenter stated that the systems were being used for example purposes and that subjects might have other ideas on how the systems should operate. Subjects were told to express their ideas/conceptions of how the systems should operate.

Following the introductory material, a practice scenario (P1) was used to familiarize subjects with the PVPA procedure. During this step (phase), subjects were given the general instruction "to imagine themselves in the context of the scenario and to relate everything they could think of to the experimenter." The practice scenario incorporated IRANS and IMSIS functions and paralleled the NAVMATE video example. A portion of the practice scenario was read, then the experimenter asked the following questions:

Table 8. Functional characteristics used for the Prospective Verbal Protocol Analysis.

The subjects' responses were recorded via microcassette (and in writing). Each portion of the scenario was read through, with the same set of questions asked after each portion. The experimenter did not try to ask additional questions at that time. After reading through the whole scenario one time and collecting the subjects' responses, the experimenter went back to the first portion of the scenario and asked the subjects to expand on their responses. This typically involved the experimenter asking for additional information as to how the subjects would "request information" and how the system would "provide information." This basic procedure was used for the remaining responses for each portion of the scenario. When the practice scenario protocol was finished, the subjects had an opportunity to ask questions of the experimenter. Each of the remaining scenarios (P8, P12, P16, and P22) were then read in separate portions with the same set of initial questions being asked for each portion of each scenario.

Problems concerning PVPA execution. Some concerns about the execution of the PVPA arose during the administration process. First, the materials used to familiarize the subjects with ATIS, both written and videotaped, may have biased subjects' responses toward what they learned from the materials (even though they were told these were examples and they might have other ideas regarding how systems could work). Second, the initial questions used to elicit subjects' responses were very general, which may have resulted in largely more general responses. More specific questions, however, might also have further biased subjects' responses. Third, greater bias was introduced by asking second-level protocol questions specifically related to each subject's first-level responses. By doing this, though, the subjects' responses stayed their own. This too suggests that PVPA may have been more appropriate when subjects are more familiar with the systems being evaluated (e.g., ATIS).