Skip to contentUnited States Department of Transportation - Federal Highway Administration FHWA Home
Research Home
Report
This report is an archived publication and may contain dated technical, contact, and link information
Publication Number: FHWA-RD-03-065
Date: September 2004

In-Vehicle Display Icons and Other Information Elements: Volume I

PDF Version (8.33 MB)

PDF files can be viewed with the Acrobat® Reader®

CHAPTER 9: TUTORIALS

TUTORIAL 1: ANALYSIS OF RANK ORDER DATA

In this tutorial, we describe the types of information that can be obtained from rank-order data. We assume that the data is collected in the following manner. A total of nj judges are asked to rank order ns stimuli (e.g., icons) with respect to some attribute (e.g., appropriateness for a specific message). Usually the whole set of stimuli is presented together, and the judges are allowed to proceed as they wish as long as each judge comes through with a single rank order along the attribute specified. If many stimuli are presented, it might be useful for the judges to sort them roughly into grades before attempting the final ranking. A hypothetical data set is presented in table 9-1. These hypothetical data might be obtained from 10 individuals who rank ordered 5 icons (icons A, B, C, D, and E) in appropriateness for a specific message. The icons would be assigned ranks of 1-5, with "1" corresponding to the icon considered to be the most appropriate 1.

Judge Ranks Assigned
Icon A Icon B Icon C Icon D Icon E
1 4 2 1 3 5
2 4 2 1 3 5
3 2 1 3 4 5
4 3 1 2 4 5
5 3 4 1 2 5
6 3 2 1 4 5
7 4 3 1 2 5
8 4 2 3 1 5
9 4 2 1 3 5
10 4 1 2 3 5

Table 9-1. Sample Data Set: Ranks Assigned to 5 Icons by 10 Judges


Our goal is to investigate whether and how much the judged stimuli differ with respect to the attribute of interest. This goal can be accomplished at different levels using different methods of analysis. In the following sections, we discuss some of these methods. First, we indicate a quick way to determine a composite rank order for the stimuli. Second, we discuss the use of mean ranks. Third, we outline two methods of obtaining scores for the stimuli on an interval scale. In addition to a general rank ordering, these scores provide an indication of relative differences among the stimuli. Finally, we discuss the use of more advanced and complex statistical methods that allow us to test whether the stimuli are truly different with respect to the attribute of interest.

1.  DERIVING RANK ORDER FROM SUMS OF ASSIGNED RANKS

A composite ranking for the stimuli can be obtained by computing the sums of the rank values assigned to each of the stimulus. The order of the magnitudes of the sums will indicate an ordering of the stimuli, with the lowest sum corresponding to the stimulus judged as the best. For example, for the raw data of table 9-1, we obtain the sums presented in table 9-2, which indicate the following composite rank order from the most to least appropriate icon: C, B, D, A, and E.

Table 9-2. A Composite Ranking of Icons Obtained from Sums of Assigned Ranks

  Icon A Icon B Icon C Icon D Icon E
Sum of assigned ranks (S) 35 20 16 29 50
Composite rank 4 2 1 3 5

2.  OBTAINING MEAN RANKS

Mean ranks are obtained for each icon by dividing the sum of the ranks assigned to that icon by the number of judges. For example, for the raw data of table 9-1, the mean rank of icon A is 35/10 = 3.5. Mean ranks corresponding to all five icons in the sample data set are presented in table 9-3. In general, mean ranks will somewhat agree with the composite ranks in both order and spacing.

Table 9-3. Mean Rank of Each of the Icons

  Icon A Icon B Icon C Icon D Icon E
Sum of assigned ranks (S) 35 20 16 29 50
Mean Rank (M r) = S/nj 3.5 2 1.6 2.9 5

It should be noted that mean ranks should never be used to judge the absolute differences between the stimuli. For example, on the basis of the data presented in table 9-3, we could not conclude that icon E was judged to be 1.5 times more inappropriate than icon A. At best, mean ranks provide an indication of relative differences. For example, mean ranks in table 9-3 indicate that the difference between icons B and A is the same as the difference between icon A and E. However, the reader should be warned that rank values are strictly ordinal numbers2 and there is little numerical meaning to be attached to means of such values. To obtain more precise relative differences between the stimuli, we recommend the use of the scaling methods discussed in the next section.

3. OBTAINING INTERVAL-SCALE VALUES

As mentioned above, rank data (i.e., the rank orders produced by the judges) are on an ordinal scale. This scale arranges things in order of magnitude, but does not reveal the magnitude of the differences between them. An interval scale, on the other hand, allows statements about how much difference there is between two objects. A good example for an interval scale is the Fahrenheit scale of temperature. Here, it is meaningful to say that there is as much difference between 60 °F and 70 °F as there is between 70°F and 80°F. Note, however, that interval-scale values are not absolute magnitudes. For instance, it is not accurate to say that 80°F is twice as high as 40°F.

Using rank orders generated by a group of judges, it is possible to obtain interval-scale values for the ranked items. Below, we discuss two different methods of obtaining interval-scale values using rank order data: (1) Choice Score Method, and (2) Torgerson's Categorical Scaling Method. The reader should keep in mind that the values obtained using either of the scaling methods are not meaningful in absolute terms, but that they give only an indication of relative differences between the stimuli. As an example, let us assume that we have obtained scale values of 2.5, 5, and 7.5 for icons A, B, and C, respectively. Based on these values, we cannot conclude that icon A is three times as good (or appropriate) as icon C. However, we can say that there is as much difference between icons A and B as there is between B and C. Such information can be important to icon designers, who frequently must make tradeoffs among design issues such as cost, driver performance, and driver preferences. Converting the rank-order data to an interval scale allows for more systematic and rigorous tradeoff analyses to be conducted.

A. Choice Score Method

This scaling procedure, described in Engen (1971), converts rank orders to choice frequencies, then to p values, and finally to z scores (unit normal deviates). The z scores obtained represent scale values for the stimuli on a psychological scale with equal intervals on the assumption that the rankings are normally distributed. The procedure is outlined below; the computations for the sample data set are shown in table 9-4.

Table 9-4. Computations for the Choice Score Method

Step   Icon A Icon B Icon C Icon D Icon E Mean*
1 Mean Rank (Mr) 3.5 2 1.6 2.9 5 3
2 Mc = ns - Mr 1.5 3 3.4 2.1 0 2
3 p = Mc/(ns - 1) 0.38 0.75 0.85 0.53 0 0.5
4 z -0.31 0.67 1.04 0.08 - -
5 R = 5z + 6 4.45 9.35 11.2 6.4 - -

* The following can be used to check the calculations: (1) mean Mr must be equal to (ns+1)/2; (2) mean choice (Mc) must be equal to (ns-1)/2; (3) mean p must be equal to 0.5.

Step 1. Calculate the mean rank (Mr) assigned to each stimulus (see table 9-2).

Step 2. Calculate a mean choice score (Mc) for each stimulus by subtracting the mean rank from the number of stimuli (ns). For example, the mean choice Mc score for icon A of the sample data set is 5 - 3.5 = 1.5.

Step 3. Convert the mean choice scores (Mc) into p values by dividing them by (ns - 1). For example, the p value for icon A of the sample data set is 1.5/4 = 0.38.

The computations performed so far can be checked with the following:

  1. (1) Mean Mr must equal (ns + 1)/2,
  2. (2) Mean Mc must equal (ns - 1)/2, and
  3. (3) Mean p must equal 0.5

For example, in table 9-4,

  1. (1) Mean Mr = (3.50 + 2.00 + 1.60 + 2.90 + 5.00) / 5 = 3.00,
  2. (2) Mean Mc = (1.50 + 3.00 + 3.40 + 2.10 + 0.00) / 5 = 2.00, and
  3. (3) Mean p= (0.38 + 0.75 + 0.85 + 0.53 +0.00) /5 = 0.50.

Step 4. Convert the p values into z scores using table 9-5.

The z scores obtained in step 4 represent scores for the stimuli on an interval scale. Because the z values are generally awkward numbers to use, we recommend a linear transformation of the z values to a more convenient range as explained in step 5 below. (This transformation is similar to converting °F temperature to °C temperature.)

Step 5. (Optional) Obtain a linear transformation of the z scores (i.e., multiply the z scores by a constant and add a constant). In table 9-4, we arbitrarily use the transformation R = 5z + 6; any other linear transformation (e.g., R = 2z + 1) can be used. The purpose of this transformation is to obtain scores in a more convenient range than the z scores obtained in step 4.

The R values, like the z values obtained in step 4, represent interval-scale values for the stimuli, with higher values indicating stimuli judged to be better with respect to the attribute of interest. As seen in table 9-4, icon E cannot be assigned a scale value. This results from the fact that icon E was invariably placed in ranked 5. If a stimulus is placed in the same rank by all judges, it cannot be assigned a scale value using this procedure.

Table 9-5. p-z Conversation Table

p - z values
p 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1
z -2.33 -2.05 -1.88 -1.75 -1.64 -1.55 -1.48 1.41 -1.34 -1.28
p 0.11 0.12 0.13 0.14 0.15 0.16 0.17 0.18 0.19 0.02
z -1.23 -1.18 -1.13 -1.08 -1.04 -0.99 -0.95 -0.92 -0.88 -0.84
p 0.21 0.22 0.23 0.24 0.25 0.26 0.27 0.28 0.29 0.3
z -0.81 -0.77 -0.74 -0.71 -0.67 -0.64 -0.61 -0.58 -0.55 -0.52
p 0.31 0.32 0.33 0.34 0.35 0.36 0.37 0.38 0.39 0.4
z -0.5 -0.47 -0.44 -0.41 -0.39 -0.36 -0.33 -0.31 -0.28 -0.25
p 0.41 0.42 0.43 0.44 0.45 0.46 0.47 0.48 0.49 0.5
z -0.23 -0.2 -0.18 -0.15 -0.13 -0.1 -0.08 -0.05 -0.03 0
p 0.51 0.52 0.53 0.54 0.55 0.56 0.57 0.58 0.59 0.6
z 0.03 0.05 0.08 0.1 0.13 0.15 0.18 0.2 0.23 0.25
p 0.61 0.62 0.63 0.64 0.65 0.66 0.67 0.68 0.69 0.7
z 0.28 0.31 0.33 0.36 0.39 0.41 0.44 0.47 0.5 0.52
p 0.71 0.72 0.73 0.74 0.75 0.76 0.77 0.78 0.79 0.8
z 0.55 0.58 0.61 0.64 0.67 0.71 0.74 0.77 0.81 0.84
p 0.81 0.82 0.83 0.84 0.85 0.86 0.87 0.88 0.89 0.9
z 0.88 0.92 0.95 0.99 1.04 1.08 1.13 1.18 1.23 1.28
p 0.91 0.92 0.93 0.94 0.95 0.96 0.97 0.98 0.99 0.995
z 1.34 1.41 1.48 1.55 1.64 1.75 1.88 2.05 2.33 2.58

B. Torgerson's Categorical Scaling Method

This scaling method is based on Torgerson's Law of Categorical Judgment. (Torgerson, 1962). While it involves more computational steps than the Choice Score Method, it also seems to reflect stronger theoretical foundations. For the raw data of table 9-1, the two scaling methods produce similar scores; however, we cannot conclude that the scale values will be the same in all cases.

This method is as follows:

Step 1. Prepare a matrix (matrix A) that indicates the frequency with which each stimulus (icon) is placed in each rank. For the raw data of table 9-1, matrix A is as seen in table 9-6. The rows of this matrix represent the stimuli and the columns, the ranks. In each column is the number of times a given icon received that rank. Icons are treated as j in the equations; ranks as g. For example, the element in the fourth row, third column (4) is the number of times icon D was placed in rank 3.

Table 9-6 Matrix A

 
Icon Rank
1 2 3 4 5
A 0 1 3 6 0
B 3 5 1 1 0
C 6 2 2 0 0
D 1 2 4 3 0
E 0 0 0 0 10

Torgensons law equation. Click here for more detail.

  1. where:

    n = number of stimuli

    m+1 = number of categories

    sj = scale value of stimulus j

    tg = mean location of the gth category boundary

    s j = discriminal dispersion of stimulus j

    s g = dispersion of the gth category boundary

    rjg = correlation between momentary values associated with stimuli j and category boundary g

    xjg = unit normal deviate corresponding to the proportion of times stimulus j is sorted below boundary g

Step 1. Prepare a matrix (matrix A) that indicates the frequency with which each stimulus (icon) is placed in each rank. For the raw data of table 9-1, matrix A is as seen in table 9-6. The rows of this matrix represent the stimuli and the columns, the ranks. In each column is the number of times a given icon received that rank. Icons are treated as j in the equations; ranks as g. For example, the element in the fourth row, third column (4) is the number of times icon D was placed in rank 3.

Table 9-6 Matrix A

Icon Rank
1 2 3 4 5
A 0 1 3 6 0
B 3 5 1 1 0
C 6 2 2 0 0
D 1 2 4 3 0
E 0 0 0 0 10

 

Step 2. Obtain matrix B by identifying any elements of matrix A that are equal to the number of judges (nj) and removing the corresponding rows and columns from the matrix. For example, in matrix A of table 9-6, we identify one element that is equal to 10 (nj), and we delete the corresponding row and column (fifth row, and fifth column) to obtain matrix B of table 9-7. This step removes from the data matrix any stimuli that have been placed in the same rank by all judges. Such stimuli cannot be assigned an interval-scale value using this procedure, and their exclusion simplifies the computations.

Table 9-7 Matrix B

Icon Rank
1 2 3 4
A 0 1 3 6
B 3 5 1 1
C 6 2 2 0
D 1 2 4 3

 

Step 3. Construct matrix C by cumulating the rows of matrix B from the left. For matrix B of table 9-7, matrix C is as seen in table 9-8. The first row of this matrix is computed as follows: 0, 0 + 1 = 1, 0 + 1 + 3 = 4, 0 + 1 + 3 + 6 = 10. The elements in the last column should be equal to the number of judges.

Table 9-8 Matrix C

Icon Rank
1 2 3 4
A 0 1 4 10
B 3 8 9 10
C 6 8 10 10
D 1 3 7 10

 

Step 4. Construct matrix D by dividing each element of matrix C by the rightmost element of the corresponding row, and then obtain row totals. For matrix C of table 9-8, matrix D is as seen in table 9-9. The first row of this matrix is computed as follows: 0/10 = 0, 1/10 = 0.1, 4/10 = 0.4, 10/10 = 1. The row total of this row is 0 + 0.1 + 0.4 + 1 = 1.5.

Table 9-9 Matrix D

Icon Rank

Row

Total

1 2 3 4
A 0 0.1 0.4 1 1.5
B 0.3 0.8 0.9 1 3.0
C 0.6 0.8 1 1 3.4
D 0.1 0.3 0.7 1 2.1

 

Step 5. Construct matrix E by ordering the rows of matrix D in decreasing magnitude of row totals, and leaving out the rightmost column. For the matrix D of table 9-9, matrix E is as seen in table 9-10. The row that corresponds to icon C has the highest row total (3.4), therefore, it is the first row of matrix E. Rows corresponding to icons B, D, and A follow in that order.

Table 9-10 Matrix E

Icon Rank
1 2 3
C 0.6 0.8 1
B 0.3 0.8 0.9
D 0.1 0.3 0.7
A 0 0.1 0.4

 

Step 6. Construct matrix F, whose elements are equal to the z-scores (from table 9-5) of the corresponding elements of matrix E. For matrix E of table 9-10, matrix F is as seen in table 9-11. The element in the second row, third column (1.28) is the z-score corresponding to 0.9 in table 9-5. The elements of this matrix will be negative for all proportions below 0.5, and positive for all proportions above 0.5. Any cells of matrix E that contain proportions of 0 or 1 cannot be transformed into a z-score, and are left vacant.

Table 9-11 Matrix F

Icon Rank
1 2 3
C 0.25 0.84 -
B -0.52 0.84 1.28
D -1.28 -0.52 0.52
A - -1.28 -0.25

 

Step 7. Construct matrix G, which shows the differences in the rows of matrix F, and obtain row means. The first row of matrix G is computed by subtracting row 2 of matrix F from row 1 of matrix F; the remaining rows are computed similarly. For matrix F of table 9-11, matrix G is as seen in table 9-12. The first row of matrix F is constructed as follows: 0.25 - (-0.52) = 0.77, 0.84 - 0.84 = 0, vacant. In computing the row means, vacant cells are excluded. For instance, the mean of row 1 in table 9-12 is (0.77 + 0)/2 = 0.39.

Table 9-12 Matrix A

Icon Rank

Row

Mean

1 2 3
C-B 0.77 0.1 - 0.39
B-D 0.76 1.37 0.76 0.96
D-A - 0.76 0.78 0.77

Step 8. Compute interval-scale values for the stimuli by listing the stimuli in the same order as in matrix F, assigning a score of 0 to the last stimulus in the list, and cumulating toward the top using the row means of matrix G. For matrix G of table 9-12, the scale values are computed as shown in table 9-13. The first column lists the icons in the same order they were listed in matrix F of table 9-11; the second column lists the row means of matrix G of table 9-12. We first assign a scale value of 0 to icon A. We then compute the scale value of icon D as follows: 0+0.77 = 0.77. Scale values for icons B and C are assigned in a similar manner.

Table 9-13. Step 8: Assigning Interval-Scale Values

Icon

Row Means of Matrix G

(from table9-12)

Scale Value  
C 0.39 (C-B) 0 + 0.77 + 0.96 + 0.39 = 2.12 Black arrow pointing up indicate cumulating scale values from bottom to top

Assign

values by

cumulating

from

bottom to

top

 

B 0.96 (B-D) 0 + 0.77 + 0.96 = 1.73
D 0.77 (D-A) 0 + 0.77 = 0.77
A   0

Note that higher scale values indicate better (i.e., more appropriate) icons. Note also that we cannot assign a scale value for icon E, but we know that its score would be lower than the scores of all other icons, as it was placed in rank 5 by all judges.

4. CONDUCTING A TEST OF SIGNIFICANCE

Using the methods discussed so far, we might obtain a composite rank order for the stimuli with no ties, and a different interval-scale score for each stimulus. Does that mean that the stimuli are truly different from each other with respect to the attribute of interest? Not necessarily. To answer this question, we need to perform a test of significance. A test of significance allows us to determine, with a specified risk of error, whether the observed difference is really meaningful, or whether we might expect the same difference to occur merely because of chance factors.

On rank-order data from a group of judges (e.g., the raw data of table 9-1), we can conduct a significance test based on Friedman Rank Sums to determine whether there are or are not real differences between the stimuli. A detailed explanation of this statistical test is beyond the scope of this tutorial, but the general outline is as follows:

  1. Compute a summary score from the data. This summary score reflects the differences between the ranks assigned to each stimulus. The more different the ranks of each stimulus, the greater will be this summary score.

  2. Compare this summary score to a criterion value. This criterion indicates how big the summary score can get if there are no true differences between the stimuli. The criterion will vary depending on the number of observations in the data set and the risk of error one is willing to take. If the summary score is greater than the criterion, conclude that the stimuli are truly different with respect to the attribute of interest.

Using the Friedman test, we can test the equivalence of all stimuli, or perform pairwise comparisons. For a detailed explanation of this statistical test, readers should refer to a text on nonparametric statistical methods (e.g., Hollander & Wolfe, 1973). The Friedman test is also available in several commercial statistical software packages.

SUMMARY

We have discussed several methods of analyzing rank order data. These are summarized in table 9-14. This table provides a guide for the designer in determining which method(s) would be appropriate for the design problem at hand.

Table 9-14. Summry of Methods Discussed for Analyzing Rank Order Data

Method Outcome Computational Complexity
Compute sum of assigned ranks Provides a composite ranking Simple
Compute mean ranks Provides a composite ranking; will somewhat agree with the spacing of the composite ranks Simple
Compute interval-scale values for the stimuli using one of the following:
  1. 1) Choice score method
  2. 2) Torgerson's categorical scaling method
Provides an indication of relative differences between the stimuli Requires a series of simple computations
Conduct a test of significance Provides evidence for concluding, with a specified risk of error, that there are or are not real differences between the stimuli Complex, can be performed using statistical software packages

References:

  1. Engen, T. (1971). Psychophysics: Scaling methods. In J. W. Kling and L. A. Riggs (Eds.), Woodworth and Schlosberg's experimental psychology (3rd. ed.) (pp. 47-86). New York: Holt, Rinehart, & Winston, Inc.

  2. Hollander, M., and Wolfe, D. A. (1973). Nonparametric statistical methods. New York: J. Wiley & Sons, Inc.

  3. Torgerson, W. S. (1962). Theory and methods of scaling. New York: J. Wiley & Sons, Inc.

TUTORIAL 2: PROVIDING SUBJECTS WITH CONTEXT DURING ICON EVALUATIONS

This tutorial provides some guidance and examples of ways to provide appropriate context to experimental subjects during comprehension evaluations of candidate icons. In real-world driving, icons are presented in the context of certain in-vehicle capabilities and driving circumstances. As such, evaluations of in-vehicle icons should include a description of the context in which icons will be presented and used. However, icon evaluations should avoid providing either too little or too much context to experimental subjects. If too little context is provided, unrealistically low comprehension scores may result because the subjects may be unable to connect a visual icon with the many possible icon meanings. If too much context is provided, unrealistically high comprehension scores may result because the subjects have been cued for a certain response by the specificity of the context. Both these extremes should be avoided.

The context provided to subjects should describe the: (1) general capabilities of the in-vehicle system that will be used to present the icons and (2) general driving circumstances associated with the presentation of the icon by the in-vehicle system. For example, an evaluation of an icon intended to warn drivers of a problem with the passive restraint system on their vehicle might provide the following "system capabilities" information to subjects:

  1. Our purpose is to investigate issues related to the use of graphical icons in the in-vehicle environment. Recent advances in automotive technology have allowed the development of various In-Vehicle Information Systems (IVIS) that present a wide range of trip, vehicle, and safety information to drivers. Much of this information will be provided to drivers through color displays located on the instrument panel or center console of the vehicle. These display systems might present information such as:

  • Adaptive Cruise Control: An improved cruise control system that maintains adequate spacing between the driver's car and the vehicle ahead.

  • Augmented Signage: System that displays regulatory and road condition icons for the road on which the driver is presently driving. Icons would be similar to the signs posted along the roadside.

  • Collision Avoidance System: On-board sensor and display systems that warn the driver of an impending collision with another vehicle or object, vehicle lane departure, and other motion-based hazards.

  • Motorist Services: Information that provides an in-vehicle "Yellow Pages" function to drivers. This directory would present information regarding the availability and locations of motorist services such as gas stations, restaurants, hotels, or recreational activities.

  • Trip Navigation: Information that provides the driver with route assistance and trip planning. This includes driving directions, cautions about certain routes, current traffic conditions, road construction, or weather updates.

  • Vehicle Condition Monitoring: Device that informs the driver of problems with the vehicle or vehicle systems (oil, tire pressure, engine, etc.).

The following " driving circumstances" information could be provided to drivers:

Figure 9-1. Sample Icon of person with seat belt attachment. Click here for more detail.

Figure 9-1. Sample Icon

You have just started your car.

A group of icons appears on your In-Vehicle Information System, including this icon.

What do you think this icon means?

Similarly, in the context of evaluating comprehension for a specific Motorists Services icon, the following "driving circumstances" information could be provided:

Figure 9-2. Motorists Services Icon which illustrates a bridge being raised above water.

Figure 9-2. Motorists Services Icon

You are driving on a highway.

This icon appears on the In-Vehicle Information System installed in your car.

What do you think this icon means?

Previous | Table of Contents | Next

ResearchFHWA
FHWA
United States Department of Transportation - Federal Highway Administration