5. Quality Measures: ACcuracy and Precision

Previous | Table of Contents | Next

INTRODUCTION

There are several quality measures that can be used in an acceptance plan. In early QA specifications in the late 1960s and 1970s, the mean (or average), or the average deviation from a target value, was often used as the quality measure. However, the use of the average alone provides no measure of variability, and it is now recognized that variability is often an important predictor of performance. The problems associated with basing acceptance on only the mean are well documented. Therefore, there was no need to consider this approach in the analyses that were performed.

Several quality measures, including percent defective (PD) and PWL, have been preferred in recent years because they simultaneously measure both the average level and the variability in a statistically efficient way. Other measures that have been used by some agencies include AAD and the moving average. An additional measure that may be considered by some agencies is the CI.

PD and PWL are, in reality, the same quality measure since they are directly related by the simple relationship PWL = 100 - PD. Therefore, analyses reported using PWL also apply equally to PD, and vice versa.

The quality measures that were evaluated during the project include:

• PWL.
• AAD.
• CI.

ACCURACY AND PRECISION

Since each of the above measures provides an estimate for a true population value, it is important to know that the estimator provides an unbiased estimate for the population parameter. It is known that this is true for the procedure for estimating PWL. Computer simulation was used to evaluate whether there was bias in any of the quality measures. A lack of bias means that the estimator is accurate. Another important factor to be considered is the variability associated with the quality measure (i.e., how much do individual estimated values vary about the long-term average value for the quality measure?) This, too, was investigated by computer simulation studies. Low variability for the estimated values is a sign that the estimator is precise.

Computer simulation studies were performed to evaluate the accuracy and precision of the estimated quality measures for various sample sizes. The results of these studies are presented for each measure in the following sections.

PWL EVALUATION

The PWL for a lot can be estimated by using the quality index, Q. The Q-statistic is used with a PWL table to determine the estimated PWL for the lot. A PWL table is shown in table 7.

Conceptually, the Q-statistic, or quality index, performs the same function as the Z-statistic, except that the reference point is the mean of an individual sample, , instead of the population mean, , and the points of interest with regard to areas under the curve are the specification limits.

(1)

and

(2)

where: = quality index for the lower specification limit

= quality index for the upper specification limit

LSL = lower specification limit

USL = upper specification limit

= sample mean for the lot

s = sample standard deviation for the lot

is used when there is a one-sided lower specification limit, while is used when there is a one-sided upper specification limit. For two-sided specification limits, the PWL value is estimated as:

(3)

where: PWL_U = percent below the upper specification limit (based on )

PWL_L = percent above the lower specification limit (based on )

PWL_T = percent within the upper and lower specification limits

Simulation Analyses

Computer simulation was used to generate samples of various sizes from known populations. These sample results were then used to estimate the PWL for the population. The mean and standard deviation for the estimated PWL values were then calculated. The difference between the mean estimated PWL was subtracted from the known population PWL to provide a measure of accuracy for the PWL estimates. The standard deviation of the estimated PWL values was used as a measure of precision (i.e., the amount of variability) for the PWL estimates.

Table 7. Quality index values for estimating PWL.

Table 7. Quality index values for estimating PWL (continued).

PWL	n = 12 to 14	n = 15 to 18	n = 19 to 25	n = 26 to 37	n = 38 to 69	n = 70 to 200	n = 201 to
100	2.83	3.03	3.20	3.38	3.54	3.70	3.83
99	2.09	2.14	2.18	2.22	2.26	2.29	2.31
98	1.91	1.93	1.96	1.99	2.01	2.03	2.05
97	1.77	1.79	1.81	1.83	1.85	1.86	1.87
96	1.67	1.68	1.70	1.71	1.73	1.74	1.75
95	1.58	1.59	1.61	1.62	1.63	1.63	1.64
94	1.50	1.51	1.52	1.53	1.54	1.55	1.55
93	1.44	1.44	1.45	1.46	1.46	1.47	1.47
92	1.37	1.38	1.39	1.39	1.40	1.40	1.40
91	1.32	1.32	1.33	1.33	1.33	1.34	1.34
90	1.26	1.27	1.27	1.27	1.28	1.28	1.28
89	1.21	1.22	1.22	1.22	1.22	1.22	1.23
88	1.17	1.17	1.17	1.17	1.17	1.17	1.17
87	1.12	1.12	1.12	1.12	1.12	1.13	1.13
86	1.08	1.08	1.08	1.08	1.08	1.08	1.08
85	1.04	1.04	1.04	1.04	1.04	1.04	1.04
84	1.00	1.00	1.00	1.00	0.99	0.99	0.99
83	0.96	0.96	0.96	0.96	0.95	0.95	0.95
82	0.92	0.92	0.92	0.92	0.92	0.92	0.92
81	0.89	0.88	0.88	0.88	0.88	0.88	0.88
80	0.85	0.85	0.85	0.84	0.84	0.84	0.84
79	0.82	0.81	0.81	0.81	0.81	0.81	0.81
78	0.78	0.78	0.78	0.78	0.77	0.77	0.77
77	0.75	0.75	0.75	0.74	0.74	0.74	0.74
76	0.72	0.71	0.71	0.71	0.71	0.71	0.71
75	0.69	0.68	0.68	0.68	0.68	0.68	0.67
74	0.66	0.65	0.65	0.65	0.65	0.64	0.64
73	0.62	0.62	0.62	0.62	0.62	0.61	0.61
72	0.59	0.59	0.59	0.59	0.59	0.58	0.58
71	0.57	0.56	0.56	0.56	0.56	0.55	0.55
70	0.54	0.53	0.53	0.53	0.53	0.53	0.52
69	0.51	0.50	0.50	0.50	0.50	0.50	0.50
68	0.48	0.48	0.47	0.47	0.47	0.47	0.47
67	0.45	0.45	0.45	0.44	0.44	0.44	0.44
66	0.42	0.42	0.42	0.42	0.41	0.41	0.41
65	0.40	0.39	0.39	0.39	0.39	0.39	0.39
64	0.37	0.36	0.36	0.36	0.36	0.36	0.36
63	0.34	0.34	0.34	0.34	0.33	0.33	0.33
62	0.31	0.31	0.31	0.31	0.31	0.31	0.31
61	0.29	0.29	0.28	0.28	0.28	0.28	0.28
60	0.26	0.26	0.26	0.26	0.26	0.25	0.25
59	0.23	0.23	0.23	0.23	0.23	0.23	0.23
58	0.21	0.21	0.20	0.20	0.20	0.20	0.20
57	0.18	0.18	0.18	0.18	0.18	0.18	0.18
56	0.16	0.15	0.15	0.15	0.15	0.15	0.15
55	0.13	0.13	0.13	0.13	0.13	0.13	0.13
54	0.10	0.10	0.10	0.10	0.10	0.10	0.10
53	0.08	0.08	0.08	0.08	0.08	0.08	0.08
52	0.05	0.05	0.05	0.05	0.05	0.05	0.05
51	0.03	0.03	0.03	0.03	0.03	0.03	0.02
50	0.00	0.00	0.00	0.00	0.00	0.00	0.00

Source: "Specification Conformity Analysis," FHWA Technical Advisory T5080.12, June 23, 1989

As was known in advance, the quality index provided an unbiased estimate of the population mean, as evidenced by the lack of bias in the mean of the estimated PWL values. Also as expected, the amount of variability in the PWL estimates decreased as the sample size increased. This is shown in table 8. Also shown in the table is the fact that the standard deviation of the estimated PWL values is a maximum at an actual PWL of 50, and decreases to a minimum as the actual PWL approaches 100 or zero. This is also shown in the plot in figure 8.

The reason for this is illustrated in figures 9 through 11, which plot histograms of the distribution of estimated PWL values for sample sizes = 3, 5, and 10, and for actual PWL values = 90, 70, and 50. As shown in the figures, as the actual PWL approaches the natural boundary imposed by 100 PWL, the spread of estimated values decreases since the estimated values cannot exceed 100 (see figure 9). As the actual PWL value moves away from the natural boundary of 100 to a value of 70, there is a greater spread available for the estimated values (see figure 10). When the actual PWL value is 50, there is the maximum opportunity for the estimated values to spread out before hitting the natural boundaries of 100 and 0 PWL (see figure 11).

To further illustrate the lack of bias and how the variability of PWL estimates varies with sample size, figure 12 plots the average estimated PWL values from the simulation results versus the actual PWL values. The straight line at a 45-degree angle, which indicates that the estimated values vary only very slightly from the actual PWL values, shows the lack of bias in the estimator. The other lines on each plot represent the 10^th and 90^th percentiles (i.e., the values that 10 percent of the estimates are either below or exceed, respectively). As the sample size increases, these bounds become narrower, indicating a reduction in variability that is also reflected in the standard deviation values in table 7. Also, figure 13 plots the average differences versus the actual PWL value to show that the differences are centered on zero and show no consistent positive or negative bias.

To investigate the effect of project size, for sample sizes = 3, 5, and 10, various numbers of lots per project, ranging from 1 to 90, were also considered. As would be expected, since it represented an increase in the total number of tests, the amount of variability in the average estimated PWL decreased as the number of lots in the project increased. This is shown in tables 9 through 11 and in figure 14.

Table 8. Accuracy and precision for PWL estimates (based on the results of 10,000 simulated lots).

Figure 8. Plot of how the standard deviation of PWL estimates varies with the population PWL.

Figure 9a. Histogram illustrating the distribution of estimated PWL values for 1000 simulated lots from a population with 90 PWL, sample 3.

Figure 9b. Histogram illustrating the distribution of estimated PWL values for 1000 simulated lots from a population with 90 PWL, sample 5.

Figure 9c. Histogram illustrating the distribution of estimated PWL values for 100 simulated lots from a population with 90 PWL, sample 10.

Figure 10a. Histogram illustrating the distribution of estimated PWL values for 1000 simulated lots from a population with 70 PWL, sample 3.

Figure 10b. Histogram illustrating the distribution of estimated PWL values for 1000 simulated lots from a population with 70 PWL, sample 5.

Figure 10c. Histogram illustrating the distribution of estimated PWL values for 1000 simulated lots from a population with 70 PWL, sample 10.

Figure 11a. Histogram illustrating the distribution of estimated PWL values for 1000 simulated lots from a population with 50 PWL, sample 3.

Figure 11b. Histogram illustrating the distribution of estimated PWL values form 1000 simulated lots from a population with 50 PWL, sample 5.

Figure 11c. Histogram illustrating the distribution of estimated PWL values from 1000 simulated lots from a population with 50 PWL, sample 10.

Figure 12a. Illustration 1 of accuracy and precision of PWL estimates.

Figure 12b. Illustration 2 of accuracy and precision of PWL estimates.

Figure 12c. Illustration 3 of accuracy and precision of PWL estimates.

Avg. Diff. = average difference between the average estimated PWL and the actual PWL for each simulated project

2.5% = value for which 2.5 percent of the simulated average differences were less than or equal

97.5% = value for which 97.5 percent of the simulated average differences were less than or equal

Avg. Diff. = average difference between the average estimated PWL and the actual PWL for each simulated project

2.5% = value for which 2.5 percent of the simulated average differences were less than or equal

7.5% = value for which 97.5 percent of the simulated average differences were less than or equal

Table 11. Results of simulation analyses with actual PWL = 50 (distribution of the results from 1000 simulated projects).

Avg. Diff. = average difference between the average estimated PWL and the actual PWL for each simulated project

2.5% = value for which 2.5 percent of the simulated average differences were less than or equal

97.5% = value for which 97.5 percent of the simulated average differences were less than or equal

Figure 13. Plot of average difference of simulated PWL values versus actual PWL values for sample sizes = 3, 5, and 10.

Figure 14a. Plots of the 95^th percentile for the average estimated PWL minus the actual PWL at 50 versus the number of lots per project.

Figure 14b. Plots of the 95^th percentile for the average estimated PWL minus the actual PWL at 70 versus the number of lots per project.

Figure 14c. Plots of the 95^th percentile for the average estimated PWL minus the actual PWL at 90 versus the number of lots per project.

AAD EVALUATION

For specifications that have a target value, the average deviation from the target value has, in the past, sometimes been used as a means for determining the acceptability of the product. This approach can have the effect of encouraging the contractor to manipulate its process during the production of a lot. For example, if two test results in the morning are below the target value, there is a strong incentive for the contractor to increase the process mean in the afternoon in an effort to get two higher test results so that the average of the four tests for the lot will be near the target value. In essence, this acceptance approach encourages the contractor to increase process variability by making frequent adjustments to the process mean. To avoid the problem of overadjusting the process in response to early test results, the average absolute deviation from the target can be used for the acceptance decision.

The equation for calculating AAD is as follows:

(4)

where: X_i = individual test results

T = target value

n = number of tests per lot

Simulation Analyses

A computer simulation program was developed to evaluate the accuracy and precision of estimated AAD values. The program generates 1000 simulated lots. The mean, or average, estimated AAD value and the standard deviation for the estimated AAD values are then calculated.

The program can represent populations that are centered on the target value, but can also generate data for populations with means that are offset from the desired target. The actual AAD value for a population centered on the target is 0.798 multiplied by the standard deviation of the population. The actual AAD value increases as the population mean is offset from the target value. Without sacrificing generality, the simulation studies were performed for populations with standard deviation values of 1.00. For a population with a standard deviation other than 1.00, the simulated average AAD values would simply be multiplied by the standard deviation of the population in question.

For the study of AAD accuracy and precision, populations were simulated with offsets from the target that ranged from 0.00 to 2.50 in increments of 0.25. The populations from which the samples were generated were normal distributions with a standard deviation of 1.00.

The shape of the distribution of the sample AAD values varies with the sample size. For small mean offsets, the distribution is skewed to the right, with the degree of skewness determined by the sample size. For a sample size = 1, the peak of the distribution occurs at AAD values near zero. As the sample size increases, the value for the peak of the distribution increases and the degree of skewness decreases, because as the sample size increases, it becomes more and more difficult to have average AAD values near zero. This relationship is shown in table 12.

For a given sample size, the shape of the distribution of the sample AAD values also varies with the offset of the population mean from the target value. The skewness of the distribution stems from the fact that the deviations from the target cannot be below zero, thereby yielding distributions skewed away from zero. As the offset departs farther from the target, the distribution of the sample AAD values approaches symmetry since the deviation values are always greater than zero. This relationship is shown in table 13 for a sample size = 3 and for mean offsets = 0.50 to 2.50.

Sample Size, n	Distribution of Sample AAD Values
1
3
5
10

Table 13. Distributions of sample AAD values for sample size = 3 and population means offset from the target by 0.50, 1.00, 1.50, 2.00, and 2.50 standard deviations.

Mean Offset	Distribution of Sample AAD Values
0.50
1.00
1.50
2.00
2.50

Table 14 presents a summary of the simulation results for AAD. The table includes sample sizes = 3, 5, and 10, and populations with means offset from the target by 0.00 to 2.50 standard deviations in increments of 0.25. The table also includes the actual AAD values for each population, which are based on the mean offset values. Finally, the table includes the average differences and the standard deviations for the estimated AAD values.

The results in table 14 show that the sample AAD is an unbiased estimator of the population AAD. It also shows that the standard deviation values decrease as the sample size increases and the standard deviation values increase as the offset from the target value increases. This is caused by the fact that as the offset increases, a deviation value of zero no longer presents a barrier, thereby allowing the deviations to spread evenly on both sides of the AAD value. These two trends are illustrated in figures 15 and 16.

CI EVALUATION

Conceptually, the CI is similar to AAD. While AAD uses the average of the absolute values of the individual deviations from the target value, the CI uses the squares of the individual deviations from the target value. The CI is also similar in concept to the standard deviation. The standard deviation is the root mean square of differences from the mean, whereas the CI is the root mean square of differences from a target. Like AAD, the CI discourages mid-lot process adjustments by not allowing positive and negative deviations from the target to cancel out one another. The CI is calculated as follows:

(5)

where: X_i = individual test results

T = target value

n = number of tests per lot

Simulation Analyses

A computer simulation program was developed to evaluate the accuracy and precision of estimated CI values. The program generated 10,000 simulated lots. The mean estimated CI value and the standard deviation for the estimated CI values are then calculated.

The simulation studies present the CI in terms of population standard deviation. Without sacrificing generality, the simulation studies were performed for populations with standard deviations of 1.00. For a population with a standard deviation other than 1.00, the simulated average CI values would simply be multiplied by the standard deviation of the population in question. The program can represent populations that have various CI values. If the population mean is centered on the target, then the CI is the same as the standard deviation. Therefore, the CI cannot be less than zero.

Table 14. Accuracy and precision for AAD estimates (based on the results of 10,000 simulated lots).

Figure 15. Plot of average difference of simulated AAD values versus actual AAD values for sample sizes = 3, 5, and 10.

Figure 16. Plot of how the standard deviation of AAD estimates varies with the population AAD value.

For the study of CI accuracy and precision, populations were simulated with CI values ranging from 1.00 to 3.00. The populations from which the samples were generated were normal distributions with a standard deviation of 1.00.

The shape of the distribution of the sample CI values varies with the sample size in a similar manner to that of AAD. For small CI values, the distribution is skewed to the right, with the degree of skewness determined by the sample size. As the sample size increases, the degree of skewness decreases, because as the sample size increases, it becomes more and more difficult to have average CI values near zero.

Similar to that of AAD, for a given sample size, the shape of the distribution of the sample CI values varies with the CI of the population. For small values of CI, the skewness of the distribution stems from the fact that the squared deviations from the target cannot be below zero, thereby yielding distributions skewed away from zero. As the CI increases, meaning that the population mean is farther from the target, the distribution approaches symmetry since the squared deviation values are always greater than zero.

The results in table 15 show that the sample CI appears to be a slightly biased estimator of the population CI since the average of the CI estimates is always lower than the population CI. This bias decreases as the sample size increases. The table also shows that the standard deviation values decrease as the sample size increases and the standard deviation values increase as the CI value for the population increases. These results are illustrated in figures 17 and 18.

CONCLUSION

Both the PWL and AAD estimators were unbiased and exhibited similar trends with respect to the variability of their individual estimates. The CI performed essentially the same as AAD, except that it appeared to be a slightly biased estimator. Since the CI offered no benefits compared to AAD and since it appeared slightly biased, it was decided to eliminate the CI and to concentrate further study on only PWL and AAD.

Figure 17. Plot of average difference of simulated CI values versus actual CI values for sample sizes = 3, 5, and 10.

Figure 18. Plot of how the standard deviation of CI estimates varies with the population CI value.