Appendix M

Previous | Table of Contents | Next

Introduction

This appendix includes a brief explanation and discussion of the OCPLOT computer simulation program for developing OC and EP curves. This program was developed as part of FHWA Demonstration Project 89, and a much more thorough discussion of the OCPLOT program is provided in the manual for that project. ⁽¹⁸⁾

In the case of pass/fail acceptance procedures, OC curves can be computed directly or constructed with the aid of specialized mathematical tables. For acceptance procedures with adjusted payment schedules, the construction of EP curves usually requires computer assistance. Program OCPLOT uses computer simulation to develop both OC and EP curves.

Features of Program OCPLOT

Program OCPLOT is designed to analyze the types of acceptance procedures typically used in the highway field. This includes pass/fail procedures, leading to the type of OC curve shown in figure M­1, and payment adjustment procedures, leading to the type of EP curve shown in figure M­2.

Figure M­3 lists some of the options that may be selected from the primary menu. The various items appear on the screen one at a time in a logical sequence, and later items are dependent upon the responses to earlier ones. The versatility of the program is apparent from the many different ways these selections might be combined.

When the selections from the first menu are complete, the menu will appear similar to that shown in figure M­4. The prompt "Press any key to continue" at the bottom of the display provides a pause that gives the user the opportunity to use the <ESC> key to go back and change some values or press the <PrintScreen> key to save the screen to a file if a record of the menu selections is desired. Striking almost any other key will cause the second menu in figure M­4 to appear.

First Menu:

Second Menu:

Because program OCPLOT uses computer simulation to analyze whatever acceptance procedure has been specified, it is very computationally intensive and the execution speed is dependent upon the level of precision selected from the second menu. Table M­1 lists the number of replications performed for the different levels of precision.

Selection 1 provides the fastest execution, which is useful for exploratory work but may not be good enough to report as a final result. When this level is selected, 200 sample sets of the desired size are randomly generated from a normal population for each of several known levels of quality.

Each sample set is evaluated in accordance with the acceptance plan specified in the primary menu and the results are stored in memory for subsequent analysis. This is far more thorough and many times faster than testing the acceptance procedure with a field trial. (A field trial would be appropriate only if the procedure survives this initial check.)

Selection 2 from the second menu provides an intermediate level of precision for which 1000 sample sets are generated at each quality level. This level of precision is usually satisfactory to report as a final result, producing points for the OC curve, representing probability of acceptance, or the EP curve, representing the expected payment factor, that are typically accurate to within one or two units. If still better precision is desired, selection 3 will cause 5000 sample sets to be generated at each quality level. This level of precision tends to produce a smooth line when the OC or EP curve is plotted.

Once the precision level is selected from the second menu, the computational process begins. For either low or intermediate precision, program OCPLOT displays detailed information at the two key points at which risk levels are usually expressed­the AQL and the RQL­as shown in figures M­5 and M­6. This serves two important purposes. For users less familiar with statistical estimation procedures and acceptance plans, the graphical displays at the AQL and RQL are both informative and educational. It may come as a surprise to some, for example, how widely distributed the quality estimates are, especially for small sample sizes. For users more familiar with statistical acceptance procedures, these displays provide assurance that the simulation process is working properly. The actual displays on a color monitor are color­coded to clearly distinguish acceptable and rejectable results and the corresponding payment factors.

 Figure M­5. Typical Display at AQL at Intermediate Precision by
Program OCPLOT

Figure M­6. Typical Display at RQL at Intermediate Precision by Program OCPLOT

Although the AQL and the RQL are probably the most important points at which it is desirable to know how the acceptance procedure will perform, it usually is useful to have a plot of the entire OC or EP curve that provides a picture of the performance over the complete range of quality that might be encountered. The prompt at the bottom of the screen instructs the user to strike any key to continue with this step to obtain the display shown in figure M­7. The horizontal and vertical axes and the two previously calculated points at the AQL and the RQL appear on the screen immediately. The remaining points appear one at a time at a speed determined by the level of precision that has been selected and the speed of the machine running the program.

After all the points have been calculated and plotted, the user may strike any key to connect the points with a solid line. Following this, the next key stroke will add vertical and horizontal lines highlighting the performance of the acceptance plan at the AQL and RQL, as shown in figure M­8. And, like the histograms in figures M­5 and M­6, any of these displays may be saved to a file by using the <PrintScreen> key.

Following this display, striking a key will produce the menu shown in figure M­9. If the first item in this menu is selected, the output shown in figure M­10 is displayed. This permits the user to save the values of the data points shown in figure M­7 from which the OC curve was constructed. The other selections in this menu make it possible to return to earlier points in the input stage of the program or to exit the program.

Example M­1: Pass/Fail Attributes Procedure

Attributes acceptance procedures are based on measures that are counted rather than computed, such as the number of defects on an item of production or the number of test results falling outside specification limits. In contrast, variables acceptance procedures are based on statistical parameters that are computed, such as the mean and standard deviation, and lead to the estimation of PD or PWL.

Advantages of attributes procedures are that they are simple to apply and they require no assumptions about the underlying distributional form of the population being sampled. For example, a typical attributes procedure might require that a sample of size N = l0 be taken and that no more than C = 2 test results may be outside the specification limits, where "C" is referred to as the acceptance number. A disadvantage is that they require larger sample sizes to achieve the equivalent discriminating power of a variables procedure. Variables procedures, because of their inherently greater efficiency, are generally preferred whenever the requirement for a normally­distributed population is reasonably satisfied.

Figure M­8. Display of EP Curve Plotted by Program OCPLOT with AQL and RQL Performance Highlighted

Figure M­9. Third Menu for Program OCPLOT

Figure M­10. Display of Numerical Values of Points on EP Curve Computed
at High Precision by Program OCPLOT

Although the vast majority of highway construction measures tend to be normally distributed, there are some that are not. A physical constraint close to the desired target value often produces non­normality. For example, depth of cover over the top mat of reinforcing steel in a bridge deck may tend to be skewed because it is physically impossible (provided the steel is embedded) for the cover to be less than zero but there is less of a restriction on how deep the mat might be. Similarly, if the target level for percent air voids in bituminous pavement is fairly low, the physical limit of zero may tend to skew this distribution toward larger values. Another condition that can produce non­normality is the combining in the same lot of two distinctly different populations. As a general rule, a conscious effort should be made to avoid this condition when applying a statistical specification.

Because situations may arise in highway construction that warrant their use, program OCPLOT provides the capability of analyzing attributes acceptance procedures. Although it would be possible to develop a payment schedule based on an attributes procedure, their use has been limited almost exclusively to pass/fail applications. The following is presented as a generic example of such a procedure.

It is assumed for this example that the statistical quality measure is the percent defective (PD), the percentage of the lot falling outside specification limits. The acceptable quality level (AQL) and the rejectable quality level (RQL) are defined as PD = 10 and PD = 50, respectively. It is desired to develop an acceptance procedure that will balance the risks at 0.05. In other words, if the contractor provides work that is exactly at the AQL, there is to be a 0.05 chance that it will erroneously be rejected and, at the other extreme, if the work is truly at the RQL, there is to be a 0.05 chance that it will erroneously be accepted.

An acceptance plan is required, stated in terms of the sample size (N) and the acceptance number (C) that will produce the desired risks. For both risks to be 0.05, an OC curve is desired that indicates probabilities of acceptance at the AQL and the RQL of 0.95 and 0.05, respectively. Because both N and C must be integer values, the resultant risk levels vary in discrete steps, and it usually is not possible to obtain exactly the desired risk values. To find a plan that matches the desired risk levels as closely as possible, it is necessary to examine several plans.

This can easily be accomplished with program OCPLOT by selecting "Pass/Fail" and "Attributes" from the opening menu, followed by other appropriate selections and ending with a trial combination of N and C. It is usually advantageous to make the first few runs at low precision (selected from the second menu) in order to speed up the trial"and"error process.

Figure M­11 shows the completed menu for the initial try with N = 10 and C = 2 and figure
M­12 shows the resulting acceptance probabilities obtained at high precision. It can be seen in figure M­12 that the probability of acceptance of 0.052 at the RQL is close to the desired value, while the probability of acceptance of 0.927 at the AQL is too low.

Figure M­11. Completed Menu for Analysis of Pass/Fail Attributes
Acceptance Plan with N = 10 and C = 2

Figure M­12. Numerical Values of Points on EP Curve for Pass/Fail Attributes Acceptance Plan with N = 10 and C = 2

The results of several attempts are presented in table M­2. The values in this table were all obtained at high precision and here it can be seen that the plan having N = 13 and C = 3 comes the closest to meeting the desired acceptance probabilities of Prob. ³ 0.95 and Prob. £ 0.05 at the AQL and RQL, respectively. This is a trial and error process, but, with a little experience, it usually is possible to find a suitable plan relatively quickly.

The attributes acceptance plan meeting the requirements of this example requires that a total of N = 13 test specimens be taken from random locations within each lot and that, after the appropriate tests have been performed, no more than C = 3 of the test results shall be outside specification limits. The use of such a plan ensures that truly AQL work will be accepted about 95 percent of the time and that truly RQL work will be accepted only about 5 percent of the time.

Example M­2: Pass/Fail Variables Procedure

Variables procedures are based on the assumption that the population being sampled is at least approximately normally distributed. They involve the computation of statistical parameters, such as the mean and standard deviation, to estimate PD or PWL. Provided the normality assumption is sufficiently well satisfied, variables procedures can provide essentially the same discriminating power as equivalent attributes plans, but with a substantially smaller sample size.

To illustrate this last statement, a variables procedure is sought that will have essentially the same discriminating power (OC curve) as the attributes procedure developed in example M­1. Like the previous example, this involves a trial and error process with which program OCPLOT can be extremely helpful. For this example, the selections "Pass/Fail" and "Variables" are made from the opening menu, followed by other appropriate selections and ending with trial values for the sample size and acceptance limit.

This trial­and­error process proceeds more quickly if a low level of precision is selected for the initial attempts. When it appears that the appropriate combination of sample size and acceptance limit has been found, this should be checked at intermediate precision. Further minor adjustments may then be necessary before confirming the result at high precision.

Figure M­13 shows the completed menu for the variables acceptance plan that was ultimately selected. The actual numerical values on the OC curve are shown in table M­3, which summarizes the results of examples M­1 and M­2. In this table it can be seen that the performance of the variables plan very closely matches that of the attributes plan in example M­1 (table M­2, N = 13, C = 3). The dramatic difference is that this has been accomplished with a sample size of N = 8, whereas the attributes plan required a sample size of N = 13. In this case, which assumes that the normality assumption is satisfied, the use of a variables plan results in a direct savings in sampling and testing costs of nearly 40 percent.

Example M­3: Analysis of Payment Equation

Although the pass/fail acceptance procedures discussed in examples M­1 and M­2 may be useful for highway construction applications, the use of acceptance procedures with adjusted payment schedules is generally of greater interest. The proper design of such plans is critical to their performance, and poorly conceived plans may be either totally ineffective or impractically severe. Neither problem may be apparent, however, until the plan has been analyzed by constructing the EP curve.

Equation M­1 gives a payment schedule proposed for use by an owner. The AQL was defined as PWL_SPEC = 90 and, since there was no incentive payment provision, the maximum payment factor was limited to 100 percent.

where: PR = payment reduction (percent).

PWL_SPEC = specified PWL at the AQL.

PWL_COMP = PWL computed from test values.

To transform this equation into a form that can be handled by program OCPLOT, it is necessary to express it in terms of the payment factor rather than the payment reduction. By substituting PWL_SPEC = 90 and PR = 100 ­ PF into equation M­1, equation M­2 is obtained, subject to the restriction that PF £ 100.

where: PF = payment factor (percent)

PWL_COMP = PWL computed from test values.

To judge the effectiveness of this payment equation, the EP curve will be developed for a typical sample size of N = 5. The completed input menu is shown in figure M­14. Following this, an intermediate level of precision was selected from the second menu to obtain the display at the AQL shown in figure M­15. Finally, the program was run at high precision, which produced the displays shown in figures M­16 and M­17.

Figure M­15. Performance of Payment Equation at AQL

Figure M­16. EP Curve for Analysis of Payment Equation

Figure M­17. Numerical Values for Points on EP Curve for
Analysis of Payment Equation

It can be seen in figure M­15 that there is a serious problem with this acceptance procedure. A contractor who performs consistently at the AQL will receive an average payment reduction of nearly 5 percent. To emphasize the impact this would have on the construction industry, this means that a contractor responsible for $10 million worth of work under this specification over the course of a construction season would on the average be penalized approximately $500,000 for successfully providing the level of quality that was explicitly defined as acceptable in the contract documents.

This example illustrates the situation discussed previously, whereby the failure to include an incentive payment provision with this type of specification prevents the acceptance procedure from paying an average of 100 percent when the work is exactly at the AQL. The reason for this can be seen with the help of figure M­15. The upper histogram in this figure represents the distribution of 1000 PWL estimates, each obtained by randomly sampling a population that is exactly of AQL quality. These sample PWL estimates range from a low of about 47 percent to a high of 100 percent because of the inherent variability of the sampling process. However, the long­term average of these estimates will always be extremely close to the true value (the AQL in this case) because the PD/PWL estimation process is an unbiased statistical estimation procedure. The problem arises because the payment schedule does not permit this unbiased property to extend to the distribution of payment factors. Because there is no incentive payment provision, all PWL estimates that are greater than the true PWL of 90 receive the maximum payment factor of 100 percent, while all those below the true value receive some degree of payment reduction. Since approximately half the lots will receive payment reductions and the other half will receive 100 percent payment, the net result is that the average payment factor for AQL work will be substantially lower than 100 percent.

This situation is clearly misleading and unfair, yet it exists in current QA specifications. It is not difficult to correct, however, as demonstrated in the next example.

Example M­4: Effect of Incentive Payment Provision

The problem described in the preceding example, in which truly AQL work receives a substantial payment reduction, is easy to correct. All that is required is the inclusion of an incentive payment provision as part of the acceptance procedure. In equation M­2, this would mean removing the restriction that the maximum payment factor cannot exceed 100 percent.

Ordinarily, the maximum payment factor and the slope of the payment equation should be consistent with established (or estimated) performance relationships and the anticipated economic consequences of any departures from the specified AQL. To be consistent with this example, a maximum incentive payment factor of 110 percent will have to be used. In actual practice, however, most agencies have used incentive payment provisions of 105 percent or less.

To confirm that this will solve the problem, program OCPLOT was run an additional time with the identical input used for example M­3 except that when the prompt "MAXIMUM PAY FAC ­ TOR" appeared, selection 2 was chosen, indicating that no upper limit is placed on the payment equation. The program then automatically computed and displayed the maximum payment factor of PF = 110.0, as shown in the completed menu in figure M­18. The performance at the AQL is displayed in figure M­19, where it is seen that the expected payment factor is 100 percent, as desired.

Figure M­19. Performance at AQL of Acceptance Procedure with
Inventive Payment Provision

Example M­5: OC Curves for Payment Adjustment Plans

It was shown in chapter 7 that multiple OC curves could be plotted to illustrate the performance of acceptance plans with payment adjustment provisions. Each of these curves is associated not with the probability of acceptance, but with the probability of receiving greater than or equal to a given payment level. For example, suppose that equation M­3 is selected to determine the payment for a lot based on a sample of size N = 5.

From this equation it can be seen that full payment, i.e., PF = 100 percent, is obtained for a computed lot PWL of 90. Lot PWL values greater than 90 will lead to an incentive payment, while lot PWL values less than 90 will lead to a disincentive payment.

The probability of receiving a payment greater than or equal to 100 percent therefore is equivalent to the probability of receiving an individual estimated lot PWL of 90 or greater. By using the Pass/Fail option and selecting 90 as the ACCEPTANCE LIMIT, the OCPLOT program can be used to determine this OC curve. The completed OCPLOT menu for this situation is shown in figure M­20, and the corresponding OC curve is shown in figure M­21.

Similarly, the probability of receiving greater than or equal to 95 percent payment is equivalent to the probability of receiving an individual estimated lot PWL of 80 or greater (obtained by plugging PF = 90 into equation M­3 and solving for PWL). OCPLOT can once again be used with the Pass/Fail option and with 80 as the ACCEPTANCE LIMIT to determine this OC curve. This can be repeated for other payment levels to develop the family of OC curves shown in figure M­22.

Figure M­21. OC Curve for the Probability of Receiving Greater than or Equal
to 100 Percent Payment for Example M­5

Figure M­22. Family of OC Curves Showing the Performance of the Acceptance Plan Using a Sample of Size N = 5 and Payment Equation M­3