U.S. Department of Transportation
Federal Highway Administration
1200 New Jersey Avenue, SE
Washington, DC 20590
202-366-4000


Skip to content
Facebook iconYouTube iconTwitter iconFlickr iconLinkedInInstagram

Federal Highway Administration Research and Technology
Coordinating, Developing, and Delivering Highway Transportation Innovations

Report
This report is an archived publication and may contain dated technical, contact, and link information
Publication Number: FHWA-RD-02-095

Appendix M

Previous | Table of Contents | Next

Introduction

This appendix includes a brief explanation and discussion of the OCPLOT computer simulation program for developing OC and EP curves. This program was developed as part of FHWA Demonstration Project 89, and a much more thorough discussion of the OCPLOT program is provided in the manual for that project. (18)

In the case of pass/fail acceptance procedures, OC curves can be computed directly or constructed with the aid of specialized mathematical tables. For acceptance procedures with adjusted payment schedules, the construction of EP curves usually requires computer assistance. Program OCPLOT uses computer simulation to develop both OC and EP curves.

Features of Program OCPLOT

Program OCPLOT is designed to analyze the types of acceptance procedures typically used in the highway field. This includes pass/fail procedures, leading to the type of OC curve shown in figure M­1, and payment adjustment procedures, leading to the type of EP curve shown in figure M­2.

Figure M­3 lists some of the options that may be selected from the primary menu. The various items appear on the screen one at a time in a logical sequence, and later items are dependent upon the responses to earlier ones. The versatility of the program is apparent from the many different ways these selections might be combined.

When the selections from the first menu are complete, the menu will appear similar to that shown in figure M­4. The prompt "Press any key to continue" at the bottom of the display provides a pause that gives the user the opportunity to use the <ESC> key to go back and change some values or press the <PrintScreen> key to save the screen to a file if a record of the menu selections is desired. Striking almost any other key will cause the second menu in figure M­4 to appear.


Figure M-1 Diagram. click here for more detail

Figure M­1. Conventional OC Curve for Pass/Fail Acceptance Procedure


Diagram M-2 Click here for more details.

Figure M­2. Typical EP Curve for Acceptance Procedure with
Adjusted Payment Schedule

 

ACCEPTANCE METHOD
Pass/Fail TYPE OF PLAN
Pay Adjustment Attributes
Variables
   
QUALITY MEASURE
Percent Defective (PD)  
Percent Within Limits (PWL)  
   
LIMIT TYPE
Single­Sided  
Double­ Sided  
   
PAY EQUATION TYPE
Linear/Nonlinear Enter Values
   
MAXIMUM PAY FACTOR
Yes Enter Values
No  
   
ACCEPTABLE QUALITY LEVEL (AQL)
Enter Value  
   
REJECTABLE QUALITY LEVEL (RQL)
Enter Value  
   
RQL PROVISION
Yes Enter RQL Payment Factor
No  
   
RETEST PROVISION
Yes INITIAL TESTS
No Combined
Discarded
   
SAMPLE SIZE
Enter Value(s)  
   
Figure M­3. Selections Available in Program OCPLOT

First Menu:

ENTER THE FOLLOWING INFORMATION

ACCEPTANCE METHOD
Pay Adjustment

ACCEPTABLE QUALITY LEVEL
PD = 10
QUALITY MEASURE
Percent Defective
REJECTABLE QUALITY LEVEL
PD = 50
LIMIT TYPE
Double-Sided
RQL PAY FACTOR
PF = 70
PAY EQUATION
PF = 102 - .2 PD
RETEST PROVISION
None
MAXIMUM PAY FACTOR
PF = 102.0
SAMPLE SIZE
10
Press any key to continue

<ESC> = Back

<END> = Exit

Second Menu:

SELECT LEVEL OF PRECISION

(1) Low ­ Faster Execution

(2) Intermediate

(3) High ­ Slower Execution

SELECTION

<ESC> = Back <END> = Exit

Figure M­4. First and Second Menus for Program OCPLOT



Because program OCPLOT uses computer simulation to analyze whatever acceptance procedure has been specified, it is very computationally intensive and the execution speed is dependent upon the level of precision selected from the second menu. Table M­1 lists the number of replications performed for the different levels of precision.

Table M­1. Program OCPLOT Precision Levels

Precision Level

Number of Replications

1

200

2

1,000

3

5,000



Selection 1 provides the fastest execution, which is useful for exploratory work but may not be good enough to report as a final result. When this level is selected, 200 sample sets of the desired size are randomly generated from a normal population for each of several known levels of quality.

Each sample set is evaluated in accordance with the acceptance plan specified in the primary menu and the results are stored in memory for subsequent analysis. This is far more thorough and many times faster than testing the acceptance procedure with a field trial. (A field trial would be appropriate only if the procedure survives this initial check.)

Selection 2 from the second menu provides an intermediate level of precision for which 1000 sample sets are generated at each quality level. This level of precision is usually satisfactory to report as a final result, producing points for the OC curve, representing probability of acceptance, or the EP curve, representing the expected payment factor, that are typically accurate to within one or two units. If still better precision is desired, selection 3 will cause 5000 sample sets to be generated at each quality level. This level of precision tends to produce a smooth line when the OC or EP curve is plotted.

Once the precision level is selected from the second menu, the computational process begins. For either low or intermediate precision, program OCPLOT displays detailed information at the two key points at which risk levels are usually expressed­the AQL and the RQL­as shown in figures M­5 and M­6. This serves two important purposes. For users less familiar with statistical estimation procedures and acceptance plans, the graphical displays at the AQL and RQL are both informative and educational. It may come as a surprise to some, for example, how widely distributed the quality estimates are, especially for small sample sizes. For users more familiar with statistical acceptance procedures, these displays provide assurance that the simulation process is working properly. The actual displays on a color monitor are color­coded to clearly distinguish acceptable and rejectable results and the corresponding payment factors.


figure M-5
 Figure M­5. Typical Display at AQL at Intermediate Precision by
Program OCPLOT


figure M-6
Figure M­6. Typical Display at RQL at Intermediate Precision by Program OCPLOT


Although the AQL and the RQL are probably the most important points at which it is desirable to know how the acceptance procedure will perform, it usually is useful to have a plot of the entire OC or EP curve that provides a picture of the performance over the complete range of quality that might be encountered. The prompt at the bottom of the screen instructs the user to strike any key to continue with this step to obtain the display shown in figure M­7. The horizontal and vertical axes and the two previously calculated points at the AQL and the RQL appear on the screen immediately. The remaining points appear one at a time at a speed determined by the level of precision that has been selected and the speed of the machine running the program.

After all the points have been calculated and plotted, the user may strike any key to connect the points with a solid line. Following this, the next key stroke will add vertical and horizontal lines highlighting the performance of the acceptance plan at the AQL and RQL, as shown in figure M­8. And, like the histograms in figures M­5 and M­6, any of these displays may be saved to a file by using the <PrintScreen> key.

Following this display, striking a key will produce the menu shown in figure M­9. If the first item in this menu is selected, the output shown in figure M­10 is displayed. This permits the user to save the values of the data points shown in figure M­7 from which the OC curve was constructed. The other selections in this menu make it possible to return to earlier points in the input stage of the program or to exit the program.

Example M­1: Pass/Fail Attributes Procedure

Attributes acceptance procedures are based on measures that are counted rather than computed, such as the number of defects on an item of production or the number of test results falling outside specification limits. In contrast, variables acceptance procedures are based on statistical parameters that are computed, such as the mean and standard deviation, and lead to the estimation of PD or PWL.

Advantages of attributes procedures are that they are simple to apply and they require no assumptions about the underlying distributional form of the population being sampled. For example, a typical attributes procedure might require that a sample of size N = l0 be taken and that no more than C = 2 test results may be outside the specification limits, where "C" is referred to as the acceptance number. A disadvantage is that they require larger sample sizes to achieve the equivalent discriminating power of a variables procedure. Variables procedures, because of their inherently greater efficiency, are generally preferred whenever the requirement for a normally­distributed population is reasonably satisfied.


figure M-7 click her form more detail.

FigureM­7. Points on EP Curve Plotted by Program OCPLOT

Figure M-8 Click here for more details.
Figure M­8. Display of EP Curve Plotted by Program OCPLOT with AQL and RQL Performance Highlighted



SELECT DESIRED OPTION

(1) Display operating characteristic table

(2) Select precision level and run again

(3) Change some values and run again

(4) Run again with new input data

(5) Exit program

SELECTION

<ESC> = Back <END> = Exit

Figure M­9. Third Menu for Program OCPLOT

figure M-9. Third Menu for Program OCPLOT

Figure M­10. Display of Numerical Values of Points on EP Curve Computed
at High Precision by Program OCPLOT


Although the vast majority of highway construction measures tend to be normally distributed, there are some that are not. A physical constraint close to the desired target value often produces non­normality. For example, depth of cover over the top mat of reinforcing steel in a bridge deck may tend to be skewed because it is physically impossible (provided the steel is embedded) for the cover to be less than zero but there is less of a restriction on how deep the mat might be. Similarly, if the target level for percent air voids in bituminous pavement is fairly low, the physical limit of zero may tend to skew this distribution toward larger values. Another condition that can produce non­normality is the combining in the same lot of two distinctly different populations. As a general rule, a conscious effort should be made to avoid this condition when applying a statistical specification.

Because situations may arise in highway construction that warrant their use, program OCPLOT provides the capability of analyzing attributes acceptance procedures. Although it would be possible to develop a payment schedule based on an attributes procedure, their use has been limited almost exclusively to pass/fail applications. The following is presented as a generic example of such a procedure.

It is assumed for this example that the statistical quality measure is the percent defective (PD), the percentage of the lot falling outside specification limits. The acceptable quality level (AQL) and the rejectable quality level (RQL) are defined as PD = 10 and PD = 50, respectively. It is desired to develop an acceptance procedure that will balance the risks at 0.05. In other words, if the contractor provides work that is exactly at the AQL, there is to be a 0.05 chance that it will erroneously be rejected and, at the other extreme, if the work is truly at the RQL, there is to be a 0.05 chance that it will erroneously be accepted.

An acceptance plan is required, stated in terms of the sample size (N) and the acceptance number (C) that will produce the desired risks. For both risks to be 0.05, an OC curve is desired that indicates probabilities of acceptance at the AQL and the RQL of 0.95 and 0.05, respectively. Because both N and C must be integer values, the resultant risk levels vary in discrete steps, and it usually is not possible to obtain exactly the desired risk values. To find a plan that matches the desired risk levels as closely as possible, it is necessary to examine several plans.

This can easily be accomplished with program OCPLOT by selecting "Pass/Fail" and "Attributes" from the opening menu, followed by other appropriate selections and ending with a trial combination of N and C. It is usually advantageous to make the first few runs at low precision (selected from the second menu) in order to speed up the trial"and"error process.

Figure M­11 shows the completed menu for the initial try with N = 10 and C = 2 and figure
M­12 shows the resulting acceptance probabilities obtained at high precision. It can be seen in figure M­12 that the probability of acceptance of 0.052 at the RQL is close to the desired value, while the probability of acceptance of 0.927 at the AQL is too low.


ENTER THE FOLLOWING INFORMATION

ACCEPTANCE METHOD REJECTABLE QUALITY LEVEL

Pass/Fail PD = 50

TYPE OF PROCUDURE RETEST PROVISION

Attributes None

QUALITY MEASURE SAMPLE SIZE

Percent Defective 10

LIMIT TYPE ACCEPTANCE NUMBER

Single­Sided 2

ACCEPTABLE QUALITY LEVEL

PD = 10

Press any key to continue

<ESC> = Back <END> = Exit

Figure M­11. Completed Menu for Analysis of Pass/Fail Attributes
Acceptance Plan with N = 10 and C = 2

Figure M-12. Numerical Values of Points for Pass/Fail Attributes Acceptance Plan

Figure M­12. Numerical Values of Points on EP Curve for Pass/Fail Attributes Acceptance Plan with N = 10 and C = 2


The results of several attempts are presented in table M­2. The values in this table were all obtained at high precision and here it can be seen that the plan having N = 13 and C = 3 comes the closest to meeting the desired acceptance probabilities of Prob. ³ 0.95 and Prob. £ 0.05 at the AQL and RQL, respectively. This is a trial and error process, but, with a little experience, it usually is possible to find a suitable plan relatively quickly.

Table M­2. Performance of Attributes Acceptance Plans

Percent Defective (PD)

Probability of Acceptance

N = 10

C = 2

N = 12

C = 3

N = 13

C = 3

N = 14

C = 3

0

1.000

1.000

1.000

1.000

10 (AQL)

0.927

0.975

0.966

0.960

20

0.681

0.810

0.747

0.698

30

0.380

0.491

0.430

0.350

40

0.167

0.226

0.163

0.130

50 (RQL)

0.052

0.079

0.048

0.028

60

0.013

0.014

0.006

0.011

The attributes acceptance plan meeting the requirements of this example requires that a total of N = 13 test specimens be taken from random locations within each lot and that, after the appropriate tests have been performed, no more than C = 3 of the test results shall be outside specification limits. The use of such a plan ensures that truly AQL work will be accepted about 95 percent of the time and that truly RQL work will be accepted only about 5 percent of the time.

Example M­2: Pass/Fail Variables Procedure

Variables procedures are based on the assumption that the population being sampled is at least approximately normally distributed. They involve the computation of statistical parameters, such as the mean and standard deviation, to estimate PD or PWL. Provided the normality assumption is sufficiently well satisfied, variables procedures can provide essentially the same discriminating power as equivalent attributes plans, but with a substantially smaller sample size.

To illustrate this last statement, a variables procedure is sought that will have essentially the same discriminating power (OC curve) as the attributes procedure developed in example M­1. Like the previous example, this involves a trial and error process with which program OCPLOT can be extremely helpful. For this example, the selections "Pass/Fail" and "Variables" are made from the opening menu, followed by other appropriate selections and ending with trial values for the sample size and acceptance limit.

This trial­and­error process proceeds more quickly if a low level of precision is selected for the initial attempts. When it appears that the appropriate combination of sample size and acceptance limit has been found, this should be checked at intermediate precision. Further minor adjustments may then be necessary before confirming the result at high precision.

Figure M­13 shows the completed menu for the variables acceptance plan that was ultimately selected. The actual numerical values on the OC curve are shown in table M­3, which summarizes the results of examples M­1 and M­2. In this table it can be seen that the performance of the variables plan very closely matches that of the attributes plan in example M­1 (table M­2, N = 13, C = 3). The dramatic difference is that this has been accomplished with a sample size of N = 8, whereas the attributes plan required a sample size of N = 13. In this case, which assumes that the normality assumption is satisfied, the use of a variables plan results in a direct savings in sampling and testing costs of nearly 40 percent.

ENTER THE FOLLOWING INFORMATION

ACCEPTANCE METHOD REJECTABLE QUALITY LEVEL

Pass/Fail PD = 50

TYPE OF PROCUDURE RETEST PROVISION

Variables None

QUALITY MEASURE SAMPLE SIZE

Percent Defective 8

LIMIT TYPE ACCEPTANCE LIMIT

Single­Sided PD = 26

ACCEPTABLE QUALITY LEVEL

PD = 10

Press any key to continue

<ESC> = Back <END> = Exit

Figure M­13. Completed Menu for Pass/Fail Variables Acceptance Plan Example


Table M­3. Comparison of Operating Characteristics of Attributes
and Variables Acceptance Plans

Percent Defective

Probability of Acceptance

Attributes Plan
(N =13, C= 3)

Variables Plan
(N = 8,
Acc Limit = 26 PD)

0

1.000

1.000

5

0.997

0.996

10 (AQL)

0.966

0.954

15

0.884

0.841

20

0.744

0.693

25

0.596

0.544

30

0.430

0.376

35

0.279

0.256

40

0.163

0.157

45

0.091

0.088

50 (RQL)

0.048

0.048

55

0.024

0.024

60

0.006

0.009

Example M­3: Analysis of Payment Equation

Although the pass/fail acceptance procedures discussed in examples M­1 and M­2 may be useful for highway construction applications, the use of acceptance procedures with adjusted payment schedules is generally of greater interest. The proper design of such plans is critical to their performance, and poorly conceived plans may be either totally ineffective or impractically severe. Neither problem may be apparent, however, until the plan has been analyzed by constructing the EP curve.

Equation M­1 gives a payment schedule proposed for use by an owner. The AQL was defined as PWLSPEC = 90 and, since there was no incentive payment provision, the maximum payment factor was limited to 100 percent.

PR = PWL SPEC PWLCOMP

(M­1)

where: PR = payment reduction (percent).

PWLSPEC = specified PWL at the AQL.

PWLCOMP = PWL computed from test values.

To transform this equation into a form that can be handled by program OCPLOT, it is necessary to express it in terms of the payment factor rather than the payment reduction. By substituting PWLSPEC = 90 and PR = 100 ­ PF into equation M­1, equation M­2 is obtained, subject to the restriction that PF £ 100.

PF = 10 + PWLCOMP

(M­2)

where: PF = payment factor (percent)

PWLCOMP = PWL computed from test values.

To judge the effectiveness of this payment equation, the EP curve will be developed for a typical sample size of N = 5. The completed input menu is shown in figure M­14. Following this, an intermediate level of precision was selected from the second menu to obtain the display at the AQL shown in figure M­15. Finally, the program was run at high precision, which produced the displays shown in figures M­16 and M­17.

ENTER THE FOLLOWING INFORMATION

ACCEPTANCE METHOD ACCEPTABLE QUALITY LEVEL

Pay Adjustment PWL = 90

QUALITY MEASURE REJECTABLE QUALITY LEVEL

Percent Within Limits PWL = 50

LIMIT TYPE RQL PROVISION

Single­Sided None

PAY EQUATION RETEST PROVISION

PF = 10 + 1 PWL None

MAXIMUM PAY FACTOR SAMPLE SIZE

PF = 100 5

Press any key to continue

<ESC> = Back <END> = Exit

Figure M­14. Completed Menu for Analysis of Payment Equation


equation M15. click here for more detail
Figure M­15. Performance of Payment Equation at AQL

equation M16 - click here for more detail.
Figure M­16. EP Curve for Analysis of Payment Equation


equation M17 - click here for more detail.
Figure M­17. Numerical Values for Points on EP Curve for
Analysis of Payment Equation

It can be seen in figure M­15 that there is a serious problem with this acceptance procedure. A contractor who performs consistently at the AQL will receive an average payment reduction of nearly 5 percent. To emphasize the impact this would have on the construction industry, this means that a contractor responsible for $10 million worth of work under this specification over the course of a construction season would on the average be penalized approximately $500,000 for successfully providing the level of quality that was explicitly defined as acceptable in the contract documents.

This example illustrates the situation discussed previously, whereby the failure to include an incentive payment provision with this type of specification prevents the acceptance procedure from paying an average of 100 percent when the work is exactly at the AQL. The reason for this can be seen with the help of figure M­15. The upper histogram in this figure represents the distribution of 1000 PWL estimates, each obtained by randomly sampling a population that is exactly of AQL quality. These sample PWL estimates range from a low of about 47 percent to a high of 100 percent because of the inherent variability of the sampling process. However, the long­term average of these estimates will always be extremely close to the true value (the AQL in this case) because the PD/PWL estimation process is an unbiased statistical estimation procedure. The problem arises because the payment schedule does not permit this unbiased property to extend to the distribution of payment factors. Because there is no incentive payment provision, all PWL estimates that are greater than the true PWL of 90 receive the maximum payment factor of 100 percent, while all those below the true value receive some degree of payment reduction. Since approximately half the lots will receive payment reductions and the other half will receive 100 percent payment, the net result is that the average payment factor for AQL work will be substantially lower than 100 percent.

This situation is clearly misleading and unfair, yet it exists in current QA specifications. It is not difficult to correct, however, as demonstrated in the next example.

Example M­4: Effect of Incentive Payment Provision

The problem described in the preceding example, in which truly AQL work receives a substantial payment reduction, is easy to correct. All that is required is the inclusion of an incentive payment provision as part of the acceptance procedure. In equation M­2, this would mean removing the restriction that the maximum payment factor cannot exceed 100 percent.

Ordinarily, the maximum payment factor and the slope of the payment equation should be consistent with established (or estimated) performance relationships and the anticipated economic consequences of any departures from the specified AQL. To be consistent with this example, a maximum incentive payment factor of 110 percent will have to be used. In actual practice, however, most agencies have used incentive payment provisions of 105 percent or less.

To confirm that this will solve the problem, program OCPLOT was run an additional time with the identical input used for example M­3 except that when the prompt "MAXIMUM PAY FAC ­ TOR" appeared, selection 2 was chosen, indicating that no upper limit is placed on the payment equation. The program then automatically computed and displayed the maximum payment factor of PF = 110.0, as shown in the completed menu in figure M­18. The performance at the AQL is displayed in figure M­19, where it is seen that the expected payment factor is 100 percent, as desired.


ENTER THE FOLLOWING INFORMATION

ACCEPTANCE METHOD ACCEPTABLE QUALITY LEVEL

Pay Adjustment PWL = 90

QUALITY MEASURE REJECTABLE QUALITY LEVEL

Percent Within Limits PWL = 50

LIMIT TYPE RQL PROVISION

Single-Sided None

PAY EQUATION RETEST PROVISION

PF = 10 + 1 PWL None

MAXIMUM PAY FACTOR SAMPLE SIZE

PF = 110.0 5

Press any key to continue

<ESC> = Back <END> = Exit

Figure M­18. Completed Menu for Analysis of Acceptance Procedure with Incentive Payment Provision


Figure M-19 Click here for more details
Figure M­19. Performance at AQL of Acceptance Procedure with
Inventive Payment Provision

Example M­5: OC Curves for Payment Adjustment Plans

It was shown in chapter 7 that multiple OC curves could be plotted to illustrate the performance of acceptance plans with payment adjustment provisions. Each of these curves is associated not with the probability of acceptance, but with the probability of receiving greater than or equal to a given payment level. For example, suppose that equation M­3 is selected to determine the payment for a lot based on a sample of size N = 5.

PF = 55 + (0.5 x PWL) (M­3)

From this equation it can be seen that full payment, i.e., PF = 100 percent, is obtained for a computed lot PWL of 90. Lot PWL values greater than 90 will lead to an incentive payment, while lot PWL values less than 90 will lead to a disincentive payment.

The probability of receiving a payment greater than or equal to 100 percent therefore is equivalent to the probability of receiving an individual estimated lot PWL of 90 or greater. By using the Pass/Fail option and selecting 90 as the ACCEPTANCE LIMIT, the OCPLOT program can be used to determine this OC curve. The completed OCPLOT menu for this situation is shown in figure M­20, and the corresponding OC curve is shown in figure M­21.

ACCEPTANCE METHOD REJECTABLE QUALITY LEVEL
Pass/Fail PWL=50
   
TYPE OF PROCEDURE RETEST PROVISION
Variables None
   
QUALITY MEASURE SAMPLE SIZE
Percent Within Limits 5
   
LIMIT TYPE ACCEPTANCE LIMIT
Single-Sided PWL=90
   
ACCEPTABLE QUALITY LEVEL  
PWL=90  
Press any key to continue
<ESC> = Back
<END> = Exit

Figure M­20. Completed Menu for Using OCPLOT to Determine the Probability of Receiving Greater than or Equal to 100 Percent Payment for Example M­5

Similarly, the probability of receiving greater than or equal to 95 percent payment is equivalent to the probability of receiving an individual estimated lot PWL of 80 or greater (obtained by plugging PF = 90 into equation M­3 and solving for PWL). OCPLOT can once again be used with the Pass/Fail option and with 80 as the ACCEPTANCE LIMIT to determine this OC curve. This can be repeated for other payment levels to develop the family of OC curves shown in figure M­22.

fig M21. Click here for more detail.
Figure M­21. OC Curve for the Probability of Receiving Greater than or Equal
to 100 Percent Payment for Example M­5

figure M22. Click here for more detail.

Figure M­22. Family of OC Curves Showing the Performance of the Acceptance
Plan Using a Sample of Size N = 5 and Payment Equation M­3

 

Previous | Table of Contents | Next

Federal Highway Administration | 1200 New Jersey Avenue, SE | Washington, DC 20590 | 202-366-4000
Turner-Fairbank Highway Research Center | 6300 Georgetown Pike | McLean, VA | 22101