U.S. Department of Transportation
Federal Highway Administration
1200 New Jersey Avenue, SE
Washington, DC 20590
202-366-4000


Skip to content
Facebook iconYouTube iconTwitter iconFlickr iconLinkedInInstagram

Federal Highway Administration Research and Technology
Coordinating, Developing, and Delivering Highway Transportation Innovations

Report
This report is an archived publication and may contain dated technical, contact, and link information
Publication Number: FHWA-RD-02-095

Chapter 7. Evaluating Risks

Previous | Table of Contents | Next

This chapter continues the discussion of Phase II of the specification development process. This chapter is intended to provide "how to use" best practices in evaluating the risks associated with the initial acceptance procedures that have been developed up to this point. The steps that are involved in this part of the process are identified in the flowchart in figure 25. The numbers in boxes before the titles of the following sections refer to the corresponding box in the flowchart.


39 - Develop OC Curves and Evaluate Risks

Establishing the limits to be used for acceptance is an important step. Making the limits too restrictive deprives the contractor of a reasonable opportunity to meet the specification. Making them not sufficiently restrictive makes them ineffective in controlling quality. Selection of the limits relates to the determination of risks. The concept of risks for acceptance is similar to that discussed in chapter 5 for verification testing to evaluate whether test results came from the same population. The two types of risk discussed in chapter 5 are the seller's (or contractor's) risk, a, and the buyer's (or agency's) risk, b. The a risk is also called a Type I risk, and the b risk is also called a Type II risk. A well-written QA acceptance plan takes these risks into consideration in a manner that is fair to both the contractor and the agency. Too large a risk for either party undermines credibility.

39.1. Risks: Definitions and Concepts

39.1.1. Risks. Before proceeding further, some terms need to be formally defined. The TRB glossary (2) includes the following definitions:

The a and b risk levels that might be appropriate vary depending upon the material or construction process that is involved. The appropriate risk level is a subjective decision that can vary from agency-to-agency. In reality, it is likely that few agencies have developed and evaluated the risk levels associated with their acceptance plans. While risk levels are an agency decision, AASHTO R-9, "Acceptance Sampling Plans for Highway Construction," suggests the risk levels indicated in table 21. (22) It should be noted that large sample sizes, on the order of 10 to 20 or more, may be required to achieve some of the risk levels stipulated in this table.

 

Risk Analysis Flowchart. Click here for more detail
Figure 25. Flowchart for Risk Analysis Portion of Phase II

 

Table 21. Risk Levels Suggested in AASHTO R-9 (22)

Criticality 1

Recommended a

Recommended b

Critical0.0500.005
Major0.0100.050
Minor0.0050.100
Contractual0.0010.200

1Critical: when the requirement is essential to preservation of life.
Major: when the requirement is necessary for the prevention of substantial financial loss.

Minor: when the requirement does not materially affect performance.
Contractual: when the requirement is established only to provide uniform standards for bidding.

As noted in the section on verification testing in chapter 5, the concept of a and b risks derives from statistical hypothesis testing where there is either a right or wrong decision. As such, when a and b risks are applied to materials or construction they are only truly appropriate for the case of a pass/fail or accept/reject decision and, in fact, may lead to considerable confusion if an attempt is made to apply them to the payment adjustment case. When materials not only can be accepted or rejected, but can also be accepted at an adjusted payment, then additional interpretations or clarifications must be applied to the definitions for these risks in an effort to manipulate them to apply to the payment adjustment situation.

For example, in the definition for buyer's risk above, it states that b is the probability that RQL material will be accepted at 100 percent payment or greater. The definition must then go on to point out that there is a much greater probability that the RQL material will receive some reduced payment. While it is not stated as directly, the same reasoning is true for the seller's risk. The definition indicates that a is the probability that AQL material will be rejected. Although not stated in the definition, it is also true that there is a much greater probability that the AQL material will be accepted at a reduced payment.

39.1.2. OC Curves. The buyer's and seller's risks are very narrowly defined to occur at only two specific quality levels. The buyer's risk is the probability of accepting material that is exactly at the RQL level of quality, while the seller's risk is the probability of rejecting material that is exactly at the AQL level of quality. These definitions do not therefore provide a very good indication of the risks over a wide range of possible quality levels. To evaluate how the acceptance plan will actually perform in practice, it is necessary to construct an OC curve. The TRB glossary (2) includes the following definition:

An example of an OC curve for a pass/fail or accept/reject acceptance plan, case (a) in the above definition, is shown in figure 26. Probability of acceptance is shown on the vertical axis for the range of quality levels indicated on the horizontal axis. An example of an OC curve for an acceptance plan with payment adjustment provisions, case (b) in the above definition, is shown in figure 27. The axes are the same as for figure 26, but there are multiple curves, one for each of several selected payment levels, plotted.

Each curve plotted in figure 27 represents the probability of receiving a payment factor equal to or greater than the one indicated for the line. For example, for the OC curves in figure 27, material that is of exactly AQL quality has approximately a 45 percent chance of receiving a payment factor of 1.04 (104 percent) or greater. This same AQL material has approximately a 55 percent chance of receiving full payment (100 percent) or greater, which also means that it has approximately a 45 percent chance of receiving less than 100 percent payment. This AQL material has essentially a 100 percent chance of receiving a payment factor of 0.80 (80 percent) or greater.

 

Level of Quality Graph. Click here for more detail.
Figure 26. Typical OC Curve for an Accept/Reject Acceptance Plan

 

Text Box: Prob. of Receiving greater than or equal or equal to PFClick here for alternative text
Figure 27. Typical OC Curves for an Acceptance Plan with Payment Adjustments

On the other hand, for the OC curves in figure 27, material that is of exactly RQL quality has approximately a 30 percent chance of receiving a payment factor of 0.80 (80 percent) or greater, and nearly an 80 percent chance of receiving a payment factor of 0.70 (70 percent) or greater. Similar payment probabilities can be determined for any level of actual quality, and additional curves could be developed for any specific value of payment factor.

39.1.3. Expected Payment Curves. Figure 27 clearly shows that consideration of onlya and brisks is clearly not sufficient when payment adjustments are used. From figure 27 it can also be seen that using multiple OC curves is not an easy way to evaluate an acceptance plan. It would be convenient to have a single curve that can represent the operation of the plan as opposed to many different curves for each plan. Another way to present the payment performance for an acceptance plan is with what is call an expected payment (EP) curve. The TRB glossary (2) includes the following definition:

An example of an EP curve is shown in figure 28. Quality levels are indicated on the horizontal axis in the usual manner, but instead of probability of acceptance, the vertical axis gives the expected (long-term average) payment factor as a percent of the contract price.

Although the risks have a different interpretation when associated with EP curves than with OC curves, the same type of information is provided. For the example in figure 28, AQL work receives an expected payment of 100 percent, as desired, while truly superior work that is better than the AQL receives an expected payment of 102 percent. At the other extreme, RQL work corresponds to an expected payment of 70 percent. For still lower levels of quality, the curve levels off at a minimum expected payment of 50 percent.

 

EP Curve Graph. Click here for more detail.
Figure 28. Typical EP Curve


Simplified Example: a and b Risks and an OC Curve

A simplified example of how risks are related to specification limits can be illustrated by considering primitive acceptance plans that were based on measuring and accepting a property based on only one test. Suppose that an accept/reject acceptance plan for asphalt content has been developed based on the definitions for AQL material and RQL material that follow.

Define AQL Material. It is assumed that asphalt content follows a normal distribution. It has been determined that for asphalt content, acceptable material has a standard deviation of about 0.20 percent when the mean is close to the target JMF value. If the JMF has established the target as 6.0 percent asphalt content, the AQL is therefore a lot (population) with a mean of 6.0 percent and a standard deviation of 0.20 percent. Figure 29 shows an AQL population.

Define RQL Material. Additionally, unacceptable material might be defined as that for which the mean differs from the target value by 0.4 percent or more, as long as the standard deviation does not exceed 0.20 percent. (Other definitions would be equally valid.) The RQL is therefore a lot (population) with a standard deviation of 0.20 percent and a mean of 5.6 percent or lower, or 6.4 percent or higher. Examples of RQL populations are shown in figure 30.

Determine a Risk. Suppose the agency has established the specification limits, i.e., the limits within which individual asphalt content results must fall, as the JMF ± 0.40. For a JMF target value of 6.0 percent, this establishes the specification limits as 5.60 percent and 6.40 percent. An AQL population is shown along with the specification limits in figure 31. The a risk is the probability that a single test result from this AQL lot would be outside of the allowable specification range of 5.60 percent to 6.40 percent. This is the a risk to the contractor because if a test falls outside these limits the agency will erroneously reject the material. The Z-statistics can be calculated and used in conjunction with the standard normal distribution table (table 7) to determine this probability to be 0.0456 or 4.56 percent.

Determine b Risk. Figure 32 shows an RQL population with its mean at 5.60 percent and standard deviation of 0.20 percent. The RQL population could also have its mean at 6.40 percent. The RQL population can either be too low or too high, but not both at the same time. The b risk is the probability that a single test result from this RQL lot would be within of the allowable specification range of 5.60 percent to 6.40 percent. This is the risk to the agency because if a test result falls in this range, the agency will erroneously accept the material. From figure 32 this probability can be seen to be 0.50 or 50 percent.

Develop the OC Curve. Similarly, the probabilities of acceptance for lots with means of any value, e.g., 5.20 percent, 5.40 percent, 5.60 percent, etc., can be calculated and plotted to form the OC curve shown in figure 33. The AQL and RQL are also noted on the figure. It should be noted that it is purely coincidental that the OC curve in figure 33 has the appearance of a normal curve.

 

AQL Population Curve. Click here for more detail.
Figure 29. AQL Population for Simplifieda and b Risks Example


 

RQL Populations Curve
Figure 30. RQL Populations for Simplifieda and bRisks Example


 

Figure 21: Illustration of the a Risk
Figure 31. Illustration of the a Risk for Simplified Example

 

Illustration of the b Risk
Figure 32. Illustration of the b Risk for Simplified Example

 

Diagram - OC Curve for Simplified Alpha and Beta Risk. Click here for additional information
Figure 33. OC Curve for Simplified a and b Risks Example

39.2. OC Curves for PWL or PD Acceptance Plans

As with any acceptance plan that bases the acceptance decision on a sample, there are risks associated with PWL or PD acceptance plans. The above example demonstrated the calculation of risks for a simple acceptance plan based on an assumed known standard deviation, and with the acceptance decision based on only a single test result. The risks associated with PWL or PD acceptance plans cannot be calculated so easily.

For PWL or PD acceptance plans the risks are almost always determined by means of computer simulation. It is, however, possible to illustrate the risks associated with using a sample to estimate PWL by means of a simplified attributes example.


Simplified Example

Assume that we have a bag that has 100 marbles. Further assume that the bag has 70 white marbles and 30 blue marbles. Also assume that we wish to take a sample of 10 marbles to estimate the percentage of the marbles in the bag that are blue.

It is easy to estimate the percentage of blue marbles from a sample of 10 marbles. However, each sample of 10 marbles will not yield the same percentage of blue marbles. The first sample of 10 marbles might contain 3 blue marbles, thereby yielding an estimate of 30 percent blue marbles. However, it could also have only one blue marble, or five blue marbles. In each of these cases the estimate from the sample will be fairly far from the true value of 30 percent.

The histogram in figure 34 shows the results of 100 samples, each with 10 marbles. While the individual sample results could be quite far from the actual percentage in the population, the average of the 100 samples is quite close to the true population value. Also, most of the sample values are close to the actual population percentage, with fewer values as the estimate becomes farther from the actual population percentage. Although simplified, this example clearly shows how the PWL values estimated from samples can vary. The long-run average of the sample averages will tend to equal the true population PWL value, but there is a risk that any individual estimate may either over-estimate or underestimate the true population PWL value.


 

Histogram of PWL Estimates. Click here for more detail.
Figure 34. Histogram of PWL Estimates for Simplified PWL Example

39.2.1. Computer Simulation. As noted above, calculating the risks for actual PWL acceptance plans is much more involved than the simplified example from figure 34. In fact, computer simulation is almost always used to developa and brisks, as well as OC and EP curves. OCPLOT, a user-friendly program that develops OC and EP curves by computer simulation, was developed as part of FHWA Demonstration Project No. 89. This program is explained in detail in the report for that project, (18) and is also presented in appendix M along with some examples.

OCPLOT can be used to develop OC curves for accept/reject acceptance plans. It can also be used with a stipulated payment equation to determine the probability of receiving a lot payment factor greater than or equal to any specific value. In this way, it can be used to plot multiple OC curves similar to those in figure 27. The program can also develop EP curves for a given payment equation. The program is also capable of simulating acceptance plans containing retest provisions.

39.3. Evaluating the Risks

39.3.1. Accept/Reject Acceptance Plans. How potential risks are evaluated depends upon the type of acceptance plan that is used. The evaluation of risks is rather straightforward for accept/reject (pass/fail) acceptance plans. As noted above,a and brisks and OC curves were developed specifically for this type of situation. Therefore, they can be directly used to assess the risks to both parties.

To reiterate, the a risk is the probability that AQL material will be rejected; while the b risk is the probability that RQL material will be accepted. However, since contractors will not operate at only these two quality levels, to fully consider risks the OC curve, which illustrates probability of acceptance for any quality level, must be developed for the acceptance plan under consideration. An example will help to illustrate how this can be done.


Example: Accept/Reject Acceptance Plans-OC Curves

The previously discussed OCPLOT program can be used to determine thea and brisks and to plot the OC curve for a sample acceptance plan. Suppose that an agency decides to use asphalt content as an accept/reject property for an HMAC pavement (note, this is not recommended, but is used here solely for the purpose of illustrating the use of an OC curve for an accept/reject situation). Further suppose that the agency has established for asphalt content a lower specification limit of 5.60 percent and an upper specification limit of 6.40 percent. The agency has decided to use the PWL, based on a sample of size 4, as the quality measure. The agency has selected 90 PWL for the AQL and 50 PWL for the RQL. The lot will be accepted if the estimated PWL is greater than or equal to 70. Table 22 and figure 35 show the results of the OCPLOT analysis of this proposed acceptance plan.

From table 22 it can be seen that the seller's risk is a = 1.000 - 0.905 = 0.095 (or 9.5 percent) and the buyer's risk is b = 0.144 (or 14.4 percent). Further, both table 22 and figure 35 show the probability of acceptance over the total range of possible lot quality levels, as defined by the actual PWL for the lot. The agency would need to decide whether or not it considers these levels of risk to be appropriate.


39.3.2. Payment Adjustment Acceptance Plans. The evaluation of risks becomes much more complicated when the acceptance plan includes payment adjustment provisions. The concepts of a and b risks, which were developed from hypothesis testing where there is a yes or no decision, i.e., reject or fail to reject the null hypothesis, are not sufficient when the decision involves not only accept or reject, but also accept at an adjusted payment level.

Table 22. OC Table from OCPLOT for the Example Problem

Population PWL

Probability of Acceptance
100 1.000
95 0.976
90 (AQL) 0.905 (a = 0.095)
85 0.810
800.696
75 0.579
70 0.466
65 0.363
60 0.288
55 0.200
50 (RQL) 0.144 (b = 0.144)
45 0.093
40 0.066
35 0.038
30 0.021
25 0.013
20 0.000

 

 

Line Graph: OC Curve from OCPLOT. Click here for more detail.
Figure 35. OC Curve from OCPLOT for the Accept/Reject Acceptance Plan Example

The TRB glossary (2) definitions for seller's and buyer's risks that are presented above do not attempt to incorporate the concept of payment adjustments. The seller's risk is defined as the probability that an acceptance plan will erroneously reject AQL material or construction. This disregards the fact that the material or construction can be accepted at full payment, increased payment, or decreased payment. In other words, whether or not a lot received 105 percent, 100 percent, 75 percent, or 50 percent payment would have no impact with regard to the seller's risk based on this definition. Obviously, however, these different payment levels would have quite an impact on how the contractor perceived its risks.

Similarly, the buyer's risk is defined as the probability that an acceptance plan will erroneously fully accept (100 percent or greater) RQL material or construction. Once again, this definition disregards the impact of partial payments when determining the buyer's risk. However, when considering its risks the agency will certainly be interested in the probability of accepting RQL material at reduced payment levels as well as at 100 percent payment or greater.

The use ofa and brisks to evaluate payment adjustment acceptance plans is simply not sufficient. Some additional method or methods is/are necessary to properly evaluate the risks when payment adjustments are added to the acceptance decision options. The expected payment, or EP, (see figure 28) is another method for considering the payment adjustment aspects of the acceptance plan. However, EP alone is also not sufficient to fully evaluate the risks that are involved. Multiple OC curves for various payment levels (see figure 27) should also be developed when evaluating acceptance plans with payment adjustment provisions. An example will help to illustrate the evaluation of risks for payment adjustment acceptance plans.


Example: Payment Adjustment Acceptance Plans-EP Curves

Consider the previous asphalt content example where the sample size was 4, the allowable specification range was 5.60 percent to 6.40 percent, and the AQL and RQL were defined as 90 PWL and 50 PWL, respectively. However, instead of a simple accept/reject acceptance plan, the agency chooses to use equation 28 to establish the payment factor for a lot:

PF = 55 + (0.50 x PWL) (28)

where: PF = payment factor for the lot, as a percent of contract price.

PWL = estimated PWL value for the lot.

From the above equation, it can be seen that the maximum payment factor is 105 percent at 100 PWL, while the payment factor at the AQL will be 100 percent and the payment factor at the RQL will be 80 percent. It is generally accepted that the average payment for AQL material should be 100 percent. In this example, the payment factor at the AQL is 100 percent, exactly as intended. However, if the payment equation is not developed properly, the average payment factor may turn out to be above or below 100 percent at the AQL. If this is the case, the agency should determine if an expected payment other than 100 percent for AQL material is acceptable.

With the above information, the OCPLOT program can be used to develop the EP curve shown in figure 36. It can be seen in this figure that, as desired, the EP for AQL material is 100 percent. This means that a contractor that consistently produces material that just meets the minimum requirements, i.e., AQL material, will receive an average payment factor of 100 percent in the long-run. Similarly, the EP for RQL material is 80 percent as desired from the payment equation.


 

Line Graph: EP Curve from OCPLOT. Click here for more detail.
Figure 36. EP Curve from OCPLOT for the Payment Adjustment Acceptance Plan Example

The EP curve has the advantage of combining all of the possible payment levels into a single expected, or long-term average, payment for each given level of quality. While it is a major improvement over only considering a and b risks, the use of the EP alone still has some serious deficiencies. The primary deficiency in the use of EP alone is that, while it considers the average long-term payment factor, it fails to consider for a given quality level the variability of the individual lot payment factors that comprise this long-term average. This variability is directly related to the sample size. That is, the variability about the average payment factor decreases as the size of the individual samples increases. To fully evaluate the risks it is necessary to also consider this variability about the expected payment values.

The OCPLOT program output can be used to demonstrate this variability of the individual lot payment factors. Figure 37 shows for an AQL population a histogram that displays the individual lot PWL estimates along with their corresponding payment values for 1,000 simulated lots using a sample of size of 4 for each individual lot. Figure 38 shows similar information for an RQL population. The high degree of variability of the individual lot payment factors is obvious from this histogram. However, over a large number of lots, the high and low estimates for lot PWL will tend to balance out to give the correct average payment factor.

If, however, there are only a small number of lots on a project, then it will be possible that a significantly low estimated PWL value could negatively impact the payment that the contractor should have received. Similarly, larger PWL estimates could be obtained that would provide a larger payment than is deserved. A contractor would be wise to target a quality level above the AQL, particularly on smaller projects, to ensure that this variability of individual lot PWL estimates does not create a problem. However, as discussed elsewhere in this manual, it is often the practice of contractors to bid projects with the anticipation of receiving the maximum incentive payments. If this is the case, it is unlikely that contractors will target their processes at the AQL. It is more likely that they will target their processes for greater than AQL quality to try to maximize their incentive payment. In this event, the variability of the individual lot payment factors will not likely pose a serious problem to either the contractor or the agency.

The variability associated with the estimate of the lot PWL can be reduced by increasing the size of the sample obtained from each lot. Figure 39 shows a histogram that displays for an AQL population the individual lot PWL estimates along with their corresponding payment values for 1,000 simulated lots using a sample of size of 20 for each individual lot. Figure 40 shows similar information for an RQL population. When these figures are compared with figures 37 and 38, for samples of size 4, the smaller spread of the individual PWL and payment factor estimates is apparent.

Even if it reduces the variability of the PWL estimates, and hence the risk to both the contractor and the agency, it may not be practical or economical to use large sample sizes unless correspondingly large lot sizes are also used. The use of a very large lot, possibly even the total project, will allow larger sample sizes, but also introduces problems of its own. As noted in chapter 6, a major assumption that is required is that all of the material and/or construction processes remain consistent throughout the total lot.

Over the course of a long project, changes in weather, materials, rolling patterns, mix designs, etc., are likely to lead to variations throughout the project. Combining all of these together may result in a normal distribution, albeit one with a larger variability than the individual production lots, but this may not be the best method to evaluate a project. If there are "bad" segments on a project, it might be better to see them penalized on a lot-by-lot basis than to have them lumped together with the "good" material from all of the other lots.

While figures 37 through 40 clearly illustrate the relative variabilities of the individual PWL and payment factor estimates associated with different sample sizes, they do not provide any quantitative measure for the variabilities. One way to quantify these variabilities would be to calculate the standard deviation of the individual PWL or payment estimates. This is not discussed in this manual, but is presented and discussed in the technical report for this project.(17)

 

Histrogram for AQL Population Click here for more detail.
Figure 37. Histogram for an AQL Population Showing Variability of Individual PWL and Payment Factor Estimates for a Sample Size of 4

 

Histrogram for and RWL Populaton. Click here for more detail.
Figure 38. Histogram for an RQL Population Showing Variability of Individual PWL and Payment Factor Estimates for a Sample Size of 4

 

Histrogram for an AQL Population Showing Re PWL and Payment Factor Estimates
Figure 39. Histogram for an AQL Population Showing Variability of Individual PWL and Payment Factor Estimates for a Sample Size of 20


 

histogram for an RQL Population. Click here for more detail.
Figure 40. Histogram for an RQL Population Showing Variability of Individual PWL and Payment Factor Estimates for a Sample Size of 20


Example: Payment Adjustment Acceptance Plans-Multiple OC Curves

Another step that is necessary to evaluate fully the risks for a payment adjustment acceptance plan is to plot OC curves, such as those shown in figure 27, associated with receiving various payment factors. As shown in appendix M, the OCPLOT program can be used to develop these curves, although each curve must be developed individually and then manually combined onto a single set of axes.

Suppose that the OCPLOT program is used to develop multiple OC curves for the asphalt content acceptance plan from the previous example. Figure 41 shows OC curves for the probability of receiving greater than or equal to various levels of payment factor for a sample of size of 4 using the payment relationship shown in equation 28. These OC curves would be considered along with the EP curve from the previous example to evaluate the risks associated with the acceptance plan.

While the EP curve in figure 36 shows that the average long-term payment is 100 percent for AQL material, the OC curves in figure 41 show that the probability is less than 60 percent that any individual lot of AQL material will receive 100 percent payment or greater. This means that there is nearly a 40 percent chance that a contractor would receive less than full payment for a lot that was of AQL quality. This risk, which would be considered to be a (if a is defined as the probability that AQL material will receive less than full payment), seems high. However, it is somewhat offset by the fact that the OC curves also indicate that there is over a 40 percent chance of receiving a payment of 104 percent or greater.


40 - Determine Whether or Not the Risks Are Acceptable

The OC curves and EP curves describe the operation of the acceptance plan such that the risks can be evaluated throughout the entire quality regime. If the risks are considered acceptable, no modifications to the initial acceptance plan are necessary. However, if the risks are considered unacceptable in terms of being too high for both or either party, a reassessment of the acceptance plan is necessary.

As shown in the previous section, there is no easy answer to the question "Are the risks acceptable?" since this is to a great extent a subject of opinion, and opinions may vary from agency-to-agency. Table 21 can provide some guidance regardinga andb risk levels, but these risks are not very useful when price adjustment acceptance plans are used. Even in accept/reject acceptance plans, thea andb risks apply at only two specific levels of quality. An OC curve is still necessary to evaluate the risks over the full range of possible quality levels.

 

Line Graph for OC Curves. Click here for more details.
Figure 41. OC Curves for a Sample of Size 4 Using the Payment Relationship in Equation 28

Recommendation

When a price adjustment acceptance plan is used it is essential that the agency develop both an EP curve and OC curves for the probability of receiving various payment factors over the total range of quality levels. The agency may also wish to look at histograms of individual payment factors to obtain a picture of how much variability is associated with the payment factor determination. This is shown in figures 37 through 40.

The decision regarding what does or does not constitute an acceptable level of risk will to a great extent be a subjective one. There is, however, one factor that is not subjective. There is generally universal agreement that the expected payment should be 100 percent for quality that is at exactly the AQL. Although it should not be confused with the statistical risk,a, the agency may wish to consider the "average payment" risk to the contractor, if the EP is less than 100 percent at the AQL, or to the agency, if the EP is greater than 100 percent at the AQL. The EP at the RQL quality level is another point that is often specifically considered.

It must be remembered that the EP alone is not a complete measure, particularly of the likelihood that any individual lot will receive a correct payment factor. The variability of the individual payment factors about the EP curve must also be considered. Ultimately, the decision regarding what constitutes acceptable or unacceptable risks rests with the individual agency. While the determination of acceptable risks rests solely with the agency, by way of the joint industry/agency task force discussed in earlier chapters, there should be contractor input into this decision.

Word of Caution

The procedures that have been presented in the previous sections, as well as the OCPLOT program that is discussed, are primarily for the case of acceptance based on a single property. When, as will often be the case, there are multiple acceptance properties it will be necessary for the agency to develop sophisticated computer simulation methods to complete a full analysis of the risks. These analyses will be quite involved and will be dependent upon the quality characteristics chosen for acceptance and whether or not a performance model for predicting service life has been adopted by the agency. Another factor that will impact the analysis is whether a composite quality measure has been developed, or whether the individual quality measures will in some way, perhaps by adding, multiplying, or averaging, be combined into a composite payment factor. All of the possibilities cannot possibly be covered in this manual.

It is very likely that the agency will need to seek outside assistance to help in developing the simulation routines necessary to fully evaluate the risks. Universities as well as other agencies and consultants who have already developed such procedures are potential sources for this outside assistance. Once the agency has developed appropriate OC and EP curves for the acceptance plan, it should supply this information to the contractors that work in the State. Otherwise, each contractor will individually be required to seek outside help to fully understand the risks associated with the new acceptance plan, and this is not very cost effective for the contractors or ultimately for the agency.


41 - Modify Acceptance Plan, if Necessary

If the risks are considered unacceptable they are likely to be too high rather than too low. To reduce the risks it may be possible to change the specification limits, the acceptance limits, and/or increase the sample size. The most straightforward approach would be to increase the sample size per lot.

An increase in the sample size may be accomplished by either increasing the lot size or increasing the sampling frequency. For example, if the lot size were 1800 Mg of HMAC, and the sampling frequency was one test per 450 Mg, then the number of tests per lot could be increased from 4 to 8 by increasing the lot size to 3600 Mg. On the other hand, the number of tests per lot could be increased from 4 to 8 by keeping the lot size as 1800 Mg but increasing the sample frequency to one test per 225 Mg.

Another way to change the risk levels would be to change the specification limits or the acceptance limits. This may be related to the definition of AQL and/or RQL material. For example, for the example presented above for asphalt content, the AQL was defined as a population with a mean of 6.00 percent and a standard deviation of 0.20 percent. Using this definition, the specification (and, in this case, acceptance) limits for an accept/reject decision based on a single test result were set at plus or minus two standard deviations from the target value of 6.00 percent, i.e., 6.00 percent ± (2 ´ 0.20 percent) or 5.60 percent to 6.40 percent. This provided an a risk to the seller of 0.0456, or 4.56 percent. This risk can be reduced to nearly zero by setting the specification (and acceptance) limits at ± 3s rather than ± 2s. However, this will also increase the b risk, unless the definition of RQL is changed.

For accept/reject acceptance plans based on PWL, the acceptance limit could be reduced, say from 90 PWL to 85 PWL, to lower the a risk. It could also be raised, say from 90 PWL to 95 PWL, to increase the a risk. It must be noted that whenever a is changed, b will also change unless the sample size is changed as well. For acceptance plans with price adjustments, the payment equation could be changed to increase or decrease the expected payment values. While these changes would impact the EP values and the "payment" risks at the AQL and RQL, they would not necessarily change the "statistical" risks,a and b.

New OC curves and EP curves must be developed for any changes that are proposed to the initial acceptance plan provisions. This is the only way to determine what impact the changes will have on the risks to both the contractor and the agency. The agency should not proceed with developing the finalized draft specification until an acceptance plan has been developed for which the agency believes the risks are appropriate.


42 - Finalize the Initial Draft Specification

Once all of the preceding steps have been completed, the agency can move forward to finalize the wording for the initial draft specification. At this point the agency is ready to move forward to the implementation phase of the specification development process.

It is obvious from the above discussions that a great deal of thought should be put into the development of an acceptance plan. There are many "pieces" to the puzzle that must fit together for the acceptance plan to be well-written and to work as intended. However, there are many resources that can be used to help accomplish this goal. QA acceptance plans have been under development and evolution for over three decades. This history can be an invaluable resource for any agency that is in the process of developing QA acceptance plans.

 

Previous | Table of Contents | Next

Federal Highway Administration | 1200 New Jersey Avenue, SE | Washington, DC 20590 | 202-366-4000
Turner-Fairbank Highway Research Center | 6300 Georgetown Pike | McLean, VA | 22101