USDOT Federal Highway Administration USDOT Home | FHWA Home | Feedback
Skip to main content

P3 Toolkit

P3-VALUE Webinars

Transcript - P3 Project Risk Assessment

Presentation: PDF  HTML
Webinar recording: Audio

IPD Academy Web-Based Course
June 13, 2013

Presented by
Patrick DeCorla-Souza
P3 Program Manager
Office of Innovative Program Delivery
FHWA

Contents

Introduction to P3 Project Risk Assessment

Slide 1 - P3 Project Risk Assessment

Operator - Ladies and gentlemen, thank you for standing by. Welcome to the P3 Project Risk Assessment 201 conference call. At this time, all participants are in a listen-only mode. Later we will conduct a question-and-answer session. Instructions will be given at that time. If you should require assistance during the call, please press star then zero. I would now like to turn the conference over to your host, Mr. Michael Kay, please go ahead, sir.

Michael Kay - Thank you, Janice. And on behalf of the Federal Highway Administration's Office of Innovative Program Delivery I'd like to welcome everyone to today's IPD Academy Webinar, P3 Project Risk Assessment. My name is Michael Kay and I'm with the U.S. DOT's Volpe Center in Cambridge, Massachusetts. I'll be moderating today's webinar along with the help of my colleague Victoria Farr, as well as facilitating our question-and-answer period and helping to address any technical problems. Our presenter for today is Patrick DeCorla-Souza. Patrick is the P3 Program Manager in the Office of Innovative Program Delivery.

Before he gets started, I just want to point out a couple of key features of our webinar room. On the top left side of your screen, you'll find the audio call in information and the list of attendees. On the bottom left, is a chat box that you can use to submit questions to Patrick throughout the webinar. We will encourage you to submit those questions via the chat box today. Time permitting, we may open the phone lines towards the later end of the webinar, but again, please do submit your questions through the chat box. If you have any technical difficulties, please use that chat box to send a private message to myself, Michael Kay. Our webinar will run until about 3:30 p.m. Eastern time today and the course is divided into six sections. We'll queue up any questions for Patrick at the end of each section and we also anticipate having a few minutes at the end of the course for additional Q&A. There are people unable to participate in today's webinar, so I wanted to mention that we are recording today's session, so they can listen at a later date.

Before we begin, we have four poll questions we'd like to pose, and I'll pull them up on your screen now. The first is simply, what is your affiliation, whether you're with the Federal Highway Administration in the division office, or outside of the division, a state DOT, another federal agency, another state agency, MPO, private sector, or other. To the right of that, how many people are participating along with you today, just to give us a sense of whether you're joining us along or in a larger group. On the bottom-left, what is your current level of knowledge and experience with project risk assessment, whether you have no prior knowledge, a very basic understanding, some knowledge and experience, already have a lot of direct knowledge and experience but still have some more to learn, or whether you're already an expert. And then on the bottom-right, we're curious to know if you have already accessed the P3-VALUE Risk Assessment Tool. Patrick will go into detail about the tool itself as well as information on how you may access it. We're just curious to know how many of you have already accessed that tool? We'll leave those up for another 15 seconds or so.

And while doing so, I'll just mention that accompanying Patrick today is Jim Sinnette in the Project Delivery Team and he'll be assisting us in answering questions during our Q&A section. Jim is the Project Delivery Team Leader in the Office of Innovative Program Delivery. I'll go ahead and move on to our presentation. And with that, I'll turn the webinar over to Patrick, who will begin with the course outline and objectives and begin the first section of the course on P3s and risk. Patrick.

Slide 2 - FHWA's P3 Toolkit

Patrick DeCorla-Souza - Thank you, Michael.  And I just want to point out that I'm very fortunate that Jim Sinnette from our office who has been conducting cost estimate reviews on major projects for many years is able to join me to help answer any questions. So here is a little background. I see some of you have already accessed our P3-VALUE tools. This slide gives you a background of where the tool fits in, in our overall P3 Toolkit, which includes four general areas relating to P3: first legislation and policy; second planning and evaluation; the third procurement; and the fourth monitoring oversight. And so far we've done a lot of work on the planning and evaluation and the P3-VALUE tools are a part of that phase of P3 implementation. We are also beginning work on the other aspects and we'll have more on our websites soon on the other areas referenced in this slide.

Slide 3 - P3-VALUE Webinars

A little more about P3-VALUE is on this slide. It's a set of four tools, individual Excel-based analytical tools that help the users understand how to conduct value for money analysis, to evaluate public-private partnership options. We did have an introductory webinar providing an overview of P3-VALUE and P3 evaluation, in general and that was held about a month ago on May 2. The slide shows you the URL to listen to a recording and by the way, all of these slides will be available to you for download so you can click on that link and listen to the recording if you missed it. Today, we are going to do the second of the series of four webinars on P3-VALUE, on project risk assessment. We have scheduled a webinar on value for money analysis on July 11. And then financial assessment will be discussed on August 7. So feel free to register for these webinars if you are interested.

Slide 4 - P3-VALUE Tools

Here on this slide is the list of four tools. The risk assessment is the one we will be talking about today. It uses your information on identified risks, the allocation or risks, your response strategies and potential cost and schedule impacts for each risk and comes up with estimates of aggregate risk financial impacts. So you get values that you can then use to add to your project costs for contingencies. The Public Sector Comparator Tool and Shadow Bid Tool we will be talking about in the next webinar in July on value for money analysis. The Public Sector Comparator Tool is a tool that helps you estimate the risk-adjusted life cycle cost of conventional procurement. So basically the risks that we will talk about today are inputs into the Public Sector Comparator Tool. The risks we will be estimating today will also be input into the Shadow Bid Tool and we will discuss the Shadow Bid Tool, which is basically an estimate of a public-private partnership option. And that tool provides estimates of costs for P3 procurement and also is able to estimate something called availability payments, those are annual payments that you might be making to the private partner over the term of a concession. And finally, the Financial Assessment Tool is the fourth tool in this package of tools. And what it does is it takes the outputs from the PSC Tool and the Shadow Bid Tool and calculates value for money. That is the difference between the two and percentage difference and also helps you evaluate financial viability.

Slide 5 - P3-VALUE Tools

This graphic shows you graphically how the tools are related. You've got project assumptions that go into your Risk Assessment Tool, which then calculates risk values, which are then input in to the PSC Tool and the Shadow Bid Tool. And then the outputs from the PSC Tool and Shadow Bid Tool are input into the Financial Assessment Tool to get you your financial viability results as well as value for money results.

Slide 6 - Risk Assessment Tool

So the tool we're going to talk about today is the Risk Assessment Tool. As I indicated earlier, we use this tool to estimate the aggregate financial impacts of risks. But this is entirely dependent on data that you input. So you identify risks. You decide on whether they should be allocated to the private partner or to the public agency. You decide on risk response strategies. And you put in estimates of potential cost impacts and schedule impacts. And the tool then is able to use this information to calculate an aggregate contingency amount that you can add to your baseline project costs. The tool also helps you figure out the breakdown between what are called retained risks, or risks that are retained by the public agency, and transferable risks, or risks that could be transferred to the private partner in a P3 project delivery option. In order to run the tool, you need certain information. You need basic information on preliminary design scope and alignment. You also need to know the schedule for the project and life cycle costs. That is not just the construction costs, but also the operations and maintenance costs over the life of the project. You also need to have some idea as to what your procurement options are both in the base case (that is for the public agency, if it were to be delivering the project in a public method) or if it is choosing to use a P3 method or is trying to evaluate a P3 method there are variety of P3 options. And you would need to know which particular option you are interested in and we'll discuss this later on in this session here.

Slide 7 - Course Outline

So this slide shows you what we are going to go through in this webinar. The lesson one is a brief recap on what we have discussed so far in the prior webinar on P3 evaluation. Lesson two, we get into a little background on where risk assessment fits in, in an overall risk management process. So we will talk about a risk management process, which applies not only to P3s, but to any project. Lesson three, we will go into more detail on risk assessment and how do you actually calculate the financial impacts of risks. Lesson four, we will talk about factors you would need to consider in deciding whether to allocate certain risks to the private partner. Lesson five, we take you through an example to show you how risk is incorporated into value for money analysis. And in lesson six we will introduce you to the Risk Assessment Tool so that those of you that haven't tried it already can begin to use it, and see how it works.

Slide 8 - Course Objectives

So we hope that by the end of this webinar, you will be able to describe the various transportation project delivery models, including the various P3 options available to you. That you will be able to identify types of risks in the project's life cycle. You will be able to explain how risks are quantified and monetized, that is how a value is placed on risk so that you can get a contingency amount. And you will be able to describe how these values of risk are incorporated into value for money analysis. And finally, hopefully by the end of the webinar, you will have the courage to go in and test out our Risk Assessment Tool.

Lesson 1: P3s and Risk

Slide 10 - What is a P3?

So a little background. This is a recap of what P3s are and what risk is. So P3 stands for public private partnerships. It's a contractual arrangement between a public agency and a private entity covering more than a single project phase or function. So whereas in design-bid-build, the public agency contracts with a separate contractor for each phase; in the case of a P3, the public agency contracts with a single private entity to do both design and build and sometimes even operations and maintenance or even financing. So several different phases might be involved in the contract. The purpose is to increase public-private participation, that is to allow greater participation in the private delivery of a project's financing as well as construction, operations and maintenance in order to create value and in order to benefit the public interest.

Slide 11 - Common Types of P3s

The most common types of public-private partnerships are shown in this slide. So you have a greenfield, which stands for basically new facilities, brownfields are existing facilities that might be turned over on a lease basis to a private partner to operate and usually this isn't the case of toll facilities so that they can collect the tolls and use the tolls to operate, maintain, rehabilitate the project over the term of the concession. So greenfields are the ones that P3-VALUE tends to address, where we're talking about new facilities so there are several options here, starting with just design-build. But our office, the Office of Innovative Program Delivery, is involved in projects that also include financing. So design-build-finance and design-build-finance-operate-maintain are the two options the P3-VALUE tools can help you evaluate.  Now, under design-build-finance-operate-maintain, also called DBFOM, there are two options. One is when you have a toll facility and the toll revenues go to the private partner and that's called a toll concession. If you have a toll facility and the revenues do not go to the private partner but come back to the public agency, you would be paying the private partner through something called an availability payment, which is an annual payment made to compensate the private partner for its investment in the facility. Now, availability payments can also work if there are no tolls on a facility since the revenues to pay the private partner could come from tax dollars rather than from tolls.

Slide 12 - Potential Benefits and Drawbacks

This slide summarizes succinctly the benefits and the drawbacks of P3s. So on the left side you have potential benefits. The big one is the financial capacity that public agencies now have access to through a P3. Since private investors are willing to put in extra money you could accelerate delivery of a project. And if you are at your debt capacity limit you would be able to use the private partner as an alternative way so that you do not have to use your debt capacity and go ahead and build a project, even if you don't have debt capacity available. Other benefits relate to the fact that when you combine all of these phases together you could create efficiencies. So, for example, with design-build it's quite well known that when you combine these phases, the contractor can coordinate the construction methods with the design much better and bring the costs down, for example. Or the contractor can make allowances for lowering the cost of maintenance, for example, by maybe making pavement stronger to begin with. So bring down the overall cost of the project by doing that. So these are benefits of combining phases. The other benefit to the public agency relates to risk transfer. And, of course, if you are able to transfer all of these risks to the private partner for a fixed price, you have budget and cost certainty which is a benefit to the public agency. And generally some risks are better managed by the private partner. So the costs of those risks can be reduced, benefiting the cost of the project.

Of course, on the right-hand side you see the potential drawbacks. The main one raised is the loss of flexibility of a public agency. Priorities change over time and when contracts are written they are generally for a period of 30, 40, or 50 years and it becomes very difficult then to make changes to the contract. The other issue relates to the fact that if you have one part of your network under the responsibility of the private partner it may be more difficult to integrate that facility into the overall network for the most efficient operations, for example. It is well known that the private sector does have higher costs for acquiring debt and the rate of return demanded by equity investors is also much higher than the borrowing rate of the public agency, for example. So you will find financial costs higher in a P3. And the procurement process tends to be more complex. You have higher costs for procurement. These contracts are very detailed, require expensive legal help, require expensive financial experts to weigh in and provide advice. And the negotiation process with the private partner requires expertise within the public agency.  It's not enough to simply hire expertise from outside. You also need some expertise within your agency.

Slide 13 - Types of Project Risks

So the P3-VALUE Risk Assessment Tool addresses risks of certain types. And we have here the two phases addressed by the P3-VALUE Risk Assessment Tool, that is the design-build phase and the operations phase. Now, there are also risks related to the procurement phase prior to design-build. And these may not be considered by P3-VALUE, but they are obviously considered by the private partner and you need to take that into consideration. So under design-build you have various site risks. You have construction risks. In the operations phase, the chief risk if it's a toll revenue project, it is a performance risk. Performance risk relates to the possibility of inflation, raising operation maintenance costs, and latent effects from construction, et cetera. Now, there are also general risks that apply to both phases. There are political risks, economic risks and other risks.

Slide 14 - Financial Impacts of Project Risks

Financial impacts of project risks include the costs, obviously, but also the delays. You have scheduled delays that can impact costs as well as revenues on a toll project. And finally, if your traffic is lower than expected, you would have toll revenue impacts and that's another kind of financial risk.

Slide 15 - Purpose of Risk Assessment in P3s

So the reason you would do a risk assessment prior to considering a P3 is first, if you want to evaluate whether P3 is a good option to use. And the way you do that is by conducting a value for money analysis. Now, in order to do a value for money analysis, you need to know the risk-adjusted life cycle costs of both the P3 option as well as the PSC or the public sector comparator. So risk assessment is the first step before you can do a value for money analysis. After you have done your value for money analysis and you feel you want to go through a P3 option, you have to start developing a request for proposals. And you have to develop a draft agreement and that draft agreement will have to outline who is going to bear the various risks. So your risk assessment helps you in that process, in deciding what risks to transfer to the private partner. After you have selected the best value, there will inevitably be a negotiation phase where you have to dot the I's and cross the T's and at that point it's also very helpful to know what the real value and price of a risk is, so that you are in a better position in these negotiations. And finally, a risk assessment helps you manage risks. You create something called a risk register where you monitor risks and be ready in case the risks do materialize.

Slide 16 - Construction Phase Uncertainties

So in order to understand what P3-VALUE Risk Assessment Tool does, it's important to understand uncertainties and the types of uncertainties in creating cost estimates. And so there is something called base variability. It's uncertainty, but it's not caused by risk events. So, for example, you know you're going to need something, but you just don't have enough information to accurately cost it out. So in early phases of planning a project, for example, you might realize you need a culvert but you don't know the size of the culvert. And so the size of the culvert affects the costs. So this is part of you know you need a culvert, but you don't know exactly what it's going to cost. And so that's called base variability. Risk is actually an uncertain event that may or may not occur. So it may not happen. And if it happens, it may have a negative effect, or it may have a positive effect. So if it has a negative effect it's called a threat. If it has a positive effect, it's called an opportunity. And here, again, just to put it in the P3-VALUE framework what P3-VALUE Risk Assessment Tool does is look at something that you might call known unknowns. In other words, you know something might happen. There's a probability it might happen. It may not be 100 percent probability, but something less than 100 percent and you're aware of it and you make an attempt to cost it out and figure out what its impact might be if it were to happen. On the other hand, there are some risks that you are not going to be able to cost out because you haven't even identified these risks. So you see the unknowns are risks that you didn't really think about when you were doing your cost estimates, but that materialize. And so you need to set aside a contingency for these unknowns. Again, the P3-VALUE Risk Assessment Tool will not give you a way to estimate that. That is something that you have to estimate and add to your cost estimation.

Slide 17 - Construction Cost Uncertainty

So this graphic tries to summarize what I've just told you. The lowest line is your basic, lower-bound base cost estimate. The base variability is the extra amount you set aside just in case the culvert needs to be bigger than you thought it might be and that's called allowances. And then what P3-VALUE tries to estimate is the known-unknowns. And, by the way, different people have different terminology, so you might see a different term used somewhere else. But the third segment is what P3-VALUE estimates. And it helps you estimate your contingency. And then you have the unknowns that you have to still figure out. So what you're seeing in this graphic is as you move along the phases from planning to preliminary design, to final design, you have more information, more detail and you can set aside a smaller allowance. The risks, on the other hand, continue because you don't know whether they're going to happen right up until construction is complete and then you know, of course, at least in the construction situation, you know that you have built the project, so there's no more risk to worry about. And so that's why you see that line going all the way to the end of construction.

Slide 18 - risk Magnitude over Concession Term

Now, how does the private sector look at this? Remember, I said, from the private sector standpoint, there's a lot or risk in the procurement phase, prior even to construction. And in developing its bids, for example, they might put a lot of effort into a bid and then find out that they're not the winner, or even worse the entire bid process is cancelled because some new administration has come into the state and there's a change of plans and they don't want to do a P3 anymore. So a lot of risk that the private partner sees in the early phases. And therefore, that risk requires a risk premium. In other words, the rate of return that they demand when they see higher risk is a higher premium. So as you go down towards the later phases you see that the risk premium demanded is reduced in the construction phase. And once the project is completed and maybe traffic is beginning to ramp-up you may still have a lot of risk, but once traffic has stabilized and toll revenues coming in, the risk is reduced even further. And finally, in the operation phase there's little or no traffic risk but there might be other risks such as inflation, et cetera. So the risks are even lower. So I turn it over to you, Mike.

Slide 19 - Audience Feedback

Michael Kay - Thank you, Patrick. We just wanted to pull up a couple of quick questions by way of feedback and I'll open them up on your screen. We're curious to know, on average, how much contingency does your agency include in cost estimates to account for risk in the planning phase?  That's on the top there, as distinguished from the design-phase on the bottom. So, again, on average how much contingency does your agency include in cost estimates to account for risk in the planning phase, on top and in the design phase on the bottom? And we'll leave that open for another 15 seconds or so.  In the meantime, feel free to submit any questions through the chat box on the bottom-left of your screen. And I'll go ahead and close those out. Patrick, anything to say about these poll questions? Or shall we move on?

Patrick DeCorla-Souza - Let's move on.

Slide 20 - Questions?

Michael Kay - Okay. Sounds good. I will go to our Q&A layout and we'll just take a couple of questions. Two comments from William and I'll just read them out for you, Patrick, and I think they're just suggestions maybe for the next iteration to include but "under benefits, by combining design and construction, one accelerates completion of a project by at least a year and eliminates most of the change orders by shifting the public risk to the P3." So that may be something you might want to consider adding in as a corollary going forward. And then second, with regard to procurement risks, "most public projects have been low-bid; with P3s the emphasis is on best value, which may not be the low bid. The public, press and legislatures need to understand the difference and importance of best value. It is all critical in all P3s that transparency is present.  Failure to address the procurement risk can seriously undermine P3 opportunities." So I appreciate those comments, William.

Patrick DeCorla-Souza - Thanks, William.

Michael Kay - And Patrick, seeing no other questions and in the interest of time, we're a little behind schedule, so I want to make sure we continue on. Let's move on to lesson two.

Lesson 2: Risk Management Process

Slide 22 - Risk Management Process Overview

Patrick DeCorla-Souza - All right, so hopefully we can go through this pretty quickly because this is something that is really part of normal risk management for any project, not necessarily just for P3s. So this is the process. It's a standard process used by public agencies for a long time now in the United States. You identify the risk. You assess it, analyze it, in terms of the probability of it happening and what the consequence might be either for cost or schedule. You prepare to respond to it, try to figure out how you can reduce its impact. And then one response technique is actually to allocate it to the private partner. And, of course, with that comes a cost that the private partner is going to charge you. And then after you are in the process of construction and operations, you have to constantly monitor these risks so that you are ready to respond if and when something might happen.

Slide 23 - 1. Risk Identification

So very quickly, the risk identification process is pretty standard. Our office helps with cost estimate reviews on major projects. And we have one early in the project development process and one at a later stage closer to the record of decision. So what happens here is you have a facilitator and you have various subject matter experts that provide their opinions on the probability and the probable impact or a range of impacts if a certain risk were to occur. So they might say, for example, the probability of certain risk occurring is 80 percent. And if it does happen, then the cost might be somewhere between $1 million and $3 million. And so you get this information from all of the different disciplines: design, environmental, geological, financial planning, et cetera, and you put it all together in the risk register.

Slide 24 - 1. Risk Identification (Cont.)

Of course, you need certain basic information on the project scope and design. You need to know what kind of procurement option you're looking at, is it a P3? Or is it a design-bid-build or is it design-build because this can change even some of the magnitudes of the risks. And how do you capture all of this information? You capture it in a risk register, which we will be talking about later in this webinar because it's part of P3-VALUE. But one of the tools you might use is a risk checklist to help you go through all of the different types of risks to see if they apply to your project. In risk assessment, of course, the key thing is trying to figure out what the probability of a certain risk might be. And then if it were to occur, how would it impact costs, schedule and in the case of a toll project, revenue from tolls because of the delay in beginning operations.

Slide 25 - 2. Risk Assessment & Analysis

There are two types of assessments, qualitative and quantitative and P3-VALUE does both. It first does a qualitative assessment. And then the more important risks are taken through a quantitative assessment.

Slide 26 - 3. Risk Response Planning

So once you've assessed the risks in terms of probability and impact you need to figure out how to respond to those risks. You could try to reduce the probability of occurring by taking precautions. You could plan to mitigate them, that is if they happen you can reduce their impact if you're prepared for them. And, of course, a third way is to transfer the risk and that's one thing you can do through a P3, is the risks you think that the private sector is more capable of handling you can allocate to the private sector.

Slide 27 - 4. Risk Allocation

So that's where the concept of transferable risks comes in. So risks that you can transfer to the private sector are called transferable risks. Those that you retain as a public agency are called retained risks. And then there are some risks that neither party would be the best to handle. And those risks can be shared.

Slide 28 - 5. Risk Monitoring & Control

And as I indicated earlier, monitoring risks throughout the project's life is an extremely important thing to do. You use performance metrics to monitor the risk. And especially in a P3 it's important to understand which risks have been transferred to the private sector so that you do not inadvertently take back the risk. So it's important to understand the P3 agreement because usually it's one set of individuals that develop the agreement. And then there's another set of individuals that are responsible for monitoring and oversight. And so it's important for the two sets of individuals to be talking to one another so it's clear as to what risk the private sector has taken responsibility for. So I turn it over to you, Michael.

Slide 29 - Audience Feedback

Michael Kay - Thank you, Patrick. And, once again, we wanted to get some audience feedback so I'll pull up our question: In your view, which of the following risks may be managed at a lower cost by the private sector? And for this question you can check all that apply. So environmental risk, land acquisition, utilities, financial, design or geotechnical, construction, traffic and revenue,  and operations and maintenance. Again, which of these risks do you feel may be managed at a lower costs by the private sector? And we'll leave that open for another 15 seconds or so.

Slide 30 - Questions?

Again, I encourage you to submit questions through the chat box on the bottom-left. And I'll go ahead and close that out. Thanks so much for your participation in that poll. I didn't see any questions coming through the chat box Patrick, so you've been very thorough thus far and I'm sure we'll attend to some additional comments coming forward. We'll wait just ten seconds or so just to see if any questions do come in, but that said, we will certainly have additional Q& A going forward. So with that, Patrick, let's move on to lesson three.

Lesson 3: Risk Assessment

Patrick DeCorla-Souza - All right, so in this lesson we're going to get into the meat of P3-VALUE. This is the focus of this webinar and we're going to talk about how P3-VALUE actually does its risk assessment. And, again, I want to highlight the fact that this is an educational tool. It's simply meant to help you understand the process. Our office, and Jim Sinnette in particular who leads the Project Delivery Team, has very detailed models that are used to help estimate risk and these are done on major projects all over the country. And those are the types of models that you would actually use in practice. The model that we have here in P3-VALUE is simply to help you understand the process that you have to go through.

Slide 32 - Risk Assessment Process Overview

So like I said, risk assessment is done in two phases with P3-VALUE. The first is a qualitative assessment. And the second is a quantitative assessment. So the qualitative assessment simply divides risks that you have identified into high, medium and low categories. And then it takes the medium and high and very high risks and does a quantitative assessment. The very low and low-rated risk really does nothing unless you do something with them. So you can either say that these risks are really minor and you could discard them. Or you could take these low-rated risks and aggregate them into some aggregate risk that you can then use in the further assessment process, which I'm going to talk about, the quantitative assessment process. The quantitative assessment process attempts to get an actual value of the risk that you can use in your cost estimates. And the way it does that is by taking a probability percentage, so what do you think the probability of that risk occurring, and the impact of the risk, so what is the cost or the schedule impact that you think that risk might have, and then you can take these estimates and either as done in P3-VALUE you can throw them into a Monte Carlo simulation, which I will talk about in a little bit, or if you don't want to do Monte Carlo simulation, there are simpler methods; for example, the Virginia DOT does a formula-based calculation and I'll talk about that formula, which is a little simpler to understand. And that then gives you costs of these risks and helps you arrive at a total value for all of these high-rated risks.

Slide 33 - Key Inputs

So as I indicated the probability is one of the key inputs. If you're doing qualitative, you divide it into very low, low, medium, high, very high. In case of quantitative you need to know the exact probability something between 0 percent and 100 percent. And you need to know the scale.  So in the case of qualitative you just simply say very low, low, medium, high, very high. In the case of quantitative you need to provide a dollar amount or the number of days of delay that you think would be the impact if the risk were to occur.

Slide 34 - Qualitative Risk Assessment

So in a qualitative assessment, what you do is you try to figure out how important a risk is. And so there are two parameters that go into that calculation. First is the scale of probability, the probability of the risk happening. And if you look at that first matrix on top, 5 is more than 70 percent. And so if I look at that row and if I say the potential cost consequence is that it would have a 3 percent to 10 percent increase in my cost for the project, that in this case, this matrix gives me a three and I would go down that column and I see that it's a high risk. On the other hand, if the cost impact were less than one percent I would rate that risk as a one, in terms of cost consequence. And even though the probability is high, 70 percent, the impact is very low and so overall the risk is not as important and it's categorized as a low-rated risk. You do the same thing with schedule impacts for the same risk. And, of course, it may have a different impact on schedule. And so with the tool itself, all you have to do is put in what you think is the schedule consequence, what you think is the probability (which will be the same as what you had for the cost consequence of that risk, so you don't have to input that again), but it will then tell you whether the risk is an important risk or not.

Slide 35 - Quantitative Risk Assessment

So with the important risks or the high- and medium-rated risks, you can do a quantitative risk assessment and Virginia DOT uses a formula. The formula basically takes the probability and multiplies it by an estimate of the cost consequence. And the cost consequence is a composite of the minimum possible cost, the maximum possible cost and four times the most likely cost. So you take these, you divide them by six and you multiply that by the probability to get the estimate of the cost of that risk. And in the table is an example and in column one there is a 50 percent probability of risk 1 happening. The consequence could be $1 million or as high as $5 million. So that would be the max. But the most likely is $4.5 million. So I take the $4.5 million, multiply it by four, which gives me $18 million. And I add $5 million, the maximum, and I add $1 million, the minimum, for a total of $24 million divided by six, multiplied by the probability of 0.50 and I get $2 million. So I do that with risk number 2 and I get a total of $2.8 million. So that's basically how we would estimate the aggregate risks. You would compute individual risk costs and then aggregate them to come up with the total amount that you need for your contingency.

Slide 36 - Quantitative Risk Assessment (Cont.)

In P3-VALUE it's a slightly more sophisticated process and there are some benefits to that, which I will explain. Monte Carlo simulates a large number of scenarios. And P3-VALUE can do from 300 scenarios to 1,000 scenarios. And in each scenario, you pick the risk and the program picks a certain set of risks and a certain cost that goes with that risk. And if you're looking at risk 1, for example, so 50 percent of the scenarios in that 1,000 scenario run would include risk 1. And then what would it use for the cost of that risk?  Well, if you do a triangular distribution it's a probability distribution. And the next slide shows you what that looks like.

Slide 37 - Simple Probability Distributions

This is a triangular distribution on the left. And the minimum is on the extreme left. The maximum is on the extreme right. The most likely value is the apex of the triangle. And the Monte Carlo simulation attempts to replicate this distribution so that the 1,000 values would all be distributed to fill up this triangle. And so what you would get as a result of this is something on the next page. It does it for both risks. So it does it for risk 1 and it does it for risk 2 and risk 2 had a uniform distribution. A uniform distribution basically says every risk has an equal probability of cost throughout the range. There is no most likely value for scheduled delay or for cost.

Slide 38 - Quantitative Risk Assessment

So you throw all of these risks into the scenario analysis called Monte Carlo simulation. And what you get is an output that looks like this. Basically, if you look at the slide you see that all of the risk 1 costs appear in the histogram to the right. The bars to the left are risk 2 costs, which are lower and between $1 million and $3 million. And the risk 1 costs were between $1 million and $5 million those are distributed over to the right. And then when you add up all of these you get the cumulative distribution. So the line right along the entire graph there provides the cumulative distribution. You see at the end it only goes up to 90 percent because of the fact that we had only two risks that we were evaluating.  It should normally go up to 100 percent total.  So if you remember when we added up the aggregate risks in the VDOT, or Virginia DOT, formula-based approach, we got $2.8 million.  So where does that appear on this slide here, and what you need to do is look at 50 percent cumulative probability, I'm looking on the right-hand vertical axis, 50 percent probability.  Go down to where it meets the cumulative distribution curve and then go vertically down and what you see there, it's somewhere around $2.8 million.  So basically the VDOT procedure gives you the same thing as the Monte Carlo Simulation for a 50 percent probability.  So why would we want to do a Monte Carlo simulation?  Well, there's a good reason, because sometimes you want to be more confident than 50 percent.  So what you can do is go down and let's say you want 70 percent confidence.  You can go to the right axis, the vertical axis again, go horizontally across and then go down, and what you see there, you're getting something like $4.5 million.  So if you want to be a little more confident, which is of course something you want to be, FHWA recommends a 70 percent confidence level, you can get that through Monte Carlo simulation.  This is something you cannot get through the formula-based approach.

Slide 39 - Quantitative Risk Assessment (Cont.)

So I wanted to show you what you will get if you run P3-VALUE, and so this is an example of the results.  On the top rows you see P10, which means that you would have a risk cost of $530,000, 10 percent or less of the time.  So if you want to have only 10 percent confidence in your results, you would use $530,000.  If you want 70 percent confidence, then you would use $3.9 million.  If you want 90 percent, you want to be very confident, you would use something like $5.1 million.  So it just depends on how much confidence you want to have that you will have the cash available if the risk materializes.  So on the right-hand side, because P3-VALUE also asks you who is allocated the risk, and so for example, for that risk 1, we said it's a design-build risk and 20 percent of that risk would be born by the public sector, and 80 percent by the private sector, and simply that got repeated because that was the only design-build risk.  If there were more than one design-build risk, it would average out all of the allocations to give you the final percentage there.  Same thing with operations, we said the second risk was an operations risk and we said it was 100 percent belonging to the private sector, and this is the output since there's only one risk.  It simply spat out the same 100 percent.  So what P3-VALUEs does is calculates these overall risks and the total at the bottom, if you see the bottom line total is the same as what you have in the corner on the upper-left, and these are the total risks calculated by Monte Carlo simulation for the P10 confidence level, P70 and P90, and then of course, it's broken down by design-build versus operations because obviously when you're doing a financial plan, you want to know when these risks are going to materialize so you have the adequate amount of cash available.  One thing I wanted to draw your attention to is the fact that all of this is in real dollars.  So if you see up there on the top it says cost risk results, real dollars.  So when you're actually doing your financial plan, you would need to convert everything to nominal dollars and distribute them over the appropriate years when these risk costs occur.  So, in a nutshell that's how P3-VALUE does the analysis.

Slide 40 - Risk Assessment Challenges

I want to draw your attention to some issues.  Remember, this is an educational tool.  There are still several issues and what this analysis will do is help you understand the issues.  So the first is estimating risk impacts and probabilities is not an easy thing to do.  So you need lots of data, what has happened in previous projects, et cetera and there is sometimes a danger of the estimators in the various disciplines being too optimistic.  In other words, estimating that the probabilities of the risks are lower than they actually might be, and to some extent I think that FHWA's P70 requirement helps to address some of that optimism bias.  Now in some cases, not in the United States, but I know in Canada for example, there have been concerns that the opposite is the case.  In other words, the public agency procurement risks are overestimated and the theory is that these estimates are done by an agency that is in the position of promoting public-private partnerships so the concern is that they might be overestimating the Public Sector Comparator risk in order to make the P3 option look good.  So that's something to be aware of.  So you need to account for correlations among risks.  P3-VALUE does not account for that.  As I mentioned, only the risks you have identified are going to be evaluated and monetized by P3-VALUE but the unknowns are not going to be evaluated, so you will need to account for those somehow in your contingency.  Double-counting of risks is an important issue because P70 and P90 actually account for risks.  So you are including those in your cost estimate.  Now, you will find that P3 financial plans and financial models include premiums for financing and you have to be aware that these premiums also try to account sometimes for the same risks.  So you have to be very careful to not double-count premiums, that is the interest rate on debt or the equity rate of return might be accounting for this risk and at the same time you're using a P90 or a P70, which also accounts for the same risk.  So you have to be very careful about that.  The accounting for revenue risks you have to do that separately because the P3-VALUE Risk Assessment Tool cannot do that for you.  You might try to do that, but you might get some strange results.  You know, revenue risks can be a threat or they can be an opportunity in some cases.  It could be an opportunity for more revenue, but P3-VALUE cannot handle that well at this point.  Perhaps you can address that through things such as the discount rate.  Aggregating low probability, low consequence risks, that's something you would have to do manually because P3-VALUE will not do that automatically for you, and finally I indicated there are some procurement fee risks that are borne by the private sector, for example, in preparing bids, and they put in all this money and there's only a one in three or one in four chance that they will actually win the bid.  Well, that's a cost to them that they are bearing and they have to make it up in some other proposals that they might win.  So they have to make it up and so there will be a risk premium that they will add to their equity rate of return to account for those bids that they have lost.  That's it for this lesson, Michael.

Slide 41 - Test Your Knowledge

Michael Kay - Thank you, Patrick.  At this point we have one test your knowledge question that we wanted to pose to you just to make sure that you're understanding the main concepts of the webinar and to see that you're understanding the material.  So true or false, in quantitative risk assessment every risk, whether low, medium or high, is quantified individually with regard to probability and scale of impact.  I'm going to read that one more time for everybody.  True or false, in quantitative risk assessment, every risk whether low, medium or high, is quantified individually with regard to probability and scale of impact.  I will keep that out for about 15 seconds or so, so you can give it some thought.  But clearly, Patrick, we need to clarify a bit.

Patrick DeCorla-Souza - Yes.

Michael Kay - I don't think the vote suggests that we have about as many true as we do false.

Patrick DeCorla-Souza - Yeah, it's equally balanced here so it was a good question to ask.  I think the key issue here was the word quantified individually, and whereas you do try to assess all risk qualitatively, you do not necessarily quantify the lower valued risks.  It's just too much effort and may not be worth your effort.  That's why those who said false are correct. 

Slide 42 - Questions?

Michael Kay - Great.  Thank you, Patrick.  With that, let's take a couple of questions.  I see two that have come in, first from Wesley.  How would you address a risk that could occur after construction?  For example, a design that the private sector supported for economic development potential may not be the safest design and could result in operational lawsuits over the long-term. 

Patrick DeCorla-Souza - That's a good question.  I've got Jim Sinnette on the line.  Jim, any thoughts?

Jim Sinnette - That may be one of your unknown risks.  I mean, that's something that you may want to talk about when you do the risk assessment and it doesn't have to really cover construction risks.  It could cover operation risks and that would be something that would be probably tough to predict.

Patrick DeCorla-Souza - Thanks, Jim.

Michael Kay - And from Irene, with regard to qualitative assessments, what is the base that you compare and rate the risk with?  In other words, low compared to what?

Patrick DeCorla-Souza - Well, if you look at the matrix on the top and to the extreme left, you have one, two, three, four and five and one is the lowest and it says if the cost impact is less than 1 percent, that's rated low.  If the cost in fact is higher than 25 percent, that would be rated high.  So it's basically what you think the impact might be that helps you identify the rating to use.  So it's really a comparison of the impact relative to the baseline cost, and in the case of schedule, it is how many days.  If it's one or two days, that's not much less than a week, not much, but if it's a year, that's a significant delay.  That's a high risk.

Michael Kay - Patrick, we had a question from New York State with regard to showing the poll results and I said that we weren't going to do so to protect everyone's anonymity and not influence the answers, but it did dawn on me that everyone may be interested in the results from our earlier poll question, so why don't we go ahead and post those just briefly.  Again, this was the question with regard to risks that may be managed at a lower cost by the private sector, so operation and maintenance risk was among the highest ones as well as construction risk and design and geotechnical.  So I don't want to spend much time on that, but I thought it was worth pointing that out, and then the others regarding contingency, it appears for the planning stage more than 15 percent is pretty typical if not 10 to 15 percent and then a little bit lower with the design phase.  So I appreciate that question from New York and I do not see any additional questions, Patrick.  So let's move on to lesson 4. 

Lesson 4: Risk Allocation

Slide 44 - Risk Transfer Principles

Patrick DeCorla-Souza - Well, this is really something that you have to do.  It's not that P3-VALUE does it for you.  So all we are going to talk about is some risk transfer principles here.  First, you do not transfer all project risk with a P3 and only those risks that can be managed better by the private sector are the ones that transferred over to the private sectors.  So it does mean that the public agencies still will be retaining some risk and maybe quite a lot of risk and it's important to remember that risk transfer is not free.  The private sector is going to charge you for it.  I mean, there's a price and that'll increase the bid price but of course, you have to remember that you won't have to come up with all these extra contingencies and have them available because now you transferred that risk to the private sector.  Now certain risks when you transfer them, the private sector is very capable of trying to reduce the impact of these risks and often does a better job because it has an incentive to save money.  So it incentivizes better management of risk, and some risks, neither party is capable of managing and neither party has control over force majeure, for example.  So in that case it may be appropriate to share those risks and those risks have costs that vary over time.

Slide 45 - Financial Impact of Risk Transfer

We talked about how as you move closer to construction you have better information and those risk values may actually be lower or may be higher.  The benefit of risk transfer is graphically explained here.  You can reduce a lot of these individual costs of construction and operations and overall capital costs.  The amount you have to borrow can be reduced if you can reduce all of these costs and that could have an impact on your cost of capital.

Slide 46 - Risk Transfer by Procurement Type

Different procurement options are displayed here.  In a design-build, you only transfer design and construction risks, but in a design-build-finance, things such as interest rate risk are transferred to the private sector.  In the case of DBFOM, with an availability payment you are also transferring operation and maintenance risk.  You know, if there were any latent defect for example, in a DB the public agency would still be responsible after a year or so of warranty.  But in the case of a DBFOM, that risk is transferred to the concessionaire.  DBFOM with toll concession is a very risky and if you could get the private sector to take it, it might be good.  Of course, they're going to charge you extra in terms of risk premiums.

Slide 47 - Port of Miami Tunnel Risk Allocation Example

Port of Miami Tunnel was not a toll concession, but this matrix is instructive, it gives you an idea of how the Florida Department of Transportation (DOT) distributed the risks, or allocated the risks, between itself and the private sector.  Some risks, as you can see, planning and permitting and utilities are shared because neither party could better manage those risks and force majeure is a shared risk, but on the other hand construction, operations, and hand-back are 100 percent private risks.  So I turn it over to you, Michael.

Slide 48 - Test Your Knowledge

Michael Kay - Thank you, Patrick, and we have one more test your knowledge question, again, it's true/false.  The goal in risk allocation is to transfer as much risk as possible to the private sector in a P3.  Again, true or false, the goal in risk allocation is to transfer as much risk as possible to the private sector in a P3, and we'll leave that open for another 10 seconds or so, and again, please do submit questions to the chat box if you have any questions for Patrick or for Jim, and we'll go ahead and close that out and a majority of our participants answered that correctly.  The answer is false.

Patrick DeCorla-Souza - That's right.

Slide 49 - Questions?

Michael Kay - We'll take a couple of questions and comments as well.  First, a comment from William, four up from the bottom: "once one makes the risk transfer, it is critical that a dispute resolution process is in place and agreed to prior to the contract being signed."  So I think we would agree with that and may want to look into adding a note about that in future iterations of the webinar.  A question from Abe, "when you pass the risk to the private sector they charge you for it.  If the risk does not materialize, you pay for it.  Isn't that in and of itself a risk to be considered?" 

Patrick DeCorla-Souza - I guess, but it's just like your insurance premium.  You pay your insurance premium to your home owner's insurance company and you pay it to the auto insurance company and you don't hope for an accident.  So, if they are able to manage that risk and it doesn't occur, that's part of the insurance.

Michael Kay - Excellent.  A comment from William, "one critical risk is the owner changes their mind about the project.  This risk will cost the public sector a great deal.  P3s require the public side to know what it wants from the beginning and that it is negotiated in the contract."  That's a good point there, William.  Thank you and it looks like we have a question also from William.  "If one does a design-build or a DBFOM, which has finance, operations and maintenance, what is the Federal Highway Administration's position on the public sector's use of long-term warranties or guarantees?"

Patrick DeCorla-Souza - That's something I will see if Jim Sinnette can answer. 

Jim Sinnette - I actually don't know if we have a position, whether we support the use of long-term warranties or guarantees.  That would be something that would be up to the owner and a decision that they would make that would be in the best interest for the owner.

Patrick DeCorla-Souza - Thanks, Jim.

Michael Kay - Thanks indeed, Jim.  With that, we will move back to our presentation for lesson 5.

Lesson 5: Risk and Value for Money

Slide 51 - What is Value for Money?

Patrick DeCorla-Souza - We figured out how to value risks.  Now we just need to figure out how to incorporate that into our value for money analysis.  First I need to explain a little bit about what value for money analysis is, we will have a whole webinar on it on July 11th, but here is a real mini-version of what value for money is, an optimum combination of life cycle costs and quality of a good or service to meet the user's requirements.  So if you're going out to buy a refrigerator, you're looking not just at the cost you pay up front, but you're looking at electricity over the 10-year-life and then making comparisons between refrigerators that meet your requirements.  I mean, if it's too small, that doesn't meet your requirement, for example.  So that's basically what value for money is, it's looking at the cost difference, it's purely an evaluation of costs.  It's important to understand and we are doing some research on how to incorporate benefits into P3 evaluation.  So VfM analysis looks at only financial impacts.  For example, if you have user benefits as a result of delivering a project early, you do not try to monetize and include that in value for money analysis, at least that has not been done so far and that's why they're looking at benefit-cost analysis to help accomplish that.  VfM compares the present value of costs.  A important thing is all of these payments and costs and revenues are occurring throughout the life of a project.  So if you want to compare them, you have to all bring them down to year zero, that is today, in order to make a fair comparison and that's where you consider the time value of money, which is done using a discount rate, which we will talk about in future courses, and of course, the important thing with regard to this webinar is it considers the transferred and retained risks under different procurement options.

Slide 52 - Value for Money Analysis Steps

So here are the steps very briefly: you identify your options, the public option and the private option; you identify, monetize and allocate risks, which we have just done; and you apply these values to the cash flows over the project life cycle (and you do this both for the Public Sector Comparator and the P3 option).  You'll have to attend the next webinar for information about that, so all we're going to show you in this lesson is how to calculate the values of these expected cash flows for risks.  So you discount the cash flows from future years and then you compare the Public Sector Comparator to the P3 option and consider qualitative factors.  So that in a nut shell is value for money analysis.

Slide 53 - Hypothetical Example

So we're going to show you in this lesson how you actually come up with the present value of risks that occur over the project life cycle.  So we will look at a design-bid-build as a conventional procurement, we will look at a DBFOM for the P3 option, and we will look at availability payment and toll, so two P3 options.  The assumptions are that we will transfer risk to the P3 concessionaire and those risks will be managed at a 50 percent lower cost.  So they will cost the concessionaire only 50 percent of what it would cost the public agency.  Again, this is just to make calculations simple.  It is not by any means what is out there for actual experience.  The P3 concessionaire may increase total revenues by 5 percent for a toll concession, which is an opportunity.  Again, I don't know whether that is realistic or not, but this is just to see how to do the analysis, and finally, we will discount the future cash flows of the public sector borrowing rate, which we assume to be 5 percent.

Slide 54 - Valuation of Transferrable Risks

So here is an example we calculated from our P3-VALUE Risk Assessment Tool, where we got the value of threats under the conventional procurement to be $100 million, and because the P3 option was able to manage it at 50 percent lower cost, it's only $50 million for availability payment for the total concession.  Now, the first thing is to distribute these costs over the construction period.  So you have 20 percent in year one, 70 percent in year two, and 10 percent in year three.  Likewise, the same costs are distributed in the P3 options, but they're lower costs because they're lower by 50 percent, and then you have the opportunities, which are the opportunities to get toll revenue, which is positive.  So you got a plus $200 million, which is distributed $10 million per year from year 4 through year 23.  So these are your inputs to help you calculate the present value of transferrable risks.

Slide 55 - Nominal Cash Flows of Transferrable Risks

So first step is to calculate the nominal values.  If you recall, I said that the P3-VALUE output will be real values, that is the value today.  Well, you have something called inflation and if you assume a 3 percent inflation rate, that $20 million in year one becomes $20.6, the $70 million becomes $74.3, et cetera.  So you calculate these, that are called nominal values, when you apply the inflation rate, the result is called a nominal value.  You apply them to all the numbers you generated in the previous table and you come up with estimates.  For example, in the last column for P3-Toll, the $10 million now is $11.3 million in nominal dollars and it goes up to $19.7 million in year 23 because of inflation.

Slide 56 - Present Value of Transferrable Risks

So now we need to get present values.  The way you get present values is you apply a discount rate, and I said we are going to apply a 5 percent discount rate, which is the borrowing rate of the public agency, and we'll further discuss discount rates in future webinars, but this is simply illustrating how in P3-VALUE the risk costs are accounted for in the cash flows in the financial model.  So now you remember the $20.6 million nominal cash flow is discounted by 5 percent and it becomes only $9.6 in present value.  Likewise, the $74 million becomes $67.4, et cetera.  So now we've got present values of all these risks after we've applied the discount rate.  You do the same for the opportunities and you see that what was $200 million that we had estimated, is now only $155 million because the discount rate of time value of money reduces the value of revenues that we are receiving in future years.

Slide 57 - How Does Risk Affect VfM?

So how does value for money take all of these into account?  Well, the Public Sector Comparator, or the public option, has the lowest initial cost because it has a lot of retained risk, which is then increased for cost and make it a lot higher than the other options, and the reason this happens, you see the initial cost in the P3 availability model is actually higher than the PSC, but you have transferred a lot of the risks that are incorporated into that initial cost and so you are left with very little retained risks that the public sector has to pay for and other costs are things such as procurement cost and monitoring cost.  So overall, you find that the P3 availability may actually turn out to be less expensive even though the initial cost is higher than the PSC.  And finally the P3 toll option you might find is actually lower than the P3 availability in terms of initial cost because of all those extra revenues that are available to the concessionaire, and because of that, the concessionaire might be willing to reduce its payment requirements and so the retained risk is still the same as in the case of the availability payment.  So overall, the P3 toll, at least in this hypothetical example, which demonstrates how all of these risk values impact the tolls, what you see is adding the risk cost can make a difference to the final comparison.  So Michael, it's over to you.

Slide 58 - Audience Feedback

Michael Kay - Thank you, Patrick, and I wanted to get everyone's feedback regarding Value for Money Analysis, whether you have done one before in your agency, either personally or whether your colleague has.  So the question is quite simply, has your agency conducted a value for money analysis, and we'll give everyone a little bit of time to answer that, another 15 seconds or so, and after that we'll take questions very quickly and then we want to move on to our last substantive lesson on using the Risk Assessment Tool.  But I'll go ahead and close that out, and among those who were sure whether or not their agency had done one, about three-quarters had done one.

Slide 59 - Questions?

I will go back to our Q&A layout.  We had a question from Irene related to value for money analysis in general.  The question is how do we account for inflation risk in value for money, and if Irene needs to clarify for us, please do. 

Patrick DeCorla-Souza - And Jim, I know you've dealt with that in your cost estimate reviews, and would you be willing to talk about that?

Jim Sinnette: Yes.  You caught me a little off guard.  I was looking back to our long-term warranties and guarantees policy, but with respect to inflation risk, what we normally do is, a lot of the state DOTs have offices that actually predict inflation for construction projects and a lot of times we use that and there's an uncertainty to that inflation rate so, and a lot of times some state DOTs actually give us that uncertainty rate.  So they would say for the next two years the inflation rate varies between 1 and 3 percent, most likely 2 percent say, and then for the next four years after that they use a rate of 4 percent.  So we can model that uncertainty inflation as a risk and we do that on our construction projects.

Patrick DeCorla-Souza - Thanks, Jim.

Michael Kay - And we had a clarification from our friend, QC, that 23 CFR 635.413 reflects current regulation on long-term warranties and that they are in fact allowed in P3s.  So Jim, no need to go back through all the CFRs, but any clarification from other view on that?

Jim Sinnette: Yeah, I was just going to say, the big issue is Federal-aid participation and warranties that cover maintenance activities and it was really hard to distinguish, to make sure that featherweight funds were not participating in something that would be maintenance.  So if you can have a warranty for something that's very specific that doesn't cover maintenance activities, like you wouldn't do a warranty for a guard rail because somebody hits the guard rail and damages it, that's considered routine maintenance.  So if you follow what was in 635.413 then yes, it's fine to go with warranties.

Michael Kay - Great.  Thank you for that clarification.  So looking at the clock, Patrick, it's about quarter after.  Let's take about 10 minutes or so to talk about the Risk Assessment Tool and then we want to leave a few minutes for the course summary and evaluation.

Lesson 6: Using the Risk Assessment Tool

Slide 61 - Accessing the Risk Assessment Tool

Patrick DeCorla-Souza - Sure.  This should be fairly straightforward.  Here is the link to the Risk Assessment Tool. I strongly encourage you to use it and test it out, and if you have questions and comments, we'd like to hear from you and the e-mail address, P3-VALUE@dot.gov is the place to send your comments.

Slide 62 - Supporting Documentation

If you go to that website, we've got lots of good information: an orientation guide, user manuals, primers.  We have a Frequently Asked Questions document and Troubleshooting Guide currently under review and we should have it up there in a few days, and I just wanted to alert you that we are developing a guidebook on risk assessment and allocation for public-private partnerships and this will provide a lot more detail than what I've been able to give you here or what we have in our primers and user guides and it'll get into a lot of those issues such as discount rates and double-counting of risks and all of those difficult issues that I've left out of this webinar. 

Slide 63 - Getting Started with the Tool

So the tool is organized into several tabs.  As you see here, these are the various tabs in different colors.  An introductory one, always remember to click on enable macros and I accept.  These are the two things you have to click before you can get started on the first tab.  Then you have all the model assumptions, you have the definitions, you have that risk assessment matrix, the colored matrix that I showed you, and the risk register, which I will be showing you in the following slides, and then the outputs from the model, some of which I've already shown you.

Slide 64 - Using the P3-VALUE Tool

So here's the risk register.  You put in the number of the risk, type of risk, what phase it is in, is it a threat or an opportunity, describe the risk and the consequence of the risk.

Slide 65 - Using the P3-VALUE Tool

Then you do the qualitative risk assessment and then here is where you look at those matrices to find out what probability rating to use.  Five is obviously a very high rating.  Two as a consequence is relatively low.  The result, of course, the yellow is what the tool produces for you.

Slide 66 - Using the P3-VALUE Tool

Then you put in an input or consequence.  Again you say, two is the scheduled consequence and the yellow cell is what is produced by P3-VALUE.  Then again you have lots of inputs and all of these blue cells are actually inputs that you put into the model, probability percentage, the type of distribution for Monte Carlo simulation, minimum, most likely and maximum, both for the scheduled impacts and the cost impacts.

Slide 67 - Using the P3-VALUE Tool

And you have to tell them what your allocation is going to be, public versus private, and of course, you can run the tool twice, once for the public sector option, once for the P3 option with different allocations and you would get different results obviously.

Slide 68 - Risk Assessment Tool Limitations

I want to point out some of the limitations.  Again, the Monte Carlo simulation does not address revenue risks, so there are other ways you can address it.  I'll talk about that in the next webinar.  The Risk Assessment Tool assumes that the risks are independent with no correlation between risks, so that's something to keep in mind.  It does not aggregate the lower-rated risks.  You have to do that manually in an off-sheet calculation and then put them back into the tool if you want the Monte Carlo simulation to take account of those aggregated low-cost risks.

Slide 69 - P3-VALUE Tools

I showed this graphic earlier, so this is a recap of the outputs of the Risk Assessment Tool, the risk values are input into these other tools, which I will be talking about in the next webinar.  Next is question time, but maybe I should move on and summarize and then open it up for questions.

Michael Kay - Yes, let's do that, Patrick.

Course Summary

Slide 72 - Course Recap

Patrick DeCorla-Souza - So summary, of course we went through types of P3s, overall risk management process, I talked in a little detail about how you actually quantify risks, how you allocate risks between the public and private sector and I showed you an example of how the risks are then converted into cash values for your financial model, and then I showed you outputs from the tool itself.

Slide 73 - Resources

There are several resources, and again, any of these links would take you to these additional resources that provide more information.  I particularly encourage you to go to that second-to-last link on the Risk Assessment Tool to actually try out the tool and send us your comments.

Slide 74 - Upcoming P3-VALUE Training

And as I indicated, we will have a value for money webinar, which is also called Public Sector Comparator/Shadow Bid webinar on July 11th.  That'll be followed by a financial assessment webinar on August 7, and then we will repeat the entire series, which began with a P3 101 webinar back in April, and we are going to repeat the entire series. 

Slide 75 - Contact Information

And here's my contact information.

Slide 76 - Contact Information

Here's Jim's contact information.

Slide 77 - Contact Information

And Thay Bishop is our coordinator for all IPD academy activities, and you can contact her, too.  So that's it, Michael.  Time for questions.

Slide 78 - Questions?

Michael Kay - Great.  Thanks, Patrick.  I just wanted to go back briefly to the calendar of upcoming training, and just to mention with regard to the July 11th and August 7th webinars, registration is currently full for those webinars.  Some slots may open up so I encourage you to check back, and many of you, I'm sure, are already registered for those webinars.  But if not, they will be given again.  So I just wanted to mention that, and we look forward to having you participate in the rest of this series of P3-VALUE webinars as well as the next series.  We will now take a couple of questions and yes, TIFIA is still occurring on July 17th, that would be the office hours for TIFIA, and we were just showing the P3-VALUE training in that slide.  We will now go to the chat box and a couple questions are coming in regarding getting a copy of the presentation.  Yes, that will be provided momentarily.  I did want to take a question from Wesley.  Patrick, I'm looking about six up from the bottom.  "Can you provide some examples of P3 risk sharing that you would consider good models to review? Projects?"

Patrick DeCorla-Souza - Now Virginia DOT has published a lot of its agreements as well as its value for money analysis.  I know that Florida DOT has also done so for its two P3 projects, the Port of Miami Tunnel and I-595.  Presidio Parkway has also published its value for money analysis.  So these are some sources that you can go to.  You can just Google them and you can find them on the web.  Jim, is there any other advice you might give?

Jim Sinnette - I don't know, sometimes the procurement documents for these P3 contracts discuss the specific risk sharing and if those are available from the state DOT, I know sometimes they're hard to get, but there might be some examples there.

Patrick DeCorla-Souza - Yeah, well most P3s that have been done in the United States, their agreements are accessible online.  So, we are hopefully going to publish pretty soon some research on P3 model contracts and we've been researching these contracts.  So once we publish that, the P3 model contract information, which should be in a couple of months, you will have a listing of all the websites from which we got the P3 agreements.  So that will be one source you can look at.

Michael Kay - All right.  We've had a couple of additional questions come in, Patrick, and I do want to tackle those, but first I want to do a couple things simultaneously.  So if you'll bear with me, I'm going to open up our evaluation layout.  We're still going to take those questions that are on the bottom-left there and you can still submit, but I did want to reserve 5 minutes for people to concurrently complete the evaluation that now appears on your screen.  It's a series of 8 or 9 multiple choice questions and then an open-ended question at the end, and it really helps us to get feedback on this webinar and on our webinar series in general.  A lengthy question came in, Patrick, but I'm going to go ahead and read it verbatim: "Is a sensitivity check performed during the project planning / budgeting process to understand the risk associated with greater than anticipated traffic volumes from impacts to the infrastructure from an O&M perspective (for example, pavement, since the design is based on forecasted data and not based on the "what if" sensitivity data)? In other words, is an analysis done (or should it be done) to evaluate if additional revenue generated because of greater than projected traffic volumes adequate to cover the life cycle maintenance cost for the impact. If revenue is not adequate, should that risk be identified and allocated?"

Patrick DeCorla-Souza - Again, a good question.  I don't know of any such analysis that has been done, but I would defer to Jim.  Jim, have you heard of such analysis?

Jim Sinnette - No.  We haven't done anything like that, but it is an interesting scenario and again, normally when we do our cost estimate reviews, we don't really look at the operation and maintenance portion of it. But yeah, that's an interesting question.

Patrick DeCorla-Souza - Right.  I mean, I come from a traffic modeling background and I know that if there's extra traffic on a toll project everybody is really happy, nobody is unhappy.  So it appears that the extra toll revenue that comes in more than compensates for any extra road damage caused by the extra traffic, but of course that's not based on any analysis.  It's just based on the fact that nobody has complained about extra traffic on toll projects.

Michael Kay - Thanks, Patrick, and I would be remiss if I didn't mention it's now at this point that you can also download today's presentation.  I know we had a lot of questions about that.  It's available on the top-left of your screen, there's a box there that says presentation download.  Simply click on that file name and then save to my computer and follow the prompts on your screen.  Again, that's in the top-left of your screen.  The presentation is available for download.  A question from Mohammed, is this course eligible for continuing professional education credit or professional development hours, and I would refer you to Thay Bishop, who is our Senior Capacity Builder and the leader of the webinar series.  Essentially the Office of Innovative Program Delivery is willing to attest to the fact that you have participated in this webinar, but we don't have a mechanism in place yet to gauge your understanding of the concepts and such, but if you need us to at least verify the fact that you were here and participated, IPD would be able to verify that.  But I would encourage you to contact Thay, and her contact information is included in the presentation download.  We've got about one minute left.  Let's take William's question and then we'll conclude.  "Has Federal Highway collected examples of unexpected issues or problems that states have experienced?"

Patrick DeCorla-Souza - Good question.  Jim, any thoughts on that?

Jim Sinnette - No, we haven't done that, but I think Washington State DOT has done that and I think they have put that on their website because they were one of the leaders in this risk-based evaluation of cost estimates.  So unfortunately no, we don't have something like this, but I do believe there might be something on Washington State DOT's website that has that.

Michael Kay - Great, thanks.  We're right at 3:30 p.m. Eastern.  Patrick, any parting words for us today?

Patrick DeCorla-Souza - Well first of all, I want to thank Jim Sinnette for taking time out of his busy day to help answer the questions on this webinar, and I want to thank you, Michael and Victoria Farr, for helping coordinate this webinar and also the operator who has been helping out on the phone.  And of course, thanks for all the questions from the audience that helped make this more interesting for us and I encourage folks to come to the next webinar on July 11th.

Michael Kay - Thanks, Patrick.  Real quick for Linda, make sure you click on the file name first, P3 Project Risk Assessment, then click save to my computer and that should work.  Thanks everyone and have a great afternoon.

Patrick DeCorla-Souza - Thank you.

Operator - That does conclude our conference for today.  Thank you for your participation and for using AT&T Teleconference Service.  You may now disconnect.