Presentation: PDF HTML
Homework Assignment 4: Risk Assessment for a Non-tolled Project - Assessing Risk
Webinar recording: Audio
P3-VALUE Webinar - September 20, 2013
P3 Program Manager
Office of Innovative Program Delivery
Victoria Farr: On behalf of the Federal Highway Administration's Office of Innovative Program Delivery, I would like to welcome everyone to today's IPD Academy Webinar, P3 Project Risk Assessment. My name is Victoria Farr. I'm with the U.S. DOT's Volpe Center in Cambridge, Massachusetts and I will be moderating today's webinar as well as facilitating our question and answer periods and addressing any technical problems along with my colleague Aaron Jette. Our presenter for today is Patrick DeCorla-Souza. Patrick is the P3 Program Manager in the Office of Innovative Program Delivery. Our webinar will run until about 3:30 P.M. eastern today and the course is divided into six lessons. We will queue up any questions for Patrick at the end of each lesson and as I mentioned, we anticipate having a few minutes at the very end for additional Q&A. Today's presentation and related materials including a homework assignment will be available for download at the conclusion of the webinar and further instructions will be provided at that time for download. I also wanted to let you know that we are recording today's webinar so that anyone who is unable to join us may review the material at a later time. Finally, we have four quick poll questions that we'd like you to answer just to help our presenter better understand our audience today.
The review has changed slightly. In the upper left hand corner we are asking "What is your affiliation? Are you with the FHWA Division Office, FHWA Non-Division Office, Safety O.T., other non FHWA federal agency, other state agency, N.P.O., private sector, or other?" And in the upper right hand corner, "How many people are participating with you today? Are you alone or are there as many as up to ten or more people with you in a conference room?" The bottom left hand question is "What is your current level of knowledge and experience with risk assessment? Whether you have no prior knowledge or experience, a basic understanding, some knowledge and experience, a lot of direct knowledge and experience but have more to learn, or if you are an expert." And then on the bottom right "Which of the following P3 value tools have you accessed?" And this is a check all that apply question and Patrick will talk more about the P3 Value Tool Kit but for those who have participated on prior webinars or checked out the I.P.D. website, let us know if you have accessed the risk assessment tool, the public sector comparator tool, the shadow bid tool, the financial assessment tool, or if you have not accessed any of those tools. So I will close those out and at this time I will turn things back over to Patrick who will begin with a brief introduction. Patrick?
Patrick DeCorla-Souza: Thanks, Victoria and good afternoon, everybody. I'm going to first talk a little bit about FHWA's P3 toolkit which the risk assessment tool is a part of. So the public private partnership's toolkit is an educational tool and it's an analytical tool to help users understand how risk assessment is done and how value for money analysis is done. The overall toolkit of the Federal Highway Administration on public and private partnerships also includes other documents relating to legislation and policy, planning and evaluation, procurement, and monitoring and oversight and the P3-VALUE tool which the risk assessment tool is a part of is part of item two there, planning and evaluation.
We have been scheduling several webinars to get P3-VALUE out to the state and local government and so this webinar is the first of a series of webinars on the tools themselves. We did have one webinar which provided an overview of P3 evaluation on September 5th and the recording of that webinar is available by accessing the link shown on your slide. You will be able to download these slides at the end of my presentation. As I said the first tool is the risk assessment tool and we're going to talk about it today. We also have scheduled a webinar on value for money analysis in January and financial structuring and assessment in March but we have had webinars on these two topics previously in July and August. And if you can't wait until January and March, feel free to listen to the recording which is available by accessing the link shown on the slide.
So the P3-VALUE tools consist of four tools. The first one is the risk assessment tool that I will be talking about today. And it assists in identifying risk, risk allocation, developing risk response strategies and evaluating cost and schedule impacts. The public sector comparator, shadow bid, and financial assessment tools are all related to value for money analysis and as I indicated we will be talking about them in future webinars.
So here is how the risk assessment tool fits in with the set of four tools. You first do a risk assessment from which you are able to develop values, cost estimates of risks, and allocation, percentages, public versus private which then are input into the value for money analysis tools as shown in this slide. The public sector comparator tool takes in risks that you estimate for the conventional procurement. The shadow bid tool takes as input any risk values that you've estimated for the P3 procurement, and then these are combined with other analysis processes in the tools and the output of those two tools which are the public sector comparator and the shadow bid are compared in the financial assessment tool and the tools are available at the link shown below.
So this provides an outline of what we are going to cover today. First I will do a very brief overview of P3s and risk. Much of it is a review of what we have already discussed in the earlier webinar on P3 overview of P3 evaluation. I will provide an overview of the risk management process in lesson two. And then we will go on and discuss in detail two components of the risk management process that is risk analysis and valuation, and risk allocation. Finally I will demonstrate how the risk estimates that you produce are incorporated into value-for-money analysis. In the final lesson I will introduce you to the P3-VALUE risk assessment tool and provide a homework assignment which will help you get familiar with the tool and then we have also scheduled a complimentary webinar two weeks from today where we will review the homework assignment results.
The slide here shows what you should be able to do by the end of this webinar. You should be able to identify different types of risks, categories of risks in the lifecycle of a project. You should be able to list the steps in the risk management process. You should be able to explain the methods that I use to quantify and monetize the values of individual risks. You should be able to explain how financial impacts of risks are incorporated into value for money analysis. And finally you should be able to access the P3-VALUE tools and be in a position to begin using the P3-VALUE risk assessment tool.
Slide 9 - Common Types of P3s
So this is the first lesson and this slide simply provides a recap of what we have discussed earlier, different types of P3s. P3s may be broken down into new facilities called Greenfield or existing facilities called Brownfield. Greenfield projects might be split up into two types of P3s: design-build with financing and design-build with operations and maintenance in addition to financing also called DBFOM. And DBFOM which is a long term contract with a concessionaire may be of two types. It may be a toll concession where toll revenues all go to the concessionaire or it might be an availability payment concession where the public sector pays the concessionaire periodically usually annually or semiannually through availability payments throughout the term of the concession. And if there are any tolls those tolls go to the public sector.
Slide 10 - Potential Benefits and Drawbacks
Again, this is a recap of benefits and drawbacks of P3s. We discussed before the main reasons that states are looking to P3s are because they can provide additional financial capacity to build major projects. They could reduce costs over the entire project lifecycle, and they provide an opportunity to transfer risks which the state may not be willing or may feel that the private sector can manage better. Drawbacks also exist. There is some loss of flexibility to the public agency. The procurement process can be long and complex and requires a lot of expertise, and P3 introduces additional financing costs which in a way are compensation to the private sector for taking on extra risks from the public sector.
Slide 11 - Purpose of Risk Assessment
The reason we do risk assessment and we do it at several points in the development of a project so several purposes of risk assessment are listed here. First, if you want to understand that the project is financially viable you need to know what the project would cost, not just base cost but also what the potential risk cost might be so that you can compare it with the revenues you have available to pay for the project. So risk assessment is extremely important to understand if the project is financially viable. If you are trying to figure out whether a P3 would provide good value for money, that is it would be a better deal for the public sector, then you also need to do a fair comparison. As I indicated earlier some of the risks are transferred over to the private sector and get reflected in the financing costs. So value-for-money allows you to account for risks that are transferred so that you can do a fair comparison between the P3 and the public sector conventional delivery. Understanding the risks in a project is extremely important in crafting your draft P3 agreement before you issue an R.F.P. for a P3 procurement. Finally after you select a bidder that provides best value, there may still be points to be negotiated and understanding the costs of risks at that point will help you negotiate a lot better and get the best deal. And finally knowing what your risks are and their impacts and possibilities and probabilities helps in monitoring risks over the project life cycle and helps you address any risks should they occur by preparing for them.
Slide 12 - Financial Impacts of Project Risks
When you are doing financial evaluation such as value-for-money analysis, risks, understanding risks help you get a good handle on the costs, the true costs both for capital and operation and maintenance. You also on the revenue side if you understand the risks of toll revenue for example, it helps you understand the potential financial impacts should some unforeseen event occur or even if things that you thought might happen actually do happen. And finally any risks that affect schedule, delays for example can affect cost. You might be faced with higher inflation, other indirect costs, and if this is a toll project, you might also suffer a loss in revenues. So that's why you need to understand your risks to the extent possible.
Slide 13 - Categorization of Risks
This is a categorization of risks that try to focus on what risks you might actually be able to value or quantify and monetize. So if you look at a project basically there are two types of risk: exogenous risks, that is risks due to some external event that you don't have a control over for example an accident on the construction site caused by some drivers not obeying signs, for example. Or there might be risks that are endogenous in the sense that some decision maker may change their minds because of some opposition to a certain feature of a project and therefore the scope may change, etcetera, So what we are talking about is in risk valuation is generally the exogenous risk because the endogenous risks are something you would deal with in scenario analysis and not in the process that we are going to talk about for project risk analysis. Then when you get to these exogenous risks you also have two types. One type is process risks and the other project risks. And the process risks are risks that occur or that could occur any time prior to the award of a contract to the P3 concessionaire. So those are risks that obviously you need to be aware of, to manage etcetera, but when you are talking about the P3 and risks that you might transfer to the concessionaire, all that you are concerned with for your evaluation is risks that might occur from the point at which you award the contract. So those are called project risks and those are again of two types. They might be systematic risks or non-systematic risks. And again, the systematic risks are risks that you don't have control over. An example is inflation, market financing rates, interest rates, and the economy etcetera. The non-systematic risks are risks that you may have some control over. Either the probability of the risk happening or you might have it via the ability to manage the risk, to reduce its cost. Systematic risks which are these market risks, you really have no way of changing what happens to the economy for example, but on your specific project perhaps by taking better precautions on the site, you might be able to reduce the probability of drivers having an accident for example. So that would be a non-systematic risk. Now non-systematic risks may also be categorized into two categories. One category is your regular uncertainties where you know that you are going to have a cost for example to build a culvert but you don't know the size of the culvert because it's too early in the process. Detailed designs haven't been developed etcetera so you can't really estimate the exact cost so that is something that is categorized as regular uncertainty. You know you need a culvert, you just don't know exactly how much it might cost. Pure risk on the other hand is something that may not happen. There's a chance it will happen, there might be a small chance, there might be a high chance, but you don't know for sure. And again, giving the example of the accident, you don't know whether an accident is going to happen and you might be able to estimate the probabilities based on past projects and you might be able to estimate the cost impact to your project again based on prior data that you might have for similar projects and so you might be able to estimate some cost for that risk.
Slide 14 - Project Risks
So project risks as I said are the risks that we are going to try to value and they include bold systematic and nonsystematic risks. And in P3 this graphic shows you how these risks might be distributed between the various parties to a P3. So the public agencies or the public sector and the partners in the P3 which include the special purpose vehicle which is the concessionaire, the equity providers and the banks that provide debt all of these in some manner shoulder those risks shown on the right hand side. So inflation risk, interest rate risk, which are basically systematic risks based on the economy that nobody has control over. But then there are some risks such as performance of the project, integration of the various components of the project, design-build, operations and maintenance, coordinating the various contractors, subcontractors. And if this is a toll project, there might be toll risk which might be project specific but some of it might also be related to the economy so it's partially systematic and partially nonsystematic. And then below the dashed line you see the subcontractors to the concessionaire, so you've got the design-build consortium, you have a maintenance contractor and you may have a toll operator. All of these take on some responsibilities through agreements with the concessionaire. And to some extent the concessionaire transfers down a lot of the project risks to these subcontractors. So to a large extent these subcontractors include the cost of the risks in their prices that they quote to the concessionaire. And so what we mean by that is the risks are included in the cash flows. On the other hand the risks above the dash line are a little different. You might not be able to include all of those risks above the dash line and the cash flows of the concessionaire for example. Toll risk is something that is difficult to include in cash flows so it might be dealt with differently. Same way with inflation risk and interest rate risk so there might be different ways of addressing those perhaps through discount rates or financing costs.
Slide 15 - Regular Uncertainties vs. Pure Risk
Now we talked about regular uncertainties versus pure risk. Regular uncertainties again it's something that has to be estimated so you can set aside an allowance within your cost estimate and it is a function of where you are in the design process. If you are at the planning stage you have very little information. Once you are detailed design stage you might have more information. So based on that, the allowance may be bigger or smaller depending on the stage of the process. Risk is different than uncertainty in the sense that it may or may not occur and also it can be either negative or positive. So by that I mean if something negative happens, you would have to account for it as a cost. On the other hand if there's a positive effect, you know, for example you might get lower bids because the market there's a lot of competition in the market because the contractors are hungry whatever, that's a positive from your standpoint and that would reduce your cost and that's an opportunity, positive from your standpoint. So what we do is estimate the impacts of these risks based on probability and estimate of the impact they might have and then we calculate a contingency amount. So that's a little different from the allowance, but it is an amount that needs to be accounted for.
Slide 16 - Identified vs. Unidentified Risks
And here we get to the difference between identified and unidentified risks. The identified risks are the risks that you know about. You know for example in the past there have been accidents on the construction site so you know there's a possibility but you're aware of it. But there might be some risks that you have no way of figuring out whether they might happen or not. It might be for example the earthquake that happened in Washington D.C. a couple of years ago. I mean nobody would have forecasted that and not much damage was done but these are the kinds of things that you would not be able to identify and they are sometimes called unknowns. So both types of risk however, need to be considered in your contingency amount. The identified risks you can account for and calculate the potential value of those risks by creating a risk register and to an estimation process that I will talk about a little later, but unidentified risks, you really have no way of estimating how much contingency you should put aside for them. So it's only by looking at some past projects that you might come up with some kind of ball park estimate. But this is important because the P3 concessionaire has to account for not just these identified risks which some of them they might be able to push down to their subcontractors, but they also have to include in their bids an amount for these unidentified risks which really is in a way the premiums that are charged for financing. So here is really the rationale for why the financing costs are higher because that is the way the P3 concessionaire is able to recuperate any cost that they may incur due to these unidentified risks.
Slide 17 - Base Estimate vs. Risk Contingency
This graphic visually shows you how these estimates might vary based on the phase of the project you are in. So the planning phase you don't have a lot of information and you might make a cost estimate in that the arrow there, the red arrow shows that you might have a base estimate and then add an allowance for the fact that you don't really know exactly what the quantities required might be. So that's your base estimate, and then you need to calculate a contingency which would incorporate both the identified risks and the unidentified risks. For a conventional delivery, you would need to make sure that you set aside a budget for each of these components or a sufficient budget to account for and support all of these components of risk.
Slide 18 - Risk Premium over Concession Term
On the private sector side, as I said the way the concessionaire accounts for these risks is through risk premiums, and these risk premiums are added to the financing costs. So you might see for example equity in the early development phase of a project. The equity risk is high similar to what we saw in that graphic and the equity participants or those who provide equity funding will demand a higher return. So that's what this is reflecting. As you go further down into the construction phase and then you transition, you begin operating the project, there's a ramp up and tolling begins and then everything settles down, everything settles down and then risk is reduced in the operations phase. So what you're seeing here is if the equity providers at that point want to sell their equity shares, the new equity providers at this point will not require a high premium so their rate of return required at that point if they enter the concession as equity providers at that point and this is generally what happens in a P3 concession, those who provide risk capital early in the project they might sell their equity stakes to other equity investors such as pension funds or insurance companies that don't want to take a lot of risk but want a stable rate of return and so that's what you see here is the premium required at that stage commensurate with the fact that the risk is reduced is also lower. That concludes this lesson. I turn it over to you, Victoria.
Victoria Farr: Thank you, Patrick so I'm going to modify your view slightly and just so you know we do have a few audience input or poll questions for you to answer at the end of each lesson today or at least after the first five lessons. So the questions on your screen are in regards to "On average how much contingency does your agency include in cost estimate to account for risks on top, the planning phase and on the bottom in the design phase?" So it's the same question but we're asking it for two separate phases of project delivery and I'll give you about ten more seconds. We will next turn to a Q&A period. There are currently no questions in the chat box but just a reminder that you can use that chat pod in the lower left hand corner to submit your questions to Patrick throughout the webinar and we will respond to them at the end of each lesson. So I am going to close out the polls now and we'll take a look at the results. So let's see here, it looks about the majority of respondents or about a third of the respondents said that in the planning phase the accounts were about ten to fifteen percent and another third more than fifteen percent. Patrick do you have any comments on the results here?
Patrick DeCorla-Souza: No I think it looks good. I mean it's good feedback to know what agencies are using.
Victoria Farr: Okay so I do see that someone is typing in the chat box. We did just get a question that came in so I'm going to now convert to our Q&A layout so it's going to expand the question box. The question is from Steve, Patrick. He says that "You mentioned an earthquake risk event but this is a very low probability, very high impact event so why model it?"
Patrick DeCorla-Souza: That's correct, I don't think you would model an earthquake risk but I will defer to Jim Sinnette who is the Major Projects team leader here in our office and, Jim, do you have any thoughts on that?
Jim: No, I agree, I normally you wouldn't model an earthquake event because it's certainly hard to predict and the impacts can vary greatly obviously. But there are some events and I think you'll talk about these later that have low probability and high impact like for example finding archeological sites. You know, if you had no idea there was no surveys done or anything but you sort of knew that the area could potentially have an archeological site, you may assign a low risk, low percentage, and low chance of that occurring but then it obviously would have some significant schedule impacts and I've seen things like that modeled before. But no I agree with you on the earthquake and probably not something you would model as a project risk.
Patrick DeCorla-Souza: Great. There's another question, Jim, "What type of risk is covered by management reserve?"
Jim: Normally those are your unknown-unknown risks that you are talking about. There was I'll use an example of let me think of a good one. I'll use a simple example, not from a major project or anything but you may be doing work and you are doing a lot of earth work in the area and the houses next to you have in-ground swimming pools and because it's a high profile project and dirt is getting in everybody's swimming pools. You're going to have to go out and buy pool covers for some of the people there. So that couldn't have been predicted. Normally it's not predicted. Usually management reserves cover those for the political risks not related to the project but something that's made at an executive level so that could be an example. Another example is you are doing a project and you're having a significant impact on the city transit to city bus system and they have to make some adjustments in the routes and move some bus stops or whatever and again, it's as a result of the project. Again those decisions are usually made at a higher level so normally for the large complex projects, a management reserve is used to cover those types of risks.
Patrick DeCorla-Souza: Thanks, Jim.
Victoria Farr: Yes, thank you, Jim, very much for your input. So seeing no other questions in the chat pod, Patrick we will move on to lesson two but just a reminder that you can input questions throughout this lesson and we'll respond to them at the very end of this section. Patrick?
Patrick DeCorla-Souza: Thanks, Victoria. So this lesson is simply to give you an idea as to where risk analysis fits, in the risk management process. And as this graphic shows there are five steps starting with identification. Analysis is the second step. Mitigation and planning for mitigation is the third step. You then go to risk allocation, and finally monitoring and control.
So in the identification step what you would do is normally hold a workshop with a facilitator and subject matter experts who can raise or identify risks in their area of expertise. And then you keep track of all the risks that are identified in a checklist or risk register that I will be presenting in little bit.
The second step is risk analysis and valuation and this is really what the P3 value risk assessment tool is designed to do so I'll spend more time on this later on. But briefly, the purpose of this step is to attempt to get an estimate of the value or cost of a risk.
In risk response planning you try to figure out what you should do to respond to a risk or to prepare for a risk. I'll give you an example of driving. You want to avoid an accident so you might drive carefully, slow down your speed. If you want to mitigate you still might have an accident. Somebody might hit you so to mitigate that you might make sure you wear your seatbelt in case you do get hit. And finally the third possible response is you might have high cost. Your car might be totaled, you might not be able to afford it so you get car insurance that will pay for your car in case you get into an accident. So you are transferring that risk over to the insurance company in that case. And similar things are what you can do with your various phases in design build and operations in a project.
Risk allocation is the process whereby you decide which risks you should transfer or you could transfer over to a concessionaire. Those are called transferable risks. The ones that you retain because you don't think the concessionaire could manage them any better than you could, those are called retained risks, and then there are some risks that might be shared. And again we will talk more about risk allocation later.
Finally once you've developed this risk register, you monitor the situation, the project as things go on. Some risks may no longer exist. You passed the phase or whatever so you update your risk register periodically and you use it to monitor potential risks and be ready in case they do happen. In a P3 one thing to be very careful about is to make sure that when you transfer a risk, you actually do transfer it to the private sector because it's possible that you might take back a risk that you transferred if you tell the concessionaire to do something a certain way and then the concessionaire can come back and blame you if something happens even though that was the concessionaire's risk. That concludes this session. It's a short one. Victoria? On to you.
Victoria Farr: Great, thanks, Patrick. So again, I will change your layout just slightly and this is a multiple answer question. "In your view, which of the following risks may be managed at a lower cost by the private sector?" So again you can select more than one option here and we'll give you about ten more seconds to answer that and then we'll broadcast the results. I'm going to close that now and broadcast. So it seems like the top three risks that people think might be managed at a lower cost include construction, operation maintenance, and design/geotechnical. Patrick do you have any comments on our responses that we received?
Patrick DeCorla-Souza: Not really. How about you, Jim?
Jim: I think those are pretty typical.
Patrick DeCorla-Souza: Good.
Victoria Farr: Sorry, Patrick?
Patrick DeCorla-Souza: Okay.
Victoria Farr: Seeing no other questions in the chat pod I think it's best to move on.
Patrick DeCorla-Souza: Alright.
Victoria Farr: Great.
Patrick DeCorla-Souza: Alright. Okay so now we are getting into the lesson which will really introduce you to how P3 values risk assessment tool operates and that's what this lesson is going to talk about, the risk assessment process as used in P3 value. To give you an overview, you first would do a qualitative assessment, basically get a rating one to five where very low is one and very high risk would be five. And then you take the medium high and very high rated risks so the three highest levels of risks and you do a quantitative assessment on them. So the ones that are lower, that is the very low and low rated risks, even though they may not be very important, you might still want to deal with them by aggregating them so you at least account for them in your budget or contingency. And it's a challenge though to figure out how to figure out the range of impact and the probability of an aggregate risk occurring. The quantitative step may be done in one of two ways. One is all the deterministic or formula-based calculation and the other is called simulation which is called probabilistic or stochastic simulation. In other words it's using probability estimates and distributions, probability distributions to try and get a range of potential estimates of risk cost. So the results of this estimation process would be cost estimates, delay estimates, or revenue impact estimates.
As we indicated earlier there might be a qualitative analysis or there could be at the second stage a quantitative analysis which might be of two types: deterministic or probabilistic.
And the inputs depend on the type of analysis you are doing. If it's qualitative you just need a five scale one to five as shown there, very low, low, medium, high, very high. If you're doing a quantitative analysis your probability of occurrence has to be specific but you need to have a specific probability identified you know, 25 percent, whatever it might be. With regard to scale of impact, you would again in a qualitative assessment simply have a rating scale, very low to very high. Quantitative assessment, you nail down the exact dollar amounts so you nail down the range of potential impacts in dollars or in number of days of delay.
The way you do a qualitative assessment is depicted in this graphic here. The upper matrix is a matrix to do cost qualitative assessment and the lower one gives you scheduled assessment. And you might see the scale goes from five down to one but five being the most severe for the percentages for the costs range from about twenty-five percent of the base project cost that you have estimated down to less than one percent and then the probability ratings again, five down to one and there's a guideline of roughly what five means. Five means something seventy percent or higher, and one means something in the range of zero to five percent probability. The same risk might have a schedule impact in the lower graph simply shows you what is severe or very high in terms of number of days and so one year is considered very high and less than a week is considered very low and by looking across, by looking across the rows, so for example if you look at that first row there with the five you have seventy percent probability and if you go down and you look at the lowest ranked severity of that risk which is very low in the cell there the result is that even though the probability is high, because the impact would be low that risk has very low priority or impact. And likewise you do it for the others and you can see a very high impact risk which would be five, more than twenty-five percent cost estimate gives you if it has a very low probability, zero to five percent, you still count it as a low impact risk.
So we said there are two ways you can analyze risk and value risk. This is one method, the formula-based method and there's an example of three risks here. You identify the probability. Risk one has a 90 percent probability. The cost could be anywhere between five million and twenty million. And you think that the most likely cost is ten million so that's a bit of a skewed distribution there but in the case of a formula application, you simply use the formula shown at the top, probability times, minimum plus maximum plus four times more most likely and divided by six. And use that in the formula and you come up with an impact of nine point seven five million. And you can do the same for the other risks, except in risk two there is no most likely. You don't have an estimate of-- you know you think all of the impacts are equally likely for example. If all of the impacts are equally likely, you can't use the formula at the top. You simply take the average risk impact, multiply it by the probability which is fifty percent and you get two point seven five million. So if you want to get the aggregate risk cost so that you can calculate your contingency, simply add up the costs of the three risks that you have estimated and you come up with thirteen point six million in aggregate risk cost.
In the probabilistic analysis, you have to do a simulation called a Monte Carlo simulation. And the first thing you need to know is what is the shape of your risk or the shape of the impact of the risk in terms of the potential distribution of the risk should it occur. So that's the first step. You select a distribution type. It can be normal, these are different types. In P3-VALUE we just have two types. And then you determine the impact levels. These are again the dollar costs just like we did in the formula-based calculation. You run the simulation and then you determine-- the simulation provides you with the result. Obviously you're not going to do it by hand so you need a computer to do it and that's what P3-VALUE does. It does as many iterations as you specify which we will see in a little bit.
But here are the two types of distributions that are used in P3-VALUE. You see the triangular distribution is the one where you know something is more likely. It might be either in the center or as the dashed red line shows it might be skewed towards the high end of the cost so that can affect the value of the risk. In the case of a uniform distribution that's what it looks like and you don't have a most likely value.
So here is how you would input it into P3-VALUE. You would input the probability of the risk just like for the formula-based calculations so risk one is ninety percent. You would input the consequence the minimum, five million, the maximum twenty million, and the most likely consequence of ten million. And you would tell the software what distribution you want to use so it's a triangular distribution in this case. Risk two was uniform so you see that there is no most likely since all are equally likely.
You take this information and you put it into P3-VALUE and you tell it how many iterations, in other words what it is doing is it runs, it does the calculation once for each scenario, and each scenario is comprised of picking a risk so for example risk two had a fifty percent probability so it would pick risk two for every other scenario. So maybe scenario one, then scenario three, etcetera. In the case of the impact when that risk does occur so if risk two did occur it would look at the distribution and it would try to replicate that distribution over the five hundred iteration so basically trying to use your input in simulating all of the potential outcomes for five hundred different scenarios. In this case, of course you can specify a thousand. I think it can go up to a couple of thousand iterations.
So what it gives you out of this software, out of this simulation is a table that tells you what the probable cost of the risks for different confidence levels. So if you want to be ninety percent confident so P90, you would want to set aside twenty-one million dollars. If you want to be very aggressive and take a lot of risk you would set aside only six million dollars. But if your risk tolerance is sort of moderate you might choose P70 and that's seventeen million dollars. The FHWA recommends the P70 level. The advantage of this simulation is it doesn't give you a single number like in case of the formula-based calculation. You recall that the number the value we got for the aggregate estimate was about thirteen point six million but that is a single point estimate and it is equivalent to the P50 that is basically the midpoint. But basically the midpoint means half the time you are going to have less budget than needed for paying for your project costs and that doesn't seem like a very good position to be in. So you might want to go to the P70 level or the P90 level to be more confident that you will have the money to pay for the project. And the advantage of this simulation is that you can figure out how much you need to set aside if you want more confidence that your budget is going to be adequate. Now the concessionaire may be more risk friendly or be willing to take more risks. In their estimates they may actually use lower confidence levels but then they make it up on their risk premiums. So you know the cost never disappears, it just shows up somewhere else. Now the breakdown is simply based on and P3-VALUE has the different phases and where the risk occurs in which phase and does give you a breakdown by phase also.
So P3-VALUE also provides a graphic and this shows you how the risk costs are distributed. This is the combined aggregate value and I can't read the numbers but sort of something if you go across on the right hand side you will see it says cumulative distribution or cumulative probability. And so that scale if you look at ten percent on the right hand side and you go down and follow that line and you then you look at where the blue line, I'm talking about not the bars but the blue line intersects that ten percent and then go down and you should see something like the six thousand we saw on the table. Similarly the seventy percent go on the right hand side, go across on the seventy percent, go to the blue line and then go down and then you should see something like the seventeen million dollars that we saw on the table. So the advantage of this graphic is if you want something instead of ninety percent or seventy percent is not good enough for you. Maybe you want ninety-five percent you can go to this graph and pull it out. Or maybe you want sixty-five percent. You can still go to this graph and read the value there. You can get whatever for whatever level of confidence or for your level of risk appetite; you could select the value that best suits your need.
Of course there are several challenges. It's not simple mathematics. We have issues and challenges in estimating probabilities with limited data. You are using probability distributions so you have to have a history of data and from which you can create those probability distributions. Sometimes there is optimism bias so you underestimate your risks or there might be overestimation in case somebody wants to make a P3 look better, all they have to do is overestimate the risk and the public sector comparator. So accounting for correlation among risk is an issue, accounting for identified risk. Remember I said only the risks you have identified can be quantified with this tool so you still have to worry about the unidentified risks. Issues related to double counting of risks. So I mentioned that some risks are included in the risk premium of the concessionaire, and others are included in the cash flows and so you need to be very careful if you are creating a shadow bid or a bid of the concessionaire for to try and make a comparison with a public or conventional procurement. You have to be careful not to double count risks. So accounting for revenue risks is always difficult. It's very difficult to get a distribution of revenue risks and so in practice people simply account for revenue risk using a discount rate. The aggregating risks and then trying to create a distribution of impacts might be a problem in a combined probability and then remember that all we are accounting for in this risk register for P3 procurement evaluation is everything that might occur after the award of the contract. And there are some procurement phase risks that are unique to P3. For example you might have three bidders and so the chance of getting the bid or accepted is one in three so the concessionaire is taking a risk at that point but that's not included anywhere in your cash flows. You haven't accounted for that so how you account for it is an issue if you are trying to create a shadow bid. That's it, Victoria. I turn it over to you.
Victoria Farr: Thanks, Patrick. So here is another poll question and it's true false. It's a bit lengthy. So "A deterministic or formula-based risk valuation can provide varying estimates of aggregate risk values from which an analyst can choose depending on the confidence level desired by decision makers." Again the key phrase there is a deterministic or formula-based risk valuation can provide varying estimates of aggregate risk values. I'll give you about another ten seconds or so and we'll close that out. Patrick, do you want to inform our audience what the correct answer is?
Patrick DeCorla-Souza: Yeah the correct answer is false so those who said false are correct. Reason is you need a stochastic or probabilistic risk valuation that is the simulation to get the graphs, the cumulative distribution that I showed you and from that is where you can pick out or choose the confidence level or the cost associated with the confidence level you desire. A deterministic approach, the formula that I used probably gives you one single estimate which is probably close to the P50 that you would get in a simulation but you aren't able to get if you want a higher confidence level than P50 you are not going to be able to get that with a deterministic formula-based approach.
Victoria Farr: Thank you for clarifying, Patrick. So we've had about three questions come into the chat pod. So looking from the bottom the questions from the Create program. "Is an aggregate impact calculation template spreadsheet available on the FHWA website?"
Patrick DeCorla-Souza: Well the one, the P3-VALUE will help you calculate the aggregate impact using the risk assessment tool and that tool is available on the Office of Innovative Program Delivery's website under the P3 toolkit. Now that is a learning tool. Now FHWA also uses another software tool called Crystal Ball but it's not on our website because it's proprietary. Jim, do you want to address that?
Jim: Yeah, you mentioned Crystal Ball. There's another software program called At Risk I believe, and those are the programs that actually run the Monte Carlo simulation so depending on what program you use will that's what your spreadsheet will have to be based upon so the spreadsheet for Crystal Ball is going to look a little different than the spreadsheet for At Risk. The other thing is it's very flexible how you model the spreadsheet so you can make it very complicated, very complex, or you can make it very simple. The good thing about the Monte Carlo simulation is that it is very scalable. So we don't have any sample spreadsheets out there for the Crystal Ball program we use just because it can vary a lot based on the application, the project that you are using it for.
Victoria Farr: The next question is from Steve, three up from the bottom. "Does the P3-VALUE schedule model treat each activity as a discrete duration or does it model the average project duration?"
Patrick DeCorla-Souza: As I said, what we are trying to get at is the aggregate delay so that's what you can hang your hat out of P3-VALUE. Now P3-VALUE also for the design bid phase, actually aggregates each individual phase within design build such as planning, construction and transition, so there are several phases that are included there and if you provide P3 value with the subcategory of the risk, then it will actually do it and give you the results by these individual subcategory phases.
Victoria Farr: And a second question from Steve, "Should low probability, low consequence risk be ignored or just rolled up into a variance on the base estimate?"
Patrick DeCorla-Souza: That's a good question. Jim, do you want to handle that?
Jim: Yeah, it depends. What we do a lot of times do with low probability, low consequence risk is add them together and then make it one aggregate risk. It all depends on how you want to model your estimate. Sometimes we sort of ignore low probability, low consequence risks, but most of the time we do try to aggregate them into one specific risk that we will just call low impact, low probability risks.
Victoria Farr: Thanks, Jim and the final question that we have right now from John has to do with the VDOT formula, Patrick.
Patrick DeCorla-Souza: Uh-hum.
Victoria Farr: He's asking "How did you arrive at a multiple of four for the most likely risk, and is that unique to transportation P3s?"
Patrick DeCorla-Souza: I don't know how VDOT that is Virginia DOT came up with the four but I understand that this multiple of four is used in the construction industry by contractors who are trying to estimate the price to bid on a project. Now how the industry came to arrive at four as being the best multiple, I don't know. It just seems to make intuitive sense to me.
Victoria Farr: Thanks, Patrick. And the last question that we'll take before moving on is from Maryanne; she's asking "Can this tool be used to determine when it makes sense for the owner agency to buy insurance for example for damage to facilities by weather or vandalism?"
Patrick DeCorla-Souza: Okay for one thing, you're going to get estimates of aggregates out of this tool. What you are going to be doing is you are going to actually be evaluating each individual risk yourself with your experts. So your experts are the ones who are going to produce those probability distributions and range of impacts so you could obviously use that formula, the Virginia DOT formula based on the estimate of for individual risk because what P3-VALUE is doing is simply aggregating risk together. To do an individual risk, you would have to use that formula if you think that formula is a good one and have your experts determine the probability distribution of the impacts and the probability of the risk occurring and then you simply do the multiplication as we showed on that slide. And then you compare it with the insurance costs.
Victoria Farr: Alright, thanks Patrick. It looks like Aaron might be typing in some clarification but in the interest of time we'll move on to the next lesson and we can address Aaron's comment or any additional questions after lesson four.
Patrick DeCorla-Souza: Alright, so we go to risk allocation and this is going to be quite brief. This table here simply shows you how risk is transferred for the various functions in a project so from in the case of design build it's only the design and the construction risks that are transferred to a P3 design build finance. You also are transferring financial risk. The ones that we are focusing on in our office and with P3 value are the DBFOM availability payment and the DBFOM toll concession. And as you see since you are looking at the long term now the concessionaire is taking on O&M risk and in the case of toll concession it's also taking on traffic and revenue risk.
This shows you a little more clearly how you are transferring risk. This is somewhat of a typical risk transfer and risk sharing graphic and the blue is the public and you see as you go down to the P3 the DBFOM P3s which are availability, payment, and toll concession, most risks are transferred. There are very few that are retained. Over here archaeological findings and the graphic information systems is retained as public risks. In the case of the availability payment you retain the financing risk. Remember this is the one where you are paying the concessionaire through an availability payment so if there are any tolls any shortfall and revenue is your risk because you are going to be collecting all the tolls. The others that are shared here for example, delay in payments neither side is able to manage it any better than the other and so those are shared.
So important to understand that P3s do not transfer all project risks; as you saw in that slide the public still continues to retain some risks because the public sector might be able to manage those risks better. For example, we saw permits in that previous slide. Risk is allocated to the party most capable of managing the risk. Risk transfer will increase the bid price of the private sector simply because they have to account for it either in their cash flow or in the premiums that they charge on the financing, the equity rate of return for example. The advantage of transferring risk is that the private partner is incentivized to reduce that risk of course to make more profit and increase their return but they have an incentive to reduce the occurrence, to reduce the impact of that risk which is of course good in the end good for society. A risk may be shared as we said, if neither party is more capable of managing it and risk costs may vary over time. We saw that in that graphic where risk premium changes based on where we are in the life of a project.
So in deciding whether to allocate a risk to a concessionaire, you might go through three steps. The first is to see whether the concessionaire can reduce the likelihood of the risk occurring. If that's not true, then go to step two and see if the risk occurs, could the concessionaire reduce its impact better than the public sector? And step three is even if step one and step two are not true and it can neither reduce the likelihood nor the impact, can it afford to bear it better than the public sector? Can it absorb the risk better than the public sector? This is sort of the consideration you use when you buy insurance, you know. You buy insurance because you figure that if there is an accident at least you won't have to worry about your balance sheet and whether you have enough money in the bank to pay for a new car. The insurance company can bear that impact better than you because it diversifies that risk.
So this is what happens when you do a P3. You do it because it can reduce costs. It can be more efficient because it has incentives to reduce the cost of construction, cost of operation, and cost of capital.
This is a typical risk allocation for the Port of Miami Tunnel and you see that not all risks are transferred to the private sector. Several risks have been retained by Florida DOT and there are several risks that are shared and this is because Florida DOT determined that it would be more cost effective. It wouldn't provide any extra benefit or value to them to transfer certain risks to the private sector. One of the big ones that is shared is the geotechnical risks. Neither one was better in handling that risk so they decided to share it. Over to you, Victoria.
Victoria Farr: Thanks, Patrick. So we have another true false question for you. True or false, "The goal in risk allocation is to transfer all risks to the private partner in a P3." So we'll give you another five seconds and we'll close that out and broadcast the results. It looks like the vast majority of people got that correct that the answer was indeed false. Patrick, do you have any additional comments on this?
Patrick DeCorla-Souza: No, most people got this correct so that's good. False is the correct answer.
Victoria Farr: So Aaron Jette did indeed provide a follow up to Maryanne's question regarding or I'm sorry to John's question regarding the VDOT formula. And he's saying that "By multiplying the most likely risk value by four, he believes that it allows the risk distribution to mimic a standard normal distribution in a simple formula. So in simplified terms the shape of the distribution is such that the height of the triangle is roughly the same height as the bell-curved shape found in a normal distribution." So Patrick I don't know if you have any follow-up comments on that.
Patrick DeCorla-Souza: I agree, that's a good rationale, it makes sense.
Victoria Farr: Okay. Since there are no other questions that have come in and especially in the interest of time let's move on to the next lesson.
Patrick DeCorla-Souza: Alright so at least from the point of view of P3-VALUE the reason we are doing risk assessment is so that we can conduct value for money analysis with the remaining tools. And so I guess before I introduce you to how risk assessment enters into value for money analysis I need to explain what it is. So it is a way to value for money is simply the cost difference between a conventional and a P3 procurement and it can be either dollars or percent difference. And you conduct a value for money analysis by creating what is called a public sector comparator which represents the conventional procurement. Then you create something called a shadow bid which represents the P3 option. And you then figure out the present values of these two options and look at the difference. So that's what we are going to try and demonstrate in a little bit.
So here are the steps you need to know the options that you are comparing. Is it design build versus DBFOM? So you begin by identifying your options to be compared. You go through the risk assessment for both of those options and they might be different, you know the risk values can be different in the P3 option because the concessionaire may be able to manage it better and that's what P3-VALUE risk assessment tool will help you do. Then you take those risk values from the risk assessment tool and apply them to the cash flows in the public sector comparator and the shadow bid, and then you take those totals that is the cash flows of the risk plus whatever other cash flows such as for operations and maintenance or financing or whatever, and you then discount all of these cash flows to get a present value and then you can make a straight comparison between the public sector comparator and the shadow bid and finally you try to account for other things that you may not have considered in this financial evaluation using qualitative analysis.
So we are going to demonstrate how this works. I'm very quickly going to show you a comparison between a conventional procurement of design bid build, and a DBFOM which could be either an availability payment or a toll concession and I'll assume that transferred risks will be reduced under the P3 by fifty percent. This is just a gross assumption just to demonstrate how this works. And I will assume that the concessionaire may be able to increase toll revenue so that's like an opportunity risk by five percent if you have a toll concession.
So you start off and you've done your risk assessment using P3 value and for the public sector side you've got a hundred million dollars for the threats, value of threats and then for the P3s you got fifty million which didn't vary between availability and toll since those are simply functions of the payment mechanism so they don't affect the construction costs or any other operations. So we have reduced the risk by fifty percent. We distribute them by year. There's a three-year period of construction. We distribute them in the same manner as the costs, the baseline costs are distributed which is what you see here. And then we try to account for the potential increase in revenue from the P3 toll concession and assuming that ten million dollars extra is possible and that's what we've estimated over the twenty year period after construction we get a total of two hundred million dollars.
These estimates were all in today's dollars so we have to work them to year of expenditure dollars in order to incorporate them into our financial models. So we apply inflation factors which is what you see here. We have simply taken the same numbers that we had previously in the previous line and applied an inflation factor of three percent so that gets applied not just to the cost but also to the revenues which is what you see in the P3 toll option all the way from year four to year twenty-three.
We take those what we call nominal values or year of expenditure values and then we create present values by using a discount rate. In this case we have used a discount rate of five percent which is applied to the nominal values that we estimated in the previous slide and so now we get a present value, it doesn't matter when the cash flow or expenditure occurs. We can account for it in today's dollars in real dollars, simply by discounting them by five percent and so that's what we've done for all of the negative risks and then there are positive risks only for the toll concessions so we have done that for year four through twenty-three and we aggregate those and come up with a present value of one hundred and fifty-five million dollars. So then we take the positives and the negatives and we combine them and that's what you see in the bottom row.
So to make the comparison in value for money analysis you look at your public sector comparator, look the initial cost, add the retain risks and any other costs such as procurement costs. And the bar on the left shows you those costs. For the P3 availability payment you can see the initial cost is actually higher than the public sector comparator and this is because the transferred risks are included in that initial cost. However the retained risk which is a risk born by the public agency is smaller because most risks have been transferred and other costs such as procurement costs, oversight and monitoring costs possibly might be higher so we added that to the P3 availability option. The P3 toll option, it's sort of similar to P3 availability but because we have subtracted the positive risks from the initial costs we have a lower initial cost than in the case of the P3 availability payment. The retained and other costs are similar. So overall with this value for money analysis you can see the P3 toll comes out looking a lot better than the PSC and a little better than the P3 availability payment concession. That's it. Over to you, Victoria.
Victoria Farr: Thanks, Patrick and this is our last poll question for you today for lesson five and the question is "Has your agency conducted a value for money analysis?" So yes, no, or I don't know. I'll give you a few more seconds to answer that. I will close that so it looks like the majority of responding agencies do conduct a value for money analysis, well not the majority but it's kind of even between do or don't. Patrick do you have any comments?
Patrick DeCorla-Souza: No, that's good to know. Thanks.
Victoria Farr: Okay well in the interest of time and seeing there are no questions in the chat pod, thank you, Steve for inserting that comment and continuing the conversation regarding the formula so there are no questions so we can move on, Patrick.
Patrick DeCorla-Souza: Okay, great.
Victoria Farr: To lesson six.
Patrick DeCorla-Souza: Alright so we are on the last leg here and ready to move on. This is to get you started on the P3-VALUE risk assessment tool so that you can get your hands on it and actually learn because unless you actually operate the tool it's difficult to understand some of these concepts related to Monte Carlo simulation and how it works so this lesson will help you actually do the homework which we will provide at the end of the webinar.
So first all the tools are available on the website of our office and available at that link along with supporting documentation.
So the one that you will be using for your homework is the risk assessment tool.
There's a lot of supporting documentation that you see listed here. We are developing a more detailed guidebook which is not available yet. It should be by the end of October.
So here is the key component of the P3-VALUE tool on risk assessment. It is the risk register and here is where you put in the data on your risks. As you can see you tell it the category of risk, the impact phase, very important, the type of risk, is it a threat or an opportunity? You put in a description of the risk and what might be the consequence of the risk and this is simply to help you since you are going to manage this over the life of the project it's good to document all of this information.
Then you get to the qualitative assessment and as you can see here you need to in rating one to five you provide the probability rating for the risk. You provide the cost consequence as a rating, again one to five. In this case it's rated a two, and the scheduled consequences has a similar rating of two. And what you see the yellow portions are what the tool calculates based on the five and the two so it is able to go into that matrix that I showed you earlier which had all the colors in it and estimate that the combination of the probability rating and the consequence that you provided results in a medium overall rating for this risk.
Now since this is a medium-rated risk we want to go ahead and do a quantitative assessment so we need a specific probability percentage and so we've said it's eighty percent is what our experts have told us is the probability of this risk occurring. They tell us its distribution is triangular with a delay impact of somewhere between fifteen days and thirty days but the most likely is twenty days so it's sort of skewed a little to the left, the distribution is skewed to the left. Similarly on the cost in fact, our experts tell us it's a triangular distribution and the minimum is about eight point three million. The maximum eighty-three million, pretty high there, but the most likely is pretty small, about sixteen million so that's also kind of skewed to the left. So on the low side, in other words.
So we take all this information and based on who we think can better manage the risk we decide to better allocate it twenty percent to the public sector, eighty percent to the private sector, and we estimate what we can do to mitigate that risk and we write it into that risk mitigation cell.
So the results as I indicated earlier will come out. The ones I showed you earlier in the lesson three or four showed you the results that come out when you actually run the software. So of course the limitations of this tool, it's not suitable for all types of potential scenarios and the very first lesson I indicated basically what risks you can quantify and monetize and what risks you can't. So there are still unidentified risks and there are still things that you are not going to be able to estimate through this tool. It won't give you the number that comes out is not-- it's the aggregate of all the risks that you have identified but you still need unidentified risks and other things such as process risks and endogenous risks that are not accounted for. The risk tool assumes that all risks are independent and there is no correlation between the risks and it does not aggregate the lower rated risks for you. You have to do it yourself and put in enter in a new aggregate risk into the register with a probability and a distribution and a range. That's it. That concludes this session so I guess we can go on.
Victoria Farr: Yeah, Patrick, we don't have any new questions so since I know we definitely want to make the homework assignment available to participants. I think we should quickly more through the core summary and get to the download and course evaluation and whatnot and we can take additional Q&A at the very end.
Patrick DeCorla-Souza: Okay so this is simply what we focused on. The most important thing is the risk analysis and valuation that P3-VALUE focuses on and hopefully we have equipped you to begin using P3-VALUE.
We have more resources available on the website available through these links.
There are additional webinars and also a workshop, a live workshop available on site in the Washington D.C. Metro area on October 29th and 30th, then several future webinars. We will be going over the homework on risk assessment at the October fourth webinar.
Here is my contact information. Here is Jim Sinnette's contact information, and Thay Bishop who oversees the I.P. Academy if you need to contact her.
And now we have the important information you need to get to the homework so the risk tool is available on the website but it is an early version of the tool. The one you are getting is an updated revised version which is a lot easier to use. And so we are suggesting that you download this version. We are calling it one point two and instructions are available here so-
Victoria Farr: And Patrick, I'm going to switch the layout so that and I can provide instructions again. So in the middle left of your screen is where the files are located that Patrick just described, the version one point two of the tool and the homework assignment itself. It's a PDF as well as a PDF of the presentation today. On the middle left of your screen you can select a file name, click save to my computer, and follow the prompts on your screen. A new window or tab may open in your browser to complete the download. If you have any trouble accessing those files, please let me know. Also on your screen here is an evaluation that we would appreciate your feedback on today. We have eight multiple choice questions and one open-ended question so at this time, Patrick, we can we have about nine minutes left of our scheduled time. I don't know if there's anything additional you want to say about the homework. I'm sorry for interrupting you. I just wanted to make sure that people weren't confused that the slide advertised the materials as being available for download on the left hand side but I had to change the layout so that would actually be accurate. So we don't see the presentation at this point but was there anything else that you wanted to cover with the homework while we wait for any additional Q&A?
Patrick DeCorla-Souza: No, I just the only thing is I want to encourage the participants to do the homework and then if you have any questions you have my number and of course you can send an email to P3-VALUE as indicated on the slide. Finally we will go over the entire homework so even if you haven't had a chance to complete it or if you ran into problems while trying to do it not to worry. We will lead you through the entire homework. It will get you familiar with the intricacies of developing risk values or a public sector comparator that is a conventional procurement and the shadow bid and how you would actually go about estimating risk values for each of these situations. So it's a good exercise. I encourage you to do it.
Victoria Farr: Okay Patrick, well I don't see any questions coming in the chat pod. I'm going to ask our operator, Tim to provide instructions if any participants want to ask questions over the phone.
Operator: At this time if you'd like to ask a question over the phone line please press star then the number one on your telephone keypad. Once again that's star one to ask a question.
Victoria Farr: Okay thank you, Tim. I'm seeing a comment from Alexander regarding the tool version not allowing you to save. I'm not sure if that means in terms of to your computer from the tool downloader or once you actually open it on your screen so if you can provide a little more clarification in the chat box "saved to my computer." Let's see here.
Patrick DeCorla-Souza: I was able to do it.
Victoria Farr: Yeah I just did it myself. I would advise Alexander if you can see if perhaps there's a new tab that's opening or window or if you're actually getting an error message and Carla is having the same problem. Okay well I'm not sure how to address this at this time. We can certainly keep the Web room open for several days. We won't change it and we'll try to address this issue. We can always contact you but I'm able to save to my computer and Patrick you are as well?
Patrick DeCorla-Souza: Yes, yes.
Victoria Farr: Okay so we will look into this and get back to the participants. Patrick in the meantime actually I will type an email address into the chat pod for you to contact and if you send a request for the tool to that email we can respond. The reason why we don't usually email it is quite a large file so just be aware that it might clog up your inbox but that email address I just entered in is P3firstname.lastname@example.org and if you send us a request for the tool we will do what we can to accommodate you. Philip is saying he had no problem with is too, I apologize. It might have to do with possibly the version of Adobe you have on your computer. Without seeing the screen I can't provide a lot more feedback other than perhaps just try again later or again send a request to that email and we will send you the tool directly. Patrick do you have any additional comments?
Patrick DeCorla-Souza: No I think I look forward to participation in the Webinar on October 4th which is exactly two weeks from today and we will leave the telephone lines open at that time so you will be able to talk to the instructors and discuss your issues.
Victoria Farr: Great, okay well I want to give a big thank you to Patrick for presenting the material today. I know it was a lot to get through and we always appreciate his expertise. We also want to thank Jim Sinnette who joined us for responding to all the questions that our audience posed. Also want to thank Aaron Jette for providing background support as well as to Daniel Williams and Thay Bishop who helped set up the webinar and make it possible. Again if you would like to request the tool please do send an email that to that address P3email@example.com. I apologize for any technical problems. I'm not sure why you're not able to download it but hopefully we can email it to you directly and you will be able to complete the homework in time for our October fourth webinar. So that's all I have. Thank you everyone. We look forward to seeing you on October fourth.
Patrick DeCorla-Souza: Thank you.