P3-VALUE Webinar - September 5, 2013
P3 Program Manager
Office of Innovative Program Delivery
Victoria Farr: On behalf of the Federal Highway Administration's Office of Innovative Program Delivery I would like to welcome everyone to today's IPD Academy Webinar, P3 Evaluation Overview. My name is Victoria Farr. I'm with the U.S. DOT Volpe Center in Cambridge, Massachusetts and I will be moderating today's webinar as well as facilitating our question and answer period and addressing any technical problems along with my colleague Aaron Jette. Our presenter for today is Patrick DeCorla-Souza. Patrick is a P3 program manager in the Office of Innovative Program Delivery. Before he begins I would just like to point out a few key features of our webinar room. On the top left of your screen you'll find the audio call in information. If you are disconnected from our webinar at any time, please use that call in information to reconnect to our audio. Below the audio information is a list of attendees and below that in the lower left corner is a chat box that you can use to submit questions to Patrick throughout the webinar. We may also take questions over the phone later in the webinar and further instruction will be given at that time. If you experience any technical difficulties, please use the chat box in the lower corner to send a private message to myself, Victoria Farr. Our webinar is scheduled to run until 3:30 p.m. Eastern today and the course is divided into four lessons. We will queue up any questions for Patrick at the end of each lesson and we also anticipate having about ten minutes at the end of the webinar for additional Q&A. I also wanted to let you know that we are recording today's webinar so that anyone who is unable to join us may review the material at a later time. Also, if you note in the chat box I have informed you that the download of the presentation will be available at the very conclusion of this webinar.
And finally before we begin we have a few quick poll questions that we want to pose that will just help Patrick understand our audience today. So the first poll question in the upper left corner asks what is your affiliation. And you have multiple options there, you can say if you are either with the FHWA Division Office, non-Division Office, state DOT, other federal agency, other state agency, NPO, private sector or other. And then upper right corner we are asking you how many people are participating along with you today, if you either at your desk alone or if you have a group of people up to ten or more joining you in a conference room. In the bottom left corner we're asking what is your current level of knowledge and experience with P3s, so whether you have no prior knowledge or experience, a basic understanding, some knowledge and experience, a lot of direct knowledge experience or if you're an expert. So Patrick, you can see those results there yourself and I think we will now turn it over to you to begin presentation. Thank you.
Patrick DeCorla-Souza: Thanks Victoria. And again, my name is Patrick DeCorla-Souza. I'm the P3 program manager in the Office of Innovative Program Delivery. And today we are going to talk about evaluating P3s and this is just an overview webinar. What you see on this slide is the definition of P3, that is a public, private partnership. We will be talking about a software tool called P3-VALUE which is a set of four analytical tools that are linked together to help you do a P3 evaluation. And also I want to let you know that this the first of four webinars that form a series on P3 evaluation. So what you're going to get today is a very brief overview. We will go into greater detail on risk assessment on September 20th and we will be presenting a value for money analysis webinar in January and a financial structuring and assessment webinar in March of next year. However, if you are interested in getting immediate access to the information in those webinars you can also listen to recordings of the webinars and you see the links over there that lead you to the recordings. And as Victoria mentioned, the presentation will be available to you for download at the end of this webinar so you can connect to these recordings. The difference between the recordings and what you will get early next year is some updates that we are doing to the P3-VALUE tools that will be incorporated in the versions that we will be presenting next year.
So as Victoria mentioned we are going to present four lessons. The first is an introduction, I will then talk about P3 financial evaluation, I will go into greater detail into value for money analysis which is a type of financial evaluation and then I will talk a little bit about FHWA's P3 Toolkit and P3-VALUE.
So hopefully after taking this course you will be able to identify the purpose of P3 evaluation, you will be able to describe to your peers what P3 evaluation involves, you will be aware of the challenges and limitations involved in P3 evaluation, you will be able to explain the role of P3-VALUE tools, the four integrated tools and you will be able to access the tools and supporting information. So I'll go right into lesson one, giving you an introduction.
So by way of introduction, evaluation can be done at the project level or it can be done at the procurement level. At the project level all we are trying to find out is if a project is worth doing. We are not concerned about the method procuring the project, we just want to know if the benefits of the project exceed the cost and we are taking a societal perspective. Now the definition of society can vary, it might be the people in a particular city, it could be the people in an entire state, it can be people in the entire nation and depending on what perspective you take, the benefits and the cost that you account for can be different. You can even go into considering benefits and costs to future generations if you consider those part of this term society. Procurement evaluation on the other hand assumes that you've already done project evaluation and you have determined that the project is worth doing so now all you are trying to find out is whether a P3 procurement would be preferable to a conventional delivery for example using Design-Bid-Build. And if you think that or the analysis shows that P3 might be a good idea you might want to evaluate several different P3 options with varying levels of allocation of risk to find out which of the various P3 options would be the best.
So procurement evaluation actually can be done at two levels, a financial level and an economic assessment level. And the difference between the two is in a financial assessment or evaluation you simply look at cash flows, you are not looking at anything that does not involve cash. An example is travel time savings to people who use the highway would not be considered in financial assessment. So the other point is generally we are looking at the perspective of the procuring agency and impact on the budget of the procuring agency. This would be the state DOT or the local DOT, that's all we are considering. And the tool we use to do this kind of evaluation is called Value for Money Analysis also VfM. In the case of economic assessment on the other hand, we don't just consider cash flows, we go beyond cash flows and we consider all costs and benefits whether or not they involve cash. So for example we might consider environmental costs and we might consider user benefits. And if there are differences between P3 procurement and conventional procurement that affects environmental cost or user benefits we would take that into consideration. So the tool that is used to do this kind of economic assessment is called Benefit Cost Analysis or BCA. Now you should be aware that in conventional practice so far the only type of assessment that has been done in evaluating P3s is financial assessment. That's the current state of the practice, only financial concerns or cash flows are considered. The economic assessment has not been done in the U.S. or internationally and FHWA has embarked on a research effort to see how this can be done and the result of this research effort will result in an updated version of the P3-VALUE tools that incorporates economic efficiency assessment.
So what are we trying to find out with financial evaluation? So the first question, is the project affordable to the public agency? Will the project need a subsidy or are the revenues coming from the project itself, for example, in the form of tolls, sufficient to pay for the project? So that is called financial feasibility analysis or financial viability analysis. Second form of financial evaluation is value for money analysis and that answers the question whether a P3 procurement would be more beneficial to the public sponsor than conventional procurement. Now there is a third level of financial evaluation that you might want to do and that is called asset valuation where you are trying to instead of just figuring out whether a P3 option is better than a conventional procurement you're also trying to get an estimate of the value of for example the toll revenue stream and what the concessionaire, the private sector would be willing to pay for the opportunity to collect tolls for a specific period of time. And so the term used for that is asset valuation and that would generally be done when you are trying to value an existing toll facility that you want to lease to the private sector and you're trying to determine how much money the private sector should be willing to pay the public sector for collecting tolls over a certain period of time. It may also be used to figure out whether the availability payment and this is if there aren't any tolls on a facility the private sector would still need to be compensated so this type of valuation can be used to figure out whether the amount of availability payment being asked for by the private sector is a good price, is a fair price and is reasonable.
If you're doing economic evaluation on the other hand as we said we are trying to determine first if the project provides benefits that exceeds the cost to society as a whole. We might also use such economic efficiency evaluation to find out the best time to build a project. If you're building it too early perhaps there's not enough demand and the benefits then may not be greater than the cost. On the other hand, if you delay the project a little bit maybe demand will be high enough at that time and the benefits might exceed the cost. You can also use this type of benefit cost analysis to scope a project. Perhaps the project has too many bells and whistles. Perhaps if you scope it down a little bit you reduce the cost significantly without reducing the benefits too much and thereby you can have an economically viable project. So at the procurement level, we might use economic efficiency evaluation to ask the question whether the P3 option increases net benefits to society over and above what the conventional procurement might produce. And of course one way a P3 option can do that is by reducing costs and the other way it can do that is by increasing benefits. Of course in this type of an evaluation we are looking at not only the cash amount or the financial amount we are looking at non-financial impacts such as user benefits and environmental costs.
In summary then, there are two basic questions we need to answer if we are evaluating a P3 option. The first, and the one that has been used in current practice is value for money which is a financial analysis approach and it answers the questions what are the financial benefits to a procuring agency in going with a P3 instead of the conventional approach. The second type of analysis approach is benefit cost analysis also known as BCA and it attempts to answer the question whether the P3 option is better from the perspective of society at large and whether all benefits, not just financial benefits exceed the cost of the project. So now we have a test question and I turn it over to Victoria.
Victoria Farr: Thank you, Patrick. So throughout the presentation, at the end of each lesson we will bring up a little poll question here like we did at the beginning. So the first question here is a true false and it asks whether financial evaluation considers the full range of costs and benefits to society. So we'll give you about five or ten more seconds to respond and then we will review the answer with you. I'll close that out. So it looks like the majority of you got that correct. Patrick, do you have any further comment on the question?
Patrick DeCorla-Souza: Just I wanted to reiterate, financial analysis simply looks at cash flows and does not look at the user benefits such as travel time savings. So if you want to do a full benefit cost analysis you have to use economic evaluation rather than financial evaluation. So those who said false are correct.
Victoria Farr: Thank you, Patrick. It does not look like anyone has submitted any questions yet in the chat box. Just to remind you, that is available on the bottom left corner of your screen. If you type questions as they come to you throughout Patrick's presentation, we will answer them at the conclusion of each lesson after we do the poll question. So Patrick, seeing no questions, we can move on now to lesson two.
Patrick DeCorla-Souza: All right. Thank you, Victoria. So the second lesson we'll now discuss financial evaluation in a little more detail and as we show here, financial evaluation can be done at several stages in the life of a project, starting with the planning phase, going onto project development then at the procurement level then during design and construction and finally operations and hand back. So why would you do financial evaluation at the planning stage?
Here are some questions you might want to ask right at the planning stage where you're developing a program of projects maybe for your long range plan and you're trying to determine whether there are any projects that might be good candidates for P3s. And the reason is that you want to do this type of screening is your further development of the project can be influenced by whether or not the project is amenable to P3. So that's why you want to do this kind of screening early on. Some of the things that affect whether or not a P3 will be appropriate for a certain project is the scale of a project. Since you have pretty high cost for transaction which means the cost to do the procurement itself can be pretty high, you want to look at mainly projects that are large in scale that are major projects or projects that at least have a cost about $100 million or more for highway projects. And this is simply because right now the procurement process is quite in its early stages of development and we do not have standardized contracts and we tend to spend a lot more resources, time and expertise in simply developing contracts with a P3 concessionaire. And in order to justify those costs there has to be a big benefit in terms of value for money and so you won't be able to get that if your project is only let's say for example $10 million. So you need to have a project that's pretty high in cost. If a project has long term service implications it may be a good candidate, P3s generally involve not just design and build but also operations and maintenance and if the same contractor is responsible for both design, build and operations and maintenance they might be able to achieve some efficiencies by for example designing the project to be easier to maintain or to be less expensive to maintain. So a project requiring long term service might be a good candidate. The complexity of the risk profile might justify consideration as a P3. What the public sector gets by doing this is reduces its own risks and is able to transfer some of this risk over to the private partner. Of course, the private partner charges a price for it but the benefit to the public sector is that it knows up front what that project is going to cost either in terms of availability payments over the life of the project or in terms of some type of upfront subsidy for a project that involves a toll concession. Finally projects that allow for innovation can be good candidates. For P3s, the private sector is more willing to take risks in order to achieve efficiencies or to increase revenues where the public sector does not have those incentives to the extent that the private sector does and so tends to shy away from innovation even if rewards might be significant from taking those risks in being the first to do something. So the final question is can we, through integration of design, build and operations and maintenance achieve some reduction in lifecycle cost. Then again the long term nature of the service would affect that.
So you would want to go through some screening process in the planning phase. The first, of course as we said in the last slide we have to look at project characteristics, that's the first item. So the other issues you have to look at is whether or not you have a legal framework in place to do this type of a P3, whether you have institutional capacity and the organizational capacity, do you have the expertise in-house to do a P3? And finally is the market interested, does the private sector think that the project that you are wanting to do as a P3 is attractive to them in terms of achieving the rates of return that they are seeking?
We have developed, FHWA has developed a screening tool called P3 Screen that you can use and that's available on our Web site. So it is something that you can help you answer several questions at the planning stage to see if a project would be viable as a P3 and it is a tool that you can use to satisfy the requirements in Map 21 relating to development of an initial financial plan. There's a provision that requires that major projects be considered for-- to find out whether they are appropriate for a P3.
At the project development phase, here is basically during your environmental process, you want to find out whether a P3 would add value. Just because a project has passed all the screening questions and you've done this type of qualitative analysis, that might not be adequate to make a final decision as to whether the P3 route is the route you want to go. So you may want to at this stage do a much more detailed quantitative analysis and that's where value for money analysis comes in. Once you determine that you may want to use value for money analysis to find out what form of P3 is best. Should you do an availability payment where you keep the risk of toll revenues for example but transfer performance risk to the private sector or do you want to do a toll concession where you don't keep the toll revenue risk, you transfer to the private sector as well as the performance risk? Of course the quantitative analysis can also help answer the question as to whether a private entity would be interested in your project and also if you would need to provide a subsidy to the private sector if tolls were not sufficient. At this stage you might also be able to use value for money analysis and financial evaluation to determine what is the best length of concession if you decide to go the P3 route.
So in this project development phase you have to do a lot more detailed analyses. Traffic and revenue to figure out how much of revenue is coming in and all of these studies can be very expensive and time consuming. You need to do more detailed cost estimates, you need to do risk assessment, you need to do a value for money analysis to compare the P3 option with the conventional delivery. And of course a financial evaluation to find out whether the project would be affordable, would be subsidies that the private sector would ask for or would the availability payment that the private sector would ask for, would that be something that the public sector could afford. So lots of considerations and detailed analysis at this stage. Also you do some initial market outreach, before you spend a lot of money on doing these detailed studies you want to find out if the market thinks they would like to come into your state. A lot of concessionaires don't consider some states as good candidates simply because they don't have confidence that the legislation is strong enough or if they think that there are other risks that they don't want to deal with. So before you go into all these evaluations it's better to find out even if you have a project that's viable, whether the market would be interested in coming into your state.
So we talked about value for money analysis and this is a very simple manner with a bar graph explains what it is. The conventional approach on the left hand side shows in purple the cost that you normally estimate, the initial cost that you calculate for your project. On the right hand side you have the P3 option and the purple bar shows you the value of the P3 bid. This is what the private sector would want either in the form of availability payments or in the form of a combination of toll revenue and a public subsidy. Now you will notice that the purple bar is actually higher, taller, the P3 option is taller than the purple bar on the conventional approach and this is quite likely. And the reason is that with the P3 option you are transferring many risks over to the private sector and they are going to charge you for those risks. So the price that they ask for is going to be higher than the cost you estimate with conventional procurement. So the red bar, the red portion of the bar on the conventional approach is the retained risk that the public sector normally does not calculate over the entire lifecycle of the project. Now with our major projects' requirements, large projects over 500 million actually do some of this analysis but they still don't evaluate the lifecycle retained risks. So what you see on the right hand side with the P3 option, the retained risks are a lot lower and that's simply because a lot of the retained risks from the conventional approach are transferred over to the concessionaire. So if you add the risks, the retained risks to the base cost, you might actually get a difference, a positive difference which would be the value for money that the P3 option would bring to your project.
At the procurement stage, now you've made a decision that you want to go with the P3 and you've decided to develop an RFP. So while you're doing that you prepare your proposed contract terms for the RFP and the contract basically allocates risks or shows what risks you want to transfer over to the concessionaire. So at this point you've gone through a little bit of more detailed evaluation, you've made some decisions as to which risks you want to allocate and to what extent and it would be helpful to conduct a value for money analysis at that time again to see if the contract or the proposed contract that you end up with still provides value for money in comparison with the conventional procurement. Now after you've issued the RFP and you've gotten bids you will get a proposal and you would need to compare that proposal again to your conventional procurement and see if the preferred bid in this case would provide value for money. You would want to ask if the availability payments or the completion payments being asked for by the concessionaire are reasonable and then you go through a process of negotiation with the preferred bidder and maybe there are some refinements to the risk allocations. So at that time you want to do another value for money analysis to see if you're still getting value for money. So value for money does not end at the project development phase, it continues throughout the procurement process to help you not just develop the RFP but to negotiate terms and allocate risks between the concessionaire and the public agency.
So this graphic simply shows you what you would do at the procurement phase is simply rerun the evaluation, you have better estimates of cost, you have better estimates of financing cost which are a big part of this evaluation and you want to see if now that you have much better information on all of these costs and whether the private bid is still providing value for money.
The benefit of going through and doing this type of evaluation not just in the project development phase but also in the procurement phase is that it provides transparency, the process of selection of a concessionaire is crystal clear, you see where the value for money is and it can convince your elected officials and the public that you're doing the right thing. From the private sector side, they also would like you to go through this process because then they understand better your criteria and it gives them more confidence and they know that if they do the right things, introducing costs and providing value that they are going to get credit for it and they see that your process accounts for these benefits.
So you would want to maintain this transparency throughout the procurement process. For example, Virginia DOT has such a process. There are some issues relating to confidentiality but they can be dealt with by explaining exactly why certain things have to be kept confidential. The important thing is as long as you are able to explain to the public why certain things have to be kept confidential they're going to be less concerned, less opposed as long as they understand and can see you're providing as much information as possible.
So in summary, we need to do financial evaluation at the planning stage. This is more of a qualitative analysis, preliminary screening about financial viability et cetera. In the project development phase, we do much more detailed analyses leading up to value for money assessment. At the procurement stage we have even better information on cost and financing terms and we try to confirm the value for money analyses that we have done in the project development phase.
Now it doesn't end at that point because if you want to improve your processes for future projects, you want to see what you did right and what you did wrong in your evaluation process, what kinds of risks did you estimate correctly, where did you make mistakes. And so the way to do that is to see how things actually happen and compare that with your initial evaluations. And so it helps you throughout the process to either confirm that you actually made the right decision or if you didn't, to find out how you can improve your decision making process in future. So it behooves you to continue with this type of evaluation in the design and construction phase as well as in the operations and hand back phase, you know, what risks did you avoid, did you end up better off. You can see many P3 projects actually are-- not many, I should say a couple have run into tough straits and those are the risks that the public sector would have faced and would have affected the public sector. So you can see where you did a good job of transferring risks and did well. So it's good to continue to evaluate and compare against your base analyses. So with that, I turn it over to Victoria.
Victoria Farr: Thank you, Patrick. So here is our lesson two poll question. Your screen will adjust slightly. And the question is true, false; the value for money analysis may be used to make a decision on whether to use a P3? Give you about five more seconds there to answer this question. I will close that out and you can see the results here. Patrick, do you have any comments on the participants' answers?
Patrick DeCorla-Souza: Yeah, most people got it right, so it is true, value for money analysis may be used in making a decision on whether or not to use a P3 and that's basically what it has been used for in existing practice, that's correct.
Victoria Farr: Okay, and we have a couple of questions that have come in through the chat box, so I'm going to change our view again to expand the chat box. So, Patrick, I'm looking three up from the bottom and the question is how do you identify and assess market interests?
Patrick DeCorla-Souza: Well again, I'm not a person with a lot of practical experience but I can tell you the way this is done is you issue something called a request for information. And a lot of times a request for information, you issue that to potential concessionaries and they basically let you know if they're interested or not or whether they would come in under certain conditions, et cetera. So that's one conventional way it is done but it seems to me that even earlier on in the process you would know if concessionaries are interested if like I indicated in your project screening or planning stage you issue a list of projects. And Virginia DOT has actually begun to do that and they've gone through their long range plan and figured out which of the projects they were proposing to build might be good candidates for P3s and they simply put that out on their Web site. And when these projects are out there, concessionaires express interest and so that's how you know whether the market is interested in your projects or not. So there are several ways to do it and I'm sure we can have some practitioners perhaps speak to it some more at the end of this webinar when we open the lines.
Victoria Farr: Okay. Thank you, Patrick. The second question, it's two up from the bottom from Lizzie is are there any less costly models under development for quote unquote, "investment grade analysis" as an alternative to proprietary traffic and revenue models?
Patrick DeCorla-Souza: Yeah, you are absolutely right; the proprietary traffic and revenue models are expensive but to do at the project development stage if you want to assure potential concessionaires that there is revenue in the project you really have to do a pretty good thorough analysis. Now if you are simply doing at the planning stage, perhaps there are less detailed sketch planning models that you could use. And one of them is the TRUCE model, the tool for rush hour user evaluation which FHWA has on its Web site that simply provides very crude estimates of revenue from projects. But that's not something that you can-- it might be fine at the planning stage to simply find out is there sufficient revenue to make this project a good candidate for a P3 but it would definitely not satisfy the markets that this project has enough revenue in it.
Victoria Farr: Right, thank you. And the final question in the chat box is from Kathy. What are retained risks and how do you calculate them?
Patrick DeCorla-Souza: Okay, so retained are the risks that the public sector retains in a conventional procurement it's the risk of cost overruns, the risk of delays which also lead to cost overruns. If it's a toll project it can be also revenue risk. Now of course there's several factors that play into these cost increases and delays that are factors that affect these three primary risks and the way you calculate them is by going through a risk assessment and I think that is going to be part of, at least we will introduce risk assessment in the next lesson so perhaps we can wait until then to talk about it.
Victoria Farr: Sure and one more question just came in the chat box, Patrick from Mohamad. Is VfM specific to the infrastructure projects or are there any synonyms?
Patrick DeCorla-Souza: Well VfM or value for money is a term that has been used in practice and I think it started in the UK, in Australia and Canada and all other countries have adopted that term. So I don't know of any other synonyms for VfM. Benefit cost analysis is the closest but as I mentioned, benefit cost analysis includes much more than financial evaluation which is what value for money is. The term money really is cash, so it's saying value for cash. And so that's the term that has been used conventionally all over the world.
Victoria Farr: Thank you for that clarification Patrick. We are now actually right on schedule to move onto lesson three. There are no further questions in the chat box so we'll just go ahead and move on.
Patrick DeCorla-Souza: Okay so now we get into what is value for money and this lesson will also go into the process which includes risk assessment and I'll talk about that. So value for money as it says here is the optimum combination of life cycle cost and quality of a good or service to meet the user's requirements. And I like to think of that as the kind of decision you make for example when buying a refrigerator. There's a certain upfront cost, your investment in the refrigerator and then it uses electricity over its life and so you would normally or should take into consideration the entire cost over its entire life. So if you're comparing refrigerators, of course you want to make sure first the capacity of the refrigerators that you're looking at are all equivalent, they have the same features, et cetera. And once you find say three or four refrigerators that meet your requirements then you want to buy the one that gives you value for money. So you would do a lifecycle cost of all of these refrigerators, some may have a high upfront cost but then they use less electricity over the life of the refrigerator. And so in order to do this comparison you do lifecycle cost analysis and since some of these costs that you're bearing in future for electricity for example are in what we call nominal dollars or year-of- expenditure dollars you have to find out their equivalent value in today's dollars so you do what's called a present value calculation. So that's what value for money involves is figuring out amongst all of your different options, and one of them is the conventional procurement and the others are various P3 options, of all these options, which one has the lowest cost to you in terms of financial outlay. So it's expressed as a dollar difference or a percent difference between the P3 option and the conventional delivery. And value for money analysis is simply the process that you use to compare these costs over the lifecycle. So it's a quantitative analysis, I indicated earlier, it's a financial analysis, not an economic analysis or not a cost benefit analysis. Looking only at cash flows and it's only looking at the balance sheet of the procuring agency. And this is very important to understand because when you are thinking of a toll facility, the procuring agency might decide that it will allow higher tolls on the facility in order to reduce the impact on its balance sheet. So by allowing higher tolls that go to the concessionaire, the concessionaire then says, "Okay, I'm getting more toll revenue so I can reduce the subsidy that I'm asking for from the public agency. " Well what has happened is that the cost has simply gotten transferred, it's not the-- the cost of that toll revenue is still being borne by someone, but it's not being borne the procuring agencies, it's being borne by the users. But since it does not appear on the balance sheet of the procuring agency, the procuring agency does not have to worry about it. This really points out the limitation of value for money analysis; it is only looking at the perspective of the procuring agency. Other benefits or other costs to users are not quantitatively considered but of course you could consider them in your qualitative evaluation which is part of value for money analysis which will talk about in a little bit. Some other definitions here that you will come across is first the public sector comparator and it is simply what I've been calling conventional procurement so far.
The cost of that conventional procurement is termed public sector comparator or PSC. So it's the baseline against which the P3 option is compared. And the P3 option before you get bids is called a shadow bid, that's the bid that you create, you create it yourself, it's not something that the market provides but you calculate it based on your best guesses so it's called a shadow bid. Now after you issue the RFP and you get bids, that will be called the actual bid and the best of all the actual bids will be called the preferred bid. So you're making comparisons between the PSC, public sector comparator and these various types of bids.
What bid you are comparing to depends on when you are making the comparison. So at the project development phase, we don't have a bid yet so we compare the PSC to the shadow bid to find out which one is the better deal which provides value for money. At the procurement phase we do a comparison first with a shadow bid created from the terms in the RFP but then when we get actual bids we make the comparisons with the actual bid so the PFC is always the baseline but you're comparing it with different kinds of bids, shadow bid, RFP type of bid, actual bid. And finally in the implementation phase you are looking at actual experience, so what did you actually have to pay in the form of availability payments of subsidies, et cetera and how does that compare with what the PSC would have cost you and did you get the value for money that you were expecting to have.
So the process for doing a value for money analysis involves six steps shown here. You first identify your procurement options both the baseline as well as the P3 options. You then go into calculating the value of risks and you have to of course identify the risks first and then calculate them and then decide which ones you're going to transfer to the private sector. You then, in step three develop the public sector comparator, step four, the P3 option called the shadow bid, step five make a comparison and finally bring in the qualitative factors.
So in identifying procurement options, first you have to figure out what your baseline is going to be, your conventional procurement options are generally design-bid-build but in many states, design-build has become conventional so in that case design-build would be our baseline. In some states like Florida design-build finance is becoming quite common so maybe that option would be the conventional procurement. And of course other various options such as contractor, manager, at risk. P3 procurement options and here you're talking about long term contracts, so they might be first, design build, finance, operate and maintain with a toll concession or with a toll concession or with an availability payment. And you can have an availability payment on a toll road but when you have an availability payment or you pay the contractor with an availability payment all toll revenues go to the public sector. And then you can have an availability payment also on facilities that don't have tolls. And finally, a form that hasn't been used in the U.S. yet, at least not on P3s is design, build operate, finance, operate, maintain with shadow tolls and shadow tolls are similar to real tolls except the tolls are paid not by the users of the facility but by the public agency. So you multiply the volume of the facility by the number of users on the facility and you calculate the total and that total is what is paid to the concessionaire. So it's kind of an availability payment calculated differently.
Risk assessment is the second step, you first have to identify the risks first and you do it for all phases, not just design and construction phase but also operations phase. You quantify those risks by figuring out what is the probability of their occurrence and what is the range of impact that they can have. And this information is thrown into an analysis process generally a Monte Carlo simulation that samples from all of these risks and goes through a number or iterations, picking different risks for each combination of risk, so it's basically a number of scenarios that are being modeled and what would their impact be on the total aggregate risk cost. So as a result of this analysis you come out with one total number, total aggregate risk cost for the project. And then once you have that you can then look at risk allocation and in this case for each individual risk you decide whether the private sector would be able to better manage that risk and if that's true you transfer that risk over to the private sector, if not, you retain it. And then if you want to figure out what is the aggregate value of the risks that you want to transfer to the private sector, then you can take those risks only and then go through a Monte Carlo simulation and figure out what the value of those risks are. If you use the same probabilities and the same range of cost impacts that you used for your conventional procurement, the result of that analysis will be the value of those risks to you. Now, when the concessionaire takes over these risks, they might be able to manage these risks better, either they might be able to reduce the cost impacts or they might be able to reduce the probability of occurrence. So you try to figure that out, you put those numbers into again the Monte Carlo simulation and you are able to then value the total impact of the risks on the concessionaire. So we have a tool within P3-VALUE that leads you through this process and the next webinar on September 20th, we'll go into how P3-VALUE does that and then we hope to have a homework assignment for you where you can actually use the P3-VALUE tool to see how this type of risk assessment is actually done. So I encourage you, if you're interested to attend the webinar on September 20th.
The public sector comparator is simply now aggregating different types of costs over the entire lifecycle. So you have the base costs which are not just for design and construction but also operations and maintenance over the entire project's life. You have to also add in financing costs. Remember the P3, the concessionaire is going to have financing costs in terms of the debt it incurs and the equity investment that has to be paid. So those are financing costs that are incorporated into the bid that it submits. So you need to also incorporate not just the baseline costs which I know we do that in our major project cost estimate review but what we also need to incorporate now in the public sector comparator is the financing cost and this is over the entire lifecycle. We need to incorporate not only the costs of the risks but if you are going to have to borrow to pay for these costs you need to include the cost of financing for those risks. Procurement and oversight costs and they might be actually lower, at least the procurement costs might be lower in the case of the public sector comparator because we all know how to do it, it's not something new, you don't need to hire experts, things of that type so procurement costs can be lower. And then something called competitive neutrality adjustments and what this does is simply adjusts for certain things that are incorporated in the concessionaire's bid but would rightly also should be incorporated into the PSC in order to have neutrality. An example is tax payments, concessionaires being the private sector they pay corporate taxes and some of those corporate taxes are paid to the state government and so if you're doing conventional procurement, the state government will not get those taxes because of course it's doing it in the conventional way and so there's no P3. So it is losing those taxes, something called an opportunity cost and so that needs to be incorporated into the PSC in order to have equivalency. Another example is insurance costs, a lot of times the state might self-insurance against liabilities, accidents and things like that or Workman's Compensation whereas the private sector has to actually go out and get insurance for the project. So we need to incorporate some of these costs into the PSC. One important thing is again for equivalency with the P3 option we need to estimate all these costs as if the conventional delivery would achieve the same standards as what we want the P3 to deliver. So if you want a certain quality from the P3 that we are including in the RFP we should estimate what it would cost the public sector to achieve those same standards. We also assume the same timeframe for construction and implementation. So if the P3 can be done sooner because it can bring in financing sooner, than a public sector delivery approach, it's going to be very difficult to make a comparison so you will have to actually assume that the public sector could deliver the project on the same schedule, otherwise you are not going to be able to make a fair comparison simply because of the way value for money analysis is done. Present values, if you assume that the public sector will deliver the project let's say after ten years and you try to discount the cost that you would be spending ten years from now, the project is going to look a lot less expensive simply because of discounting. So that won't be a fair comparison.
The next step is developing a shadow bid. So the shadow bid includes three components, the payments that would be made by the public sector to the concessionaire and these would be either a subsidy or an availability payment or it could be a shadow toll payment. Then it would include the retained risk because not all risks are transferred, some risks are still retained by the public sector and any other project costs for example the procurement costs in the case of a P3 option might be higher and then you need to incur some costs for oversight of the concessionaire and those need to be incorporated.
So once you add up all these costs you get something that looks like this, the public sector comparator on the left hand side. You see the various components, base cost, financing, other public costs, retained risks and competitive neutrality. On the right hand side, the shadow bid has the baseline costs and the financing costs which are the equivalent of what the concessionaire will actually bid and that's the amount that you would pay, the public sector would pay the private sector. And then you add in other public costs such as oversight and procurement costs and retained risks which would be a lot smaller and then the difference is the value for money.
Now it's possible that rather than having a positive value for money you might actually have a negative value for money but that doesn't mean necessarily that the decision is clear cut and you have to do a conventional procurement because you have other qualitative considerations that need to be brought into the evaluation. If the P3 for example accelerates project delivery you might get more user benefits from all the extra years of project life that you have. I mean it's earlier delivery, users get to use it sooner, they have benefits. There might be safety benefits, service quality benefits, reliability benefits from turning over the project to a concessionaire. You usually have performance standards. They try to maintain those standards and therefore, you might have better quality all around. Now, there are other issues that need to be considered that are contract related. Everything is not just quantitative in terms of this evaluation, and not everything about the project itself, but also contract related considerations. So, you might think that you can actually transfer all these risks, but in reality, if you can't create a contract that's sound enough to transfer all these risks that means that you're not going to be successful transferring those risks. So, contract issues are important. Can the contract or will the contract allow you a water tight way of transferring these risks? Can the contract give the concessionaires sufficient flexibility to innovate so that they can actually produce efficiencies? Is the contract maximizing the private sector's capabilities, and does the contract give the public agency flexibility to coordinate regional network policies? If you're turning over the entire project to the private sector, it could tie your hands in terms of region-wide coordination. For example, a simple thing such as a toll rates for example. If you want to have a network of managed lanes, and how do you coordinate that region-wide if you have different concessionaires operating all of these lanes with different rules, etc. So, some issues like that need to be considered in developing your contracts. And with that, I think it's the end.
So, what we're going to do is very quickly go through a hypothetical illustration. We'll show you a design build dimensional approach and show you how the two P3 options compare against them with a value for money analysis, and the assumptions are that we are going to achieve certain reductions in lifecycle costs with the P3 and maybe even a little extra revenue.
So, you start off with the initial estimates. Here, you see the baseline cost that you calculated for your PSE and for the availability payment, you, as a concessionaire, is including financing costs, as well as transferred risks. The reason with the toll concession, the P3 toll is the full concession. The reason the financing costs are higher is because lenders perceive that option is more risky and therefore charge higher interest rates.
Now, when you are looking at the total cost, however, you have to bring in retained risks and other costs such as procurement costs. So, the total cost to the public agency now appear to be slightly higher than the P3 options because of the high amount of retained risk.
Then, you consider the adjustments, and this is not to scale. This is only the upper part of the bars to more clearly show the differences. What we've done here is added tax adjustments and toll risk adjustments. In other words, in a public sector delivery, the total risk would remain with the public sector and with an availability payment situation, toll risk would also remain with the public sector.
We need to add the value of those risks to those bars, and that's what we've done here. So, you see that both the P3 options provide value for money; in the case of the availability payments, slightly more. But remember, with a toll concession, there are opportunities to try and get more revenues by designing the project to attract more users or other methods which the private sector can innovate in order to maximize revenue. And if the private sector is successful and has good ideas, they can bring in more revenue and therefore, reduce the public subsidy that they will require, which you see in this graph. In this situation, you have more value for money with the toll concession because of the ability of the concessionaire to bring in more toll revenue.
So, we need to make sure you understand that there are still a lot of VfM analysis. Although it's a state of the practice methodology, there are still a lot of challenges. First, as I said, the conventional delivery option is very difficult to configure. You need to look at lifecycle cost, something that public agencies are not used to looking at. You need to consider financing costs. You need to consider risks, and you need to make sure that the time frames are comparable for both P3 and conventional delivery. Pricing risk is not easy. There's a lot of estimation that's going on sometimes, a lot of guessing. So, you need to do much more sensitivity analysis on your pricing of risk. Allocation of risk to the party best able to handle it is important. Predicting traffic and revenue as was pointed out by the questioner earlier is very expensive to do, but it's an important part of ensuring that you do get market interest and are able to do a good analysis of the value of that toll revenue. You need to do sensitivity analysis because a lot of these estimates are not very clear cut; so, sensitivity analysis on a lot of the costs, on the discount rate used. This means that doing a Value for Money analysis is quite resource intensive and requires expertise from outside the state DOT in most cases.
Limitations are, again, the issues relating to risk evaluation and transfer of risk. The discount rate and how you choose the right discount rate is very important, and we are issuing some guidance on that, which I will talk about later. And finally, keep in mind, again, that non-financial costs such as user costs and environmental costs are not considered in Value for Money analysis. So, you really need to make sure you consider them in your qualitative evaluation. So, I turn it over to you, Victoria.
Victoria: Thank you, Patrick. So, another question here on the poll. It is another true/false. "The PSE assumes that the project can be delivered and operated to the same quality as the P3 option. " I'll give you about another five seconds to complete that poll, and I will close that out and make those results public. And it looks like the vast majority of people got the answer correct. Patrick, do you want to review it?
Patrick DeCorla-Souza: Yes. Yeah, I mean most of the people got it correct and the assumption that you have the same quality is important because that can affect your cost estimates, especially we know that public agencies are not very diligent in making sure that certain standards are maintained in the operations and maintenance of a facility. So, the quality can differ in actual practice. So, if you use your historical data to estimate your operations and maintenance costs, in other words what you've actually spent in the past on typical facilities, those may not be the right numbers because in your concession terms, you may be asking the concessionaire to maintain a certain higher quality. And so, you need to estimate what the cost would be to you if you maintain that higher quality.
Victoria: Thank you for that clarification and we did receive quite a number of questions in the chat pod. So, I've expanded that view. So, Patrick, I'm looking at - one, two, three - about four from the bottom from Richard. The question is, "As the answer to the P3 VfM analysis can be subject to assumptions, is it vetted and if so, who by and what stages?" So, I think he's asking if or when and by whom the assumptions for the VfM analysis are vetted.
Patrick DeCorla-Souza: That's the reason, of course, we are doing these Webinars because you need to be aware of the assumptions, the key assumptions that your consultant will be using to do the Value for Money analysis. It's a very complicated analysis involving very detailed financial models. So, it requires a lot of financial expertise. So, even if the financial model itself is pretty good and vetted, the assumptions that go into it may be used just based on whatever the consultant thinks is a good assumption. So, the reason we're doing these Webinars is so that the public agencies understand what these key assumptions are and they need to evaluate whether these assumptions that are used in the Value for Money analysis are good assumptions. And, of course, if you have an open process, and that's where the transparency issue comes in; if you have this open process, you will ensure that the assumptions you are making have gone through public and stakeholder review and everybody is comfortable with those assumptions. So, it's important to have a transparent process.
Victoria: Thank you, Patrick and the next question from Hamid is, "Isn't the VDOT NoVA outer beltway project a DBFOM with shadow tools?" He's not sure if that project is complete.
Patrick DeCorla-Souza: Well, I don't know whether you are talking about the existing-- the outer beltway, as far as I know, is still on the drawing boards. It has gone through several cycles of up and down, and I don't know where it's at now, but my understanding is it has been resuscitated. But now, it's nowhere near being completed, absolutely. I mean I think it's back on the plan, but I have no idea where it's at in terms of even project development.
Victoria: Okay and the next question from Mary. She's assuming that costs incurred by a project sponsor to do a Value for Money analysis would be eligible for reimbursement.
Patrick DeCorla-Souza: Well, again, this is something that obviously if it's in the planning stage, you can use, I'm sure. I'm not in the planning office here, but I assume it would be eligible for your normal planning funds if you're doing it at the planning stage and project development stage just like your EIS' are eligible for federal funding. Yeah, it would be. I don't see any reason why it wouldn't. But again, I'm not in the planning office. So, I can't give a definitive answer to that question.
Victoria: Okay and we got what looks like a comment from Michael regarding our most recent poll question, which just to remind you was a true or false that the PSE assumes that the project can be delivered and operated to the same quality as a P3 option. And for that, we said that the answer was true and Michael is saying that he emphatically disagrees, that false is incorrect, that the VfM analysis does not account for all the costs and benefits such as environmental costs. Thus, the PSE and P3 comparator are fundamentally different from a quality perspective.
Patrick DeCorla-Souza: Okay. So, I think what Michael is saying is actually correct, which is why we are going into the benefit cost analysis. All I am saying is that current practice is like this. I'm not saying it is correct, mind you. I never said that VfM analysis is the best way to do project or P3 evaluation. I actually agree with Michael because it's only a financial analysis. It does not account for everything. But, the assumption that is made by the financial experts that are doing this analysis is that the project is delivered to the same quality. This is simply for the cash flow issues. So, basically, where this comes in is the cash flows are affected by costs, that is construction costs, design costs and operations and maintenance costs. That's all the VfM analysis can handle. So, when the VfM analysis then, in order to calculate those costs for the public sector side is assuming that the road is of the same quality as the P3 option. But, obviously, you have to go much beyond Value for Money analysis in order to account for any differences in environmental costs, in user benefits, etc. That's why we have started a research project on using benefit cost analysis to do P3 evaluation, and we are going to update the Value for Money analysis tool we have. I mean it's currently called P3 Value, but we will be incorporating a module on benefit cost analysis so the kinds of things that Michael is talking about can be incorporated. So, I'm just reporting on the state of the practice. I hope I've made that clear.
Victoria: Thank you, Patrick, and we will be discussing that P3 Value tool kit in the next lesson. But before we get to that, we have a few more questions. The first one is from Mohammad, about four up from the bottom. He's asking, "How likely is it that a project qualifies in a traditional benefit cost analysis, but not in Value for Money?"
Patrick DeCorla-Souza: Well, since we haven't really done these analyses, you know, I have no way of knowing. But, we need to -- it really depends on the project I would assume, you know. I think one possibility is when a project would not be undertaken in the same year or would not be completed in the same year as the P3 option would complete the project. Maybe it would be delayed by ten years and then for Value for Money, since you can't do the analysis unless you assume that the project is delivered conventionally in the same year as the P3 option, you make a wrong assumption. Otherwise, you can't do a Value for Money analysis. So, the only way then you can do a correct analysis is to do a benefit cost analysis because with benefit cost analysis, you can account for the fact that the benefits now are being brought forward. Instead of the benefits being received ten years from now under conventional procurement, they are achieved in year one. And so, you have all these extra benefits that you can account for and a BCA could theoretically, in a case where the conventional procurement might delay the project, a BCA, a benefit cost analysis, might actually show a positive result for a P3 because of these extra benefits relative to the Value for Money.
Victoria: All right, Patrick and we have one more question that we'll take before moving on, and it's from Richard. "Are there any quantitative benefits from taking the project costs "off" the balance sheet for the agency?"
Patrick DeCorla-Souza: Well, again, that's an issue for the public agencies, but if they go above a certain debt capacity, my understanding is that the rating agencies downgrade their debt and of course, that has financial-- debt for the entire state and then that has a financial consequence for the state, which I'm sure you could calculate the value of that loss, or the extra amount that the state would have to pay on their debt, on their future debt. Now, my understanding is rating agencies are becoming smarter and may be actually now considering the fact that commitments are being made for things like availability payments, and they may be accounting for these future commitments to the cash flows of the agency. These are the kinds of things that were actually done in Europe, to take the project off the balance sheet and the agencies and that's why we have situations like Greece occurring where basically they were trying to not account for it in their budget.
Victoria: Okay, Patrick. So, we are nearing towards the end of our scheduled time here. So, let's move on to the fourth and final lesson here on the P3 Toolkit.
Patrick DeCorla-Souza: All right. Yeah, this should be fairly brief. I just want to let the audience be aware of our various tools that we have put up on our Web site and a lot of other information. And so, we've broken it down into four phases as you see here, the legislation and the policy phase, a couple of-- and you can download these at the end of the presentation.
So, this is some documents on legislation and policy, developing legislation for P3s.
For the evaluation phase, we have fact sheets, primers. Guide books are under development and should be up in a couple of months. We've got the screening tool I mentioned and we've got the P3 Value tool that I've also mentioned.
For procurement and oversight, we are developing model contracts for both toll concessions and availability payments, and we should have the one on toll concession up on our Web site by November. We're also developing best practices and some of that should be available by the middle of next year.
So, P3 Value is a set of four analytical tools as I mentioned and it helps people understand what the key assumptions are in Value for Money analysis in order to be able to guide your consultants.
So, here are the four tools: Risk assessment, public sector comparator tool, shadow bid tool, and financial assessment tool. So, the financial assessment tool simply brings in what has been calculated for the public sector comparator and what has been calculated for the shadow bid and compares the two.
Here's how the tools relate to one another. You have the risk assessment tool that creates these risk values and allocations that go into the development of the public sector comparator and the shadow bid, and then these two, the shadow bid and the PSE, are taken into the financial assessment tool to compare the two options.
There are some limitations you need to be aware of. These tools are for educational purposes only and not intended to guide decisions and actual projects. So, you may not evaluate your actual projects using this tool. You can use the tool to understand what key assumptions are. The reason is, as I mentioned, financial models are very complex and not only are the models complex, but the instruments, financial instruments are complex. I mean you have different kinds of debt and you've got equity and you've got bonds and bank loans and you've got different maturity periods, and all of these can be very complicated and P3 Value simply cannot handle the variety of financing instruments. So, it's simply to help you understand how Value for Money analysis is done. It is not applicable for brownfield projects and brownfield is projects that are existing facilities, generally toll facilities that you're trying to estimate what they might be valued at for putting on the market for a concessionaire to bid on. And we only do financial analysis. It does not include benefit cost analysis in the current version, but we are going to update it and then we will have a more complete tool that accounts for the kinds of things that Michael Replogle was talking about. All right. Over to you, Victoria.
Victoria: All right, Patrick. So, we now have our fourth and final poll question for you. True/false, "P3 Value is a learning tool and should not be used to evaluate actual projects?" I will give you about five more seconds to answer that poll and I will close that out now. Patrick, do you want to respond?
Patrick DeCorla-Souza: It is a learning tool and should not be used to evaluate actual projects. So, those who said it's true, they are right. You can obviously put whatever data you want in there to learn and understand your project and what key assumptions are, but you may not use this as an actual evaluation tool.
Victoria: Okay, thank you, Patrick. So, I'm going to do several things now. I'm going to change our layout so that now in the center of the screen, the participants can now view and evaluation, which is just a brief eight multiple choice questions and one open ended question to provide feedback on this Webinar. In the upper left corner are some instructions for downloading the presentation, which is available in the middle left side of your screen in a box entitled "Presentation download. " Essentially, if you just click that file name for the PDF, you then click, "Save to my computer," and follow the prompts on your screen. So again, that presentation is available now for download, although, Patrick, I'm realizing that we skip the course summary. Do you want to go through that very quickly, or should we just proceed with Q&A?
Patrick DeCorla-Souza: Why don't we proceed with Q&A.
Victoria: Okay, great. So, I'm looking two questions up from the bottom. It's a quite lengthy question from Michael again and he asks, "Is FHWA working on how environmental performance agreements might be incorporated into P3 contracts to enable them to deliver superior VfM compared to a PSE through incentive payments that ensure delivery of superior transportation projects and environment performance? For example, increasing vehicle occupancies, shifting to low carbon mode, improved congestion management, reduce air and water pollution, etc. "
Patrick DeCorla-Souza: Well, you know, we are working on this benefit cost analysis module and I guess I welcome anybody, including Michael, to participate on the review panel. We are looking for folks out there who might be interested in reviewing draft documents. Of course, these are privileged documents not to be shared. So, it's only for review, and I'd be happy to include Michael on my list of reviewers of these documents so that he can assist us in ensuring that these types of considerations are included in the benefit cost approach.
Victoria: Okay and the most recent entry in the chat box from Mary; she says that she hasn't seen any guidance stipulating the required sophistication of a P3 analysis for major projects. Is this addressed somewhere that she's not familiar with?
Patrick DeCorla-Souza: Well, there is no guidance on the requirements. We are issuing some sort of guidance on financial planning and major projects and it will reference our P3 screen tool. So, the P3 screening tool is all that a person might want to use to evaluate the potential of their major project for a P3. So, we are not requiring any detailed analysis, but simply screening evaluation, which is not a quantitative detailed evaluation.
Victoria: All right, Patrick. We just received another question from Mohammad. "Is there any particular reason why shadow tolls are not practiced much, especially in the U.S. ?"
Patrick DeCorla-Souza: Well, the U.K. used it a lot and they have dispensed with it and I think that there are probably some good reasons in the sense that when you're doing a shadow toll, you're actually transferring some risk over to the concessionaire, and it's sort of like a half risk. And in situations where you are trying to get the best deal from the private sector and you want to reduce financing costs, by increasing their risk, by shifting this so-called demand risk over to them, their financing costs are going to increase and therefore, their bids are going to be higher. So, there hasn't been much need to do a shadow toll because if you want to transfer risk, you just do a complete toll concession. Why do this halfway house. There's really no benefit. I mean if you're really worried about risk, you just do a real toll concession.
Victoria: Thank you, Patrick and I did want to remind our audience since there were several questions regarding the presentation, that it is currently available for download in the middle of your screen. I have also provided a hyperlink in the chat box where we have course archives of all of our previous Webinars on P3 and other related topics. So, the recording for this particular presentation as well as the presentation will be posted to that site pretty soon. So again, if you're FHWA staff, you may access that site. If you are non-FHWA, but you are interested in receiving the recording, you may contact Thay Bishop and her contact information is available in the presentation, as is Patrick's, which is available for download. And I just saw a comment from Eric that the link isn't working. If you are not FHWA staff, you will not have access to that hyperlink. So, Patrick, we are nearing the conclusion. Do you have any additional closing remarks that you'd like to share?
Patrick DeCorla-Souza: Have we dealt with all the questions? Are we missing some?
Victoria: No. We've answered all the questions that are in the chat box. So, we can certainly stick around if there's any additional Q&A, but--
Patrick DeCorla-Souza: Well, I guess what you might do is put up the upcoming P3 Value training slide.
Victoria: Okay, sure. So, we're going to return to the presentation briefly and again, this is available for download. But right now, we're going to navigate to a particular slide for Patrick to review.
Patrick DeCorla-Souza: Okay. So, I just want to mention the September 20th Project Assessment 201 Webinar for those that are interested and then if you have had some experience in Value for Money and want to get advanced training, we have a workshop scheduled, this is a live workshop, not a Webinar, on October 29th and 30th. And this workshop will be held in the Washington, D.C. metropolitan area. We are not going to charge any registration fees, but your travel is your own responsibility. So, there's no funding available for that. And then, of course, I've mentioned the updates to the P3 Value and we are going to release those late December/early January, and then we will have the Value for Money Analysis Webinar on the 23rd and we will have P3 Financial Structuring Assessment Webinar on March 13th. And you can register for those using the links shown at the bottom. I guess with that, I just want to thank all of you for your participation and for your questions and I want to thank the operator, Danielle and I want to thank, Victoria Farr, our moderator, and Aaron Jette who has been helping in the background. So with that, I turn it over back to you, Victoria.
Victoria: Thank you, Patrick. So, I have brought the evaluation back onto your screen as is the download, and we'll leave this up. So, for about a day, you can return to this link and continue to access and download materials. So, Patrick, thank you very much for taking the time today to present. Thank you very much to all of our audience members who joined us, and we look forward to seeing you at the September 20th or other later Webinars. Thank you.
Patrick DeCorla-Souza: Thank you.