Appendix C: Evaluation Methodology and Evaluation Criteria
Congress requested the U.S. DOT, in cooperation with the DHS, to "review and assess Federal and State evacuation plans (including the costs of the plans) for catastrophic hurricanes and other catastrophic events impacting the Gulf Coast Region and report its findings and recommendations to Congress." To meet the Congressional requirements for this study, the U.S. DOT developed a systematic, analytical process to evaluate the readiness and adequacy of State and local jurisdictions to move people in the Gulf Coast region away from catastrophic events and to safe shelter. The U.S. DOT's methodology included several major steps to collect information, evaluate and validate the criteria and results, and develop findings and recommendations to improve catastrophic evacuation planning in the Gulf Coast region and nationwide. These steps are described in more detail later in this chapter.
Hurricane evacuation plans are prepared at various levels of government including State, county, and municipal, and may be prepared by a variety of agencies ranging from a State DOT, to a State emergency management agency, to a municipal public works agency, or even to a State highway patrol agency. For the evaluation of the adequacy of State and local evacuation plans, the U.S. DOT collected and assessed the plans from various agencies within a State as well as neighboring jurisdictions that were involved, and also assessed how they coordinated their plans across geographic and political boundaries.
The assessment process was built on the following steps:
- Assessing against criteria. The team evaluated the State and local evacuation plans against common criteria, not against each other. The team's analysts began by carefully identifying the seven key areas that are most important in achieving evacuation planning goals — decision making and management, evacuation planning, public communications and preparedness, operations, sheltering considerations, and training and exercise. In each area, the team then identified the characteristics of effectively managed evacuation processes and framed these as questions that became the rating factors.
- Refining the factors. The team then identified subfactors, again framed as questions, that further refined each of the general evaluation questions. For example, a state that manages decision making well would have established processes for large-scale evacuations. A state with strong training and exercise programs would have plans with provisions for training a volunteer cadre.
- Collecting the most important information on the questions. The team then assembled information related to each of the questions including current Federal guidelines and examples of good practice from various sources.
- Analyzing the information through a collaborative process. The team of analysts and subject matter experts jointly analyzed and discussed the data and its implications. They combined their information and jointly assigned ratings for each of the plans.
- Conducting the process in transparent fashion. From the very beginning, the evaluation team committed to a process of transparency. The project's researchers and analysts consulted extensively with all involved officials before defining the criteria. Along the way, the team provided regular status updates on its progress.
Developing the evaluation questions against which to assess the Gulf Coast region's evacuation plans requires (among other things) an understanding of the transportation requirements needed to support evacuations, the characteristics of the underlying incident, and the transportation resources available to respond to the incident. A decision to evacuate residents, visitors, and businesses due to an approaching hurricane is not a decision for transportation agencies. Emergency management and senior elected officials make that decision. However, transportation personnel are a key resource for planning and operating a safe and successful evacuation.
The U.S. DOT developed evaluation questions that focus on State and local actions necessary to plan for and implement a mass evacuation. These actions include written policy directives; coordination of planning processes; and provisions made to communicate information to evacuees before, during, and after evacuation. The developed questions were used for the detailed review of individual written evacuation planning documents from the jurisdictions in the Gulf Coast region. The criteria were also reviewed and validated during the discussions with the State and local officials during the site visits.
Questions based on current Federal guidance were drawn from multiple sources including FEMA SLG 101: Guide for All-Hazard Emergency Operations Planning, HSPD-5: Management of Domestic Incidents, NIMS, and NRP, Volunteer and Donations Management Support Annex. Additional questions were developed based on factors cited in SAFETEA-LU; the FY 2006 DOT Appropriations Act; findings from recent government reports including the Task Force Report to the Texas Governor and the White House, House of Representatives, Senate, and GAO reports on the Hurricane Katrina response. Section 10204 of SAFETEA-LU and the FY 2006 DOT Appropriations Act included specific factors that were a part of this evaluation, such as the potential modes of transportation in evacuations, coordination with neighboring jurisdictions, and communications before and during evacuations. Additional transportation factors developed by the U.S. DOT ensure full consideration of transportation agency roles in mass evacuations.
The eight factors identified in the legislation that were reflected in the evaluation questions are as follows:
- All safe and practical modes of transportation available for evacuations;
- The extent to which evacuation plans are coordinated with neighboring States and adjoining jurisdictions'
- Methods of communicating evacuation plans and preparing citizens in advance of evacuations;
- Methods of coordinating communication with evacuees during plan execution;
- The availability of food, water, restrooms, fueling stations, and shelter opportunities along the evacuation routes;
- The time required to evacuate under the plan;
- The physical and mental strains associated with the evacuation; and
- The cost of the plan.
Table C-1 shows how Congressional study criteria apply to seven key elements.
|Factors from Section 10204 SAFETEA-LU and FY 2006 DOT Appropriations Act||Evaluation Questions|
|Key Element: Mass Evacuation Decision Making and Management|
|Key Element: Evacuation Planning|
|Key Element: Public Communications and Preparedness for Mass Evacuations|
|Key Element: Evacuation of People with Special Needs in a Catastrophic Event|
|Key Element: Mass Evacuation Operations|
|Key Element: Mass Evacuation Sheltering Considerations|
|Key Element: Mass Evacuation Training and Exercises|
Table C-2 lists the 20 criteria developed by the U.S. DOT for this study. For each of the criteria, a list of attributes (stated as questions) was developed to define and support the parameters of each criterion. The criteria were grouped according to the seven key elements of catastrophic evacuation planning and operations. These elements are major categories of activities or functions that local, State, and Federal emergency management agencies would conduct to plan and execute a catastrophic evacuation.
|Decision Making and Management|
|D1||Does the plan describe direction and control with respect to catastrophic evacuation?||a. Does the plan identify organizations and individuals (by position) responsible for providing data to support the evacuation decision?|
|b. Does the plan identify organizations and individuals (by position) responsible for reviewing these data, including the condition of the transportation infrastructure, and making evacuation recommendations to decision maker(s)?|
|c. Does the plan identify organizations and individuals (by position) responsible for issuing and amending evacuation orders?|
|d. Does the plan identify organizations and individuals (by position) responsible for executing and coordinating overall evacuation operations?|
|e. Does the plan identify organizations and individuals (by position) responsible for directing resources to execute evacuation orders?|
|f. Does the plan identify organizations and individuals (by position) responsible for authorizing/releasing information to media and the public?|
|g. Does the plan identify roles and responsibilities of different levels of government, and how these government agencies coordinate their efforts with each other?|
|h. Does the plan identify specific roles of the State and local transportation and law enforcement agencies in evacuation operations?|
|i. Does the plan identify variation(s) in direction and control for different types of catastrophic events that require evacuation?|
|D2||Does the plan describe the provisions needed to execute a large-scale evacuation?||a. Does the plan define specific criteria for issuing either voluntary or mandatory evacuation orders?|
|b. Does the plan contain pre-approved drafts of executive orders for evacuations?|
|c. Does the plan address amending or revising the evacuation orders?|
|d. Does the plan describe time-phasing of evacuation execution (i.e., sequential and concurrent activities) for different levels of evacuation response?|
|e. Does the plan identify specific contingency plans that can be used if conditions change during the course of an evacuation?|
|f. Does the plan contain provisions to announce closings of schools and businesses in the risk area?|
|g. Does the plan contain provisions to identify the number of evacuees that will require transportation assistance?|
|h. Does the plan contain a strategy for special needs populations?|
|i. Does the plan contain a strategy for securing and restricting access to evacuated areas?|
|P1||Does the plan address evacuation planning considerations (i.e., decision-making, communications, available transportation modes, special needs, and sheltering) with regard to catastrophic events?||a. Does the plan address the catastrophic hazards that would cause a large-scale evacuation?|
|b. Based on the catastrophic hazards, does the plan identify and prioritize the communities that should be evacuated?|
|c. Based on the catastrophic hazards, does the plan identify the number of people and vehicles to be evacuated?|
|d. Based on the catastrophic hazards, does the plan identify decision points/triggers for implementation of evacuation?|
|e. Based on the catastrophic hazards, does the plan identify the estimated time needed to complete the evacuation, and is this coordinated with highway, law enforcement, and transit agencies?|
|f. Based on the catastrophic hazards, does the plan identify the distance evacuees must travel to ensure their safety?|
|g. Based on the catastrophic hazards, does the plan identify the estimated time public sheltered evacuees will need support?|
|h. Does the plan require tasked organizations to identify, obtain, and maintain resources and equipment required to support all aspects of the evacuation (e.g., evacuating people that do not have their own means of transport, evacuating special needs populations, communicating information to the public, managing traffic flow, etc.)?|
|i. Does the plan include host communities in disaster declarations?|
|j. Does the plan require periodic reviews and updates of the plan, exercises, and/or drills, and after-action reports as part of the planning process?|
|P2||Does the plan require organizations to prepare standard operating procedures that contain the detailed instructions that responsible individuals need to follow to accomplish assigned tasks?||a. Does the plan require organizations to prepare standard operating procedures that contain the detailed instructions that responsible individuals need to follow to accomplish assigned tasks?|
|P3||b. Does the plan address procedures in place for handling re-entry, and communicating these to the public?|
|c. Does the plan include provisions for transporting evacuees back to their place of residence?|
|d. Does the plan identify personnel resources and equipment required to support re-entry?|
|e. Does the plan include provisions for assisting evacuees in reuniting with family members?|
|Public Communications and Preparedness|
|C1||Does the plan describe the provisions and methods for alerting citizens that evacuation may be necessary?||a. Does the plan contain provisions to manage timely communication of evacuation instructions to prepare citizens in advance of the order to evacuate?|
|b. Does the plan identify contingency plans for use if normal means of public communications are unavailable?|
|c. Does the plan provide for communicating with people that have limited English proficiency?|
|d. Does the plan include provisions for communicating with special needs evacuees (e.g., people with hearing, vision, and physical impairments) and the means to communicate?|
|e. Does the plan provide for keeping the public informed of the estimated travel times to safe haven under current conditions?|
|f. Does the plan address the communication of alternative routes and/or transportation modes?|
|g. Does the plan contain sample/prescripted messages for use?|
|C2||Does the plan identify what will be done to keep evacuees informed during evacuation to reduce their level of mental and physical stress?||a. Does the plan address informing evacuees of available transportation modes, how to access them, and any restrictions on what evacuees may carry with them?|
|b. Does the plan address informing evacuees about when transportation assistance will begin and end and the frequency of departure at designated pick-up locations?|
|c. Does the plan establish times for public officials to provide updates, and does the plan address informing the public on when to expect such updates?|
|d. Does the plan address informing evacuees of their destination before boarding public transport?|
|e. Does the plan address informing the public and/or family members of evacuee destination?|
|f. Does the plan address providing the public with information about available services along evacuation routes and at shelters?|
|g. Does the plan identify established web sites, hotlines, etc. where people can get answers to their questions/concerns?|
|h. Does the plan address communicating security measures to the public so they are not concerned about possessions left behind?|
|i. Does the plan identify support services that are available for those with special needs?|
|C3||Does the plan describe the means the government will use to keep evacuees and the public informed on the specific actions they should take after the evacuation has started?||a. Does the plan indicate that highway conditions are reported on emergency radio announcements for travelers en route?|
|b. Does the plan identify Intelligent Transportation System (ITS) resources that are used to communicate to the public (e.g., 511, changeable message signs, highway advisory radio, web sites, etc.)?|
|c. Does the plan address contingency plans in place if normal means of communications are unavailable?|
|Evacuation of People with Special Needs|
|N1||Does the plan describe provisions for evacuating special needs populations including those in assisted living facilities, hospitals, and those living independently (e.g., people with physical, mental, cognitive, and developmental disabilities)?||a. Does the plan include provisions for movement of required assisted devices such as wheelchairs, life support systems, service animals, and communication equipment?|
|b. Does the plan identify special needs populations in the community by geographic location (e.g., hospitals, nursing homes, assisted care facilities, etc.)?|
|c. Does the plan contain provisions for coordination with the most appropriate community outreach agencies to transport special needs populations?|
|d. Does the plan contain a policy with respect to bringing service animals and pets to shelters or on board transit vehicles, planes, helicopters, etc.?|
|e. Does the plan include provision for sheltering these evacuees?|
|N2||Does the plan describe provisions for evacuating other special needs populations (e.g., people in schools, day centers, mobile home parks, prisons, and detention centers, as well as people that do not speak English or who are tourists, seasonal workers, or homeless)?||a. Does the plan make provisions for schools and day care centers?|
|b. Does the plan make provisions for mobile trailer parks and campgrounds?|
|c. Does the plan make provisions for incarcerated residents (e.g., those in prisons, jails, juvenile facilities, and drug treatment centers)?|
|d. Does the plan make provisions for transient populations (e.g., vacationers, seasonal workers, homeless, etc.)?|
|e. Does the plan make provisions for non-English speaking evacuees?|
|O1||Does the plan include provisions to ensure availability of public and private transport modes and necessary transport operators?||a. Does the plan include monitoring evacuation operations of motorized transport, rail, air, water, and other modes of transportation to determine the adequacy of available resources?|
|b. Does the plan address contingency plans for obtaining additional evacuee transport vehicles, if needed?|
|c. Does the plan provide coordination to prevent over-tasking of transportation resources where neighboring jurisdictions also require support from the same resource provider?|
|d. Does the plan address coordination with the next higher level of government (local to State, and State to Federal) to ensure unmet transportation resource needs are identified and requested to support evacuations?|
|e. Does the plan address the use of standby contracts with motor coach companies, paratransit providers, ambulance companies, railroads, air carriers, etc., to obtain drivers and vehicles to fill identified transportation shortfalls?|
|O2||Does the plan describe the safe and practical transportation modes that will be available to move evacuees that cannot transport themselves (other than special needs populations)?||a. Is the plan coordinated with the agencies and organizations that are tasked to provide transportation resources?|
|b. Does the plan include the use of public transit vehicles, school buses, and paratransit vehicles?|
|c. Does the plan include the use of non-government agency vehicles (e.g., church buses, volunteer group vehicles, etc.) as a contingency?|
|d. Does the plan identify accommodations for the transport of luggage, etc., and how evacuees are informed about any restrictions or limitations?|
|e. Does the plan address provisions made to transport evacuees to pick-up points/staging areas?|
|O3||Does the plan identify evacuation routes?||a. Does the plan require routes to be designated, marked, and communicated to the public?|
|b. Does the plan address the safety of routes (e.g., roads, bridges, railways, waterways, airstrips, etc.) that will be used?|
|c. Does the plan identify the provisions made to monitor traffic conditions on the evacuation routes and make operational adjustments as appropriate to maximize throughput?|
|d. Does the plan require critical operational changes be communicated to the Emergency Operations Center (EOC) and the public?|
|e. Does the plan identify provisions to control access to evacuation routes and manage traffic flow?|
|f. Does the plan address strategies and responsibilities for maintaining evacuation route capacity, especially with regard to work zones, toll collection, vehicle incidents, etc.?|
|g. Does the plan identify a system for notifying and coordinating with neighboring jurisdictions about what routes will be used and when?|
|h. Does the plan address strategies and responsibilities for providing food, water, restrooms, fueling stations, and rest stations for evacuees along the evacuation routes, including those for special needs populations?|
|i. Does the plan include resource lists to support evacuation routes (including all resource needs identified above)?|
|O4||Does the plan address the use of contraflow measures?||a. Does the plan contain specific contraflow plans that have been developed and coordinated with all affected parties?|
|b. Does the plan address strategies for emergency responders, transit vehicles, and other essential equipment to move inbound against the predominant outbound flow?|
|c. Does the plan address communicating information to the public on the contraflow plans and the beginning and ending times for contraflow operations?|
|d. Does the plan identify a system for communicating and coordinating contraflow operations to neighboring jurisdictions?|
|e. Does the plan indicate that the contraflow plans have been tested in an exercise or drill, or in a real evacuation?|
|S1||Does the plan require the establishment of mutual-aid agreements with other jurisdictions to formalize access to and use of shelters?||a. Does the plan indicate whether the agreements provide for sufficient spaces to house the worst-case estimate for the number of evacuees that will need shelter?|
|b. Does the plan include provisions for notification and coordination with host communities, non-governmental organizations, and other risk communities to address shelter activation and operations?|
|S2||Does the plan include provisions for informing shelter operators and evacuees about the locations of public shelters outside of the evacuation area and their status (e.g., full, accepting evacuees, accepting evacuees with pets, special needs shelters, etc.)?||a. Does the plan contain a system for keeping shelter information current during the evacuation?|
|b. Does the plan contain a system for acquiring and communicating information to evacuees on the availability of non-public shelters such as hotels?|
|c. Does the plan include provisions for keeping shelter operators informed on the location and status of alternate shelters?|
|d. Does the plan require travel routes to the shelters and the shelters to be marked?|
|S3||Does the plan address strategies and responsibilities for shelter operations?||a. Does the plan provide for food, water, and sanitation at shelters?|
|b. Does the plan provide for first aid and mental health support at shelters?|
|c. Does the plan provide for security at shelters?|
|d. Does the plan provide for disaster welfare information at shelters?|
|e. Does the plan provide for special needs population sheltering?|
|f. Does the plan provide for distinct population sheltering?|
|g. Does the plan include a resource listing and capabilities to support shelter operations included in the above elements?|
|h. Does the plan provide for refuges of last resort?|
|S4||Does the plan include provisions for the care and protection of animals?||a. Does the plan include provisions to evacuate people and their companion animals?|
|b. Does the plan contain policy with respect to sheltering pets?|
|c. Does the plan address communicating to the public the locations of shelters that accept pets and if they are equipped to meet the needs of pets?|
|d. Does the plan address measures for trying to reconnect families with their pets?|
|e. Does the plan address provisions for identifying and evacuating other animals (e.g., animals in farms, kennels, veterinary hospitals, zoos, theme parks, pet stores, and university laboratories)?|
|Mass Evacuation Training and Exercises|
|T1||Does the plan include provisions for training a volunteer cadre to support shelter management operations, transport of evacuees, and first aid stations along the evacuation routes, etc.?||a. Does the plan provide for trained personnel to support shelter management operations?|
|b. Does the plan provide for trained personnel to support the evacuation route (food, first aid, information, etc.)?|
|T2||Does the plan require periodic reviews and updates of the plan, exercises, and after-action reports as part of the planning process?||a. Does the plan require periodic reviews and updates of the plan, exercises, and after-action reports as part of the planning process?|
The study methodology, illustrated in Figure C-1, started with an assessment of individual plans, followed by an assessment and comparison of the family of plans, which included the individual plans and the coordination with neighboring jurisdictions. The complete list of plans included in the assessment is presented in Appendix C.
Figure C-1. Evacuation Plan Assessment Approach
The individual plan reviews included three steps that began with a detailed assessment of individual attributes, and then "rolled up" the information to increasingly higher levels for an overall assessment. The first step examined how the plan addressed the evaluation attributes of each criterion. The second step weighted the attributes for each criterion to provide an assessment of that criterion. The third step evaluated the 20 criteria to give an overall assessment of how well the plan meets the criteria. The individual steps are described in more detail below.
Step 1: Detailed assessment of how the plans address the individual attributes for each question.
Each attribute is scored on a scale of 0 to 3.
0 = the attribute was not met in the plan
1 = the attribute was only partially met in the plan
2 = the attribute was completely met in the plan
3 = the plan exceeds the requirements of the attribute, and is a potential best practice.
Step 2: Scoring of the degree to which the plan addresses each question.
The scores from each attribute were sorted into essential and non-essential. The scores from Step 1 for each essential attribute were added and then divided by the total number of essential attributes within a given question to determine the average weighted score for that question. This produced a weighted score for each of the 20 questions for the plan.
Step 3: Assessment of plans based on the cumulative results of the ratings.
Each question is given a rating, and the plans are assessed based on the cumulative results of the ratings. The overall evacuation plan criteria ratings are listed in Table C-3.
|4||All essential attributes are rated as "Meets" or higher, and all non-essential attributes are rated as "Partially meets" or higher.|
|3||At least 75% of essential attributes are rated as "Meets" or higher, and non-essential attributes are rated as "Does not meet."|
|1||No more than 50% of the essential attributes are rated as "Partially meets," and less than 25% of the non-essential attributes are rated as "Does not meet."|
|0||More than 25% of the essential attributes are rated as "Does not meet."|
|Attributes:||Detailed questions to describe and define the parameters of each criterion.|
|Essential:||If the attribute is not met, fulfilling the criteria and executing the evacuation is at serious risk of failure.|
|Non-Essential:||Fulfilling the criteria and executing the evacuation may be impaired or degraded if the attribute is not met, but will not result in failure by itself.|
The evaluation forms, guidelines and instructions, a keyword search list, daily team review and reporting, and evaluation methodology training ensure consistency across the evaluations. Before beginning actual reviews, all team members reviewed and assessed a single "family of plans" to test the template and determine whether it is being applied consistently by the reviewers. Adjustments were made to the instructions and template as necessary.
The team identified keywords for searching the documents to allow the reviewers to focus on key aspects of the evacuation plan and, therefore, reduce the amount of time spent on reviewing non-key portions of the plans. A worksheet template was developed for plan reviewers' use in capturing their evaluations, and was used to document relevant facts applicable to the rating assigned to each plan or family of plans assessed. The plan review team compiled a qualitative assessment of their evaluation findings against the defined list of evaluation criteria using a ranking tool to indicate the degree of integration of each question in the plan. The result of the assessments yielded a snapshot of the strengths and weaknesses of each plan reviewed. The team has also compiled the rating across the plans to observe trends and identify common problems that exist throughout the region.