Chapter 3: Evacuation Plan Assessment Methodology
This chapter describes the U.S. DOT's study methods used to evaluate the local, State, and Federal plans for evacuations related to hurricanes or other catastrophic incidents in the Gulf Coast region. The U.S. DOT, in coordination with the DHS, collected and evaluated information from State and local plans; held discussions with State and local emergency management, transportation, and law enforcement officials; conducted a review of literature and research on mass evacuations; and vetted concepts with representatives of transportation associations. The U.S. DOT also evaluated its own response to Hurricanes Katrina and Rita and incorporated these findings with the results of the State and local assessments to develop lessons learned, best practices, and findings and recommendations for this report.
Section 10204 of the SAFETEA-LU and the FY 2006 DOT Appropriations Act included specific factors that were a part of this evaluation, such as the potential modes of transportation in evacuations, coordination with neighboring jurisdictions, and communications before and during evacuations. Figure 3-1 lists the eight factors identified in the legislation. The U.S. DOT added other factors to develop a comprehensive view of mass evacuation planning in the region. These factors are discussed in more detail below.
The objective of the evaluation was to assess the status of evacuation planning processes in the Gulf Coast region to determine how well they address the various aspects of evacuation planning and operations, the constraints that State and local jurisdictions face in preparing and implementing evacuation plans, and actions that could be taken to improve evacuation capabilities. The purpose of the evaluation was not to grade the individual local and State plans, but rather to assess and gather information from those plans that will help local, State, and Federal agencies improve their evacuation processes in the future.
The study was led by a U.S. DOT team of representatives from the Office of the Secretary (OST), FHWA, FTA, FRA, FAA, and FMCSA, with assistance from a consultant team. The U.S. DOT also sought comments on the criteria from representatives of a number of its key partners and stakeholder agencies and associations. These included AASHTO, APTA, the American Trucking Associations, American Association of Railroads (AAR), American Bus Association (ABA), Amtrak and the National Council on Disability.
Scope of the Plan Evaluation
Hurricane evacuation plans are prepared at various levels of government including State, county, and municipal, and may be prepared by a variety of agencies ranging from a State DOT, to a State emergency management agency, to a municipal public works agency, or even to a State highway patrol agency. For the evaluation of the adequacy of State and local evacuation plans, the U.S. DOT collected and assessed the plans from various agencies within a State, as well as neighboring jurisdictions that were involved, and assessed how the agencies coordinated their plans across geographic and political boundaries.
Figure 3-2: States Included in Assessment
The review included the State and local jurisdictions in the Gulf Coast region that were involved in major hurricane evacuations (i.e., the States of Alabama, Florida, Louisiana, Mississippi, and Texas). The State and local evaluation reviewed plans for moving people away from areas threatened by catastrophic incidents and for accepting evacuees from other areas, including emergency evacuation plans that were available from State, county, and parish emergency management agencies. The U.S. DOT reviewed evacuation plans for coastal counties and the adjacent counties along the Gulf Coast and many of the counties of Florida. In addition, the evaluation included an assessment of sheltering plans for the States of Arkansas, Georgia, Oklahoma, and Tennessee, because these states may be among those to receive large numbers of evacuees from a catastrophic hurricane. Figure 3-2 shows the States included in the assessments.
The U.S. DOT requested copies of the evacuation plans through three means:
- The DHS Nationwide Plan Review
- The Secretary of Transportation's letter to the Governors (see Appendix A)
- Direct contact with State and local emergency managers and transportation officials.
The U.S. DOT also consulted with the State emergency management agencies to validate that the plans provided included all relevant plans available in each State. While the U.S. DOT did not receive evacuation plans from every jurisdiction in time for the study, the Department collected and assessed evacuation plans from 63 jurisdictions in the Gulf Coast region. This included plans from each of the 5 Gulf Coast States and 58 plans from county and parish jurisdictions from Florida and the coastal counties of Alabama, Louisiana, Mississippi, and Texas. The set of collected plans provided a representative base from which to assess the overall adequacy of evacuation planning for those areas in the Gulf Coast region with the greatest threat of catastrophic hurricanes. Appendix B lists the jurisdictions and agencies included in the assessment of the 5 Gulf Coast States.
Coordination with DHS Study
The U.S. DOT and DHS worked together to coordinate the U.S. DOT study of mass evacuation planning for the Gulf Coast region and the DHS nationwide review of catastrophic emergency operations plans. The U.S. DOT and DHS coordinated the methodologies, criteria, and assessment tools for each study. The U.S. DOT and DHS study teams coordinated site visits and discussed feedback and results and findings and recommendations from the site visits.
Although the U.S. DOT and DHS studies are related, the scope and focus of the two are different. The DHS study assessed the overall emergency operations plans for catastrophic incidents for States, territories, and large urban areas nationwide. The U.S. DOT study focused on mass evacuation planning elements for States, counties, and parishes in the Gulf Coast region. Evacuation plans are generally a subset of or an annex to overall emergency operations plans.
Methodology
To meet the Congressional requirements for this study and assess the range of factors included in effectively planning and executing a mass evacuation, the U.S. DOT developed a systematic, analytical process to evaluate the readiness and adequacy of State and local jurisdictions to move people in the Gulf Coast region away from catastrophic incidents and to safe shelter. The U.S. DOT's methodology included identifying the major components of a comprehensive evacuation planning and implementation program, collecting current practices and information on evacuations, developing criteria to evaluate current plans, conducting on-site discussions with State and local emergency management officials, assessing the plans, identifying lessons learned and best practices from recent evacuations, and developing findings and recommendations to improve mass evacuation planning in the Gulf Coast region and nationwide. Figure 3-3 highlights the study process.
Identify Key Elements of Evacuation Planning and Implementation
The body of the report is been organized by the key elements related to mass evacuation planning and operations for catastrophic incidents. These elements are major categories of activities or functions that local, State, and Federal emergency management agencies would conduct to plan and execute a mass evacuation. Figure 3-4 shows the seven key elements of mass evacuation planning and operations.
Review Current Evacuation Practices
To establish the current framework that guides evacuation planning and implementation, the U.S. DOT collected and reviewed guidance, literature and research, cost data, and other information on evacuations.
- Review Existing Guidance: The U.S. DOT reviewed the current guidance for local, State, and Federal agencies regarding evacuations. This included the more recent NRP and NIMS from DHS as well as SLG 101, NFPA 1600, and NRC evacuation guidance that has been used for a number of years. These documents provide the framework, guidance, and considerations used by agencies throughout the government as well as NGOs such as the American Red Cross to prepare for and implement a mass evacuation. This guidance is discussed earlier in this report.
- Conduct Literature Review: The literature review provided additional information to guide the methodology and evaluation criteria for the study. The U.S. DOT collected relevant domestic and international evacuation reference materials, plans, policies, procedures, newspaper and magazine articles, journals, industry publications, and other documents. It reviewed and analyzed the documents and compiled the results into a bibliography. The literature review also identified lessons learned and best practices from after-action reports and case studies.
The literature review addressed both anecdotal information and documented assessments of hurricane evacuation experiences, lessons learned, and guidance for transportation, public safety, and emergency management agencies that may have been developed following evacuation events. It collected documentation on pre-event public information campaigns that agencies implement to inform the public of evacuation routes and procedures and to manage public expectations. The review also included documents related to both rural and urban situations. - Analyze Cost: The purpose of the cost analysis was to examine the cost of mass evacuation plans in the Gulf Coast region. The cost analysis was used to identify and define the costs associated with and incurred in mass evacuation plans. Very little data are available on the costs of developing and implementing evacuation plans, and the limited cost information received for this study is incomplete. State officials noted that determining the amount of funding allocated and spent for evacuation planning would be difficult since funding comes from many sources at the local, State, and Federal level and many agencies within the various levels of government have agencies involved in evacuation planning. Information from the Gulf Coast States on the costs of evacuation is provided in Chapter IV, the Planning Key Element section, later in this report.
Develop Plan Evaluation Criteria
The U.S. DOT developed evaluation criteria that focus on local and State actions necessary to plan for and implement a mass evacuation. These actions include written policy directives; coordination of planning processes; and provisions made to communicate information to evacuees before, during, and after evacuation. The criteria were used for the detailed review of individual written evacuation planning documents from the jurisdictions in the Gulf Coast region. The criteria were also reviewed and validated during the discussions with the State and local officials during the site visits.
The U.S. DOT developed the evaluation criteria from information gathered from current Federal guidance on evacuation planning and operations as well as recent questions raised after the evacuations for the catastrophic hurricanes in 2005. Specifically, questions used to evaluate plans were drawn from:
- FEMA SLG 101: Guide for All-Hazard Emergency Operations Planning
- HSPD-5: Management of Domestic Incidents
- NIMS
- NRP, Volunteer and Donations Management Support Annex.
- SAFETEA-LU
- FY 2006 DOT Appropriations Act
- Findings from recent government reports including the White House, House of Representatives, and GAO reports and the Task Force Report to the Texas Governor on the Hurricane Katrina response.
Table 3-1 lists the 20 general questions developed by the U.S. DOT to evaluate evacuation plans for this study. For each of the questions, a list of attributes (stated as questions) was developed to define and support the parameters of each question. For example, within the category of communicating evacuation considerations, one of the questions is: “Does the plan describe the provisions and methods for alerting citizens that evacuation may be necessary?” Within this general question, several more specific questions were used to evaluate the plan (e.g., Does the plan identify contingency plans to use if normal means of public communications are unavailable? Does the plan include provisions for communicating with special needs evacuees such as hearing, vision, and physically impaired?). Appendix E provides the detailed list of questions used to evaluate State and local evacuation plans.
Each attribute was also characterized into two distinct categories—essential and non-essential. Essential means that if the attribute were not addressed in the plan, executing key aspects of the evacuation would at risk. Non-essential means that failure to address the attribute could impair execution of the plan but would not be expected to cause key aspects of an evacuation to fail. These two categories were used to weigh individual attributes and prioritize activities..
| Plan Evaluation Questions for Mass Evacuation from Catastrophic Incidents | |
|---|---|
| Decision Making and Management | |
| D1 | Does the plan describe direction and control with respect to catastrophic evacuation? |
| D2 | Does the plan describe the provisions needed to execute a large-scale evacuation? |
| Planning | |
| P1 | Does the plan address evacuation planning considerations (e.g., decision making, communications, available transportation modes, special needs, and sheltering) with regard to catastrophic hurricanes and other catastrophic events? |
| P2 | Does the plan require organizations to prepare standard operating procedures that contain the detailed instructions that responsible individuals need to follow to accomplish assigned tasks? |
| P3 | Does the plan include provisions for returning evacuees to their homes? |
| Public Communication and Preparedness | |
| C1 | Does the plan describe the provisions and methods for alerting citizens that evacuation may be necessary? |
| C2 | Does the plan identify what will be done to keep evacuees informed during evacuation to reduce their level of mental and physical stress? |
| C3 | Does the plan describe the means the government will use to keep evacuees and the public informed on the specific actions they should take after the evacuation has started? |
| Evacuation of People with Special Needs | |
| N1 | Does the plan describe provisions for evacuating special needs populations including those in assisted living facilities, hospitals, and those living independently (e.g., people with physical, mental, cognitive, and developmental disabilities)? |
| N2 | Does the plan describe provisions for evacuating other special needs populations (e.g., people in schools, day centers, mobile home parks, prisons, and detention centers, as well as, people that do not speak English or who are tourists, seasonal workers, or homeless)? |
| Operations | |
| O1 | Does the plan describe the safe and practical transportation modes that will be available to move evacuees that cannot transport themselves (other than special needs populations)? |
| O2 | Does the plan include provisions to ensure availability of public and private transport modes and necessary transport operators? |
| O3 | Does the plan identify evacuation routes? |
| O4 | Does the plan address the use of contraflow measures? |
| Sheltering Considerations | |
| S1 | Does the plan require the establishment of mutual aid agreements with other jurisdictions to formalize access to and use of shelters? |
| S2 | Does the plan include provisions for informing shelter operators and evacuees about the locations of public shelters outside of the evacuation area and their status (e.g., full, accepting evacuees, accepting evacuees with pets, special needs shelters, etc.)? |
| S3 | Does the plan address strategies and responsibilities for shelter operations? |
| S4 | Does the plan include provisions for the care and protection of animals? |
| Training and Exercises | |
| T1 | Does the plan include provisions for training a volunteer cadre to support shelter management operations, transport of evacuees, and first aid stations along the evacuation routes, etc.? |
| T2 | Does the plan require periodic reviews and updates of the plan, exercises and/or drills, and after-action reports as part of the planning process? |
Evaluation of Written Plans
Evacuation planning and operations is a complex process that involves many emergency operation functions. Evacuation plans are generally included in a "family of plans." This means that evacuation plans are often components of the all-hazard emergency response plans developed by State and local governments. Requirements relating to evacuation planning are generally addressed in several parts of the plan, rather than only in an evacuation section. Evacuation plans are included in the basic plan, direction and control, evacuation, emergency public information, resource management, and mass care or the ESF annexes in the emergency response plan. In some jurisdictions, stand-alone hurricane evacuation plans have been prepared. Whether a single plan or a family of plans, these are the primary documents that governments rely on to detail what they will do in catastrophic incidents that warrant mass evacuations.
For this study, the U.S. DOT assembled and assessed these families of plans relating to evacuations for the Gulf Coast jurisdictions to determine how they address the evacuation planning and operational questions developed for this evaluation. The study also assessed how the jurisdictions' plans coordinated with their neighboring jurisdictions and then how all of the jurisdictions in the Gulf Coast region coordinate together to prepare for catastrophic hurricanes. This included assessment of the plans of the neighboring “host” states (i.e., Arkansas, Georgia, Oklahoma, and Tennessee) for sheltering evacuees from the Gulf Coast region.
Figure 3-5 illustrates how the study methodology starts with the assessment of individual plans and then assesses and compares the family of plans that include the individual plan. The evaluation then assesses the coordination with neighboring jurisdictions.
Figure 3-5: Study Methodology for Assessment of Individual Plans
The individual plan assessments included three steps:
Step 1: The first step is a detailed assessment of how a given plan addresses individual attributes for each of the 20 general evaluation questions:
- Each attribute is scored on a scale of 0 to 3.
0 = the attribute was not addressed in the plan
1 = the attribute was only partially addressed in the plan
2 = the attribute was completely addressed in the plan
3 = the plan exceeds the requirements of the attribute, and is a potential best practice.
Step 2: The second step is to score the degree to which the plan fulfills each general evaluation question.
- The attributes are sorted into essential and non-essential categories.
- The scores for each attribute from Step 1 are converted into weighted scores, using different weights for essential and non-essential attributes.
The weighted scores for all attributes are added and divided by the total of their weights, to determine the average weighted score for each question.
Step 3: The final step is to roll-up the average weighted scores from all of the plans for each general question. Based on the cumulative score, each question is then given a rating, based on the levels described in Table 3-2.
| Rank | Cumulative Score | Description | Representation |
|---|---|---|---|
| 4 | 1.51 – 2.00 | Very Effective |  |
| 3 | 1.01 – 1.50 | Effective |  |
| 2 | 0.51 – 1.00 | Partially Effective |  |
| 1 | 0.00 – 0.50 | Marginally Effective |  |
Conduct Site Visits
A key step in the U.S. DOT's study methodology was the on-site discussions with key State and local emergency management, transportation, law enforcement, and non-governmental organization officials. In these meetings, the U.S. DOT discussed and validated the initial results from the evaluation of the written evacuation plans. Written State and local plans did not contain all of the information needed to assess key issues pertaining to evacuation plans. For example, plans do not usually include the cost of implementing the plans, constraints on what can be included in the plans, and recommendations for actions to improve State and local evacuation capabilities beyond the scope of the plan.
The U.S. DOT conducted one visit in each State, and coordinated whenever possible with the DHS staff conducting the on-site peer reviews for Phase II of the DHS Nationwide Plan Review. The U.S. DOT was also able to participate in some of the DHS meetings, and its objectives for each meeting included:
- Review and validate the evaluation questions developed for the review
- Discuss any remaining questions from the review of State and local written evacuation plans
- Identify any other documents that may be available that further elaborate on key aspects of evacuation plans and generally confirm how key components of evacuation plans are handled
- Discuss costs of planning and implementing the plans with State and local officials and identify the extent to which costs are a significant constraint on evacuation planning and execution
- Discuss any other constraints that State and local officials identified as affecting their ability to effectively plan and execute evacuations
- Discuss actions that State and local officials believed could improve their ability to plan and conduct evacuations
- Discuss initial best practices and lessons learned from the literature review and identify potential additional best practices
- Discuss any updates to evacuation plans that may be underway or planned and the nature of changes expected in those updates
- Vet initial findings and recommendations with subject matter experts at the State and local level to clarify and refine the assessment.
The on-site meetings were a valuable source of additional information and provided a more comprehensive picture of the status of mass evacuation planning and operations in the region. The attendees at the State and local meetings were experts in evacuation planning and operations. Many had years of first-hand experience managing evacuations under very difficult conditions. Their knowledge, expertise, and experience were invaluable to this study. The U.S. DOT prepared a summary of each State visit, and a group of separate reports documents the constraints, recommendations, and best practices. Appendix D includes a list of the State and local agencies that participated in the on-site meetings.
Identify Lessons Learned and Best Practices
From the literature review, the U.S. DOT identified a number of lessons learned and best practices. These lessons learned and best practices were discussed with the participants in the State visits and coordinated with the partners and stakeholders to help identify the highest priority issues. Lessons learned identify gaps, mistakes, or problems encountered that can or should be fixed. A best practice is an effective practice that can be replicated in other jurisdictions. It is typically based on an actual event, so that it has been proven effective in the field.
After-action reports are prepared after drills and exercises and major events. The reports summarize the events and identify what did and did not perform efficiently and effectively. Emergency responders use this analysis of their capabilities to continually improve their plans and processes. The U.S. DOT evaluated a number of after-action reports from the 2005 hurricane season to glean lessons learned that would have broad applicability for jurisdictions in the Gulf Coast region.
Exercises test emergency response capabilities and the performance and skills of the agencies and individuals. Sometimes, exercises and drills are designed to severely strain the current capabilities of the emergency response agencies to the point of failure to help identify potential gaps and weaknesses and ways to improve plans, operations, and training. These types of challenging exercises provide valuable lessons learned. Likewise, actual events provide insights into the capabilities and needs of emergency responders.
The identified lessons learned and corrective actions will be shared with the DHS Lessons Learned Information Sharing portal, and Corrective Action Program associated with the DHS National Exercise Program, to ensure that they influence ongoing Federal, State, and local government preparedness programs and planning activities.
In addition, the U.S. DOT's study benefited from several other recent governmental reviews on emergency response to Hurricane Katrina. This included the Final Report of the Select Bipartisan Committee to Investigate the Preparation for and Response to Hurricane Katrina by the U.S. House of Representatives; The Federal Response to Hurricane Katrina: Lessons Learned from the Assistant to the President for Homeland Security and Counterterrorism; the Governors Task Force on Evacuation, Transportation, and Logistics in Texas; and several reports by the GAO. These studies identified a number of lessons learned that were useful for this study.
