Appendix C: Evaluation Methodology and Evaluation Criteria

Congress requested the U.S. DOT, in cooperation with the DHS, to "review and assess Federal and State evacuation plans (including the costs of the plans) for catastrophic hurricanes and other catastrophic events impacting the Gulf Coast Region and report its findings and recommendations to Congress." To meet the Congressional requirements for this study, the U.S. DOT developed a systematic, analytical process to evaluate the readiness and adequacy of State and local jurisdictions to move people in the Gulf Coast region away from catastrophic events and to safe shelter. The U.S. DOT's methodology included several major steps to collect information, evaluate and validate the criteria and results, and develop findings and recommendations to improve catastrophic evacuation planning in the Gulf Coast region and nationwide. These steps are described in more detail later in this chapter.

Hurricane evacuation plans are prepared at various levels of government including State, county, and municipal, and may be prepared by a variety of agencies ranging from a State DOT, to a State emergency management agency, to a municipal public works agency, or even to a State highway patrol agency. For the evaluation of the adequacy of State and local evacuation plans, the U.S. DOT collected and assessed the plans from various agencies within a State as well as neighboring jurisdictions that were involved, and also assessed how they coordinated their plans across geographic and political boundaries.

Overall Methodology

The assessment process was built on the following steps:

• Assessing against criteria. The team evaluated the State and local evacuation plans against common criteria, not against each other. The team's analysts began by carefully identifying the seven key areas that are most important in achieving evacuation planning goals — decision making and management, evacuation planning, public communications and preparedness, operations, sheltering considerations, and training and exercise. In each area, the team then identified the characteristics of effectively managed evacuation processes and framed these as questions that became the rating factors.
• Refining the factors. The team then identified subfactors, again framed as questions, that further refined each of the general evaluation questions. For example, a state that manages decision making well would have established processes for large-scale evacuations. A state with strong training and exercise programs would have plans with provisions for training a volunteer cadre.
• Collecting the most important information on the questions. The team then assembled information related to each of the questions including current Federal guidelines and examples of good practice from various sources.
• Analyzing the information through a collaborative process. The team of analysts and subject matter experts jointly analyzed and discussed the data and its implications. They combined their information and jointly assigned ratings for each of the plans.
• Conducting the process in transparent fashion. From the very beginning, the evaluation team committed to a process of transparency. The project's researchers and analysts consulted extensively with all involved officials before defining the criteria. Along the way, the team provided regular status updates on its progress.

Criteria Development

Developing the evaluation questions against which to assess the Gulf Coast region's evacuation plans requires (among other things) an understanding of the transportation requirements needed to support evacuations, the characteristics of the underlying incident, and the transportation resources available to respond to the incident. A decision to evacuate residents, visitors, and businesses due to an approaching hurricane is not a decision for transportation agencies. Emergency management and senior elected officials make that decision. However, transportation personnel are a key resource for planning and operating a safe and successful evacuation.

The U.S. DOT developed evaluation questions that focus on State and local actions necessary to plan for and implement a mass evacuation. These actions include written policy directives; coordination of planning processes; and provisions made to communicate information to evacuees before, during, and after evacuation. The developed questions were used for the detailed review of individual written evacuation planning documents from the jurisdictions in the Gulf Coast region. The criteria were also reviewed and validated during the discussions with the State and local officials during the site visits.

Questions based on current Federal guidance were drawn from multiple sources including FEMA SLG 101: Guide for All-Hazard Emergency Operations Planning, HSPD-5: Management of Domestic Incidents, NIMS, and NRP, Volunteer and Donations Management Support Annex. Additional questions were developed based on factors cited in SAFETEA-LU; the FY 2006 DOT Appropriations Act; findings from recent government reports including the Task Force Report to the Texas Governor and the White House, House of Representatives, Senate, and GAO reports on the Hurricane Katrina response. Section 10204 of SAFETEA-LU and the FY 2006 DOT Appropriations Act included specific factors that were a part of this evaluation, such as the potential modes of transportation in evacuations, coordination with neighboring jurisdictions, and communications before and during evacuations. Additional transportation factors developed by the U.S. DOT ensure full consideration of transportation agency roles in mass evacuations.

The eight factors identified in the legislation that were reflected in the evaluation questions are as follows:

1. All safe and practical modes of transportation available for evacuations;
2. The extent to which evacuation plans are coordinated with neighboring States and adjoining jurisdictions'
3. Methods of communicating evacuation plans and preparing citizens in advance of evacuations;
4. Methods of coordinating communication with evacuees during plan execution;
5. The availability of food, water, restrooms, fueling stations, and shelter opportunities along the evacuation routes;
6. The time required to evacuate under the plan;
7. The physical and mental strains associated with the evacuation; and
8. The cost of the plan.

Table C-1 shows how Congressional study criteria apply to seven key elements.

Table C-2 lists the 20 criteria developed by the U.S. DOT for this study. For each of the criteria, a list of attributes (stated as questions) was developed to define and support the parameters of each criterion. The criteria were grouped according to the seven key elements of catastrophic evacuation planning and operations. These elements are major categories of activities or functions that local, State, and Federal emergency management agencies would conduct to plan and execute a catastrophic evacuation.

General Approach

The study methodology, illustrated in Figure C-1, started with an assessment of individual plans, followed by an assessment and comparison of the family of plans, which included the individual plans and the coordination with neighboring jurisdictions. The complete list of plans included in the assessment is presented in Appendix C.

Figure C-1. Evacuation Plan Assessment Approach

The individual plan reviews included three steps that began with a detailed assessment of individual attributes, and then "rolled up" the information to increasingly higher levels for an overall assessment. The first step examined how the plan addressed the evaluation attributes of each criterion. The second step weighted the attributes for each criterion to provide an assessment of that criterion. The third step evaluated the 20 criteria to give an overall assessment of how well the plan meets the criteria. The individual steps are described in more detail below.

Step 1: Detailed assessment of how the plans address the individual attributes for each question.

Each attribute is scored on a scale of 0 to 3.

0 = the attribute was not met in the plan

1 = the attribute was only partially met in the plan

2 = the attribute was completely met in the plan

3 = the plan exceeds the requirements of the attribute, and is a potential best practice.

Step 2: Scoring of the degree to which the plan addresses each question.

The scores from each attribute were sorted into essential and non-essential. The scores from Step 1 for each essential attribute were added and then divided by the total number of essential attributes within a given question to determine the average weighted score for that question. This produced a weighted score for each of the 20 questions for the plan.

Step 3: Assessment of plans based on the cumulative results of the ratings.

Each question is given a rating, and the plans are assessed based on the cumulative results of the ratings. The overall evacuation plan criteria ratings are listed in Table C-3.

Evaluation Consistency

The evaluation forms, guidelines and instructions, a keyword search list, daily team review and reporting, and evaluation methodology training ensure consistency across the evaluations. Before beginning actual reviews, all team members reviewed and assessed a single "family of plans" to test the template and determine whether it is being applied consistently by the reviewers. Adjustments were made to the instructions and template as necessary.

The team identified keywords for searching the documents to allow the reviewers to focus on key aspects of the evacuation plan and, therefore, reduce the amount of time spent on reviewing non-key portions of the plans. A worksheet template was developed for plan reviewers' use in capturing their evaluations, and was used to document relevant facts applicable to the rating assigned to each plan or family of plans assessed. The plan review team compiled a qualitative assessment of their evaluation findings against the defined list of evaluation criteria using a ranking tool to indicate the degree of integration of each question in the plan. The result of the assessments yielded a snapshot of the strengths and weaknesses of each plan reviewed. The team has also compiled the rating across the plans to observe trends and identify common problems that exist throughout the region.