September is recognized as National Preparedness Month, which serves as a reminder that we all have a role in preparing, now and throughout the year, for the types of emergencies that could affect us where we live, work, and also where we travel. As part of EMSI’s goal to help clients prepare for incidents, this article will provide an overview of how to design, conduct, and evaluate an exercise. As noted in our previous Preparedness Month article, exercises are generally conducted after you’ve identified a need for planning, developed a plan, and conducted training to prepare your members to enact the plan.
But before you can effectively exercise your plan, you need to establish an exercise and evaluation team. This ensures that those personnel designated to evaluate the exercise understand its purpose, as well as being able to assist in the development of the scenario to ensure that the objectives can be evaluated. To help develop your exercise, an evaluation team leader is needed for all exercise planning discussions from the very start. Additionally, other personnel will serve as facilitators and controllers to keep the exercise moving and on course, while a Safety Officer is required to prevent or stop any dangerous activity. Although rare, the Safety Officer must have complete authority to immediately stop the exercise if he or she observes or becomes aware of any action or condition that jeopardizes the safety or welfare of anyone (observer, participant, or bystander) involved.
As a member of the exercise design team, the exercise evaluation team leader ensures that the objectives are valid and can be verified during whatever type of exercise is conducted. The team leader determines the manner in which the exercise will be reviewed, identifies members of the evaluation team, defines what documents need to be developed and used, and coordinates the planning of the post-exercise meetings (e.g., “hot wash” or lessons learned).
One purpose of the exercise design team is to determine exercise scope by:
- Getting intent and direction from senior officials;
- Gathering input from the exercise planning team; and Identifying:
- Exercise design requirements and conditions (e.g., assumptions and artificialities);
- Exercise objectives;
- Participant extent of play; and
- Scenario variables (e.g., time, location, hazard selection).
The exercise design team is also used to develop exercise documentation by obtaining the planning team’s input on exercise location, schedule, duration, and other relevant details. Exercise planning team members are assigned responsibility for activities associated with designing and developing exercise documents, such as the Exercise Plan (ExPlan) and the Situation Manual (SitMan), and coordinating exercise logistics.
Using a Master Scenario Events List (MESL), the exercise design team establishes the scenario’s timelines. A MESL is a descriptive timeline of all events and actions to take place during the exercise. This timeline serves as a script of what is to happen and includes injects to prod the exercise along its intended path. For facilitators and controllers, the MESL’s script provides a guideline of what is expected to happen during the exercise, thereby helping them ensure that the participants’ responses guide them towards the intended results and learning points. Use of a MESL assures that all exercise objectives are evaluated during the exercise and documents the results for inclusion in lessons learned/after action reporting.
After design and development activities are complete, the exercise is ready for prime time. Activities essential to conducting individual exercises include preparing for exercise play, managing exercise play, and conducting immediate exercise wrap-up activities. For discussion-based exercises, conduct also entails presentation, facilitation, and discussion. For operations-based exercises, conduct encompasses all operations occurring between the designated Start of Exercise (StartEx) and End of Exercise (EndEx).
A government and industry best practice is to develop and evaluate exercises in accordance with the Homeland Security Exercise and Evaluation (HSEEP) standards. EMSI’s cadre has years of experience using HSEEP, to include the Master Exercise Practitioner designation. HSEEP uses a common methodology for planning and conducting individual exercises. This methodology applies to exercises in support of all national preparedness mission areas. A common methodology ensures a consistent and interoperable approach to exercise design and development, conduct, evaluation, and improvement planning (see graphic below). Exercise practitioners are encouraged to apply and adapt HSEEP doctrine on exercise conduct to meet their specific needs. Throughout these efforts, the engagement of senior officials by practitioners will ensure that the exercise is addressing the guidance and intent of affected officials.
A key to any successful exercise is the inclusion of all interested parties. The time to find a friend is not when you need one. Your county or state Office of Emergency Management can also provide guidance and possibly assistance. Another reason to include partner organizations is that many agencies and organizations receive credit for participation in exercises towards their requirements. Cooperation with your partners also ensures that your exercise does not conflict with any other exercises that may be conducted at the same time in a nearby location. An unstated goal of any exercise should be to reduce redundancies, whether it’s exercises or duplication of effort during a response.
Evaluation is the cornerstone of an exercise and must be considered throughout all phases of the exercise planning cycle, beginning when the exercise planning team meets to establish objectives and initiate exercise design. Effective evaluation assesses performance against exercise objectives, and identifies and documents strengths and areas for improvement relative to core capabilities. An exercise evaluation guide (EEG) is developed to be used in reviewing the results of the exercise. This document tracks all activities performed during the exercise to ensure that they are consistent with the goals and objectives that were intended to be validated.
A Hot Wash provides an opportunity for exercise participants to discuss exercise strengths and areas for improvement immediately following the conduct of an exercise while their thoughts are still fresh. The Hot Wash should be led by an experienced facilitator who can ensure that the discussion remains brief and constructive. The information gathered during a Hot Wash can be used during the After Action Report/Improvement Plan (AAR/IP) process and exercise suggestions can be used to improve future exercises. Hot Washes also provide opportunities to distribute Participant Feedback Forms, which when completed by players, can be used to help generate the AAR/IP. Remember, the purpose of the hot wash is not to criticize any agency, organization, or individual, but to learn from it; exercises should always be considered a no-fault exercise. In addition to the general hot wash, each participating functional group should conduct their own specific debriefing (e.g., Operations Section), addressing issues unique to their own areas of responsibility. These functional specific debriefings should be conducted by the controller who oversaw that group during the exercise.
Once all the feedback, evaluation forms, and EEGs are received, the AAR can be completed. The goal of data analysis is to evaluate the ability of exercise participants to perform core capabilities and to determine if exercise objectives were met. During data analysis, the evaluation team consolidates the data collected during the exercise and determines whether participants performed critical tasks and met capability targets. Evaluators consider participant performance against all targets to determine the overall ability to perform core capabilities. Additionally, the evaluation team takes notes on the course of exercise play, demonstrated strengths, and areas for improvement. This provides the evaluators with not only what happened, but why events happened. Lessons learned from exercises can also be used to enhance future exercises in addition to real world responses.
After this initial data analysis, evaluators examine each critical task not completed as expected and each target not met, with the aim of identifying a root cause. A root cause is the source of or underlying reason behind an identified issue toward which the evaluator can direct an improvement. When conducting a root-cause analysis, the evaluator should attempt to trace the origin of each event back to earlier events and their respective causes. EMSI’s seasoned experts are able to apply years of firsthand field experience to the data interpretation that can help an organization identify opportunities for improvement, as well as a plan of action to affect those improvements.
Overall, the AAR should address if the objectives of the exercise were met, not just how well everyone performed. AAR focusing on how the systems or agencies performed, instead of individuals, the AAR provides a path for preparedness improvement so that your organization is ready when something bad happens.
EMSI’s seasoned cadre of exercise experts is ready and able to help your organization exercise and evaluate its preparedness. Visit our website at emsics.com to learn more or contact us with your specific questions.