Evaluation

Exercise Evaluation 

Discussion-based exercises tend to focus on higher-level capability issues involving an entity’s plans, policies, and procedures. As such, many discussion-based exercises use breakout sessions and other exercise techniques different from those used in operations-based exercises. In the breakout session approach, a lead facilitator frames the scenario and poses discussion questions; players then break into subgroups, based on discipline or jurisdiction, to discuss the questions with a table facilitator. In such exercises, there must be evaluators and/or note-takers present in each breakout group. It may be desirable to assign both an evaluator and a note-taker to each breakout group so that the evaluator can focus on addressing issues related to exercise objectives and the note-taker can focus on capturing general discussion issues.

Exercise Evaluation Guides (EEGs) help evaluators collect and interpret relevant exercise observations. EEGs provide evaluators with information on what tasks they should expect to see accomplished or discussed during an exercise, space to record observations, and questions to address after the exercise as a first step in the analysis process and development of the AAR/IP.

During the exercise, each evaluator uses the evaluation form to record data for critical topics and subjects that the lead evaluator has assigned him/her to assess. Exercise objectives—and the associated capabilities, activities, and tasks—determine the type of evaluation form used. Since these forms are based on the exercise objectives and EEGs, the content of these forms drives the facilitated discussion and also provide evaluators with guidelines and additional space for collecting relevant data while they observe exercise discussions. Evaluation forms should include questions linked to the capabilities, activities, and tasks within the EEGs to produce an effective evaluation that supports an overall capabilities-based exercise program.

Facilitators help evaluators collect useful data by keeping discussions focused on capabilities and activities relevant to the questions provided in discussion-based versions of the EEGs. Strategies for keeping discussion focused and constructive may be recorded in a SitMan or appendix to the SitMan, which is provided to facilitators prior to an exercise.

Evaluators generally record the following types of information from player discussions:
  • What plans, policies, and procedures would players implement to prevent, protect against, respond to, or recover from the incident described in the exercise scenario?
  • Are roles and responsibilities of the various agencies/organizations clearly defined?
  • How are various decisions made? Who has authority to make decisions?
  • What information about the scenario, the hazard, the victims, and the risks to participants and the public is collected? Who collects it, and what do they do with it?
  • How is information shared with other agencies and with the public? What information is shared?
  • What are the roles of participating agencies/organizations? 
  • How are the resources distributed and controlled?
  • What recommendations for improvements are made by the group?
  • Which issues are unresolved or require follow-up?
  • What actions do players plan to take in order to address outstanding issues?
After breakout sessions take place, the entire group typically reconvenes to address any key issues, cross-disciplinary issues, or conflicting recommendations that were identified during breakout group discussions. Although individual evaluators are assigned to record discussions within a designated group, all evaluators should capture information on cross-cutting issues.

A debrief with the exercise planning team, facilitators, and evaluators should be held immediately following the exercise. The purpose of this debrief is to collect observations and thoughts about exercise conduct. The debrief also provides evaluators the opportunity to clarify points or collect any missing information. Following an exercise, evaluators may also supplement the data collected on their observation forms by collecting additional data from participants through discussions, Participant Feedback Forms, and facilitator notes. The lead evaluator should assign one or more members of the evaluation team to take detailed notes during the debrief. The debrief discussion is reflected in the preliminary analysis.

 For more information about how to develop EEGs, refer to the following FEMA HSEEP website: https://hseep.dhs.gov/pages/1002_EEGLi.asp