A Systematic Review of the Efficacy of Full-Scale Simulation Exercises in Enhancing Hospital Disaster Preparedness

Introduction

the deteriorating situation (1). In recent years, the world has seen an increase in extreme scenarios, including both man-made and natural disasters, conflicts, and societal upheavals. The Third UN World Conference on Disaster Risk Reduction in 2015 affirmed the widespread impact of such events on a global scale, resulting in more than 700,000 deaths and 1.4 million injuries. Additionally, it was projected that these calamities affected more than 1.5 billion individuals, resulting in economic losses surpassing 1.3 trillion US dollars (2,3). It is therefore critical to prioritise timely, effective, and long-term interventions for vulnerable populations. Recognising the importance of the effective management of catastrophic events will ensure greater emphasis on generating and sharing information and training in this field to assure preparedness and competence in such crucial situations (4,5). According to the World Health Organization's Hospital and Health Facility Emergency Exercises Guidelines, there are five commonly utilised types of emergency exercise: orientation, tabletop, drill, functional, and full-scale simulation exercises (6). Simulation is a representation of real-life events and can take many forms, including computer software, case studies, written clinical scenarios, simulated patients (SPs), role-playing, or replicating reality with simple or advanced functionalities (7,8). This wide range of modalities allows for diverse and thorough simulation of real-world scenarios (7,8). Real simulation with actors has been used for almost 50 years, demonstrating its continued utility and efficacy in a variety of sectors (7,8). Full-scale exercises are characterised by their leadership (typically conducted by controllers) and the involvement of participants across all levels of personnel (6). These exercises should simulate a realistic setting, including the activation of emergency operations centres, and their duration can range from two hours to more than one day (6). The preparation for such exercises is extensive, often requiring 1–1.5 years of development (6). This preparation includes significant time, effort, and resources, as well as the incorporation of preparatory drills, tabletop exercises, and functional exercises (6).

Full-scale simulation exercises in hospital disaster preparedness are immersive, all-encompassing drills designed to imitate real-world emergency circumstances, and involving the active participation of various hospital departments, including healthcare professionals, administrative workers, and support personnel (8–10). The scenarios are meticulously designed to simulate natural disasters, mass casualties, or infectious disease epidemics, putting the hospital’s reaction and coordination capabilities to the test (11). The multidisciplinary approach ensures that teams collaborate effectively, and scenarios frequently vary to measure adaptation. Communication systems are exhaustively tested, and the entire exercise is analysed and followed by a thorough debriefing (10,11). Understanding the influence of full-scale simulations on hospital disaster preparedness is critical for a number of reasons. For starters, it provides a detailed examination of the strengths and shortcomings of current practices, allowing hospitals to discover gaps and opportunities for development in their preparedness plans. The findings can be used to design more resilient and adaptive protocols to handle real-world difficulties, ultimately improving the resilience of healthcare systems in the face of calamities. This systematic review aimed to illustrate these points, and to provide further recommendations for future researchers. The findings of this study were reported in accordance with the PRISMA statement requirements (12), and all procedures were carried out in exact conformity with the Cochrane Handbook of Systematic Reviews and Meta-Analysis of Interventions (version 5.1.0) (13). Due to the heterogeneity of the studies, the narrative synthesis approach was used to report the findings.

Eligibility criteria

Table 1. Description of inclusion criteria Arms Description Population Hospitals and their staff who have participated in disaster and emergency preparedness training. p420

Methods

Intervention Full-scale simulation exercises are the intervention of major interest. Outcome Recognising the challenges associated with noncompliance, evaluating the time required to assemble a crisis management team, and investigating the effect of full-scale simulations on the quality of the disaster plan.

Study Design

posters, studies from overlapped databases, and various non-primary research sources such as reviews, book chapters, theses, comments, letters, and editorials. Information sources and search strategy We performed a comprehensive search of four 434 All Fields: ((“Full-scale simulations“ OR “Full-scale regional simulations“ OR “Full-scale regional simulation“ OR FSRE OR “Simulations Scenarios“ OR “Full-scale exercise“ OR FSEs OR “Simulations Training“ OR “Simulations system“ OR “Simulations“ OR “High-fidelity simulation“ OR “Online simulations“ OR “Simulating“ OR “Simulator“ OR “Simulation modelling“ OR “Simulation modeling“) AND (“Preparedness“ OR “Readiness“) AND (“Hospital“)). Cochrane 138 Keywords, Title: ((“Full-scale simulations“ OR “Full-scale regional simulations“ OR “Full-scale regional simulation“ OR FSRE OR “Simulations Scenarios“ OR “Full-scale exercise“ OR FSEs OR “Simulations Training“ OR “Simulations system“ OR “Simulations“ OR “High-fidelity simulation“ OR “Online simulations“ OR “Simulating“ OR “Simulator“ OR “Simulation modelling“ OR “Simulation modeling“) AND (“Preparedness“ OR “Readiness“) AND (“Hospital“)).

Risk of bias assessment

According to the Cochrane Handbook of Systematic Reviews of Interventions 5.1.0 (updated March 2011), we used the National Institutes of Health (NIH) quality evaluation tools for cross-sectional, cohort, case-control, and intervention studies to assess the quality of quantitative studies (15). We used Noyes et al.'s recommendations for qualitative investigations, and the Mixed Methods Appraisal Tool (MMAT) for mixed-method studies (16). Two reviewers worked separately on the evaluation, and any inconsistencies were handled through discussion with a senior reviewer (17). Each tool had its own set of criteria, on the basis of which the studies were graded as good, fair, or poor.

Data synthesis

Results

Literature search results

A 36-Hour Unplugged Full-Scale Exercise: Closing the Gaps in Interagency Collaboration between the Disaster Medical Assistance Team and Urban Search and Rescue Team in Disaster Preparedness in Taiwan Evaluate preparedness and performance of DMATs in a simulated crisis scenario Foo 2021 (29) Compare effectiveness of simulated versus actual actor-patients in MCI drills Simulation in a Disaster Drill: Comparison of High-Fidelity Simulators Versus Trained Actors Gillett 2008 (30) Evaluation of Medical Command and Control Using Performance Indicators in a Full-Scale, Major Aircraft Accident Exercise Assess medical command and control in major aircraft accident using FSE at local and strategic levels Gryth 2010 (31) Test feasibility of incorporating patient's perspective into traditional evaluations of Developing the Fourth Evaluation Dimension: A Protocol for Evaluation of Video from the Patient’s Haverkort 2017 (32) Disaster scenario involving medical rescue teams showed effective command, control, and communication, and highlighted instances of noncompliance in health examinations. DMATs were successful but communication and evaluation skills need improvement. Lack of comprehensive analysis, potential subjectivity Non-validated survey, Likert scale issues, lack of inter-rater reliability testing, non-identical paired cases, risk of learning bias High-fidelity simulators and live actors produced equivalent results in mass casualty drills, emphasising the effectiveness of both. Simulators are as effective as live actors in MCI drills. Performance indicators assessed medical command and control. Management and staff skills scored 15 and 17 at the strategic level, and 17 and 21 at the hospital level, respectively, revealing feasibility for future evaluations and real incidents. Potential differences in exercise vs. real incidents, lack of absolute distinction between scores Performance indicators are effective for evaluation of command and control in FSE. Equipping mock patients with point-of-view cameras in trauma drills enhanced communication training and identified areas for Preliminary nature of study, images used by multiple observers Point-of-view camera enhances trauma drill evaluation. Protocols should be optimised Perspective During Major Incident Exercises hospital using simulated training Healthcare professionals’ experience of using in situ simulation training in preparation for the COVID-19 pandemic: a qualitative focus group study from a Danish hospital Investigate impact of ISS on COVID-19 preparedness Juelsgaard 2022 (33) Kohlhoff 2012 (34) Use of an Emergency Preparedness Disaster Simulation with Undergraduate Nursing Students Evaluate impact of EPDS on nursing students Kaplan 2012 (35) Full-scale regional exercises: Closing the gaps in disaster preparedness Assess FSRE in mass casualty disaster preparedness Klima 2012 (36) improvement. The approach improved non-medical aspects and identified areas for training and process enhancement. for effective utilisation of such footage. Positive outcomes from simulations for healthcare professionals, emphasising improved teamwork and skills in PPE use. ISS is valuable for pandemic preparation, at both organisational and individual levels. Focus on specific hospital, potential response bias NR EPDS received positive evaluations from 95% of students, indicating increased knowledge, confidence, and preparedness for disaster situations. The hybrid approach proved cost-effective and time efficient. EPDS enhances disaster preparedness, skills, and confidence, providing a cost-effective educational experience. NR Assessment of 16 hospitals showed widespread non-compliance, with communication problems Lack of training with technologies, communication issues FSE is crucial for MCI readiness, as a communication gap Operation continued care: A large mass-casualty, full-scale exercise as a test of regional preparedness McElroy 2019 (37) Assess coordination in mass-casualty FXS Utilization of Functional Exercises to Build Regional Emergency Preparedness among Rural Health Organizations in the US Design and implement disaster exercises in rural areas to test efficacy Obaid 2017 (38) Combining performance and outcome indicators can be used in a standardized way: a pilot study of two multidisciplinary, full-scale major aircraft exercises Demonstrate feasibility of evaluating multidisciplinary, full-scale disaster exercises using combined indicators Rådestad 2012 (39) Use of simulation models when developing and testing hospital evacuation plans: a tool for improving emergency preparedness Evaluate hospital evacuation preparedness for MCI using simulation real-time exercise Rådestad 2023 (40) (94%) and staffing issues (63%) being common. remains in tabletop exercises. Triage analysis revealed strengths but highlighted communication failures and areas for improvement in data entry and resource requests. FXS aids preparedness, highlights communication and coordination flaws. NR Functional exercises vital for testing decision-making and response, offering insights beyond tabletop or short, full-scale exercises. Functional exercises evaluated command staff, fostering regional collaboration. Revealed barriers to unified command and documentation issues. Limited resources Improved performance indicators, including preventable deaths and complications, were observed. Additionally, there were an adequate number of ambulances and injured victims considered. Standardised evaluation using combined indicators allows for meaningful comparisons between exercises. Competency variations, need for real incident validation

Tabletop exercises improved evacuation plans and critical operations. In two exercises, 71.2% and 52.9% of patients were evacuated within 7 and 4 hours. Positive feedback on survey results and a desire for more exercises. Limited generalisability due to varied exercises; low response rates; exercise did not include treatment during transport Simulation exercises help planners create more efficient evacuation plans, protocols, and procedures for MCI. Evaluate hospital incident command, triage, and patient transfer in FSS for MCI Full-scale simulation exercise—A preparedness for trauma mass casualty incident: Nepal Shrestha 2022 (10) Full-scale simulations to improve disaster preparedness in hospital pharmacies Evaluate impact of FSS on pharmacy emergency readiness Schumacher 2022 (41) Waterworks, a Full-Scale Chemical Exposure Exercise: Interrogating Pediatric Critical Care Surge Capacity in an Inner-City Tertiary Care Medical Center Assess PICU surge plan in chemical exposure exercise Shah 2013 (42) or upon arrival at facility Simulation exercise revealed gaps in knowledge and skills, indicating room for improvement in incident command, [mean score of 161 out of 220 (73.2%)], triage, and patient transferal. Simulation identifies gaps and highlights areas for improvement. Single site, observational design

Simulation exercises improved task completion. Four hospital pharmacies initially completed 69%±6% of assigned tasks, increasing to 84%±7% in the second round. Average quality of actions improved from 3.0/5 to 3.6/5, and crisis team assembly time reduced from 23 to 5 minutes with challenges in communication and crisis management. Small number of participating pharmacies, subjective assessments FSS significantly improves readiness and staff knowledge. Effective patient care transitions in chlorine overexposure simulation. Identified shortcomings in ED care. Lack of post-instruction testing, potential ED overstaffing bias PICU surge plan effective, but ED areas need attention. Randomized controlled trial of high-fidelity patient simulators compared to actor patients in a pandemic influenza drill scenario Compare resuscitation times in pandemic influenza high-fidelity simulation Wallace 2010 (43) Lessons Learned from an Active Shooter Full-Scale Functional Exercise In a Newly Constructed Emergency Department Assess hospital response to active-shooter scenario Wexler 2017 (44) ISS: Institutional Simulation Scenario, HOSPEX: Hospital Exercise, PEMAF: Emergency Plan for Massive Influx of Injured, MCI: Mass Casualty Incident(s), EPDS: Emergency Preparedness Disaster Simulation, FSE: Full-Scale Exercise, FSRE: Full-Scale Regional Exercise, ED: Emergency Department, DMAT: Disaster Medical Assistance Team, FSS: Full-Scale Simulation, PICU: Paediatric Intensive Care Unit, MGE: Mass Gathering Event, MACSIM®: Mass Casualty Simulation Tool High-fidelity simulator patients posed challenges in real-time disaster response training, with significantly longer procedures. Small sample size, lack of real patient comparison Potential challenges using simulators for real-time training. Noted strengths and weaknesses, including a lack of experience in mass casualty triage and communication breakdowns. Identified educational opportunities. Conducting an active shooter exercise provided valuable insights for improvement without impacting patient care. Artificial nature of drills, potential staff response discrepancies Kern Medical Center Emergency Medicine Department Bakersfield, CA Facho 2021 United States Qualitative study Adult ED of the Persian Gulf General Hospital in Bushehr, Iran Farahi 2021 Iran Qualitative study Nan Hospital of China Medical University Foo 2021 China Observational study Kings County Hospital Center in Brooklyn, New York, and Tampa Gillett 2008 United Prospective cohort study States iterative development of cognitive aids Instrumental skills for physicians, other tasks for nurses and non-physicians Three patients in round 1; another three in round 2 NR NR ED scenarios, regular operations, crisis response, workload integration, scenarios shaped by historical data, patient interarrival times adjustment for diverse intensity levels 2021 NR NR Technical proficiency of Disaster Medical Assistance Teams (DMATs), barriers to cooperation between DMATs and Urban Search and Rescue (USAR) teams during a simulated earthquake exercise 30physicians, 65 nurses, and 74 logistical personnel 2018 November NR Different scenarios, tasks such as triage, obtaining vital signs, applying oxygen, initiating stabilising December 2006 and March 2007 NR 78 hospital staff members General Hospital in Tampa, Florida Gryth 2010 Sweden Qualitative study NR 2008 October NR NR NR Major Incident Hospital (MIH) in Utrecht, Netherlands Haverkort 2017 Netherlands Proof-of-concept study Juelsgaard 2022 Denmark Qualitative focus group study Danish University Hospital American Association of Colleges of Nursing and associated Hospitals Kaplan 2012 United States Qualitative study Klima 2012 United States Qualitative study Multi-agency over 16 areas NR Kohlhoff 2012 United States Qualitative focus group study Brooklyn, New York hospitals NR interventions, completing surveys, providing appropriate disposition Review of captured footage to gain insights into patient experience 2014 to 2015 Two mock cases NR ABCDE approach, PPE and medical equipment, transportation, communication, decision making in critical cases 2020 April-may NR 22 healthcare professionals Communication, supplies usage, paediatric safety, triage, treatment 2008 to 2011 NR 90 undergraduate nursing students Communication, command structure, decontamination, staffing, patient tracking 280 volunteer patients NR 354 volunteer victims NR 7 trauma centres, 30 acute care hospitals and free-standing emergency departments, and 42 emergency medical services agencies McElroy 2019 United States Qualitative study Three rural Nebraska (USA) regions by the Center for Preparedness Education (CPE) at the University of Nebraska Medical Center (Omaha, Nebraska USA) Obaid 2017 United States Quasi-experimental design Two multidisciplinary FSE (major aircraft accident) in two regions in Sweden Pilot study (quantitative evaluation study) Rådestad 2012 Sweden Patients’ entry, triage accuracy, communication, coordination, performance, points of improvement, resources request, patient tracking, OHTrac utilisation, communication assessment 2017 April 445 volunteer patients NR Command staff, identification of areas for improvement, advancing regional collaboration among diverse response partners and other parameters of Homeland Security Exercise Evaluation Program 2010 to 2013 NR 83 command centres, including 667 participants prehospital, regional, and hospital command and control, as well as regional and hospital staff procedure skills. Key points include response time, communication effectiveness, resource allocation, media interaction, and staff 2008 to 2010 99 victims 131 and 69 health care workers, respectively Two emergency hospitals within region of Stockholm Rådestad 2023 Sweden Qualitative study Group of Swiss French-speaking hospital pharmacists (Groupe ment des Pharmacies Hospitallers Romans) Schumacher 2022 Switzerland Prospective multicentre design Kings County Hospital Center’s (KCHC) Paediatric Intensive Care Unit (PICU) Prospective single-centre simulated disaster drill Shah 2013 Switzerland Shrestha 2022 Nepal Observational study Six hospitals in Nepal 2021 NR NR coordination during disaster exercises. Transfer, patients evacuated, time to evacuate, decision-making, leadership, resource management, simulation system effectiveness, proposed evacuation plans, communication procedures, participants' perceptions 145 participants in Exercise 1; 95 participants in Exercise 2 2019 NR Simulation responses, communication, disaster management, logistical activities, pharmaceutical assistance, quality of actions using Likert scale 2018 August to 2020 August NR NR 22 patients with 14 family members needed evaluation 36 (medical students or emergency medicine residents), 9 of these patients Transportation, delivery of appropriate therapy, ED care, discharge process, educational materials 2012 July Hospital incident command area, patient care, triage, treatment areas, emergency signal, internal and external Emergency Department at Kings County Hospital Center, Brooklyn Wallace 2010 United States RCT ED at WellSpan York Hospital, a Level 1 trauma centre in south-central Pennsylvania Wexler 2017 United States Qualitative study coordination mechanism, organisation issues Identification of respiratory and circulatory compromise, provision of definitive airway, initiation of vasopressors through a central line, disposition 2008 July NR 12 cases Communication, recognition instructions response, evacuation preparedness NR NR NR Full-scale exercise 1 day Participants (university professionals), controller, observer Kohlhoff 2012 Influenza pandemic Emergency preparedness disaster simulation (EPDS) Mass causality (tornado disaster) Kaplan 2012 30-45 minutes Full-scale regional exercise (FSRE) Klima 2012 Mass causality NR Operation Continued Care full-scale exercise McElroy 2019 Mass causality NR Six functional exercises 3 hours Command centres, participants, controllers, observers Obaid 2017 General disaster preparation Rådestad 2012 Major aircraft accident Full-scale exercise 292 minutes Participants, controller, observers Simulating pandemic influenza outbreak with adapted case definition Evaluation using standardised forms Pre-and post-simulation surveys with self-reported, anonymous responses Team leader, triage staff nurse, bedside assistants, controller Tornado hitting an assisted living facility with injuries and fire Simulated terrorist assault with train derailment, involving 281 volunteer patients Participants, controllers, exercise design teams, evaluator Third-party evaluators, after-action evaluation Post-exercise analysis in after-action report/improvement plan Participants, communication team, resource request team, triage team, OHTrac input team Staged terrorist assaults at three locations, involving 445 mock victims Homeland Security Exercise Evaluation Program (HSEEP; Washington DC, USA) Six functional exercises to improve regional planning and collaboration External evaluators were registered using standardised performance indicators and patient outcome indicators. Aircraft crash scenarios with 99 and 100 victims in 2008 and 2010 Mass Casualty Simulation system (MACSIM) 11 hours Command group, physicians, nurses, support staff Rådestad 2023 Mass casualty incidents (MCI) Trauma mass-casualty Incidents Full-scale simulation exercise Shrestha 2022 30 minutes Participants, controller, evaluator Mass causality (road traffic accident and terrorist attack) Full-scale simulation exercise Schumacher 2022 3-4 hours Participants, assessors 90 minutes (patient briefing from 24 hours prior) Mass causality (influx of critically ill children) Full-scale chemical exposure exercise Shah 2013 High-fidelity simulator patients and human actor patients in a surge setting Wallace 2010 Influenza pandemic NR Full-scale active shooter exercise Wexler 2017 Mass casualty (active shooting) NR Participants, controllers NR: Not reported. Post-exercise surveys and open-ended questions for feedback Two exercises testing evacuation plans and application Assessment of disaster management capabilities in six hospitals Standardised evaluation tool Harvard School of Public Health's Emergency Exercise Evaluation Toolkit Simulated road traffic accident and terrorist attack scenarios Homeland Security Exercise and Evaluation Program (HSEEP) terms for a full-scale exercise Drill involving victims of chemical exposure, evaluating surge response and medical management Patients, participants, evaluators, controller Comparison of resuscitation times for simulator and human actor patients in ED Participants (physician, residents, nurse, technician), 12 patients Standardised data collection tool, checklist Closed-circuit video and portable video cameras were utilised to provide feedback for debriefing

Full-scale exercise in ED with an active shooter scenario use of training in real-world circumstances. Furthermore, their net benefits extended beyond individual skills development. Additionally, several studies, such as those by Kohlhoff et al. (34), and Obaid et al. (38), emphasised a significant impact on healthcare systems, with a crucial role in evaluating decision-making, encouraging regional collaboration, and providing insights. Simulation not only validates readiness, but also acts as a diagnostic tool for identifying the ideal disaster response. According to Shrestha 2022 (10) and Cristal 2019 (24), high-quality simulations provide opportunities for improvement by identifying issues such as communication breakdowns, equipment deficiencies, and weaknesses in specific emergency plans. As evidenced by Foo (29), the integration of disaster medical assistance teams (DMATs) ensures effective command, control, and communication, while allowing for improvement in health assessments. The wide variety of simulated scenarios, ranging from trauma centre stress management to mass casualty catastrophes involving earthquakes, pandemics, and active shooter scenarios, exemplifies these exercises’ adaptability in dealing with diverse crisis situations and ensures a comprehensive approach to disaster preparedness and prediction, which improves response in real-life situations. Furthermore, the range of exercise durations, from 30-minute simulations to multi-day events, provides flexibility in fulfilling diverse training requirements. The variety of evaluation measures used in different studies of high-stress simulations, including checklists, quantitative and qualitative surveys, and specialised evaluation tools, reflects the multifaceted nature of disaster preparedness. In addition to the conventional focus on technical abilities, nontechnical factors such as communication skills, teamwork, decision-making, healthcare infrastructure, and operational readiness are also examined. Radestad (40) demonstrates the utility of a simulation model in refining hospital evacuation preparations for mass causality crises. Their system makes it easier to build effective and practical evacuation plans, protocols, and procedures for actual crisis scenarios. This is the first systematic review of its kind, and highlights the effectiveness of full-scale simulation (FSS) in enhancing participants' skills and uncovering hidden safety issues in hospital settings during disaster preparedness exercises. It systematically identifies crucial areas for improvement that are addressed by simulation Effectiveness of simulations Identification of gaps, challenges, and limitations

However, it is essential to approach the findings with a nuanced understanding, considering the limitations across various studies. When evaluating the results, potential biases, short sample sizes, and actor-related issues should all be considered. Recognising the diversity in simulation approaches—from high fidelity simulators to functional exercises and in-situ simulations—enhances our understanding of the broad landscape of disaster preparedness research.

Recommendations and implications

These studies go beyond theoretical concerns to provide practical ideas for improving hospital readiness. Suggestions include mandating hospital-wide involvement, appointing dedicated Emergency Management Officers, and utilising simulation-optimisation methodologies. As we work our way through these collective findings, it becomes clear that disaster preparation simulations not only increase education and readiness, but also serve as a platform for continual improvement within the healthcare system.

Discussion

This systematic review confirms the benefits of full-scale simulation exercises in disaster preparation and improving healthcare system readiness. A wide range of methodological approaches were reviewed, involving different crisis scenarios and training projects. Many areas for improvement were identified, and our evidence supports the notion of enhancement and sustainability of disaster preparedness through simulation and evidence-based practice. Protocol registration: The protocol of this study was registered on the PROSPERO website with reference number CRD42023481418. Funding: No funding was received for this research. Conflict of interest: The authors declare no conflict of interest. Ethical approval: This article does not include any studies by any of the authors involving human participants or animals.

study. Prehosp Disaster Med. 2014 Aug 4;29(5):441–7. 6. World Health Organization Regional Office for the Western Pacific. Hospital and Health Facility Emergency Exercises: Guidance Materials. 2010.