“Never doubt that a small group of thoughtful, committed citizens can change the world. Indeed, it is the only thing that ever has.”
Cureus is on a mission to change the long-standing paradigm of medical publishing, where submitting research can be costly, complex and time-consuming.
The SIQ for this article will be revealed
once 2 ratings are submitted.
digital scholarship, free open access medical education, education needs assessment, emergency medicine, simulation in medical education
Anson Dinh, Teresa M. Chan, Kyla Caners, Andrew K. Hall, Andrew Petrosoniak, Tim Chaplin, Christopher Heyd, Jared B. Baylis
Published: July 12, 2022 (see history)
DOI: 10.7759/cureus.26799
Cite this article as: Dinh A, Chan T M, Caners K, et al. (July 12, 2022) The Future of Emergency Medicine (EM) Sim Cases: A Modified Massive Online Needs Assessment. Cureus 14(7): e26799. doi:10.7759/cureus.26799
Emergency Medicine (EM) Sim Cases was initially developed in 2015 as a free open-access simulation resource. To ensure the future of EM Sim Cases remains relevant and up to date, we performed a needs assessment to better define our audience and facilitate long-term goals.
We delivered a survey using a modified massive-online-needs-assessment methodology through an iterative process with simulation experts from the EM Simulation Educators Research Collaborative. We distributed the survey via email and Twitter and analyzed the data using descriptive statistics and thematic analysis.
We obtained 106 responses. EM Sim Cases is commonly used by physicians primarily as an educational resource for postgraduate level trainees. Perceived needs included resuscitation, pediatrics, trauma, and toxicology content. Prompted needs included non-simulation-case educational resources, increased case database, and improved website organization.
Data collected from our needs assessment has defined our audience allowing us to design our long-term goals and strategies.
Emergency Medicine (EM) Sim Cases (https://emsimcases.com) is an online bank of simulation cases and educational content on a free open-access medical education (FOAM) platform. It was established to provide easy access to high-quality peer-reviewed simulation cases and to reduce the duplication of effort among simulation educators. EM Sim Cases currently serves over 10,000 visitors per month from over 200 countries.
To respond to the expanding and evolving profile of users and contributors and to inform new educational content moving forward, an intervention was needed to ensure that the needs of the EM Sim Cases audience were being met. As in most curricular or education innovation reforms, a formal needs assessment is critical to ensure the program in question continues to have its desired impact [1]. Using the Massive Online Needs Assessment (MONA) methodology [2], we set out to define both the reader’s perceived needs and the needs that readers were prompted to identify via an online survey. The aim of this process was to define and establish long-term goals for the site guided by participant needs. There are many FOAM resources in existence, however, there is a paucity of literature describing formal processes for determining content and ensuring the content is meeting the needs of participants.
This project was selected as a top-seven education innovation moderated poster presentation for the Canada Association of Emergency Physicians Annual Conference 2020 but could not be presented due to the COVID-19 pandemic.
We made our survey available from October 15, 2019, to January 26, 2020. This project underwent review by the HiREB of McMaster University who waived full review, as it is considered program evaluation.
Participants were recruited via three avenues: 1) the EM Sim Cases email list, 2) a blog post on EM Sim Cases, and 3) via Twitter, with an infographic and link to the survey. We generated an email list of 407 participants from both subscribers to website updates and a pre-existing list of Canadian EM simulation experts [3]. One reminder was sent to this list. Twitter posts occurred three times through the EM Sim Cases Twitter account and twice through the email list of simulation experts.
We developed the survey tool (Appendix 1) in keeping with established best practices in survey design [4]. This included an iterative process with simulation experts from the Canadian EM Simulation Educators Research Collaborative [5]. The survey included questions about user demographics (location, specialty, education, years of training, and level of website use) and the educational impact of EM Sim Cases as perceived by the respondent (how EM Sim Cases is used, for whom, and the level of involvement in local curricula). We piloted the survey with a registered nurse, two emergency physicians, and a medical student, all of whom were not otherwise involved in the study. This served to ensure understanding and evidence of content and face validity with modifications made accordingly.
Our survey was designed as a modification of the previously described the MONA methodology [2]. The original MONA method is well-suited to eliciting the needs of a diverse audience that exists in online communities. Specifically, our modified MONA contained:
Phase 1: Elicitation of perceived needs. During this phase, the reader’s perceived needs were assessed with a given list of topics and asked if they would like to see related content. In this project, we asked about simulation-related educational topics.
Phase 2: Defining prompted needs. During this section, our modified MONA used free text to elicit needs that the reader prompted themselves.
Originally, the MONA method contained a third phase wherein educators attempted to measure unperceived needs characterized by the following statement, “I do not know what I don’t know”, using methods like multiple-choice questions. This third phase is used to identify knowledge gaps [2] in curriculum development regarding a specific topic. While useful in teacher-centered methods of instruction, we deemed it would be inappropriate for our more general project, and therefore we did not include this phase.
The analysis excluded “item nonresponses” to capture constructive comments from respondents. Quantitative data from demographics, educational impact, and phase one of the modified MONA were analyzed in Microsoft Excel (Microsoft Corporation, Redmond, WA) using descriptive statistics.
Thematic analysis of the qualitative data from educational impact and phase two of the modified MONA was performed by two investigators (AD, JB), one of whom has extensive experience in simulation and another who was blinded to survey development. Authors independently analyzed and coded survey responses using an open coding technique [6]. They subsequently compared their results and agreed on a set of thematic codes that served as ‘suggestions’ to be improved upon. Data were recoded a second time and then sent to the author group for approval before manuscript production. We set our cut-offs at 20-40% for moderate priority and >40% for high priority to stratify our results to suit our curriculum development needs.
We received 106 completed surveys from 18 countries, with the majority from Canada (n=41), the USA (n=27), and Australia (n=9). Email recruitment yielded 53 responses for a response rate of 13% while our blog post yielded 30 responses for a response rate of 26%. The remainder of the responses were obtained through Twitter, where a response rate calculation was not possible. Respondents consisted of physicians (n=59), nurses (n=12), and postgraduate trainees (n=10), with the majority specialized in emergency medicine (n=60). Other respondents (n=10) included paramedics, respiratory therapists, and medical students. Respondents most frequently reported six to 10 years of experience in their clinical career.
Notably, respondents reported that EM Sim Cases changed their clinical practice in that they added to their knowledge or provided a refresher (n=21) and overall positively impacted clinical practice (n=40). Many respondents noted specific impacts in free-form text responses (Table 1).
The majority (n=75) used EM Sim Cases as a resource for simulation facilitation and/or exam preparation. Many respondents (n=57) used completed published cases and a minority (n=23) used our template for their own development of cases. Respondents (n=92) used the site as part of their simulation curriculum targeting postgraduate trainees (n=67), nurses (n=53), and medical students (n=46). Four respondents noted that 76-100% of their simulation curriculum was made up entirely of EM Sim Cases material. Cases were also used for the continued professional development of staff physicians (n=35).
In phase one of our MONA, resuscitation, pediatrics, trauma, and toxicology cases were reported as high-priority perceived needs (Figure 1). Phase two of our MONA found faculty development, facilitating simulations, and assessment tools to be high-priority prompted needs for additional resources. Other high priorities were more cases and better website organization (e.g. improved searchable keywords). A notable request for more rural and disaster medicine cases was identified in our survey, however, this did not meet outcome percentage endorsement cut-offs.
MONA: Massive Online Needs Assessment; EM: emergency medicine
Our data demonstrated that EM Sim Cases is frequently used to support educational curricula at the program and institutional levels. EM Sim Cases may serve to overcome some of the commonly reported barriers to simulation-based education, including lack of time and medical simulation expertise [7-8]. Specifically, a standardized template reduces the time required to start from scratch, an intimidating undertaking for those unfamiliar with simulation case design. Pre-written simulation cases reduce the work required by educators thereby simplifying the design and delivery of simulation scenarios. In addition, it may enhance trust in the simulation educator that the simulation session will go as intended since it has been previously trialed elsewhere. The utility of the site will continue to benefit from increased usage, much like Wikipedia has harnessed the wisdom of crowds.
Our MONA identified priorities that will improve EM Sim Cases as an educational resource. First, respondents identified perceived needs of priority case topics: resuscitation, pediatrics, trauma, and toxicology. These are in line with a recent survey of Canadian EM training programs [7] and previous reports of perceived deficiencies in residency education [9-10]. Second, respondents indicated that additional website content and improved organization would be beneficial. Proposed organizational improvements included an improved search function for cases (e.g., improved keywords) and a more appealing layout. To achieve these goals with a limited workforce, EM Sim Cases may need to adopt a post-publication peer review system incorporating both comments and expert feedback [11].
We identified a signal toward the need for more disaster medicine content that did not meet prespecified cutoffs. This result may have been augmented had the survey been completed during the coronavirus disease 2019 (COVID-19) global pandemic. In-situ simulation played a key role in many emergency departments to improve the safety of workers during the COVID-19 pandemic [12]. On Feb 18, 2020, EM Sim Cases published a high-acuity COVID-19 simulation case, which received significant interest with 18,265-page views and 7,543 case downloads over the span of four months. This heavily utilized case was downloaded and used across the globe for healthcare provider training and system improvement and serves as an example of how identified needs can be met in short order by an online community of practice such as EM Sim Cases.
Our modified MONA has several limitations. Our response rate was low, attributed to our poor email response rate, which may lead to a selection bias in those who respond versus the typical user. Additionally, our respondents were primarily physicians, and future needs assessment surveys will need to reflect interdisciplinary education in a more robust manner.
As EM Sim Cases continues to expand, the needs and expectations of its users continue to evolve. Re-evaluation was necessary to quantify and qualify these needs. Data gathered from our survey will assist to support the redesign of this educational resource and align our priorities with those of our users.
1.     How often do you use EM Sim Cases? 
a.     Weekly 
b.     Monthly 
c.     Several times per year 
d.     Once per year 
e.     Never 
2.     Do you use EM Sim Cases solely for the simulation cases, solely for the other blog content, or for both? 
a.     Solely for other blog content
b.     Solely for the simulation cases 
c.     Both 
d.     Neither
3.     What do you use EM Sim Cases material for (e.g. education, in situ simulation, quality improvement, studying, assessment, etc)? 
4.     How much (if any) of your simulation curriculum is made up of EM Sim Cases material?
a.     None
b.     <25% 
c.     25-50%
d.     51-75% 
e.     >75% 
f.      Unsure
5.     How has EM Sim Cases impacted your educational practice (if at all)?
6.     How has EM Sim Cases impacted your clinical practice (if at all)? 
7.     We currently have the following categories of simulation cases. Of this list, please select the three that interest you the most? 
a.     Resuscitation 
b.     Pediatrics 
c.     Trauma 
d.     Toxicology 
e.     Respiratory
f.      OBGYN
g.     GI
h.     Neurology
i.       Endocrine
j.       Communication
k.     Cardiology
8.     What other resources do you use when sourcing simulation cases aside from EM Sim Cases? 
9.     When referring to sources other than EM Sim Cases, what do you like and/or dislike about those resources?
10.  Are there any simulation cases that you would like to see developed that have not yet been covered? 
11.  We are trying to expand our simulation-related blog content. In addition to content about simulation cases, what else would like to see?
12.  What could EM Sim Cases do to be more useful to you?
13.  In addition to your suggestions thus far, do you have any general suggestions for future directions of EM Sim Cases?
14.  What do you dislike about the website?
15.  What are your suggestions for how to improve the website?
16.  Have you ever submitted a case or a post for EM Sim Cases? Why or why not?
17.  What field do you come from?
a.     Staff physician
b.     Resident
c.     Medical student
d.     Respiratory therapist
e.     Physician assistant
f.      Simulation technologist
g.     Registered nurse
h.     Licensed practical nurse
i.       Nurse practitioner
j.       Primary care paramedic
k.     Advanced care paramedic
l.       Critical care paramedic
m.    Other
18.  How many years into your career are you?
a.     In training
b.     0-5 years
c.     6-10 years
d.     11-15 years 
e.     16-20 years
f.      21+ years
19.  If you are a physician, what specialty are you from (e.g. emergency medicine, family medicine, general surgery, etc)? 
20.  What is your training background in simulation-based education?
a.     Completed a simulation fellowship
b.     Masters degree with a focus on simulation-based education
c.     Completed a simulation debriefing course (e.g. the Harvard debriefing course)
d.     Informal training
21.  Are you a simulation leader/educator who is involved in simulation curriculum design and/or delivery? 
a.     Yes 
b.     No 
22.  If you answered yes to the previous question, which group of learners does this pertain to (check all that apply)?
a.     Residents
b.     Medical students
c.     Nursing
d.     Paramedicine
e.     Respiratory therapy
f.      Staff physician continuing professional development
g.     Physician assistants
23.  Where did you discover this survey? 
a.     Email
b.     EM Sim Cases blog post
c.     Twitter
24.  What city and country do you live in primarily?
College of Medicine, University of Saskatchewan, Saskatoon, CAN
Emergency Medicine, McMaster University, Hamilton, CAN
Emergency Medicine, Queen’s University, Kingston, CAN
Emergency Medicine, University of Ottawa, Ottawa, CAN
Emergency Medicine, University of Toronto, Toronto, CAN
Emergency Medicine, Queen’s University, Kingston, CAN
Emergency Medicine, McMaster University, Hamilton, CAN
Emergency Medicine, The University of British Columbia, Kelowna, CAN
Human subjects: All authors have confirmed that this study did not involve human participants or tissue. Animal subjects: All authors have confirmed that this study did not involve animal subjects or tissue. Conflicts of interest: In compliance with the ICMJE uniform disclosure form, all authors declare the following: Payment/services info: All authors have declared that no financial support was received from any organization for the submitted work. Financial relationships: All authors have declared that they have no financial relationships at present or within the previous three years with any organizations that might have an interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.
Dinh A, Chan T M, Caners K, et al. (July 12, 2022) The Future of Emergency Medicine (EM) Sim Cases: A Modified Massive Online Needs Assessment. Cureus 14(7): e26799. doi:10.7759/cureus.26799
Peer review began: June 20, 2022
Peer review concluded: July 08, 2022
Published: July 12, 2022
© Copyright 2022
Dinh et al. This is an open access article distributed under the terms of the Creative Commons Attribution License CC-BY 4.0., which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
MONA: Massive Online Needs Assessment; EM: emergency medicine
Scholarly Impact Quotient&#8482 (SIQ&#8482) is our unique post-publication peer review rating process. Learn more here.
This link will take you to a third party website that is not affiliated with Cureus, Inc. Please note that Cureus is not responsible for any content or activities contained within our partner or affiliate websites.
Scholarly Impact Quotient™ (SIQ™) is our unique post-publication peer review rating process. SIQ™ assesses article importance and quality by embracing the collective intelligence of the Cureus community-at-large. All registered users are invited to contribute to the SIQ™ of any published article. (Authors cannot rate their own articles.)
High ratings should be reserved for work that is truly groundbreaking in its respective field. Anything above 5 should be considered above average. While all registered Cureus users can rate any published article, the opinion of domain experts is weighted appreciably more than that of non-specialists. An article’s SIQ™ will appear alongside the article after being rated twice and is recalculated with each additional rating.
Visit our SIQ™ page to find out more.
Scholarly Impact Quotient™ (SIQ™) is our unique post-publication peer review rating process. SIQ™ assesses article importance and quality by embracing the collective intelligence of the Cureus community-at-large. All registered users are invited to contribute to the SIQ™ of any published article. (Authors cannot rate their own articles.)
Already have an account? Sign in.
Please note that by doing so you agree to be added to our monthly email newsletter distribution list.
By joining Cureus, you agree to our Privacy Policy and Terms of Use.