An Evaluation of the 2020 Change to the Saudi Emergency Residency Program Assessment

Introduction

Medical and surgical trainees across the globe must undergo vigorous training and assessment during their residency years before qualifying to work independently. Traditionally, clinical practice and time spent in the clinical setting are considered the predominant structure for postgraduate training, with additional final examinations including written exams, oral exams, and objective structured clinical exams (OSCEs). Assessment tools include written or standardised tests, simulation-based assessments, and direct observation [1]. The movement of competency-based medical education (CBME) has given rise to multiple bodies and initiatives within medical education, such as the Saudi Commission for Health Specialties (SCFHS) in Saudi Arabia [2]; the General Medical Council (GMC) in the United Kingdom; the Accreditation Council for Graduate Medical Education (ACGME) in the United States; and the Educating Future Physicians of Ontario (EFPO) and CanMEDS competency initiatives in Canada [1]. Emergency medicine (EM) residency programs accredited by the Royal College of Physicians and Surgeons of Canada (RCPSC) formally implemented their CBME evaluation procedure for residents commencing postgraduate training in July 2018. This assessment program consists of 28 entrustable professional activities (EPAs), assessed on a five-point entrustment scale and divided into four stages (Transition to Discipline, Foundations of Discipline, Core of Discipline, and Transition to Practice) spread over five years of training, all of which were predefined centrally by the RCPSC EM Speciality Committee [3]. The Outcome Project, initiated in the United States by the Accreditation Council for Graduate Medical Education (ACGME), delineated six domains to determine medical competence: patient care, medical knowledge, professionalism, practice-based learning and improvement, interpersonal communication skills, and systems-based practice. According to one survey, direct observation and global assessment are used, and a few programs include standardised patient examinations and drug prescription practice reviews in their evaluations [4].

In the United Kingdom, standards for postgraduate medical education and training are established by The General Medical Council (GMC). All specialist training programs describe knowledge, skills, and behaviours according to the General Medical Council’s "Good Medical Practice": knowledge, skills, and performance; safety and quality; communication, partnerships, and teamwork; and sustaining trust. Supervised learning events (SLEs), summative evaluations of performance and tests, and triangu-lated judgment by the educational supervisor are used to measure competence progression. After a successful speciality training program, a Certificate of Completion of Training (CCT) is awarded and permits entry into the specialist list [5].

In the Netherlands, competence is used to evaluate postgraduate medical education. All new residents work with the director of the training program to create an "individual training plan" based on abilities gained before speciality training, such as during previous employment as a resident-not-in-training. During speciality training, residents create a portfolio documenting their progress across all skill areas, and this portfolio is the foundation for each resident’s progress reviews [5].

Such international training programs lack agreement on a standardised method of assessment. The variation in assessment, although agreed on the need for impartial assessment using different means and methods, still renders the number of examinations controversial. In Saudi Arabia, the assessment model includes a yearly written examination, in addition to OSCE and evaluation per rotation.

In 2020, a change was made to the number of written examinations in the emergency board training – from one per year to three, and a logbook became a requirement for mini-clinical evaluation exercises (mini-CEX) and direct observation of procedural skills (DOPS). The latter two requirements were based on evidence of their valuable utility in medical education [6-8], and an argument could be made for their informal application before the effective change date. However, while the change in the number of assessments was intended to engage residents more with the educational material, its efficiency from the residents’ perspective still needs to be evaluated. This study, therefore, aimed to assess the perspective of EM residents on the recent change in the number of examinations, explore its impact on their knowledge gain, time management, and reported stress and anxiety, and compare those factors between different training provinces.

Methods

This is a cross-sectional study carried out from September 2022 to October 2022. The survey was created by the authors and distributed using an online platform. A convenient sample was sought targeting emergency medicine resident trainees in Saudi Arabia. Sampling was done through contacting residents’ representatives from various geographical regions and asking them to distribute the survey within their centres. Trainees’ level donated by R1-R4 represents the level of training from junior to senior level. We used Fisher’s Exact Test to explore any significant association that might influence our research inquiry. We considered a level of significance to be less than 0.05. We used SPSS version 25.0 for statistical analysis. The study was approved by the IRB committee, with log number 22-407.

Results

Demographics: One hundred and nine emergency residents responded to the survey. The participants’ demographics are illustrated in Table 1. Attitude toward the change from one exam to three exams: More than half of the participants, 56% (n=61), supported having one exam instead of three, whereas 14.7% (n=16) were neutral in supporting or opposing three exams.

There was a significant difference between provinces regarding the attitude toward having three exams; Fisher exact=24.967, p <0.005. Trainees from the Central Province were more supportive of having three exams, at 78.1%, than trainees from exams. Another explanation might be that the idea of these exams being opportunities to learn, rather than tests per se, has not been adequately conveyed. Nonetheless, a gradual implementation of any changes, and involving the residents in those changes, fosters commitment and greater acceptance [10]. The true impact of the changes to the assessment can be measured objectively using the median knowledge curve before and after the change. This can be obtained by comparing test scores from before and after the change.

Time management appears to have a central role in the residents’ attitude toward the change. A focus on time management should be part of any quality improvement project, particularly when it influences a busy training curriculum like that of emergency medicine. The residents might, for example, be asked to agree upon scheduling the exams at the beginning of the academic year, which might accommodate the time constraints and the effect on annual leave and social life. Furthermore, productivity and well-being are enhanced after time management workshops are conducted for residents [11].

Although less than one third supported the new changes, support was higher in the Central region than in the other provinces. The Central Province also refutes any negative influence on time management or the gain of clinical skills. This raises questions regarding the implementation of the three exams or the similarity in teaching style between different provinces. More research into this notion is justified. The convenient sample used for this analysis represents a limitation thereof. We believe more inquiry is needed into how the changes to the assessment should occur, including seeking the perspectives of both educators and trainees. Overall, support was low for having three exams throughout the year. Further assessment of the benefit of three exams should be undertaken, and it should be noted that the involvement of the residents in the education and assessment process is paramount to fostering commitment.