Specific Learning Objectives in the Form of Self- assessment to Facilitate Rehabilitation Residency Training Competency: A Prospective Cohort Study

Rinlada Pongratanakul, M.D., Waraporn Kanthanet, Kanda Sungkharuk, Sumana Srisoongnern, M.D.

Department of Rehabilitation Medicine, Faculty of Medicine Siriraj Hospital, Mahidol University, Bangkok 10700, Thailand.

ABSTRACT

Objective: To explore the benefits of rotation- and milestone-specific learning objectives in the form of self-assessment questionnaires to facilitate rehabilitation residency training.

Materials and Methods: Control and intervention groups were formed during academic years 2020 and 2021, respectively. The group participants were residents undergoing training in rehabilitation medicine. Before entering their rotations, the intervention group participants were provided rotation- and milestone-specific learning objectives in the form of self-assessment questionnaires. The participants of both groups self-assessed their confidence levels for each learning objective after the rotations, using a 5-level Likert scale. The self-rated scores of the 2 groups were compared.

Results: Twenty-five and 28 residents were enrolled in control and intervention groups, respectively. There were 12 sets of learning objectives (4 rotations; 3 milestones). Only 2 sets revealed higher self-rated scores for the intervention group (the pediatric and the pain rotations of the second-year residents). The groups’ scores for Entrustable Professional Activities (EPAs) and tests of participants’ knowledge of the learning objectives showed no differences. Conclusion: Providing specific learning objectives in the form of self-assessment alone was not found to significantly improve residents’ self-perceived learning achievement, particularly in the knowledge-based activities. The advantages of specific learning objectives might be enhanced when combined with other measures that support residency training.

Keywords: Learning objectives; residency training; self-assessment (Siriraj Med J 2023; 75: 275-281)

INTRODUCTION

Rehabilitation medicine is an extensive and diverse field. During training, residents are inundated with information and practices related to developing the skillsets needed to perform the job. We are interested in implementing an intervention that will help residents achieve their learning objectives and gain the most out of their 3 years of residency training.

Our hospital’s rehabilitation residency core curriculum complies with the guidelines of the Royal College of Physiatrists of Thailand and is outcome-based.1 New

versions of Entrustable Professional Activities (EPAs) and the Workplace-Based Assessment (WPBA) were launched in 20202 to support competency-based training. To assist both learners and instructors in achieving the required competencies, the rehabilitation residency core curriculum and the EPAs are written in a milestone fashion, as required by the Next Accreditation System3 of the Accreditation Council of Graduate Medical Education.4 Introducing the entire 3-year curriculum at the beginning of residency training might be too complex, thereby risking that residents would disregard the

Corresponding author: Rinlada Pongratanakul E-mail: rinlada.p@gmail.com

Received 2 February 2023 Revised 22 February 2023 Accepted 28 February 2023 ORCID ID:http://orcid.org/0000-0001-7808-7749 https://doi.org/10.33192/smj.v75i4.261160

All material is licensed under terms of the Creative Commons Attribution 4.0 International (CC-BY-NC-ND 4.0) license unless otherwise stated.

provided curriculum. Consequently, a strategy that could guide trainees through their residency training was launched. Motivation dictates human behavior and is a key to successful learning.5 Knowing a goal or learning objective is one way to promote motivation in medical education.5 As described by Mager in 1984, a learning objective is ‘...the description of a performance you want learners to be able to exhibit before you consider them competent.’ This study therefore investigated whether learning outcomes are improved by using a self-assessment process to communicate learning objectives corresponding to residents’ tasks and milestones (level of knowledge and skills).

MATERIALS AND METHODS

This questionnaire-based pilot study was conducted at the Department of Rehabilitation Medicine, Faculty of Medicine Siriraj Hospital, Mahidol University. The intervention was provided to residents trained in academic year 2021. Those residents’ learning outcomes were compared with those of residents trained in the preceding academic year (2020), when no intervention was given. The research coordinators invited the participation of all first- to third-year residents via an online platform. The inclusion criterion was residents undertaking training in rehabilitation medicine during academic years 2020 and 2021. Once residents voluntarily agreed to enroll in the study, their informed consent was obtained electronically. There were no exclusion criteria at enrollment. However, the analysis did not include residents in academic year 2021 who did not provide pre- and postintervention responses. Before this research began, the Siriraj Institutional Review Board approved its protocol (COA no. Si 755/2021). The study followed the Helsinki Declaration of 1964 and subsequent amendments.

Rehabilitation residency clinical rotations

In Thailand, residency training in rehabilitation medicine takes 3 years. At Siriraj Hospital, clinical rotations are divided into 5 major blocks, 4 of which are specific to particular illnesses (cardiopulmonary, pediatric, geriatric, and pain). The other rotation was excluded because it includes various fields of study including general rehabilitation, prosthesis-orthosis, urodynamic studies, etc. This precludes the formation of specific learning objectives. In each of the 3 academic years, the first- to third-year residents perform all 5 clinical rotations, with individual rotations lasting 4 weeks.

Interventions

Rotation-specific learning objectives were provided

to residents before entering each rotation. The learning objectives were provided in the form of self-assessment questionnaires and were listed as items. Residents were required to read and self-assess their confidence in each task. Participants in academic year 2020 were requested to complete the questionnaires after each rotation. However, participants in academic year 2021 were required to respond to the questionnaires before and after each rotation. As there were 3 years of training with 4 specific clinical rotations each year, there were 12 sets of learning objectives. In addition, tests were given to determine participants’ knowledge of the learning objectives.

Questionnaires and tests

Questionnaire 1: Self-assessment of the learning objectives (the intervention)

The learning objectives complied with the core curriculum of the Royal College of Physiatrists of Thailand of 20181 and the original and revised versions of EPAs and WPBA of 2020 and 2021.2,6 The objectives were formulated in accordance with the concepts of “SMART” (specific, measurable, attainable, relevant, and time-bound)7 and “ABCD” (audience, behavior, condition, and degree of independence).8 Faculty educators specializing in the medical fields reviewed and approved the learning objectives.

For the self-rated questionnaires, the participants used a 5-level Likert scale (1=not at all confident, 2=slightly confident, 3=moderately confident, 4=quite confident and, 5=extremely confident)9 to rate how confident and competent they considered they were in each objective. The rating system was presented as an online Google form. Between 4 and 13 items were assessed per objective.

Tests on knowledge of the learning objectives

Tests were given to assess whether the intervention helped participants become informed about the learning objectives. There were 3 sets of tests, one for each year of training. Residents of both academic years 2020 and 2021 took the tests at the end of the year after completing all clinical rotations. The tests were in addition to the residents’ routine academic examinations, and the test scores were used for this research only.

Questionnaire 2: Participant satisfaction with the intervention

Participants rated their satisfaction with the intervention on a 5-level Likert scale (1=not at all satisfied; 5=extremely satisfied).9 Only the participants of academic year 2021 were administered Questionnaire 2, which was given at the end of the study period.

EPA and WPBA evaluation results

In all, there are 13 EPAs, and each has 3 milestones. EPAs that were related to the rotations in this study were selected. The EPA score for “overall actual performance” was used; scores were graded into 4 levels: “good,” “pass,” “borderline,” and “fail.”

The baseline demographic data that was recorded consisted of age, sex, number of postgraduate internship years, grade point average during medical school, and scores for the first and second steps of the national medical licensing examination.

The study’s primary endpoint was the benefits of the intervention assessed by the achievement of the rotation-specific learning objectives. The Questionnaire-1 scores of the participants in academic years 2020 and 2021 were compared. In the case of the intervention group, the pre- and postrotation questionnaires had to be completed to be eligible for analysis.

Statistical analysis

Demographic data and results are described using descriptive statistics. Categorical data are presented as numbers with percentages. Normally distributed continuous data are presented as the means ± standard deviations. Comparisons between groups were made using the chi- squared test for categorical data and the independent t-test for normally distributed continuous data. The internal consistency reliability of the Questionnaire 1 (the intervention) was analysed with Cronbach’s alpha coefficients. The level of significance for all tests was

set at 0.05. The analyses were performed using PASW Statistics for Windows, version 18.0 (SPSS Inc, Chicago, IL, USA).

RESULTS

Twenty-eight and 30 residents were enrolled in the rehabilitation-medicine training program during academic years 2020 and 2021, respectively. Of those, 89% (25/28) and 93% (28/30) residents enrolled in the study. However, not all participants responded to all of the study questionnaires and tests. For the primary outcome (Questionnaire 1), the response rates were 80% (80/100) and 65% (73/112) for academic years 2020 and 2021, respectively.

The participants’ characteristics are summarized in Table 1. The residents in academic years 2020 and 2021 had mean ages of 29 and 28, respectively. Although there was only a 1-year difference in the mean ages, the difference was statistically significant (P = 0.01). Both groups were similar in terms of grade point averages, and scores for the first and second steps of the national medical licensing examination.

Table 2 details the participants’ self-assessment scores for their confidence and competency in each learning objective. The mean scores are categorized by rotation and year of residency. The intervention group achieved higher self-rated scores than the control group for only 2 of the 12 assessment sets: the pediatric and the pain rotations of the second-year residents (P = 0.02 and 0.02, respectively). The scores for the 4 clinical

TABLE 1. Demographic data of participants.

*= P value < 0.05

Academic year 2020

Academic year 2021

Mean difference

TABLE 2. Scores for self-assessment of the learning objectives test (Questionnaire 1), categorized by 4 clinical rotations and 3 levels of residency training.

# Scores are rated on a 5-level Likert scale (1=not at all confident; 5=extremely confident).

* P value < 0.05.

rotations were also compared holistically, i.e., without a breakdown by year of residency. This analysis revealed that the intervention group had a significantly higher self-rated score for the pediatric rotation than the control group (P = 0.002).

In addition to the residents self-evaluating their confidence levels, they were tested on their knowledge of the learning objectives. For all 3 years of residency, the participants in the control and intervention groups had similar scores.

EPA assessment scores were obtained as a means of objectively assessing the intervention. Unfortunately, not all learning objectives had a corresponding EPA assessment. Moreover, not all participants underwent an EPA assessment during their rotation period. For some assessments, scores were available for only 20% of the participants. Based on the available data, there were no significant differences in the performance scores of the control and intervention groups for any of the EPA

assessments (Table 3).

During the academic year 2021, self-rated scores were obtained from the participants before and after their rotations. For all 3 years of residency and all 4 clinical rotations, the participants gave significantly higher ratings after than before the rotations (all P < 0.05; data not shown).

The internal consistency reliability of the Questionnaire 1 was analysed with Cronbach’s alpha coefficients. In 12 sets of learning objectives, Cronbach’s alpha coefficients ranged from .891 to .965, all of which exceeded .70, indicating acceptable values.10

Questionnaire 2 investigated participant satisfaction with the intervention. On a Likert scale of 1 to 5, the mean scores for the 5 aspects of satisfaction that were examined ranged between 3.6 and 4.0. Over 80% of the residents gave scores ≥ 4.0 when asked if they wished to have the learning objectives routinely provided as part of their training program (Table 4).

TABLE 3. Scores from the EPA assessments, categorized by rotation and milestone level.

# Scores range from 1–4 (4=good, 3=pass, 2=borderline, and 1=fail).

* No responses = although there was an EPA assessment that matched the rotation and milestones, no assessments were performed during the study period.

** No assessments = no EPA assessment matched the rotations or milestones. NA = not applicable

TABLE 4. Satisfaction with having learning objectives (Questionnaire 2).

# Scores are rated on a 5-level Likert scale (1=not at all satisfied; 5=extremely satisfied).

DISCUSSION

Learning goals are the foundation of self-regulated learning and enhance learner motivation, reinforcing each aspect and contributing to successful learning.11-14 However, while using goals is vital, having medical residents set effective goals for themselves has been reported to be challenging.15 A study on the learning obstacles faced by family-medicine residents found that they yearned for external motivation and guidance.16 One form that can be implemented relatively easily is the communication of specific learning objectives, which is the technique that this study investigated. Goals that are well written and proximal (relating to the near future) are more effective than general or long-term goals.14 By giving residents rotation-specific objectives, we hypothesized that the scores of the residents in academic year 2021 would be higher. Although the self-rated confidence scores were slightly higher in academic year 2021 than in 2020 for almost all learning objectives, only the pediatric rotation showed statistical significance (P = 0.002).

Research has been undertaken to determine whether self-evaluation can accurately rate clinical skills and knowledge. Some studies claimed that self-assessment was more accurate when used to assess skills or clinical performance rather than knowledge or cognitive aspects. In contrast, other studies found no differences.17 In the present study, most learning objectives were related to knowledge-based activities. Only the pediatric rotation had more skills-based than knowledge-based learning outcomes. This may have limited the apparent benefits of our intervention.

Self-evaluation is a well-established method of self-reflection.18,19 However, its accuracy is questionable, with most studies finding that it is limited compared with external standards.17,20,21 Several factors influence

self-evaluation accuracies, for example, sex and learner insight.17 Meta-analysis of the association between self- assessment and external evaluation reported that external validation is still needed while self-assessment is used.21 Consequently, the current study applied EPAs and tests (the tests on the participants’ knowledge of the learning objectives) as an external standard. Unfortunately, only a minority of participants obtained EPA assessments, which limited our ability to detect score differences. Tests of the participants’ knowledge of the curriculum’s learning objectives were also administered and the results were negative. Given the negative findings of the self-rated scores and the 2 external standards, the benefits of the intervention seemed negligible.

Moreover, we obtained self-rated scores before and after the rotations of the intervention group. The scores were all significantly higher for the postrotation evaluations. However, this finding more likely reflects the benefits of the training rather than the intervention.

The study aimed to guide residents through residency training with effectively written learning objectives combined with a process of self-reflection. Nevertheless, because self-evaluation accuracy is restricted, other measures should be incorporated, for example, feedback, performance benchmarking,17,22 experience, and external measurement.21-23 In addition, as monthly self-evaluations might have burdened the residents with extra work, participant satisfaction was obtained; the results were quite positive. Consequently, strategies to provide rotation-specific learning objectives should be continued. In conclusion, providing specific learning objectives in the form of self-assessment alone was not found to significantly improve residents’ self-perceived learning achievement, particularly in the knowledge- based activities.

Limitations

As it was a pilot study, the number of participants enrolled was small, which might have limited the study’s power. Moreover, the number of respondents to the questionnaire was not constant. Both factors could obscure the full benefits of the intervention. Another limitation was the EPA assessments. Being workplace-based evaluations, they were difficult to obtain. Another concern was the history threat to internal validity as the participants of the control and the intervention groups were from different academic years. The training process between the 2 years might not be absolutely identical. In addition, it should be noted that this study has limited generalizability as it was performed in a rehabilitation residency training setting at only 1 institution. Therefore, the study’s results might not apply to other academic institutions or residency training programs. More positive outcomes might be identified via a larger sample size coupled with a study design that better engages participants throughout the study and draws upon EPA evaluations.

ACKNOWLEDGMENTS

The authors thank Miss Julaporn Pooliam of the Faculty of Medicine Siriraj Hospital, Mahidol University, for her assistance with the statistical analyses. The authors are also indebted to Mr David Park for the English- language editing of this paper.

Conflicts of interest: none Funding: none REFERENCES

1. The Board of Training and Examination in Rehabilitation Medicine The Royal College of Physiatrists of Thailand. Residency Training Curriculum in Rehabilitation Medicine 2018. Available from: http://rehabmed.or.th/main/wp-content/ uploads/2019/09/%E0%B8%9B%E0%B8%81_Rehab-Med- Curriculum-Revised-2561_March-20-2019_Final-OK.pdf.

2. The Board of Training and Examination in Rehabilitation Medicine The Royal College of Physiatrists of Thailand. Entrustable Professional Activities (EPAs) and Workplace- based Assessment (WPBA) in Rehabilitation Medicine Residency Training 2020 2020. Available from: http://rehabmed.or.th/ main/wp-content/uploads/2020/09/EPAs-and-WPBA_Proof- IV_July-15-2020_Final-for-print.pdf.

3. Nasca TJ, Philibert I, Brigham T, Flynn TC. The Next GME Accreditation System — Rationale and Benefits. N Engl J Med. 2012;366(11):1051-6.

4. Carraccio C, Iobst WF, Philibert I. Milestones: Not Millstones but Stepping Stones. J Grad Med Educ. 2014;6(3):589-90.

5. Cook DA, Artino AR, Jr. Motivation to learn: an overview of contemporary theories. Med Educ. 2016;50(10):997-1014.

6. The Board of Training and Examination in Rehabilitation

   Medicine The Royal College of Physiatrists of Thailand. Entrustable Professional Activities (EPAs) and Workplace-based Assessment (WPBA) in Rehabilitation Medicine Residency Training 2021 2021. Available from: https://www.rehabmed.or.th/main/wp- content/uploads/2021/07/Final_EPA_WPBA_2021-Revision-1. pdf.

7. Chatterjee D, Corral J. How to Write Well-Defined Learning Objectives. J Educ Perioper Med. 2017;19(4):E610.

8. Heinich R, Molenda M, Russell JD, Smaldino SE. Instructional Media and Technologies for Learning. 7th ed: Pearson; 2002.

9. Artino AR, Jr., La Rochelle JS, Dezee KJ, Gehlbach H. Developing questionnaires for educational research: AMEE Guide No. 87. Med Teach. 2014;36(6):463-74.

10. Tavakol M, Dennick R. Making sense of Cronbach’s alpha. Int J Med Educ. 2011;2:53-5.

11. Collins NS. Motivation and Self-Regulated Learning: Theory, Research, and Applications. The Journal of Higher Education. 2009;80:476-9.

12. Kusurkar RA, Ten Cate TJ, van Asperen M, Croiset G. Motivation as an independent and a dependent variable in medical education: A review of the literature. Medical Teacher. 2011;33(5):e242-e62.

13. Panadero E. A Review of Self-regulated Learning: Six Models and Four Directions for Research. Front Psychol. 2017;8:422.

14. Zimmerman BJ, Risemberg R. Chapter 4 - Self-Regulatory Dimensions of Academic Learning and Motivation. In: Phye GD, editor. Handbook of Academic Learning. San Diego: Academic Press; 1997.p.105-25.

15. Sawatsky AP, Halvorsen AJ, Daniels PR, Bonnes SL, Issa M, Ratelle JT, et al. Characteristics and quality of rotation-specific resident learning goals: a prospective study. Med Educ Online. 2020;25(1):1714198.

16. Nothnagle M, Anandarajah G, Goldman RE, Reis S. Struggling to Be Self-Directed: Residents’ Paradoxical Beliefs About Learning. Acad Med. 2011;86(12):1539-44.

17. Colthart I, Bagnall G, Evans A, Allbutt H, Haig A, Illing J, et al. The effectiveness of self-assessment on the identification of learner needs, learner activity, and impact on clinical practice: BEME Guide no. 10. Med Teach. 2008;30(2):124-45.

18. Brydges R, Butler D. A reflective analysis of medical education research on self-regulation in learning and practice. Med Educ. 2012;46(1):71-9.

19. van Houten-Schat MA, Berkhout JJ, van Dijk N, Endedijk MD, Jaarsma ADC, Diemers AD. Self-regulated learning in the clinical context: a systematic review. Med Educ. 2018;52(10):1008- 15.

20. Brewster LP, Risucci DA, Joehl RJ, Littooy FN, Temeck BK, Blair PG, et al. Comparison of resident self-assessments with trained faculty and standardized patient assessments of clinical and technical skills in a structured educational module. Am J Surg. 2008;195(1):1-4.

21. Davis DA, Mazmanian PE, Fordis M, Van Harrison R, Thorpe KE, Perrier L. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA. 2006;296(9):1094-102.

22. Zevin B. Self versus external assessment for technical tasks in surgery: a narrative review. J Grad Med Educ. 2012;4(4):417- 24.

23. Blanch-Hartigan D. Medical students’ self-assessment of performance: results from three meta-analyses. Patient Educ Couns. 2011;84(1):3-9.