Are Technical Skills Assessed Using Medical Knowledge Associated with Non-technical Skill Knowledge in Anaesthesia Resident Training?

Maliwan Oofuvong, M.D., Ph.D.*, Ngamjit Pattaravit, M.D., Orarat Kanjanawanichkul, M.D., Sirikarn Siripruekpong, M.D., Kanjana Nuanjun, R.N., Boonthida Suwannarat, R.N.

Department of Anesthesiology, Faculty of Medicine, Prince of Songkla University, Hat Yai, Songkhla 90110, Thailand.

ABSTRACT

Objective: We examined the association between midterm examinations and anaesthetists’ non-technical skills (ANTS) knowledge using multiple choice questions (MCQs) for anaesthesia training.

Materials and Methods: A prospective cross-sectional study was implemented based on two cohort studies conducted in November 2017 and November 2019 at the university hospital in southern Thailand. Each cohort consisted of MCQs, short answer questions (SAQs), objective structured clinical examinations (OSCEs), and MCQs after ANTS simulation workshops during the midterm examinations. The main exposure variable was the midterm examination (MCQs/ SAQs/ OSCEs) whereas the ANTS MCQs were the outcome. The potential predictors were the residents’ role, sex, PGY (1-3), and competency. Associations between midterm examinations and ANTS MCQ scores were analysed using Pearson’s correlation coefficients (r) and multivariate linear regression analysis, and presented as beta coefficient (β) and 95% confidence limit (CL).

Results: Forty-eight anaesthesia residents were recruited for the study. After adjusting for PGY, knowledge score, and attitude evaluated by the staff, OSCE was found to be significantly associated with the pretest ANTS scores (β [95% CL]= 1.02 (0.06, 1.98)) and MCQ scores were significantly associated with the posttest ANTS scores (β [95% CL]= 0.14 (0.04, 0.24)). SAQ scores had negative associations with the pretest (β [95% CL]=-0.11 (-0.21, -0.01)) and posttest (β [95% CL]=-0.16 (-0.27, -0.05)) ANTS scores. It was found that there was a significant relationship between midterm examinations and posttest ANTS scores (r=0.52).

Conclusion: Technical skills using OSCE and midterm MCQ examination scores were associated with non-technical skill knowledge in anaesthesia training.

Keywords: Technical skill; anaesthesia’s non-technical skill knowledge; multiple choice questions; short answer questions; objective structured clinical examination (Siriraj Med J 2022; 74: 844-856)

INTRODUCTION

Anaesthetists’ non-technical skills (ANTS) are a crucial component of anaesthesia resident training because they emphasise communication skills and the ability to work in a multidisciplinary team during anaesthesia practise.1 It is also part of the anaesthesia training

requirement in conjunction with the technical skills required to answer multiple choice questions (MCQs), short answer questions (SAQs), objective structured clinical examinations (OSCEs), and oral examinations. Because it is a behavioural marker tool, at least two evaluators are required to assess residents’ behaviour during the

Corresponding author: Maliwan Oofuvong E-mail: oomaliwa@gmail.com

Received 23 July 2022 Revised 5 November 2022 Accepted 6 November 2022 ORCID ID:http://orcid.org/0000-0001-5840-5965 http://dx.doi.org/10.33192/Smj.2022.99

All material is licensed under terms of the Creative Commons Attribution 4.0 International (CC-BY-NC-ND 4.0) license unless otherwise stated.

simulation workshop to accurately evaluate performances related to 15 elements of ANTS.2 Some predictors – such as the number of evaluators, residents’ skill level range, and the observational process – can affect the reliability of the evaluators.3 Other predictors associated with higher non-technical skills were trainees’ experience, recent training in the life support protocols,4 and higher technical skills.5,6 Most studies have demonstrated the correlation between technical and non-technical skills in simulation workshops.5,6 To date, no studies have reported other evaluated tools besides the behaviour maker tool to examine the association between non-technical and technical skills in anaesthesia. Not all institutes like ours might be able to evaluate ANTS behaviour in all residents in one time period workshop; therefore, through midterm examinations, we applied the ANTS assessment by using the debrief session and evaluating ANTS knowledge to ensure ANTS comprehension and understanding. Moreover, we examined whether the technical skills assessed using medical knowledge during midterm examinations were associated with ANTS knowledge, based on the MCQ scores of residents who either participated in or observed the ANTS simulation workshop.

MATERIALS AND METHODS

This prospective observational study was conducted after receiving approval from the Ethics Committee, Faculty of Medicine, Prince of Songkla University on 1 October 2019 (REC 62-265-8-1). We recruited anaesthesia residents from different post-graduate years (PGY-1, PGY-2, and PGY-3) who attended a simulation-based ANTS training workshop at the simulation centre, Faculty of Medicine, Prince of Songkla University, in November 2017 and November 2019. Since the workshop was a part of the anaesthesia training programme, written informed consent from the participants was waived by the Ethics Committee.

Standard operating procedure

Midterm examinations held annually in the month of November and consist of MCQs, SAQs, and OSCEs (Fig 1). Residents’ performance in terms of knowledge, skill, and attitude are simultaneously evaluated by the anaesthesia staff. The simulation ANTS workshop has been held biennially since 2015. All residents were informed of the schedule of the workshop one month in advance. The three case scenarios were based on the dominant ANTS category themes. Meanwhile, situation awareness (SA), teamwork and communication (TW), task management (TM), and decision making (DM), were developed by

three different anaesthesia staff members for each theme. For 2019, the scenario ‘Missing tooth’ for the TW theme, the scenario ‘Anaesthesia emergency service’ for the TM theme, and the scenario ‘Decision making in on call resident’ for the DM theme, were created by SS, MO, and NP, respectively. All the residents (PGY-1, PGY-2, PGY-3) received the ANTS material one week before the workshop. The manikins and workstations were prepared and checked to ensure readiness one day before the workshop. On the day of the workshop, the residents received a ten-minute briefing on overall concepts of the ANTS by MO and completed a pretest consisting of 20 MCQs (Appendix 1). Each scenario consisted of a 30-minute simulation workshop and a 30-minute debriefing session. One or two residents from PGY-1, PGY-2, or PGY-3 were randomly selected to participate in the scenario theme (Fig 1), while the remaining residents observed the sessions through a monitor in the observer room. Each debriefing session consisted of de-rolling and applying the ANTS, and was facilitated by the anaesthesia staff who created the respective scenarios. After finishing the workshop, all residents completed the posttest (Appendix 1). An anaesthesia staff member (MO) discussed the MCQ answers with the residents after the posttest.

Operational definition

Scenario participant was defined as a first-, second- or third-year resident who participated as a team leader or team member in the scenario. Observer was defined as a first-, second- or third-year resident who observed the session from the observation room. Technical skills in anaesthesia training were represented by MCQ, SAQ, and OSCE scores during the same period of the simulation ANTS training (midterm examinations in November 2017 and November 2019).

Outcome of the study and outcome measurement

The primary outcome of the study was the posttest MCQ score after the ANTS workshop. The secondary outcome was the pretest MCQ score before the ANTS workshop. The pretest and posttest MCQs consisted of the same 20-item questions. The reliability of these pretest and posttest MCQ items were 0.95 and 0.98, respectively, as assessed by two anaesthesiologists (NP and SS). The MCQ consisted of 80% (16/20 items) application taxonomy-related questions and 20% (4/20) recalled taxonomy-related questions, with the ratio of ANTS content being 25% in SA (5/20 items), 25% in TM (5/20 items), 30% in TW (6/20 items), and 20% in DM (4/20 items).7

Fig 1. Midterm examinations of anaesthesia training.

Note: PGY: post-graduate year, ANTS: anaesthetists’ non-technical skills, MCQ: multiple-choice question, SAQ: short answer question, OSCE: objective structured clinical examination.

Predictors and potential confounders

The main exposure variable was the midterm examination, which comprised 180 MCQs, 12 SAQs, and 24 OSCEs (Appendix 2). The average score of the anaesthesia residents’ competency for each year in terms of knowledge, skills, and attitude was also evaluated by all anaesthesia staffs before the examination (Appendix 2). Potential predictors were the residents’ role in the scenario (participant/observer), sex, PGY (1-3), and percentage of self-study prior to attending the workshop.

The step of statistical analysis

First, the basic comparison test to compare the residents’ characteristics and the midterm examination scores as well as MCQ ANTS scores among PGY (1-3) were performed. Continuous variables were presented using the median and interquartile range for non-normally distributed data and using mean and standard deviation (SD) for normally distributed data. Categorical variables were presented using frequency and percentage, and were compared using Fisher’s exact test or Pearson’s

APPENDIX 1

MCQ ANTS pretest and posttest

1. You are an anesthesiologist on call and handed over a case of hypotension. The inotrope was given but the hypotension persisted. You did not notice that the line of inotrope was clamped. Which ANTS category is related to this event?

   1. Task management

   2. Teamwork & communication

   3. Situation awareness

   4. Decision-making

2. You are an anesthesiologist and a team leader of a fire situation in OR. You need to discuss with the surgeon, a chairman of the surgery department, to rapidly manage a patient. The surgeon has refused to abandon surgery until he has completed the operation. Which key element would you deal with in this situation?

   1. Providing & maintaining standards

   2. Co-ordinating activities with team

   3. Using authority & assertiveness

   4. Recognising & understanding

3. According to question 2, which ANTS category was referred to?

   1. Task management

   2. Teamwork & communication

   3. Situation awareness

   4. Decision-making

4. You are a third-year resident working in the emergency room. There is a pregnancy case with fetal distress; however, all 3 emergency rooms are occupied. Which ANTS category applies in this situation?

   1. Task management

   2. Teamwork & communication

   3. Situation awareness

   4. Decision making

5. From question 4, you managed the pregnancy case with fetal distress by taking the first available elective room. To which key element does this situation referred?

   1. Identifying & utilizing resources

   2. Co-ordinating activities with the team

   3. Reconising & understanding

   4. Balancing risks & selecting options

6. What is the key element of situation awareness?

   1. Providing & maintaining standards

   2. Assessing capabilities

   3. Recognising & understanding

   4. Identifying options

7. What is the key element of decision making?

   1. Planning & preparing

   2. Supporting others

   3. Anticipating

   4. Re-evaluating

      
      

      
      

      APPENDIX 1 (Continued)

      
      

8. Which is FALSE regarding the key element of teamwork & communication?

   1. Identifying & utilizing resources

   2. Co-ordinating activities with the team

   3. Exchanging information

   4. Using authority & assertiveness

9. Failure of which ANTS category could lead to communication failure between two colleagues?

   1. Task management

   2. Teamwork & communication

   3. Situation awareness

   4. Decision making

10. What tool is the most important for the simulation of Anaesthetists' Non-Technical skills (ANTS) training?

    1. Simulation center

    2. Standardized patient

    3. Real-life case scenario

    4. Feedback or debriefing

11. You handed over the details, except for the information of morphine allergy, of a case of total abdominal hysterectomy with bilateral salpingo-oophorectomy to your colleague who was on call. This mistake could refer to which ANTS category(ies)?

    1. Teamwork & communication

    2. Teamwork & communication and task management

    3. Teamwork & communication and situation awareness

    4. Teamwork & communication and decision making

12. After placing an epidural catheter for painless labor at the labor room, you noticed that there was no basic monitoring. You managed this by informing an anesthetist nurse to palpate a radial pulse and a nurse aid to urgently get a monitor from the OR. What ANTS category are you concerned with the most?

    1. Task management

    2. Teamwork & communication

    3. Situation awareness

    4. Decision making

13. The anesthesiologist in charge did not have any plan for postoperative pain control for patient receiving total abdominal hysterectomy with bilateral salpingo-oophorectomy. The patient developed severe pain at the PACU. This error did not refer to which ANTS category?

    1. Situation awareness

    2. Teamwork & communication

    3. Task management

    4. Decision making

14. A surgeon would like to perform surgery in a case of immediate post-cardiac arrest in the OR. As the anesthesiologist in charge, you managed to discuss with him regarding the risk of performing surgery. Which ANTS category best reflects your action?

    1. Situation awareness

    2. Teamwork & communication

    3. Task management

    4. Decision-making

       
       

       
       

       APPENDIX 1 (Continued)

       
       

15. You are an anesthesia resident who administered the wrong type of packed red cell to a patient with massive bleeding. Before you transfused the packed red cell, you checked only the patient's name without checking the hospital number nor cross-checking with an anesthetist nurse. This mistake refers to which ANTS category?

    1. Teamwork & communication

    2. Decision making

    3. Situation awareness

    4. Task Management

16. The mistake from question 15 was relevant to which key element of that particular ANTS category?

    1. Balancing risks & selecting options

    2. Providing & maintaining standards

    3. Co-ordinating activities with the team

    4. Anticipating

17. You were a team leader of advanced cardiac life support (ACLS) resuscitation. You noticed that your team member could not perform chest compression effectively, so you managed to exchange a chest compressor with a mask ventilation member. Your decision correlates with which key element of ANTS?

    1. Co-ordinating activities with the team

    2. Exchanging information

    3. Using authority & assertiveness

    4. Assessing capabilities

18. You are giving anesthesia to a patient with a history of difficult airway. After the patient loses consciousness, you cannot ventilate him, so you call a senior anesthesiologist staff to stand by in the room. Your action could refer to which ANTS category(ies)?

    1. Situation awareness

    2. Situation awareness and teamwork & communication

    3. Situation awareness and task management

    4. Situation awareness and decision making

19. An anesthesia resident who was on call was about to intubate a non-difficult airway patient when he overheard a CPR code. The resident arranged for an anesthetist nurse to do the intubation instead. The resident then informed the anesthesiologist staff by phone while he ran to the CPR scene. His action was relevant to which key element?

    1. Prioritising

    2. Recognising & understanding

    3. Co-ordinating activities with the team

    4. Balancing risks & selecting options

20. As an anesthesiologist staff, you informed an anesthetist nurse to administer intravenous succinylcholine to a child. You suspected the child might develop a laryngospasm since there was no reading of oxygen saturation on the monitor. After you checked the oxygen saturation probe, you found that it was not connected to the child’s finger. This error was relevant to which ANTS category?

    1. Task management

    2. Teamwork & communication

    3. Situation awareness

    4. Decision making

APPENDIX 2

Combination of midterm examination score of resident training and resident’s competency evaluated by staff in 2017 and 2019

Abbreviation: PGY: post-graduate year

chi-square test. Continuous variables were compared using Kruskal–Wallis tests and analysis of variance, as appropriate. Second, to determine the degree of correlation between midterm examinations and MCQ ANTS scores, Pearson’s correlation coefficients (r) were used. Third, the r used among the main exposures and outcomes in the second step was considered a univariate analysis, and therefore, a directed acyclic graph (DAG) was used to select the potential confounders that could affect the main exposures and the outcomes into the multivariate model. Then, the DAGitty software version 3.0. was applied. Potential confounding variables including resident competencies (knowledge, skills, and attitude) suggested by the DAG were then selected for a multivariate linear regression model and were retained irrespective of their statistical significance.8,9 The outcomes were tested to ensure that they fit the residual of linear distribution assumption. The association between the posttest/pretest MCQ scores and midterm examinations was presented as an adjusted beta coefficients with 95% confidence limits (CL) and considered significant if the F-test p values were <0.05. Finally, the effect modification between the potential predictors and midterm examinations on the outcomes were evaluated for each final model.

Sample size calculation

The sample size was calculated based on the correlation between midterm MCQ scores and posttest ANTS scores. A correlation coefficient (r) of 0.4, which represented a medium level of correlation, was used to calculate the required sample size of 47 patients under a significance

level of 0.05. Since a total pool of 24 residents were available from PGY-1 to PGY-3, we used data from two cohorts – collected in 2017 and in 2019 – to include in this study project.

RESULTS

Forty-eight anaesthesia residents were recruited from among the two cohorts of ANTS training in 2017 and 2019, including 16 PGY-1, 16 PGY-2, and 16 PGY- 3 residents. Table 1 compares the characteristics of the various three-year anaesthesia residents, which reveals no significant differences. Table 2 compares the ANTS pretest/posttest scores, midterm examination scores, and anaesthesia residents’ competency as evaluated by the staff. There were no significant differences between the ANTS pretest/posttest scores and midterm examination scores among groups; however, the skill competency increased by year of resident training (P<0.001).

Table 3 shows the correlation between midterm examination scores, residents’ competency as evaluated by staff, and MCQ ANTS scores. The correlation between the midterm examination scores and pretest/posttest ANTS scores was low (r<0.3). SAQ scores had a negative correlation with posttest ANTS scores but were not significantly different (r=-0.24, P=0.10).

Analysis of pretest MCQ score

Nine variables (sex, PGY, self-study, knowledge score, skill competency, attitude, MCQ score, SAQ score, and OSCE) as suggested by a previous literature review were related to ANTS scores.4,5,7 There was an effect

TABLE 1. Comparison of characteristics of anaesthesia residents.

*Numbers in the table represent frequency (%) unless stated otherwise.

TABLE 2. Comparison of ANTS pretest and posttest scores, midterm examination scores, and anaesthesia residents’ competency as evaluated by staff.

*Mean (standard deviation) and P value by analysis of variance. †Median (interquartile range) and P value by Kruskal–Wallis test. Abbreviations: ANTS: anaesthetists’ non-technical skills, MCQ: multiple choice question, SAQ: short answer question, OSCE: objective structured clinical examination

TABLE 3. Correlation between midterm examination scores of anaesthesia training, residents’ competency as evaluated by staff, and multiple choice question ANTS scores.

P value by Pearson's correlation coefficients (r), OSCE: Objective structured clinical examination.

modification between sex and midterm MCQ scores in the pretest MCQ scores (P<0.05). Three potential biasing variables (PGY, knowledge score, and attitude) of the total effect of midterm examinations indicated by the DAG (Fig 2A) were included as the minimally sufficient adjustment set with midterm examination variables (MCQ*sex/SAQ/OSCE) (Table 4). Among males, MCQ scores had a negative correlation with pretest ANTS scores (P=0.024). Regardless of sex, SAQ scores had a

negative correlation with pretest ANTS scores (P=0.033) whereas OSCE was positively correlated with pretest ANTS scores (P=0.038).

Analysis of posttest MCQ scores

Eleven variables (sex, PGY, self-study, resident roles, knowledge score, skill competency, attitude, pretest ANTS score, MCQ score, SAQ score, and OSCE) suggested by the previous literature review were related to ANTS scores.4,5,7

Fig 2. Hypothesised causal relationship between midterm examinations and pretest (A) and posttest (B) non-technical skill scores.

Note: PGY: post-graduate year, ANTS: anaesthetists’ non-technical skills, MCQ: multiple choice question, SAQ: short answer question, OSCE: objective structured clinical examination.

There was no evidence of effect modification between sex and any other variables in relation to the midterm examinations in the posttest MCQ model. Three potential biasing variables (PGY, knowledge score, and attitude) of the total effect of midterm examinations indicated by the DAG (Fig 2B) were included as the minimally sufficient

adjustment set with midterm examination variables (MCQ/SAQ/OSCE) (Table 4). The midterm MCQ scores showed a positive correlation with ANTS posttest scores (P=0.007) whereas the SAQ scores showed a negative correlation with ANTS posttest scores (P=0.004).

TABLE 4. Multiple linear regression by total effect model predicting relative probability of pretest and posttest ANTS scores based on midterm examinations.

Male (ref= female)

8.14 (-0.11, 16.4)

8.04 (-0.07, 16.15)

0.052

Main exposure variables

Pretest ANTS score† Crude β

(95% CL)

Adjusted β

(95% CL)

P value*

MCQ score 0.07 (0, 0.14) 0.14 (0.04, 0.24) 0.006

SAQ score

0 (-0.07, 0.07)

-0.11 (-0.21, -0.01)

0.033

MCQ score: male -0.12 (-0.23, 0) -0.13 (-0.24, -0.02) 0.024

r of final model

0.38

Main exposure variables

Posttest ANTS score‡ Crude β

(95% CL)

Adjusted β

(95% CL)

P value*

OSCE score 0.5 (-0.17, 1.18) 1.02 (0.06, 1.98) 0.038

SAQ score

-0.07 (-0.16, 0.01)

-0.16 (-0.27, -0.05)

0.004

MCQ score 0 (-0.07, 0.07) 0.14 (0.04, 0.24) 0.007

r of final model

0.52

OSCE score -0.51 (-1.37, 0.36) -0.25 (-1.29, 0.8) 0.634

*F-test. β: Beta coefficients, CL: confidence limits, MCQ: multiple choice question, SAQ: short answer question, OSCE: objective structured clinical examination, ref: reference.

†Minimally sufficient adjusted set suggested by a directed acyclic graph: post-graduate year of anaesthesia residents, knowledge score, attitude competency.

‡Minimally sufficient adjusted set suggested by a directed acyclic graph: post-graduate year of anaesthesia residents, knowledge score, attitude competency.

DISCUSSION

We examined whether technical skills in anaesthesia training using midterm MCQ/SAQ/OSCE assessments were associated with non-technical skill knowledge based on the pretest/posttest MCQ ANTS simulation workshop. According to the one-day workshop with three ANTS scenarios for each midterm examination, only one or two residents, and not all, could participate in each workshop. The rest were observed in the observer room. Instead of assessing their ANTS behavior one- by-one in the workshop, the debrief session, which was attended by all the residents, was performed to ensure a better and understanding of ANTS scenarios. Therefore, an assessment of ANTS knowledge using MCQs was feasible and helped the residents understand more about the ANTS elements incorporated in their

learning by performing them daily. Therefore, the residents gradually gained non-technical skills during the training. Since we used two prospective cohorts (2017 and 2019), eight residents who were PGY1 in 2017 become PGY3 in 2019. Approximately 17% (8/48) of the residents might have been familiar with pretest MCQ ANTS. To minimise this bias, PGY was one of the adjusted sets for confounders in the final multivariate model of pretest and posttest MCQ ANTS. Our findings from Table 3 reveal that the relationship between MCQ/SAQ/OSCE scores and pretest/posttest MCQ ANTS scores (univariate analysis) were low (r<0.3), but the final multivariate linear regression models revealed a significant relationship not only between midterm examinations and pretest ANTS scores (r=0.38) but also between midterm examinations and posttest ANTS scores (r=0.52).

This proves that the relationship between technical and non-technical skills in anaesthesia is not simply linear due to the environment and many other circumstances during training, such as sex, PGY, self-study, residents’ competencies, and residents’ roles during ANTS workshop. This non-linear network can help researchers interpret the complex relationships within it.10,11 Therefore, DAG was performed to construct the causal relationship between the two anaesthesia skills and to minimise confounding variables. We used midterm examinations (MCQ/SAQ/ OSCE) to represent technical skill in anaesthesia training, which was similar to what was done in Raksamani’s study.12 The difference was that the MCQs, which comprised all four ANTS categories (SA/TM/TW/DM), was implemented instead of assessing ANTS behaviours among residents to correspond with non-technical skills. Moreover, to minimise the measurement bias, technical skills, as assessed by midterm examinations and the ANTS workshop, were performed in the same evaluation period (Fig 1).

Before the ANTS workshop, we found that OSCE had a positive correlation with pretest MCQ ANTS scores. This may be attributable to the fact that OSCE demonstrates clinical skills/competency in a standardised medical scenario, which represents routine clinical practise.13 Thus, this skill may have been applied in the scenario in the pretest MCQ before the workshop. Furthermore, our results revealed that midterm MCQ scores had a positive correlation with pretest MCQ scores but, among males, it had negative correlation. Regardless of sex, the reason for a positive correlation between the midterm MCQ scores and pretest ANTS MCQ scores could be similar types of questions. However, compared to females, MCQ scores revealed a reverse association with pretest MCQ scores among males. A previous study revealed that males had higher pretest MCQ scores than females.7 Therefore, it was found that sex could somehow affect the correlation between midterm MCQ scores and pretest ANTS scores and was considered a common confounding variable in other models.14,15

After the ANTS workshop, there was no effect modification between the midterm examinations and posttest ANTS scores. We found midterm MCQ scores to have an obvious positive correlation with posttest ANTS scores (P<0.007). Perhaps, the ANTS lecture before the workshop promoted ANTS basic knowledge simultaneously, while enhancing ANTS skills through different case scenarios (SA/TW/TM/DM) during the workshop and debriefing sessions. This possibly led to a higher positive correlation between midterm MCQ scores and posttest ANTS score. A previous study revealed that participants or observers can gain similar knowledge of

ANTS based on the pretest/posttest MCQ scores.7

The reasons we implemented the 20-item MCQs to represent non-technical skills in anaesthesia training were as follows: first, multiple choice tests can be an effective and simple way to measure learning.16 Second, assessments can be completed quickly, providing students with prompt feedback. Third, MCQs can assess knowledge and cognition and cover large topics in the curriculum.17 Moreover, the application of MCQs can vary from simple recall to problem solving. Well-written MCQs can go beyond testing rote facts and might measure higher cognitive abilities.18 However, from our results (Table 2), it was found that the posttest MCQ ANTS scores among residents (14.3) improved compared to pretest scores (12.6) but not greatly. This might be because the nature of four-answer choices makes it difficult to decide the best answer among four ANTS categories from each ANTS situation, especially an ANTS training beginner. Therefore, we discussed the appropriate answer with the residents after the posttest to stimulate higher cognitive function during clinical practise.

Surprisingly, SAQ scores revealed a negative correlation with pretest MCQ ANTS scores, and this finding was confirmed by posttest MCQ ANTS scores when SAQ scores indicated a more significant negative correlation. This could be because SAQs need to be related to a common or specific topic depending on the objective of the curriculum subjects whereas the ANTS MCQ test is broader and non-specific to a particular component of ANTS. Moreover, SAQs are more similar to essay questions that tend to be more difficult than MCQs since there are more cues to stimulate memory and because recall rather than recognition is required.19 A good attitude, which was our most important confounder, is necessary in anaesthesia training and could be the key to becoming proficient in both technical and non-technical anaesthesia skills, especially the learners’ attitudes towards medical ethics education.20

Strengths and limitations of the study

We formed a multivariate analysis model using DAG to adjust indicated confounders in the final model in order to determine whether the midterm examinations were independent predictors of the outcomes. The limitations of the study are as follows: first, the assessment of ANTS knowledge was used instead of ANTS behavioural skills of each resident owing to the limitation of time for the midterm ANTS workshop. Second, because we used two cohorts from two different years, the periods of training might have affected the midterm examination scores because the training environment changes with time.

CONCLUSION

Our findings revealed that technical skills based on midterm MCQ and OSCE assessments were associated with non-technical skill knowledge in anaesthesia. An assessment of ANTS knowledge during the workshop can help residents comprehend detailed ANTS information and foresee adverse situations in advance. Residents can gain non-technical skills by performing routine anaesthesia practise by collaborating with other anaesthesia personnel (nurse anaesthetists/anaesthesia technicians/anaesthesia staffs/surgeons) daily. A good attitude, which was our most important confounder, is necessary in anaesthesia training and could be the key to becoming proficient in both technical and non-technical anaesthesia skills. A further research regarding non-technical skill knowledge and non-technical skill behaviour of all residents should be performed to improve our ANTS training workshop.

ACKNOWLEDGMENTS

We would like to thank Editage for assistance with editing the manuscript.

Funding: Faculty of Medicine, Prince of Songkla University, Hat Yai, Thailand.

Conflict of interest: The authors have no conflicts of interest.

REFERENCES

1. Flin R, Patey R. Non-technical skills for anaesthetists: developing and applying ANTS. Best Pract Res Clin Anaesthesiol 2011;25: 215-27.

2. Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R. Anaesthetists’ non-technical skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth 2003;90:580-8.

3. Mete I, Brannick MT. Estimating the reliability of nontechnical skills in medical teams. J Surg Educ 2017;74:596-611.

4. Fletcher GC, McGeorge P, Flin RH, Glavin RJ, Maran NJ. The role of non-technical skills in anaesthesia: a review of current literature. Br J Anaesth 2002;88:418-29.

5. Riem N, Boet S, Bould MD, Tavares W, Naik VN. Do technical skills correlate with non-technical skills in crisis resource management: a simulation study. Br J Anaesth 2012;109:723-8.

6. Brunckhorst O, Shahid S, Aydin A, Khan S, McIlhenny C, Brewin J, et al. The relationship between technical and nontechnical skills within a simulation-based ureteroscopy training environment.

   J Surg Educ 2015;72:1039-44.

7. Oofuvong M, Pattaravit N, Kanjanawanichkul O, Siripruekpong S, Nuanjun K, Suwannarat B. Do different roles of anesthesia residents improve knowledge retention after non-technical skills workshop? J Med Assoc Thai 2021;104:1519-27.

8. Shrier I, Platt RW. Reducing bias through directed acyclic graphs. BMC Med Res Methodol 2008;8:70. doi: 10.1186/1471- 2288-8-70.

9. Suttorp MM, Siegerink B, Jager KJ, Zoccali C, Dekker FW. Graphical presentation of confounding in directed acyclic graphs. Nephrol Dial Transplant 2015;30:1418-23.

10. Evidence scan: Complex adaptive systems. [homepage on the Internet]. The health Foundation [updated 2010 August; cited 2022 Jul 1]. Available from: http://www.health.org.uk/sites/ default/files/ComplexAdaptiveSystems.pdf.

11. Pype P, Mertens F, Helewaut F, Krystallidou D. Healthcare teams as complex adaptive systems: understanding team behaviour through team members’ perception of interpersonal interaction. BMC Health Serv Res 2018;18:570. doi: 10.1186/ s12913-018-3392-3.

12. Raksamani K, Jirativanont T, Sareenun P. Correlation of medical knowledge and non-technical skills assessment in anesthesia residents. Siriraj Med J 2020;72:483-7.

13. Gormley G. Summative OSCEs in undergraduate medical education. Ulster Med J 2011;80:127-32.

14. Shapiro JR, Klein SL, Morgan R. Stop ‘controlling’ forsex and gender in global healthresearch. BMJ Global Health 2021;6:e005714. doi:10.1136/bmjgh-2021-005714.

15. Groenwold RH, Klungel OH, Grobbee DE, Hoes AW. Selection of confounding variables should not be based on observed associations with exposure. Eur J Epidemiol 2011;26:589-93.

16. Farley JK. The multiple-choice test: writing the questions. Nurse Educ 1989;14:10-2, 39.

17. Salih KMA, Alshehri MAA, Elfaki OA. A comparison between students’ performance in multiple choice and modified essay questions in the MBBS pediatrics examination at the College of Medicine, King Khalid University, KSA. Journal of Education and Practice 2016;7:116-20.

18. Scully D. Constructing multiple-choice items to measure higher-order thinking practical assessment. Assessment, Research, and Evaluation [serial on the Internet]. 2017 May [cited 2022 Jul 5]; 22(4): [about 14 p.]. Available from:https://scholarworks. umass.edu/cgi/viewcontent.cgi?article=1355&context=pare

19. Farooqui F, Saeed N, Aaraj S, Sami MA, Amir M. A comparison between written assessment methods: multiple-choice and short answer questions in end-of-clerkship examinations for final year medical students. Cureus 2018;10:e3773. doi: 10.7759/cureus.3773.

20. Sathirareuangchai S. Iramaneerat C. Medical students’ and interns’ attitudes toward medical ethics education in a Thai medical school. Siriraj Med J 2016;68:97-103.