Development and Validation of the Anesthesia Critical- Thinking Test

Abstract

Objective: Despite critical thinking being a crucial skill in anesthesia practice, no specific tool exists to evaluate the skill in anesthesia training. The present study measured the validity, reliability and practicality of the Anesthesia Critical-Thinking Test (ACTT) for measuring the stages of critical thinking in anesthesia settings. Materials and methods: Part I Tool Development: We developed the ACTT to evaluate critical-thinking skills by ranking learners into six critical-thinking stages: 1) unreflective thinker, 2) beginning critical thinker, 3) practicing critical thinker, 4) advanced critical thinker, 5) accomplished critical thinker and 6) challenged thinker. Five educational experts assessed content validity. Part II Tool Evaluation: We retrospectively and randomly assessed 47 long-essay assignments written by 1st-3rd-year anesthesia residents in 2018. The assignments were subsequently blindly reviewed by two researchers to determine the critical-thinking stages. Two evaluators for tested concurrent validity, reliability, and practicality. Results: The ACTT’s items were adjusted for content validity until the item-objective congruence was >0.5. Concurrent validity between the scores and the critical-thinking stages showed moderate correlation by both evaluators (Evaluator 1 rs 0.574, P<0.001; Evaluator 2 rs 0.603, P<0.001). High-performance learners (stages 3, 4 and 5) had significantly higher scores than low-performance learners (stages 1 and 2), with P=0.001. Interrater reliability was strongly correlated (rs 0.866, P<0.001). Both evaluators were satisfied with the tool’s practicality. Conclusion: The ACTT provides a valid, reliable, and practical means of evaluating the critical-thinking skills of anesthesia residents.