The Validity and Reliability of Clinical Performance Assessment of Residents in the General Surgery Residency Program

Abstract

A retrospective study of clinical performance assessment of residents in the general surgery residency program at the Department of Surgery, Faculty of Medicine Siriraj Hospital, during the 200-2001 and 2001-2002 academic years was done to evaluate the inter-rater reliability, internal structure, and ducational discriminant validity of the assessment. The inter-rater reliabilities were determined by using the intraclass correlation. Correlations between individual performance ratings were checked to determine how well faculty members differentiate their ratings with individual clinical skills. Finally, the percentage of marginal or unsatisfactory ratings was checked to demonstrate how sensitive the ratings were in identifying performance deficiencies among residents.
   From the evaluation of 42 residents in the 2000-2001 academic year, inter-rater reliability coefficients of individual ratings ranged from 0.06 to 0.93 with an average of 0.51. From the evaluation of 47 residents in the 2001-2002 academic year, inter-rater reliability coefficients ranged from 0.04 to 0.85 with an average of 0.49. Inter-rater reliabilities of performance ratings of third-year residents were at an acceptable standard for a medium-stake assessment. However, those of first and second-year residents should need some improvement. Tasks performed by first and second-year residents seem to have been inadequate for faculty members to evaluate their performance reliably. The correlation study between individual performance ratings demonstrated the lack of discrimination between many clinical skills in faculty ratings. Faculty members need some guidelines to differentiate between many items including relationship with patients, relationship with other doctors, and relationship with other workers; knowledge and judgment; work concentration and work effectiveness; and punctuality and responsibility. About ten percent of ratings were marginal or unsatisfactory which indicated that these ratings were sensitive enough to identify performance deficiencies among residents.