Improving the Reliability of OSCEs for the Royal Thai College of Obstetricians and Gynecologists Board Examination

Abstract

Background Since 1990, the Royal Thai College of Obstetricians and Gynecologists (RTCOG) has launched an OSCE as the main assessment of skills in Obstetrics and Gynecology. This OSCE comprised of twenty 5-minute stations, including three pathology and cytology stations. Previous psychometric analysis of the OSCE revealed a moderate reliability. However, pursuing a higher reliability of the OSCE has always been the main goal of the RTCOG Board Examination Committee.

Methods A descriptive study of the reliabilities (Cronbach u) of OSCEs for the RTCOG Board Examination between 1992 and 1998 was performed by using the computerized SPSS program under window operation. From the previous studies, the reliability of OSCEs between 1992 and 1994 ranged from 0.54 to 0.75. In order to improve such reliabilities, the Board Examination Committee decided to replace all stations of pathology and cytology with other stations for data interpretations and treatments. Simultaneously, a new set of 10-item pathology as well as cytology examination was launched.

Results The reliabilities of the new OSCEs between 1995 and 1998 were much higher than those between 1992 and 1994. They ranged from 0.79 to 0.82. Interestingly, the reliabilities of the 10-item pathological examination ranged from 0.55 to 0.66. The average failure in an examination of the original and the new OSCEs were 7.11% and 4.08% respectively.

Conclusion Replacement of pathological and cytological stations with other stations for data interpretations and treatments in the original OSCEs has much improved the reliabilities of the Royal Thai College of Obstetricians and Gynecologists OSCEs. The higher reliabilities of the new OSCEs has resulted in fewer failed candidates.