Generalizability Coefficients for Measuring Coding Competencies of 9th Grade Students with the Application of Generalizability Theory
Keywords:
Generalizability Coefficients, Coding Competency Measurement, Competency Rubrics, Generalizability TheoryAbstract
The generalizability coefficient ensures the reliability of test results when measured under varying conditions, such as different test items, different raters, or other differing circumstances. The purpose of this research was to validate tasks for measuring coding competency for 9th grade students with the application of generalizability theory. The sample consists of 90 9th grade students from Chumphon province, selected through a multi-stage random sampling. Two facets were investigated, and the r : ( p x i ) design was adopted, where p denotes student component (90 students), i denotes task (4 tasks), and r denotes rater (3 raters). Experimental instruments were scoring criteria and tasks to measure coding competency. Statistics used in data included variance component analysis under the generalizability theory. The results showed that the interrater reliability coefficients among the 1st, 2nd, and 3rd raters were .991, .994, and .997, respectively. According to the variance component, the highest source of variability was the student within the examiners for both non-computerized and computerized coding performance assessments (91.5% and 90.7%, respectively). The estimation of generalizability coefficients suggested acceptable reliability coefficients for the relative and absolute decision of non-computerized coding and computerized coding (= .7612 and .7557, = .7612 and .7557, respectively). It is concluded that students should be assigned two tasks each for both non-computer-based and computer-based formats, evaluated by a single rater.
References
ชนิสรา สงวนไว้ และศิริชัย กาญจนวาสี. (2559). ความเที่ยงของการวัดในระบบการประเมินสมรรถนะเชิงปฏิบัติ: การประยุกต์ทฤษฎีการสรุปอ้างอิง. วารสารวิจัยทางการศึกษา, 9(2), 26–43.
นลินี ณ นคร. (2561). วิธีการวัดและประเมินการคิดตามสภาพจริง (พิมพ์ครั้งที่ 2). สันติศิริการพิมพ์.
สำนักงานเลขาธิการสภาการศึกษา. (2564). แนวทางการส่งเสริมการจัดการเรียนการสอนวิทยาการคำนวณ Coding เพื่อพัฒนาทักษะผู้เรียนในศตวรรษที่ 21. 21 เซ็นจูรี่.
Balanskat, A., & Engelhardt, K. (2015). Computing our future: Computer programming and coding priorities, school curricula and initiatives across Europe.
Brackmann, C. P., Román-González, M., Robles, G., Moreno-León, J., Casali, A., & Barone, D. (2017, November 8–10). Development of computational thinking skills through unplugged activities in primary school. In Proceedings of the 12th workshop on primary and secondary computing education (pp. 65–72). Nijmegen, Netherlands. https://doi.org/10.1145/3137065.3137069
Brennan, K., & Resnick, M. (2012, April 13–17). New frameworks for studying and assessing the development of computational thinking. In Proceedings of the 2012 annual meeting of the American Educational Research Association (pp. 1–25). Vancouver, Canada. http://scratched.gse.harvard.edu/ct/files/AERA2012.pdf
Brennan, R. L. (2001). Generalizability theory. Springer-Verlag.
Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. Wiley.
Csizmadia, A., Curzon, P., Dorling, M., Humphreys, S., Ng., T., Selby, C., & Woollard, J. (2015). Computational thinking: A guide for teachers (Computing at School). https://eprints.soton.ac.uk/424545/1/150818_Computational_Thinking_1_.pdf
Djambong, T. (2016, July 6). Computational thinking in connection with the acquisition of key skills for the 21st century: Reflection from literature review. In Proceedings of 8th international conference on education and new learning technologies (pp. 500–509). Barcelona, Spain. https://doi.org/10.21125/edulearn.2016.1096
Ebel, R. L., & Frisbie, D. A. (1986). Essentials of educational measurement (4th ed.). Prentice Hall.
Hopkins, K. D., & Stanley, J. C. (1981). Educational and psychological measurement and evaluation (6th ed.). Prentice-Hall.
Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review, 2(2), 130–144. https://doi.org/10.1016/j.edurev.2007.05.002
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012
Korkmaz, Ö. (2016). The effect of Scratch- and Lego Mindstorms EV3-based programming activities on academic achievement, problem-solving skills and logical-mathematical thinking skills of students. Malaysian Online Journal of Educational Sciences, 4(3), 73–88. https://eric.ed.gov/?id=EJ1106444
Mehrens, W. A., & Lehmann, I. J. (1991). Measurement and evaluation in education and psychology (4th ed.). Houghton Mifflin Company.
National Research Council. (2010). Report of a workshop on the scope and nature of computational thinking. National Academies Press. https://www.nae.edu/24767/Report-of-a-Workshop-on-The-Scope-and-Nature-of-Computational-Thinking
Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Sage.
Thorndike, R. M., Cunningham, G. K., Thorndike, R. L., & Hagen, E. P. (1991). Measurement and evaluation in psychology and education (5th ed.). Macmillan Publishing.
Tsarava, K., Moeller, K., Pinkwart, N., Butz, M., Trautwein, U., & Ninaus, M. (2017). Training computational thinking: Game-based unplugged and plugged-in activities in primary school. https://www.researchgate.net/publication/320491120
Unahalekhaka, A., & Bers, M. U. (2022). Evaluating young children’s creative coding: Rubric development and testing for ScratchJr projects. Education and Information Technologies, 27, 6577–6597. https://doi.org/10.1007/s10639-021-10873-w
Wing, J. M. (2010). Computational thinking: What and why?. https://www.cs.cmu.edu/~CompThink/resources/TheLinkWing.pdf
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Journal of Multidisciplinary Academic Research and Development (JMARD)
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
