Developing and Validating a Test of Higher Order Thinking Skills for Future Teachers
Abstract
Higher-order thinking skills (HOTS) are extremely needed to be possessed by future teachers but there is less work on the development of suitable measuring instruments for HOTS (Zhou et al., 2023). The present study is intended to develop an instrument to test the higher order thinking skills (HOTS) of future teachers at university level. The instrument named as Maimoona Higher Order Thinking Skills Test (MHOT). The developed test is delimited to two main HOTS: Creative and Critical thinking skills. These HOTS are distinct as well as have inter-dependency. There are total fourteen interpretive exercises/items in the instrument. To meet the psychometric criteria the construct validity evidence (n = 200) was gathered through exploratory factor analysis (EFA). The convergent validity evidence was also collected and established by using the College Assessment Academic Proficiency of Critical Thinking (CAAP) test with the average variance 0.65 and composite reliability of 0.92 between MHOT and CAAP. The inter-rater reliability value of Kappa was 0.724 and the Cronbach’s alpha value of the entire instrument was 0.91. The content validation process involved four experts in total, three content experts from the discipline of Education and one from the discipline of Psychology. Scale content validity index (SCVI) was 0.93. Item analysis was also conducted to ensure the discrimination index of the items and all items were falling under the acceptable criteria except two items.
References
Abdullah, A. H., Mokhtar, M., Abd Halim, N. D., Ali, D. F., Mohd Tahir, L., & Abdul Kohar, U. H. (2016). Mathematics teachers’ level of knowledge and practice on the implementation of higher-order thinking skills (HOTS). EURASIA Journal of Mathematics, Science and Technology Education, 13(1). https://doi.org/10.12973/eurasia.2017.00601a
Abosalem, Y. (2016). Assessment techniques and students’ higher-order thinking skills. International Journal of Secondary Education, 4(1), 1-11.
ACT. (2008). Collegiate Assessment Academic Proficiency Critical Thinking (CAAP) Technical Handbook. American College Testing Program. www.act.org
Anderson, L. W. (2005). Objectives, evaluation, and the improvement of education. Studies in Educational Evaluation, 31(2–3), 102–113.
Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives: complete edition. Addison Wesley Longman, Inc. https://doi.org/10.1016/j. stueduc.2005.05.004
Arievitch, I. M. (2020). The vision of developmental teaching and learning and Bloom’s taxonomy of educational objectives. Learning, Culture and Social Interaction. https://doi.org/10.1016/j.lcsi.2019.01.007
Astuti, A. P., Aziz, A., Sumarti, S. S., & Bharati, D. A. L. (2019). Preparing 21st century teachers: Implementation of 4C character’s pre-service teacher through teaching practice. Journal of Physics, 1233(1).
Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of Educational Objectives: The Classification of Educational Goals by a Committee of College and University Examiners. Handbook I: Cognitive Domain. New York: Longmans Publishing.
Brandt, J.-. O., Barth, M., Merritt, E., & Hale, A. (2021). A matter of connection: The 4 Cs of learning in pre-service teacher education for sustainability. Journal of Cleaner Production. https://doi.org/10.1016/j.jclepro.2020.123749
Brookhart, S. M. (2010). How to assess higher-order thinking skills in your classroom. ASCD.
Bruner, J. (1966). Toward a theory of instruction. Massachusetts. Harvard University Press.
Creswell, J. W., & Creswell, J. D. (2022). Research design: Qualitative, quantitative, and mixed methods approaches. (6th ed.). Sage.
Din, A. M., & Jabeen, S. (2014). Scenario-based assessment exercises and the perceived learning of mass communication students. Asian Association of Open Universities Journal, 9(1), 93-103.
Field, A. (2018). Discovering statistics using IBM SPSS statistics (5th ed.). Sage.
Gay, L. R. (1985). Educational Evaluation & Measurement. Competencies for analysis and application.
Hamlin, M.D. (2022). Developing self-directed learning skills for lifelong learning (pp. 209–234). https://doi.org/10.4018/978-1-7998-7661-8.ch012. https://doi.org/10.11591/edulearn.v13i1.11218
Hussain, Z. (2021). Paradigm of technological convergence and digital transformation. The challenges of CH sectors in the global COVID-19 pandemic and commencing resilience-based structure for the post-COVID-19 era. Digital Applications in Archaeology and Cultural Heritage, 21, e00182. https://doi.org/10.1016/j. daach.2021.e00182
Jansen, T., & M¨oller, J. (2022). Teacher judgments in school exams. Influences of students’ lower-order-thinking skills on the assessment of students’ higher-order-thinking skills. Teaching and Teacher Education, 111, https://doi.org/10.1016/j.tate.2021.103616
Kaiser, H. F. (1970). A second generation little jiffy. Psychometrika, 35(4), 401–415.
Kaiser, H. F. (1974). Little jiffy, mark IV. Educational and Psychological Measurement, 34(1), 111–117. https://doi.org/10.1177/001316447403400115
Karunarathne, W., & Calma, A. (2024). Assessing creative thinking skills in higher education: deficits and improvements. Studies in Higher Education, 49(1), 157-177.
Kroth, M., Carr-Chellman, D. J., & Rogers-Shaw, C. (2022). Formation as an organizing framework for the processes of lifelong learning. New Horizons in Adult Education and Human Resource Development, 34(1), 26–36. https://doi.org/10.1002/nha3.20348
Lewis, A., & Smith, D. (1993). Defining higher order thinking. Theory Into Practice, 32(3), 131–137. https://doi.org/10.1080/00405849309543588
McHugh, M. L. (2012). Interrater reliability: the kappa statistic. Biochemia medica, 22(3), 276-282.
Miyazaki, R. (2024). Using Revised Bloom’s Taxonomy to Evaluate Higher Order Thinking Skills (HOTS) in Tasks From an 8th Grade English Language Textbook in Japan (Master's thesis, Southeast Missouri State University.
Nagappan, R. (2002). Using constructivist approach to teach higher order thinking skills. Transforming Teaching Practice to Facilitate Mindful Learning. In 10th International Conference on Thinking (pp. 15-19).
Naeem, M., & Rana, R. A. (2023). An Investigation about Creative and Critical Thinking Skills of Prospective Teachers: A Preliminary Study. Bulletin of Education and Research, 45(3), 35-47.
Piaget, J. (1973). To Understand is to Invent. New York: Grossman. (http://curriculum.calstatela.edu/faculty/psparks/theorists/501const.htm)
Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Focus on research methods is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Researchin Nursing & Health, 30, 459–467.
Resnick, L. (1987). Education and Learning to Think. National Academy Press.
Rosba, E., Zubaidah, S., Mahanal, S., & Sulisetijono, S. (2021). College students’ critical thinking skills and creativity. AIP Conference Proceedings, 2330 (1).
Salih, M., & Abdelbagi, O. (2022). Scenario-Based, Single Best, Multiple-Choice Questions (SB-SB-MCQs) in basic medical sciences. An exploratory study about the staff awareness, knowledge and difficulties encountered. Journal of Biosciences and Medicines, 10 (9), 79-85.
Schmidt, E. (2022). AI, great power competition & national security. Daedalus, 151(2), 288–298. https://doi.org/10.1162/daed_a_01916
Silva Pacheco, C., & Iturra Herrera, C. (2021). A conceptual proposal and operational definitions of the cognitive processes of complex thinking. Thinking Skills and Creativity, 39. https://doi.org/10.1016/j.tsc.2021.100794
Stayanchi, J. (2018). Higher order thinking through Bloom’s taxonomy. Kwansei Gakuin University Humanities Review, 22, 117-124.
Suparman, S., Juandi, D., & Tamur, M. (2021). Does problem-based learning enhance students’ higher order thinking skills in mathematics learning? A systematic review and meta-analysis. 4th International Conference on Big Data and Education (pp. 44–51). https://doi.org/10.1145/3451400.3451408
Tambunan, H., & Naibaho, T. (2019). Performance of mathematics teachers to build students’ high order thinking skills (HOTS). Journal of Education and Learning (EduLearn), 13(1), 111–117.
Tibbetts, G. G. (2013). Elements of scientific thinking. How the Great Scientists Reasoned (pp. 5–24). Elsevier. https://doi.org/10.1016/B978-0-12-398498-2.00002-3
Tight, M. (2021a). Twenty-first century skills: Meaning, usage and value. European Journal of Higher Education, 11(2), 160–174. https://doi.org/10.1080/ 21568235.2020.1835517
Tijsma, G., Hilverda, F., Scheffelaar, A., Alders, S., Schoonmade, L., Blignaut, N., et al. (2020). Becoming productive 21st century citizens. A systematic review uncovering design principles for integrating community service learning into higher education courses. Educational Research, 62(4), 390–413. https://doi.org/ 10.1080/00131881.2020.1836987
Ülger, K., Morsünbül, Ü. (2016). The differences in creative thinking: the comparison of male and female students. The Online Journal of Counselling and Education, 5(4), 1-12. http://www.tojce.com/frontend//articles/pdf/v5i4/1-12-ulger-morsunbulpdf.pdf
Vincent-Lancrin, S. et al. (2019), Fostering Students’ Creativity and Critical Thinking: What it means in school, OECD Publishing, Paris, https://doi.org/10.1787/62212c37-en.
Vygotsky, L. S. (1978). Mind in society. Harvard University Press.
Wiyaka, W., Prastikawati, E. F., & Kusumo Adi, A. P. (2020). Higher-order thinking skills (hots)-based formative assessment: a proposed model for language learning assessment. Vision: Journal for Language and Foreign Language Learning, 9(2), 115-130.
Wechsler, S. M., Saiz, C., Rivas, S. F., Vendramini, C. M. M., Almeida, L. S., Mundim, M. C., & Franco, A. (2018). Creative and critical thinking: Independent or overlapping components? Thinking Skills and Creativity, 27, 114-122.
Yusoff, M. S. B. (2019). ABC of content validation and content validity index calculation. Education in Medicine Journal, 11(2), 49-54.
Zhou, Y., Gan, L., Chen, J., Wijaya, T. T., & Li, Y. (2023). Development and validation of a higher-order thinking skills assessment scale for pre-service teachers. Thinking Skills and Creativity, 48, 101272.
Zohar, Anat, & Dori, Y. J. (2003). Higher order thinking skills and low-achieving students: Are they mutually exclusive? Journal of the Learning Sciences, 12(2), 145–181. https://doi.org/10.1207/S15327809JLS1202_1
