ChatGPT in higher education: Measurement instruments to assess student knowledge, usage, and attitude

Carmen Köhler; Johannes Hartig

doi:10.30935/cedtech/15144

Carmen Köhler ¹ ^* , Johannes Hartig ¹

More Detail

CONT ED TECHNOLOGY, Volume 16, Issue 4, Article No: ep528

https://doi.org/10.30935/cedtech/15144

Submitted: 23 July 2024, Published Online: 10 September 2024

OPEN ACCESS 499 Views 399 Downloads

Download Full Text (PDF)

Abstract

Since ChatGPT-3.5 has been available to the public, the potentials and challenges regarding chatbot usage in education have been widely discussed. However, little evidence exists whether and for which purposes students even apply generative AI tools. The first main purpose of the present study was to develop and test scales that assess students’ (1) knowledge about ChatGPT, (2) actual ChatGPT usage and perceived value of use, and (3) attitude towards ChatGPT. Our second aim was to examine the intercorrelations between these scales, and to investigate differences (a) across five academic fields (i.e., human sciences, social sciences, teaching profession, health sciences, and law and economics) and (b) between stages of education (i.e., number of semesters). N = 693 students from various German universities participated in our online survey. Quality checks (Cronbach’s alpha, MacDonald’s omega, and confirmatory factor analyses) show satisfactory results for all scales. The scales all positively relate to each other, except for the knowledge and attitude scales. This means that more knowledge about ChatGPT is connected to a less favorable attitude regarding the generative AI tool. Lastly, MANOVA and subsequent Bonferroni corrected ANOVA tests show that ChatGPT is mostly used by law and economics students, and most frequently by students in the third year of higher education.

Keywords: ChatGPT in higher education, student knowledge, student use, student attitude, scale development, assessment, large language models (LLMs)

References

Abdaljaleel, M., Barakat, M., Alsanafi, M., Salim, N. A., Abazid, H., Malaeb, D., Mohammed, A. H., Hassan, B. A. R., Wayyes, A. M., Farhan, S. S., Khatib, S. E., Rahal, M., Sahban, A., Abdelaziz, D. H., Mansour, N. O., AlZayer, R., Khalil, R., Fekih-Romdhane, F., Hallit, R., …, & Sallam, M. (2024). A multinational study on the factors influencing university students’ attitudes and usage of ChatGPT. Scientific Reports, 14(1983), 1–14. https://doi.org/10.1038/s41598-024-52549-8
Alkaissi, H., & Mcfarlane, S. (2023). Artificial hallucinations in ChatGPT: Implications in scientific writing. Cureus, 15. https://doi.org/10.7759/cureus.35179
Banihashem, S. K., Kerman, N. T., Noroozi, O., Moon, J., & Drachsler, H. (2024). Feedback sources in essay writing: Peer-generated or AI-generated feedback? International Journal of Educational Technology in Higher Education, 21(1), Article 23. https://doi.org/10.1186/s41239-024-00455-4
Bernaards, C. A., & Jennrich, R. I. (2005). Gradient projection algorithms and software for arbitrary rotation criteria in factor analysis. Educational and Psychological Measurement, 65(5), 676–696. https://doi.org/10.1177/0013164404272507
Bonsu, E. M., & Baffour-Koduah, D. (2023). From the consumers’ side: Determining students’ perception and intention to use ChatGPT in Ghanaian higher education. Journal of Education, Society & Multiculturalism, 4(1), 1–29. https://doi.org/10.2478/jesm-2023-0001
Chan, C. K. Y., & Hu, W. (2023). Students’ voices on generative AI: Perceptions, benefits, and challenges in higher education. International Journal of Educational Technology in Higher Education, 20(1). https://doi.org/10.1186/s41239-023-00411-8
Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2023). Chatting and cheating: Ensuring academic integrity in the era of ChatGPT. Innovations in Education and Teaching International, 61(2), 228–239. https://doi.org/10.1080/14703297.2023.2190148
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008
Farrokhnia, M., Banihashem, S. K., Noroozi, O., & Wals, A. (2023). A SWOT analysis of ChatGPT: Implications for educational practice and research. Innovations in Education and Teaching International, 61(3), 460–474. https://doi.org/10.1080/14703297.2023.2195846
Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41(4), 1149–1160. https://doi.org/10.3758/BRM.41.4.1149
Fauzi, F., Tuhuteru, L., Sampe, F., Ausat, A. M. A., & Hatta, H. R. (2023). Analysing the role of ChatGPT in improving student productivity in higher education. Journal on Education, 5(4), 14886–14891. https://doi.org/10.31004/joe.v5i4.2563
Firat, M. (2023). What ChatGPT means for universities: Perceptions of scholars and students. Journal of Applied Learning and Teaching, 6(1), 57–63. https://doi.org/10.37074/jalt.2023.6.1.22
Gasser, U., Cortesi, S., Malik, M. M., & Lee, A. (2012). Youth and digital media: From credibility to information quality. SSRN. https://doi.org/10.2139/ssrn.2005272
Grájeda, A., Burgos, J., Córdova, P., & Sanjinés, A. (2023). Assessing student-perceived impact of using artificial intelligence tools: Construction of a synthetic index of application in higher education. Cogent Education, 11(1). https://doi.org/10.1080/2331186X.2023.2287917
Hessisches Kultusministerium. (2023). Handreichung “Künstliche Intelligenz (KI) in Schule und Unterricht” [Handout “artificial intelligence (AI) in schools and teaching”]. Digitale Schule Hessen. https://digitale-schule.hessen.de/unterricht-und-paedagogik/handreichung-kuenstliche-intelligenz-ki-in-schule-und-unterricht
Hu, L., & Bentler, P. M. (1998). Fit indices in covariance structure modeling: Sensitivity to underparameterized model misspecification. Psychological Methods, 3(4), 424–453. https://doi.org/10.1037/1082-989X.3.4.424
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., …, & Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for education. Learning and Individual Differences, 103. https://doi.org/10.1016/j.lindif.2023.102274
Koltay, T. (2011). The media and the literacies: Media literacy, information literacy, digital literacy. Media, Culture & Society, 33(2), 211–221. https://doi.org/10.1177/0163443710393382
Kooli, C. (2023). Chatbots in education and research: A critical examination of ethical implications and solutions. Sustainability, 15(7), Article 5614. https://doi.org/10.3390/su15075614
Küchemann, S., Steinert, S., Kuhn, J., Avila, K., & Ruzika, S. (2024). Large language models—Valuable tools that require a sensitive integration into teaching and learning physics. The Physics Teacher, 62(5), 400–402. https://doi.org/10.1119/5.0212374
Laupichler, M. C., Aster, A., Haverkamp, N., & Raupach, T. (2023). Development of the “scale for the assessment of non-experts’ AI literacy” – An exploratory factor analysis. Computers in Human Behavior Reports, 12, Article 100338. https://doi.org/10.1016/j.chbr.2023.100338
Lim, W. M., Gunasekara, A., Pallant, J. L., Pallant, J. I., & Pechenkina, E. (2023). Generative AI and the future of education: Ragnarök or reformation? A paradoxical perspective from management educators. The International Journal of Management Education, 21(2). https://doi.org/10.1016/j.ijme.2023.100790
Liu, S.-H., Liao, H.-L., & Pratt, J.-A. (2008). Impact of media richness and flow on e-learning technology acceptance. Computers and Education, 52(3), 599–607. https://doi.org/10.1016/j.compedu.2008.11.002
Maurer, M., Schemer, C., Zlatkin-Troitschanskaia, O., & Jitomirski, J. (2019). Positive and negative media effects on university students’ learning: Preliminary findings and a research program. In O. Zlatkin-Troitschanskaia (Ed.), Frontiers and advances in positive learning in the age of information (PLATO) (pp. 109–119). Springer. https://doi.org/10.1007/978-3-030-26578-6_8
Mills, A., Bali, M., & Eaton, L. (2023). How do we respond to generative AI in education? Open educational practices give us a framework for an ongoing process. Journal of Applied Learning and Teaching, 6(1), 16–30. https://doi.org/10.37074/jalt.2023.6.1.34
Mogali, S. R. (2023). Initial impressions of ChatGPT for anatomy education. Anatomical Sciences Education, 17(2), 444–447. https://doi.org/10.1002/ase.2261
Molerov, D., Zlatkin-Troitschanskaia, O., Nagel, M.-T., Brückner, S., Schmidt, S., & Shavelson, R. J. (2020). Assessing university students’ critical online reasoning ability: A conceptual and assessment framework with preliminary evidence. Frontiers in Education, 5. https://doi.org/10.3389/feduc.2020.577843
Murray, M. C., & Pérez, J. (n.d.). Unraveling the digital literacy paradox: How higher education fails at the fourth literacy. Issues in Informing Science and Information Technology, 11, 85–100. https://doi.org/10.28945/1982
Nagel, M.-T., Schäfer, S., Zlatkin-Troitschanskaia, O., Schemer, C., Maurer, M., Molerov, D., Schmidt, S., & Brückner, S. (2020). How do university students’ web search behavior, website characteristics, and the interaction of both influence students’ critical online reasoning? Frontiers in Education, 5. https://doi.org/10.3389/feduc.2020.565062
National Research Council. (2012). Education for life and work: Developing transferable knowledge and skills in the 21^st century. The National Academies Press. https://doi.org/10.17226/13398
Neumann, M., Rauschenberger, M., & Schön, E.-M. (2023). “We need to talk about ChatGPT”: The future of AI and higher education. In Proceedings of the IEEE/ACM 5^th International Workshop on Software Engineering Education for the Next Generation (pp. 29–32). IEEE. https://doi.org/10.1109/SEENG59157.2023.00010
Noroozi, O., Soleimani, S., Farrokhnia, M., & Banihashem, S. K. (2024). Generative AI in education: Pedagogical, theoretical, and methodological perspectives. International Journal of Technology in Education, 7(3), 373–385. https://doi.org/10.46328/ijte.845
O’Dea, X., & O’Dea, M. (2023). Is artificial intelligence really the next big thing in learning and teaching in higher education? A conceptual paper. Journal of University Teaching and Learning Practice, 20(5). https://doi.org/10.53761/1.20.5.05
OpenAI. (2023). ChatGPT. https://chat.openai.com
PISA. (2023). Learning in the digital world–PISA. https://www.oecd.org/pisa/innovation/learning-digital-world/
R Core Team. (2023). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/
Rasul, T., Nair, S., Kalendra, D., Robin, M., Santini, F., Ladeira, W., Sun, M., Day, I., Rather, A., & Heathcote, L. (2023). The role of ChatGPT in higher education: Benefits, challenges, and future research directions. Journal of Applied learning & Teaching, 6(1). https://doi.org/10.37074/jalt.2023.6.1.29
Revelle, W. (2023). psych: Procedures for psychological, psychometric, and personality research. Northwestern University, Evanston, Illinois. https://CRAN.R-project.org/package=psych
Rosseel, Y. (2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1–36. https://doi.org/10.18637/jss.v048.i02
Rudolph, J., Tan, S., & Aspland, T. (2023). Fully automated luxury communism or Turing trap? Graduate employability in the generative AI age. Journal of Applied Learning and Teaching, 6(1), 7–15. https://doi.org/10.37074/jalt.2023.6.1.35
Rudolph, J., Tan, S., & Tan, S. (2023a). ChatGPT: Bullshit spewer or the end of traditional assessments in higher education? Journal of Applied Learning and Teaching, 6(1), 342–363. https://doi.org/10.37074/jalt.2023.6.1.9
Rudolph, J., Tan, S., & Tan, S. (2023b). War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education. Journal of Applied Learning and Teaching, 6(1), 364–389. https://doi.org/10.37074/jalt.2023.6.1.23
Sallam, M. (2023). ChatGPT utility in healthcare education, research, and practice: Systematic review on the promising perspectives and valid concerns. Healthcare, 11(6). https://doi.org/10.3390/healthcare11060887
Schepman, A., & Rodway, P. (2020). Initial validation of the general attitudes towards artificial intelligence Scale. Computers in Human Behavior Reports, 1, Article 100014. https://doi.org/10.1016/j.chbr.2020.100014
Schön, E.-M., Neumann, M., Hofmann-Stölting, C., Baeza-Yates, R., & Rauschenberger, M. (2023). How are AI assistants changing higher education? Frontiers in Computer Science, 5. https://doi.org/10.3389/fcomp.2023.1208550
Shen, Y., Heacock, L., Elias, J., Hentel, K. D., Reig, B., Shih, G., & Moy, L. (2023). ChatGPT and other large language models are double-edged swords. Radiology, 307(2). https://doi.org/10.1148/radiol.230163
Sindermann, C., Sha, P., Zhou, M., Wernicke, J., Schmitt, H. S., Li, M., Sariyska, R., Stavrou, M., Becker, B., & Montag, C. (2021). Assessing the attitude towards artificial intelligence: Introduction of a short measure in German, Chinese, and English language. KI – Künstliche Intelligenz, 35(1), 109–118. https://doi.org/10.1007/s13218-020-00689-0
Sok, S., & Heng, K. (2023). ChatGPT for education and research: A review of benefits and risks. SSRN. https://doi.org/10.2139/ssrn.4378735
Sparks, J. R., Katz, I. R., & Beile, P. M. (2016). Assessing digital information literacy in higher education: A review of existing frameworks and assessments with recommendations for next-generation assessment. ETS Research Report Series, 2, 1–33. https://doi.org/10.1002/ets2.12118
Strzelecki, A. (2023). To use or not to use ChatGPT in higher education? A study of students’ acceptance and use of technology. Interactive Learning Environments. https://doi.org/10.1080/10494820.2023.2209881
Sullivan, M., Kelly, A., & McLaughlan, P. (2023). ChatGPT in higher education: Considerations for academic integrity and student learning. Journal of Applied Learning and Teaching, 6(1), 31–40. https://doi.org/10.37074/jalt.2023.6.1.17
U.S. Department of Education, Office of Educational Technology. (2023). Artificial intelligence and the future of teaching and learning. https://www2.ed.gov/documents/ai-report/ai-report.pdf
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425–478. https://doi.org/10.2307/30036540
Venkatesh, V., Thong, J. Y. L., & Xu, X. (2012). Consumer acceptance and use of information technology: Extending the unified theory of acceptance and use of technology. MIS Quarterly, 36(1), 157–178. https://doi.org/10.2307/41410412
Wang, B., Rau, P.-L. P., & Yuan, T. (2022). Measuring user competence in using artificial intelligence: Validity and reliability of artificial intelligence literacy scale. Behaviour & Information Technology, 42(9), 1324–1337. https://doi.org/10.1080/0144929X.2022.2072768
Xames, M. D., & Shefa, J. (2023). ChatGPT for research and publication: Opportunities and challenges. Journal of Applied Learning & Teaching, 6(1). https://doi.org/10.37074/jalt.2023.6.1.20
Zlatkin-Troitschanskaia, O., Hartig, J., Goldhammer, F., & Krstev, J. (2021). Students’ online information use and learning progress in higher education – A critical literature review. Studies in Higher Education, 46(10), 1996–2021. https://doi.org/10.1080/03075079.2021.1953336

Citation

Köhler, C., & Hartig, J. (2024). ChatGPT in higher education: Measurement instruments to assess student knowledge, usage, and attitude. Contemporary Educational Technology, 16(4), ep528. https://doi.org/10.30935/cedtech/15144

Köhler, C., and Hartig, J. (2024). ChatGPT in higher education: Measurement instruments to assess student knowledge, usage, and attitude. Contemporary Educational Technology, 16(4), ep528. https://doi.org/10.30935/cedtech/15144

Köhler C, Hartig J. ChatGPT in higher education: Measurement instruments to assess student knowledge, usage, and attitude. CONT ED TECHNOLOGY. 2024;16(4), ep528. https://doi.org/10.30935/cedtech/15144

Köhler, Carmen, and Johannes Hartig. "ChatGPT in higher education: Measurement instruments to assess student knowledge, usage, and attitude". Contemporary Educational Technology 2024 16 no. 4 (2024): ep528. https://doi.org/10.30935/cedtech/15144

Köhler, Carmen et al. "ChatGPT in higher education: Measurement instruments to assess student knowledge, usage, and attitude". Contemporary Educational Technology, vol. 16, no. 4, 2024, ep528. https://doi.org/10.30935/cedtech/15144

Köhler C, Hartig J. ChatGPT in higher education: Measurement instruments to assess student knowledge, usage, and attitude. CONT ED TECHNOLOGY. 2024;16(4):ep528. https://doi.org/10.30935/cedtech/15144