Identification of Cognitive Learning Complexity of Assessment Questions Using Multi-class Text Classification

Syaamantak Das; Shyamal Kumar Das Mandal; Anupam Basu

doi:10.30935/cedtech/8341

Syaamantak Das ¹ ^* , Shyamal Kumar Das Mandal ¹, Anupam Basu ² ³

More Detail

CONT ED TECHNOLOGY, Volume 12, Issue 2, Article No: ep275

https://doi.org/10.30935/cedtech/8341

OPEN ACCESS 3441 Views 2745 Downloads

Download Full Text (PDF)

Abstract

Cognitive learning complexity identification of assessment questions is an essential task in the domain of education, as it helps both the teacher and the learner to discover the thinking process required to answer a given question. Bloom’s Taxonomy cognitive levels are considered as a benchmark standard for the classification of cognitive thinking (learning complexity) in an educational environment. However, it was observed that some of the action verbs of Bloom’s Taxonomy are overlapping in multiple levels of the hierarchy, causing ambiguity about the real sense of cognition required. The paper describes two methodologies to automatically identify the cognitive learning complexity of given questions. The first methodology uses labelled Latent Dirichlet Allocation (LDA) as a machine learning approach. The second methodology uses the BERT framework for multi-class text classification for deep learning. The experiments were performed on an ensemble of 3000+ educational questions, which were based on previously published datasets along with the TREC question corpus and AI2 Biology How/Why question corpus datasets. The labelled LDA reached an accuracy of 83% while BERT based approach reached 89% accuracy. An analysis of both the results is shown, evaluating the significant factors responsible for determining cognitive knowledge.

Keywords: multi-class text classification, labelled LDA, pre-trained BERT (Bidirectional Encoder Representations from Transformers), question classification

References

Agrawal, R., Gollapudi, S., Kannan, A., & Kenthapadi, K. (2014). Study navigator: An algorithmically generated aid for learning from electronic text-books. Journal of Educational Data Mining, 6(1), 53-75.
Andre, T. (1979). Does answering higher-level questions while reading facilitate productive learning? Review of Educational Research, 49(2), 280-318. https://doi.org/10.3102/00346543049002280
Bhatia, P., Celikkaya, B., Khalilia, M., & Senthivel, S. (2019). Comprehend medical: a named entity recognition and relationship extraction web service. arXiv preprint arXiv: 1910.07419. https://doi.org/10.1109/ICMLA.2019.00297
Bicalho, P., Pita, M., Pedrosa, G., Lacerda, A., & Pappa, G. L. (2017). A general framework to expand short text for topic modeling. Information Sciences, 393, 66-81. https://doi.org/10.1016/j.ins.2017.02.007
Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). Latent dirichlet allocation. Journal of machine Learning research, 3(Jan), 993-1022.
Bloom, B. S., et al. (1956). Taxonomy of educational objectives. vol. 1: Cognitive domain. New York: McKay, 20-24.
Dalton, J., & Smith, D. (1989). Extending children’s special abilities: strategies for primary classrooms. Office of Schools Administration, Ministry of Education, Victoria.
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018). Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. https://doi.org/10.18653/v1/N19-1423
Hamilton, R. (1992). Application adjunct post-questions and conceptual problem solving. Contemporary Educational Psychology, 17(1), 89-97. https://doi.org/10.1016/0361-476X(92)90050-9
Hamilton, R. J. (1985). A framework for the evaluation of the effectiveness of adjunct questions and objectives. Review of Educational Research, 55(1), 47-85. https://doi.org/10.3102/00346543055001047
Howard, J., & Ruder, S. (2018). Universal language model ne-tuning for text classification. arXiv preprint arXiv:1801.06146. https://doi.org/10.18653/v1/P18-1031
Jain, M., Beniwal, R., Ghosh, A., Grover, T., & Tyagi, U. (2019). Classifying question papers with bloom’s taxonomy using machine learning techniques. In International conference on advances in computing and data sciences (pp. 399-408). https://doi.org/10.1007/978-981-13-9942-8_38
Jansen, P., Surdeanu, M., & Clark, P. (2014). Discourse complements lexical semantics for non-factoid answer reranking. In Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 1: Long papers) (pp. 977-986). https://doi.org/10.3115/v1/P14-1092
Jones, K. O., Harland, J., Reid, J. M., & Bartlett, R. (2009). Relationship between examination questions and bloom’s taxonomy. In 2009 39th IEEE frontiers in education conference (pp. 1-6). https://doi.org/10.1109/FIE.2009.5350598
Krathwohl, D. R. (2002). A revision of bloom’s taxonomy: An overview. Theory into practice, 41(4), 212-218. https://doi.org/10.1207/s15430421tip4104_2
Krathwohl, D. R., & Anderson, L. W. (2010). Merlin c. wittrock and the revision of bloom’s taxonomy. Educational psychologist, 45(1), 64-65. https://doi.org/10.1080/00461520903433562
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 159-174. https://doi.org/10.2307/2529310
Lee, Y.-J., Kim, M., Jin, Q., Yoon, H.-G., & Matsubara, K. (2017). Revised blooms taxonomy the swiss army knife in curriculum research. In East-asian primary science curricula (pp. 11-16). Springer. https://doi.org/10.1007/978-981-10-2690-4
Li, X., & Roth, D. (2002). Learning question classifiers. In Proceedings of the 19th international conference on computational linguistics-volume 1 (pp. 1-7). https://doi.org/10.3115/1072228.1072378
Long, G., Chen, L., Zhu, X., & Zhang, C. (2012). Tcsst: transfer classification of short & sparse text using external data. In Proceedings of the 21st ACM international conference on information and knowledge management (pp. 764-772). https://doi.org/10.1145/2396761.2396859
Luo, L., & Wang, Y. (2019). Emotionx-hsu: Adopting pre-trained bert for emotion classification. arXiv preprint arXiv:1907.09669.
Massey, L. (2011). Autonomous and adaptive identification of topics in unstructured text. In International conference on knowledge-based and intelligent information and engineering systems (pp. 1-10). https://doi.org/10.1007/978-3-642-23863-5_1
Pan, S. J., & Yang, Q. (2009). A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10), 1345-1359. https://doi.org/10.1109/TKDE.2009.191
Peverly, S. T., & Wood, R. (2001). The effects of adjunct questions and feed-back on improving the reading comprehension skills of learning-disabled adolescents. Contemporary Educational Psychology, 26(1), 25-43. https://doi.org/10.1006/ceps.1999.1025
Phan, X.-H., Nguyen, L.-M., & Horiguchi, S. (2008). Learning to classify short and sparse text & web with hidden topics from large-scale data collections. In Proceedings of the 17th international conference on world wide web (pp. 91-100). https://doi.org/10.1145/1367497.1367510
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1(8).
Ramage, D., Hall, D., Nallapati, R., & Manning, C. D. (2009). Labeled lda: A supervised topic model for credit attribution in multi-labeled corpora. In Proceedings of the 2009 conference on empirical methods in natural language processing, 1(1), 248-256. https://doi.org/10.3115/1699510.1699543
Redfield, D. L., & Rousseau, E. W. (1981). A meta-analysis of experimental research on teacher questioning behavior. Review of educational research, 51(2), 237-245. https://doi.org/10.3102/00346543051002237
Rothkopf, E. Z. (1970). The concept of mathemagenic activities. Review of educational research, 40(3), 325-336. https://doi.org/10.3102/00346543040003325
Stanny, C. (2016). Reevaluating blooms taxonomy: What measurable verbs can and cannot say about student learning. Education Sciences, 6(4), 37. https://doi.org/10.3390/educsci6040037
Swart, A. J., & Daneti, M. (2019). Analyzing learning outcomes for electronic fundamentals using blooms taxonomy. In 2019 IEEE global engineering education conference (educon) (pp. 39-44). https://doi.org/10.1109/EDUCON.2019.8725137
Uys, J., Du Preez, N., & Uys, E. (2008). Leveraging unstructured information using topic modelling. In Picmet’08-2008 portland international conference on management of engineering & technology (pp. 955-961). https://doi.org/10.1109/PICMET.2008.4599703
Wang, P., Xu, B., Xu, J., Tian, G., Liu, C.-L., & Hao, H. (2016). Semantic ex-pansion using word embedding clustering and convolutional neural network for improving short text classification. Neurocomputing, 174, 806-814. https://doi.org/10.1016/j.neucom.2015.09.096
Yahya, A. A., Toukal, Z., & Osman, A. (2012). Blooms taxonomy-based classi-cation for item bank questions using support vector machines. In Modern advances in intelligent systems and tools (pp. 135-140). Springer. https://doi.org/10.1007/978-3-642-30732-4_17
Zarei, F., & Nik-Bakht, M. (2019). Automated detection of urban flooding from news. In Proceedings of the 36th international symposium on automation and robotics in construction (pp. 515-521). https://doi.org/10.22260/ISARC2019/0069
Zhang, H., & Zhong, G. (2016). Improving short text classification by learning vector representations of both words and hidden topics. Knowledge-Based Systems, 102, 76-86. https://doi.org/10.1016/j.knosys.2016.03.027

Citation

Das, S., Das Mandal, S. K., & Basu, A. (2020). Identification of Cognitive Learning Complexity of Assessment Questions Using Multi-class Text Classification. Contemporary Educational Technology, 12(2), ep275. https://doi.org/10.30935/cedtech/8341

Das, S., Das Mandal, S. K., and Basu, A. (2020). Identification of Cognitive Learning Complexity of Assessment Questions Using Multi-class Text Classification. Contemporary Educational Technology, 12(2), ep275. https://doi.org/10.30935/cedtech/8341

Das S, Das Mandal SK, Basu A. Identification of Cognitive Learning Complexity of Assessment Questions Using Multi-class Text Classification. CONT ED TECHNOLOGY. 2020;12(2), ep275. https://doi.org/10.30935/cedtech/8341

Das, Syaamantak, Shyamal Kumar Das Mandal, and Anupam Basu. "Identification of Cognitive Learning Complexity of Assessment Questions Using Multi-class Text Classification". Contemporary Educational Technology 2020 12 no. 2 (2020): ep275. https://doi.org/10.30935/cedtech/8341

Das, Syaamantak et al. "Identification of Cognitive Learning Complexity of Assessment Questions Using Multi-class Text Classification". Contemporary Educational Technology, vol. 12, no. 2, 2020, ep275. https://doi.org/10.30935/cedtech/8341

Das S, Das Mandal SK, Basu A. Identification of Cognitive Learning Complexity of Assessment Questions Using Multi-class Text Classification. CONT ED TECHNOLOGY. 2020;12(2):ep275. https://doi.org/10.30935/cedtech/8341