Applying Kano’s two-factor theory to prioritize learning analytics dashboard features for learning technology designers

Tobias Alexander Bang Tretow-Fish; Md Saifuddin Khalid

doi:10.30935/cedtech/14286

Tobias Alexander Bang Tretow-Fish ¹ ^* , Md Saifuddin Khalid ¹

More Detail

CONT ED TECHNOLOGY, Volume 16, Issue 2, Article No: ep496

https://doi.org/10.30935/cedtech/14286

Submitted: 29 August 2023, Published Online: 26 February 2024

OPEN ACCESS 1309 Views 938 Downloads

Download Full Text (PDF)

Abstract

Existing methods for software requirements elicitation, five-point Likert scales and voting methods for requirements prioritization, and usability and user experience evaluation methods do not enable prioritizing the learning analytics dashboard requirements. Inspired by management and product design field, this research applies Kano’s two-factor theory to prioritize the features of learning analytics dashboards (LADs) of adaptive learning platform (ALP) called Rhapsode^TM learner, based on students’ perceived usefulness to support designers’ decision-making. Comparing usability and user experience methods for evaluating LAD features, this paper contributes with the protocol and a case applying Kano method for evaluating the perceived importance of the dashboards in ALP. The paper applies Kano’s two-factor questionnaire on the 13 LADs features of Rhapsode^TM learner. Responses from 17 students are collected using a questionnaire, which is used to showcase the strength of the two-factor theory through five tabular and graphical techniques. Through these five tabular and graphical techniques, we demonstrate the application and usefulness of the method as designers and management are often carried away by the possibilities of insights instead of actual usefulness. The results revealed a variation in the categorization of LADs depending on the technique employed. As the complexity of the techniques increases, additional factors that indicate data uncertainty are gradually incorporated, clearly highlighting the growing requirement for data. In the case of RhapsodeTM learner platform, results based on the students responses show that 11 of 13 LADs being excluded due to low significance level in categorization (technique 1) and low response rate.

Keywords: adaptive learning platforms, Kano’s two-factor theory, learning analytics dashboard, design methods

References

Ahn, J., Campos, F., Hays, M., & Digiacomo, D. (2019). Designing in context: Reaching beyond usability in learning analytics dashboard design. Journal of Learning Analytics, 6(2), 70-85. https://doi.org/10.18608/jla.2019.62.5
Area9 (2024, January 1). "Rhapsode^TM Learner": Area9 Rhapsode^TM. https://area9lyceum.com/the-platform/
Bauk, S., Šćepanović, S., & Kopp, M. (2014). Estimating students’ satisfaction with web based learning system in blended learning environment. Education Research International, 2014, 731720. https://doi.org/10.1155/2014/731720
Baxter, K., Courage, C., & Caine, K. (2015). Choosing a user experience research activity in understanding your users: A practical guide to user research methods. Morgan Kaufmann. https://doi.org/10.1016/B978-0-12-800232-2.00005-5
Benyon, D. (2019). Designing user experience. Pearson.
Berger, C., Blauth, R., Boger, D., Bolster, C., Burchill, G., DuMouchel, W., Pouliot, F., Richter, R., Rubinoff, A., Shen, D., Timko, M., & Walden, D. (1993). A special issue on Kano’s methods for understanding customer defined quality. Center for Quality of Management Journal, 2(4), 3-28.
Bodily, R., Ikahihifo, T. K., Mackley, B., & Graham, C. R. (2018). The design, development, and implementation of student-facing learning analytics dashboards. Journal of Computing in Higher Education, 30, 572-598. https://doi.org/10.1007/s12528-018-9186-0
Chen, L.-S., & Hsu, J.-Y. (2019). Discover users’ needs in e-learning by Kano analysis and decision trees. In Proceedings of the 2019 IEEE6^th International Conference on Industrial Engineering and Applications (pp. 572-576). https://doi.org/10.1109/IEA.2019.8714873
Chen, W.-K., Chang, J.-R., Chen, L.-S., & Hsu, R.-Y. (2022). Using refined kano model and decision trees to discover learners’ needs for teaching videos. Multimedia Tools and Applications, 81(6), 8317-8347. https://doi.org/10.1007/s11042-021-11744-9
de Kock, E., van Biljon, J., & Pretorius, M. (2009). Usability evaluation methods: Mind the gaps. In Proceedings of the 2009 Annual Research Conference of the South African Institute of Computer Scientists and Information Technologists (pp. 122-131). https://doi.org/10.1145/1632149.1632166
Earnshaw, Y., Tawfik, A. A., & Schmidt, M. (2018). User experience design. In R. E. West (Ed.), Foundations of learning and instructional design technology. EdTech Books.
Few, S. (2006). Information dashboard design, the effective visual communication of data. O’Reilly.
Fong, D. (1996). Using the self-stated importance questionnaire to interpret Kano questionnaire results. Center for Quality Management Journal, 5, 21-24.
Fontaine, G., Cossette, S., Maheu-Cadotte, M.-A., Mailhot, T., Deschênes, M.-F., Mathieu-Dupuis, G., Côté, J., Gagnon, M.-P., & Dubé, V. (2019). Efficacy of adaptive e-learning for health professionals and students: A systematic review and meta-analysis. BMJ Open, 9(8), e025252. https://doi.org/10.1136/bmjopen-2018-025252
Hofstede, G. (2024). Country comparison tool. The Culture Factor Group. https://www.hofstede-insights.com/country-comparison-tool
Hornbæk, K. (2006). Current practice in measuring usability: Challenges to usability studies and research. International Journal of Human-Computer Studies, 64(2), 79-102. https://doi.org/10.1016/j.ijhcs.2005.06.002
Iordache, D. D., & Pribeanu, C. (2009). A comparison of quantitative and qualitative data from a formative usability evaluation of an augmented reality learning scenario. Informatica Enconomica [Economic Informatics], 13(3), 67-74.
Kano, N., Seraku, N., Takahashi, F., & Tsuji, S. (1984). Attractive quality and must-be quality. Journal of the Japanese Society for Quality Control, 41, 39-48.
Khalid, M. S., Mustafa, A., & Haque, I. (2008). Application of Kano’s model for evaluating information quality of university websites. In H. R. Arabnia & A. Marsh (Eds.), Proceedings of the 2008 International Conference on Semantic Web & Web Services (pp. 277-280). CSREA Press.
Kim, J., Jo, I. H., & Park, Y. (2016). Effects of learning analytics dashboard: Analyzing the relations among dashboard utilization, satisfaction, and learning achievement. Asia Pacific Education Review, 17, 13-24. https://doi.org/10.1007/s12564-015-9403-8
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2006). Evaluating training programs: The four levels. Berrett-Koehler.
Lesemann, E., Woletz, N., & Koerber, S. (2007). Combining methods to evaluate mobile usability. In Proceedings of the9^th International Conference on Human Computer Interaction with Mobile Devices and Services (pp. 444-447). ACM. https://doi.org/10.1145/1377999.1378051
Lewis, J. R. (2018). The system usability scale: Past, present, and future. International Journal of Human-Computer Interaction, 34(7), 577-590. https://doi.org/10.1080/10447318.2018.1455307
Löfgren, M. (2005). Kano’s theory of attractive quality and packaging. Quality Management Journal, 2(3), 7-20. https://doi.org/10.1080/10686967.2005.11919257
Löfgren, M., & Witell, L. (2008). Two decades of using Kano’s theory of attractive quality: A literature review. Quality Management Journal, 15(1), 59-75. https://doi.org/10.1080/10686967.2008.11918056
Madzík, P., Budaj, P., Mikuláš, D., & Zimon D. (2009). Application of the Kano model for a better understanding of customer requirements in higher education–A pilot study. Administrative Sciences, 9(1), 11. https://doi.org/10.3390/admsci9010011
Matzler, K., Hinterhuber, H. H., Bailom, F., & Sauerwein, E. (1996). How to delight your customer. Journal of Product & Brand Management, 5(2), 6-18. https://doi.org/10.1108/10610429610119469
Nakamura, W. T., de Oliveira, E. H. T., & Conte, T. (2017). Usability and user experience evaluation of learning management systems. In Proceedings of the 19^th International Conference on Enterprise Information Systems (pp. 97-108). SciTePress. https://doi.org/10.5220/0006363100970108
Park, Y., & Jo, I. (2015). Development of the learning analytics dashboard to support students’ learning performance. Journal of Universal Computer Science, 21(1), 110-133.
Roberts, L. D., Howell, J. A., & Seaman, K. (2017). Give me a customizable dashboard: Personalized learning analytics dashboards in higher education. Technology, Knowledge and Learning, 22, 317-333. https://doi.org/10.1007/s10758-017-9316-1
Salas, J., Chang, A., Montalvo, L., Nunez, A., Vilcapoma, M., Moquillaza, A., Murillo, B., & Paz, F. (2019). Guidelines to evaluate the usability and user experience of learning support platforms: A systematic review. In P. Ruiz, & V. Agredo-Delgado (Eds.), Human-computer interaction (pp. 238-254). Springer. https://doi.org/10.1007/978-3-030-37386-3_18
Sauro, J., & Lewis J. R. (2016). Standardized usability questionnaires in quantifying the user experience: Practical statistics for user research. Morgan Kaufmann. https://doi.org/10.1016/B978-0-12-802308-2.00008-4
Schwendimann, B. A., Rodríguez-Triana, M. J., Vozniuk, A., Prieto, L. P., Boroujeni, M. S., Holzer, A., Gillet, D., & Dillenbourg, P. (2017). Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies, 10(1), 30-41. https://doi.org/10.1109/TLT.2016.2599522
Sedrakyan, G., Mannens, E., & Verbert, K. (2019). Guiding the choice of learning dashboard visualizations: Linking dashboard design and data visualization concepts. Journal of Computer Languages, 50, 19-38. https://doi.org/10.1016/j.jvlc.2018.11.002
Tretow-Fish, T. A. B., & Khalid, M. S. (2023). Methods for evaluating Learning Analytics and Learning Analytics Dashboards in Adaptive Learning Platforms: A systematic review. Electronic Journal of E-Learning, 21(5), 430-449. https://doi.org/10.34190/ejel.21.5.3088
Verbert, K., Ochoa, X., De Croon, R., Dourado, R. A., & De Laet, T. (2020). Learning analytics dashboards: The past, the present and the future. In Proceedings of the 10^th International Conference on Learning Analytics & Knowledge (pp. 35-40). ACM. https://doi.org/10.1145/3375462.3375504
Witell, L., Löfgren, M., & Dahlgaard, J. J. (2013). Theory of attractive quality and the Kano methodology–The past, the present, and the future. Total Quality Management and Business Excellence, 24(11-12), 1241-1252. https://doi.org/10.1080/14783363.2013.791117
Xie, H., Chu, H.-C., Hwang, G.-J., & Wang, C.-C. (2019). Trends and development in technology-enhanced adaptive/personalized learning: A systematic review of journal publications from 2007 to 2017. Computers & Education, 140, 103599. https://doi.org/10.1016/j.compedu.2019.103599
Yoo, M., & Jin, S. (2020). Development and evaluation of learning analytics dashboards to support online discussion activities. Educational Technology & Society, 23(2), 1-18.

Citation

Tretow-Fish, T. A. B., & Khalid, M. S. (2024). Applying Kano’s two-factor theory to prioritize learning analytics dashboard features for learning technology designers. Contemporary Educational Technology, 16(2), ep496. https://doi.org/10.30935/cedtech/14286

Tretow-Fish, T. A. B., and Khalid, M. S. (2024). Applying Kano’s two-factor theory to prioritize learning analytics dashboard features for learning technology designers. Contemporary Educational Technology, 16(2), ep496. https://doi.org/10.30935/cedtech/14286

Tretow-Fish TAB, Khalid MS. Applying Kano’s two-factor theory to prioritize learning analytics dashboard features for learning technology designers. CONT ED TECHNOLOGY. 2024;16(2), ep496. https://doi.org/10.30935/cedtech/14286

Tretow-Fish, Tobias Alexander Bang, and Md Saifuddin Khalid. "Applying Kano’s two-factor theory to prioritize learning analytics dashboard features for learning technology designers". Contemporary Educational Technology 2024 16 no. 2 (2024): ep496. https://doi.org/10.30935/cedtech/14286

Tretow-Fish, Tobias Alexander Bang et al. "Applying Kano’s two-factor theory to prioritize learning analytics dashboard features for learning technology designers". Contemporary Educational Technology, vol. 16, no. 2, 2024, ep496. https://doi.org/10.30935/cedtech/14286

Tretow-Fish TAB, Khalid MS. Applying Kano’s two-factor theory to prioritize learning analytics dashboard features for learning technology designers. CONT ED TECHNOLOGY. 2024;16(2):ep496. https://doi.org/10.30935/cedtech/14286