Assumption Validation Process for the Assessment of Technology-Enhanced Learning

Denise Pauline-Graf; Susan E. Mandel; Heather W. Allen; Lynne E. Devnew

doi:10.30935/cedtech/11071

Denise Pauline-Graf ¹ ² ^* , Susan E. Mandel ³, Heather W. Allen ⁴, Lynne E. Devnew ⁵

More Detail

CONT ED TECHNOLOGY, Volume 13, Issue 4, Article No: ep316

https://doi.org/10.30935/cedtech/11071

Submitted: 06 November 2020, Published Online: 13 July 2021

OPEN ACCESS 2469 Views 1188 Downloads

Download Full Text (PDF)

Abstract

Technology-enhanced learning (TEL) is a broad term to denote the incorporation of digital technology to mediate activities that support education. Educational researchers need to validate critical assumptions about any new system that involves TEL early and throughout its development course to make quick, informed, de-risked decisions about the progress of TEL. We introduce an assumption validation process for educational researchers to consider utilizing when evolving ideas or prototypes of TEL. We present a preliminary study conducted in Russia of a knowledge sharing (KS) training program using digital games to illustrate the proposed four-stage process. The first stage is listing the assumptions that apply to initiatives of TEL. The second is identifying the methods most useful for testing those assumptions. The third is executing tests on each of those assumptions. The last stage is determining assumption validity. In the illustrative study, a single pilot trial was considered the appropriate approach to validate the assumptions selected in the first stage. We found that determining assumption validity requires testing many of the assumptions individually and in aggregate. Educational researchers can use this assumption validation process to assess the potential of TEL in a variety of settings before investing resources into its further development.

Keywords: instructional digital games, assumption validation, knowledge sharing, pilot trial, self-efficacy, technology-enhanced learning

References

Al-Azawi, R., Al-Faliti, F., & Al-Blushi, M. (2016). Educational gamification vs. game based learning: Comparative study. International Journal of Innovation, Management and Technology, 7(4), 132-136. https://doi.org/10.18178/ijimt.2016.7.4.659
Apiola, M., Lokkila, E., & Laakso, M.-J. (2019). Digital learning approaches in an intermediate-level computer science course. International Journal of Information & Learning Technology, 36(5), 467-484. https://doi.org/10.1108/IJILT-06-2018-0079
Arnold, D., Burns, K., Adhikari, N., Kho, M., Meade, M., & Cooke, D. (2009). The design and interpretation of pilot trials in clinical research in critical care. Critical Care Medicine, 37(1 Suppl), 69-74. https://doi.org/10.1097/CCM.0b013e3181920e33
Baack, D., Dow, D., Parente, R., & Bacon, D. (2015). Confirmation bias in individual-level perceptions of distance: An experimental investigation. Journal of International Business Studies, 46(8), 938-959. https://doi.org/10.1057/jibs.2015.19
Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84(2), 191-215. https://doi.org/10.1037/0033-295X.84.2.191
Bartolomé, A., Castañeda, L., & Adell, J. (2018). Personalisation in educational technology: The absence of underlying pedagogies. International Journal of Educational Technology in Higher Education, 15(14), 1-17. https://doi.org/10.1186/s41239-018-0095-0
Bayne, S. (2015). What’s the matter with ‘technology enhanced learning’? Learning, Media and Technology, 40(1), 5-20. https://doi.org/10.1080/17439884.2014.915851
Beckstead, J. (2013). On measurements and their quality: Paper 2: Random measurement error and the power of statistical tests. International Journal of Nursing Studies, 50(10), 1416-1422. https://doi.org/10.1016/j.ijnurstu.2013.04.010
Bennett, S., & Oliver, M. (2011). Talking back to theory: The missed opportunities in learning technology research. Research in Learning Technology, 19(3), 179-189. https://doi.org/10.3402/rlt.v19i3.17108
Black, J., Kim, K., Rhee, S., Wang, K., & Sakchutchawan, S. (2019). Self-efficacy and emotional intelligence: Influencing team cohesion to enhance team performance. Team Performance Management: An International Journal, 25(1/2), 100-119. https://doi.org/10.1108/TPM-01-2018-0005
Chen, M.-L., & Lin, C.-P. (2013). Assessing the effects of cultural intelligence on team knowledge sharing from a socio-cognitive perspective. Human Resource Management, 52(5), 675-695. https://doi.org/10.1002/hrm.21558
Cook, D., & Ellaway, R. (2015). Evaluating technology-enhanced learning: a comprehensive framework. Medical Teacher, 37(10), 961-970. https://doi.org/10.3109/0142159X.2015.1009024
Courtright, S., Colbert, A., & Choi, D. (2014). Fired up or burned out? How developmental challenge differentially impacts leader behavior. Journal of Applied Psychology, 99(4), 681-696. https://doi.org/10.1037/a0035790
Cózar-Gutiérrez, R., & Sáez-López, J. (2016). Game-based learning and gamification in initial teaching training in social sciences: an experiment with MinecraftEdu. International journal of Educational technology in Higher Education, 13(1), 2. https://doi.org/10.1186/s41239-016-0003-4
Crook, C., & Gu, X. (2019). How new technology is addressed by researchers in educational studies: Approaches from high-performing universities in China and the UK. British Journal of Educational Technology, 50(3), 1173-1188. https://doi.org/10.1111/bjet.12750
Donaldson, S., & Grant-Vallone, E. (2002). Understanding self-report bias in organizational behavior research. Journal of Business and Psychology, 17(2), 245-260. https://doi.org/10.1023/A:1019637632584
Ellaway, R. (2014). Virtual patients as activities: Exploring the research implications of an activity theoretical stance. Perspectives on Medical Education 3(4), 266-277. https://doi.org/10.1007/s40037-014-0134-z
Ellaway, R., Pusic, M., Gailbraith, R., & Cameron, T. (2014a). Developing the role of big data and analytics in health professional education. Medical Teaching, 36(3), 216-222. https://doi.org/10.3109/0142159X.2014.874553
Ellaway, R., Pusic, M., Yavner, S., & Kalet, A. (2014b, March 9). Context matters. Emergent variability in an effectiveness trial of online teaching modules. Medical Education, 48(4), 386-396. https://doi.org/10.1111/medu.12389
Filippova, T. (2015). Priority fields of e-learning development in Russia. Procedia – Social and Behavioral Sciences, 206, 348-353. https://doi.org/10.1016/j.sbspro.2015.10.063
Frye, W., & Hemmer, P. (2012). Program evaluation models and related theories: AME guide no. 67. Medical Teacher, 34(5) e288-e299. https://doi.org/10.3109/0142159X.2012.668637
Giannakas, F., Kambourakis, G., Papaslouros, A., & Gritzalis, S. (2018). A critical review of 13 years of mobile game-based learning. Educational Technology Research and Development, 66(2), 341-384.
Goodyear, P., & Retalis, S. (2010). Learning, technology and design. In P. Goodyear & S. Retalis (Eds.), Technology-enhanced learning: Design patterns and pattern languages (pp. 1-28). Sense Publishers.
Gresalfi, M., & Barnes, J. (2016). Designing feedback in an immersive videogame: supporting mathematical engagement. Educational Technology Research and Development, 64(1), 65-86. https://doi.org/10.1007/s11423-015-9411-8
Hagemeier, N., Hess, R., Hagen, K., & Sorah, E. (2014). Impact of an interprofessional communication course on nursing, medical, and pharmacy students’ communication skill self-efficacy beliefs. American Journal of Pharmaceutical Education, 78(10), 186-196. https://doi.org/10.5688/ajpe7810186
Harvey, J. (1998). Evaluation cookbook. Edinburgh: Learning Technology Dissemination Initiative. Heriot-Watt University, Edinburgh.
Hussein, M., Ow, S., Cheong, L., Thing, M., & Ebrahim, N. (2019). Effects of digital game-based learning on elementary science learning: A systematic review. IEEE Access, 7, 62465-62478. https://doi.org/10.1109/ACCESS.2019.2916324
Hwa, S. (2018). Pedagogical change in mathematics learning: Harnessing the power of digital game-based learning. Journal of Educational Technology & Society, 21(4), 259-276.
Khan, A., Ahmad, F., & Malik, M. (2017). Use of digital game based learning and gamification in secondary school science: The effect on student engagement, learning and gender difference. Education and Information Technologies, 22(6), 2767-2804. https://doi.org/10.1007/s10639-017-9622-1
Kiili, K. (2005). Digital game-based learning: Towards an experiential gaming model. Internet and Higher Education, 8(1), 13-24. https://doi.org/10.1016/j.iheduc.2004.12.001
Kirkwood, A., & Price, L. (2013). Examining some assumptions and limitations of research on the effects of emerging technologies for teaching and learning in higher education. British Journal of Educational Technology, 44(4), 536-543. https://doi.org/10.111/bjet.12049
Kirkwood, A., & Price, L. (2014). Technology-enhanced learning and teaching in higher education: what is ‘enhanced’ and how do we know? A critical literature review. Learning, Media and Technology, 39(1), 6-36. https://doi.org/10.1080/17439884.2013.770404
Koh, C., Tan, H., Tan, K., Fang, L., & Fong, F., Kan, D., Lye, S., & Wee, M. (2010). Investigating the effect of 3D simulation-based learning on the motivation and performance of engineering students. Journal of Engineering Education, 99(3), 237-251. https://doi.org/10.1002/j.2168-9830.2010.tb01059.x
Kurvinen, E., Kaila, E., Laakso, M.-J., & Salakoski, T. (2020). Long term effects on technology enhanced learning: The use of weekly digital lessons in Mathematics. Informatics in Education, 19(1), 51-75. https://doi.org/10.15388/infedu.2020.04
Larionova, V., Brown, K., Bystrova, T., & Sinitsyn, E. (2018). Russian perspectives of online learning technologies in higher education: An empirical study of a MOOC. Research in Comparative & International Education, 13(1), 70-91. https://doi.org/10.1177/1745499918763420
Lee, Y-H. (2015). Does digital game interactivity always promote self-efficacy? Cyberpsychology, Behavior, and Social Networking, 18(11), 669-673. https://doi.org/10.1089/cyber.2015.0165
Li, Z., Cheng, Y., & Liu, C. (2013). A constructionism framework for designing game-like learning systems: Its effect on different learners. British Journal of Educational Technology, 44(2), 208-224. https://doi.org/10.1111/j.1467-8535.2012.01305.x
Liu, Z.-Y., Shaikh, Z., & Gazizova, F. (2020). Using the concept of game-based learning in education. International Journal of Emerging Technologies in Learning, 15(14), 53-64. https://doi.org/10.39991/ijet.v15i14.14675
Niederhauser, D., Howard, S., Voogt, J., Agyei, D., Laferriere, T., Tondeaur, J., & Cox, M. (2018). Sustainability and scalability in educational technology initiatives: Research-informed practice. Technology, Knowledge and Learning, 23(3), 507-523. https://doi.org/10.1007/s10758-018-9382-z
Ojedokun, O., & Idemudia, E. (2014). Examining the roles of gender and personal dispositions in attitudes toward knowledge sharing of senior administrators. Gender & Behaviour, 12(3), 5857-5867.
Oliver, E. (2018). Digital game-based learning and technology-enhanced learning for theological education. Verbum et Ecclesia, 39, 1-8. https://doi.org/10.4102/ve.v39i1.1900
Oliver, M. (2011). Technological determinism in educational technology research: some alternative ways of thinking about the relationship between learning and technology. Journal of Computer Assisted Learning, 27(5), 373-384. https://doi.org/10.111/j1365-2729.2011.00406.x
Olowodunoye, S. (2015). Knowledge sharing behavior: The role of self-efficacy, organizational justice and organizational tenure. European Scientific Journal, 11(17), 254-264.
Osman, S., Sauid, M., & Azizan, N. (2015). Knowledge sharing patterns among undergraduate students in Universiti Teknologi MARA (UiTM) Johor, Malaysia. International Journal of Management, Accounting and Economics, 2(3), 167-178.
Passey, D. (2019). Technology-enhanced learning: Rethinking the term, the concept, and its theoretical background. British Journal of Educational Technology, 50(3), 972-986. https://doi.org/10.1111/bjet.12783
Pauline-Graf, D., & Mandel, S. E. (2019). Defining preliminary research for digital game-based learning evaluation: Best practices. International Journal of Educational Methodology, 5(4), 623-635. https://doi.org/10.12973/ijem.5.4.623
Persico, D., Pozzi, F., & Goodyear, P. (2018). Teachers as designers of TEL programs. British Journal of Educational Technology, 49(6). https://doi.org/10.1111/bjet.12706
Pickering, J., Lazarus, M., & Hallam, J. (2019). A practitioner’s guide to performing a holistic evaluation of technology-enhanced learning in medical education. Medical Science Educator, 29(4), 1095-1102. https://doi.org/10.1007/s40670-019-00781-7
Prosser, M., Trigwell, K., & Taylor, P. (1994). A phenomenographic study of academics’ conceptions of science learning and teaching. Learning and Instruction, 4(3), 217-232. https://doi.org/10.1016/0959-4752(94)90024-8
Qian, M., & Clark, K. (2016). Game-based learning and 21^st century skills: A review of recent research. Computers in Human Behavior, 63, 50-58. https://doi.org/10.1016/j.chb.2016.05.023
Samuelowicz, K., & Bain, J. (1992). Conceptions of teaching held by academic teachers. Higher Education, 24(1), 93-111. https://doi.org/10.1007/BF00138620
Sanders, J. (2012). Technology-enhanced learning. Education for Primary Care, 23(3), 228-229. https://doi.org/10.1080/14739879.2012.11494107
Shaari, R., Rahman, S., & Rajab, A. (2014). Self-efficacy as a determined factor for knowledge sharing awareness. International Journal of Trade Economics and Finance, 5(1), 39-42. https://doi.org/10.7763/IJTEF.2014.V5.337
Shahriarpour, N., & Kafi, Z. (2014). On the effect of playing digital games on Iranian intermediate EFL learners’ motivation toward learning English vocabularies. Procedia – Social and Behavioral Sciences, 98, 1738-1734. https://doi.org/10.1016/j.sbspro.2014.03.601
Shao, Z., Wang, T., & Feng, Y. (2015). Impact of organizational culture and computer self-efficacy on knowledge sharing. Industrial Management & Data Systems, 115(4), 590-611. https://doi.org/10.1108/IMDS-12-2014-0377
Swinkels, M., Koopman, M., & Beijaard, D. (2013). Student teachers’ development of learning focused conceptions. Teaching and Teacher Education, 34, 26-37. https://doi.org/10.1016/j.tate.2013.03.003
Tang, S., Hanneghan, M., & Carter, C. (2013). A platform independent game technology model for model driven serious game development. The Electronic Journal of e-Learning, 11(1), 61-79.
Tasa, K., Taggar, S., & Seijts, G. (2007). The development of collective self-efficacy in teams: A multilevel and longitudinal perspective. Journal of Applied Psychology, 92(1), 17-27. https://doi.org/10.1037/0021-9010.92.1.17
Van Driel, J., & Berry, A. (2012). Teacher professional development focusing on pedagogical content knowledge. Educational Researcher, 41(1), 26-28. https://doi.org/10.3102/0013189X11431010
Vas de Carvalho, C., Lopes, M., & Ramos, A. (2014). Lean games approaches — Simulation games and digital serious games. International Journal of Advanced Corporate Learning, 7(1), 11-16. https://doi.org/10.3991/ijac.v7il.3433
Van Genderen, E. (2014). Strategic knowledge sharing: Culture acting as an inhibitor. Middle East Journal of Business, 9(4), 3-8.
Vetushinskiy, A., & Zhukov, P. (2019). Gamification of school education in Russia: Case study. In J. Arnedo-Moreno, C. S. González, & A. Mora (Eds.), Proceedings of the 3rd International Symposium on Gamification and Games for Learning (GamiLearn’19), Barcelona, Spain. http://ceur-ws.org
Walter, O., Shenaar-Golan, V., & Greenberg, Z. (2015). Effect of short-term program on academic self-efficacy in higher education. Psychology, 6(10), 1199-1215. https://doi.org/10.4236/psych.2015.610118
Zawadzki, M., Danube, C., & Shields, S. (2012). How to talk about gender inequity in the workplace: Using WAGES as an experiential learning tool to reduce reactance and promote self-efficacy. Sex Roles, 67(11-12), 605-616. https://doi.org/10.1007/s11199-012-0181-z

Citation

Pauline-Graf, D., Mandel, S. E., Allen, H. W., & Devnew, L. E. (2021). Assumption Validation Process for the Assessment of Technology-Enhanced Learning. Contemporary Educational Technology, 13(4), ep316. https://doi.org/10.30935/cedtech/11071

Pauline-Graf, D., Mandel, S. E., Allen, H. W., and Devnew, L. E. (2021). Assumption Validation Process for the Assessment of Technology-Enhanced Learning. Contemporary Educational Technology, 13(4), ep316. https://doi.org/10.30935/cedtech/11071

Pauline-Graf D, Mandel SE, Allen HW, Devnew LE. Assumption Validation Process for the Assessment of Technology-Enhanced Learning. CONT ED TECHNOLOGY. 2021;13(4), ep316. https://doi.org/10.30935/cedtech/11071

Pauline-Graf, Denise, Susan E. Mandel, Heather W. Allen, and Lynne E. Devnew. "Assumption Validation Process for the Assessment of Technology-Enhanced Learning". Contemporary Educational Technology 2021 13 no. 4 (2021): ep316. https://doi.org/10.30935/cedtech/11071

Pauline-Graf, Denise et al. "Assumption Validation Process for the Assessment of Technology-Enhanced Learning". Contemporary Educational Technology, vol. 13, no. 4, 2021, ep316. https://doi.org/10.30935/cedtech/11071

Pauline-Graf D, Mandel SE, Allen HW, Devnew LE. Assumption Validation Process for the Assessment of Technology-Enhanced Learning. CONT ED TECHNOLOGY. 2021;13(4):ep316. https://doi.org/10.30935/cedtech/11071