Systematic Review on the Application of Multimodal Learning Analytics to Personalize Students' Learning
Main Article Content
Keywords
multimodal learning analytics, multimodal data, personalised learning, systematic review
Abstract
In personalized learning (PL), learning processes are customized to account for student skills and preferences. However, as PL is generally based on a single data type, it cannot wholly represent students' learning behaviors and progress. Hence, it is crucial to leverage Multimodal Learning Analytics (MMLA) in PL to alleviate these restrictions. A systematic literature review was conducted to explore the use of MMLA in PL and investigate its benefits across several contexts and approaches. The underexplored aspects of MMLA in PL, like the gaps in topics, pedagogies, learning settings and environments, populations, and modalities studied, are addressed, and MMLA’s potential to provide real-time tailored feedback and improve engagement is discussed.
References
Alwahaby, H., Cukurova, M., Papamitsiou, Z., Giannakos, M. (2022). The evidence of impact and ethical considerations of multimodal learning analytics: A systematic literature review. In M. Giannakos, D. Spikol, D. Di Mitri, K. Sharma, X. Ochoa, & R. Hammad (Eds.), The Multimodal Learning Analytics Handbook (pp. 289-325). Springer. https://doi.org/10.1007/978-3-031-08076-0_12
Amarasinghe, I., Hernández-Leo, D., & Jonsson, A. (2019). Data-informed design parameters for adaptive collaborative scripting in across-spaces learning situations. User Modeling and User-Adapted Interaction, 29, 869-892. https://doi.org/10.1007/s11257-019-09233-8
Blikstein, P., & Worsley, M. (2016). Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks. Journal of Learning Analytics, 3(2), 220-238. http://dx.doi.org/10.18608/jla.2016.32.11
Chan, M.C.E., Ochoa, X., Clarke, D. (2020). Multimodal learning analytics in a laboratory classroom. In M. Virvou, E. Alepis, G. Tsihrintzis, & L. Jain (Eds.), Machine Learning Paradigms. Intelligent Systems Reference Library, Volume 158. Springer. https://doi.org/10.1007/978-3030-13743-4_8
Cukurova, M., Luckin, R., Millán, E., & Mavrikis, M. (2018). The NISPI framework: Analysing collaborative problemsolving from students’ physical interactions. Computers & Education, 116, 93-109. https://doi.org/10.1016/j.compedu.2017.08.007
D'Mello, S., & Graesser, A. (2012). Dynamics of affective states during complex learning. Learning and Instruction, 22(2), 145–157. https://doi.org/10.1016/j.learninstruc.2011.10.001
Di Mitri, D., Scheffel, M., Drachsler, H., Börner, D., Ternier, S., & Specht, M. (2017). Learning pulse: A machine learning approach for predicting performance in self-regulated learning using multimodal data. In A. Wise (Eds.), Proceedings of the Seventh International Learning Analytics & Knowledge Conference (pp. 188-197). Association for Computing Machinery. https://doi.org/10.1145/3027385.3027447
Dumont, H., & Ready, D. D. (2023). On the promise of personalized learning for educational equity. Npj Science of Learning, 8(1), 26. https://doi.org/10.1038/s41539-023-00174-x
Ezen-Can, A., Grafsgaard, J. F., Lester, J. C., & Boyer, K. E. (2015). Classifying student dialogue acts with multimodal learning analytics. In J. Baron (Eds.), Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (pp. 280-289). Association for Computing Machinery. https://doi.org/10.1145/2723576.2723588
Florian-Gaviria, B., Glahn, C., & Fabregat Gesa, R. (2013). A software suite for efficient use of the European qualifications framework in online and blended courses. IEEE Transactions on Learning Technologies, 6(3), 283-296. https://doi.org/10.1109/tlt.2013.18
Giannakos, M. N., Sharma, K., Pappas, I. O., Kostakos, V., & Velloso, E. (2019). Multimodal data as a means to understand the learning experience. International Journal of Information Management, 48, 108-119. https://doi.org/10.1016/j.ijinfomgt.2019.02.003
Giannakos, M., Spikol, D., Di Mitri, D., Sharma, K., Ochoa, X., Hammad, R. (2022). Introduction to Multimodal Learning Analytics. In: Giannakos, M., Spikol, D., Di Mitri, D., Sharma, K., Ochoa, X., Hammad, R. (Eds.), The Multimodal Learning Analytics Handbook (pp. 3-28). Springer. https://doi.org/10.1007/978-3-031-08076-0_1
Grawemeyer, B., Mavrikis, M., Holmes, W., Gutiérrez-Santos, S., Wiedmann, M., & Rummel, N. (2017). Affective learning: Improving engagement and enhancing learning with affect-aware feedback. User Modeling and User-Adapted Interaction, 27(1), 119-158. https://doi.org/10.1007/s11257-017-9188-z
Junokas, M. J., Lindgren, R., Kang, J., & Morphew, J. W. (2018). Enhancing multimodal learning through personalized gesture recognition. Journal of Computer Assisted Learning, 34(4), 350-357. https://doi.org/10.1111/jcal.12262
Kaklauskas, A., Kuzminske, A., Zavadskas, E. K., Daniunas, A., Kaklauskas, G., Seniut, M., Raistenskis, J., Safonov, A., Kliukas, R., Juozapaitis, A., Radzeviciene, A., & Cerkauskiene, R. (2015). Affective tutoring system for built environment management. Computers & Education, 82, 202-216. https://doi.org/10.1016/j.compedu.2014.11.016
Khor, E. T., & Looi, C. K. (2019). A learning analytics approach to model and predict learners' success in digital learning.
In Y. W. Chew, K. M. Chan, and A. Alphonso (Eds.), 2019: ASCILITE 2019 Conference Proceedings: Personalised Learning. Diverse Goals. One Heart (pp. 476-480). https://doi.org/10.14742/apubs.2019.315
Khor E. T., & Mutthulakshmi K. (2023). A systematic review of the role of learning analytics in supporting personalized learning. Education Sciences, 14(1), 1-18. https://doi.org/10.3390/educsci14010051
Kosmas, P., Ioannou, A., & Retalis, S. (2018). Moving bodies to moving minds: A study of the use of motion-based games in special education. TechTrends, 62(6), 594–601. https://doi.org/10.1007/s11528-018-0294-5
Mangaroska, K., Sharma, K., Giannakos, M., Trætteberg, H., & Dillenbourg, P. (2018). Gaze-driven design insights to amplify debugging skills: A learnercentered analysis approach. Journal of Learning Analytics, 5(3), 98-119. https://doi.org/10.18608/jla.2018.53.7
Mangaroska, K., Vesin, B., & Giannakos, M. (2019). Cross-platform analytics: A step towards personalization and adaptation in education. In D. Azcona, R. Chung (Eds.), Proceedings of the 9th International Conference on Learning Analytics & Knowledge (pp. 71-75). Association for Computing Machinery. https://doi.org/10.1145/3303772.3303825
Martinez-Maldonado, R., Echeverria, V., Santos, O. C., Santos, A. D., & Yacef, K. (2018). Physical learning analytics: A multimodal perspective. In A. Pardo, K. Bartimote-Aufflick, G. Lynch, S. B. Shum, R. Ferguson, A. Merceron, X. Ochoa (Eds.), Proceedings of the 8th International Conference on Learning Analytics and Knowledge (pp. 375-379). Association for Computing Machinery. https://doi.org/10.1145/3170358.3170379
Martinez-Maldonado, R., Power, T., Hayes, C., Abdiprano, A., Vo, T., Axisa, C., & Buckingham Shum, S. (2017). Analytics meet patient manikins: Challenges in an authentic smallgroup healthcare simulation classroom. In A. Wise, P. H. Winne, G. Lynch, X. Ochoa, I. Molenaar, S. Dawson, M. Hatala (Eds.), Proceedings of the Seventh International Learning Analytics & Knowledge Conference (pp. 90-94). Association for Computing Machinery. https://doi.org/10.1145/3027385.3027401
Maseleno, A., Sabani, N., Huda, M., Ahmad, R., Azmi Jasmi, K., & Basiron, B. (2018). Demystifying learning analytics in personalised learning. International Journal of Engineering & Technology, 7(3), 1124-1129. https://doi.org/10.14419/ijet.v7i3.9789
Nguyen, Q., Huptych, M., & Rienties, B. (2018). Using temporal analytics to detect inconsistencies between learning design and students' behaviours. Journal of Learning Analytics, 5(3), 120-135. https://doi.org/10.18608/jla.2018.53.8
Noel, R., Riquelme, F., Lean, R. M., Merino, E., Cechinel, C., Barcelos, T. S., Villarroel, R., & Munoz, R. (2018). Exploring collaborative writing of user stories with multimodal learning analytics: A case study on a software engineering course. IEEE Access, 6, 67783-67798. https://doi.org/10.1109/access.2018.2876801
Ochoa, X., Domínguez, F., Guamán, B., Maya, R., Falcones, G., & Castells, J. (2018). The rap system: Automatic feedback of oral presentation skills using multimodal analysis and low-cost sensors. In A. Pardo, K. BartimoteAufflick, G. Lynch, S. B. Shum, R. Ferguson, A. Merceron, & X. Ochoa (Eds.), Proceedings of the 8th International Conference on Learning Analytics and Knowledge. Association for Computing Machinery. https://doi.org/10.1145/3170358.3170406
Pardo, A., Han, F., & Ellis, R. A. (2017). Combining university student self-regulated learning indicators and engagement with online learning events to predict academic performance. IEEE Transactions on Learning Technologies, 10(1), 82-92. https://doi.org/10.1109/tlt.2016.2639508
Prieto, L. P., Sharma, K., Dillenbourg, P., & Jesús, M. (2016). Teaching analytics: Towards automatic extraction of orchestration graphs using wearable sensors. In D. Gašević, G. Lynch, S. Dawson, H. Drachsler, C. P. Rosé (Eds.), Proceedings of the Sixth International Conference on Learning Analytics & Knowledge - LAK ’16 (pp. 148-157). Association for Computing Machinery. https://doi.org/10.1145/2883851.2883927
Prieto, L. P., Sharma, K., Kidzinski, RodríguezTriana, M. J., & Dillenbourg, P. (2018). Multimodal teaching analytics: Automated extraction of orchestration graphs from wearable sensor data. Journal of Computer Assisted Learning, 34(2), 193-203. https://doi.org/10.1111/jcal.12232
Rodríguez-Triana, M. J., Prieto, L. P., MartínezMonés, A., Asensio-Pérez, J. I., & Dimitriadis, Y. (2018). The teacher in the loop: Customizing multimodal learning analytics for blended learning. In A. Pardo, K. Bartimote-Aufflick, G. Lynch, S. B. Shum, R. Ferguson, A. Merceron, & X. Ochoa (Eds.), Proceedings of the 8th International Conference on Learning Analytics and Knowledge (pp. 417-426). Association for Computing Machinery. https://doi.org/10.1145/3170358.3170364
Santos, O. C., Saneiro, M., Salmeron-Majadas, S., & Boticario, J. G. (2014). A methodological approach to eliciting affective educational recommendations. In D. G. Sampson, J. M. Spector, N.-S. Chen, R. Huang, Kinshuk (Eds.), 2014 IEEE 14th International Conference on Advanced Learning Technologies (pp. 529-533). IEEE. https://doi.org/10.1109/icalt.2014.234
Sharma, K., Papamitsiou, Z., & Giannakos, M. (2019). Building pipelines for educational data using AI and multimodal analytics: A "grey‐box” approach. British Journal of Educational Technology, 50(6), 3004-3031. https://doi.org/10.1111/bjet.12854
Spikol, D., Avramides, K., & Cukurova, M. (2016). Exploring the interplay between human and machine annotated multimodal learning analytics in handson STEM activities. In D. Gašević, G. Lynch, S. Dawson, H. Drachsler, C. P. Rosé (Eds.), Proceedings of the Sixth International Conference on Learning Analytics & Knowledge - LAK’16 (pp. 522-523). Association for Computing Machinery. https://doi.org/10.1145/2883851.2883920
Spikol, D., Ruffaldi, E., Dabisias, G., & Cukurova, M. (2018). Supervised machine learning in multimodal learning analytics for estimating success in project-based learning. Journal of Computer Assisted Learning, 34(4), 366-377. https://doi.org/10.1111/jcal.12263
Taylor, D. L., Yeung, M., & Bashet, A. Z. (2021). Personalized and adaptive learning. In J. Ryoo & K. Winkelmann (Eds.), Innovative Learning Environments in STEM Higher Education (pp.17-34). Springer. https://doi.org/10.1007/978-3-030-58948-6_2
Worsley, M., & Blikstein, P. (2018). A multimodal analysis of making. International Journal of Artificial Intelligence in Education, 28(3), 385–419. https://doi.org/10.1007/s40593-017-0160-1