Systematic Review on the Application of Multimodal Learning Analytics to Personalize Students' Learning

Main Article Content

Khor Ean Teng
Tan Le Ping
Chan Shi Hui Leta

Keywords

multimodal learning analytics, multimodal data, personalised learning, systematic review

Abstract

In personalized learning (PL), learning processes are customized to account for student skills and preferences. However, as PL is generally based on a single data type, it cannot wholly represent students' learning behaviors and progress. Hence, it is crucial to leverage Multimodal Learning Analytics (MMLA) in PL to alleviate these restrictions. A systematic literature review was conducted to explore the use of MMLA in PL and investigate its benefits across several contexts and approaches. The underexplored aspects of MMLA in PL, like the gaps in topics, pedagogies, learning settings and environments, populations, and modalities studied, are addressed, and MMLA’s potential to provide real-time tailored feedback and improve engagement is discussed.

Abstract 24 | PDF Downloads 19

References

Alexander, B., Ashford-Rowe, K., Barajas-Murph, N., Dobbin, G., Knott, J., McCormack, M., Pomerantz, J., Seilhamer, R., & Weber, N. (2019). Horizon report 2019 higher education edition. EDU19. https://www.learntechlib.org/p/208644/report_20/

Alwahaby, H., Cukurova, M., Papamitsiou, Z., Giannakos, M. (2022). The evidence of impact and ethical considerations of multimodal learning analytics: A systematic literature review. In M. Giannakos, D. Spikol, D. Di Mitri, K. Sharma, X. Ochoa, & R. Hammad (Eds.), The Multimodal Learning Analytics Handbook (pp. 289-325). Springer. https://doi.org/10.1007/978-3-031-08076-0_12

Amarasinghe, I., Hernández-Leo, D., & Jonsson, A. (2019). Data-informed design parameters for adaptive collaborative scripting in across-spaces learning situations. User Modeling and User-Adapted Interaction, 29, 869-892. https://doi.org/10.1007/s11257-019-09233-8

Blikstein, P., & Worsley, M. (2016). Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks. Journal of Learning Analytics, 3(2), 220-238. http://dx.doi.org/10.18608/jla.2016.32.11

Chan, M.C.E., Ochoa, X., Clarke, D. (2020). Multimodal learning analytics in a laboratory classroom. In M. Virvou, E. Alepis, G. Tsihrintzis, & L. Jain (Eds.), Machine Learning Paradigms. Intelligent Systems Reference Library, Volume 158. Springer. https://doi.org/10.1007/978-3030-13743-4_8

Cukurova, M., Luckin, R., Millán, E., & Mavrikis, M. (2018). The NISPI framework: Analysing collaborative problemsolving from students’ physical interactions. Computers & Education, 116, 93-109. https://doi.org/10.1016/j.compedu.2017.08.007

D'Mello, S., & Graesser, A. (2012). Dynamics of affective states during complex learning. Learning and Instruction, 22(2), 145–157. https://doi.org/10.1016/j.learninstruc.2011.10.001

Di Mitri, D., Scheffel, M., Drachsler, H., Börner, D., Ternier, S., & Specht, M. (2017). Learning pulse: A machine learning approach for predicting performance in self-regulated learning using multimodal data. In A. Wise (Eds.), Proceedings of the Seventh International Learning Analytics & Knowledge Conference (pp. 188-197). Association for Computing Machinery. https://doi.org/10.1145/3027385.3027447

Dumont, H., & Ready, D. D. (2023). On the promise of personalized learning for educational equity. Npj Science of Learning, 8(1), 26. https://doi.org/10.1038/s41539-023-00174-x

Ezen-Can, A., Grafsgaard, J. F., Lester, J. C., & Boyer, K. E. (2015). Classifying student dialogue acts with multimodal learning analytics. In J. Baron (Eds.), Proceedings of the Fifth International Conference on Learning Analytics And Knowledge (pp. 280-289). Association for Computing Machinery. https://doi.org/10.1145/2723576.2723588

Florian-Gaviria, B., Glahn, C., & Fabregat Gesa, R. (2013). A software suite for efficient use of the European qualifications framework in online and blended courses. IEEE Transactions on Learning Technologies, 6(3), 283-296. https://doi.org/10.1109/tlt.2013.18

Giannakos, M. N., Sharma, K., Pappas, I. O., Kostakos, V., & Velloso, E. (2019). Multimodal data as a means to understand the learning experience. International Journal of Information Management, 48, 108-119. https://doi.org/10.1016/j.ijinfomgt.2019.02.003

Giannakos, M., Spikol, D., Di Mitri, D., Sharma, K., Ochoa, X., Hammad, R. (2022). Introduction to Multimodal Learning Analytics. In: Giannakos, M., Spikol, D., Di Mitri, D., Sharma, K., Ochoa, X., Hammad, R. (Eds.), The Multimodal Learning Analytics Handbook (pp. 3-28). Springer. https://doi.org/10.1007/978-3-031-08076-0_1

Grawemeyer, B., Mavrikis, M., Holmes, W., Gutiérrez-Santos, S., Wiedmann, M., & Rummel, N. (2017). Affective learning: Improving engagement and enhancing learning with affect-aware feedback. User Modeling and User-Adapted Interaction, 27(1), 119-158. https://doi.org/10.1007/s11257-017-9188-z

Junokas, M. J., Lindgren, R., Kang, J., & Morphew, J. W. (2018). Enhancing multimodal learning through personalized gesture recognition. Journal of Computer Assisted Learning, 34(4), 350-357. https://doi.org/10.1111/jcal.12262

Kaklauskas, A., Kuzminske, A., Zavadskas, E. K., Daniunas, A., Kaklauskas, G., Seniut, M., Raistenskis, J., Safonov, A., Kliukas, R., Juozapaitis, A., Radzeviciene, A., & Cerkauskiene, R. (2015). Affective tutoring system for built environment management. Computers & Education, 82, 202-216. https://doi.org/10.1016/j.compedu.2014.11.016

Khor, E. T., & Looi, C. K. (2019). A learning analytics approach to model and predict learners' success in digital learning.
In Y. W. Chew, K. M. Chan, and A. Alphonso (Eds.), 2019: ASCILITE 2019 Conference Proceedings: Personalised Learning. Diverse Goals. One Heart (pp. 476-480). https://doi.org/10.14742/apubs.2019.315

Khor E. T., & Mutthulakshmi K. (2023). A systematic review of the role of learning analytics in supporting personalized learning. Education Sciences, 14(1), 1-18. https://doi.org/10.3390/educsci14010051

Kosmas, P., Ioannou, A., & Retalis, S. (2018). Moving bodies to moving minds: A study of the use of motion-based games in special education. TechTrends, 62(6), 594–601. https://doi.org/10.1007/s11528-018-0294-5

Mangaroska, K., Sharma, K., Giannakos, M., Trætteberg, H., & Dillenbourg, P. (2018). Gaze-driven design insights to amplify debugging skills: A learnercentered analysis approach. Journal of Learning Analytics, 5(3), 98-119. https://doi.org/10.18608/jla.2018.53.7

Mangaroska, K., Vesin, B., & Giannakos, M. (2019). Cross-platform analytics: A step towards personalization and adaptation in education. In D. Azcona, R. Chung (Eds.), Proceedings of the 9th International Conference on Learning Analytics & Knowledge (pp. 71-75). Association for Computing Machinery. https://doi.org/10.1145/3303772.3303825

Martinez-Maldonado, R., Echeverria, V., Santos, O. C., Santos, A. D., & Yacef, K. (2018). Physical learning analytics: A multimodal perspective. In A. Pardo, K. Bartimote-Aufflick, G. Lynch, S. B. Shum, R. Ferguson, A. Merceron, X. Ochoa (Eds.), Proceedings of the 8th International Conference on Learning Analytics and Knowledge (pp. 375-379). Association for Computing Machinery. https://doi.org/10.1145/3170358.3170379

Martinez-Maldonado, R., Power, T., Hayes, C., Abdiprano, A., Vo, T., Axisa, C., & Buckingham Shum, S. (2017). Analytics meet patient manikins: Challenges in an authentic smallgroup healthcare simulation classroom. In A. Wise, P. H. Winne, G. Lynch, X. Ochoa, I. Molenaar, S. Dawson, M. Hatala (Eds.), Proceedings of the Seventh International Learning Analytics & Knowledge Conference (pp. 90-94). Association for Computing Machinery. https://doi.org/10.1145/3027385.3027401

Maseleno, A., Sabani, N., Huda, M., Ahmad, R., Azmi Jasmi, K., & Basiron, B. (2018). Demystifying learning analytics in personalised learning. International Journal of Engineering & Technology, 7(3), 1124-1129. https://doi.org/10.14419/ijet.v7i3.9789

Nguyen, Q., Huptych, M., & Rienties, B. (2018). Using temporal analytics to detect inconsistencies between learning design and students' behaviours. Journal of Learning Analytics, 5(3), 120-135. https://doi.org/10.18608/jla.2018.53.8

Noel, R., Riquelme, F., Lean, R. M., Merino, E., Cechinel, C., Barcelos, T. S., Villarroel, R., & Munoz, R. (2018). Exploring collaborative writing of user stories with multimodal learning analytics: A case study on a software engineering course. IEEE Access, 6, 67783-67798. https://doi.org/10.1109/access.2018.2876801

Ochoa, X., Domínguez, F., Guamán, B., Maya, R., Falcones, G., & Castells, J. (2018). The rap system: Automatic feedback of oral presentation skills using multimodal analysis and low-cost sensors. In A. Pardo, K. BartimoteAufflick, G. Lynch, S. B. Shum, R. Ferguson, A. Merceron, & X. Ochoa (Eds.), Proceedings of the 8th International Conference on Learning Analytics and Knowledge. Association for Computing Machinery. https://doi.org/10.1145/3170358.3170406

Pardo, A., Han, F., & Ellis, R. A. (2017). Combining university student self-regulated learning indicators and engagement with online learning events to predict academic performance. IEEE Transactions on Learning Technologies, 10(1), 82-92. https://doi.org/10.1109/tlt.2016.2639508

Prieto, L. P., Sharma, K., Dillenbourg, P., & Jesús, M. (2016). Teaching analytics: Towards automatic extraction of orchestration graphs using wearable sensors. In D. Gašević, G. Lynch, S. Dawson, H. Drachsler, C. P. Rosé (Eds.), Proceedings of the Sixth International Conference on Learning Analytics & Knowledge - LAK ’16 (pp. 148-157). Association for Computing Machinery. https://doi.org/10.1145/2883851.2883927

Prieto, L. P., Sharma, K., Kidzinski, RodríguezTriana, M. J., & Dillenbourg, P. (2018). Multimodal teaching analytics: Automated extraction of orchestration graphs from wearable sensor data. Journal of Computer Assisted Learning, 34(2), 193-203. https://doi.org/10.1111/jcal.12232

Rodríguez-Triana, M. J., Prieto, L. P., MartínezMonés, A., Asensio-Pérez, J. I., & Dimitriadis, Y. (2018). The teacher in the loop: Customizing multimodal learning analytics for blended learning. In A. Pardo, K. Bartimote-Aufflick, G. Lynch, S. B. Shum, R. Ferguson, A. Merceron, & X. Ochoa (Eds.), Proceedings of the 8th International Conference on Learning Analytics and Knowledge (pp. 417-426). Association for Computing Machinery. https://doi.org/10.1145/3170358.3170364

Santos, O. C., Saneiro, M., Salmeron-Majadas, S., & Boticario, J. G. (2014). A methodological approach to eliciting affective educational recommendations. In D. G. Sampson, J. M. Spector, N.-S. Chen, R. Huang, Kinshuk (Eds.), 2014 IEEE 14th International Conference on Advanced Learning Technologies (pp. 529-533). IEEE. https://doi.org/10.1109/icalt.2014.234

Sharma, K., Papamitsiou, Z., & Giannakos, M. (2019). Building pipelines for educational data using AI and multimodal analytics: A "grey‐box” approach. British Journal of Educational Technology, 50(6), 3004-3031. https://doi.org/10.1111/bjet.12854

Spikol, D., Avramides, K., & Cukurova, M. (2016). Exploring the interplay between human and machine annotated multimodal learning analytics in handson STEM activities. In D. Gašević, G. Lynch, S. Dawson, H. Drachsler, C. P. Rosé (Eds.), Proceedings of the Sixth International Conference on Learning Analytics & Knowledge - LAK’16 (pp. 522-523). Association for Computing Machinery. https://doi.org/10.1145/2883851.2883920

Spikol, D., Ruffaldi, E., Dabisias, G., & Cukurova, M. (2018). Supervised machine learning in multimodal learning analytics for estimating success in project-based learning. Journal of Computer Assisted Learning, 34(4), 366-377. https://doi.org/10.1111/jcal.12263

Taylor, D. L., Yeung, M., & Bashet, A. Z. (2021). Personalized and adaptive learning. In J. Ryoo & K. Winkelmann (Eds.), Innovative Learning Environments in STEM Higher Education (pp.17-34). Springer. https://doi.org/10.1007/978-3-030-58948-6_2

Worsley, M., & Blikstein, P. (2018). A multimodal analysis of making. International Journal of Artificial Intelligence in Education, 28(3), 385–419. https://doi.org/10.1007/s40593-017-0160-1