Abstract
Personalized gamification aims to address shortcomings of the one-size-fits-all (OSFA) approach in improving students’ motivations throughout the learning process. However, studies still focus on personalizing to a single user dimension, ignoring multiple individual and contextual factors that affect user motivation. Unlike prior research, we address this issue by exploring multidimensional personalization compared to OSFA based on a multi-institution sample. Thus, we conducted a controlled experiment in three institutions, comparing gamification designs (OSFA and Personalized to the learning task and users’ gaming habits/preferences and demographics) in terms of 58 students’ motivations to complete assessments for learning. Our results suggest no significant differences among OSFA and Personalized designs, despite suggesting user motivation depended on fewer user characteristics when using personalization. Additionally, exploratory analyses suggest personalization was positive for females and those holding a technical degree, but negative for those who prefer adventure games and those who prefer single-playing. Our contribution benefits designers, suggesting how personalization works; practitioners, demonstrating to whom the personalization strategy was more or less suitable; and researchers, providing future research directions.


Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Notes
That is, personalized to the learning task, users’ gaming habits/preferences, and demographics simultaneously; see “Experimental Conditions” section for details.
Moderators are factors that increase/decrease an intervention’s effect (Landers et al., 2018).
The amotivation of all participants of this subgroup was 1.
References
Abt, K. (1987). Descriptive data analysis: a concept between confirmatory and exploratory data analysis. Methods of Information in Medicine, 26(02), 77–88.
Altmeyer, M., Lessel, P., Muller, L., & Krüger, A. (2019). Combining behavior change intentions and user types to select suitable gamification elements for persuasive fitness systems. In International conference on persuasive technology (pp. 337–349).
Armstrong, R.A. (2014). When to use the B onferroni correction. Ophthalmic and Physiological Optics, 34(5), 502–508.
Bai, S., Hew, K.F., & Huang, B. (2020). Is gamification “bullshit”? Evidence from a meta-analysis and synthesis of qualitative data in educational contexts. Educational Research Review, 100322.
Baldeón, J., Rodríguez, I., & Puig, A. (2016). LEGA: A learner-centered gamification design framework. In Proceedings of the XVII international conference on human computer interaction. https://doi.org/10.1145/2998626.2998673 (pp. 45:1–45:8). ACM: New York.
Batsell Jr, W.R., Perry, J.L., Hanley, E., & Hostetter, A.B. (2017). Ecological validity of the testing effect: The use of daily quizzes in introductory psychology. Teaching of Psychology, 44(1), 18–23.
Blandford, A., Furniss, D., & Makri, S. (2016). Qualitative HCI research: Going behind the scenes. Synthesis Lectures on Human-Centered Informatics, 9(1), 1–115.
Bloom, B.S. (1956). Taxonomy of educational objectives: The classification of educational goals. New York: Longman.
Bovermann, K., & Bastiaens, T.J. (2020). Towards a motivational design? Connecting gamification user types and online learning activities. Research and Practice in Technology Enhanced Learning, 15(1), 1–18. https://doi.org/10.1186/s41039-019-0121-4.
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101.
Cairns, P. (2019). Doing better statistics in human-computer interaction. Cambridge: Cambridge University Press.
Canty, A., & Ripley, B.D. (2021). boot: Bootstrap R (S-plus) functions [Computer software manual]. R package version 1.3-28.
Carpenter, J., & Bithell, J. (2000). Bootstrap confidence intervals: When, Which, What? A practical guide for medical statisticians. Statistics in Medicine, 19(9), 1141–1164.
Cockburn, A., Dragicevic, P., Besançon, L., & Gutwin, C. (2020). Threats of a replication crisis in empirical computer science. Communications of the ACM, 63(8), 70–79.
Dash, S. (2019). Google classroom as a learning management system to teach biochemistry in a medical school. Biochemistry and Molecular Biology Education, 47(4), 404–407.
Deci, E.L., & Ryan, R.M. (2000). The “What” and “Why” of goal pursuits: human needs and the self-determination of behavior. Psychological Inquiry, 11 (4), 227–268. https://doi.org/10.1207/S15327965PLI1104_01.
Deterding, S., Dixon, D., Khaled, R., & Nacke, L. (2011). From game design elements to gamefulness: defining gamification. In Proceedings of the 15th international academic MindTrek conference: Envisioning future media environment (pp. 9–15), DOI https://doi.org/10.1145/2181037.2181040, (to appear in print).
Dichev, C., & Dicheva, D. (2017). Gamifying education: what is known, what is believed and what remains uncertain: a critical review. International Journal of Educational Technology in Higher Education, 14(1), 9.
Dragicevic, P. (2016). Fair statistical communication in HCI. In Modern statistical methods for HCI (pp. 291–330). Springer.
Gamboa, V., Valadas, S.T., & Paixão, O. (2013). Validação da versão portuguesa da Situational Motivation Scale (SIMS) em contextos académicos. In Atas do XII Congresso Galego-Português de Psicopedagogia, 11-13 de setembro de (pp. 4868–4882).
Gari, M., Walia, G., & Radermacher, A (2018). Gamification in computer science education: A systematic literature review. In American society for engineering education.
Gliem, J.A., & Gliem, R.R. (2003). Calculating, interpreting, and reporting Cronbach’s alpha reliability coefficient for likert-type scales.
Grin, E., Burgos, F., Fernandes, G., & Bresciani, L. (2021). O mapa regional das múltiplas desigualdades e do desenvolvimento humano no Brasil. In: (pp 99–122).
Guay, F., Vallerand, R.J., & Blanchard, C. (2000). On the assessment of situational intrinsic and extrinsic motivation: The situational motivation scale (SIMS). Motivation and Emotion, 24(3), 175–213.
Hajarian, M., Bastanfard, A., Mohammadzadeh, J., & Khalilian, M. (2019). A personalized gamification method for increasing user engagement in social networks. Social Network Analysis and Mining, 9(1), 47.
Hallifax, S., Audrey, S., Jean-Charles, M., Guillaume, L., & Elise, L. (2019). Factors to consider for tailored gamification. In CHI play. https://hal.archives-ouvertes.fr/hal-02185647 (pp. 559–572). Barcelona, Spain: ACM.
Hallifax, S., Serna, A., Marty, J.C., & Lavoué, É (2019). Adaptive gamification in education: A literature review of current trends and developments. In M. Scheffel, J. Broisin, V. Pammer-Schindler, A. Ioannou, & J. Schneider (Eds.) Transforming learning with meaningful technologies. https://doi.org/10.1007/978-3-030-29736-7_22 (pp. 294–307). Cham: Springer International Publishing.
Hamari, J., Koivisto, J., & Sarsa, H. (2014). Does gamification work?-A literature review of empirical studies on gamification. In HICSS, (Vol. 14 pp. 3025–3034).
Hanus, M.D., & Fox, J. (2015). Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation, social comparison, satisfaction, effort, and academic performance. Computers & Education, 80, 152–161. ttp://www.sciencedirect.com/science/article/pii/S0360131514002000, https://doi.org/10.1016/j.compedu.2014.08.019.
Hothorn, T., Hornik, K., & Zeileis, A. (2006). Unbiased recursive partitioning: A conditional inference framework. Journal of Computational and Graphical Statistics, 15(3), 651–674. https://doi.org/10.1198/106186006X133933.
Hox, J.J., Moerbeek, M., & Van de Schoot, R. (2010). Multilevel analysis: Techniques and applications. Evanston: Routledge.
Huang, R., Ritzhaupt, A.D., Sommer, M., Zhu, J., Stephen, A., Valle, N., & Li, J. (2020). The impact of gamification in educational settings on student learning outcomes: a meta-analysis. Educational Technology Research and Development, 1–27.
Hyrynsalmi, S., Smed, J., & Kimppa, K. (2017). The dark side of gamification: How we should stop worrying and study also the negative impacts of bringing game design elements to everywhere. In GamiFIN (pp. 96–104).
Klock, A.C.T., Gasparini, I., Pimenta, M.S., & Hamari, J. (2020). Tailored gamification: A review of literature. International Journal of Human-Computer Studies, 102495.
Kocadere, S.A., & Çağlar, Ş. (2015). The design and implementation of a gamified assessment. Journal of e-Learning and Knowledge Society, 11(3).
Krathwohl, D.R. (2002). A revision of bloom’s taxonomy: An overview. Theory into Practice, 41(4), 212–218. https://doi.org/10.1207/s15430421tip4104_2.
Landers, R.N., Auer, E.M., Collmus, A.B., & Armstrong, M.B. (2018). Gamification science, its history and future: Definitions and a research agenda. Simulation & Gaming, 49(3), 315–337.
Landers, R.N., Bauer, K.N., Callan, R.C., & Armstrong, M.B. (2015). Psychological theory and the gamification of learning. In T. Reiners L.C. Wood (Eds.) Gamification in education and business. https://doi.org/10.1007/978-3-319-10208-5_9 (pp. 165–186). Cham: Springer International Publishing.
Lavouė, E., Monterrat, B., Desmarais, M., & George, S. (2018). Adaptive gamification for learning environments. IEEE Transactions on Learning Technologies, 12(1), 16–28.
Lessel, P., Altmeyer, M., Müller, M., Wolff, C., & Krüger, A. (2017). Measuring the effect of “bottom-up” gamification in a microtask setting. In Proceedings of the 21st international academic Mindtrek conference (pp. 63–72).
Liu, D., Santhanam, R., & Webster, J. (2017). Toward meaningful engagement: A framework for design and research of gamified information systems. MIS Quarterly, 41(4), 1011–1034.
Lopez, C.E., & Tucker, C.S. (2021). Adaptive gamification and its impact on performance. In International conference on human-computer interaction (pp. 327–341).
Loughrey, K., & Broin, D. (2018). Are we having fun yet? Misapplying motivation to gamification. In 2018 IEEE games, entertainment, media conference (GEM). https://doi.org/10.1109/GEM.2018.8516535 (pp. 1–9).
Mair, P., & Wilcox, R. (2018). Robust statistical methods using WRS2. The WRS2 Package.
Mora, A., Riera, D., Gonzalez, C., & Arnedo-Moreno, J. (2015). A literature review of gamification design frameworks. In 2015 7th international conference on games and virtual worlds for serious applications (VS-Games) (pp. 1–8).
Mora, A., Tondello, G.F., Nacke, L.E., & Arnedo-Moreno, J. (2018). Effect of personalized gameful design on student engagement. In 2018 IEEE global engineering education conference (EDUCON). https://doi.org/10.1109/EDUCON.2018.8363471(pp. 1925–1933).
Mpungose, C.B. (2020). Emergent transition from face-to-face to online learning in a South African University in the context of the Coronavirus pandemic. Humanities and Social Sciences Communications, 7(1), 1–9.
Norman, D.A. (2004). Emotional design: Why we love (or hate) everyday things. Basic Civitas Books.
Oliveira, W., Toda, A., Toledo, P., Shi, L., Vassileva, J., Bittencourt, I.I., & Isotani, S. (2020). Does tailoring gamified educational systems matter? The impact on students’ flow experience. In Proceedings of the 53rd Hawaii international conference on system sciences (pp. 1226–1235). ScholarSpace.
Orji, R., Oyibo, K., & Tondello, G.F. (2017). A comparison of system-controlled and user-controlled personalization approaches. In Adjunct publication of the 25th conference on user modeling, adaptation and personalization. https://doi.org/10.1145/3099023.3099116 (pp. 413–418). New York: ACM.
Palomino, P., Toda, A., Rodrigues, L., Oliveira, W., & Isotani, S. (2020). From the lack of engagement to motivation: Gamification strategies to enhance users learning experiences. In 19th Brazilian symposium on computer games and digital entertainment (SBGames)-GranDGames BR Forum (pp. 1127–1130).
Pedro, L.Z., Lopes, A.M., Prates, B.G., Vassileva, J., & Isotani, S. (2015). Does gamification work for boys and girls? An exploratory study with a virtual learning environment. In Proceedings of the 30th annual ACM symposium on applied computing (pp. 214–219).
Pereira, F.D., Fonseca, S.C., Oliveira, E.H., Cristea, A.I., Bellhäuser, H., Rodrigues, L., & Carvalho, L.S. (2021). Explaining individual and collective programming students’ behaviour by interpreting a black-box predictive model. IEEE Access.
Pereira, F.D., Oliveira, E.H., Oliveira, D.B., Cristea, A.I., Carvalho, L.S., Fonseca, S.C., & Isotani, S. (2020). Using learning analytics in the Amazonas: understanding students’ behaviour in introductory programming. British Journal of Educational Technology, 51(4), 955–972.
Pintrich, P.R. (2003). A motivational science perspective on the role of student motivation in learning and teaching contexts. Journal of Educational Psychology, 95(4), 667.
Polo-Peña, A.I., Frías-Jamilena, D.M., & Fernández-Ruano, M.L. (2020). Influence of gamification on perceived self-efficacy: gender and age moderator effect. International Journal of Sports Marketing and Sponsorship.
Recabarren, M., Corvalán, B., & Villegas, M. (2021). Exploring the differences between gamer and non-gamer students in the effects of gamification on their motivation and learning. Interactive Learning Environments, 1–14.
Rey, G.D. (2012). A review of research and a meta-analysis of the seductive detail effect. Educational Research Review, 7(3), 216–237.
Rodrigues, L., Oliveira, W., Toda, A., Palomino, P., & Isotani, S. (2019). Thinking inside the box: How to tailor gamified educational systems based on learning activities types. In Proceedings of the Brazilian symposium of computers on education. https://doi.org/10.5753/cbie.sbie.2019.823 (pp. 823–832). SBC.
Rodrigues, L., Palomino, P.T., Toda, A.M., Klock, A.C.T., Oliveira, W., Avila-Santos, A.P., & Isotani, S. (2021). Personalization improves gamification: Evidence from a mixed-methods study. In Proc. ACM hum.-comput. interact., 5 (CHI PLAY). https://doi.org/10.1145/3474714.
Rodrigues, L., Toda, A.M., Oliveira, W., Palomino, P.T., Avila-Santos, A.P., & Isotani, S. (2021). Gamification works, but how and to whom? An experimental study in the context of programming lessons. In Proceedings of the 52nd ACM technical symposium on computer science education (pp. 184–190).
Rodrigues, L., Toda, A.M., Oliveira, W., Palomino, P.T., & Isotani, S. (2020). Just beat it: Exploring the influences of competition and task-related factors in gamified learning environments. In Anais do XXXI Simpósio Brasileiro de Informática na Educação (pp. 461–470).
Rodrigues, L., Toda, A.M., Oliveira, W., Palomino, P.T., Vassileva, J., & Isotani, S. (2021). Automating gamification personalization: To the user and beyond.
Rodrigues, L., Toda, A.M., Palomino, P.T., Oliveira, W., & Isotani, S. (2020). Personalized gamification: A literature review of outcomes, experiments, and approaches. In F.J. García-Peñalvo (Ed.) Proceedings of the 8th international conference on technological ecosystems for enhancing multiculturality (TEEM 2020) (Salamanca, Spain, October 21-23, 2020).
Roediger-III, H.L., & Karpicke, J.D. (2006). Test-enhanced learning: Taking memory tests improves long-term retention. Psychological Science, 17 (3), 249–255.
Rowland, C.A. (2014). The effect of testing versus restudy on retention: a meta-analytic review of the testing effect. Psychological Bulletin, 140 (6), 1432.
Ryan, R.M., & Deci, E.L. (2017). Self-determination theory: Basic psychological needs in motivation, development, and wellness. Guilford Publications.
Sailer, M., & Homner, L. (2020). The gamification of learning: a meta-analysis. Educational Psychology Review, 32, 77–112. https://doi.org/10.1007/s10648-019-09498-w.
Sanchez, D.R., Langer, M., & Kaur, R. (2020). Gamification in the classroom: Examining the impact of gamified quizzes on student learning. Computers & Education, 144, 103666. https://doi.org/10.1016/j.compedu.2019.103666.
Schubhan, M., Altmeyer, M., Buchheit, D., & Lessel, P. (2020). Investigating user-created gamification in an image tagging task. In Proceedings of the 2020 CHI conference on human factors in computing systems (pp. 1–12).
Seaborn, K., & Fels, D.I. (2015). Gamification in theory and action: A survey. International Journal of Human-Computer Studies, 74, 14–31. https://doi.org/10.1016/j.ijhcs.2014.09.006.
Sheskin, D.J. (2003). Handbook of parametric and nonparametric statistical procedures. London: Chapman and Hall/CRC.
Stuart, H., Lavoué, E., & Serna, A. (2020). To tailor or not to tailor gamification? An analysis of the impact of tailored game elements on learners’ behaviours and motivation. In 21th International conference on artificial intelligence in education.
Toda, A.M., do Carmo, R.M., da Silva, A.P., Bittencourt, I.I., & Isotani, S. (2019). An approach for planning and deploying gamification concepts with social networks within educational contexts. International Journal of Information Management, 46, 294–303. https://doi.org/10.1016/j.ijinfomgt.2018.10.001.
Toda, A.M., Klock, A.C., Oliveira, W., Palomino, P.T., Rodrigues, L., Shi, L., & Cristea, A.I. (2019). Analysing gamification elements in educational environments using an existing gamification taxonomy. Smart Learning Environments, 6(1), 16. https://doi.org/10.1186/s40561-019-0106-1.
Toda, A.M., Valle, P.H.D., & Isotani, S. (2018). The dark side of gamification: An overview of negative effects of gamification in education. In A.I. Cristea, I.I. Bittencourt, & F. Lima (Eds.) Higher education for all. From challenges to novel technology-enhanced solutions. https://doi.org/10.1007/978-3-319-97934-2_9 (pp. 143–156). Cham: Springer International Publishing.
Tondello, G.F. (2019). Dynamic personalization of gameful interactive systems (Unpublished doctoral dissertation). University of Waterloo.
Tondello, G.F., Mora, A., & Nacke, L.E. (2017). Elements of gameful design emerging from user preferences. In Proceedings of the annual symposium on computer-human interaction in play. https://doi.org/10.1145/3116595.3116627 (pp. 129–142).
Tondello, G.F., & Nacke, L.E. (2020). Validation of user preferences and effects of personalized gamification on task performance. Frontiers in Computer Science, 2, 29.
Tondello, G.F., Orji, R., & Nacke, L.E. (2017). Recommender systems for personalized gamification. In Adjunct publication of the 25th conference on user modeling, adaptation and personalization. https://doi.org/10.1145/3099023.3099114(pp. 425–430).
Van Houdt, L., Millecamp, M., Verbert, K., & Vanden Abeele, V. (2020). Disambiguating preferences for gamification strategies to motivate pro-environmental behaviour. In Proceedings of the annual symposium on computer-human interaction in play (pp. 241–253).
Van Roy, R., & Zaman, B. (2018). Need-supporting gamification in education: An assessment of motivational effects over time. Computers & Education, 127, 283–297. https://doi.org/10.1016/j.compedu.2018.08.018.
Vansteenkiste, M., Sierens, E., Soenens, B., Luyckx, K., & Lens, W. (2009). Motivational profiles from a self-determination perspective: The quality of motivation matters. Journal of Educational Psychology, 101(3), 671.
Venter, M. (2020). Gamification in stem programming courses: State of the art. In 2020 IEEE global engineering education conference (EDUCON) (pp. 859–866).
Vornhagen, J.B., Tyack, A., & Mekler, E.D. (2020). Statistical significance testing at CHI play: Challenges and opportunities for more transparency. In Proceedings of the annual symposium on computer-human interaction in play (pp. 4–18).
Wilcox, R.R. (2011). Introduction to robust estimation and hypothesis testing. New York: Academic Press.
Wohlin, C., Runeson, P., Hst, M., Ohlsson, M.C., Regnell, B., & Wessln, A. (2012). Experimentation in software engineering. Springer Publishing Company, Incorporated.
Zainuddin, Z., Chu, S.K.W., Shujahat, M., & Perera, C.J. (2020). The impact of gamification on learning and instruction: A systematic review of empirical evidence. Educational Research Review, 100326.
Funding
This research received financial support from the following agencies: Brazilian National Council for Scientific and Technological Development (CNPq) - processes 141859/2019-9, 163932/2020-4, 308458/2020-6, 308395/2020-4, and 308513/2020-7; Coordination for the Improvement of Higher Education Personnel (CAPES) - Finance Code 001; and São Paulo State Research Support Foundation (FAPESP) - processes 2018/15917- 0 and 2013/07375-0. Additionally, this research was carried out within the scope of the Samsung-UFAM Project for Education and Research (SUPER), according to Article 48 of Decree \(\text {n}^{\underline {\text {o}}}\); 6.008/2006 (SUFRAMA), and partially funded by Samsung Electronics of Amazonia Ltda., under the terms of Federal Law \(\text {n}^{\underline {\text {o}}}\); 8.387/1991, through agreements 001/2020 and 003/2019, signed with Federal University of Amazonas and FAEPI, Brazil.
Author information
Authors and Affiliations
Contributions
L. Rodrigues: Conceptualization, Methodology, Formal Analysis, Investigation, Data Curation, Writing - Original Draft, Visualisation; P. T. Palomino: Conceptualization, Methodology, Validation, Data Curation, Writing - Review & Editing; A. M. Toda: Conceptualization, Methodology, Validation, Data Curation, Writing - Review & Editing; A. C. T. Klock: Conceptualization, Methodology, Validation, Writing - Review & Editing; M. Pessoa: Resources, Data Curation, Writing - Review & Editing; F. D. Pereira: Resources, Data Curation, Writing - Review & Editing; E. H. T. Oliveira: Resources, Writing - Review & Editing; D. F. Oliveira: Resources, Writing - Review & Editing; A. I. Cristea: Validation, Writing - Review & Editing; I. Gasparini: Validation, Writing - Review & Editing; S. Isotani: Conceptualization, Methodology, Writing - Review & Editing, Supervision
Corresponding author
Ethics declarations
Competing interests
The authors have no relevant financial or non-financial interests to disclose.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Rodrigues, L., Palomino, P.T., Toda, A.M. et al. How Personalization Affects Motivation in Gamified Review Assessments. Int J Artif Intell Educ 34, 147–184 (2024). https://doi.org/10.1007/s40593-022-00326-x
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40593-022-00326-x