A review of automated feedback systems for learners: Classification framework, challenges and opportunities
Introduction
Providing feedback is a critical factor for improving knowledge and acquiring skills. If such feedback is provided when learners need it, it can considerably improve the quality and speed of learning processes (Economides, 2005; Hattie & Timperley, 2007). However, students report serious deficiencies in the amount and quality of feedback they receive (Boud & Molloy, 2013; Ferguson, 2011). Delivering real-time feedback at an individual level is often infeasible, considering the limited teaching resources and the heterogeneous nature of the students’ profiles (Boud & Molloy, 2013; Pardo et al., 2019).
Recent technological advancements have caused a widespread uptake of various technologies in the field of education, e.g. Intelligent Tutoring Systems (ITS), which are intended to produce personalized feedback in an automated way (Pardo et al., 2019). Such systems support learning processes at any time, given that electronic devices and internet connection are broadly accessible to a large number of people (Hwang, 2014). Throughout the history of computer tutoring, it was common to express certain concerns regarding its effectiveness in general, as well as in comparison to human tutoring. For example, Bloom (1984) argued that human tutors provide a larger difference in the learning gains compared to computer tutors. However, as the remarkable advancements of computer technologies in the last decades allow for developing better and more capable computer tutors, more recent works confirm that the effect of intelligent tutoring systems providing automated feedback is comparable to that of human tutoring (VanLehn, 2011). Furthermore, as Singh et al. (2013) highlight, technologies can offer quality education to numerous learners around the world. These advantages of technology-enhanced learning have led to more and more digital learning environments being developed and implemented.
Considering this wide variety of digital learning environments, there is a need to outline and analyze the current state of the field. Previously, there have been some studies that aimed to review the literature that tackles the topic of automated feedback technologies/systems (the terms “technology” and “system” are used interchangeably throughout this paper and address the same concept). For instance, Normadhi et al. (2019) presented a systematic literature review on the most common personal traits used to create learner models within e-learning systems, as well as methods for identification of such traits, thus focusing on a specific aspect of automated feedback system architecture. From a theoretical point of view, a conceptual framework for feedback automation in smart learning environments, providing various classification categories, was proposed by Serral and Snoeck (2016) (see also a later work in Serral Asensio et al. (2019)). In another literature review, Bimba et al. (2017) presented an overview of adaptive feedback in computer-based learning environments, including its target, means, goal and strategy. Similarly, in a systematic literature review by Martin et al. (2020), adaptive strategies and adaptive learning technologies are analyzed, emphasizing the content, instructional models and learner models used in the literature. Furthermore, given that automated feedback is commonly used in the field of programming education, partially due to the fact that many aspects of programming can be formalized and thus assessed automatically, there is a substantial number of studies that provide a review of the systems designed for automatic grading (Caiza & Del Alamo, 2013), or automated feedback in programming exercises (Keuning et al., 2018; Le et al., 2013). Similarly, other literature reviews have been focused on tutoring systems within a particular domain or educational task, such as teaching computer graphics (Suselo et al., 2019), science education (Zacharia et al., 2015) and foreign language learning (Golonka et al., 2014).
These reviews provide useful information on different aspects of automated feedback systems. However, as they are focused either on particular educational topics or on specific domains, a global perspective is missing. Having a global perspective is important for anyone involved in the development of an automated feedback system. Such systems are usually initially developed or piloted in a specific application domain, tackle different educational tasks, and often are designed in very distinctive ways. A review that is tailored to a specific application domain (e.g. programming), or focuses only on a limited set of properties of such systems holds the risk of missing interesting insights from systems developed in other domains and for other applications. This could lead to reinventing the wheel. Moreover, bringing together the knowledge on designing automated feedback systems that is currently scattered across different domains and applications could be beneficial for system developers, and might be a source of inspiration and cross-fertilization. For that reason, this study aims to approach a review of automated feedback systems in a more general way without focusing on a particular domain and looking into a large number of different aspects, ranging from the educational context to the evaluation of the system effectiveness, and thus develop a general classification framework for these systems. The importance and utility of such a framework is summarized as follows. First, by looking at systems across different domains, system developers could potentially get inspiration in using new methods and approaches typical for a certain field and less common in another. Moreover, certain domains might have their own standards when describing an automated feedback system, and, as such, a more general standard for reporting such systems would improve systems’ comparison.
As such, the core contributions of this study are:
- 1)
the development of a classification framework of automated feedback technologies;
- 2)
a systematic literature review of automated feedback technologies in the period from 2008 to 2019;
- 3)
a detailed overview of the predominantly available automated feedback technologies, the educational settings in which they are applied, the properties of automated feedback they deliver, and the approaches for their design and evaluation.
The findings of this study will be relevant for educators, researchers in the educational domain, as well as developers of new automated feedback systems, both as a source of inspiration for the systematic development of new systems and as an introduction to the field, especially for interdisciplinary researchers and educational technologists.
The remainder of this paper is organized as follows. In Section 2, the survey methodology is presented, including the research questions, the selection criteria and the search process. Next, Section 3 introduces the developed classification framework of automated feedback technologies. Consequently, Section 4 provides a classification of the selected papers according to the proposed framework. Finally, Sections 5 Discussion and recommendations, 6 Conclusion discuss the main findings, implications and limitations of the study, as well as provide general recommendations.
Section snippets
Research methodology
The methodology of this paper is based on the guidelines for performing systematic literature reviews proposed by Kitchenham (2004). Following these guidelines, in this section we describe the designed review protocol, which includes the research questions, the search process, the selection criteria and the selection process. Subsequently, the process of framework development based on design science research is discussed in detail. Finally, the stage of framework validation is presented.
Technologies for Automated Feedback - classification framework (TAF-ClaF)
In this section we discuss the classification framework (TAF-ClaF) developed for the purpose of this study, and thus address the Research Questions 2 and 3.
The classification framework is shown in Fig. 3. It consists of four main components: architecture, feedback, educational context and evaluation, each characterized by a number of dimensions. The first component, architecture, comprises essential elements of a technical implementation of a system with capability of providing feedback, such
Classification of the research on automated feedback technologies
This section provides a broad overview of the predominantly available systems in the field and their main properties, and thus answers the Research Question 1.
Discussion and recommendations
In this paper, we thoroughly explored the field of automated feedback technologies for learners. With regard to Research Question 1, Section 4 provides a broad overview of the predominantly available systems in the field. Moreover, the second and third research questions are addressed in Section 3, in which a classification framework was developed that allows to categorize these systems according to key dimensions. The overall framework is illustrated in Fig. 3. The remainder of this section
Conclusion
This study reports on a structured literature review of automated feedback technologies in education. After a four-phase search process, we selected and analyzed 109 papers. Based on this analysis, we identified the most relevant dimensions for classification of automated feedback technologies and combined them into a general classification framework. Additionally, we classified the selected papers according to the framework and presented an overview of recent trends in the field of educational
Credit author statement
Galina Deeva: Conceptualization, Methodology, Writing, Analysis, Review and Editing. Daria Bogdanova: Writing, Analysis, Review and Editing, Conceptualization. Estefania Serral: Conceptualization, Review and Editing. Monique Snoeck: Analysis, Review and Editing. Jochen De Weerdt: Review and Editing, Methodology, Validation.
Acknowledgements
This research was in part funded by an internal research project at KU Leuven with grant ID C24/16/002.
References (148)
How automated feedback through text mining changes plagiaristic behavior in online assignments
Computers & Education
(2015)- et al.
Domain-specific knowledge representation and inference engine for an intelligent tutoring system
Knowledge-Based Systems
(2013) The impact of online automated feedback on students' reflective journal writing in an efl course
The Internet and Higher Education
(2017)- et al.
An extension of the oele platform for generating semantic feedback for students and teachers
Procedia-Social and Behavioral Sciences
(2010) - et al.
Adaptive feedback selection for intelligent tutoring systems
Expert Systems with Applications
(2011) - et al.
Classification of motor errors to provide real-time feedback for sports coaching in virtual realitya case study in squats and tai chi pushes
Computers & Graphics
(2018) - et al.
Employing umls for generating hints in a tutoring system for medical problem-based learning
Journal of Biomedical Informatics
(2012) - et al.
Improving medical students’ awareness of their non-verbal communication through automated non-verbal behavior feedback
Frontiers in ICT
(2016) - et al.
Interactive stories and exercises with dynamic feedback for improving reading comprehension skills in deaf children
Computers & Education
(2013) - et al.
Learning Persian grammar with the aid of an intelligent feedback generator
Engineering Applications of Artificial Intelligence
(2016)
The effect of positive feedback in a constraint-based intelligent tutoring system
Computers & Education
Exploring feedback and student characteristics relevant for personalizing feedback strategies
Computers & Education
Providing graduated corrective feedback in an intelligent computer-assisted language learning environment
ReCALL
A taxonomy for learning, teaching, and assessing: A revision of bloom's taxonomy of educational objectives
Gui-driven intelligent tutoring system with affective support to help learning the algebraic method
A systematic, tool-supported method for conducting literature reviews in information systems
Intelligent tutoring system for learning digital systems on mooc environments
An automated feedback system based on adaptive testing: Extending the model
International Journal of Emerging Technologies in Learning (iJET)
Providing real-time feedback for student teachers in a virtual rehearsal environment
Learner modeling for adaptive scaffolding in a computational thinking-based science learning environment
User Modeling and User-Adapted Interaction
Personalized feedback for self assessment in lifelong learning environments based on semantic web
Computers in Human Behavior
The effect of a web-based coding tool with automatic feedback on students' performance and perceptions
Adaptive feedback in computer-based learning environments: A review
Adaptive Behavior
Higher education learner experience with fuzzy feedback in a digital learning environment
The 2 sigma problem: The search for methods of group instruction as effective as one-to-one tutoring
Educational Researcher
Rethinking models of feedback for learning: The challenge of design
Assessment & Evaluation in Higher Education
Some insights into the impact of affective information when delivering feedback to students
Behaviour & Information Technology
Programming assignments automatic grading: Review of tools and implementations
Students' conceptions of tutor and automated feedback in professional writing
Journal of Engineering Education
A virtual reality dance training system using motion capture technology
IEEE Transactions on Learning Technologies
Efficacy of an icall tutoring system and process-oriented corrective feedback
Computer Assisted Language Learning
An intelligent tutoring system for Japanese language particles with user assessment and feedback
Supporting Chinese character educational interfaces with richer assessment feedback through sketch recognition
Enhancing feedback to students in automated diagram assessment
Assessing whether students seek constructive criticism: The design of an automated feedback system for a graphic design task
International Journal of Artificial Intelligence in Education
Ontopefege: Ontology-based personalized feedback generator
IEEE Access
Be brief, and they shall learn: Generating concise language feedback for a computer tutor
International Journal of Artificial Intelligence in Education
Forró trainer: Automated feedback for partner dance learning
Data mining to generate individualised feedback
Beetle ii: Deep natural language understanding and automatic feedback generation for intelligent tutoring in basic electricity and electronics
International Journal of Artificial Intelligence in Education
Personalized feedback in cat
WSEAS Transactions on Advances in Engineering Education
Student perceptions of quality feedback in teacher education
Assessment & Evaluation in Higher Education
Instructional message design: Principles from the behavioral sciences
Data driven automatic feedback generation in the ilist intelligent tutoring system
Technology, Instruction, Cognition and Learning
Multi-paradigm generation of tutoring feedback in robotic arm manipulation training
Adaptive training environment without prior knowledge: Modeling feedback selection as a multi-armed bandit problem
Learners use of domain-specific computer-based feedback to overcome logical circularity in deductive proving in geometry
ZDM
Roc speak: Semi-automated personalized feedback on nonverbal behavior from recorded videos
Personalized versus collective instructor feedback in the online courseroom: Does type of feedback affect student satisfaction, academic performance and perceived connectedness with the instructor?
International Journal on E-Learning
Cited by (78)
Elementary English learners’ engagement with automated feedback
2024, Learning and InstructionPredictors of middle school students’ perceptions of automated writing evaluation
2024, Computers and EducationImpact of AI assistance on student agency
2024, Computers and EducationEnhancement of efficiency of the training process with the use of digital technologies
2023, Education for Chemical EngineersThe effect of feedback on academic achievement in technology-rich learning environments (TREs): A meta-analytic review
2023, Educational Research Review