Full length articleCombining factor analysis with writing analytics for the formative assessment of written reflection
Introduction
Expectations are growing regarding the knowledge, skills and dispositions that university graduates should be able to demonstrate in readiness for a fast-changing job market. In response, universities seek increasingly to provide learners with more authentic assessments, that require them to display transferrable skill sets that are often referred to as “Graduate Attributes” (GAs), or “21st Century Competencies”, in conjunction with the more discipline-specific abilities that we have come to expect from higher education. A range of approaches are being pursued in the sector, distinctive for the rich, embodied and complex challenges that they provide, in both the social and psychological realms. This raises the question of how to track the emerging competencies of our students, who may not even be directly observable (e.g. while on internship in a company, or working in teams across multiple locations and timezones). One approach is to ensure that assessment criteria incorporate GAs, so that they can be modelled, tracked and reported across diverse assignments (e.g. Thompson, 2016). Another approach is to design more authentic assessments, following principles such as the encouragement of reflexivity and the development of evaluative judgement identified by Herrington and Herrington (2005). A more technical approach is the use of activity-based Learning Analytics, combining mobile, multimodal sensors and analytics to track embodied activity and physiological data, in combination with more conventional data from platform-mediated interaction (e.g. Ochoa & Worsley, 2016). Each of these brings their own strengths and weaknesses to educators and learners, in terms of the insights they can offer, their technical complexity, and the literacies that students and educators require.
Applicable to all of these approaches is the well-known adage that summarizes Dewey's (1933) foundational work on teaching and learning, “We do not learn from experience … we learn from reflecting on experience”. Critical self-reflection has been recognized increasingly as central to the development of agentic, self-regulated learners. When students engage meaningfully in reflection, they review the way they perceive events and issues, their beliefs, feelings and actions. Such reflective processes in learning have most impact when they are formative and future-oriented (Boud & Falchikov, 2006), which provides mechanisms to encourage meta-cognitive adaptation as students connect their thinking to the wider world (Gibson, Kitto, & Bruza, 2016).
Reflection is a complex, internal process, which leads us to an important question: how can educators gather reliable evidence of student reflection? In this regard, written reflection (in private journals, shared blogs, or formal assignments) is by far the most common approach adopted in higher education (although we must acknowledge that students often express reflective thought in other modalities, including audio/video records, giving a presentation, or re-enacting a critical incident for discussion). Reflective writing can be a powerful process for the writer, as well as capturing evidence of significant, even transformative, learning for a different reader. Consider these examples from the literature:
“It was a great surprise to me to realize that coordination was such an important aspect of engineering” (Reidsema, Goldsmith, & Mort, 2010, p.9, p.9)
“Before I came to this class I had never really thought much about gender and what it means or that it is something that is fluid. Taking this course was completely eye opening and really made me think about things I have never had the chance to think about.” (Buckingham Shum, Sándor, Goldsmith, Bass, & McWilliams, 2017, p.76, p.76)
“I had never previously given thought to this idea, as I had thought that a patient's medications and medical conditions are fine to discuss with other family members.” (Lucas, Gibson, & Buckingham Shum, 2019, p.1267, p.1267)
Despite its evident potential, a growing body of evidence shows that students find reflective writing hard to learn, and moreover, that educators (who often include casual tutors and teaching assistants) also find it hard to teach and assess (Ryan, 2013). Writing in the first person, acknowledging uncertainties and failures, disclosing emotions and feelings, and showing insight into how one is changing as a learner and professional, is an unfamiliar genre for many educators and students. Writing in this way challenges students to share their weaknesses, which goes against almost every other educational experience and form of assessment they have been schooled in. Furthermore, there are rarely clearly ‘correct’ answers as to how one should act in complex human dilemmas, or how one should make sense of an experience. On what basis, therefore, can written reflection be assessed, and how will students know what the difference is between good and poor reflection?
Research into written reflection for learning has devoted much attention to these questions. One strand of work has focused on the evaluation of individual written reflection on a single scale, such as Mezirow's (1991) three levels of reflection: non-reflection, reflection and critical reflection (and see also Plack et al., 2007; Wong, Kember, Chung, & Yan, 1995). This evaluation is often based on the presence of multiple reflective elements, such as the description, feelings and outcomes elements in Boud et al.‘s. (1985) reflection model, or in a modified Bloom's taxonomy (Plack et al., 2007). Other research adds a more formative assessment dimension, where a written reflection can be assessed based on the presence of several important reflective elements, and the assessment of the depth of each (Birney, 2012; Lucas et al., 2017; Poldner, Schaaf, Simons, Tartwijk, & Wijngaards, 2014). These approaches seek indicators of both the overall depth of reflection, and individual aspects of reflection. These frameworks provide the language we need to talk more precisely about what good reflective writing looks like, as a proxy for the quality of the author's reflection. However, a significant limiting factor impedes both the empirical validation and the wider adoption of these frameworks in teaching practice: assessing reflective writing is extremely time-consuming.
Learning Analytics is defined in 2011 on the First LAK conference (https://tekri.athabascau.ca/analytics/) as “the measurement, collection, analysis and reporting to data about learners and their contexts, for purposes of understanding and optimizing learning the environments in which it occurs”. While it offers a new generation of tools for educational and learning science researchers to study learning processes, when deployed as an educational technology tool, it also enables new ways to augment learning and teaching as it unfolds, by closing the feedback loop to educators and students. Specifically, Writing Analytics (Buckingham Shum et al., 2016) emphasises the analysis of written text for the purpose of generating automated feedback to support personal learning, and within that field, Reflective Writing Analytics (RWA) uses recent advances in text analytics (i) to automatically identify reflective elements at the level of sentence segment level (e.g. Kovanović et al., 2018) or sentences (e.g. Gibson et al., 2017; Ullmann, 2015, 2019), and (ii) to evaluate reflection depth, at either the sentence level (e.g. Ullmann, 2019) or document level (e.g. Liu, Buckingham Shum, Mantzourani, & Lucas, 2019). Compared to other established fields, such as automated essay evaluation, Writing Analytics focuses on not only the computational evaluation of the students’ written text, but also the learning design for better integration of the writing analytics tools into classrooms (Liu, Goldsmith, Ahuja, & Huang, 2019; Shibani, Knight, Buckingham Shum, & Ryan, 2017).
Recently, Jung and Wise (2020) developed a multi-label classifier which extracted more than 100 textual features from a reflective statement, comparing them with the reflective elements that were identified and evaluated at the document level. These machine learning approaches corroborate earlier corpus-based studies reporting that some of these linguistic textual features were important indicators for the quality of written reflections (Birney, 2012). In particular, Cui, Wise, and Allen (2019) proposed a theoretical framework for reflective writing analytics which attempted to link textual features to conceptual elements of reflection. Despite this conceptual advance, that model fails to elaborate upon how strongly the identified textual features affect the quality of the reflective elements, or indeed, whether they impact upon the final quality of the overall reflection.
The aim of this paper is twofold. Firstly, we combine these two streams of work (written reflection assessment and writing analytics) by synthesizing a theoretical assessment model for what we term the Capability for Written Reflection (CWRef). Secondly, this is evaluated using confirmatory factor analysis that links the textual features that can be extracted automatically from texts using writing analytics, to CWRef. We will argue that the analytic model proposed here is more explainable than reflective element classification (e.g. Ullmann, 2019) or depth detection (e.g. Jung & Wise, 2020) because the model measures not only the overall reflection depth of a document, but also the depth of the individual latent reflective factors underpinning this overall assessment — which parts of the writing are stronger and weaker. We will argue that this therefore provides new possibilities for the formative assessment of written reflection.
Two research questions drive the work reported here:
RQ1: How can we quantify and validate the relative contributions that textual features make to the different latent factors underpinning the quality of written reflection?
RQ2: To what degree does this model of reflective writing generalise to different reflective writing contexts?
We make two contributions in responding to these questions. Firstly, we contribute to writing analytics by extending Cui et al.’s (2019) work, which linked low level textual features to reflective elements. We add higher order rhetorical move textual features, and then quantify and validate the relative contributions of these features to the different reflective elements or factors through confirmatory factor analysis, which extends previous work on reflection detection (Jung & Wise, 2020; Kovanović et al., 2018; Liu,; Buckingham Shum, Ferguson, & Martinez-Maldonado, 2019; Ullmann, 2019). Secondly, we contribute to the assessment of written reflection by providing a method for automating this process. In comparison with Birney's (2012) work, which developed a reflective writing assessment instrument based on the judgement of human experts, we propose an automated writing analytics approach. We develop a model comprising five factors, whereby each factor is correlated with textual features that can be extracted using writing analytics. Both the model and the textual features it relies upon are then evaluated based on two reflective writing datasets.
In the remainder of the paper, Section 2 reviews in more depth the existing literature and frameworks related to reflection quality assessment and reflective writing analytics, and describes a more practical written reflection evaluation model, called Capability of Written Reflection, derived from this literature. Section 3 describes the methodology linking writing analytics to this new model. Sections 4 Reflection contexts: Pharmacy and Data Science masters, 5 Results, 6 Discussion present the empirical validation of this model against two writing datasets from Pharmacy and Data Science postgraduate students. Section 7 discusses how this approach could in principle help to improve automated feedback, before Section 8 identifies directions for future work.
Section snippets
Synthesizing the literature to derive a model of written reflection
This section reviews literature on reflection models for assessing the quality of written reflection, from which is synthesised a practical written reflection assessment model. This provides the conceptual foundation for making sense of the textual features that reflective writing analytics can identify.
Methodology
This section first describes the concept of Confirmatory Factor Analysis (CFA), before detailing a methodology which shows how CFA was used to link recent advances in writing analytics to the formative assessment of reflective writing.
Reflection contexts: Pharmacy and Data Science masters
This section describes the empirical evaluation of the CWRef model that was performed using two independent datasets collected in authentic reflective learning environments. We followed the process described in the previous section to fit CWRef to these two datasets, each of which was generated from different learning designs and assessment regimes for reflective writing, described next.
Results
The goodness-of-fit indices shown in Table 5 from the CFA demonstrate a strong fit of the data collected in the Pharmacy and Data Science contexts to the five-factor measurement model. The RMSEA values are 0.071 in Pharmacy and .062 in Data Science, which is considered an acceptable fit (Fabrigar, MacCallum, Wegener, & Strahan, 1999). The CFI in Pharmacy and Data Science exceeds 0.9, and the TLI in Pharmacy, 0.865, is close to 0.9. Based on these indices, these two samples can be said to
Discussion
We now return to our research questions to reflect upon what we have learned during our investigations of CWRef in two authentic learning contexts.
This paper has proposed a four-stage process that links writing behaviours extracted from student texts using writing analytics, to four latent written reflection factors. These have been demonstrated to contribute to a second-order reflective thinking factor, using confirmatory factor analysis to evaluate the validity and reliability of the
Implications for improving automated feedback
The principal goal of writing analytics is not just to automatically assess texts, but to deeply understand students’ potential reflective thinking skills in a learning context, such that we are able to generate actionable feedback to improve their writing (Simon Buckingham Shum et al., 2016; Knight, Buckingham Shum, Ryan, Sándor, & Wang, 2018; Lucas, Gibson, & Buckingham Shum, 2019). We now turn to a discussion of how the CWRef model might be used to improve feedback.
Hattie and Timperley (2007)
Conclusions, limitations and future work
Reflective writing is a widespread practice to help learners reflect on challenging experiences. However, the evidence is that it is both challenging to teach, and to learn. Furthermore, the assessment of reflective writing is quite different to that of other genres. Central to improving reflective writing (as with any skill) is the provision of timely, actionable feedback (Lucas, Gibson, & Buckingham Shum, 2019; Lucas, Smith, et al., 2019). Automating textual analysis approaches opens new
References (86)
The use of reflective journals in the promotion of reflection and learning in post-registration nursing students
Nurse Education Today
(2007)- et al.
Developing reflection analytics for health professions education: A multi-dimensional framework to align critical concepts with data features
Computers in Human Behavior
(2019) - et al.
Development of a template to facilitate reflection among student pharmacists
Research in Social and Administrative Pharmacy
(2018) - et al.
Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success
Internet and Higher Education
(2016) - et al.
Inter-rater reliability of a reflective rubric to assess pharmacy students' reflective thinking
Currents in Pharmacy Teaching and Learning
(2017) - et al.
Pharmacy students' utilization ofc an online tool for immediate formative feedback on reflective writing tasks
American Journal of Pharmaceutical Education
(2019) - et al.
Can a reflective rubric be applied consistently with raters globally? A study across three countries
Currents in Pharmacy Teaching and Learning
(2019) Assessing teaching practicum reflections: Distinguishing discourse features of the “high” and “low” grade reports
System
(2008)- et al.
Assessing reflective writing on a pediatric clerkship by using a modified Bloom's taxonomy
Ambulatory Pediatrics
(2007) - et al.
A model for assessing reflective practices in pharmacy education
American Journal of Pharmaceutical Education
(2015)
Assessment of complex performances in digital environments
The Annals of the American Academy of Political and Social Science
Comparative fit indexes in structural models
Psychological Bulletin
Reflective writing: Quantitative assessment and identification of linguistic features [Waterford Institute of technology]
Structural equations with latent variables. In structural equation models
Reflection. Turning experience into learning
Aligning assessment with long-term learning
Assessment & Evaluation in Higher Education
Reflection: Turning experience into learning
Becoming a critically reflective teacher
Learning analytics for 21st Century competencies
Journal of Learning Analytics
Human-centred learning analytics
Journal of Learning Analytics
Critical perspectives on writing analytics
Towards reflective writing analytics: Rationale, methodology and preliminary results
Journal of Learning Analytics
Structural equation modeling with AMOS: Basic concepts, applications, and programming, second edition. Structural equation modeling with AMOS: Basic concepts, Applications, and Programming
How we think. In Great books in philosophy
Progress feedback effects on students' writing mastery goal, self-efficacy beliefs, and performance
Educational Research and Evaluation
Evaluating the use of exploratory factor analysis in psychological research
Psychological Methods
Counting clicks is not enough: Validating a theorized model of engagement in learning analytics
Evaluating structural equation models with unobservable variables and measurement error
Journal of Marketing Research
Learning by doing: A guide to teaching and learning methods
Reflective writing analytics for actionable feedback
Proceedings of the Seventh International Learning Analytics & Knowledge Conference on - LAK
Towards the discovery of learner metacognition from reflective writing
Journal of Learning Analytics
Coh-metrix: Analysis of text on cohesion and language
Behavior Research Methods Instruments & Computers: A Journal of the Psychonomic Society, Inc
Multivariate data analysis: A Global perspective. In multivariate data analysis: A Global perspective
Partial least squares structural equation modeling (PLS-SEM)
The power of feedback
Review of Educational Research
Authentic learning environments in higher education
Exploratory and confirmatory factor analysis: Guidelines, issues, and alternatives
Journal of Organizational Behavior
How and how well do students Reflect ?: Multi-dimensional automated reflection assessment in health professions education
Proceedings of the 10th International Conference on Learning Analytics & Knowledge (LAK’20)
Development of a questionnaire to measure the level of reflective thinking
Assessment & Evaluation in Higher Education
Principles and practice of structural equation modeling
Designing academic writing analytics for civil law student self-assessment
International Journal of Artificial Intelligence in Education
AcaWriter: A learning analytics tool for formative feedback on academic writing
Journal of Writing Research
Experiential learning: Experience as the source of learning and development
Cited by (18)
Towards strengthening links between learning analytics and assessment: Challenges and potentials of a promising new bond
2022, Computers in Human BehaviorAdaptive formative assessment system based on computerized adaptive testing and the learning memory cycle for personalized learning
2022, Computers and Education: Artificial IntelligenceCitation Excerpt :It can dynamically adjust student and item parameters during the test; thus, it is more convenient for use in practical settings. Other approaches for modeling students' knowledge acquisition, such as knowledge tracing (Zhou et al., 2021) and performance factor analysis (Liu et al., 2021), have been developed; however, these methods are impractical because they employ complex models and require additional information for parameter calibration. Although the aforementioned CAT-based techniques have been used for creating formative assessments, most of them have been used for student evaluation rather than for learning (McCallum & Milner, 2021).
Evaluating Reflective Writing in Pre-Service Teachers: The Potential of a Mixed-Methods Approach
2023, Education SciencesImpact of combining human and analytics feedback on students’ engagement with, and performance in, reflective writing tasks
2023, International Journal of Educational Technology in Higher EducationUsing causal models to bridge the divide between big data and educational theory
2023, British Journal of Educational TechnologyPrompting Reflection: Using Corpus Linguistic Methods in the Local Assessment of Reflective Writing
2023, Written Communication