Full length article
Combining factor analysis with writing analytics for the formative assessment of written reflection

https://doi.org/10.1016/j.chb.2021.106733Get rights and content

Highlights

  • Presentation of a five-factor assessment model (CWRef) for written reflection.

  • Alignment of factors with textual features extracted from writing analytics.

  • Empirical evaluation of CWRef using factor analysis in reflective writing contexts. .

  • Implication for automatic formative assessment/feedback of written reflection.

Abstract

The formative assessment of written reflection provides opportunities for students to improve their practice in an iterative manner using reflective writing. However, manual formative assessment of written reflection is time consuming and subjective. While progress has been made in deploying writing analytics tools to provide automated, formative feedback, few approaches to automated assessment are grounded in a validated, theory-based, formative assessment model. To address this, we propose a five-factor model of the Capability for Written Reflection (CWRef), grounded in the scholarship of reflective writing pedagogy. This paper uses Confirmatory Factor Analysis to validate the CWRef model by examining the relative contributions of textual features, derived from writing analytics, to each factor in the model, and their contributions to CWRef. The model was evaluated with two reflective writing corpora, showing which textual features, derived using Academic Writing Analytics and Linguistic Inquiry & Word Count, were significant indicators of factors in both corpora. In addition, it was found that the reflective writing context was an important factor influencing the validity of the CWRef model. Finally, we consider how this new analytical assessment model could enable improved tracking of progression in reflective writing, providing the basis for improved formative feedback.

Introduction

Expectations are growing regarding the knowledge, skills and dispositions that university graduates should be able to demonstrate in readiness for a fast-changing job market. In response, universities seek increasingly to provide learners with more authentic assessments, that require them to display transferrable skill sets that are often referred to as “Graduate Attributes” (GAs), or “21st Century Competencies”, in conjunction with the more discipline-specific abilities that we have come to expect from higher education. A range of approaches are being pursued in the sector, distinctive for the rich, embodied and complex challenges that they provide, in both the social and psychological realms. This raises the question of how to track the emerging competencies of our students, who may not even be directly observable (e.g. while on internship in a company, or working in teams across multiple locations and timezones). One approach is to ensure that assessment criteria incorporate GAs, so that they can be modelled, tracked and reported across diverse assignments (e.g. Thompson, 2016). Another approach is to design more authentic assessments, following principles such as the encouragement of reflexivity and the development of evaluative judgement identified by Herrington and Herrington (2005). A more technical approach is the use of activity-based Learning Analytics, combining mobile, multimodal sensors and analytics to track embodied activity and physiological data, in combination with more conventional data from platform-mediated interaction (e.g. Ochoa & Worsley, 2016). Each of these brings their own strengths and weaknesses to educators and learners, in terms of the insights they can offer, their technical complexity, and the literacies that students and educators require.

Applicable to all of these approaches is the well-known adage that summarizes Dewey's (1933) foundational work on teaching and learning, “We do not learn from experience … we learn from reflecting on experience”. Critical self-reflection has been recognized increasingly as central to the development of agentic, self-regulated learners. When students engage meaningfully in reflection, they review the way they perceive events and issues, their beliefs, feelings and actions. Such reflective processes in learning have most impact when they are formative and future-oriented (Boud & Falchikov, 2006), which provides mechanisms to encourage meta-cognitive adaptation as students connect their thinking to the wider world (Gibson, Kitto, & Bruza, 2016).

Reflection is a complex, internal process, which leads us to an important question: how can educators gather reliable evidence of student reflection? In this regard, written reflection (in private journals, shared blogs, or formal assignments) is by far the most common approach adopted in higher education (although we must acknowledge that students often express reflective thought in other modalities, including audio/video records, giving a presentation, or re-enacting a critical incident for discussion). Reflective writing can be a powerful process for the writer, as well as capturing evidence of significant, even transformative, learning for a different reader. Consider these examples from the literature:

“It was a great surprise to me to realize that coordination was such an important aspect of engineering” (Reidsema, Goldsmith, & Mort, 2010, p.9, p.9)

“Before I came to this class I had never really thought much about gender and what it means or that it is something that is fluid. Taking this course was completely eye opening and really made me think about things I have never had the chance to think about.” (Buckingham Shum, Sándor, Goldsmith, Bass, & McWilliams, 2017, p.76, p.76)

“I had never previously given thought to this idea, as I had thought that a patient's medications and medical conditions are fine to discuss with other family members.” (Lucas, Gibson, & Buckingham Shum, 2019, p.1267, p.1267)

Despite its evident potential, a growing body of evidence shows that students find reflective writing hard to learn, and moreover, that educators (who often include casual tutors and teaching assistants) also find it hard to teach and assess (Ryan, 2013). Writing in the first person, acknowledging uncertainties and failures, disclosing emotions and feelings, and showing insight into how one is changing as a learner and professional, is an unfamiliar genre for many educators and students. Writing in this way challenges students to share their weaknesses, which goes against almost every other educational experience and form of assessment they have been schooled in. Furthermore, there are rarely clearly ‘correct’ answers as to how one should act in complex human dilemmas, or how one should make sense of an experience. On what basis, therefore, can written reflection be assessed, and how will students know what the difference is between good and poor reflection?

Research into written reflection for learning has devoted much attention to these questions. One strand of work has focused on the evaluation of individual written reflection on a single scale, such as Mezirow's (1991) three levels of reflection: non-reflection, reflection and critical reflection (and see also Plack et al., 2007; Wong, Kember, Chung, & Yan, 1995). This evaluation is often based on the presence of multiple reflective elements, such as the description, feelings and outcomes elements in Boud et al.‘s. (1985) reflection model, or in a modified Bloom's taxonomy (Plack et al., 2007). Other research adds a more formative assessment dimension, where a written reflection can be assessed based on the presence of several important reflective elements, and the assessment of the depth of each (Birney, 2012; Lucas et al., 2017; Poldner, Schaaf, Simons, Tartwijk, & Wijngaards, 2014). These approaches seek indicators of both the overall depth of reflection, and individual aspects of reflection. These frameworks provide the language we need to talk more precisely about what good reflective writing looks like, as a proxy for the quality of the author's reflection. However, a significant limiting factor impedes both the empirical validation and the wider adoption of these frameworks in teaching practice: assessing reflective writing is extremely time-consuming.

Learning Analytics is defined in 2011 on the First LAK conference (https://tekri.athabascau.ca/analytics/) as “the measurement, collection, analysis and reporting to data about learners and their contexts, for purposes of understanding and optimizing learning the environments in which it occurs”. While it offers a new generation of tools for educational and learning science researchers to study learning processes, when deployed as an educational technology tool, it also enables new ways to augment learning and teaching as it unfolds, by closing the feedback loop to educators and students. Specifically, Writing Analytics (Buckingham Shum et al., 2016) emphasises the analysis of written text for the purpose of generating automated feedback to support personal learning, and within that field, Reflective Writing Analytics (RWA) uses recent advances in text analytics (i) to automatically identify reflective elements at the level of sentence segment level (e.g. Kovanović et al., 2018) or sentences (e.g. Gibson et al., 2017; Ullmann, 2015, 2019), and (ii) to evaluate reflection depth, at either the sentence level (e.g. Ullmann, 2019) or document level (e.g. Liu, Buckingham Shum, Mantzourani, & Lucas, 2019). Compared to other established fields, such as automated essay evaluation, Writing Analytics focuses on not only the computational evaluation of the students’ written text, but also the learning design for better integration of the writing analytics tools into classrooms (Liu, Goldsmith, Ahuja, & Huang, 2019; Shibani, Knight, Buckingham Shum, & Ryan, 2017).

Recently, Jung and Wise (2020) developed a multi-label classifier which extracted more than 100 textual features from a reflective statement, comparing them with the reflective elements that were identified and evaluated at the document level. These machine learning approaches corroborate earlier corpus-based studies reporting that some of these linguistic textual features were important indicators for the quality of written reflections (Birney, 2012). In particular, Cui, Wise, and Allen (2019) proposed a theoretical framework for reflective writing analytics which attempted to link textual features to conceptual elements of reflection. Despite this conceptual advance, that model fails to elaborate upon how strongly the identified textual features affect the quality of the reflective elements, or indeed, whether they impact upon the final quality of the overall reflection.

The aim of this paper is twofold. Firstly, we combine these two streams of work (written reflection assessment and writing analytics) by synthesizing a theoretical assessment model for what we term the Capability for Written Reflection (CWRef). Secondly, this is evaluated using confirmatory factor analysis that links the textual features that can be extracted automatically from texts using writing analytics, to CWRef. We will argue that the analytic model proposed here is more explainable than reflective element classification (e.g. Ullmann, 2019) or depth detection (e.g. Jung & Wise, 2020) because the model measures not only the overall reflection depth of a document, but also the depth of the individual latent reflective factors underpinning this overall assessment — which parts of the writing are stronger and weaker. We will argue that this therefore provides new possibilities for the formative assessment of written reflection.

Two research questions drive the work reported here:

RQ1: How can we quantify and validate the relative contributions that textual features make to the different latent factors underpinning the quality of written reflection?

RQ2: To what degree does this model of reflective writing generalise to different reflective writing contexts?

We make two contributions in responding to these questions. Firstly, we contribute to writing analytics by extending Cui et al.’s (2019) work, which linked low level textual features to reflective elements. We add higher order rhetorical move textual features, and then quantify and validate the relative contributions of these features to the different reflective elements or factors through confirmatory factor analysis, which extends previous work on reflection detection (Jung & Wise, 2020; Kovanović et al., 2018; Liu,; Buckingham Shum, Ferguson, & Martinez-Maldonado, 2019; Ullmann, 2019). Secondly, we contribute to the assessment of written reflection by providing a method for automating this process. In comparison with Birney's (2012) work, which developed a reflective writing assessment instrument based on the judgement of human experts, we propose an automated writing analytics approach. We develop a model comprising five factors, whereby each factor is correlated with textual features that can be extracted using writing analytics. Both the model and the textual features it relies upon are then evaluated based on two reflective writing datasets.

In the remainder of the paper, Section 2 reviews in more depth the existing literature and frameworks related to reflection quality assessment and reflective writing analytics, and describes a more practical written reflection evaluation model, called Capability of Written Reflection, derived from this literature. Section 3 describes the methodology linking writing analytics to this new model. Sections 4 Reflection contexts: Pharmacy and Data Science masters, 5 Results, 6 Discussion present the empirical validation of this model against two writing datasets from Pharmacy and Data Science postgraduate students. Section 7 discusses how this approach could in principle help to improve automated feedback, before Section 8 identifies directions for future work.

Section snippets

Synthesizing the literature to derive a model of written reflection

This section reviews literature on reflection models for assessing the quality of written reflection, from which is synthesised a practical written reflection assessment model. This provides the conceptual foundation for making sense of the textual features that reflective writing analytics can identify.

Methodology

This section first describes the concept of Confirmatory Factor Analysis (CFA), before detailing a methodology which shows how CFA was used to link recent advances in writing analytics to the formative assessment of reflective writing.

Reflection contexts: Pharmacy and Data Science masters

This section describes the empirical evaluation of the CWRef model that was performed using two independent datasets collected in authentic reflective learning environments. We followed the process described in the previous section to fit CWRef to these two datasets, each of which was generated from different learning designs and assessment regimes for reflective writing, described next.

Results

The goodness-of-fit indices shown in Table 5 from the CFA demonstrate a strong fit of the data collected in the Pharmacy and Data Science contexts to the five-factor measurement model. The RMSEA values are 0.071 in Pharmacy and .062 in Data Science, which is considered an acceptable fit (Fabrigar, MacCallum, Wegener, & Strahan, 1999). The CFI in Pharmacy and Data Science exceeds 0.9, and the TLI in Pharmacy, 0.865, is close to 0.9. Based on these indices, these two samples can be said to

Discussion

We now return to our research questions to reflect upon what we have learned during our investigations of CWRef in two authentic learning contexts.

This paper has proposed a four-stage process that links writing behaviours extracted from student texts using writing analytics, to four latent written reflection factors. These have been demonstrated to contribute to a second-order reflective thinking factor, using confirmatory factor analysis to evaluate the validity and reliability of the

Implications for improving automated feedback

The principal goal of writing analytics is not just to automatically assess texts, but to deeply understand students’ potential reflective thinking skills in a learning context, such that we are able to generate actionable feedback to improve their writing (Simon Buckingham Shum et al., 2016; Knight, Buckingham Shum, Ryan, Sándor, & Wang, 2018; Lucas, Gibson, & Buckingham Shum, 2019). We now turn to a discussion of how the CWRef model might be used to improve feedback.

Hattie and Timperley (2007)

Conclusions, limitations and future work

Reflective writing is a widespread practice to help learners reflect on challenging experiences. However, the evidence is that it is both challenging to teach, and to learn. Furthermore, the assessment of reflective writing is quite different to that of other genres. Central to improving reflective writing (as with any skill) is the provision of timely, actionable feedback (Lucas, Gibson, & Buckingham Shum, 2019; Lucas, Smith, et al., 2019). Automating textual analysis approaches opens new

References (86)

  • J.T. Behrens et al.

    Assessment of complex performances in digital environments

    The Annals of the American Academy of Political and Social Science

    (2019)
  • P.M. Bentler

    Comparative fit indexes in structural models

    Psychological Bulletin

    (1990)
  • R. Birney

    Reflective writing: Quantitative assessment and identification of linguistic features [Waterford Institute of technology]

    (2012)
  • K. Bollen

    Structural equations with latent variables. In structural equation models

    (1989)
  • D. Boud

    Reflection. Turning experience into learning

    (1985)
  • D. Boud et al.

    Aligning assessment with long-term learning

    Assessment & Evaluation in Higher Education

    (2006)
  • D. Boud et al.

    Reflection: Turning experience into learning

    (1985)
  • S. Brookfield

    Becoming a critically reflective teacher

    (1995)
  • S. Buckingham Shum et al.

    Learning analytics for 21st Century competencies

    Journal of Learning Analytics

    (2016)
  • S. Buckingham Shum et al.

    Human-centred learning analytics

    Journal of Learning Analytics

    (2019)
  • S. Buckingham Shum et al.

    Critical perspectives on writing analytics

  • S. Buckingham Shum et al.

    Towards reflective writing analytics: Rationale, methodology and preliminary results

    Journal of Learning Analytics

    (2017)
  • B.M. Byrne

    Structural equation modeling with AMOS: Basic concepts, applications, and programming, second edition. Structural equation modeling with AMOS: Basic concepts, Applications, and Programming

    (2013)
  • J. Dewey

    How we think. In Great books in philosophy

    (1933)
  • H. Duijnhouwer et al.

    Progress feedback effects on students' writing mastery goal, self-efficacy beliefs, and performance

    Educational Research and Evaluation

    (2010)
  • L.R. Fabrigar et al.

    Evaluating the use of exploratory factor analysis in psychological research

    Psychological Methods

    (1999)
  • E. Fincham et al.

    Counting clicks is not enough: Validating a theorized model of engagement in learning analytics

  • C. Fornell et al.

    Evaluating structural equation models with unobservable variables and measurement error

    Journal of Marketing Research

    (1981)
  • G. Gibbs

    Learning by doing: A guide to teaching and learning methods

    (1988)
  • A. Gibson et al.

    Reflective writing analytics for actionable feedback

    Proceedings of the Seventh International Learning Analytics & Knowledge Conference on - LAK

    (2017)
  • A. Gibson et al.

    Towards the discovery of learner metacognition from reflective writing

    Journal of Learning Analytics

    (2016)
  • A.C. Graesser et al.

    Coh-metrix: Analysis of text on cohesion and language

    Behavior Research Methods Instruments & Computers: A Journal of the Psychonomic Society, Inc

    (2004)
  • J. Hair et al.

    Multivariate data analysis: A Global perspective. In multivariate data analysis: A Global perspective

    (2010)
  • J.F. Hair et al.

    Partial least squares structural equation modeling (PLS-SEM)

    (2014)
  • J. Hattie et al.

    The power of feedback

    Review of Educational Research

    (2007)
  • T. Herrington et al.

    Authentic learning environments in higher education

    (2005)
  • A.E. Hurley et al.

    Exploratory and confirmatory factor analysis: Guidelines, issues, and alternatives

    Journal of Organizational Behavior

    (1997)
  • Y. Jung et al.

    How and how well do students Reflect ?: Multi-dimensional automated reflection assessment in health professions education

    Proceedings of the 10th International Conference on Learning Analytics & Knowledge (LAK’20)

    (2020)
  • D. Kember et al.

    Development of a questionnaire to measure the level of reflective thinking

    Assessment & Evaluation in Higher Education

    (2000)
  • R.B. Kline

    Principles and practice of structural equation modeling

  • S. Knight et al.

    Designing academic writing analytics for civil law student self-assessment

    International Journal of Artificial Intelligence in Education

    (2018)
  • S. Knight et al.

    AcaWriter: A learning analytics tool for formative feedback on academic writing

    Journal of Writing Research

    (2020)
  • D.A. Kolb

    Experiential learning: Experience as the source of learning and development

    (1984)
  • Cited by (18)

    • Adaptive formative assessment system based on computerized adaptive testing and the learning memory cycle for personalized learning

      2022, Computers and Education: Artificial Intelligence
      Citation Excerpt :

      It can dynamically adjust student and item parameters during the test; thus, it is more convenient for use in practical settings. Other approaches for modeling students' knowledge acquisition, such as knowledge tracing (Zhou et al., 2021) and performance factor analysis (Liu et al., 2021), have been developed; however, these methods are impractical because they employ complex models and require additional information for parameter calibration. Although the aforementioned CAT-based techniques have been used for creating formative assessments, most of them have been used for student evaluation rather than for learning (McCallum & Milner, 2021).

    View all citing articles on Scopus
    View full text