Abstract
A legal casenote essay is a commonly assigned writing task to first-year law students aiming to promote their understanding of legal reasoning and help them acquire writing skills in a legal domain. To ensure law students master the legal casenote writing, it is critical that instructors monitor and evaluate students’ progress, and provide a timely and specific feedback. This is, however, a challenging task to many instructors as they often need to dedicate excessive time and effort to evaluate writing of and provide formative feedback to each individual student. We posit a computational tool that can afford at-scale evaluation of legal casenote writing may help remedy this challenge. Although quite some automatic writing evaluation (AWE) tools have been applied in the domain of education, the AWE tool that can analyse rhetoric of a legal casenote essay (i.e., specific rhetorical elements required by this task) is yet to be developed. We made the first step towards developing such a tool. We manually annotated each sentence in a corpus of 1,020 authentic casenote essays written over 6 offerings of the first-year legal writing course and developed one traditional machine learning classifier (Random Forest) and two deep learning classifiers (based on vanilla BERT and Legal BERT pre-trained language models). We found that the deep learning classifier based on Legal BERT could correctly identify more than 86% of rhetorical moves in a casenote. Our findings may be of a particular interest for educational researchers and practitioners who seek to use the methods of artificial intelligence to support legal writing education.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
Source files of the casenote classifier developed in this study are publicly available at https://bit.ly/3roDWTC.
References
Ädel, A.: Selecting quantitative data for qualitative analysis: a case study connecting a lexicogrammatical pattern to rhetorical moves. J. Engl. Acad. Purp. 16, 68–80 (2014)
Allen, L.K., Jacovina, M.E., McNamara, D.S.: Computer-based writing instruction(2016)
Almatrafi, O., Johri, A., Rangwala, H.: Needle in a haystack: identifying learner posts that require urgent response in MOOC discussion forums. Comput. Educ. 118, 1–9 (2018)
Anthony, L., Lashkia, G.V.: Mover: a machine learning tool to assist in the reading and writing of technical papers. IEEE TPC 46(3), 185–193 (2003)
Artstein, R., Poesio, M.: Inter-coder agreement for computational linguistics. Comput. Linguist. 34, 555–596 (2008)
Barbosa, G., et al.: Towards automatic cross-language classification of cognitive presence in online discussions. In: LAK, pp. 605–614 (2020)
Bhatia, V.K.: Simplification v. easification-the case of legal texts1. Appl. Linguis. 4(1), 42–54 (1983)
Bhatia, V.K.: Analysing genre: Language use in professional settings. Routledge (2014)
Bird, S., Loper, E.: Nltk: the natural language toolkit. ACM (2004)
Bransford, J.D., Barclay, J.R., Franks, J.J.: Sentence memory: a constructive versus interpretive approach. Cogn. Psychol. 3(2), 193–209 (1972)
Breiman, L.: Random forests. Mach. Learn. 45(1), 5–32 (2001)
Caines, A., Pastrana, S., Hutchings, A., Buttery, P.J.: Automatically identifying the function and intent of posts in underground forums. Crime Sci. 7(1), 1–14 (2018). https://doi.org/10.1186/s40163-018-0094-4
Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N., Androutsopoulos, I.: Legal-bert: The muppets straight out of law school (2020). arXiv preprint arXiv:2010.02559
Chawla, N.V., Bowyer, K.W., Hall, L.O., Kegelmeyer, W.P.: Smote: synthetic minority over-sampling technique. J. Artifi. Intell. Res. 16, 321–357 (2002)
Cotos, E., Huffman, S., Link, S.: Understanding graduate writers’ interaction with and impact of the research writing tutor during revision. J. Writing Res. 12(1), 187–232 (2020)
Crossley, S.: A chronotopic approach to genre analysis: an exploratory study. Engl. Specif. Purp. 26(1), 4–24 (2007)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding (2018). arXiv preprint arXiv:1810.04805
Driessen, E., Van Der Vleuten, C.: Matching student assessment to problem-based learning: lessons from experience in a law faculty. Stud. Contin. Educ. 22(2), 235–248 (2000)
Dunlosky, J., Hartwig, M.K., Rawson, K.A., Lipko, A.R.: Improving college students’ evaluation of text learning using idea-unit standards. Quart. J. Exp. Psychol. 64(3), 467–484 (2011)
Ferreira, M., Rolim, V., Mello, R.F., Lins, R.D., Chen, G., Gašević, D.: Towards automatic content analysis of social presence in transcripts of online discussions. In: LAK, pp. 141–150 (2020)
Hao, Y., Dong, L., Wei, F., Xu, K.: Visualizing and understanding the effectiveness of bert (2019). arXiv preprint arXiv:1908.05620
Haykin, S., Lippmann, R.: Neural networks, a comprehensive foundation. Int. J. Neural Syst. 5(4), 363–364 (1994)
Hendry, G.D., Bromberger, N., Armstrong, S.: Constructive guidance and feedback for learning: the usefulness of exemplars, marking sheets and different types of feedback in a first year law subject. Ass. Evalu. High. Educ. 36(1), 1–11 (2011)
Knight, S., Abel, S., Shibani, A., Goh, Y.K., Conijn, R., Gibson, A., Vajjala, S., Cotos, E., Sándor, Á., Shum, S.B.: Are you being rhetorical? a description of rhetorical move annotation tools and open corpus of sample machine-annotated rhetorical moves. J. Learn. Analy. 7(3), 138–154 (2020)
Knight, S., Shibani, A., Abel, S., Gibson, A., Ryan, P., Sutton, N., Wight, R., Lucas, C., Sandor, A., Kitto, K., et al.: Acawriter: a learning analytics tool for formative feedback on academic writing. J. Writing Res. 12(1), 141–186 (2020)
Knight, S., Shum, S.B., Ryan, P., Sándor, Á., Wang, X.: Designing academic writing analytics for civil law student self-assessment. Int. J. AIED 28(1), 1–28 (2018)
Kovanović, V., et al.: Towards automated content analysis of discussion transcripts: A cognitive presence case. In: LAK, pp. 15–24 (2016)
Levine, J.M.: Legal writing as a discipline: Past, present, and future. ABA Legal Writing Sourcebook, 3rd ed., Duquesne University School of Law Research Paper (2020)
Lucas, C., Shum, S.B., Liu, M., Bebawy, M.: Implementing acawriter as a novel strategy to support pharmacy students’ reflective practice in scientific research. In: JPE (2021)
Neto, V., et al.: Automated analysis of cognitive presence in online discussions written in portuguese. In: Pammer-Schindler, V., Pérez-Sanagustín, M., Drachsler, H., Elferink, R., Scheffel, M. (eds.) EC-TEL 2018. LNCS, vol. 11082, pp. 245–261. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-98572-5_19
Neumann Jr., R.K., Margolis, E., Stanchi, K.M.: Legal reasoning and legal writing. Aspen Publishers (2021)
Nguyen, T.-S., Nguyen, L.-M., Tojo, S., Satoh, K., Shimazu, A.: Recurrent neural network-based models for recognizing requisite and effectuation parts in legal texts. Arti. Intell. Law 26(2), 169–199 (2018). https://doi.org/10.1007/s10506-018-9225-1
Pardo, A., Jovanovic, J., Dawson, S., Gašević, D., Mirriahi, N.: Using learning analytics to scale the provision of personalised feedback. Br. J. Edu. Technol. 50(1), 128–138 (2019)
Pennebaker, J.W., Francis, M.E., Booth, R.J.: Linguistic inquiry and word count: Liwc 2001, vol. 71. Lawrence Erlbaum Associates, Mahway (2001)
Ren, H., Li, Y.: A comparison study on the rhetorical moves of abstracts in published research articles and master’s foreign-language theses. Engl. Lang. Teach. 4(1), 162–166 (2011)
Robson, R.: Law students as legal scholars: an essay/review of scholarly writing for law students and academic legal writing. NY City L. Rev. 7, 195 (2004)
Roscoe, R.D., McNamara, D.S.: Writing pal: Feasibility of an intelligent writing strategy tutor in the high school classroom. J. Educ. Psychol. 105(4), 1010 (2013)
Sándor, Á.: Modeling metadiscourse conveying the author’s rhetorical strategy in biomedical research abstracts. Revue française de linguistique appliquée 12(2), 97–108 (2007)
Sha, L., et al.: Which hammer should i use? a systematic evaluation of approaches for classifying educational forum posts. In: EDM (2021)
Shibani, A., Knight, S., Shum, S.B.: Contextualizable learning analytics design: a generic model and writing analytics evaluations. In: LAK, pp. 210–219 (2019)
Silveira, R., Fernandes, C., Neto, J.A.M., Furtado, V., Pimentel Filho, J.E.: Topic modelling of legal documents via legal-bert. In: Proceedings CEUR 1613 (2021)
Speiser, J.L., Miller, M.E., Tooze, J., Ip, E.: A comparison of random forest variable selection methods for classification prediction modeling. Expert Syst. Appl. 134, 93–101 (2019)
Strobl, C., Ailhaud, E., Benetos, K., Devitt, A., Kruse, O., Proske, A., Rapp, C.: Digital support for academic writing: a review of technologies and pedagogies. Comput. Educ. 131, 33–48 (2019)
Swales, J.M.: Aspects of article introductions. No. 1. University of Michigan Press (2011)
Tessuto, G.: Investigating English legal genres in academic and professional contexts. Cambridge Scholars Publishing (2013)
Tessuto, G.: Generic structure and rhetorical moves in English-language empirical law research articles: sites of interdisciplinary and interdiscursive cross-over. Engl. Specif. Purp. 37, 13–26 (2015)
Vihikan, W.O., Mistica, M., Levy, I., Christie, A., Baldwin, T.: Automatic resolution of domain name disputes. In: NLP Workshop 2021, pp. 228–238 (2021)
Wenestam, A.: Labelling factual information in legal cases using fine-tuned bert models (2021)
Xing, W., Tang, H., Pei, B.: Beyond positive and negative emotions: looking into the role of achievement emotions in discussion forums of MOOCS. Internet High. Educ. 43, 100690 (2019)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Raković, M. et al. (2022). Towards the Automated Evaluation of Legal Casenote Essays. In: Rodrigo, M.M., Matsuda, N., Cristea, A.I., Dimitrova, V. (eds) Artificial Intelligence in Education. AIED 2022. Lecture Notes in Computer Science, vol 13355. Springer, Cham. https://doi.org/10.1007/978-3-031-11644-5_14
Download citation
DOI: https://doi.org/10.1007/978-3-031-11644-5_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-11643-8
Online ISBN: 978-3-031-11644-5
eBook Packages: Computer ScienceComputer Science (R0)