Skip to main content
Springer Nature Link
Log in

BERT-CNN based evidence retrieval and aggregation for Chinese legal multi-choice question answering

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Legal question answering is an important natural language processing application in the legal domain. The Judicial Examination of Chinese Question Answering dataset is the most prominent and more challenging legal question answering dataset, which offers many multiple-choice legal questions and meta-information about the questions labelled by skilled humans. The current approaches to this task rely solely on pre-trained language models and do not find effective ways to utilise legal knowledge. We propose a retrieving-then-answering framework for the task. Its core is the Graph-Based Evidence Retrieval and Aggregation Network. The network enhances the model’s ability to answer a question by leveraging the legal knowledge relevant to the question and its answer options. The experimental results show that our model outperforms the existing state-of-the-art methods. The results also indicate that our proposed approach to using evidence is practical.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

The JEC-QA dataset used in the current study are available at https://jecqa.thunlp.org.

Notes

  1. https://jecqa.thunlp.org/.

  2. https://jecqa.thunlp.org.

  3. https://github.com/KKenny0/torchKbert.

References

  1. Bagherian-Marandi N, Ravanshadnia M, Akbarzadeh-T MR (2021) Two-layered fuzzy logic-based model for predicting court decisions in construction contract disputes. Artif Intell Law 29:453–484

    Article  Google Scholar 

  2. Bastings J, Titov I, Aziz W, et al (2017) Graph convolutional encoders for syntax-aware neural machine translation. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp 1957–1967

  3. Carvalho DS, Nguyen MT, Tran CX, et al (2015) Lexical-morphological modelling for legal text analysis. In: New Frontiers in Artificial Intelligence: JSAI-isAI 2015, Lecture Notes in Computer Science, vol 10091. Springer, p 295–311

  4. Chen D, Fisch A, Weston J, et al (2017) Reading wikipedia to answer open-domain questions. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pp 1870–1879

  5. Chen T, Van Durme B (2017) Discriminative information retrieval for question answering sentence selection. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pp 719–725

  6. Cui Y, Che W, Liu T, et al (2020) Revisiting pre-trained models for Chinese natural language processing. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pp 657–668

  7. De Cao N, Aziz W, Titov I (2019) Question answering by reasoning across documents with graph convolutional networks. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp 2306–2317

  8. De Martino G, Pio G, Ceci M (2022) PRILJ: an efficient two-step method based on embedding and clustering for the identification of regularities in legal case judgments. Artif Intell Law 30:359–390

    Article  Google Scholar 

  9. Devlin J, Chang MW, Lee K, et al (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp 4171–4186

  10. Dhingra B, Mazaitis K, Cohen WW (2017) Quasar: Datasets for Question Answering by Search and Reading. arXiv e-prints https://arxiv.org/abs/arXiv:1707.03904

  11. Do PK, Nguyen HT, Tran CX, et al (2017) Legal Question Answering using Ranking SVM and Deep Convolutional Neural Network. arXiv e-prints https://arxiv.org/abs/arXiv:1703.05320

  12. Dunn M, Sagun L, Higgins M, et al (2017) SearchQA: a new Q &A dataset augmented with context from a search engine. arXiv e-prints https://arxiv.org/abs/arXiv:1704.05179

  13. Fawei B, Pan JZ, Kollingbaum M, et al (2018) A methodology for a criminal law and procedure ontology for legal question answering. In: Proceedings of the Joint International Semantic Technology Conference, pp 198–214

  14. Gori M, Monfardini G, Scarselli F (2005) A new model for learning in graph domains. In: Proceedings of the 2005 IEEE International Joint Conference on Neural Networks, pp 729–734

  15. Green Jr BF, Wolf AK, Chomsky C, et al (1961) Baseball: an automatic question-answerer. In: Proceedings of Western Joint IRE-AIEE-ACM Computer Conference, pp 219–224

  16. Guo ZX, Deng XL (2021) Intelligent identification method of legal case entity based on BERT-BiLSTM-CRF. J Beijing Univ Posts Telecommun 44(4):129–134

    MathSciNet  Google Scholar 

  17. Harabagiu S, Moldovan D, Clark C, et al (2003) Answer mining by combining extraction techniques with abductive reasoning. In: Proceedings of the 12th Text Retrieval Conference, pp 375–382

  18. Huang Q, Luo X (2018) State-of-the-art and development trend of artificial intelligence combined with law. Comput Sci 45(12):1–11 (In Chinese)

    ADS  Google Scholar 

  19. Humphreys L, Boella G, van der Torre L et al (2021) Populating legal ontologies using semantic role labelling. Artif Intell Law 29(2):171–211

    Article  Google Scholar 

  20. Joshi M, Choi E, Weld DS, et al (2017) TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pp 1601–1611

  21. Kano Y, Hoshino R, Taniguchi R (2017) Analyzable legal yes/no question answering system using linguistic structures. EPiC Series Comput 47:57–67

    Google Scholar 

  22. Kano Y, Kim MY, Yoshioka M, et al (2018) COLIEE-2018: evaluation of the competition on legal information extraction and entailment. In: New Frontiers in Artificial Intelligence: JSAI-isAI 2018, Lecture Notes in Computer Science, vol 11717. Springer, p 177–192

  23. Kien PM, Nguyen HT, Bach NX, et al (2020) Answering legal questions by learning neural attentive text representation. In: Proceedings of the 28th International Conference on Computational Linguistics, pp 988–998

  24. Kourtin I, Mbarki S, Mouloudi A (2020) A legal question answering ontology-based system. In: Formalising Natural Languages: Applications to Natural Language Processing and Digital Humanities: NooJ 2020, Communications in Computer and Information Science, vol 1389. Springer, p 218–229

  25. Liu B, Wu Y, Zhang F et al (2022) Query generation and buffer mechanism: towards a better conversational agent for legal case retrieval. Inform Process Manag 59(5):103051

    Article  Google Scholar 

  26. Liu J, Wu J, Luo X (2021) Chinese judicial summarising based on short sentence extraction and GPT-2. In: Knowledge Science, Engineering and Management: KSEM 2021, Lecture Notes in Computer Science, vol 12816. Springer, p 376–393

  27. Liu L, Luo J (2018) A question answering system based on deep learning. In: Proceedings of the International Conference on Intelligent Computing, pp 173–181

  28. Liu Y, Luo X, Yang X (2019) Semantics and structure based recommendation of similar legal cases. In: Proceedings of the 14th International Conference on Intelligent Systems and Knowledge Engineering, pp 388–395

  29. Liu Z, Xiong C, Sun M, et al (2020) Fine-grained fact verification with kernel graph attention network. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp 7342–7351

  30. Mandal A, Ghosh K, Ghosh S et al (2022) A sequence labelling model for catchphrase identification from legal case documents. Artif Intell Law 30:325–358

    Article  Google Scholar 

  31. Marcheggiani D, Titov I (2017) Encoding sentences with graph convolutional networks for semantic role labelling. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp 1506–1515

  32. Martinez-Gil J, Freudenthaler B, Tjoa AM (2019) Multiple choice question answering in the legal domain using reinforced co-occurrence. In: Proceedings of the International Conference on Database and Expert Systems Applications, pp 138–148

  33. McElvain G, Sanchez G, Teo D, et al (2019) Non-factoid question answering in the legal domain. In: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp 1395–1396

  34. Qin L, Xu X, Che W, et al (2020) Dynamic fusion network for multi-domain end-to-end task-oriented dialogueue. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp 6344–6354

  35. Robertson S, Zaragoza H (2009) The probabilistic relevance framework: BM25 and beyond. Now Publishers Inc

  36. Šavelka J, Ashley KD (2022) Legal information retrieval for understanding statutory terms. Artif Intell Law 30:245–289

    Article  Google Scholar 

  37. Seo M, Kembhavi A, Farhadi A, et al (2016) Bidirectional attention flow for machine comprehension. arXiv e-prints https://arxiv.org/abs/arXiv:1611.01603

  38. Shao H, Chen Y, Huang S (2020) BERT-based ensemble model for statute law retrieval and legal information entailment. In: New Frontiers in Artificial Intelligence: JSAI-isAI 2020, Lecture Notes in Computer Science, vol 12758. Springer, p 226–239

  39. Silveira R, Fernandes CG, Neto JAM et al (2021) Topic modelling of legal documents via LEGAL-BERT. CEUR Workshop Proceedings 2896:64–72

  40. Su J (2020) WoBERT: word-based Chinese BERT model - ZhuiyiAI. Tech. rep., Zhuiyi Technology, https://github.com/ZhuiyiTechnology/WoBERT

  41. Sun H, Dhingra B, Zaheer M, et al (2018) Open domain question answering using early fusion of knowledge bases and text. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp 4231–4242

  42. Tagarelli A, Simeri A (2022) Unsupervised law article mining based on deep pre-trained language representation models with application to the Italian civil code. Artif Intell Law 30:417–473

    Article  Google Scholar 

  43. Vaswani A, Shazeer N, Parmar N, et al (2017) Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp 6000–6010

  44. Velickovic P, Cucurull G, Casanova A, et al (2018) Graph attention networks. In: Proceedings of the 6th International Conference on Learning Representations, pp 1–12

  45. Voorhees E (2001) The TREC question answering track. Nat Lang Eng 7(4):361–378

    Article  MathSciNet  Google Scholar 

  46. Voorhees EM et al (1999) The TREC-8 question answering track report. Trec 99:77–82

    Google Scholar 

  47. Wang C, Luo X (2021) A legal question answering system based on bert. In: Proceedings of the 5th International Conference on Computer Science and Artificial Intelligence, pp 278–283

  48. Wang S, Jiang J (2017) Machine comprehension using match-LSTM and answer pointer. In: Proceedings of the 2017 International Conference on Learning Representations, pp 1–15

  49. Wang S, Yu M, Jiang J, et al (2018) A co-matching model for multi-choice reading comprehension. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pp 746–751

  50. Wehnert S, Sudhi V, Dureja S, et al (2021) Legal norm retrieval with variations of the Bert model combined with TF-IDF vectorization. In: Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, pp 285–294

  51. Wenestam A (2021) Labelling factual information in legal cases using fine-tuned BERT models. Master’s thesis, Uppsala University, Uppsala, Sweden

  52. Wu J, Luo X (2021) Alignment-based graph network for judicial examination task. In: Knowledge Science, Engineering and Management: KSEM 2021, Lecture Notes in Computer Science, vol 12817. Springer, p 386–400

  53. Xiao C, Hu X, Liu Z et al (2021) Lawformer: a pre-trained language model for chinese legal long documents. AI Open 2:79–84

    Article  Google Scholar 

  54. Xu K, Wu L, Wang Z, et al (2018) SQL-to-text generation with graph-to-sequence model. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp 931–936

  55. Xu K, Wu L, Wang Z, et al (2018) Graph2Seq: graph to sequence learning with attention-based neural networks. arXiv e-prints https://arxiv.org/abs/arXiv:1804.00823

  56. Xu Y, Li T, Han Z (2020) The language model for legal retrieval and BERT-based model for rhetorical role labelling for legal judgments. CEUR Workshop Proceedings 2826:71–75

  57. Yu M, Yin W, Hasan KS, et al (2017) Improved neural relation detection for knowledge base question answering. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pp 571–581

  58. Zhang NN, Xing Y (2021) Questions and answers on legal texts based on BERT-BiGRU. In: Journal of Physics: Conference Series, p article id. 012035, 10.1088/1742-6596/1828/1/012035

  59. Zhang Y, Qi P, Manning CD (2018) Graph convolution over pruned dependency trees improves relation extraction. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp 2205–2215

  60. Zhong H, Xiao C, Tu C, et al (2020a) How does nlp benefit legal system: a summary of legal artificial intelligence. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp 5218–5230

  61. Zhong H, Xiao C, Tu C, et al (2020b) JEC-QA: a legal-domain question answering dataset. In: Proceedings of the 34th AAAI Conference on Artificial Intelligence, pp 9701–9708

  62. Zhong Q, Fan X, Luo X et al (2019) An explainable multi-attribute decision model based on argumentation. Expert Syst Appl 117:42–61

    Article  Google Scholar 

  63. Zhu H, Wei F, Qin B, et al (2018) Hierarchical attention flow for multiple-choice reading comprehension. In: Proceedings of the 32th AAAI Conference on Artificial Intelligence, pp 6077–6084

Download references

Acknowledgements

We want to extend our heartfelt gratitude to the anonymous reviewers for their insightful comments and constructive suggestions. Their expertise and dedication to the peer review process have significantly contributed to enhancing the quality and rigour of this manuscript. Their input was invaluable in refining our paper to its current form. We sincerely appreciate their time and effort. Also, we would like to extend our heartfelt gratitude to Guibin Chen, whose invaluable insights and diligent efforts have played a critical role in refining this paper. His expertise and guidance have been a beacon of light in enhancing this manuscript. This manuscript is an extended version of our prior work [52]. This work was supported by the National Natural Science Foundation of China (No. 61762016), the Middle-aged and Young Teachers’ Basic Ability Promotion Project of Guangxi(No. 2021KY0067) and Research Fund of Guangxi Key Lab of Multi-source Information Mining & Security (22-A-01-02).

Author information

Authors and Affiliations

  1. School of Computer Science and Engineering, Guangxi Normal University, Guilin, 541004, Guangxi, China

    Yanling Li, Jiaye Wu & Xudong Luo

  2. Key Lab of Education Blockchain and Intelligent Technology, Ministry of Education of the People’s Republic of China, Guilin, 541004, Guangxi, China

    Xudong Luo

  3. Guangxi Key Lab of Multi-Source Information Mining & Security, Guilin, 541004, Guangxi, China

    Xudong Luo

Authors
  1. Yanling Li
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Jiaye Wu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Xudong Luo
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Xudong Luo.

Ethics declarations

Conflict of interest

We declare that we do not have any commercial or associative interest that represents a conflict of interest in connection with the work submitted.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, Y., Wu, J. & Luo, X. BERT-CNN based evidence retrieval and aggregation for Chinese legal multi-choice question answering. Neural Comput & Applic 36, 5909–5925 (2024). https://doi.org/10.1007/s00521-023-09380-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-09380-5

Keywords