Andong Chen
ABSTRACT
Legal Question Answering (LQA) is a promising artificial intelligence application with high practical value. A professional and effective legal question answering (QA) agent can assist in reducing the workload of lawyers and judges, and help to achieve judicial accessibility. However, the NLP community lacks a large-scale LQA dataset with high quality, making it difficult to develop practical data-driven LQA agents. To tackle this bottleneck, this work presents EQUALS, a well-annotated real-world dataset for lEgal QUestion Answering via reading Chinese LawS. EQUALS contains 6,914 {question, article, answer} triplets as well as a pool of articles of laws that covers 10 different collections of Chinese Laws. Questions and the corresponding answers in EQUALS are collected from a professional law consultation forum. More importantly, the exact spans of law articles are annotated by senior law students as the answers. In this way, we could assure the quality and professionalism of EQUALS. Furthermore, this work proposes a QA framework that encompasses a law article retrieval module and a machine reading comprehension module for extracting accurate answers from the law article. We conduct thorough experiments with representative baselines on EQUALS, and the results indicate that EQUALS is a challenging question answering task. To the best of our knowledge, EQUALS is the largest real-world LQA dataset which shall significantly promote the research of LQA tasks. The work has been open-sourced and is available at: https://github.com/andongBlue/EQUALS.
- Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to Answer Open-Domain Questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30-August 4, Volume 1: Long Papers. Association for Computational Linguistics, 1870--1879.Google Scholar
Cross Ref
- Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. 2019. A Span-Extraction Dataset for Chinese Machine Reading Comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, 5882--5888. https://doi.org/10.18653/v1/D19-1600Google ScholarCross Ref
- Biralatei Fawei, Adam Z. Wyner, and Jeff Z. Pan. 2016. Passing a USA National Bar Exam: a First Corpus for Experimentation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation LREC 2016, Portorož, Slovenia, May 23-28, 2016. European Language Resources Association (ELRA).Google Scholar
- Yi Feng, Chuanyi Li, and Vincent Ng. 2022. Legal Judgment Prediction via Event Extraction with Constraints. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022. Association for Computational Linguistics, 648--664.Google ScholarCross Ref
- Changzhen Ji, Xin Zhou, Yating Zhang, Xiaozhong Liu, Changlong Sun, Conghui Zhu, and Tiejun Zhao. 2020. Cross Copy Network for Dialogue Generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, 1900--1910. https://doi.org/10.18653/v1/2020.emnlp-main.149Google ScholarCross Ref
- Donghong Ji, Peng Tao, Hao Fei, and Yafeng Ren. 2020. An end-to-end joint model for evidence information extraction from court record document. Inf. Process. Manag. 57, 6 (2020), 102305.Google ScholarCross Ref
- Bert F. Green Jr., Alice K. Wolf, Carol Chomsky, and Kenneth Laughery. 1961. Baseball: an automatic question-answerer. In Papers presented at the 1961 western joint IRE-AIEE-ACM computer conference, IRE-AIEE-ACM 1961 (Western), Los Angeles, California, USA, May 9-11, 1961, Walter F. Bauer (Ed.). ACM, 219--224. https://doi.org/10.1145/1460690.1460714Google ScholarDigital Library
- Yoshinobu Kano, Mi-Young Kim, Masaharu Yoshioka, Yao Lu, Juliano Rabelo, Naoki Kiyota, Randy Goebel, and Ken Satoh. 2018. COLIEE-2018: Evaluation of the Competition on Legal Information Extraction and Entailment. In New Frontiers in Artificial Intelligence - JSAI-isAI 2018 Workshops, JURISIN, AI-Biz, SKL, LENLS, IDAA, Yokohama, Japan, November 12-14, 2018, Revised Selected Papers, Vol. 11717. Springer, 177--192.Google Scholar
- Phi Manh Kien, Ha-Thanh Nguyen, Ngo Xuan Bach, Vu Tran, Minh Le Nguyen, and Tu Minh Phuong. 2020. Answering Legal Questions by Learning Neural Attentive Text Representation. In Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020. International Committee on Computational Linguistics, 988--998.Google ScholarCross Ref
- Antoine Louis and Gerasimos Spanakis. 2022. A Statutory Article Retrieval Dataset in French. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022. Association for Computational Linguistics, 6789--6803.Google ScholarCross Ref
- Luyao Ma, Yating Zhang, Tianyi Wang, Xiaozhong Liu, Wei Ye, Changlong Sun, and Shikun Zhang. 2021. Legal Judgment Prediction with Multi-Stage Case Representation Learning in the Real Court Setting. In SIGIR '21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021. ACM, 993--1002.Google ScholarDigital Library
- Jorge Martinez-Gil. 2021. A survey on legal question answering systems. arXiv preprint arXiv:2110.07333 (2021).Google Scholar
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, 2383--2392. https://doi.org/10.18653/v1/D16-1264Google ScholarCross Ref
- Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019. Association for Computational Linguistics, 3980--3990.Google ScholarCross Ref
- Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, 3980--3990. https://doi.org/10.18653/v1/D19-1410Google ScholarCross Ref
- Stephen E. Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Found. Trends Inf. Retr. 3, 4 (2009), 333--389.Google ScholarDigital Library
- Yunqiu Shao, Jiaxin Mao, Yiqun Liu, Weizhi Ma, Ken Satoh, Min Zhang, and Shaoping Ma. 2020. BERT-PLI: Modeling Paragraph-Level Interactions for Legal Case Retrieval. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020. ijcai.org, 3501--3507.Google ScholarCross Ref
- Börkur Sigurbjörnsson. 2004. Language Modeling for Information Retrieval. J. Log. Lang. Inf. 13, 4 (2004), 531--534.Google ScholarDigital Library
- Changlong Sun, Yating Zhang, Qiong Zhang, and Xiaozhong Liu. 2020. Legal Artificial Intelligence - Have You Lost a Piece from Jigsaw Puzzle?. In Proceedings of the AAAI 2020 Spring Symposium on Combining Machine Learning and Knowledge Engineering in Practice, AAAI-MAKE 2020, Palo Alto, CA, USA, March 23-25, 2020, Volume I (CEUR Workshop Proceedings, Vol. 2600). CEUR-WS.org.Google Scholar
- Yuan Sun, Chaofan Chen, Andong Chen, and Xiaobing Zhao. 2021. Tibetan question generation based on sequence to sequence model. Comput. Mater. Continua 68, 3 (2021), 3203--3213.Google ScholarCross Ref
- Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. 2018. R3: Reinforced Ranker-Reader for Open-Domain Question Answering. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018. AAAI Press, 5981--5988.Google ScholarCross Ref
- Yiquan Wu, Kun Kuang, Yating Zhang, Xiaozhong Liu, Changlong Sun, Jun Xiao, Yueting Zhuang, Luo Si, and Fei Wu. 2020. De-Biased Court's View Generation with Causality. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020. Association for Computational Linguistics, 763--780.Google ScholarCross Ref
- Feng Yao, Chaojun Xiao, Xiaozhi Wang, Zhiyuan Liu, Lei Hou, Cunchao Tu, Juanzi Li, Yun Liu, Weixing Shen, and Maosong Sun. 2022. LEVEN: A Large-Scale Chinese Legal Event Detection Dataset. In Findings of the Association for Computational Linguistics: ACL 2022. Association for Computational Linguistics, Dublin, Ireland, 183--201. https://doi.org/10.18653/v1/2022.findings-acl.17Google ScholarCross Ref
- Botao Zhong, Wanlei He, Ziwei Huang, Peter E. D. Love, Junqing Tang, and Hanbin Luo. 2020. A building regulation question answering system: A deep learning methodology. Adv. Eng. Informatics 46 (2020), 101195. https://doi.org/10.1016/j.aei.2020.101195Google ScholarCross Ref
- Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. JEC-QA: A Legal-Domain Question Answering Dataset. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020. AAAI Press, 9701--9708.Google ScholarCross Ref
Index Terms
- EQUALS: A Real-world Dataset for Legal Question Answering via Reading Chinese Laws
Recommendations
A dataset for evaluating legal question answering on private international law
ICAIL '21: Proceedings of the Eighteenth International Conference on Artificial Intelligence and LawInternational Private Law (PIL) is a complex legal domain that presents frequent conflicting norms between the hierarchy of legal sources, legal domains, and the adopted procedures. Scientific research on PIL reveals the need to create a bridge between ...
BERT-CNN based evidence retrieval and aggregation for Chinese legal multi-choice question answering
AbstractLegal question answering is an important natural language processing application in the legal domain. The Judicial Examination of Chinese Question Answering dataset is the most prominent and more challenging legal question answering dataset, which ...
Non-factoid Question Answering in the Legal Domain
SIGIR'19: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information RetrievalNon-factoid question answering in the legal domain must provide legally correct, jurisdictionally relevant, and conversationally responsive answers to user-entered questions. We present work done on a QA system that is entirely based on IR and NLP, and ...
Comments