research-article

A Reliable and Accurate Multiple Choice Question Answering System for Due Diligence

Authors:

Alexander K. HudekAuthors Info & Claims

ICAIL '19: Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law

Pages 184 - 188

https://doi.org/10.1145/3322640.3326711

Published: 17 June 2019 Publication History

Abstract

The problem of answering multiple choice questions, based on the content of documents has been studied extensively in the machine learning literature. We pose the due diligence problem, where lawyers study legal contracts and assess the risk in potential mergers and acquisitions, as a multiple choice question answering problem, based on the text of the contract. Existing frameworks for question answering are not suitable for this task, due to the inherent scarcity and imbalance in the legal contract data available for training. We propose a question answering system which first identifies the excerpt in the contract which potentially contains the answer to a given question, and then builds a multi-class classifier to choose the answer to the question, based on the content of this excerpt. Unlike existing question answering systems, the proposed system explicitly handles the imbalance in the data, by generating synthetic instances of the minority answer categories, using the Synthetic Minority Oversampling Technique. This ensures that the number of instances in all the classes are roughly equal to each other, thus leading to more accurate and reliable classification. We demonstrate that the proposed question answering system outperforms the existing systems with minimal amount of training data.

References

[1]

Jon Louis Bentley. 1975. Multidimensional Binary Search Trees Used for Associative Searching. Commun. ACM 18, 9 (1975), 509--517.

Digital Library

[2]

Christopher M Bishop. 2008. Pattern Recognition and Machine Learning. Springer.

[3]

Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002. SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research 16 (2002), 321--357.

Digital Library

[4]

Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014).

[5]

Eunsol Choi, Daniel Hewlett, Jakob Uszkoreit, Illia Polosukhin, Alexandre Lacoste, and Jonathan Berant. 2017. Coarse-to-fine question answering for long documents. In Proceedings of the Association for Computational Linguistics, Vol. 1. 209--220.

[6]

Phong-Khac Do, Huy-Tien Nguyen, Chien-Xuan Tran, Minh-Tien Nguyen, and Minh-Le Nguyen. 2017. Legal question answering using ranking SVM and deep convolutional neural network. arXiv preprint arXiv:1703.05320 (2017).

[7]

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735--1780.

Digital Library

[8]

Thorsten Joachims. 2006. Training linear SVMs in linear time. In Proceedings of the International Conference on Knowledge Discovery and Data mining. 217--226.

Digital Library

[9]

Mi-Young Kim, Ying Xu, and Randy Goebel. 2015. Applying a convolutional neural network to legal question answering. In JSAI International Symposium on Artificial Intelligence. 282--294.

[10]

John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the International Conference on Machine Learning. 282--289.

Digital Library

[11]

Huayu Li, Martin Renqiang Min, Yong Ge, and Asim Kadav. 2017. A Context-aware Attention Network for Interactive Question Answering. In Proceedings of the International Conference on Knowledge Discovery and Data mining. 927--935.

Digital Library

[12]

Fenglong Ma, Radha Chitta, Saurabh Kataria, Jing Zhou, Palghat Ramesh, Tong Sun, and Jing Gao. 2017. Long-Term Memory Networks for Question Answering. arXiv preprint arXiv:1707.01961 (2017).

[13]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems. 3111--3119.

Digital Library

[14]

Amit Mishra and Sanjay Kumar Jain. 2016. A survey on question answering systems with classification. Journal of King Saud University Computer and Information Sciences 28, 3 (2016), 345--361.

Digital Library

[15]

Naoaki Okazaki. 2007. CRFsuite: a fast implementation of Conditional Random Fields. http://www.chokkan.org/software/crfsuite.

[16]

Adam Roegiest, Alexander K Hudek, and Anne McNulty. 2018. A Dataset and an Examination of Identifying Passages for Due Diligence. In International ACM SIGIR Conference on Research & Development in Information Retrieval. 465--474.

Digital Library

[17]

Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan. 2008. Introduction to information retrieval. Vol. 39. Cambridge University Press.

[18]

James A Sherer, Taylor M Hoffman, and Eugenio E Ortiz. 2015. Merger and Acquisition Due Diligence: A Proposed Framework to Incorporate Data Privacy, Information Security, E-Discovery, and Information Governance into Due Diligence Practices. Richmond Journal of Law & Technology 21, 2 (2015), 5.

[19]

Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In Advances in Neural Information Processing Systems. 2440--2448.

Digital Library

[20]

Boyu Wang and Joelle Pineau. 2016. Online bagging and boosting for imbalanced data streams. IEEE Transactions on Knowledge & Data Engineering 1 (2016).

Digital Library

Cited By

Martinez-Gil J(2023)A survey on legal question–answering systemsComputer Science Review10.1016/j.cosrev.2023.10055248:COnline publication date: 1-May-2023
https://dl.acm.org/doi/10.1016/j.cosrev.2023.100552
Song DGao SHe BSchilder F(2022)On the Effectiveness of Pre-Trained Language Models for Legal Natural Language Processing: An Empirical StudyIEEE Access10.1109/ACCESS.2022.319040810(75835-75858)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3190408
Vo DDo-Vo ANguyen-Thi TDuong H(2022)An Approach for Multiple Choice Question Answering SystemNature of Computation and Communication10.1007/978-3-030-92942-8_7(76-82)Online publication date: 3-Jan-2022
https://doi.org/10.1007/978-3-030-92942-8_7
Show More Cited By

Index Terms

A Reliable and Accurate Multiple Choice Question Answering System for Due Diligence
1. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals

Recommendations

Quality-aware collaborative question answering: methods and evaluation
WSDM '09: Proceedings of the Second ACM International Conference on Web Search and Data Mining

Community Question Answering (QA) portals contain questions and answers contributed by hundreds of millions of users. These databases of questions and answers are of great value if they can be used directly to answer questions from any user. In this ...
Compositional question answering: A divide and conquer approach

This paper describes how questions can be characterized for question answering (QA) along different facets and focuses on questions that cannot be answered directly but can be divided into simpler ones so that they can be answered directly using ...
Automatic question-answer pairs generation and question similarity mechanism in question answering system
Abstract
With the swift growth of the information over the past few years, taking full benefit is increasingly essential. Question Answering System is one of the promising methods to access this much information. The Question Answering System lacks humans’ ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICAIL '19: Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law

June 2019

312 pages

ISBN:9781450367547

DOI:10.1145/3322640

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGAI: ACM Special Interest Group on Artificial Intelligence

In-Cooperation

Univ. of Montreal: University of Montreal
AAAI
IAAIL: Intl Asso for Artifical Intel & Law

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 June 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ICAIL '19

Sponsor:

SIGAI

ICAIL '19: Seventeenth International Conference on Artificial Intelligence and Law

June 17 - 21, 2019

QC, Montreal, Canada

Acceptance Rates

Overall Acceptance Rate 69 of 169 submissions, 41%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
152
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)2

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Martinez-Gil J(2023)A survey on legal question–answering systemsComputer Science Review10.1016/j.cosrev.2023.10055248:COnline publication date: 1-May-2023
https://dl.acm.org/doi/10.1016/j.cosrev.2023.100552
Song DGao SHe BSchilder F(2022)On the Effectiveness of Pre-Trained Language Models for Legal Natural Language Processing: An Empirical StudyIEEE Access10.1109/ACCESS.2022.319040810(75835-75858)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3190408
Vo DDo-Vo ANguyen-Thi TDuong H(2022)An Approach for Multiple Choice Question Answering SystemNature of Computation and Communication10.1007/978-3-030-92942-8_7(76-82)Online publication date: 3-Jan-2022
https://doi.org/10.1007/978-3-030-92942-8_7
Vold AConrad JMaranhão JWyner A(2021)Using transformers to improve answer retrieval for legal questionsProceedings of the Eighteenth International Conference on Artificial Intelligence and Law10.1145/3462757.3466102(245-249)Online publication date: 21-Jun-2021
https://dl.acm.org/doi/10.1145/3462757.3466102

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten