Hostname: page-component-848d4c4894-hfldf Total loading time: 0 Render date: 2024-05-16T08:33:34.471Z Has data issue: false hasContentIssue false
  • English
  • Français

Gender bias in legal corpora and debiasing it

Published online by Cambridge University Press:  30 March 2022

Nurullah Sevim ,
Furkan Şahinuç  and
Aykut Koç
Nurullah Sevim
Affiliation:
Department of Electrical and Electronics Engineering, Bilkent University, Ankara, Turkey National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara, Turkey
Furkan Şahinuç
Affiliation:
Department of Electrical and Electronics Engineering, Bilkent University, Ankara, Turkey ASELSAN Research Center, Ankara, Turkey
Aykut Koç*
Affiliation:
Department of Electrical and Electronics Engineering, Bilkent University, Ankara, Turkey National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara, Turkey
*
*Corresponding author. Email: aykut.koc@bilkent.edu.tr
Get access Rights & Permissions [Opens in a new window]

Abstract

Word embeddings have become important building blocks that are used profoundly in natural language processing (NLP). Despite their several advantages, word embeddings can unintentionally accommodate some gender- and ethnicity-based biases that are present within the corpora they are trained on. Therefore, ethical concerns have been raised since word embeddings are extensively used in several high-level algorithms. Studying such biases and debiasing them have recently become an important research endeavor. Various studies have been conducted to measure the extent of bias that word embeddings capture and to eradicate them. Concurrently, as another subfield that has started to gain traction recently, the applications of NLP in the field of law have started to increase and develop rapidly. As law has a direct and utmost effect on people’s lives, the issues of bias for NLP applications in legal domain are certainly important. However, to the best of our knowledge, bias issues have not yet been studied in the context of legal corpora. In this article, we approach the gender bias problem from the scope of legal text processing domain. Word embedding models that are trained on corpora composed by legal documents and legislation from different countries have been utilized to measure and eliminate gender bias in legal documents. Several methods have been employed to reveal the degree of gender bias and observe its variations over countries. Moreover, a debiasing method has been used to neutralize unwanted bias. The preservation of semantic coherence of the debiased vector space has also been demonstrated by using high-level tasks. Finally, overall results and their implications have been discussed in the scope of NLP in legal domain.

Keywords

BiasNLP in lawLegal text processingLawComputational law
Type
Article
Information
Natural Language Engineering , Volume 29 , Issue 2 , March 2023 , pp. 449 - 482
Check for updates
Copyright
© The Author(s), 2022. Published by Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

Aletras, N., Tsarapatsanis, D., Preotiuc-Pietro, D. and Lampos, V. (2016). Predicting judicial decisions of the European Court of Human Rights: A natural language processing perspective. PeerJ Computer Science 2, e93.10.7717/peerj-cs.93CrossRefGoogle Scholar
Aleven, V. (2003). Using background knowledge in case-based legal reasoning: A computational model and an intelligent learning environment. Artificial Intelligence 150, 183–237.10.1016/S0004-3702(03)00105-XCrossRefGoogle Scholar
Ashley, K.D. (1988). Modelling Legal Argument: Reasoning with Cases and Hypotheticals. PhD thesis, University of Massachusetts, USA. Order No: GAX88-13198.Google Scholar
Ashley, K.D. (1991). Reasoning with cases and hypotheticals in HYPO. International Journal of Man-Machine Studies 34(6), 753–796.10.1016/0020-7373(91)90011-UCrossRefGoogle Scholar
Ashley, K.D. (1992). Case-based reasoning and its implications for legal expert systems. Artificial Intelligence and Law 1, 113–208.10.1007/BF00114920CrossRefGoogle Scholar
Ashley, K.D. and Brüninghaus, S. (2009). Automatically classifying case texts and predicting outcomes. Artificial Intelligence and Law 17(2), 125–165.10.1007/s10506-009-9077-9CrossRefGoogle Scholar
Azarbonyad, H., Dehghani, M., Marx, M. and Kamps, J. (2021). Learning to rank for multi-label text classification: Combining different sources of information. Natural Language Engineering 27(1), 89–111.10.1017/S1351324920000029CrossRefGoogle Scholar
Bach, N.X., Minh, N.L., Oanh, T.T. and Shimazu, A. (2013). A two-phase framework for learning logical structures of paragraphs in legal articles. ACM Transactions on Asian Language Information Processing 12(1), 1–32.10.1145/2425327.2425330CrossRefGoogle Scholar
Bartl, M., Nissim, M. and Gatt, A. (2020). Unmasking contextual stereotypes: Measuring and mitigating BERT’s gender bias. In Proceedings of the Second Workshop on Gender Bias in Natural Language Processing, Spain (Online). Barcelona: Association for Computational Linguistics, pp. 1–16.Google Scholar
Baziotis, C. and Jafari, B. 2018. ntua-slp-semeval2018. https://github.com/cbaziotis/ntua-slp-semeval2018.Google Scholar
Bench-Capon, T., Araszkiewicz, A.M., Ashley, A.K., Atkinson, K., Bex, F., Borges, F., Bourcier, D., Bourgine, P., Conrad, J.G., Francesconi, E., Gordon, T.F., Governatori, G., Leidner, J.L., Lewis, D.D., Loui, R.P., McCarty, L.T., Prakken, H., Schilder, F., Schweighofer, E., Thompson, P., Tyrrell, A., Verheij, B., Walton, D.N. and Wyner, A.Z. (2012). A history of AI and Law in 50 papers: 25 years of the international conference on AI and Law. Artificial Intelligence and Law 20, 215–319.CrossRefGoogle Scholar
Bhardwaj, R., Majumder, N. and Poria, S. (2021). Investigating gender bias in BERT. Cognitive Computation 13, 1008–1018.CrossRefGoogle Scholar
Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V. and Kalai, A. (2016). Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Proceedings of the 30th International Conference on Neural Information Processing Systems (NIPS), Red Hook, NY, USA. Curran Associates Inc., pp. 4356–4364.Google Scholar
Branting, K.L., Yeh, A., Weiss, B., Merkhofer, E. and Brown, B. (2018). Inducing predictive models for decision support in administrative adjudication. In Pagallo, U., Palmirani, M., Casanovas, P., Sartor, G. and Villata, S. (eds), AI Approaches to the Complexity of Legal Systems. Springer International Publishing, pp. 465–477.Google Scholar
Brunet, M.-E., Alkalay-Houlihan, C., Anderson, A. and Zemel, R. (2019). Understanding the origins of bias in word embeddings. In Chaudhuri, K. and Salakhutdinov, R. (eds), Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol. 97. PMLR, pp. 803–811.Google Scholar
Buchanan, B.G. and Headrick, T.E. (1970). Some speculation about artificial intelligence and legal reasoning. Stanford Law Review 23, 40–62.10.2307/1227753CrossRefGoogle Scholar
Caliskan, A., Bryson, J.J. and Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases. Science 356(6334), 183–186.CrossRefGoogle ScholarPubMed
Cardellino, C., Teruel, M., Alemany, L.A. and Villata, S. (2017). A low-cost, high-coverage legal named entity recognizer, classifier and linker. In Proceedings of the 16th Edition of the International Conference on Articial Intelligence and Law (ICAIL), New York, NY, USA. Association for Computing Machinery, pp. 9–18.CrossRefGoogle Scholar
Casanovas, P., Pagallo, U., Palmirani, M. and Sartor, G. (eds) (2013). AI Approaches to the Complexity of Legal Systems (AICOL) , Lecture Notes in Computer Science, vol. 8929. Belo Horizonte, Brazil: Springer International Publishing.Google Scholar
Chalkidis, I. and Androutsopoulos, I. (2017). A deep learning approach to contract element extraction. In Wyner A.Z. and Casini, G. (eds), Legal Knowledge and Information Systems - (JURIX): The Thirtieth Annual Conference, Frontiers in Artificial Intelligence and Applications, vol. 302, Luxembourg. IOS Press, pp. 155–164.Google Scholar
Chalkidis, I., Androutsopoulos, I. and Michos, A. (2017). Extracting contract elements. In Proceedings of the 16th Edition of the International Conference on Articial Intelligence and Law (ICAIL), New York, NY, USA. Association for Computing Machinery, pp. 19–28.CrossRefGoogle Scholar
Chalkidis, I., Androutsopoulos, I. and Michos, A. (2018). Obligation and prohibition extraction using hierarchical RNNs. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Melbourne, Australia. Association for Computational Linguistics, pp. 254–259.CrossRefGoogle Scholar
Chalkidis, I., Fergadiotis, E., Malakasiotis, P., Aletras, N. and Androutsopoulos, I. (2019). Extreme multi-label legal text classification: A case study in EU legislation. In Proceedings of the Natural Legal Language Processing Workshop 2019, Minneapolis, Minnesota. Association for Computational Linguistics, pp. 78–87.CrossRefGoogle Scholar
Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N. and Androutsopoulos, I. (2020). Legal-bert: ‘Ppreparing the muppets for court”. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pp. 2898–2904.Google Scholar
Chalkidis, I., Jana, A., Hartung, D., Bommarito, M.J., Androutsopoulos, I., Katz, D.M. and Aletras, N. (2021). Lexglue: A benchmark dataset for legal language understanding in English. Available at SSRN 3936759.CrossRefGoogle Scholar
Chalkidis, I. and Kampas, D. (2019). Deep learning in law: Early adaptation and legal word embeddings trained on large corpora. Artificial Intelligence and Law 27(2), 171–198.CrossRefGoogle Scholar
Church, K.W. (2017). Word2vec. Natural Language Engineering 23(1), 155–162.CrossRefGoogle Scholar
Clark, K. and Manning, C.D. (2016). Improving coreference resolution by learning entity-level distributed representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany. Association for Computational Linguistics, pp. 643–653.CrossRefGoogle Scholar
Dale, R. (2019). Law and word order: NLP in legal tech. Natural Language Engineering 25(1), 211–217.CrossRefGoogle Scholar
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K. and Kalai, A.T. (2019). Bias in bios: A case study of semantic representation bias in a high-stakes setting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, FAT*’19, New York, NY, USA. Association for Computing Machinery, pp. 120–128.10.1145/3287560.3287572CrossRefGoogle Scholar
Devlin, J., Chang, M.-W., Lee, K. and Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. Association for Computational Linguistics, pp. 4171–4186.Google Scholar
Dixon, L., Li, J., Sorensen, J., Thain, N. and Vasserman, L. (2018). Measuring and mitigating unintended bias in text classification. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (AIES), New York, NY, USA. Association for Computing Machinery, pp. 67–73.CrossRefGoogle Scholar
Do, P.-K., Nguyen, H.-T., Tran, C.-X., Nguyen, M.-T. and Nguyen, M.-L. (2017). Legal question answering using ranking svm and deep convolutional neural network. arXiv preprint arXiv:1703.05320.Google Scholar
Dozier, C., Kondadadi, R., Light, M., Vachher, A., Veeramachaneni, S. and Wudali, R. (2010). Named entity recognition and resolution in legal text. In Semantic Processing of Legal Texts: Where the Language of Law Meets the Law of Language. Berlin, Heidelberg: Springer-Verlag, pp. 27–43.10.1007/978-3-642-12837-0_2CrossRefGoogle Scholar
Elnaggar, A., Otto, R. and Matthes, F. (2018). Deep learning for named-entity linking with transfer learning for legal documents. In Proceedings of the Artificial Intelligence and Cloud Computing Conference (AICCC), New York, NY, USA. Association for Computing Machinery, pp. 23–28.10.1145/3299819.3299846CrossRefGoogle Scholar
Evans, R., Piwek, P., Cahill, L. and Tipper, N. (2008). Natural language processing in CLIME, a multilingual legal advisory system. Natural Language Engineering 14(1), 101–132.10.1017/S135132490600427XCrossRefGoogle Scholar
Faruqui, M., Tsvetkov, Y., Yogatama, D., Dyer, C. and Smith, N.A. (2015). Sparse overcomplete word vector representations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Beijing, China. Association for Computational Linguistics, pp. 1491–1500.CrossRefGoogle Scholar
Francesconi, E., Montemagni, S., Peters, W. and Tiscornia, D. (eds) (2010). Semantic Processing of Legal Texts: Where the Language of Law Meets the Law of Language , Lecture Notes in Computer Science, vol. 6036. New York, NY: Springer.Google Scholar
Fu, R., Guo, J., Qin, B., Che, W., Wang, H. and Liu, T. (2014). Learning semantic hierarchies via word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Baltimore, Maryland. Association for Computational Linguistics, pp. 1199–1209.CrossRefGoogle Scholar
Galgani, F., Compton, P. and Hoffmann, A. (2012). Combining different summarization techniques for legal text. In Proceedings of the Workshop on Innovative Hybrid Approaches to the Processing of Textual Data (HYBRID), USA. Association for Computational Linguistics, pp. 115–123.Google Scholar
Garg, N., Schiebinger, L., Jurafsky, D. and Zou, J. (2018). Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences 115(16), 3635–3644.CrossRefGoogle Scholar
Gonen, H. and Goldberg, Y. (2019). Lipstick on a pig: Debiasing methods cover up systematic gender biases in word embeddings but do not remove them. Computing Research Repository, arXiv:1903.03862. version 2.Google Scholar
Hafner, C.D. and Berman, D.H. (2002). The role of context in case-based legal reasoning: Teleological, temporal, and procedural. Artificial Intelligence and Law 10(1–3), 19–64.CrossRefGoogle Scholar
Hamilton, W.L., Leskovec, J. and Jurafsky, D. (2016). Diachronic word embeddings reveal statistical laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany. Association for Computational Linguistics, pp. 1489–1501.CrossRefGoogle Scholar
Hochreiter, S. and Schmidhuber, J. (1997). Long short-term memory. Neural Computation 9(8), 1735–1780.CrossRefGoogle ScholarPubMed
Joshi, M., Levy, O., Zettlemoyer, L. and Weld, D. (2019). BERT for coreference resolution: Baselines and analysis. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China. Association for Computational Linguistics, pp. 5803–5808.CrossRefGoogle Scholar
Joulin, A., Grave, E., Bojanowski, P. and Mikolov, T. (2017). Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, Valencia, Spain. Association for Computational Linguistics, pp. 427–431.10.18653/v1/E17-2068CrossRefGoogle Scholar
Kaneko, M. and Bollegala, D. (2019). Gender-preserving debiasing for pre-trained word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. Association for Computational Linguistics, pp. 1641–1650.CrossRefGoogle Scholar
Katz, D.M., Bommarito, M.J. and Blackman, J. (2017). A general approach for predicting the behavior of the Supreme Court of the United States. PLOS ONE 12(4), 1–18.CrossRefGoogle ScholarPubMed
Kim, M.-Y., Xu, Y. and Goebel, R. (2017). Applying a convolutional neural network to legal question answering. In Otake M., Kurahashi S., Ota Y., Satoh K. and Bekki D. (eds), New Frontiers in Artificial Intelligence. Springer International Publishing, pp. 282–294.Google Scholar
Kiritchenko, S. and Mohammad, S. (2018). Examining gender and race bias in two hundred sentiment analysis systems. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, New Orleans, Louisiana. Association for Computational Linguistics, pp. 43–53.CrossRefGoogle Scholar
Kurita, K., Vyas, N., Pareek, A., Black, A.W. and Tsvetkov, Y. (2019). Measuring bias in contextualized word representations. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, Florence, Italy. Association for Computational Linguistics, pp. 166–172.CrossRefGoogle Scholar
Kusner, M.J., Loftus, J., Russell, C. and Silva, R. (2017). Counterfactual fairness. In Guyon I., Luxburg U.V., Bengio S., Wallach, H., Fergus R., Vishwanathan S. and Garnett, R. (eds), Advances in Neural Information Processing Systems 30. Curran Associates, Inc., pp. 4066–4076.Google Scholar
Lai, S., Xu, L., Liu, K. and Zhao, J. (2015). Recurrent convolutional neural networks for text classification. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence. AAAI Press, pp. 2267–2273.CrossRefGoogle Scholar
Leitner, E., Rehm, G. and Moreno-Schneider, J. (2019). Fine-grained named entity recognition in legal documents. In International Conference on Semantic Systems. Springer, pp. 272–287.Google Scholar
Liang, P.P., Li, I.M., Zheng, E., Lim, Y.C., Salakhutdinov, R. and Morency, L.-P. (2020). Towards debiasing sentence representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online. Association for Computational Linguistics, pp. 5502–5515.CrossRefGoogle Scholar
Locke, D. and Zuccon, G. (2019). Towards automatically classifying case law citation treatment using neural networks. In Proceedings of the 24th Australasian Document Computing Symposium (ADCS), New York, NY, USA. Association for Computing Machinery.CrossRefGoogle Scholar
Long, S., Tu, C., Liu, Z. and Sun, M. (2019). Automatic judgment prediction via legal reading comprehension. In Sun M., Huang X., Ji H., Liu Z. and Liu Y. (eds), Chinese Computational Linguistics (CCL), Cham. Springer International Publishing, pp. 558–572.Google Scholar
Luz de Araujo, P.H., de Campos, T.E., de Oliveira, R. R.R., Stauffer, M., Couto, S. and Bermejo, P. (2018). LeNER-Br: A dataset for named entity recognition in Brazilian legal text. In International Conference on the Computational Processing of Portuguese (PROPOR), Lecture Notes on Computer Science (LNCS), Canela, RS, Brazil. Springer, pp. 313–323.CrossRefGoogle Scholar
Manzini, T., Yao Chong, L., Black, A.W. and Tsvetkov, Y. (2019). Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. Association for Computational Linguistics, pp. 615–621.CrossRefGoogle Scholar
Martin, A.D., Quinn, K.M., Ruger, T.W. and Kim, P.T. (2004). Competing approaches to predicting Supreme Court decision making. Perspectives on Politics 2(4), 761–767.CrossRefGoogle Scholar
May, C., Wang, A., Bordia, S., Bowman, S.R., and Rudinger, R. (2019). On measuring social biases in sentence encoders. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. Association for Computational Linguistics, pp. 622–628.CrossRefGoogle Scholar
Medvedeva, M., Vols, M. and Wieling, M. (2020). Using machine learning to predict decisions of the European Court of Human Rights. Artificial Intelligence and Law 28(2), 237–266.CrossRefGoogle Scholar
Mikolov, T., Chen, K., Corrado, G. and Dean, J. (2013a). Efficient estimation of word representations in vector space. In Bengio Y. and LeCun Y. (eds), 1st International Conference on Learning Representations (ICLR), Workshop Track Proceedings, Scottsdale, Arizona, USA.Google Scholar
Mikolov, T., Sutskever, I., Chen, K., Corrado, G. and Dean, J. (2013b). Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems (NIPS) - Volume 2, Red Hook, NY, USA. Curran Associates Inc., pp. 3111–3119.Google Scholar
Mohammad, S., Bravo-Marquez, F., Salameh, M. and Kiritchenko, S. (2018). SemEval-2018 task 1: Affect in tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, New Orleans, Louisiana. Association for Computational Linguistics, pp. 1–17.CrossRefGoogle Scholar
Morimoto, A., Kubo, D., Sato, M., Shindo, H. and Matsumoto, Y. (2017). Legal question answering system using neural attention. In Satoh K., Kim M., Kano Y., Goebel R. and Oliveira T. (eds), 4th Competition on Legal Information Extraction and Entailment (COLIEE), held in conjunction with the 16th International Conference on Artificial Intelligence and Law (ICAIL) in King’s College London, UK, EPiC Series in Computing, vol. 47. EasyChair, pp. 79–89.Google Scholar
Mumcuoğlu, E., Öztürk, C.E., Ozaktas, H.M. and Koç, A. (2021). Natural language processing in law: Prediction of outcomes in the higher courts of Turkey. Information Processing & Management 58(5), 102684.CrossRefGoogle Scholar
Murphy, B., Talukdar, P. and Mitchell, T. (2012). Learning effective and interpretable semantic models using non-negative sparse embedding. In Proceedings of COLING, Mumbai, India. The COLING 2012 Organizing Committee, pp. 1933–1950.Google Scholar
Nanda, R., John, A.K., Caro, L.D., Boella, G. and Robaldo, L. (2017). Legal information retrieval using topic clustering and neural networks. In Satoh K., Kim M.-Y., Kano Y., Goebel R. and Oliveira T. (eds), 4th Competition on Legal Information Extraction and Entailment (COLIEE), EPiC Series in Computing, vol. 47. EasyChair, pp. 68–78.Google Scholar
Navigli, R. and Martelli, F. (2019). An overview of word and sense similarity. Natural Language Engineering 25(6), 693–714.CrossRefGoogle Scholar
Nejadgholi, I., Bougueng, R. and Witherspoon, S. (2017). A semi-supervised training method for semantic search of legal facts in Canadian immigration cases. In Wyner, A.Z. and Casini G. (eds), Legal Knowledge and Information Systems - (JURIX): The Thirtieth Annual Conference, Luxembourg, 13–15 December 2017, Frontiers in Artificial Intelligence and Applications, vol. 302. IOS Press, pp. 125–134.Google Scholar
Nguyen, T.-S., Nguyen, L.-M., Tojo, S., Satoh, K. and Shimazu, A. (2018). Recurrent neural network-based models for recognizing requisite and effectuation parts in legal texts. Artificial Intelligence and Law 26(2), 169–199.CrossRefGoogle Scholar
O’Neill, J., Buitelaar, P., Robin, C. and O’Brien, L. (2017). Classifying sentential modality in legal language: A use case in financial regulations, acts and directives. In Proceedings of the 16th Edition of the International Conference on Artificial Intelligence and Law (ICAIL), New York, NY, USA. Association for Computing Machinery, pp. 159–168.Google Scholar
O’Sullivan, C. and Beel, J. (2019). Predicting the outcome of judicial decisions made by the european court of human rights. In In Proceedings of the 27th AIAI Irish Conference on Artificial Intelligence and Cognitive Science, Dublin, Ireland.Google Scholar
Pennington, J., Socher, R. and Manning, C.D. (2014). Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543.CrossRefGoogle Scholar
Perez, C.C. (2019). Invisible Women: Exposing Data Bias in a World Designed for Men. Pengu in Random House, South Africa.Google Scholar
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K. and Zettlemoyer, L. (2018). Deep contextualized word representations. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana. Association for Computational Linguistics, pp. 2227–2237.CrossRefGoogle Scholar
Pittaras, N., Giannakopoulos, G., Papadakis, G. and Karkaletsis, V. (2020). Text classification with semantically enriched word embeddings. Natural Language Engineering 27(4), 391–425.CrossRefGoogle Scholar
Prost, F., Thain, N. and Bolukbasi, T. (2019). Debiasing embeddings for reduced gender bias in text classification. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, Florence, Italy. Association for Computational Linguistics, pp. 69–75.CrossRefGoogle Scholar
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D. and Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog 1(8), 9.Google Scholar
Rudinger, R., Naradowsky, J., Leonard, B. and Van Durme, B. (2018). Gender bias in coreference resolution. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, Louisiana. Association for Computational Linguistics.Google Scholar
Ruger, T., Kim, P., Martin, A. and Quinn, K. (2004). The Supreme Court forecasting project: Legal and political science approaches to predicting Supreme Court decisionmaking. Columbia Law Review 104, 1150–1210.CrossRefGoogle Scholar
Sangeetha, D., Kavyashri, R., Swetha, S. and Vignesh, S. (2017). Information retrieval system for laws. In 2016 Eighth International Conference on Advanced Computing (ICoAC), pp. 212–217.CrossRefGoogle Scholar
Sartor, G. and Rotolo, A. (2013). Agreement Technologies, Chapter AI and Law. New York: Springer, pp. 199–207.Google Scholar
Senel, L.K., Utlu, I., Şahinuç, F., Ozaktas, H.M. and Koç, A. (2020). Imparting interpretability to word embeddings while preserving semantic structure. Natural Language Engineering 27(6), 721–746.CrossRefGoogle Scholar
Shulayeva, O., Siddharthan, A. and Wyner, A. (2017). Recognizing cited facts and principles in legal judgements. Artificial Intelligence and Law 25(1), 107–126. Open access via Springer Compact Agreement.CrossRefGoogle Scholar
Sleimi, A., Sannier, N., Sabetzadeh, M., Briand, L. and Dann, J. (2018). Automated extraction of semantic legal metadata using natural language processing. In IEEE 26th International Requirements Engineering Conference (RE). IEEE, pp. 124–135.CrossRefGoogle Scholar
Soh, J., Lim, H.K. and Chai, I.E. (2019). Legal area classification: A comparative study of text classifiers on Singapore Supreme Court judgments. In Proceedings of the Natural Legal Language Processing Workshop, Minneapolis, Minnesota. Association for Computational Linguistics, pp. 67–77.CrossRefGoogle Scholar
Stanovsky, G., Smith, N.A. and Zettlemoyer, L. (2019). Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy. Association for Computational Linguistics, pp. 1679–1684.10.18653/v1/P19-1164CrossRefGoogle Scholar
Şulea, O.-M., Zampieri, M., Vela, M. and van Genabith, J. (2017). Predicting the law area and decisions of French Supreme Court cases. In Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP, Varna, Bulgaria. INCOMA Ltd., pp. 716–722.CrossRefGoogle Scholar
Tan, Y.C. and Celis, L.E. (2019). Assessing social and intersectional biases in contextualized word representations. In Wallach H., Larochelle H., Beygelzimer A., d’Alché Buc F., Fox E. and Garnett R. (eds), Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc., pp. 13230–13241.Google Scholar
Tanaka-Ishii, K. (2007). Word-based predictive text entry using adaptive language models. Natural Language Engineering 13(1), 51–74.CrossRefGoogle Scholar
Tang, D., Wei, F., Yang, N., Zhou, M., Liu, T. and Qin, B. (2014). Learning sentiment-specific word embedding for Twitter sentiment classification. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Baltimore, Maryland. Association for Computational Linguistics, pp. 1555–1565.CrossRefGoogle Scholar
Tang, G., Guo, H., Guo, Z. and Xu, S. (2016). Matching law cases and reference law provision with a neural attention model. In IBM China Research, Beijing.Google Scholar
Tezcan, A., Hoste, V. and Macken, L. (2020). Estimating word-level quality of statistical machine translation output using monolingual information alone. Natural Language Engineering 26(1), 73–94.CrossRefGoogle Scholar
Tjong Kim Sang, E.F. and De Meulder, F. (2003). Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL, pp. 142–147.CrossRefGoogle Scholar
Üstün, A. and Can, B. (2020). Incorporating word embeddings in unsupervised morphological segmentation. Natural Language Engineering 27(5), 609–629.CrossRefGoogle Scholar
Vardhan, H., Surana, N. and Tripathy, B. (2020). Named-entity recognition for legal documents. In International Conference on Advanced Machine Learning Technologies and Applications. Springer, pp. 469–479.Google Scholar
Virtucio, M.B.L., Aborot, J.A., Abonita, J.K.C., Aviñante, R.S., Copino, R. J. B., Neverida, M.P., Osiana, V.O., Peramo, E.C., Syjuco, J.G. and Tan, G.B.A. (2018). Predicting decisions of the Philippine Supreme Court using natural language processing and machine learning. In 2018 IEEE 42nd Annual Computer Software and Applications Conference (COMPSAC), vol. 02, pp. 130–135.CrossRefGoogle Scholar
Vo, N.P.A., Privault, C. and Guillot, F. (2017). Experimenting word embeddings in assisting legal review. In Proceedings of the 16th Edition of the International Conference on Articial Intelligence and Law (ICAIL), New York, NY, USA. Association for Computing Machinery, pp. 189–198.CrossRefGoogle Scholar
Zhang, B.H., Lemoine, B. and Mitchell, M. (2018). Mitigating unwanted biases with adversarial learning. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, AIES’18, New York, NY, USA. Association for Computing Machinery, pp. 335–340.CrossRefGoogle Scholar
Zhao, J., Wang, T., Yatskar, M., Cotterell, R., Ordonez, V. and Chang, K.-W. (2019). Gender bias in contextualized word embeddings. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota. Association for Computational Linguistics, pp. 629–634.CrossRefGoogle Scholar
Zhao, J., Wang, T., Yatskar, M., Ordonez, V. and Chang, K.-W. (2018a). Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, Louisiana, USA, pp. 15–20.CrossRefGoogle Scholar
Zhao, J., Wang, T., Yatskar, M., Ordonez, V. and Chang, K.-W. (2017). Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark. Association for Computational Linguistics, pp. 2979–2989.CrossRefGoogle Scholar
Zhao, J., Zhou, Y., Li, Z., Wang, W. and Chang, K.-W. (2018b). Learning gender-neutral word embeddings. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium. Association for Computational Linguistics, pp. 4847–4853.CrossRefGoogle Scholar
Zou, J. and Schiebinger, L. (2018). AI can be sexist and racist — it’s time to make it fair. Nature 559, 324–326.CrossRefGoogle Scholar