Abstract
Despite recent advances in deep learning-based language modelling, many natural language processing (NLP) tasks in the financial domain remain challenging due to the paucity of appropriately labelled data. Other issues that can limit task performance are differences in word distribution between the general corpora – typically used to pre-train language models – and financial corpora, which often exhibit specialized language and symbology. Here, we investigate two approaches that can help to mitigate these issues. Firstly, we experiment with further language model pre-training using large amounts of in-domain data from business and financial news. We then apply augmentation approaches to increase the size of our data-set for model fine-tuning. We report our findings on an Environmental, Social and Governance (ESG) controversies data-set and demonstrate that both approaches are beneficial to accuracy in classification tasks.
This research was conducted while all authors.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
https://www.ft.com/content/3f1d44d9-094f-4700-989f-616e27c89599 (accessed 2020-06-30).
- 3.
To obtain the data for replication, it can be licensed from Reuters at https://www.reutersagency.com/en/products/archive/ (accessed 2020-06-30).
- 4.
https://www.refinitiv.com/en/financial-data/company-data/esg-research-data (accessed 2020-06-30).
- 5.
https://sustainabledevelopment.un.org (accessed 2020-06-30).
- 6.
https://www.statmt.org/wmt14/translation-task.html (accessed 2020-06-30).
- 7.
F-score or F1 is the harmonic mean between precision and recall.
- 8.
http://handbook.reuters.com (accessed 2020-06-39).
References
Araci, D.: FinBERT: financial sentiment analysis with pre-trained language models. arXiv preprint arXiv:1908.10063 (2019)
Barz, B., Denzler, J.: Deep learning on small datasets without pre-training using cosine loss. In: IEEE Winter Conference on Applications of Computer Vision, WACV 2020, IEEE (2020)
Beltagy, I., Lo, K., Cohan, A.: SciBERT: a pretrained language model for scientific text. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3606–3611 (2019)
Bloice, M.D., Stocker, C., Holzinger, A.: Augmentor: an image augmentation library for machine learning. Technical report, ArXiv preprint server (2017). https://arxiv.org/pdf/1708.04680.pdf
Choi, Y., Jung, Y., Myaeng, S.-H.: Identifying controversial issues and their sub-topics in news articles. In: Chen, H., Chau, M., Li, S., Urs, S., Srinivasa, S., Wang, G.A. (eds.) PAISI 2010. LNCS, vol. 6122, pp. 140–153. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-13601-6_16
Clark, K., Khandelwal, U., Levy, O., Manning, C.D.: What does BERT look at? An analysis of BERT’s attention. In: Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 276–286. ACL, Florence (2019). https://doi.org/10.18653/v1/W19-4828
Desola, V., Hanna, K., Pri Nonis: FinBERT: pre-trained model on sec filings for financial natural language tasks. Technical report, University of California at Berkeley (2019). https://doi.org/10.13140/RG.2.2.19153.89442
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4171–4186. Association for Computational Linguistics, Minneapolis (2019). https://doi.org/10.18653/v1/N19-1423
Dori-Hacohen, S., Jensen, D., Allan, J.: Controversy detection in Wikipedia using collective classification. In: Proceedings of SIGIR, New York, NY, USA, pp. 797–800 (2016)
Edunov, S., Ott, M., Auli, M., Grangier, D.: Understanding back-translation at scale. arXiv preprint arXiv:1808.09381 (2018)
Feuerriegel, S., Prendinger, H.: News-based trading strategies. Decis. Support Syst. 90, 65–74 (2016)
Friede, G., Busch, T., Bassen, A.: ESG and financial performance: aggregated evidence from more than 2000 empirical studies. J. Sustain. Financ. Invest. 5(4), 210–233 (2015). https://doi.org/10.1080/20430795.2015.1118917
In, S.Y., Rook, D., Monk, A.: Integrating alternative data (also known as ESG data) in investment decision making. Glob. Econ. Rev. 48(3), 237–260 (2019). https://doi.org/10.1080/1226508X.2019.1643059
Jang, M., Foley, J., Dori-Hacohen, S., Allan, J.: Probabilistic approaches to controversy detection. In: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, CIKM 2016, pp. 2069–2072. ACM, New York (2016). https://doi.org/10.1145/2983323.2983911
Kitaev, N., Kaiser, Ł., Levskaya, A.: Reformer: the efficient transformer. arXiv preprint arXiv:2001.04451 (2020)
Kovaleva, O., Romanov, A., Rogers, A., Rumshisky, A.: Revealing the dark secrets of BERT. arXiv preprint arXiv:1908.08593 (2019)
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.: ALBERT: a lite BERT for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942 (2019)
Lee, J., et al.: BioBERT: pre-trained biomedical language representation model for biomedical text mining. arXiv preprint arXiv:1901.08746 (2019)
Leidner, J.L.: Survey of textual data & geospatial technology. In: Werner, M., Chiang, Y.-Y. (eds.) Handbook of Big Geospatial Data. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-55462-0_16
Lu, Y., Nakicenovic, N., Visbeck, M., Stevance, A.S.: Policy: five priorities for the UN Sustainable Development Goals. Nat. News 520(7548), 432 (2015)
Nugent, T., Leidner, J.L.: Risk mining: company-risk identification from unstructured sources. In: Domeniconi, C., et al. (eds.) IEEE International Conference on Data Mining ICDM, 12–15 December 2016, Barcelona, Spain, pp. 1308–1311 (2016)
Nugent, T., Petroni, F., Raman, N., Carstens, L., Leidner, J.L.: A comparison of classification models for natural disaster and critical event detection from news. In: 2017 IEEE International Conference on Big Data (Big Data), pp. 3750–3759. IEEE (2017)
Pennington, J., Socher, R., Manning, C.D.: GloVe: global vectors for word representation. In: Proceedings of EMNLP (2014)
Schramade, W.: Investing in the UN sustainable development goals: opportunities for companies and investors. J. Appl. Corp. Fin. 29(2), 87–99 (2017)
Sennrich, R., Haddow, B., Birch, A.: Improving neural machine translation models with monolingual data. arXiv preprint arXiv:1511.06709 (2015)
Vaswani, A., et al.: Tensor2Tensor for neural machine translation. CoRR abs/1803.07416 (2018)
Wu, Y., et al.: Google’s neural machine translation system: bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 (2016)
Xie, Q., Dai, Z., Hovy, E., Luong, M.T., Le, Q.V.: Unsupervised data augmentation. arXiv preprint arXiv:1904.12848 (2019)
Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., Hovy, E.: Hierarchical attention networks for document classification. In: Proceedings of NAACL-HLT, pp. 1480–1489. ACL, San Diego (2016). https://doi.org/10.18653/v1/N16-1174
Yu, A.W., et al.: QANet: combining local convolution with global self-attention for reading comprehension. arXiv preprint arXiv:1804.09541 (2018)
Zhang, Z., Sabuncu, M.R.: Generalized cross entropy loss for training deep neural networks with noisy labels. arXiv preprint arXiv:1805.07836 (2018)
Zhu, Y., et al.: Aligning books and movies: towards story-like visual explanations by watching movies and reading books. In: Proceedings of IEEE ICCV, pp. 19–27 (2015)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Nugent, T., Stelea, N., Leidner, J.L. (2021). Detecting Environmental, Social and Governance (ESG) Topics Using Domain-Specific Language Models and Data Augmentation. In: Andreasen, T., De Tré, G., Kacprzyk, J., Legind Larsen, H., Bordogna, G., Zadrożny, S. (eds) Flexible Query Answering Systems. FQAS 2021. Lecture Notes in Computer Science(), vol 12871. Springer, Cham. https://doi.org/10.1007/978-3-030-86967-0_12
Download citation
DOI: https://doi.org/10.1007/978-3-030-86967-0_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86966-3
Online ISBN: 978-3-030-86967-0
eBook Packages: Computer ScienceComputer Science (R0)