Abstract
This work proposes the PetroBERT, which is a BERT-based model adapted to the oil and gas exploration domain in Portuguese. PetroBERT was pre-trained using the Petrolês corpus and a private daily drilling report corpus over BERT multilingual and BERTimbau. The proposed model was evaluated in the NER and sentence classification tasks and achieved interesting results, which shows its potential for such a domain. To the best of our knowledge, this is the first BERT-based model to the oil and gas context.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Alzubi, J.A., Jain, R., Singh, A., Parwekar, P., Gupta, M.: COBERT: Covid-19 question answering system using Bert. Arabian J. Sci. Eng. 1–11 (2021). https://doi.org/10.1007/s13369-021-05810-5
Amaral, D.O.F.: Reconhecimento de entidades nomeadas na área da geologia: bacias sedimentares brasileiras. Ph.D. thesis, Programa de Pós-Graduação em Ciência da Computação (2017). http://tede2.pucrs.br/tede2/handle/tede/8035, escola Politécnica
Araci, D.: Finbert: Financial sentiment analysis with pre-trained language models (2019). http://arxiv.org/abs/1908.10063
Beltagy, I., Lo, K., Cohan, A.: SciBERT: a pretrained language model for scientific text. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3615–3620. Association for Computational Linguistics, Hong Kong, China (2019)
Cinelli, L.P.: Automatic event identification and extraction from daily drilling reports using an expert system and artificial intelligence. J. Petrol. Sci. Eng. 205, 108939 (2021)
Consoli, B., Santos, J., Gomes, D., Cordeiro, F., Vieira, R., Moreira, V.: Embeddings for named entity recognition in geoscience Portuguese literature. In: Proceedings of the 12th Language Resources and Evaluation Conference, pp. 4625–4630. European Language Resources Association, Marseille, France, May 2020
Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Burstein, J., Doran, C., Solorio, T. (eds.) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, 2–7 June 2019, Volume 1 (Long and Short Papers), pp. 4171–4186. Association for Computational Linguistics (2019)
Hoffimann, J., Mao, Y., Wesley, A., Taylor, A.: Sequence mining and pattern analysis in drilling reports with deep natural language processing. In: SPE Annual Technical Conference and Exhibition, vol. Day 3 Wed, 26 September 2018 (2018)
Huang, K., Altosaar, J., Ranganath, R.: ClinicalBERT: modeling clinical notes and predicting hospital readmission (2020)
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C.H., Kang, J.: BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36(4), 1234–1240 (2019)
Liu, Z., Huang, D., Huang, K., Li, Z., Zhao, J.: FinBERT: a pre-trained financial language representation model for financial text mining. In: Bessiere, C. (ed.) Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pp. 4513–4519. International Joint Conferences on Artificial Intelligence Organization (2020)
Ribeiro, L.C., Afonso, L.C., Colombo, D., Guilherme, I.R., Papa, J.P.: Evolving neural conditional random fields for drilling report classification. J. Petrol. Sci. Eng. 187, 106846 (2020)
Schneider, E.T.R., et al.: BioBERTpt - a Portuguese neural language model for clinical named entity recognition. In: Proceedings of the 3rd Clinical Natural Language Processing Workshop, pp. 65–72. Association for Computational Linguistics, November 2020
Shen, J.T., Yamashita, M., Prihar, E., Heffernan, N., Wu, X., Graff, B., Lee, D.: MathBERT: a pre-trained language model for general NLP tasks in mathematics education (2021)
da Silva Magalhães Gomes, D., et al.: Portuguese word embeddings for the oil and gas industry: development and evaluation. Comput. Indus. 124, 103347 (2021)
Sousa, G.J., et al.: Pattern analysis in drilling reports using optimum-path forest. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2018)
Wu, Y., et al.: Google’s neural machine translation system: bridging the gap between human and machine translation. CoRR abs/1609.08144 (2016). http://arxiv.org/abs/1609.08144
Acknowledgments
The authors would like to thank Petróleo Brasileiro (Petrobras) grant 2019/00697-8. The authors are also grateful to FAPESP grants #2013/07375-0, #2014/12236-1, #2018/15597-6, and #2019/07665-4, and CNPq grants #307066/2017-7, #309439/2020-5 and #427968/2018-6.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Rodrigues, R.B.M. et al. (2022). PetroBERT: A Domain Adaptation Language Model for Oil and Gas Applications in Portuguese. In: Pinheiro, V., et al. Computational Processing of the Portuguese Language. PROPOR 2022. Lecture Notes in Computer Science(), vol 13208. Springer, Cham. https://doi.org/10.1007/978-3-030-98305-5_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-98305-5_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-98304-8
Online ISBN: 978-3-030-98305-5
eBook Packages: Computer ScienceComputer Science (R0)