Abstract
Geological reports are records of the geological elements and survey contents found in geological exploration, but it is difficult to extract useful concepts from such reports. In the process of information extraction, accurately identification of entities in unstructured geotext is a foundational task that is known as geological named entity recognition (Geo-NER). However, the existing methods generally require a large number of annotated corpora, and face problems with long entity recognition. Therefore, this paper proposes a two-stage fine-tuning method. In the first fine-tuning stage, we use a bidirectional encoder representations from transformers language model with geological domain knowledge (GeoBERT), which combines geological domain knowledge, on a pretrained BERT model, and in the second stage, we use a small number of samples to complete the NER task in the geological report based on GeoBERT. Our proposed model achieves a very high F1-score compared to baseline models on the constructed dataset.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Akkasi A, Varoglu E (2017) Improving biochemical named entity recognition using PSO classifier selection and Bayesian combination methods. IEEE/ACM Transactions on Computational Biology and Bioinformatics 14(6):1327–1338. https://doi.org/10.1109/TCBB.2016.2570216
Atkinson J, Bull V (2012) A multi-strategy approach to biological named entity recognition. Expert Syst Appl 39(17):12968–12974. https://doi.org/10.1016/j.eswa.2012.05.033
Bao Y, Wu M, Chang S, Barzilay R (2019) Few-shot text classification with distributional signatures. ArXiv. http://arxiv.org/abs/1908.06039
Chu D, Wan B, Li H, Fang F, Wang R (2020) Geological entity recognition based on ELMO-CNN-BiLSTM-CRF model. Earth Sci:1–22
Dai Z, Wang X, Ni P, Li Y, Li G, Bai X (2019) Named entity recognition using BERT BiLSTM CRF for Chinese electronic health records. In: 2019 12th international congress on image and signal processing, BioMedical engineering and informatics (CISP-BMEI), pp 1–5. https://doi.org/10.1109/CISP-BMEI48845.2019.8965823
Devlin J, Chang MW, Lee K, Toutanova K (2018) BERT: pre-training of deep bidirectional transformers for language understanding. ArXiv. https://arxiv.org/abs/1810.04805v2
Fan R, Wang L, Yan J, Song W, Zhu Y, Chen X (2019) Deep learning-based named entity recognition and knowledge graph construction for geological hazards. ISPRS Int J Geo Inf 9(1):15. https://doi.org/10.3390/ijgi9010015
Fu R, Qin B, Liu T (2014) Generating Chinese named entity data from parallel corpora. Front Comput Sci 8(4):629–641. https://doi.org/10.1007/s11704-014-3127-5
Hofer M, Kormilitzin A, Goldberg P, Nevado-Holgado A (2018) Few-shot learning for named entity recognition in medical text. ArXiv. http://arxiv.org/abs/1811.05468
Howard J, Ruder S (2018) Universal language model fine-tuning for text classification. ArXiv. http://arxiv.org/abs/1801.06146
Huang C, Li Y, Zhu X (2006) Tokenization Guidelines of 67
Huang Z, Xu W, Yu K (2015) Bidirectional LSTM-CRF models for sequence tagging. ArXiv:1508.01991 [Cs]. http://arxiv.org/abs/1508.01991
Ji B, Liu R, Xu WS, Li SS, Tang JT, Yu J, Li Q (2018) A BILSTM-CRF method to Chinese electronic medical record named entity recognition. In: ACM international conference proceeding series, pp 1–6. https://doi.org/10.1145/3302425.3302465
Ju Z, Wang J, Zhu F (2011) Named entity recognition from biomedical text using SVM. In: 5th international conference on bioinformatics and biomedical engineering, ICBBE 2011, pp 1–4. https://doi.org/10.1109/icbbe.2011.5779984
Liu W, Yu B, Zhang C, Wang H, Pan K (2018) Chinese named entity recognition based on rules and conditional random field. In: ACM international conference proceeding series, pp 268–272. https://doi.org/10.1145/3297156.3297196
Liu H, Jun G, Zheng Y (2021) Chinese named entity recognition model based on BERT. MATEC Web of Conferences 336:06021. https://doi.org/10.1051/MATECCONF/202133606021
Luo L, Yang Z, Yang P, Zhang Y, Wang L, Lin H, Wang J (2018) An attention-based BiLSTM-CRF approach to document-level chemical named entity recognition. Bioinformatics 34(8):1381–1388. https://doi.org/10.1093/bioinformatics/btx761
Mikolov T, Chen K, Corrado G, Dean J (2013) Efficient estimation of word representations in vector space. In: 1st international conference on learning representations, ICLR 2013 - workshop track proceedings http://arxiv.org/abs/1301.3781
Naacl A (2019) Few-shot text classification with induction network. Naacl:1–10
Nadeau D, Sekine S (2007) A survey of named entity recognition and classification. International Journal of Linguistics and Language Resources Lingvisticæ Investigationes 30(1):3–26. https://doi.org/10.1075/li.30.1.03nad
Pauls A, Klein D (2011) Faster and smaller n-gram language models. In: ACL-HLT 2011 - proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies (Vol. 1) http://code.google.com/p/berkeleylm/
Qiu Q, Xie Z, Wu L, Tao L (2019a) GNER: a generative model for geological named entity recognition without labeled data using deep learning. Earth and Space Science 6(6):931–946. https://doi.org/10.1029/2019EA000610
Qiu Q, Xie Z, Wu L, Tao L, Li W (2019b) BiLSTM-CRF for geological named entity recognition from the geoscience literature. Earth Sci Inf 12(4):565–579. https://doi.org/10.1007/s12145-019-00390-3
Ratinov L, Roth D (2009) Design challenges and misconceptions in named entity recognition. In: CoNLL 2009 - proceedings of the thirteenth conference on computational natural language learning, pp 147–155. https://doi.org/10.3115/1596374.1596399
Santoro A, Bartunov S, Botvinick M, Wierstra D, Lillicrap T (2016) Meta-learning with memory-augmented neural networks. In: International conference on machine learning, pp 1842–1850
Shen Y, Yun H, Lipton ZC, Kronrod Y, Anandkumar A (2018) Deep active learning for named entity recognition. In: 6th international conference on learning representations, ICLR 2018 - conference track proceedings http://arxiv.org/abs/1707.05928
Singh A, Thakur N, Sharma A (2016) A review of supervised machine learning algorithms. In: Proceedings of the 10th INDIACom; 2016 3rd international conference on computing for sustainable global development, INDIACom 2016, pp 1310–1315
Sobhana N, Mitra P, Ghosh SK (2010) Conditional random field based named entity recognition in geological text. Int J Comput Appl 1(3):143–147. https://doi.org/10.5120/72-166
Strubell E, Verga P, Belanger D, McCallum A (2017) Fast and accurate entity recognition with iterated dilated convolutions. ArXiv. http://arxiv.org/abs/1702.02098
Tang S, Zhang N, Zhang J, Wu F, Zhuang Y (2017) NITE: a neural inductive teaching framework for domain-specific NER. In: EMNLP 2017 - conference on empirical methods in natural language processing, proceedings, pp 2652–2657. https://doi.org/10.18653/v1/d17-1280
Wang YX, Hebert M (2016) Learning to learn: model regression networks for easy small sample learning. In: Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics), 9910 LNCS, pp 616–634. https://doi.org/10.1007/978-3-319-46466-4_37
Wang C, Ma X, Chen J, Chen J (2018a) Information extraction and knowledge graph construction from geoscience literature. Comput Geosci 112:112–120. https://doi.org/10.1016/j.cageo.2017.12.007
Wang S, Zhang X, Ye P, Du M (2018b) Deep Belief Networks Based Toponym Recognition for Chinese Text. ISPRS International Journal of Geo-Information 2018 7(6):217. https://doi.org/10.3390/IJGI7060217
Wang Y, Yao Q, Kwok JT, Ni LM (2019) Generalizing from a few examples: a survey on few-shot learning. ACM Comput Surv 53(3). https://doi.org/10.1145/3386252
Wang J, Hu Y, Joseph K (2020a) NeuroTPR: a neuro-net toponym recognition model for extracting locations from social media messages. Trans GIS 24(3):719–735. https://doi.org/10.1111/TGIS.12627
Wang, Yu, Sun Y, Ma Z, Gao L, Xu Y (2020b) An ERNIE-based joint model for Chinese named entity recognition. Appl Sci 10(16):5711. https://doi.org/10.3390/APP10165711
Wu L, Liu L, Li H, Gao Y (2017) A Chinese toponym recognition method based on conditional random field. Wuhan Daxue Xuebao (Xinxi Kexue Ban)/Geomatics and Information Science of Wuhan University 42(2):150–156. https://doi.org/10.13203/J.WHUGIS20141009
Xiang X, Shi X, Applications H. Z.-C., & 2005, U. (2005) Chinese named entity recognition system using statistics-based and rules-based method. En.Cnki.Com.Cn
Yang SM, Yoo SY, Jeong OR (2020) DeNERT-KG: named entity and relation extraction model using DQN, knowledge graph, and BERT. Appl Sci 10(18):6429. https://doi.org/10.3390/APP10186429
Zaremba W, Sutskever I, Vinyals O (2014) Recurrent neural network regularization. ArXiv:1409.2329 [Cs]. http://arxiv.org/abs/1409.2329
Zhang X, Ye P, Wang S, Du M (2018) Geological entity recognition method based on deep belief networks. Yanshi Xuebao/Acta Petrologica Sinica 34(2):343–351
Zhou P, Shi W, Tian J, Qi Z, Li B, Hao H, Xu B (2016) Attention-based bidirectional long short-term memory networks for relation classification. In: 54th annual meeting of the Association for Computational Linguistics, ACL 2016 - short papers, pp 207–212. https://doi.org/10.18653/v1/p16-2034
Zhou F, Cao C, Zhong T, Geng J (2021) Learning meta-knowledge for few-shot image emotion recognition. Expert Syst Appl 168:114274. https://doi.org/10.1016/j.eswa.2020.114274
Acknowledgments
This study was supported by the National Science Foundation of China (Grant No. 41871311, 42050101), Open Research Project of The Hubei Key Laboratory of Intelligent Geo-Information Processing (No. KLIGIP-2021A01), and the Fundamental Research Funds for the Central Universities, China University of Geosciences (Wuhan) (No. CUG2106116)). The authors thank the Development and Research Center of the China Geological Survey for providing technical support. We thank the National Engineering Research Center of Geographic Information System for providing hardware support.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Declaration of interests
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Additional information
Communicated by: H. Babaie
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
• The study proposes a two-stage fine-tuning method for name entity recognition.
• Considering the problem of small sample datasets and long entity identification.
• Capturing long-distance dependency features within longer geological entities.
• The method has achieved better performances compared to other models.
Rights and permissions
About this article
Cite this article
Liu, H., Qiu, Q., Wu, L. et al. Few-shot learning for name entity recognition in geological text based on GeoBERT. Earth Sci Inform 15, 979–991 (2022). https://doi.org/10.1007/s12145-022-00775-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12145-022-00775-x