Skip to main content

Advertisement

SpringerLink
Log in

A Controlled Attention for Nested Named Entity Recognition

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Traditional methods to recognize named entities are conducted as sequence labelling or span classification. They are usually implemented on a raw input without any cue about possible named entities. This method cannot be aware of entity boundaries and learn semantic dependencies between them. Cognitive neuroscience has revealed that foveating stimuli improves the efficiency of processing in terms of acuity. Inspired by this phenomenon, we propose a controlled attention mechanism for recognizing named entities. In our method, instead of feeding a raw input into a neural network, task-related cues are implanted into each sentence to indicate boundaries of possible named entities. Then, the modified sentence is sent into a deep network to learn a discriminative entity-relevant sentence representation. In our experiments, the controlled attention is evaluated on English and Chinese corpora. Comparing with existing models, it shows significant improvement for nested named entity recognition. We achieve the state-of-the-art performance in all evaluation datasets. The controlled attention has three advantages for named entity recognition. First, it enables a neural network to become aware of entity boundaries and construct semantic dependencies relevant to possible entities. Second, implanting entity cues enables a neural network to concentrate on the task-related semantic features while disregarding nonessential information in a sentence. Third, the controlled attention also has the potentiality to be extended for other NLP tasks, e.g., entity relation extraction and event extraction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

Data Availability

All evaluation datasets in our experiments, including the ACE and the GENIA, are public datasets. They are available online.

Notes

  1. Symbols in this picture are cuneiform characters, Latin transcription, and English words.

References

  1. Yang S, Tu K. Bottom-up constituency parsing and nested named entity recognition with pointer networks. In: Proceedings of the ACL ’22. 2022. p. 2403–16.

  2. Katiyar A, Cardie C. Nested named entity recognition revisited. In: Proceedings of the NAACL ’18. New Orleans, Louisiana: ACL; 2018. p. 861–71.

  3. Straková J, Straka M, Hajic J. Neural architectures for nested NER through linearization. In: Proceedings of the ACL ’19. Florence, Italy: ACL; 2019. p. 5326–31.

  4. Lin H, Lu Y, Han X, Sun L. Sequence-to-nuggets: nested entity mention detection via anchor-region networks. In: Proceedings of the ACL ’19. Florence, Italy: ACL; 2019. p. 5182–92.

  5. Tan Z, Shen Y, Zhang S, Lu W, Zhuang Y. A sequence-to-set network for nested named entity recognition. arXiv:2105.08901 [Preprint]. 2021. Available from: http://arxiv.org/abs/2105.08901.

  6. Yuan Z, Tan C, Huang S, Huang F. Fusing heterogeneous factors with triaffine mechanism for nested named entity recognition. In: Proceedings of the ACL ’22. 2022. p. 3174–86.

  7. Lou C, Yang S, Tu K. Nested named entity recognition as latent lexicalized constituency parsing. In: Proceedings of the ACL ’22. Dublin, Ireland: Association for Computational Linguistics. 2022. p. 6183–98.

  8. Zhu E, Li J. Boundary smoothing for named entity recognition. In: Proceedings of the ACL ’22. 2022. p. 7096–108.

  9. Chen Y, Wu Y, Qin Y, Hu Y, Wang Z, Huang R, Cheng X, Chen P. Recognizing nested named entity based on the neural network boundary assembling model. IEEE Intell Syst. 2020;35(1):74–81.

    Article  Google Scholar 

  10. Chen Y, Wu L, Zheng Q, Huang R, Liu J, Deng L, Yu J, Qing Y, Dong B, Chen P. A boundary regression model for nested named entity recognition. Cogn Comput. 2022. p. 1–18.

  11. Shen Y, Ma X, Tan Z, Zhang S, Wang W, Lu W. Locate and label: a two-stage identifier for nested named entity recognition. In: Proceedings of the ACL-IJCNLP ‘21. 2021. p. 2782–94.

  12. Levy O, Goldberg Y. Neural word embedding as implicit matrix factorization. Adv Neural Inf Process Syst. 2014;27:2177–85.

    Google Scholar 

  13. Yang Z, Dai Z, Salakhutdinov R, Cohen WW. Breaking the softmax bottleneck: a high-rank RNN language model. arXiv:1711.03953 [Preprint]. 2017. Available from: http://arxiv.org/abs/1711.03953.

  14. Li B, Zhou H, He J, Wang M, Yang Y, Li L. On the sentence embeddings from pre-trained language models. arXiv:2011.05864 [Preprint]. 2020. Available from: http://arxiv.org/abs/2011.05864.

  15. Posner MI, Petersen SE. The attention system of the human brain. Annu Rev Neurosci. 1990;13(1):25–42.

    Article  Google Scholar 

  16. Groome D. An introduction to cognitive psychology: processes and disorders. Psychology Press; 1999. https://doi.org/10.4324/9780203977989.

  17. Namazi M, Thordardottir E. A working memory, not bilingual advantage, in controlled attention. Int J Biling Educ Biling. 2010;13(5):597–616.

    Article  Google Scholar 

  18. Zhou G, Su J. Named entity recognition using an HMM-based chunk tagger. In: Proceedings of the ACL ’02. ACL; 2002. p. 473–80.

  19. McCallum A, Li W. Early results for named entity recognition with conditional random fields, feature induction and web-enhanced lexicons. In: Proceedings of the HLT-NAAC ’03. ACL; 2003. p. 188–91.

  20. Huang Z, Xu W, Yu K. Bidirectional LSTM-CRF models for sequence tagging. arXiv:1508.01991 [Preprint]. 2015. Available from: http://arxiv.org/abs/1508.01991.

  21. Alex B, Haddow B, Grover C. Recognising nested named entities in biomedical text. In: Proceedings of the BioNLP ’07. USA: ACL; 2007. p. 65–72.

  22. Zhang J, Shen D, Zhou G, Su J, Tan C-L. Enhancing HMM-based biomedical named entity recognition by studying special phenomena. J Biomed Inform. 2004;37(6):411–22.

    Article  Google Scholar 

  23. McDonald R, Crammer K, Pereira F. Flexible text segmentation with structured multilabel classification. In: Proceedings of EMNLP ’05. 2005. p. 987–94.

  24. Ju M, Miwa M, Ananiadou S. A neural layered model for nested named entity recognition. In: Proceedings of the NAACL ’18. New Orleans, Louisiana: ACL; 2018. p. 1446–59.

  25. Fisher J, Vlachos A. Merge and label: a novel neural network architecture for nested NER. arXiv:1907.00464 [Preprint]. 2019. Available from: http://arxiv.org/abs/1907.00464.

  26. Wang J, Shou L, Chen K, Chen G. Pyramid: a layered model for nested named entity recognition. In: Proceedings of the ACL ’20. Online: ACL; 2020. p. 5918–28.

  27. Lu W, Roth D. Joint mention extraction and classification with mention hypergraphs. In: Proceedings of the EMNLP ’15. Lisbon, Portugal: ACL; 2015. p. 857–67.

  28. Wang B, Lu W, Wang Y, Jin H. A neural transition-based model for nested mention recognition. arXiv:1810.01808 [Preprint]. 2018. Available from: http://arxiv.org/abs/1810.01808.

  29. Finkel JR, Manning CD. Joint parsing and named entity recognition. In: Proceedings of the NAACL ’09. USA: ACL; 2009. p. 326–34.

  30. Jie Z, Muis AO, Lu W. Efficient dependency-guided named entity recognition. In: Proceedings of the AAAI ’17. AAAI Press; 2017. p. 3457–65.

  31. Jie Z, Lu W. Dependency-guided LSTM-CRF for named entity recognition. In: Proceedings of the EMNLP-IJCNLP ’19. Hong Kong, China: Association for Computational Linguistics; 2019. p. 3862–72.

  32. Xu M, Jiang H. A FOFE-based local detection approach for named entity recognition and mention detection. arXiv:1611.00801 [Preprint]. 2016. Available from: http://arxiv.org/abs/1611.00801.

  33. Sohrab MG, Miwa M. Deep exhaustive model for nested named entity recognition. In: Proceedings of the EMNLP ’18. Brussels, Belgium: ACL; 2018. p. 2843–9.

  34. Xia C, Zhang C, Yang T, Li Y, Du N, Wu X, Fan W, Ma F, Yu P. Multi-grained named entity recognition. In: Proceedings of the ACL ’19. ACL; 2019. p. 1430–40.

  35. Yongming N, Yanping C, Yongbin Q, Ruizhang H, Ruixue T, Ying H. A joint model for entity boundary detection and entity span recognition. J King Saud Univ - Comput Inf Sci. 2022;34(10):8362–9.

    Google Scholar 

  36. Li X, Feng J, Meng Y, Han Q, Wu F, Li J. A unified MRC framework for named entity recognition. In: Proceedings of the ACL-20. 2020. p. 5849–58.

  37. Li J, Fei H, Liu J, Wu S, Zhang M, Teng C, Ji D, Li F. Unified named entity recognition as word-word relation classification. In: Proceedings of the AAAI ‘22, vol. 36, no. 10. 2022. p. 10965–73.

  38. Wu D, Luo X, He Y, Zhou M. A prediction-sampling-based multilayer-structured latent factor model for accurate representation to high-dimensional and sparse data. IEEE Trans Neural Netw Learn Syst. 2022.

  39. Li J, Chiu B, Feng S, Wang H. Few-shot named entity recognition via meta-learning. IEEE Trans Knowl Data Eng. 2020.

  40. Li J, Shang S, Chen L. Domain generalization for named entity boundary detection via metalearning. IEEE Trans Neural Netw Learn Syst. 2020;32(9):3819–30.

    Article  Google Scholar 

  41. Yu J, Bohnet B, Poesio M. Named entity recognition as dependency parsing. Meeting of the Association for Computational Linguistics; 2020.

  42. Li J, Xu K, Chaudhuri S, Yumer E, Zhang H, Guibas L. Grass: generative recursive autoencoders for shape structures. ACM Trans Graph. 2017;36(4):1–14.

    Google Scholar 

  43. Shi Y, Xu X, Xi J, Hu X, Hu D, Xu K. Learning to detect 3D symmetry from single-view RGB-D images with weak supervision. IEEE Trans Pattern Anal Mach Intell. 2022.

  44. Li J, Han P, Ren X, Hu J, Chen L, Shang S. Sequence labeling with meta-learning. IEEE Trans Knowl Data Eng. 2021.

  45. Kalchbrenner N, Grefenstette E, Blunsom P. A convolutional neural network for modelling sentences. arXiv:1404.2188 [Preprint]. 2014. Available from: http://arxiv.org/abs/1404.2188.

  46. Strubell E, Verga P, Belanger D, McCallum A. Fast and accurate entity recognition with iterated dilated convolutions. In: Proceedings of the EMNLP ’17. Copenhagen, Denmark: Association for Computational Linguistics; 2017. p. 2670–80.

  47. Galassi A, Lippi M, Torroni P. Attention in natural language processing. IEEE Trans Neural Netw. 2020;1–18.

  48. Zhu Y, Wang G. CAN-NER: Convolutional Attention Network for Chinese Named Entity Recognition. In: Proceedings of the NAACL-HLT ’19. Minneapolis, Minnesota: Association for Computational Linguistics. 2019. p. 3384–93.

  49. Luo G, Yuan Q, Li J, Wang S, Yang F. Artificial intelligence powered mobile networks: from cognition to decision. IEEE Network. 2022;36(3):136–44.

    Article  Google Scholar 

  50. Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate. arXiv:1409.0473 [Preprint]. 2014. Available from: http://arxiv.org/abs/1409.0473.

  51. Devlin J, Chang M-W, Lee K, Toutanova K. Bert: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. 2018. Available from: https://arxiv.org/abs/1810.04805

  52. Muis AO, Lu W. Labeling gaps between words: recognizing overlapping mentions with mention separators. In: Proceedings of the EMNLP ’17. 2017. p. 2608–18.

  53. Wang B, Lu W. Neural segmental hypergraphs for overlapping mention recognition. arXiv:1810.01817 [Preprint]. 2018. Available from: http://arxiv.org/abs/1810.01817.

  54. Zheng C, Cai Y, Xu J, Leung H-F, Xu G. A boundary-aware neural model for nested named entity recognition. In: Proceedings of the EMNLP-IJCNLP ‘19. 2019. p. 357–66.

  55. Shibuya T, Hovy EH. Nested named entity recognition via second-best sequence learning and decoding. Trans Assoc Comput Linguist. 2020;8:605–20.

    Article  Google Scholar 

  56. Tan C, Qiu W, Chen M, Wang R, Huang F. Boundary enhanced neural span classification for nested named entity recognition. In: Proceedings of the AAAI ‘20, vol. 34, no. 05. 2020. p. 9016–23.

  57. Doddington G, Mitchell A, Przybocki M, Ramshaw L, Strassel S, Weischedel R. The automatic content extraction (ACE) program – tasks, data, and evaluation. In: Proceedings of the LREC ’04. Lisbon, Portugal: European Language Resources Association (ELRA). 2004.

  58. Kim J-D, Ohta T, Tateisi Y, Tsujii J. Genia corpus-a semantically annotated corpus for bio-textmining. Bioinformatics (Oxford, England). 2003;19(Suppl 1):i180-2.

    Google Scholar 

  59. Finkel JR, Manning CD. Nested named entity recognition. In: Proceedings of the EMNLP ’09. 2009. p. 141.

  60. Lee J, Yoon W, Kim S, Kim D, Kim S, So CH, Kang J. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. 2019.

  61. Lan Z, Chen M, Goodman S, Gimpel K, Sharma P, Soricut R. Albert: a lite Bert for self-supervised learning of language representations. arXiv:1909.11942 [Preprint]. 2019. Available from: http://arxiv.org/abs/1909.11942.

  62. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I. Attention is all you need. arXiv:1706.03762 [Preprint]. 2017. Available from: http://arxiv.org/abs/1706.03762.

  63. Chen Y, Zheng Q, Chen P. A boundary assembling method for chinese entity-mention recognition. IEEE Intell Syst. 2015;30(6):50–8.

    Article  Google Scholar 

  64. Vig J. A multiscale visualization of attention in the transformer model. arXiv:1906.05714 [Preprint]. 2019. Available from: http://arxiv.org/abs/1906.05714.

Download references

Funding

This work is supported by the Joint Funds of the National Natural Science Foundation of China Nos. 62166007 and 62066008.

Author information

Authors and Affiliations

  1. Text Computing & Cognitive Intelligence Engineering Research Center of National Education Ministry, Guizhou University, Guiyang, 550025, China

    Yanping Chen, Rong Huang, Lijun Pan & Ruizhang Huang

  2. College of Computer Science and Technology, Guizhou University, Guiyang, 550025, China

    Yanping Chen, Rong Huang, Lijun Pan & Ruizhang Huang

  3. College of Computer Science and Technology, Xi’an Jiaotong University, Xi’an, 710048, China

    Qinghua Zheng

  4. College of Science and Mathematics, University of Massachusetts Boston, Boston, MA, 02125, USA

    Ping Chen

  5. Information Communication Branch, Guizhou Power Grid Company, Guiyang, China

    Lijun Pan

Authors
  1. Yanping Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rong Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lijun Pan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ruizhang Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qinghua Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ping Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yanping Chen.

Ethics declarations

Ethical Approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Conflict of Interest

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, Y., Huang, R., Pan, L. et al. A Controlled Attention for Nested Named Entity Recognition. Cogn Comput 15, 132–145 (2023). https://doi.org/10.1007/s12559-023-10112-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-023-10112-z

Keywords

Search

Navigation