Skip to main content
Springer Nature Link
Log in

Enhancing Joint Entity and Relation Extraction with Language Modeling and Hierarchical Attention

  • Conference paper
  • First Online:
Web and Big Data (APWeb-WAIM 2019)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 11641))

  • 1684 Accesses

Abstract

Both entity recognition and relation extraction can benefit from being performed jointly, allowing them to enhance each other. However, existing methods suffer from the sparsity of relevant labels and strongly rely on external natural language processing tools, leading to error propagation. To tackle these problems, we propose an end-to-end joint framework for entity recognition and relation extraction with an auxiliary training objective on language modeling, i.e., learning to predict surrounding words for each word in sentences. Furthermore, we incorporate hierarchical multi-head attention mechanisms into the joint extraction model to capture vital semantic information from the available texts. Experiments show that the proposed approach consistently achieves significant improvements on joint extraction task of entities and relations as compared with strong baselines.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Nadeau, D., Sekine, S.: A survey of named entity recognition and classification. Lingvisticae Investigationes 30(1), 3–26 (2007)

    Article  Google Scholar 

  2. Zheng, S., Wang, F., Bao, H., Hao, Y., Zhou, P., Xu, B.: Joint extraction of entities and relations based on a novel tagging scheme. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pp. 1227–1236 (2017)

    Google Scholar 

  3. Bekoulis, G., Deleu, J., Demeester, T., Develder, C.: Adversarial training for multi-context joint entity and relation extraction. In: Empirical Methods in Natural Language Processing, pp. 2830–2836 (2018)

    Google Scholar 

  4. Roth, D., Yih, W.: A linear programming formulation for global inference in natural language tasks. In: Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL 2004) at HLT-NAACL 2004, pp. 1–8. Association for Computational Linguistics, Boston, USA (2004)

    Google Scholar 

  5. Chiu, J.P., Nichols, E.: Named entity recognition with bidirectional LSTM-CNNs. Trans. Assoc. Comput. Linguist. 4(1), 357–370 (2016)

    Article  Google Scholar 

  6. Liu, L., et al.: Empower sequence labeling with task-aware neural language model. In: Thirty-Second AAAI Conference on Artificial Intelligence, pp. 5253–5260 (2018)

    Google Scholar 

  7. Zhang, X., Cheng, J., Lapata, M.: Dependency parsing as head selection. In: Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, Valencia, Spain, pp. 665–676 (2017)

    Google Scholar 

  8. Verga, P., Strubell, E., McCallum, A.: Simultaneously self-attending to all mentions for full-abstract biological relation extraction. In: 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 872–884 (2018)

    Google Scholar 

  9. Bekoulis, G., Deleu, J., Demeester, T., Develder, C.: Joint entity recognition and relation extraction as a multi-head selection problem. Expert Syst. Appl. 114, 34–45 (2018)

    Article  Google Scholar 

  10. Le, H.-Q., Can, D.-C., Vu, S.T., Dang, T.H., Pilehvar, M.T., Collier, N.: Large-scale exploration of neural relation classification architectures. In: 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2266–2277 (2018)

    Google Scholar 

  11. Feng, J., Huang, M., Zhao, L., Yang, Y., Zhu, X.: Reinforcement learning for relation classification from noisy data. In: Thirty-Second AAAI Conference on Artificial Intelligence, pp. 5779–5786 (2018)

    Google Scholar 

  12. Gao, T., Han, X., Liu, Z., Sun, M.: Hybrid attention-based prototypical networks for noisy few-shot relation classification. In: Thirty-Third AAAI Conference on Artificial Intelligence (2019)

    Google Scholar 

  13. Ren, X., et al.: Cotype: joint extraction of typed entities and relations with knowledge bases. In: Proceedings of the 26th International Conference on World Wide Web, pp. 1015–1024. International World Wide Web Conferences Steering Committee (2017)

    Google Scholar 

  14. Wang, S., Zhang, Y., Che, W., Liu, T.: Joint extraction of entities and relations based on a novel graph scheme. In: 27th International Joint Conference on Artificial Intelligence, pp. 4461–4467 (2018)

    Google Scholar 

  15. Lafferty, J., McCallum, A., Pereira, F.C.: Conditional random fields: probabilistic models for segmenting and labeling sequence data. In: Proceedings of the 18th International Conference on Machine Learning, pp. 282–289 (2001)

    Google Scholar 

  16. Miwa, M., Bansal, M.: End-to-end relation extraction using LSTMs on sequences and tree structures. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp. 1105–1116 (2016)

    Google Scholar 

  17. Katiyar, A., Cardie, C.: Investigating LSTMs for joint extraction of opinion entities and relations. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp. 919–929 (2016)

    Google Scholar 

  18. Gupta, P., Schütze, H., Andrassy, B.: Table filling multi-task recurrent neural network for joint entity and relation extraction. In: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp. 2537–2547 (2016)

    Google Scholar 

  19. Adel, H., Schütze, H.: Global normalization of convolutional neural networks for joint entity and relation classification. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1723–1729. Association for Computational Linguistics, Copenhagen, Denmark (2017)

    Google Scholar 

  20. Xie, R., Liu, Z., Jia, J., Luan, H., Sun, M.: Representation learning of knowledge graphs with entity descriptions. In: AAAI, pp. 2659–2665 (2016)

    Google Scholar 

  21. Vaswani, A., Bisk, Y., Sagae, K., Musa, R.: Supertagging with LSTMs. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 232–237 (2016)

    Google Scholar 

  22. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)

  23. Pennington, J., Socher, R., Manning, C.: Glove: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)

    Google Scholar 

  24. Li, Q., Ji, H.: Incremental joint extraction of entity mentions and relations. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pp. 402–412 (2014)

    Google Scholar 

  25. Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., Dyer, C.: Neural architectures for named entity recognition. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 260–270 (2016)

    Google Scholar 

  26. Miwa, M., Sasaki, Y.: Modeling joint entity and relation extraction with table representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1858–1869 (2014)

    Google Scholar 

  27. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)

    Google Scholar 

  28. Li, F., Zhang, M., Fu, G., Ji, D.: A neural joint model for entity and relation extraction from biomedical text. BMC Bioinform. 18(1), 198 (2017)

    Article  Google Scholar 

  29. Katiyar, A., Cardie, C.: Going out on a limb: joint extraction of entity mentions and relations without dependency trees. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, Canada, pp. 917–928 (2017)

    Google Scholar 

  30. Bekoulis, G., Deleu, J., Demeester, T., Develder, C.: An attentive neural architecture for joint segmentation and parsing and its application to real estate ads. Expert Syst. Appl. 102, 100–112 (2018)

    Article  Google Scholar 

Download references

Acknowledgment

This work is supported by the National Key Research and Development Program of China (2018YFC0831500), National Natural Science Foundation of China (No. 61772082), National Natural Science Foundation of China (No. 61806020), the Fundamental Research Funds for the Central Universities and Big Data Research Foundation of PICC.

Author information

Authors and Affiliations

  1. Beijing Key Laboratory of Intelligent Telecommunications Software and Multimedia, Beijing University of Posts and Telecommunications, Beijing, China

    Renjun Chi, Bin Wu, Linmei Hu & Yunlei Zhang

Authors
  1. Renjun Chi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bin Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Linmei Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yunlei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Linmei Hu .

Editor information

Editors and Affiliations

  1. University of Electronic Science and Technology of China, Chengdu, China

    Jie Shao

  2. Hong Kong Polytechnic University, Hong Kong, China

    Man Lung Yiu

  3. The University of Tokyo, Tokyo, Japan

    Masashi Toyoda

  4. Zhejiang University, Hangzhou, China

    Dongxiang Zhang

  5. National University of Singapore, Singapore, Singapore

    Wei Wang

  6. Peking University, Beijing, China

    Bin Cui

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chi, R., Wu, B., Hu, L., Zhang, Y. (2019). Enhancing Joint Entity and Relation Extraction with Language Modeling and Hierarchical Attention. In: Shao, J., Yiu, M., Toyoda, M., Zhang, D., Wang, W., Cui, B. (eds) Web and Big Data. APWeb-WAIM 2019. Lecture Notes in Computer Science(), vol 11641. Springer, Cham. https://doi.org/10.1007/978-3-030-26072-9_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-26072-9_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-26071-2

  • Online ISBN: 978-3-030-26072-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics