Skip to main content
SpringerLink
Log in

Surface pattern-enhanced relation extraction with global constraints

  • Regular Paper
  • Published:
Knowledge and Information Systems Aims and scope Submit manuscript

Abstract

Relation extraction is one of the most important tasks in information extraction. The traditional works either use sentences or surface patterns (i.e., the shortest dependency paths of sentences) to build extraction models. Intuitively, the integration of these two kinds of methods will further obtain more robust and effective extraction models, which is, however, ignored in most of the existing works. In this paper, we aim to learn the embeddings of surface patterns to further augment the sentence-based models. To achieve this purpose, we propose a novel pattern embedding learning framework with the weighted multi-dimensional attention mechanism. To suppress noise in the training dataset, we mine the global statistics between patterns and relations and introduce two kinds of prior knowledge to guide the pattern embedding learning. Based on the learned embeddings, we present two augmentation strategies to improve the existing relation extraction models. We conduct extensive experiments on two popular datasets (i.e., NYT and KnowledgeNet) and observe promising performance improvements.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. The same entity pair appearing in k different sentences will be counted as k times.

  2. To simplify the code implementation, the length of \(p_j\) is either truncated or padded to \(l=20\) with “null.”

  3. Notice that the test set, i.e., the fold 5, in KnowledgeNet is unavailable.

  4. The results of PCNN+ATT+GloRE and PCNN +ATT+LoRE are from the authors’ GitHub, i.e., https://github.com/ppuliu/GloRE.

  5. For convenience, we only conduct experiments on NYT dataset.

References

  1. Auger A, Barrière C (2008) Pattern-based approaches to semantic relation extraction: A state-of-the-art. Terminology 14(1):1–19

    Article  Google Scholar 

  2. Alicante A, Corazza A, Isgrò F, Silvestri S (2016) Unsupervised entity and relation extraction from clinical records in italian. Comput Biol Med 72:263–275

    Article  Google Scholar 

  3. Alicante A, Corazza A (2011) Barrier features for classification of semantic relations. Proceedings of the international conference recent advances in natural language processing 2011, pp. 509–514

  4. Büttcher S, Clarke CL, Cormack GV (2016) Information retrieval: Implementing and evaluating search engines. MIT Press,

  5. Bekoulis G, Deleu J, Demeester T, Develder C (2018) An attentive neural architecture for joint segmentation and parsing and its application to real estate ads. Expert Syst Appl 102:100–112

    Article  Google Scholar 

  6. Bethard S, Martin JH (2008) Learning semantic links from a corpus of parallel temporal and causal relations. In: Proceedings of the 46th annual meeting of the association for computational linguistics on human language technologies: short papers, pp. 177–180. Association for computational linguistics

  7. Bordes A, Glorot X, Weston J, Bengio Y (2012) Joint learning of words and meaning representations for open-text semantic parsing. In: Artificial intelligence and statistics, pp. 127–135

  8. Bunescu RC, Mooney RJ (2005) A shortest path dependency kernel for relation extraction. In: Proceedings of the conference on human language technology and empirical methods in natural language processing, pp. 724–731. Association for Computational Linguistics

  9. Chen D, Manning C (2014) A fast and accurate dependency parser using neural networks. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 740–750

  10. Culotta A, Sorensen J (2004) Dependency tree kernels for relation extraction. In: Proceedings of the 42nd annual meeting on association for computational linguistics, p. 423. Association for computational linguistics

  11. de Sá Mesquita F, Cannaviccio M, Schmidek J, Mirza P, Barbosa D (2019) Knowledgenet: A benchmark dataset for knowledge base population. In: EMNLP/IJCNLP

  12. Devlin J, Chang M-W, Lee K, Toutanova K (2018) Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805

  13. Eichler K, Xu F, Uszkoreit H, Hennig L, Krause S (2016) Teg-rep: A corpus of textual entailment graphs based on relation extraction patterns. In: LREC

  14. Feng J, Huang M, Zhao L, Yang Y, Zhu X (2018) Reinforcement learning for relation classification from noisy data. In: Thirty-second AAAI conference on artificial intelligence

  15. Girju R, Badulescu A, Moldovan D (2003) Learning semantic constraints for the automatic discovery of part-whole relations. In: Proceedings of the 2003 conference of the north american chapter of the association for computational linguistics on human language technology-volume 1, pp. 1–8. Association for computational linguistics

  16. Hoffart J, Suchanek FM, Berberich K, Weikum G (2013) Yago2: A spatially and temporally enhanced knowledge base from wikipedia. Artif Intell 194:28–61

    Article  MathSciNet  Google Scholar 

  17. Hoffmann R, Zhang C, Ling X, Zettlemoyer L, Weld DS (2011) Knowledge-based weak supervision for information extraction of overlapping relations. In: Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies-volume 1, pp. 541–550. Association for computational linguistics

  18. Jiang X, Wang Q, Li P, Wang B (2016) Relation extraction with multi-instance multi-label convolutional neural networks. In: Proceedings of COLING 2016, the 26th international conference on computational linguistics, pp. 1471–1480

  19. Ji G, He S, Xu L, Liu K, Zhao J (2015) Knowledge graph embedding via dynamic mapping matrix. In: Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (vol 1: Long Papers), vol 1, pp. 687–696

  20. Ji G, Liu K, He S, Zhao J (2017) Distant supervision for relation extraction with sentence-level attention and entity descriptions. In: Thirty-first AAAI conference on artificial intelligence

  21. Kambhatla N (2004) Combining lexical, syntactic, and semantic features with maximum entropy models for extracting relations. In: Proceedings of the ACL 2004 on Interactive poster and demonstration sessions, p. 22. Association for computational linguistics

  22. Katiyar A, Cardie C (2017) Going out on a limb: Joint extraction of entity mentions and relations without dependency trees. In: Proceedings of the 55th annual meeting of the association for computational linguistics (Vol. 1: Long Papers), vol 1, pp. 917–928

  23. Kim Y (2014) Convolutional neural networks for sentence classification. arXiv:1408.5882

  24. Lin Y, Shen S, Liu Z, Luan H, Sun M (2016) Neural relation extraction with selective attention over instances. In: Proceedings of the 54th annual meeting of the association for computational linguistics (vol. 1: Long Papers), vol. 1, pp. 2124–2133

  25. Liu Y, Li S, Wei F, Ji H (2016) Relation classification via modeling augmented dependency paths. IEEE/ACM Trans Audio, Speech and Lang Process (TASLP) 24(9):1585–1594

    Google Scholar 

  26. Liu Y, Wei F, Li S, Ji H, Zhou M, Wang H (2015) A dependency-based neural network for relation classification. arXiv:1507.04646

  27. Mikolov T, Sutskever I, Chen K, Corrado GS, Dean J (2013) Distributed representations of words and phrases and their compositionality. In: Advances in neural information processing systems, pp. 3111–3119

  28. Mintz M, Bills S, Snow R, Jurafsky D (2009) Distant supervision for relation extraction without labeled data. In Proceedings of the joint conference of the 47th annual meeting of the ACL and the 4th international joint conference on natural language processing of the AFNLP: vol. 2, pp. 1003–1011. Association for computational linguistics

  29. Miwa M, Bansal M (2016) End-to-end relation extraction using lstms on sequences and tree structures. arXiv:1601.00770

  30. Mooney RJ, Bunescu RC (2006) Subsequence kernels for relation extraction. In: Advances in neural information processing systems, pp. 171–178

  31. Napoles C, Gormley M, Van Durme B (2012) Annotated gigaword. In: Proceedings of the joint workshop on automatic knowledge base construction and web-scale knowledge extraction, pp. 95–100. Association for computational linguistics

  32. Nguyen TH, Grishman R (2015) Combining neural networks and log-linear models to improve relation extraction. arXiv:1511.05926

  33. Nguyen TH, Grishman R (2015) Relation extraction: Perspective from convolutional neural networks. In: Proceedings of the 1st workshop on vector space modeling for natural language processing, pp. 39–48

  34. Pennington J, Socher R, Manning C (2014) Glove: Global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532–1543

  35. Qin P, Xu W, Wang WY (2018) Robust distant supervision relation extraction via deep reinforcement learning. arXiv:1805.09927

  36. Qu M, Ren X, Zhang Y, Han J (2018) Weakly-supervised relation extraction by pattern-enhanced embedding learning. In: Proceedings of the 2018 world wide web conference on world wide web, pp. 1257–1266. International world wide web conferences steering committee

  37. Rath T (2017) Word and relation embedding for sentence representation. PhD thesis, Arizona State University

  38. Ratner AJ, De Sa CM, Wu S, Selsam D, Ré C (2016) Data programming: creating large training sets, quickly. In: Advances in neural information processing systems, pp. 3567–3575

  39. Riedel S, Yao L, McCallum A (2010) Modeling relations and their mentions without labeled text. In: Joint european conference on machine learning and knowledge discovery in databases, pp. 148–163. Springer

  40. Riedel S, Yao L, McCallum A, Marlin BM (2013) Relation extraction with matrix factorization and universal schemas. In: Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: human language technologies, pp. 74–84

  41. Santos CNd, Xiang B, Zhou B (2015) Classifying relations by ranking with convolutional neural networks. arXiv:1504.06580

  42. Shanahan T, Fisher D, Frey N (2016) The challenge of challenging text. On developing readers: readings from educational leadership (EL Essentials), p. 100

  43. Shen T, Zhou T, Long G, Jiang J, Zhang C (2018) Bi-directional block self-attention for fast and memory-efficient sequence modeling. arXiv:1804.00857

  44. Snow R, Jurafsky D, Ng AY (2005) Learning syntactic patterns for automatic hypernym discovery. In: Advances in neural information processing systems, pp. 1297–1304

  45. Su Y, Liu H, Yavuz S, Gur I, Sun H, Yan X (2017) Global relation embedding for relation extraction. arXiv:1704.05958

  46. Sun X (2014) Structure regularization for structured prediction. In: Advances in neural information processing systems, pp. 2402–2410

  47. Surdeanu M, Tibshirani J, Nallapati R, Manning CD (2012) Multi-instance multi-label learning for relation extraction. In: Proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning, pp. 455–465. Association for computational linguistics

  48. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I (2017) Attention is all you need. In: Advances in neural information processing systems, pp. 5998–6008

  49. Völkel M, Krötzsch M, Vrandecic D, Haller H, Studer R (2006) Semantic wikipedia. In: Proceedings of the 15th international conference on world wide web, pp. 585–594. ACM

  50. Wang L, Cao Z, de Melo G, Liu Z (2016) Relation classification via multi-level attention cnns

  51. Wu Y, Bamman D, Russell S (2017) Adversarial training for relation extraction. In: Proceedings of the 2017 conference on empirical methods in natural language processing, pp. 1778–1783

  52. Xu K, Feng Y, Huang S, Zhao D (2015) Semantic relation classification via convolutional neural networks with simple negative sampling. arXiv:1506.07650

  53. Xu Y, Jia R, Mou L, Li G, Chen Y, Lu Y, Jin Z (2016) Improved relation classification by deep recurrent neural networks with data augmentation. arXiv:1601.03651

  54. Xu Y, Mou L, Li G, Chen Y, Peng H, Jin Z (2015) Classifying relations via long short term memory networks along shortest dependency paths. In: Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 1785–1794

  55. Yang L, Ng TLJ, Mooney C, Dong R (2017) Multi-level attention-based neural networks for distant supervised relation extraction. In: 25th Irish conference on artificial intelligence and cognitive science, Dublin, Ireland, 7-8 December 2017. Insight centre

  56. Yao L, Riedel S, McCallum A (2010) Collective cross-document relation extraction without labelled data. In: Proceedings of the 2010 conference on empirical methods in natural language processing, pp. 1013–1023. Association for computational linguistics

  57. Ye H, Chao W, Luo Z, Li Z (2016) Jointly extracting relations with class ties via effective deep ranking. arXiv:1612.07602

  58. Yu AW, Dohan D, Luong M-T, Zhao R, Chen K, Norouzi M, Le QV (2018) Qanet: Combining local convolution with global self-attention for reading comprehension. arXiv:1804.09541

  59. Zelenko D, Aone C, Richardella A (2003) Kernel methods for relation extraction. J Mach Learn Res 3(Feb):1083–1106

    MathSciNet  MATH  Google Scholar 

  60. Zeng X, He S, Liu K, Zhao J (2018) Large scaled relation extraction with reinforcement learning. Relation 2:3

    Google Scholar 

  61. Zeng D, Liu K, Chen Y, Zhao J (2015) Distant supervision for relation extraction via piecewise convolutional neural networks. In: Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 1753–1762

  62. Zeng D, Liu K, Lai S, Zhou G, Zhao J, et al. (2014) Relation classification via convolutional deep neural network

  63. Zhang X, Zhao J, LeCun Y (2015)Character-level convolutional networks for text classification. In: Advances in neural information processing systems, pp. 649–657

  64. Zhang Y, Zhong V, Chen D, Angeli G, Manning CD (2017) Position-aware attention and supervised data improve slot filling. In: Proceedings of the 2017 conference on empirical methods in natural language processing, pp. 35–45

  65. Zhou P, Xu J, Qi Z, Bao H, Chen Z, Xu B (2018) Distant supervision for relation extraction with hierarchical selective attention. Neural Netw 108:240–247

    Article  Google Scholar 

  66. Zhu W, Yao T, Ni J, Wei B, Lu Z (2018) Dependency-based siamese long short-term memory network for learning sentence representations. PloS one 13(3):e0193919

    Article  Google Scholar 

Download references

Funding

This paper was supported by Shanghai Science and Technology Innovation Action Plan (No. 19511120400).

Author information

Authors and Affiliations

  1. Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University, Shanghai, China

    Haiyun Jiang, JunTao Liu, Yanghua Xiao & Wei Wang

  2. Ping An Health Technology, Shanghai, China

    Sheng Zhang

  3. School of Data Science, Fudan University, Shanghai, China

    Deqing Yang

Authors
  1. Haiyun Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. JunTao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sheng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Deqing Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yanghua Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Wei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yanghua Xiao.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jiang, H., Liu, J., Zhang, S. et al. Surface pattern-enhanced relation extraction with global constraints. Knowl Inf Syst 62, 4509–4540 (2020). https://doi.org/10.1007/s10115-020-01502-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10115-020-01502-y

Keywords

Search

Navigation