Skip to main content
SpringerLink
Log in

Sentence Matching with Deep Self-attention and Co-attention Features

  • Conference paper
  • First Online:
Knowledge Science, Engineering and Management (KSEM 2021)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12816))

Abstract

Sentence matching refers to extracting the semantic relation between two sentences which is widely applied in many natural language processing tasks such as natural language inference, paraphrase identification and question answering. Many previous methods apply a siamese network to capture semantic features and calculate cosine similarity to represent sentences relation. However, they could be effective for overall rough sentence semantic but not sufficient for word-level matching information. In this paper, we proposed a novel neural network based on attention mechanism which focuses on learning richer interactive features of two sentences. There are two complementary components in our model: semantic encoder and interactive encoder. Interactive encoder compares sentences semantic features which are encoded by semantic encoder. In addition, semantic encoder considers the output of interactive encoder as supplementary matching features. Experiments on three benchmark datasets proved that self-attention network and cross-attention network can efficiently learn the semantic and interactive features of sentences, and achieved state-of-the-art results.

Supported by National Key Research and Development Program of China under Grant 2018YFC0831502 and State Grid Shandong Electric Power Company Science and Technology Project Funding (2020A-074).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bowman, S.R., Angeli, G., Potts, C., Manning, C.D.: A large annotated corpus for learning natural language inference (2015)

    Google Scholar 

  2. Wang, Z., Hamza, W., Florian, R.: Bilateral multi-perspective matching for natural language sentences (2017)

    Google Scholar 

  3. Gong, Y., Luo, H., Zhang, J.: Natural language inference over interaction space (2017)

    Google Scholar 

  4. Huang, P.-S., et al.: Learning deep structured semantic models for web search using clickthrough data. In: Proceedings of the 22nd ACM International Conference on Information & Knowledge Management, pp. 2333–2338 (2013)

    Google Scholar 

  5. Chen, Q., Zhu, X., Ling, Z., Wei, S., Jiang, H., Inkpen, D.: Enhanced lstm for natural language inference (2016)

    Google Scholar 

  6. Liu, P., Qiu, X., Chen, J., Huang, X.-J.: Deep fusion lstms for text semantic matching. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1034–1043 (2016)

    Google Scholar 

  7. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)

    Google Scholar 

  8. Parikh, A.P., Täckström, O., Das, D., Uszkoreit, J.: A decomposable attention model for natural language inference (2016)

    Google Scholar 

  9. Kim, S., Kang, I., Kwak, N.: Semantic sentence matching with densely-connected recurrent and co-attentive information. In: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 6586–6593 (2019)

    Google Scholar 

  10. Tan, C., Wei, F., Wang, W., Lv, W., Zhou, M.: Multiway attention networks for modeling sentence pairs. In: IJCAI, pp. 4411–4417 (2018)

    Google Scholar 

  11. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  12. Ba, J.L., Kiros, J.R., Hinton, G.E.: Layer normalization (2016)

    Google Scholar 

  13. Gai, K., Qiu, M.: Reinforcement learning-based content-centric services in mobile sensing. IEEE Network 32(4), 34–39 (2018)

    Article  Google Scholar 

  14. Dai, W., Qiu, L., Wu, A., Qiu, M.: Cloud infrastructure resource allocation for big data applications. IEEE Trans. Big Data 4(3), 313–324 (2016)

    Article  Google Scholar 

  15. Gai, K., Qiu, M., Zhao, H., Sun, X.: Resource management in sustainable cyber-physical systems using heterogeneous cloud computing. IEEE Trans. Sustainable Comput. 3(2), 60–72 (2017)

    Article  Google Scholar 

  16. Romano, L., Kouylekov, M., Szpektor, I., Dagan, I., Lavelli, A.: Investigating a generic paraphrase-based approach for relation extraction. In: 11th Conference of the European Chapter of the Association for Computational Linguistics (2006)

    Google Scholar 

  17. Conneau, A., Kiela, D., Schwenk, H., Barrault, L., Bordes, A.: Supervised learning of universal sentence representations from natural language inference data (2017)

    Google Scholar 

  18. Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543. 2014

    Google Scholar 

  19. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)

    Google Scholar 

  20. Wang, S., Jiang, J.: A compare-aggregate model for matching text sequence. arXiv preprint arXiv:1611.01747 2016

  21. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826 (2016)

    Google Scholar 

  22. Yang, R., Zhang, J., Gao, X., Ji, F., Chen, H.: Simple and effective text matching with richer alignment features (2019)

    Google Scholar 

  23. Tay, Y., Luu, A.T., Hui, S.C.: Hermitian co-attention networks for text matching in asymmetrical domains. In: IJCAI, pp. 4425–4431 (2018)

    Google Scholar 

  24. Tay, Y., Tuan, L.A., Hui, S.C.: Compare, compress and propagate: Enhancing neural architectures with alignment factorization for natural language inference (2017)

    Google Scholar 

  25. Tay, Y., Tuan, L.A., Hui, S.C.: Co-stack residual affinity networks with multi-level attention refinement for matching text sequences (2018)

    Google Scholar 

  26. Khot, T., Sabharwal, A., Clark, P.: SciTaiL: a textual entailment dataset from science question answering. In: AAAI, pp. 41–42 (2018)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications, Beijing, China

    Zhipeng Wang & Danfeng Yan

Authors
  1. Zhipeng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Danfeng Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Danfeng Yan .

Editor information

Editors and Affiliations

  1. Tsinghua University, Beijing, China

    Han Qiu

  2. Ibaraki University, Hitachi, Japan

    Cheng Zhang

  3. University of Kentucky, Lexington, KY, USA

    Zongming Fei

  4. Texas A&M University – Commerce, Commerce, TX, USA

    Meikang Qiu

  5. Princeton University, Princeton, NJ, USA

    Sun-Yuan Kung

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Wang, Z., Yan, D. (2021). Sentence Matching with Deep Self-attention and Co-attention Features. In: Qiu, H., Zhang, C., Fei, Z., Qiu, M., Kung, SY. (eds) Knowledge Science, Engineering and Management . KSEM 2021. Lecture Notes in Computer Science(), vol 12816. Springer, Cham. https://doi.org/10.1007/978-3-030-82147-0_45

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-82147-0_45

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-82146-3

  • Online ISBN: 978-3-030-82147-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation