Skip to main content
SpringerLink
Log in

Enhanced Sentence Meta-Embeddings for Textual Understanding

  • Conference paper
  • First Online:
Advances in Information Retrieval (ECIR 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13186))

Included in the following conference series:

  • 2478 Accesses

Abstract

Sentence embeddings provide vector representations for sentences and short texts, enabling the capture of contextual and semantic meaning for different applications. However, the diversity of sentence embedding techniques poses a challenge, in terms of choosing the model best suited for the downstream task. As such, meta-embeddings study different techniques for combining embeddings from multiple sources. In this paper, we propose CINCE, a principled meta-embedding framework for aggregating various semantic information, captured by different embeddings techniques, via multiple component analysis strategies. Experiments on SentEval benchmark exhibit improved performance for semantic understanding and text classification, compared to existing approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Andrew, G., Arora, R., Bilmes, J., Livescu, K.: Deep canonical correlation analysis. In: International Conference on Machine Learning (ICML), pp. 1247–1255 (2013)

    Google Scholar 

  2. Artetxe, M., Schwenk, H.: Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Trans. Assoc. Comput. Linguist. 7, 597–610 (2019)

    Article  Google Scholar 

  3. Assem, H., Dutta, S., Burgin, E.: DTAFA: decoupled training architecture for efficient FAQ retrieval. In: Annual Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL), pp. 423–430 (2021)

    Google Scholar 

  4. Bao, C., Bollegala, D.: Learning word meta-embeddings by autoencoding. In: Conference on Computational Linguistics (COLING), pp. 1650–1661 (2018)

    Google Scholar 

  5. Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017)

    Article  Google Scholar 

  6. Cer, D., et al.: Universal Sentence Encoder, arXiv preprint arXiv:1803.11175 (2018)

  7. Chelba, C., et al.: One billion word benchmark for measuring progress in statistical language modeling. In: Conference of the International Speech Communication Association (INTERSPEECH), pp. 2635–2639 (2014)

    Google Scholar 

  8. Chen, W., Sheng, M., Mao, J., Sheng, W.: Investigating word meta-embeddings by disentangling common and individual information. IEEE Access 8, 11692–11699 (2020)

    Article  Google Scholar 

  9. Coates, J.N., Bollegala, D.: Frustratingly easy meta-embedding - computing meta-embeddings by averaging source word embeddings. In: North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pp. 194–198 (2018)

    Google Scholar 

  10. Comon, P.: Independent component analysis - a new concept? Signal Process. 36, 287–314 (1994)

    Article  MATH  Google Scholar 

  11. Conneau, A., Kiela, D.: SentEval: an evaluation toolkit for universal sentence representations. In: International Conference on Language Resources and Evaluation (LREC), pp. 1699–1704 (2018)

    Google Scholar 

  12. Conneau, A., Kiela, D., Schwenk, H., Barrault, L., Borders, A.: Supervised learning of universal sentence representations from natural language inference data. In: Empirical Methods in Natural Language Processing (EMNLP), pp. 670–680 (2017)

    Google Scholar 

  13. Conneau, A., Kruszewski, G., Lample, G., Barrault, L., Baroni, M.: What you can cram into a single \$&!#* vector: probing sentence embeddings for linguistic properties. In: Annual Meeting of the Association for Computational Linguistics (ACL), pp. 2126–2136 (2018)

    Google Scholar 

  14. Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pp. 4171–4186 (2019)

    Google Scholar 

  15. Dutta, S.: “Alignment is All You Need”: analyzing cross-lingual text similarity for domain-specific applications. In: The Web Conference - Workshop on Cross-lingual Event-centric Open Analytics, pp. 85–92 (2021)

    Google Scholar 

  16. Goswami, K., Dutta, S., Assem, H., Fransen, T., McCrae, J.: Cross-lingual sentence embedding using multi-task learning. In: Empirical Methods in Natural Language Processing (EMNLP) (2021)

    Google Scholar 

  17. Hazem, A., Hernandez, N.: Meta-embedding sentence representation for textual similarity. In: Recent Advances in Natural Language Processing (RANLP), pp. 465–473 (2019)

    Google Scholar 

  18. Hirota, W., Suhara, Y., Golshan, B., Tan, W.C.: EMU: enhancing multilingual sentence embeddings with semantic specialization. In: AAAI Conference on Artificial Intelligence (AAAI), pp. 7935–7943 (2020)

    Google Scholar 

  19. Hotelling, H.: Analysis of a complex of statistical variables into principal components. J. Educ. Psychol. 24(417–441), 498–520 (1933)

    Article  MATH  Google Scholar 

  20. Hotelling, H.: Relations between two sets of variates. Biometrika 28(3–4), 321–377 (1936)

    Article  MATH  Google Scholar 

  21. Hyvärinen, A., Oja, E.: Independent component analysis: algorithms and applications. Neural Netw. 13(4–5), 411–430 (2000)

    Article  Google Scholar 

  22. Jutten, C., Hérault, J.: Blind separation of sources, Part I: an adaptive algorithm based on neuromimetic architecture. Signal Process. 24, 1–10 (1991)

    Article  MATH  Google Scholar 

  23. Kettenring, J.R.: Canonical analysis of several sets of variables. Biometrika 58, 433–451 (1971)

    Article  MathSciNet  MATH  Google Scholar 

  24. Kiela, D., Wang, C., Cho, K.: Dynamic meta-embeddings for improved sentence representations. In: Empirical Methods in Natural Language Processing (EMNLP), pp. 1466–1477 (2018)

    Google Scholar 

  25. Knapp, T.R.: Canonical correlation analysis: a general parametric significance-testing system. Psychol. Bull. 85(2), 410–416 (1978)

    Article  MathSciNet  Google Scholar 

  26. Kornblith, S., Norouzi, M., Lee, H., Hinton, G.: Similarity of neural network representations revisited. In: International Conference on Machine Learning (ICML), pp. 3519–3529 (2019)

    Google Scholar 

  27. Lample, G., Conneau, A.: Cross-lingual language model pretraining. In: Conference on Neural Information Processing Systems (NIPS), pp. 1–11 (2019)

    Google Scholar 

  28. Lange, L., Adel, H., Strötgen, J., Klakow, D.: Adversarial learning of feature-based meta-embeddings (2020). arXiv:2010.12305

  29. Li, B., Zhou, H., He, J., Wang, M., Yang, Y., Li, L.: On the sentence embeddings from pre-trained language models. In: Empirical Methods in Natural Language Processing (EMNLP), pp. 9119–9130 (2020)

    Google Scholar 

  30. Lin, Z., et al.: A structured self-attentive sentence embedding. In: International Conference on Learning Representations (ICLR), pp. 1–15 (2017)

    Google Scholar 

  31. Mika, S., Schölkopf, B., Smola, A., Müller, K.R., Scholz, M., Rätsch, G.: Kernel PCA and de-noising in feature spaces. In: Conference on Neural Information Processing Systems (NIPS), pp. 536–542 (1998)

    Google Scholar 

  32. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Conference on Neural Information Processing Systems (NIPS), pp. 3111–3119 (2013)

    Google Scholar 

  33. O’Neill, J., Bollegala, D.: Meta-embedding as auxiliary task regularization. In: European Conference on Artificial Intelligence (ECAI), pp. 2124–2131 (2020)

    Google Scholar 

  34. Patel, R.N., Burgin, E., Assem, H., Dutta, S.: Efficient multi-lingual sentence classification framework with sentence meta encoders. In: IEEE International Conference on Big Data (2021)

    Google Scholar 

  35. Pearson, K.: On lines and planes of closest fit to systems of points in space. Phil. Mag. 2(11), 559–572 (1901)

    Article  MATH  Google Scholar 

  36. Pennington, J., Socher, R., Manning, C.D.: GloVe: global vectors for word representation. In: Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)

    Google Scholar 

  37. Poerner, N., Waltinger, U., Schütze, H.: Sentence meta-embeddings for unsupervised semantic textual similarity. In: Annual Meeting of the Association for Computational Linguistics (ACL), pp. 7027–7034 (2020)

    Google Scholar 

  38. Reimers, N., Gurevych, I.: Sentence-BERT: sentence embeddings using Siamese BERT-networks. In: Empirical Methods in Natural Language Processing (EMNLP), pp. 3982–3992 (2019)

    Google Scholar 

  39. Schölkopf, B.: Nonlinear component analysis as a kernel eigenvalue problem. Neural Comput. 10(5), 1299–1319 (1998)

    Article  Google Scholar 

  40. Senel, L.K., Utlu, I., Yucesoy, V., Koc, A., Cukur, T.: Semantic structure and interpretability of word embeddings. IEEE/ACM Trans. Audio Speech Lang. Process. 26(10), 1769–1779 (2018)

    Article  Google Scholar 

  41. Shi, X.: Nonlinear ICA, pp. 97–112. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-11347-5_5

    Book  Google Scholar 

  42. Sørensen, M., Kanatsoulis, C.I., Sidiropoulos, N.D.: Generalized canonical correlation analysis: a subspace intersection approach. arXiv:2003.11205 (2020)

  43. Tsatsishvilil, V., Cong, F., Toiviainen, P., Ristaniemil, T.: Combining PCA and multiset CCA for dimension reduction when group ICA is applied to decompose naturalistic fMRI data. In: International Joint Conference on Neural Networks (IJCNN), pp. 1–6 (2015)

    Google Scholar 

  44. Wieting, J., Gimpel, K.: ParaNMT-50M: pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In: Annual Meeting of the Association for Computational Linguistics (ACL), pp. 451–462 (2018)

    Google Scholar 

  45. Wieting, J., Kiela, D.: No Training required: exploring random encoders for sentence classification. In: International Conference on Learning Representations (ICLR), pp. 1–16 (2019)

    Google Scholar 

  46. Yin, W., Schütze, H.: Learning word meta-embeddings. In: Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1351–1360 (2016)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Huawei Research Centre, Dublin, Ireland

    Sourav Dutta & Haytham Assem

Authors
  1. Sourav Dutta
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haytham Assem
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sourav Dutta .

Editor information

Editors and Affiliations

  1. Martin Luther University Halle-Wittenberg, Halle, Germany

    Matthias Hagen

  2. Leiden University, Leiden, The Netherlands

    Suzan Verberne

  3. University of Glasgow, Glasgow, UK

    Craig Macdonald

  4. University of Duisburg-Essen, Essen, Germany

    Christin Seifert

  5. University of Stavanger, Stavanger, Norway

    Krisztian Balog

  6. Norwegian University of Science and Technology, Trondheim, Norway

    Kjetil Nørvåg

  7. University of Stavanger, Stavanger, Norway

    Vinay Setty

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Dutta, S., Assem, H. (2022). Enhanced Sentence Meta-Embeddings for Textual Understanding. In: Hagen, M., et al. Advances in Information Retrieval. ECIR 2022. Lecture Notes in Computer Science, vol 13186. Springer, Cham. https://doi.org/10.1007/978-3-030-99739-7_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-99739-7_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-99738-0

  • Online ISBN: 978-3-030-99739-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation