Abstract
Learning meaningful representations for different granularities of texts is a challenging and on-going area of research in natural language processing. Recently, neural sentence modeling that learns continuous valued vector representations for sentences in a low dimensional latent semantic space has gained increasing attention. In this work, we propose a novel method to learn meaning representation for variable-sized sentence based on recursive auto-encoders. The key difference between our model and others is that we embed the sentence meaning while jointly learning evolved word representation in unsupervised manner and without using any parse or dependency tree. Our deep compositional model is not only able to construct meaningful sentence representation but also to keep pace with the words meanings evolving. We evaluate our obtained embeddings on semantic similarity task. The experimental results show the effectiveness of our proposed model and demonstrate that it can achieve a competitive performance without any feature engineering.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Harris, Z.S.: Distributional structure. Word 10(2–3), 146–162 (1954)
Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: ICLR (2013)
Pennington, J., Socher, R., Manning, C.: Glove: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, 25–29 October, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 1532–1543 (2014)
Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors with subword information. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017)
Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R.S., Torralba, A., Urtasun, R., Fidler, S.: Skip-thought vectors. In: NIPS (2015)
Kalchbrenner, N., Grefenstette, E., Blunsom, P.: A convolutional neural network for modelling sentences. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pp. 655–665 (2014)
Kim, Y.: Convolutional neural networks for sentence classification. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1746–1751 (2014)
Conneau, A., Kiela, D., Schwenk, H., Barrault, L., Bordes, A.: Supervised learning of universal sentence representations from natural language inference data. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (2017)
Subramanian, S., Trischler, A., Bengio, Y., Pal, C.J.: Learning general purpose distributed sentence representations via large scale multi-task learning. In: International Conference on Learning Representations (2018)
Socher, R., Huang, E.H., Pennin, J., Manning, C., Ng, A.Y.: Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In: Advances in Neural Information Processing Systems, USA, pp. 801–809 (2011)
Socher, R., Karpathy, A., Le, Q.V., Manning, C.D., Ng, A.Y.: Grounded compositional semantics for finding and describing images with sentences. Trans. Assoc. Comput. Linguist. 2, 207–218 (2014)
Tai, K.S., Socher, R., Manning, C.D.: Improved semantic representations from tree-structured long short-term memory networks. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pp. 1556–1566 (2015)
Bowman, S.R., Gauthier, J., Rastogi, A., Gupta, R., Manning, C.D., Potts, C.: Fast unified model for parsing and sentence understanding. ACL (2016)
Dyer, C., Kuncoro, A., Ballesteros, M., Smith, N.A.: Recurrent neural network grammars. In: NAACL, San Diego, California, 12–17 June 2016, pp. 199–209 (2016)
Hill, F., Cho, K., Korhonen, A.: Learning distributed representations of sentences from unlabelled data. In: NAACL-HLT, San Diego, California, 12–17 June 2016, pp. 1367–1377 (2016)
Grover, J., Mitra, P.: Sentence alignment using unfolding recursive autoencoders. In: Proceedings of the 10th Workshop on Building and Using Comparable Corpora ACL, Vancouver, Canada, 3 August 2017, pp. 16–20 (2017)
Kingma, D.P., Ba, J.L.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations, pp. 1–13 (2015)
Bjerva, J., Bos, J., Van der Goot, R., Nissim, M.: The meaning factory: formal semantics for recognizing textual entailment and determining semantic similarity. In: SemEval COLING, pp. 642–646 (2014)
Zhao, J., Zhu, T., Lan, M.: ECNU: one stone two birds: ensemble of heterogenous measures for semantic relatedness and textual entailment. In: SemEval COLING, pp. 271–277 (2014)
Lai, A., Hockenmaier, J.: Illinois-LH: a denotational and distributional approach to semantics. In: International Workshop on Semantic Evaluation, Dublin, Ireland (2014)
Jimenez, S., Dueñas, G., Baquero, J., Gelbukh, A.: UNAL-NLP: combining soft cardinality features for semantic textual similarity, relatedness and entailment. In: Proceedings of the 8th International Workshop on Semantic Evaluation (SemeVal), Dublin, Ireland (2014)
Banea, C., Hassan, S., Mohler, M., Mihalcea, R.: UNT: a supervised synergistic approach to semantic text similarity. In: Proceedings of SemEval 2012, pp. 635–642 (2012)
Bar, D., Biemann, C., Gurevych, I., Zesch, T.: UKP: computing semantic textual similarity by combining multiple content similarity measures. In: Proceedings of SemEval 2012, pp. 435–440 (2012)
Saric, R., Glavas, G., Karan, M., Snajder, J., Dalbelo, B.: TakeLab: systems for measuring semantic text similarity. In: Proceedings of SemEval 2012, Montreal, Canada, pp. 441–448 (2012)
Kenter, T., Borisov, A., Rijke, M.D.: Siamese CBOW: optimizing word embeddings for sentence representations. CoRR abs/1606.04640 (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Bouraoui, A., Jamoussi, S., Ben Hamadou, A. (2019). Learning Meaningful Sentence Embedding Based on Recursive Auto-encoders. In: Macintyre, J., Iliadis, L., Maglogiannis, I., Jayne, C. (eds) Engineering Applications of Neural Networks. EANN 2019. Communications in Computer and Information Science, vol 1000. Springer, Cham. https://doi.org/10.1007/978-3-030-20257-6_17
Download citation
DOI: https://doi.org/10.1007/978-3-030-20257-6_17
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-20256-9
Online ISBN: 978-3-030-20257-6
eBook Packages: Computer ScienceComputer Science (R0)