Abstract
Recently, dramatic performance improvement in computing has enabled a breakthrough in machine learning technologies. Against this background, generating distributed representation of discrete symbols such as natural languages and images has attracted considerable interest. In the field of natural language processing, word2vec, a method to generate distributed representations of words is well known and its effectiveness well reported. However, an effective method to generate the distributed representation of sentences and documents has not yet been reported.
In this study, we propose a method of generating the distributed representation of sentences by using an autoencoder based on bi-directional long short-term memory (BiLSTM). To obtain the information and findings that necessary to generate effective representations, the computational experiments are carried out.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Pennington, J., Socher, R., Manning, C.D.: Glove: global vectors for word representation. In: EMNLP, vol. 14, pp. 1532–1543 (2014)
Vinyals, O., Toshev, A., Bengio, S., Erhan, D.: Show and tell: a neural image caption generator. In: Computer Vision and Pattern Recognition (2015)
Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. CoRR, abs/1301.3781 (2013)
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Burges, C.J.C., Bottou, L., Welling, M., Ghahramani, Z., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 26, pp. 3111–3119. Curran Associates, Inc. (2013)
Logeswaran, L., Lee, H.: An efficient framework for learning sentence representations. In: International Conference on Learning Representations (2018)
Ponti, E.M., Vulic, I., Korhonen, A.: Decoding sentiment from distributed representations of sentences. CoRR, abs/1705.06369 (2017)
Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)
Gers, F.A., Schmidhuber, J.A., Cummins, F.A.: Learning to forget: continual prediction with LSTM. Neural Comput. 12(10), 2451–2471 (2000)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9, 1735–1780 (1997)
Le, Q.V., Mikolov, T.: Distributed representations of sentences and documents. CoRR, abs/1405.4053 (2014)
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, NIPS 2014, vol. 2, pp. 3104–3112. MIT Press, Cambridge (2014)
Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. Trans. Sig. Proc. 45(11), 2673–2681 (1997)
Shōsetsuka ni narō. https://syosetu.com/
Kiros, R., Zhu, Y., Salakhutdinov, R., Zemel, R.S., Torralba, A., Urtasun, R., Fidler, S.: Skip-thought vectors. arXiv preprint arXiv:1506.06726 (2015)
Acknowledgements
This work was supported by JSPS KAKENHI Grant, Grant-in-Aid for Scientific Research(C), 26330282.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Fukuda, K., Mori, N., Matsumoto, K. (2019). A Novel Sentence Vector Generation Method Based on Autoencoder and Bi-directional LSTM. In: De La Prieta, F., Omatu, S., Fernández-Caballero, A. (eds) Distributed Computing and Artificial Intelligence, 15th International Conference. DCAI2018 2018. Advances in Intelligent Systems and Computing, vol 800. Springer, Cham. https://doi.org/10.1007/978-3-319-94649-8_16
Download citation
DOI: https://doi.org/10.1007/978-3-319-94649-8_16
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-94648-1
Online ISBN: 978-3-319-94649-8
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)