Abstract
Inferencing the entailment relations between natural language sentence pairs is fundamental to artificial intelligence. Recently, there is a rising interest in modeling the task with neural attentive models. However, those existing models have a major limitation to keep track of the attention history because usually only one single vector is utilized to memorize the past attention information. We argue its importance based on our observation that the potential alignment clues are not always centralized. Instead, they may diverge substantially, which could cause the problem of long-range dependency. In this paper, we propose to facilitate the conventional attentive reading operations with two sophisticated writing operations - forget and update. Instead of utilizing a single vector that accommodates the attention history, we write the past attention information directly into the sentence representations. Therefore, higher memory capacity of attention history could be achieved. Experiments on Stanford Natural Language Inference corpus (SNLI) demonstrate the superior efficacy of our proposed architecture.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Dagan, I., Glickman, O., Magnini, B.: The PASCAL recognising textual entailment challenge. In: Quiñonero-Candela, J., Dagan, I., Magnini, B., d’Alché-Buc, F. (eds.) MLCW 2005. LNCS (LNAI), vol. 3944, pp. 177–190. Springer, Heidelberg (2006). https://doi.org/10.1007/11736790_9
Lakoff, G.: Linguistics and natural logic. Synthese 22(1), 151–271 (1970)
MacCartney, B.: Natural Language Inference. Stanford University, Stanford (2009)
Pavlick, E.: Compositional lexical semantics in natural language inference. Ph.D. dissertation, University of Pennsylvania (2017)
Bowman, S.R., Potts, C., Manning, C.D.: Recursive neural networks for learning logical semantics. CoRR, abs/1406.1827 (2014). http://arxiv.org/abs/1406.1827
Bowman, S.R., Potts, C., Manning, C.D.: Learning distributed word representations for natural logic reasoning. In: Proceedings of the Association for the Advancement of Artificial Intelligence Spring Symposium, AAAI, pp. 10–13 (2015)
Bowman, S.R., Angeli, G., Potts, C., Manning, C.D.: A large annotated corpus for learning natural language inference. CoRR, abs/1508.05326 (2015). http://arxiv.org/abs/1508.05326
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473 (2014). http://arxiv.org/abs/1409.0473
Daniluk, M., Rocktäschel, T., Welbl, J., Riedel, S.: Frustratingly short attention spans in neural language modeling. CoRR, abs/1702.04521 (2017). http://arxiv.org/abs/1702.04521
Graves, A., Wayne, G., Danihelka, I.: Neural turing machines. CoRR, abs/1410.5401 (2014). http://arxiv.org/abs/1410.5401
Parikh, A.P., Täckström, O., Das, D., Uszkoreit, J.: A decomposable attention model for natural language inference. CoRR, abs/1606.01933 (2016). http://arxiv.org/abs/1606.01933
Rocktäschel, T., Grefenstette, E., Hermann, K.M., Kociský, T., Blunsom, P.: Reasoning about entailment with neural attention. CoRR, abs/1509.06664 (2015). http://arxiv.org/abs/1509.06664
Pennington, J., Socher, R., Manning, C.: GloVe: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP, pp. 1532–1543 (2014)
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735
McDermott, K.B., Roediger, H.L.: Memory (Encoding, Storage, Retrieval). Noba Textbook Series: Psychology. DEF Publishers, Champaign (2016). https://doi.org/nobaproject.com
Meng, F., Lu, Z., Li, H., Liu, Q.: Interactive attention for neural machine translation. CoRR, abs/1610.05011 (2016). http://arxiv.org/abs/1610.05011
Wang, M., Lu, Z., Li, H., Liu, Q.: Memory-enhanced decoder for neural machine translation. CoRR, abs/1606.02003 (2016). http://arxiv.org/abs/1606.02003
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. CoRR, abs/1412.6980 (2014). http://arxiv.org/abs/1412.6980
Wang, S., Jiang, J.: Learning natural language inference with LSTM. CoRR, abs/1512.08849 (2015). http://arxiv.org/abs/1512.08849
Liu, P., Qiu, X., Chen, J., Huang, X.: Deep fusion LSTMs for text semantic matching. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, Berlin, Germany, 7–12 August 2016, vol. 1, Long Papers. The Association for Computer Linguistics (2016). http://aclweb.org/anthology/P/P16/P16-1098.pdf
Liu, P., Qiu, X., Huang, X.: Modelling interaction of sentence pair with coupled-LSTMs. CoRR, abs/1605.05573 (2016). http://arxiv.org/abs/1605.05573
Cheng, J., Dong, L., Lapata, M.: Long short-term memory-networks for machine reading. CoRR, abs/1601.06733 (2016). http://arxiv.org/abs/1601.06733
Sha, L., Chang, B., Sui, Z., Li, S.: Reading and thinking: re-read LSTM unit for textual entailment recognition. In: Calzolari, N., Matsumoto, Y., Prasad, R. (eds.) 26th International Conference on Computational Linguistics, COLING 2016. Proceedings of the Conference, Technical Papers, Osaka, Japan, 11–16 December 2016, pp. 2870–2879. ACL (2016). http://aclweb.org/anthology/C/C16/C16-1270.pdf
Weston, J., Chopra, S., Bordes, A.: Memory networks. CoRR, abs/1410.3916 (2014). http://arxiv.org/abs/1410.3916
Sukhbaatar, S., Szlam, A., Weston, J., Fergus, R.: End-to-end memory networks. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) Annual Conference on Neural Information Processing Systems. Advances in Neural Information Processing Systems, Montreal, Quebec, Canada, 7–12 December 2015, vol. 28, pp. 2440–2448 (2015). http://papers.nips.cc/paper/5846-end-to-end-memory-networks
Meng, F., Lu, Z., Tu, Z., Li, H., Liu, Q.: Neural transformation machine: a new architecture for sequence-to-sequence learning. CoRR, abs/1506.06442 (2015). http://arxiv.org/abs/1506.06442
Feng, Y., Zhang, S., Zhang, A., Wang, D., Abel, A.: Memory-augmented neural machine translation. CoRR, abs/1708.02005 (2017). http://arxiv.org/abs/1708.02005
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Liu, L., Huo, H., Liu, X., Palade, V., Peng, D., Chen, Q. (2018). Recognizing Textual Entailment with Attentive Reading and Writing Operations. In: Pei, J., Manolopoulos, Y., Sadiq, S., Li, J. (eds) Database Systems for Advanced Applications. DASFAA 2018. Lecture Notes in Computer Science(), vol 10827. Springer, Cham. https://doi.org/10.1007/978-3-319-91452-7_54
Download citation
DOI: https://doi.org/10.1007/978-3-319-91452-7_54
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-91451-0
Online ISBN: 978-3-319-91452-7
eBook Packages: Computer ScienceComputer Science (R0)