Abstract
Attention mechanism originally introduced for machine translation has a wide application in NLP tasks. By attending more important data with higher wight, the mechanism has the potential to improve neural networks’ performance. Meanwhile, along with the pursuit of higher performance is the challenge of neural networks’ interpretability. Based on quantum probability theory, quantum language model is just such an attempt and has drawn increasing attention. In this paper, we intend to investigate a balance between model’s performance and interpretability, and propose a quantum attention based language model. Density matrix which carries the appearance probability of any word is used to construct quantum attention. Applied in a typical Question Answering task—Answer Selection, our model shows an effective performance on TREC-QA and WIKI-QA datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint, arXiv:1409.0473 (2014)
Galassi, A., Lippi, M., Torroni, P.: Attention in natural language processing. IEEE Trans. Neural Netw. Learn. Syst. 32(10), 4291–4308 (2021)
Yang, Z., Yang, D., Dyer, C., He, X., Smola, A., Hovy, E.: Hierarchical attention networks for document classification. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1480–1489 (2016)
Wang, F., Tax, M.J.D.: Survey on the attention based RNN model and its applications in computer vision. arXiv preprint, arXiv:1601.06823 (2016)
Wang, Y., Huang, M., Zhu, X., Zhao, L.: Attention-based LSTM for aspect-level sentiment classification. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp. 606–615 (2016)
Hermann, K.M., et al.: Teaching machines to read and comprehend. Adv. Neural Inf. Process. Syst. 28, 1693–1701. Curran Associates Inc. (2015)
Cho, K., Courville, A., Bengio, Y.: Describing multimedia content using attention-based encoder- decoder networks. IEEE Trans. Multimed. 17(11), 1875–1886 (2015)
Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. In: Proceeding of ICML, pp. 2048–2057 (2015)
Anderson, P., et al.: Bottom-up and top-down attention for image captioning and visual question answering. In: Proceeding of CVPR, pp. 6077–6086 (2018)
Yu, L., Moritz Hermann, K., Blunsom, P., Pulman, S.: Deep learning for answer sentence selection. arXiv preprint, arXiv:1412.1632 (2014)
Dos Santos, C., Tan, M., Xiang, B., Zhou, B.: Attentive pooling networks. arXiv preprint, arXiv:1602.03609 (2016)
Melucci, M., Van Rijsbergen, K.: Quantum Mechanics and Information Retrieval, pp. 125–155. Springer, Berlin, Heidelberg (2011)
Sordoni, A., Nie, J.-Y.: Looking at vector space and language models for IR using density matrices. In: Atmanspacher, H., Haven, E., Kitto, K., Raine, D. (eds.) QI 2013. LNCS, vol. 8369, pp. 147–159. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-642-54943-4_13
Sordoni, A., Nie, J., Bengio, Y.: Modeling term dependencies with quantum language models for IR. In: Proceeding of Special Interest Group on Information Retrieval, pp. 653–662. ACM (2013)
Levine, Y., Yakira, D., Cohen, N., Shashua, A.: Deep learning and quantum entanglement: Fundamental connections with implications to network design. arXiv:1704.01552 (2017)
Zhang, P., Niu, J., Su, Z., Wang, B., Ma, L., Song, D.: End-to-end quantum-like language models with application to question answering. In: Proceeding of Association for the Advancement of Artificial Intelligence, pp. 5666–5673 (2018)
Li, Q., Wang, B., Melucci, M.: CNM: an interpretable complex-valued network for matching. In: Proceeding of North American Chapter of the Association for Computational Linguistics, pp. 4139–4148 (2019)
Zhao, Q., Hou, C., Liu, C., Zhang, P., Xu, R.: A quantum expectation value based language model with application to question answering. Entropy 22, 533 (2020)
Van Rijsbergen, C.J.: The Geometry of Information Retrieval. Cambridge University Press, Cambridge (2004)
Piwowarski, B., Frommholz, I., Lalmas, M., van Rijsbergen, K.: What can quantum theory bring to information retrieval. In: Proceeding of Conference on Information and Knowledge Management, pp. 59–68 (2010)
Zuccon, G., Azzopardi, L.: Using the quantum probability ranking principle to rank interdependent documents. In: Proceeding of European Conference on Information Retrieval, pp. 357–369 (2010)
Zhao, X., Zhang, P., Song, D., Hou, Y.: A novel reranking approach inspired by quantum measurement. In: Proceeding of European Conference on Information Retrieval, pp. 721–724 (2011)
Zhang, P., et al.: A quantum query expansion approach for session search. Entropy 18(4), 146 (2016)
Sordoni, A., Bengio, Y., Nie, J.: Learning concept embeddings for query expansion by quantum entropy minimization. In: Proceeding of Association for the Advancement of Artificial Intelligence, vol. 14, pp. 1586–1592 (2014)
Li, Q., Li, J., Zhang, P., Song, D.: Modeling multi-query retrieval tasks using density matrix transformation. In: Proceeding of Special Interest Group on Information Retrieval, pp. 871–874. ACM (2015)
Neumann, V.: Mathematical Foundations of Quantum Mechanics. Number 2. Princeton University Press, Princeton (1955)
Nielsen, M.A., Chuang, I.L.: Quantum Computation and Quantum Information. Cambridge University Press, Cambridge, New York (2010)
Gleason, A.M.: Measures on the closed subspaces of a hilbert space. J. Appl. Math. Mech. 885–893 (1957)
Hughes, R.I.: The Structure and Interpretation of Quantum Mechanics. Harvard University Press, Cambridge (1992)
Goddard, C., Wierzbicka, A.: Semantic and Lexical Universals: Theory and Empirical Findings. John Benjamins Publishing, Amsterdam (1994)
Voorhees, E.M., Tice, D.M.: Building a question answering test collection. In: Proceeding of Special Interest Group on Information Retrieval, pp. 200–207 (2000)
Yang, Y., Yih, W., Meek, C.: WikiQA: a Challenge dataset for open-Domain question answering. In: Proceeding of Empirical Methods in Natural Language Processing, pp. 2013–2018. Association for Computational Linguistics (2015)
Severyn, A., Moschitti, A.: Learning to rank short text pairs with convolutional deep neural networks. In: Proceeding of Special Interest Group on Information Retrieval, pp. 373–382. ACM (2015)
Severyn, A., Moschitti, A.: Modeling relational information in question-answer pairs with convolutional neural networks. arXiv:1604.01178 (2016)
He, H., Gimpel, K., Lin, J.: MultiPerspective sentence similarity modeling with convolutional neural networks. In: Proceeding of Empirical Methods in Natural Language Processing, pp. 1576–1586. ACL (2015)
Wang, D., Nyberg, E.: A long short-term memory model for answer sentence selection in question answering. In: Proceeding of Association for Computational Linguistics, pp. 707–712 (2015)
Miao, Y., Yu, L., Blunsom, P.: Neural variational inference for text processing. arXiv:1511.06038 (2015)
Acknowledgments
This work was partially supported by National Natural Science Foundation of China 62006062, 61876053, Shenzhen Foundational Research Funding JCYJ2018 0507183527919, China Postdoctoral Science Foundation 2020M670912.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhao, Q., Hou, C., Xu, R. (2022). Quantum Attention Based Language Model for Answer Selection. In: Pan, Y., Mao, ZH., Luo, L., Zeng, J., Zhang, LJ. (eds) Artificial Intelligence and Mobile Services – AIMS 2021. AIMS 2021. Lecture Notes in Computer Science(), vol 12987. Springer, Cham. https://doi.org/10.1007/978-3-030-96033-9_4
Download citation
DOI: https://doi.org/10.1007/978-3-030-96033-9_4
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-96032-2
Online ISBN: 978-3-030-96033-9
eBook Packages: Computer ScienceComputer Science (R0)