Abstract
An efficient method of representing and retrieving information is an essential component of open domain QA. There are question and answer models that allow for real-time responses with speed benefit and scalability. Nonetheless, due to the limitations of existing phrase models, their accuracy is low. In this paper, we improve the contextualized sparse representation to strengthen the connection between contextual information. We achieve better answer retrieval by enhancing the embedding quality of the model for phrase representation. Specifically, based on original contextualized sparse representations, we transform the single self-attention into collaborative multi-head attention so that attention heads can connect and pay attention to crucial information in different context locations. Compared with learning sparse vectors in n-gram vocabulary space by rectified self-attention, collaborative multi-head attention performs better on the SQuAD dataset. Due to the increased efficiency of critical information representation, the model improves to varying degrees on both of the two evaluation metrics.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Semnani, S.J., Pandey, M.: Revisiting the open-domain question answering pipeline. arXiv preprint arXiv:2009.00914 (2020)
Chen, D., Fisch, A., Weston, J., Bordes, A.: Reading Wikipedia to answer open-domain questions. In: ACL (1) (2017)
Yang, W., Xie, Y., Lin, A., Li, X., Tan, L., Xiong, K., Li, M., Lin, J.: End-to-end open-domain question answering with bertserini. In: NAACL-HLT (Demonstrations) (2019)
Wang, S., et al.: R 3: Reinforced ranker-reader for open-domain question answering. In: Thirty-Second AAAI Conference on Artificial Intelligence (2018)
Das, R., Dhuliawala, S., Zaheer, M., McCallum, A.: Multi-step retriever-reader interaction for scalable open-domain question answering. In: International Conference on Learning Representations (2018)
Jiang, B., Yang, J., Yang, C., Zhou, W., Pang, L., Zhou, X.: Knowledge augmented dialogue generation with divergent facts selection. Knowl. Based Syst. 210, 106, 479 (2020)
Seo, M.J., Kembhavi, A., Farhadi, A., Hajishirzi, H.: Bidirectional attention flow for machine comprehension. In: 5th International Conference on Learning Representations, ICLR (2017)
Fautsch, C., Savoy, J.: Adapting the TF IDF vector-space model to domain specific information retrieval. In: Proceedings of the 2010 ACM Symposium on Applied Computing (SAC), Sierre, Switzerland, March 22–26, 2010, pp. 1708–1712 (2010)
Kadlec, R., Schmid, M., Bajgar, O., Kleindienst, J.: Text understanding with the attention sum reader network. In: ACL (1) (2016)
Seo, M., Kwiatkowski, T., Parikh, A., Farhadi, A., Hajishirzi, H.: Phrase-indexed question answering: a new challenge for scalable document comprehension. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 559–564 (2018)
Seo, M., Lee, J., Kwiatkowski, T., Parikh, A., Farhadi, A., Hajishirzi, H.: Real-time open-domain question answering with dense-sparse phrase index. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4430–4441 (2019)
Lee, J., Seo, M., Hajishirzi, H., Kang, J.: Contextualized sparse representations for real-time open-domain question answering. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 912–919 (2020)
Lin, Y., Ji, H., Liu, Z., Sun, M.: Denoising distantly supervised open-domain question answering. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1736–1745 (2018)
Wang, Z., Ng, P., Ma, X., Nallapati, R., Xiang, B.: Multi-passage BERT: a globally normalized BERT model for open-domain question answering. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5878–5882 (2019)
Kenton, J.D.M.W.C., Toutanova, L.K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL-HLT, pp. 4171–4186 (2019)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: 3rd International Conference on Learning Representations, ICLR (2015)
Voita, E., Serdyukov, P., Sennrich, R., Titov, I.: Context-aware neural machine translation learns anaphora resolution. In: 56th Annual Meeting of the Association for Computational Linguistics, pp. 1264–1274. Association for Computational Linguistics (2018)
Voita, E., Talbot, D., Moiseev, F., Sennrich, R., Titov, I.: Analyzing multi-head self-attention: specialized heads do the heavy lifting, the rest can be pruned. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5797–5808 (2019)
Michel, P., Levy, O., Neubig, G.: Are sixteen heads really better than one? In: Advances In Neural Information Processing Systems, vol. 32 (2019)
Cordonnier, J.B., Loukas, A., Jaggi, M.: Multi-head attention: Collaborate instead of concatenate. arXiv preprint arXiv:2006.16362 (2020)
Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: Squad: 100, 000+ questions for machine comprehension of text. In: EMNLP (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Sun, M., Jiang, B., Zhou, X., Zhang, B., Yang, C. (2023). Collaborative Multi-head Contextualized Sparse Representations for Real-Time Open-Domain Question Answering. In: Sun, Y., et al. Computer Supported Cooperative Work and Social Computing. ChineseCSCW 2022. Communications in Computer and Information Science, vol 1682. Springer, Singapore. https://doi.org/10.1007/978-981-99-2385-4_32
Download citation
DOI: https://doi.org/10.1007/978-981-99-2385-4_32
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-2384-7
Online ISBN: 978-981-99-2385-4
eBook Packages: Computer ScienceComputer Science (R0)
