Abstract
Recent studies on Question Answering (QA) and Conversational QA (ConvQA) emphasize the role of retrieval: a system first retrieves evidence from a large collection and then extracts answers. This open-retrieval ConvQA setting typically assumes that each question is answerable by a single span of text within a particular passage (a span answer). The supervision signal is thus derived from whether or not the system can recover an exact match of this ground-truth answer span from the retrieved passages. This method is referred to as span-match weak supervision. However, information-seeking conversations are challenging for this span-match method since long answers, especially freeform answers, are not necessarily strict spans of any passage. Therefore, we introduce a learned weak supervision approach that can identify a paraphrased span of the known answer in a passage. Our experiments on QuAC and CoQA datasets show that the span-match weak supervisor can only handle conversations with span answers, and has less satisfactory results for freeform answers generated by people. Our method is more flexible as it can handle both span answers and freeform answers. Moreover, our method can be more powerful when combined with the span-match method which shows it is complementary to the span-match method. We also conduct in-depth analyses to show more insights on open-retrieval ConvQA under a weak supervision setting.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
References
Ahmad, A., Constant, N., Yang, Y., Cer, D.M.: ReQA: An Evaluation for End-to-End Answer Retrieval Models. ArXiv (2019)
Chen, D., Fisch, A., Weston, J., Bordes, A.: Reading wikipedia to answer open-domain uestions. In: ACL (2017)
Chen, L., Tang, Z., Yang, G.: Balancing reinforcement learning training experiences in interactive information retrieval. In: SIGIR (2020)
Chen, Y., Wu, L., Zaki, M.J.: GraphFlow: Exploiting Conversation Flow with Graph Neural Networks for Conversational Machine Comprehension. ArXiv (2019)
Choi, E., et al.: QuAC: question answering in context. In: EMNLP (2018)
Clark, C., Gardner, M.: Simple and effective multi-paragraph reading comprehension. In: ACL (2017)
Cohen, D., Yang, L., Croft, W.B.: WikiPassageQA: a benchmark collection for research on non-factoid answer passage retrieval. In: SIGIR (2018)
Das, R., Dhuliawala, S., Zaheer, M., McCallum, A.: Multi-step retriever-reader interaction for scalable open-domain question answering. In: ICLR (2019)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT (2019)
Dhingra, B., Mazaitis, K., Cohen, W.W.: Quasar: Datasets for Question Answering by Search and Reading. ArXiv (2017)
Dunn, M., Sagun, L., Higgins, M., Güney, V.U., Cirik, V., Cho, K.: SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine. ArXiv (2017)
Elgohary, A., Peskov, D., Boyd-Graber, J.L.: Can You Unpack That?. EMNLP/IJCNLP, Learning to Rewrite Questions-in-Context. In (2019)
Garg, S., Vu, T., Moschitti, A.: TANDA: transfer and adapt pre-trained transformer models for answer sentence selection. In: AAAI (2020)
Htut, P.M., Bowman, S.R., Cho, K.: Training a ranking function for open-domain question answering. In: NAACL-HLT (2018)
Huang, H.Y., Choi, E., tau Yih, W.: Flowqa: grasping flow in history for conversational machine comprehension. ArXiv (2018)
Joshi, M., Choi, E., Weld, D.S., Zettlemoyer, L.: TriviaQA: a large scale distantly supervised challenge dataset for reading comprehension. In: ACL (2017)
Karpukhin, V., et al.: Dense passage retrieval for open-domain question answering. In: EMNLP (2020)
Kratzwald, B., Feuerriegel, S.: Adaptive document retrieval for deep question answering. In: EMNLP (2018)
Krishna, K., Wieting, J., Iyyer, M.: Reformulating unsupervised style transfer as paraphrase generation. In: EMNLP (2020)
Kwiatkowski, T., et al.: Natural questions: a benchmark for question answering research. TACL 7, 453–466 (2019)
Lan, Z.Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.: ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. ArXiv (2019)
Lee, J., Yun, S., Kim, H., Ko, M., Kang, J.: Ranking paragraphs for improving answer recall in open-domain question answering. In: EMNLP (2018)
Lee, K., Chang, M.W., Toutanova, K.: Latent retrieval for weakly supervised open domain question answering. In: ACL (2019)
Li, J., Qiu, L., Tang, B., Chen, D., Zhao, D., Yan, R.: Insufficient Data Can Also Rock!. AAAI, Learning to Converse Using Smaller Data with Augmentation. In (2019)
Lowe, R., Pow, N., Serban, I., Pineau, J.: The ubuntu dialogue corpus: a large dataset for research in unstructured multi-turn dialogue systems. In: SIGDIAL (2015)
Luo, J., Dong, X., Yang, G.: Learning to reinforce search effectiveness. In: ICTIR (2015)
Luo, J., Zhang, S., Yang, G.: Win-win search: dual-agent stochastic game in session search. In: SIGIR (2014)
Nguyen, T., et al.: MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. ArXiv (2016)
Qiu, M., et al.: Reinforced history backtracking for conversational question answering. In: AAAI (2021)
Qu, C., Yang, L., Chen, C., Qiu, M., Croft, W.B., Iyyer, M.: Open-retrieval conversational question answering. In: SIGIR (2020)
Qu, C., et al.: Attentive history selection for conversational question answering. In: CIKM (2019)
Qu, C., Yang, L., Qiu, M., Croft, W.B., Zhang, Y., Iyyer, M.: BERT with history answer embedding for conversational question answering. In: SIGIR (2019)
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. OpenAI blog 1(8), 9 (2019)
Rajpurkar, P., Jia, R., Liang, P.: Know what you don’t know: unanswerable questions for SQuAD. In: ACL (2018)
Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: SQuAD: 100, 000+ questions for machine comprehension of text. In: EMNLP (2016)
Reddy, S., Chen, D., Manning, C.D.: CoQA: a conversational question answering challenge. TACL 7, 249–266 (2018)
Shrivastava, A., Li, P.: Asymmetric LSH (ALSH) for sublinear time maximum inner product search (MIPS). In: NIPS (2014)
Tao, C., Wu, W., Xu, C., Hu, W., Zhao, D., Yan, R.: Multi-representation fusion network for multi-turn response selection in retrieval-based chatbots. In: WSDM (2019)
Trischler, A., et al.: NewsQA: a machine comprehension dataset. In: Rep4NLP@ACL (2016)
Turc, I., Chang, M.W., Lee, K., Toutanova, K.: Well-Read Students Learn Better: On the Importance of Pre-training Compact Models. ArXiv (2019)
Vaswani, A., et al.: Attention is all you need. In: NIPS (2017)
Voorhees, E.M., Tice, D.M.: The TREC-8 question answering track evaluation. In: TREC (1999)
Wang, M., Smith, N.A., Mitamura, T.: What is the Jeopardy Model?. EMNLP-CoNLL, A Quasi-Synchronous Grammar for QA. In (2007)
Wang, S., et al.: R3: Reinforced ranker-reader for open-domain question answering. In: AAAI (2018)
Wieting, J., Gimpel, K.: ParaNMT-50M: pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In: ACL (2018)
Wolf, T., Sanh, V., Chaumond, J., Delangue, C.: Transfertransfo: a transfer learning approach for neural network based conversational agents. In: NeurIPS CAI Workshop (2018)
Wu, Y., Wu, W.Y., Zhou, M., Li, Z.: Sequential match network: a new architecture for multi-turn response selection in retrieval-based chatbots. In: ACL (2016)
Yan, R., Song, Y., Wu, H.: Learning to respond with deep neural networks for retrieval-based human-computer conversation system. In: SIGIR (2016)
Yan, R., Song, Y., Zhou, X., Wu, H.: “Shall i be your chat companion?": towards an online human-computer conversation system. In: CIKM (2016)
Yang, L., et al.: Response ranking with deep matching networks and external knowledge in information-seeking conversation systems. In: SIGIR (2018)
Yang, L., et al.: A hybrid retrieval-generation neural conversation model. In: CIKM (2019)
Yang, L., et al.: IART: intent-aware response ranking with transformers in information-seeking conversation systems. In: WWW (2020)
Yang, W., et al.: End-to-end open-domain question answering with BERTserini. In: NAACL-HLT (2019)
Yang, Y., Yih, W.T., Meek, C.: WikiQA: a challenge dataset for open-domain question answering. In: EMNLP (2015)
Yeh, Y.T., Chen, Y.N.: FlowDelta: Modeling Flow Information Gain in Reasoning for Conversational Machine Comprehension. ArXiv (2019)
Zhou, J., Agichtein, E.: RLIRank: learning to rank with reinforcement learning for dynamic search. In: WWW (2020)
Zhu, C., Zeng, M., Huang, X.: SDNet: Contextualized Attention-based Deep Network for Conversational Question Answering. ArXiv (2018)
Acknowledgments
This work was supported in part by the Center for Intelligent Information Retrieval and in part by NSF IIS-1715095. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect those of the sponsor. The authors would like to thank Minghui Qiu for his constructive comments on this work.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Qu, C., Yang, L., Chen, C., Croft, W.B., Krishna, K., Iyyer, M. (2021). Weakly-Supervised Open-Retrieval Conversational Question Answering. In: Hiemstra, D., Moens, MF., Mothe, J., Perego, R., Potthast, M., Sebastiani, F. (eds) Advances in Information Retrieval. ECIR 2021. Lecture Notes in Computer Science(), vol 12656. Springer, Cham. https://doi.org/10.1007/978-3-030-72113-8_35
Download citation
DOI: https://doi.org/10.1007/978-3-030-72113-8_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-72112-1
Online ISBN: 978-3-030-72113-8
eBook Packages: Computer ScienceComputer Science (R0)