Enhancing Complex Question Answering via LLM Pseudo-Document and Adaptive Retrieval

Abstract

Multi-hop complex question answering (QA) involves answering questions that require reasoning over multiple pieces of information. Despite the excellent performance of the combination of Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) techniques in many natural language understanding tasks, they still encounter challenges when tackling multi-hop complex question answering tasks. The current approach breaks down complex questions into several sub-questions and then uses the RAG framework to retrieve k documents based on semantic similarity to prompt LLMs to answer. However, this method has several limitations. Firstly, LLMs tend to excessively decompose the original multi-hop complex question while neglecting the background, which leads to an inability to focus on information that is crucial for answering the question. Secondly, current RAG often utilize a static top-k retrieval strategy, which does not adequately accommodate the varying informational demands of different questions. On one hand, insufficient retrieved documents may lead to a scarcity of essential knowledge. On the other hand, an excessive number of retrieved documents can introduce significant noise, complicating the information extraction process. To overcome these challenges, this paper introduces a novel solution. We first generate pseudo-documents for the original complex questions and integrate them with the original question to enhance question decomposition. This method resolves the problem of LLMs overly decomposing questions without consideration of the background. Additionally, to mitigate the impact of static parameters during the retrieval process, we incorporate an adaptive retrieval strategy that provides real-time relevance assessment for retrieved documents, ensuring that key information is not overlooked when solving complex multi-hop questions. Through extensive comparative experiments and ablation studies, our method has demonstrated significant effectiveness and achieved state-of-the-art (SOTA) performance in the zero-shot setting for multi-hop complex QA.