A novel multi-domain machine reading comprehension model with domain interference mitigation

Abstract

Machine reading comprehension (MRC), as an important task in natural language processing (NLP), is to automatically answer the question after reading a passage. In this aspect, dominant studies mainly focus on domain-specific models. However, domain-specific models trained only on single domain data often cannot achieve satisfactory performance. Although using data of other domains can bring improvement to some extent, building MRC models specific to each domain also makes deployment more difficult in practice. In this paper, we propose a multi-domain MRC model based on knowledge distillation (KD) with domain interference mitigation. Specifically, we employ KD to train a joint model by simultaneously using the multi-domain data and the output distributions of all domain-specific models. In this way, our joint model can better exploit multi-domain data while enabling simpler deployment at the same time. Moreover, to deal with the gradient conflict caused by using data of different domains, we resort to measuring domain-level gradient similarity, based on which an improved PCGrad (short for projecting conflicting gradients) algorithm with adaptive learning rate is proposed. The algorithm mitigates domain interference to improve our joint model across domains. Experimental results and in-depth analysis demonstrate the effectiveness of our joint model and mitigating domain interference further improves the overall performance of our model on a set of benchmark datasets.

Introduction

As one important task of natural language processing (NLP), machine reading comprehension (MRC) aims to answer questions given a passage. It has been employed in many real industrial applications such as search engines and become a research hotspot in NLP field [1], [2], [3], [4], [5], [6], [7], [8]. Nowadays, dominant MRC models are almost either extractive or generative ones, where the former attract more attention due to its advantages of simplicity and computing efficiency.

In recent years, as the basis of MRC studies, many datasets of different domains have been constructed, e.g., SQuAD [9] collected from WikiPedia, NewsQA [10] for news passages. The data of different domains usually vary greatly. In order to handle datasets of various domains, most existing approaches resort to building domain-specific models [11], [12]. However, the training data of a single domain are often limited, and thus they are insufficient to solely train a domain-specific model with satisfying performance.

To address this issue, previous study explores data augmentation to enhance domain-specific models [13]. Meanwhile, more researchers are committed to employing transfer learning to exploit multi-domain data for domain-specific models [14], [15], [16], [17]. In spite of their success to some extent, building domain-specific models makes deployment more difficult, especially when there are a large number of domains involved. There is also an exception that Xu et al. [18] explores a multi-task framework to train a unified model using the sample re-weighting technique. It is notable that such an approach only focuses on the data-level knowledge transfer, ignoring the beneficial knowledge of domain-specific models.

In this paper, inspired by the successful applications of knowledge distillation (KD) [19] in multilingual neural machine translation [20] and multi-task learning [21], we propose a KD-based multi-domain MRC model with domain interference mitigation. Specifically, we first individually train domain-specific MRC models on their own training data. Then using the mixed-domain data, we use KD to transfer the knowledge of domain-specific models to the joint model. During this process, the joint model is simultaneously supervised by the signal of training data and the output distributions of domain-specific models. More importantly, since the training samples from different domains contribute to the parameter update of the joint model differently and result in conflicting gradients, we introduce the PCGrad algorithm [22], formerly used in multi-task learning scenarios to modulate gradients from different domains. However, in the original PCGrad algorithm, the updating step size of model paramaters is independent of the domain-level gradient similarities, which leads to negative effect on model optimization. Intuitively, if there exists a great conflict between the gradients in different domains, the excessive updating step size will have a negative impact on the model performance for some domains. Therefore, the smaller the gradient similarity is, the smaller the updating step size should be. On the contrary, the larger the gradient similarity is, the larger the updating step size can be. To model this intuition, we propose to measure the gradient similarity between domains in terms of two factors: gradient direction and gradient magnitude. Then, based on the overall gradient similarity, we refine the conventional PCGrad algorithm by adaptively adjusting the learning rate to optimize model parameters. Compared to [18] which only transfers knowledge at the data level, our joint model leverages the model-level knowledge transfer for MRC. Moreover, mitigating domain interference with our improved PCGrad algorithm further enhances the performance of our joint model.

The contribution of our work lies in the following points:

• We introduce knowledge distillation to construct a multi-domain model for MRC. To the best of our knowledge, our work is the first attempt to explore knowledge transfer at both data and model levels in MRC.

• We propose to refine the conventional PCGrad algorithm by adaptively adjusting its learning rate according to our designed gradient similarity metric, which mainly depends on the similarities in gradient direction and magnitude.

• Experimental results show that our multi-domain model surpass several commonly-used baselines, including domain-specific models. In particular, further analysis demonstrates the effectiveness of our improved PCGrad algorithm.