Interpretable duplicate question detection models based on attention mechanism

Abstract

Recently, there exist growing concerns about the interpretability of deep learning models. While few of these models have been applied to a duplicate question detection task, which aims at finding semantically equivalent question pairs of question answering forum. In this paper, based on an attention mechanism, we propose two modularized interpretable deep neural network models for such tasks. During the word precessing procedure, a filter operation is employed to enhance the relevant information contained in the pre-trained word embeddings. Regarding the word matching and sentence representation process, vanilla attention and structured attention mechanisms are utilized, respectively. Benefiting from the interpretability of attention techniques, our models can illustrate how the words match between sentence pairs and what aspects of the sentences are extracted to have an effect on the final decision. The attention visualization furnishes us with detailed representation at word and sentence level. And experimental results show that our models are comparable with other reported models.

Introduction

With the development of question answering community like Quora,¹ Reddit² and Stack Overflow,³ more and more individuals tend to post and answer questions in such forums. However, more and more newly posted questions cannot be answered quickly for the reason that fews have a proper recommended answer due to the lack of popularity. In practice, many newly posted questions have the similar or even the same answers as the stored ones’. Specifically, many questions, posted in different formulations, are of semantic equivalence, thus similar answers must be well matched to the corresponding ones. Therefore, to answer a newly posted question as quickly as possible, we can directly take advantage of the answers to the duplicate or nearly duplicate questions stored in the forums, which can bring a great convenience to question answering forum.

According to [3], two questions are regarded as duplicate question pair if they share common answers. This task is similar to the text similarity one. While the text similarity task has graded outputs indicating how similar they are, the output of duplicate question detection is yes$/$No. Thus this task can be viewed as a subtask of a text similarity identification that has a binary output and only contains the interrogative text.

For the text similarity task, many recently proposed deep learning models have achieved the state-of-the-art results [8], [26], [24], [9]. However, the existing methods have such limitations: (1) In many models, the final matching score is calculated through two individual sentence vectors. Considering each sentence pair as a single input, we argue that putting two sentences together to capture the integral information must be a more natural method. (2) For most of the sentence pair modeling models, the generated sentence vectors only contain single-aspect information. Nevertheless, multi-aspect information can be extracted from one sentence focusing on different part of it, which could definitely enrich the sentence representation. (3) Recently, the popularity of model interpretability is increasing [15], [4], which is very helpful for improving the quality of the online question answering forums. But many existing models proposed in [26], [3], [1] have no ability to tell us which words or phrases of two sentences contribute more to the final decision.

To address the aforementioned limitations, in this paper, two interpretable matching models based on attention mechanism are proposed to implement sentence pairs. Our models first match words of one sentence with words of the other sentence, then perform sentence level interaction to generate similarity vectors (see Fig. 1). The final decision will be made based on the vectors. Main contributions of this work are summarized as follows: (1) A filter operation is proposed to eliminate the redundant information of pre-trained word embeddings. (2) Based on structured attention [14], two models are proposed to enhance the sentence level information. (3) The proposed models are interpretable deep models, which will provide the semantic matching process illustrated.

The rest of the paper is organized as follows. Section 2 introduces the related work. Section 3 illustrates and details the proposed model architecture. Section 4 and Section 5 show the datasets and the experimental results. Final conclusion and future work are described in Section 6.