Efficient Answer-Annotation for Frequent Questions

Abstract

Ground truth is a crucial resource for the creation of effective question-answering (Q-A) systems. When no appropriate ground truth is available, as it is often the case in domain-specific Q-A systems (e.g. customer-support, tourism) or in languages other than English, new ground truth can be created by human annotation. The annotation process in which a human annotator looks up the corresponding answer label for each question from an answer catalog ($\textsc {Sequential}$ approach), however, is usually time-consuming and costly. In this paper, we propose a new approach, in which the annotator first manually groups questions that have the same intent as a candidate question, and then, labels the entire group in one step ($\textsc {Group}\text {-}\textsc {Wise}$ approach). To retrieve same-intent questions effectively, we evaluate various unsupervised semantic similarity methods from recent literature, and implement the most effective one in our annotation approach. Afterwards, we compare the $\textsc {Group}\text {-}\textsc {Wise}$ approach with the $\textsc {Sequential}$ approach with respect to answer look-ups, annotation time, and label-quality. We show based on 500 German customer-support questions that the $\textsc {Group}\text {-}\textsc {Wise}$ approach requires 51% fewer answer look-ups, is 41% more time-efficient, and retains the same label-quality as the $\textsc {Sequential}$ approach. Note that the described approach is limited to Q-A systems where frequently asked questions occur.