Visualization-based improvement of neural machine translation

Highlights

• A visual-interactive approach for analyzing, understanding, and correcting NMT.

• Visualizations show model-specific information and metrics for translation quality.

• Our approach supports both LSTM-based NMT models and the Transformer architecture.

• The quantitative computer-based evaluation reveals reduced post-editing efforts.

Abstract

We introduce a novel visual-interactive approach for analyzing, understanding, and correcting neural machine translation. Our system supports users in automatically translating documents using neural machine translation and identifying and correcting possible erroneous translations. User corrections can then be used to fine-tune the neural machine translation model and automatically improve the whole document. While translation results of neural machine translation can be impressive, there are still many challenges such as over- and under-translation, domain-specific terminology, and handling long sentences, making it necessary for users to verify translation results. Our system aims at supporting users in this task. Our visual analytics approach combines several visualization techniques in an interactive system. A parallel coordinates plot with multiple metrics related to translation quality can be used to find, filter, and select translations that might contain errors. An interactive beam search visualization and graph- or matrix-based visualizations for attention weights can be used for post-editing and understanding machine-generated translations. The machine translation model is updated from user corrections to improve the translation quality of the whole document. We designed our approach for an LSTM-based translation model and extended it to also include the Transformer architecture. We show for representative examples possible mistranslations and how to use our system to deal with them. A user study revealed that many participants favor such a system over manual text-based translation, especially for translating large documents. Furthermore, we performed quantitative computer-based experiments that show that our system can be used to improve translation quality and reduce post-editing efforts for domain-specific documents.

Introduction

Machine learning and especially deep learning are popular and rapidly growing fields in many research areas. The results created with machine learning models are often impressive but sometimes still problematic. Currently, much research is performed to better understand, explain, and interact with these models. In this context, visualization and visual analytics methods are suitable and more and more often used to explore different aspects of these models. Available techniques for visual analytics in deep learning were examined by Hohman et al. [1]. While there is a large amount of work available for explainability in computer vision, less work exists for machine translation.

As it becomes increasingly important to communicate in different languages, and since information should be available for a huge range of people from different countries, many texts have to be translated. Doing this manually takes much effort. Nowadays, online translation systems like Google Translate [2] or DeepL [3] support humans in translating texts. However, the translations generated that way are often not as expected or like someone familiar with both languages might translate them. It may also not express someone’s translation style or use the correct terminology of a specific domain or for some occasion. Often, more background knowledge about the text is required to translate documents appropriately.

With the introduction of deep learning methods, the translation quality of machine translation models has improved considerably in the last years. However, there are still difficulties that need to be addressed. Common problems of neural machine translation (NMT) models are, for instance, over- and under-translation [4] when words are translated repeatedly or not at all. Handling rare words [5], which might be available in specific documents, and long sentences, are also issues. Domain adaption [5] is another challenge. Especially documents from specific domains such as medicine, law, or science require high-quality translations [6]. As many NMT models are trained on general data sets, their translation performance is worse for domain-specific texts.

If high-quality translations for large texts are required, it is insufficient to use machine translation models alone. These models are computationally efficient and able to translate large documents with low time effort, but they may create erroneous or inappropriate translations. Humans are very slow compared to these models, but they can detect and correct mistranslations when familiar with the languages and the domain terminology. In a visual analytics system, both of these capabilities can be combined. Such a system should provide the translations from an NMT model and possibilities for users to visually explore translation results to find mistranslated sentences, correct them, and steer the machine learning model.

We have developed a visual analytics approach to reach the goals outlined above. First, our system performs automatic translation of a whole, possibly large, document and shows the result in the Document View (Fig. 1). Users can then explore and modify the document on different views [7] (Fig. 2) to improve translations and use these corrections to fine-tune the NMT model. We support different NMT architectures and use both an LSTM-based and a Transformer architecture.

So far, visual analytics systems for deep learning were mostly available for computer vision, some text-related areas, focusing on smaller parts of machine translation [8], [9] or intended for domain experts to gain insight into the models or to debug them [10], [11]. This work contributes to visualization research by introducing the application domain of NMT using a user-oriented visual analytics approach. In our system, we employ different visualization techniques adapted for usage with NMT. Our parallel coordinates plot (Fig. 1(B)) supports the visualization of different metrics related to text quality. The interaction techniques in our graph- and matrix-based visualizations for attention (Fig. 2(B) and (C)) and tree-based visualization for beam search (Fig. 2(D)) are specifically designed for text exploration and modification. They have a strong coupling to the underlying model. Furthermore, our system has a fast feedback loop and allows interaction in real-time. We demonstrate our system’s features in a video and provide the source code¹ [12] for our system. The trained models [13] we used in our case study and evaluation are also publicly available.

This paper is an extended version of our previous work [14]. We improved our visual analytics approach by adding a second interactive attention-based visualization for sentences in the form of a matrix (Fig. 2(C)) that supports subword units. In this context, we discuss the differences between our attention visualizations and the circumstances in which one variant might be preferable. Additionally, we allow users to specify parameters in the user interface for a better analysis of translation results. This is especially useful for more experienced users who want to explore more details about the Transformer architecture or about the attention weights in both architectures we implemented. We provide more information about our implemented machine translation models and explore a different document in our case study. Finally, we include a quantitative computer-based evaluation, demonstrating both the utility of our metrics for detecting mistranslated sentences, as well as how fine-tuning LSTM-based models on domain-specific documents increases the in-domain translation quality.