Visual exploration and comparison of word embeddings

Abstract

Word embeddings are distributed representations for natural language words, and have been wildly used in many natural language processing tasks. The word embedding space contains local clusters with semantically similar words and meaningful directions, such as the analogy. However, there are different training algorithms and text corpora, which both have a different impact on the generated word embeddings. In this paper, we propose a visual analytics system to visually explore and compare word embeddings trained by different algorithms and corpora. The word embedding spaces are compared from three aspects, i.e., local clusters, semantic directions and diachronic changes, to understand the similarity and differences between word embeddings.

Introduction

The word embedding is a kind of mathematical representation of vocabulary. Usually, there are two kinds of representations: one-hot vector representation and distributed representation. One-hot vector representation easily comes to our minds, which uses the index in the dictionary to represent word uniquely. However, this representation method only separates word and does not express the semantic meanings of the word. Distributed representation is a vector of real numbers, originally proposed by Hinton [1] in 1986, and it can encode semantic information compared to the one-hot vector representation. Euclidean distance or cosine similarity between distributed vectors can be used to measure the semantic similarity of words. Due to these advantages, distributed representation is widely used in many natural language processing tasks.

In order to generate the distributed representation of words, Xu and Rudnicky [2] applied neural networks to train word embeddings in 2000. The classic training method is to build a model by a three-layer neural network proposed by Bengio et al. [3], and many subsequent algorithms are based on this work. Until 2013, Mikolov et al. [4] proposed CBOW and Skip-gram methods. Google made it as an open source project named word2vec based on these methods and made word embeddings widely accepted by users. Thus, we use word2vec to train our word embeddings in this paper. Since the distributed representation captures semantic meanings of words, it is very popular in natural language processing tasks and can significantly improve performance in downstream tasks, such as text classification [5], sentiment analysis [6], [7], and semantic analysis [8]. For linguists, word embeddings not only can help them understand the structure of language from a macroscopic perspective, but also analyze the usage and meaning of words in a fine-grained manner.

The original word embedding data is a kind of vector data of tens of dimensions or even hundreds of dimensions. The high-dimensional data is difficult to interpret. It is complicated to show the spatial information in the high-dimensional space. Although we know that semantically similar words would be together, it is still hard to imagine how these words will be distributed in the high-dimensional space. We need to employ visualization techniques to enhance our understanding of the word embedding space. The word embedding space contains semantic information, such as synonym information, and it can be interpreted by the nearest neighbor words or clustering results. It also contains other semantic information, such as the word analogy relationship, and such information requires different visualization techniques to reveal its underlying relationships and structure.

Different corpora, parameters settings and training methods may lead to different word embedding spaces, and they are difficult to evaluate mathematically. The data, what we used to train the word embeddings, is a kind of large text and usually in Gigabytes. In our paper, we use CBOW and Skip-gram method, which are totally different in model architecture, which will discuss in Section 3. Therefore, we hope to design a visual analytics system to explore and compare the similarity and differences between two word embedding spaces trained from different corpora or methods. The contributions of this paper are:

• We propose an interactive visual analytics system to understand and compare the word embedding spaces, in order to obtain an intuitive understanding between the spaces.

• A case study demonstrating insights in word embeddings with different training algorithms and corpora, which reveals some interesting results like latent semantic changes in words.