A novel word embedding learning model using the dissociation between nouns and verbs

Abstract

In recent years, there have been researches on using semantic knowledge and global statistical features to guide the learning of word embeddings. Though the syntax knowledge also plays an very important role in natural language understanding, its effectiveness on the word embedding learning is still far from well investigated. Inspired by the principle of the dissociation between nouns and verbs (DNV) in language acquisition observed in neuropsychology, we propose a novel model for word embeddings learning using DNV (named Continuous Dissociation between Nouns and Verbs Model, CDNV). CDNV uses a three-layer feed forward neural network to integrate DNV generated by auto-tagged noun/verb information into the word embeddings learning process, which can still preserve the word order of local context. The advantage of the CDNV lies in that it is able to learn high-quality word embeddings with relatively low time complexity. Experimental results show that: (1) CDNV takes about 1.5 h to learn word embeddings on a corpus of billions of words, which is comparable with CBOW and Skip-gram and more efficient than other models; (2) the nearest neighbors of some representative words derived from the word embeddings learnt by CDNV are more reasonable than other word embeddings; (3) the performance improvement on F1 measure from CDNV word embeddings is greater than other word embeddings on NER and Chunking.

Introduction

In most natural language processing (NLP) models, words are usually mapped to symbolic IDs and then are transformed to discrete 0/1 binary vectors using a one-hot representation. The feature vectors are hence of the same length as the size of the vocabulary. This representation often suffers from the curse of dimensionality [1] and data sparsity. To overcome these problems, researchers have proposed the distributed representation for words, which essentially represent words using low dimensional, continuous and dense vectors, called word embeddings, and have showed stable improvements on a variety of tasks such as paraphrase detection [2], sentiment analysis [3] and machine translation [4]. Word embeddings are usually learnt from a large-scale unlabeled corpus by a multi-layer neural network. Studies on word embeddings learning mainly focus on one of the following two aspects: (1) reducing the time complexity of the learning process, and (2) improving the quality of word embeddings. Two types of approaches were used to reduce the time complexity, i.e., decomposing output nodes and simplifying the structure of the neural network. To improve the quality of word embeddings, human knowledge or global statistical information have been introduced into the process of learning, unfortunately most of which have a relatively high time complexity. The goal of this paper is therefore to seek an efficient model to learn word embeddings of high quality.

More specifically, we want to answer the following question: can grammatical information (e.g. part-of-speech, POS), a basic component of a language [5], be directly used in models to learn word embeddings of high quality? Experiments from cognitive linguistics have shown that English-speaking children take the POS of a new word as a clue to understand it [6]. In particular, nouns and verbs play an important role in language understanding. For example, the word “love” can be either a noun or a verb. Children learn the noun “love” (“love/NN”) in a different way from the verb “love” (“love/VB”), since the meanings and usages of “love/NN” and “love/VB” are different. Experiments from neuropsychology have shown that different parts of the brain are activated when a person learns a noun and a verb from analyzing brain images taken by functional magnetic resonance [7], [8]. This phenomenon is known as the dissociation between nouns and verbs (DNV).

In this paper, we attempt to design an efficient learning framework of word embeddings to take full advantage of the DNV characteristic of language acquisition for improving the quality of word embeddings. To ensure a low time complexity, we adopt a three-layer feed forward neural network like the continuous bag-of-words model (CBOW) and the continuous skip-gram model (Skip-gram) [9] proposed by Mikolov et al. recently. For high-quality word embeddings, we preserve the word order of local context and decompose the output nodes of the neural network according to DNV. The proposed models for learning word embeddings with DNV are named Continuous Dissociation between Nouns and Verbs Models (CDNV). For evaluation, we use it to learn word embeddings on a corpus of billions of words and compare them with other word embeddings in the following two aspects: (1) qualitative analysis: check whether the nearest neighbors of some representative words derived from the word embeddings are reasonable, (2) quantitative analysis: compare the improvement gains from different word embeddings on two traditional NLP tasks, i.e. named entity recognition (NER) and Chunking. The word embeddings used for comparison include those learnt on the same corpus by CBOW and Skip-gram, and other public word embeddings learnt on the similar corpora by other models. Experimental results show that: (1) CDNV models is very efficient (taking about 1.5 h), (2) the nearest neighbors of some representative words derived from the word embeddings learnt by CDNV are more reasonable than other word embeddings, and (3) on both NER and Chunking tasks, the improvement from the word embeddings learnt by CDNV is significantly higher than other word embeddings.