Generating summary sentences using Adversarially Regularized Autoencoders with conditional context

Highlights

• We proposed a novel summary method by using code vector of autoencoder.

• Unsupervised abstractive summarization using Adversarially regularized autoencoder.

• Conditional model is proposed and enhanced performance for review summarization.

Abstract

Abstractive summarization is challenging problem, especially abstractive summarization based on unsupervised learning because it must generate whole, unique sentences. In the real world, companies use abstractive summarization to understand customer feedbacks. In many cases, this work is done by humans and so is expensive in terms of time and money. Therefore, there is an increasing demand for machine learning-based abstractive summarization systems. However, most previous abstractive summarization studies were of supervised models. In this paper, we proposed novel abstractive summarization methods that can be trained unsupervisedly. One of the proposed methods is based on Adversarially Regularized Autoencoder(ARAE) model, but abstractive summary generation method for each cluster of similar customers’ reviews, is newly proposed. We further proposed Conditional Adversarially Regularized Autoencoder(CARAE) model which is similar to the ARAE model but with the addition of condition nodes so that additional information about the cluster can be used during summarization. We first performed summary experiments based on Korean and additionally performed experiments on English. In the experiments, we set up some comparison models and used ROUGE and BLEU to evaluate our proposed models’ performance. Overall, our proposed models outperformed the comparison models and CARAE model performed better than the ARAE model.

Introduction

Text summarization is a technique for generating a compressed version of an original document. Manual text summarization is expensive, so there has been an increasing demand for automatic summarization technology.

When making decisions in business, companies usually summarizes Voice of Customer (VOC) data (e.g., review data) to improve their products to reflect customer feedback. Companies usually categorize customer response documents by topic and extract their key content by summarization (Fig. 1). In most cases, this task is usually done manually by the experts, making it expensive in terms of time and money. Furthermore, as amount of VOC data is growing faster, automatic summarization techniques become increasingly important more than ever before. In response to this trend, we proposed an efficient, unsupervised summarization method for each topic automatically.

Text summarization can be categorized in two distinct ways: extractive and abstractive. Extractive summarization extracts the important parts of a document in place of whole sentences. Abstractive summarization compresses the important content of the document and generates new sentences based on that content. Abstractive summarization is difficult, so most text summarization has been done by extractive summarization methods, including keyword extraction, similarity detection, and rule-based methods. However, recent successful work is sequence to sequence model (Sutskever, Vinyals, & Le, 2014), have shown that abstractive summarization is possible by reading and freely generating text using recursive neural networks (RNN) (Chopra, Auli, Rush, 2016, Nallapati, Zhou, dos Santos, Gulcehre, Xiang, 2016, Rush, Chopra, Weston, 2015, See, Liu, Manning, 2017, Zeng, Luo, Fidler, & Urtasun). Most of these studies used target sentences as summary sentences for learning. However, the method proposed in this study used an unsupervised abstractive summarization method that did not require the use of target sentences for learning.

Our work is a multi-document summarization and aims to summarize various review data. We summarized review data clustered by each topic and conducted text summarization on review data from beer review data which is in Korean and Opinosis data set (Ganesan, Zhai, & Han, 2010) which is in English.

The proposed abstractive summarization method in this study generated summary sentences using a code vector of autoencoder. Several studies have used autoencoders to extract summaries (Yousefi-Azar, Hamey, 2017, Zhong, Liu, Li, Long, 2015), but our method was different from those previous methods because it used an autoencoder for abstractive summarization.

We used code vectors of Adversarially Regularized Autoencoder (ARAE) (Zhao, Kim, Zhang, Rush, & LeCun, 2017) model to summarize review data categorized by topic. ARAE is known as a generative model for text data and it is an autoencoder that can represent discretely structured text data as continuous space by learning Generative Adversarial Network (GAN) (Goodfellow et al., 2014) jointly. This property was leveraged to generate abstractive summaries in this work.

In addition, we proposed the Conditional Adversarially Regularized Autoencoder (CARAE) model, which can consider the conditional probability of the review data cluster. This model was designed to improve summary sentence generation performance for each cluster.

In summary, this paper’s contributions are the followings:

• We proposed a novel and intuitive summary generation method by using code vectors of autoencoder.

• We used ARAE model for acquiring code vectors in an abstractive summarization method that can be trained unsupervisedly.

• We proposed a new autoencoder model, CARAE, which is more specialized in summarizing review data categorized by topic.

• Our proposed models performed better than the existing approaches in both English and Korean document summarization.

• Our proposed models can help many companies to understand customer feedback more easily and clearly.

The remainder of the paper is organized as follows. Section 2 discusses related work. Sections 3 and 4 explain the experimental methods and setup, respectively. Section 5 presents the experimental results. Section 6 discusses the results. Section 7 provides conclusions and ideas for future research.