Memetic feature selection for multilabel text categorization using label frequency difference

Abstract

Multilabel text categorization is an important task in modern text mining applications. Text datasets comprise an excessive number of terms, and this can degrade the accuracy. Therefore, conventional studies applied a feature selection method before text categorization. Recently, memetic feature selection methods that hybridize an evolutionary feature wrapper and a filter have gained popularity and showed promising results. However, conventional memetic text feature selection methods suffer from limited performance because the used feature filter requires problem transformation that degrades the search capability, resulting in unrefined feature subsets with poor accuracy. In this study, we propose an effective memetic feature selection method based on a novel feature filter that is highly specialized to multilabel text categorization. Our experiments demonstrate that the proposed method significantly outperforms several conventional methods.

Introduction

Text categorization involves identifying relevant categories of a given text [40]. It is a core technique for many applications such as sentiment analysis [7], [29], [48], spam detection [47], review recommendation [28], and clinical analysis [11]. A single text can be assigned to multiple categories simultaneously, and therefore, task categorization needs to be performed using multilabel classification [17], [21], [22]. Let $W\subset {\mathbb{B}}^d$ denote a set of text patterns constructed from a set of term features $F=\{f_1,\dots ,f_d\},$ where each feature can represent each term by encoding its presence or absence [14]. Then, each pattern w_i ∈ W, where 1 ≤ i ≤ |W|, is assigned to a label subset λ_i ⊆ L, where $L=\{l_1,\dots ,l_{\left|L\right|}\}$. Among thousands of features, there can be a subset of unnecessary features that encodes meaningless terms such as stop words [8]. In this situation, multilabel feature selection can be a useful preprocessing step because it highlights the important features that significantly contribute to pattern-label set association by discarding unnecessary features [23].

Multilabel feature selection involves identifying a subset S ⊂ F comprising n ≪ |F| features that is significantly relevant to label set L. Thus, an effective search strategy should be devised to locate the best feature subset because of ${\sum }_{i=1}^n\left(\genfrac{}{}{0pt}{}{d}{i}\right)$ possible feature subsets. To achieve this, conventional studies considered feature wrappers and filters. Specifically, the filter approach evaluates the importance of features based on the intrinsic characteristics of data using a score function such as χ² statistics or information gain [39]. By contrast, the wrapper approach evaluates using a specific learning algorithm, that is, a classifier. Thus, each can be distinguished according to whether a classifier is used to evaluate the superiority of the candidate feature subsets and feature wrappers that often outperform the feature filters in terms of learning accuracy because of the interaction with the learning algorithm [20].

Population-based evolutionary algorithms such as genetic algorithms were frequently used as a search method for feature wrappers owing to their stochastic global search capability [42]. However, conventional evolutionary search methods waste computational resources for fine-tuning solutions because new offspring are created by randomly combining the ancestors. A promising alternative is to use an ideal hybridization of an evolutionary feature wrapper and a filter to obtain the optimal solution, that is, a feature subset, without becoming trapped in the local optima; separately, unrefined feature subsets created randomly in the evolutionary feature wrappers can be fine-tuned using the score function of the feature filter. Consequently, recent studies have focused on hybridizing the evolutionary feature wrapper and filter for multilabel text categorization, that is, memetic feature selection, because of its excellent search capability [7], [10], [18], [27].

Few memetic feature selection methods have been considered in conventional text categorization studies [27], [47]. Those that were showed promising results demonstrating the advantages of hybridization [10]. However, their performance is limited because the score functions of used or candidate feature filters [1], [6], [10], [34], [37] for hybridization commonly require the transformation of multiple labels into a single label to obtain the feature importance scores for the fine-tuning process. This drawback leads to an unrefined final feature subset owing to the degradation of the fine-tuning capability of the feature filter, which in turn is caused by inaccurate score evaluation due to information loss from the transformation [19]. In addition, these score functions are designed without considering cooperation with the evolutionary feature wrapper. This results in a complicated hybridization process that requires additional parameters.

In this study, we propose an effective score function called “label frequency difference” (LFD) that is highly specialized for memetic multilabel text feature selection. This score function calculates the conditional frequencies of labels directly and determines the discriminating power of features based on the difference in label frequencies according to the presence/absence of terms. Thus, this scoring process can circumvent the degradation of fine-tuning capability because it does not incur any problems during transformation.