Cross-Category Defect Discovery from Online Reviews: Supplementing Sentiment with Category-Specific Semantics

Abstract

Online reviews contain many vital insights for quality management, but the volume of content makes identifying defect-related discussion difficult. This paper critically assesses multiple approaches for detecting defect-related discussion, ranging from out-of-the-box sentiment analyses to supervised and unsupervised machine-learned defect terms. We examine reviews from 25 product and service categories to assess each method’s performance. We examine each approach across the broad cross-section of categories as well as when tailored to a singular category of study. Surprisingly, we found that negative sentiment was often a poor predictor of defect-related discussion. Terms generated with unsupervised topic modeling tended to correspond to generic product discussions rather than defect-related discussion. Supervised learning techniques outperformed the other text analytic techniques in our cross-category analysis, and they were especially effective when confined to a single category of study. Our work suggests a need for category-specific text analyses to take full advantage of consumer-driven quality intelligence.

Appendix

To curate smoke terms, we utilize the CC score algorithm (Fan et al., 2005), an information retrieval technique that prior works have shown to be quite effective in smoke term curation (Abrahams et al., 2012, 2013, 2015; Goldberg & Abrahams, 2018). As the chi-squared distribution is used to test for the independence between two variables in statistics, information retrieval has utilized this principle to examine relationships between documents and words that they contain. Ng et al. (1997) first suggested a means of using a one-sided chi-squared test to select words or phrases associated with a relevant classification of documents; Fan et al. (2005) later expanded upon this technique. The CC score algorithm generates a relevance score for each term (word or phrase) in a corpus, where higher scores indicate more relevant terms that may be predictive of the target classification. We first distinguish between relevant documents (in our study, reviews) from the target classification (in our study, defect-related reviews) and non-relevant documents not from the target classification (in our study, non-defect-related reviews). Consider Table 8, which defines the relationships between document relevance and inclusion/exclusion of terms.

Table 8 Contingency table for CC score algorithm (adapted from Fan et al. (2005))

Given this contingency table, terms are given higher scores when they are especially frequent in documents that are relevant and especially infrequent in documents that are irrelevant. The CC score algorithm defines this relevance as follows for each term in the corpus:

$$ Relevance=\frac{\sqrt{N}\times \left( AD- CB\right)}{\sqrt{\left(A+B\right)\times \left(C+D\right)}} $$

(1)

The CC score algorithm generates a relevance score for each term in the corpus such that scores with high relevance scores occur frequently in relevant documents and infrequently in irrelevant documents. Thus, we may use high-scoring terms as predictors of relevance in unseen documents. After using the CC score algorithm to generate relevance scores for each smoke term, the lead author analyzed the top-ranking terms to remove any stop words (common English words like “a,” “an,” and “the”), common brands names, and/or common product (or service) categories (Abrahams et al., 2012, 2013, 2015). A coauthor further reviewed and reverified these decisions to ensure accuracy. The retained terms are referred to as smoke terms, and each set of smoke terms is referred to as a smoke term list.

When analyzing unseen reviews (e.g., our holdout set), we use the appropriate smoke term list to generate “smoke scores” for each review. For a given review, we determine this smoke score by searching for any occurrences of the smoke terms within that review. Each time we observe an occurrence of a smoke term, we increment that review’s smoke score by that smoke term’s relevance score as determined by the CC score algorithm. Finally, using these smoke scores, we can prioritize the reviews believed to refer to defects. We can sort the reviews from the highest smoke score to the lowest smoke score, where the highest smoke scores are the most likely to refer to defects.