An evaluation and annotation methodology for product category matching in e-commerce

Highlights

• Product category matching is an important task in digital marketplaces and e-commerce.

• This paper motivates, describes and formalizes the problem of product category matching.

• The paper also presents a rigorously designed methodology and guidelines for acquiring reliable and cost-effective annotations for this task.

• The utility of all methods presented is validated on three real-world e-commerce taxonomies.

Abstract

Product category matching is an important task in digital marketplaces and e-commerce, helping to power better search and recommendations in an online context. While variants of the problem have received some attention in academia, there is no documented guidance on how to efficiently acquire annotations for evaluating multiple (current and future) models, many of which rely on modern machine learning techniques such as neural representation learning. In this paper, we motivate and formalize the problem of product category matching in e-commerce, and present a rigorously designed set of guidelines and methodology for acquiring annotations in a cost-effective and reliable manner. We also present a methodology for using the annotations to compare solutions of two or more product category matching methods, including comparing models both before and after annotation. Three widely used e-commerce product category taxonomies, and multiple metrics, are used to demonstrate the utility of our proposals.

Introduction

The last decade has witnessed the rapid rise of e-commerce, including e-commerce marketplaces and platforms (such as eBay and Amazon) but also the adoption of e-commerce technologies by traditional retailers like Walmart and Target (Krishnamurthy, 2004, Chaffey, 2007, Hänninen et al, 2018, Mandel, 2017). In online marketplaces, e-commerce platforms, and even media relating thereof (e.g., product reviews and influencer blogs), product category matching between two independent webpages or platforms is a practical problem for users, advertisers and aggregators of information. While we define the problem formally in Section 2, Fig. 1(a) illustrates an intuitive example. One website (Walmart) may be talking about ‘runner rugs’, while another (Target) refers to the same concept as ‘runners’. The burden is on the user to find different mentions of the same product category through more intensive search (e.g., by posing different keywords in a search engine) or to be limited to the results that show up as relevant for a specific search phrase, even though a better product (described using a different phrase) or price may be available elsewhere.

In our own experience, we have found that there are several practical reasons why product category matching is an important problem, especially for media companies relying on advertising dollars. One reason is that when users are browsing media websites, including blog and product review sites, they may be exposed to a particular product that they would either like to immediately purchase, or research further for a future purchase. Linking the product category mentioned in the media post to retailers’ product webpages, many of whom may not refer to it in the same way (as illustrated above in the Walmart-Target example), is clearly valuable.

A key aspect of the problem is its domain-specific nature. In the e-commerce domain, product categories are arranged in a taxonomy, and along with the product category label, the ‘path’ leading to it from the root of the taxonomy is also an important structured attribute.¹ A fragment of this taxonomy is illustrated in Fig. 1(b) for both websites. We formally define a taxonomy in Section 2. Because of this structure, classic ‘unstructured’ solutions such as ‘string matching’ were found to be too noisy to be useful even in preliminary experiments (Ukkonen, 1985, Navarro, 2001). Instead, we hypothesize that techniques that take both the label and the path into account may be more successful in determining when two concepts match, compared with solutions that rely only on the label. Through experimental results, we show that a method that takes the structure of the taxonomy into account when matching concepts between two taxonomies indeed performs better than one that only takes labels into account. Specific contributions are enumerated below:

• First, we formalize and define the problem of product category matching, especially as it applies to the structured version of the problem that we intuitively described through the Walmart-Target example above.

• Second, we present a rigorous set of guidelines for acquiring annotations for the product category matching problem in an efficient, cost-effective and reliable manner. Following an experimental study, we also present feedback from actual annotators that may allow further customization and task-specific refinement of these guidelines in other enterprises.

• Third, we present a methodology for using the acquired annotations to evaluate two or more candidate solutions for product category matching. Within e-commerce, such evaluations have been conducted either behind closed doors, or using task-specific measures and datasets that may not have validity beyond that particular enterprise.² In contrast, we present a clear and replicable description of our experimental findings and evaluation methodology.

• Finally, using three widely used e-commerce product category taxonomies, we conduct an experimental study to evaluate two candidate solutions inspired by recent progress in representation learning. We also use the acquired annotations to evaluate models that may be developed and proposed after annotations have been collected.