A unified classifiability analysis framework based on meta-learner and its application in spectroscopic profiling data

Abstract

Spectroscopic profiling data (e.g., Raman spectroscopy and mass spectroscopy), combined with machine learning, have provided a data-driven approach for discriminative tasks. In these tasks, researchers often start with simple classification models. If one model doesn’t work, they will try more sophisticated models. If all models fail, the researchers will deem the data set as “inseparable.“ This “trial-and-error” practice reveals a fundamental question: does the dataset possess the necessary statistical power for the current discriminative task? This “classifiability analysis” is an implicit and often neglected step in the data-driven pipeline. This paper aims to design a unified methodological framework for classifiability analysis. In this framework, a meta-learner model combines diversified atom metrics (e.g., Bayes error rate / irreducible error, classification accuracy, information gain / mutual information) into one unified metric (d). We have successfully used the proposed framework to analyze a spectroscopic profiling dataset to discriminate vintage liquors of different ages. A significant difference (d = 1.447. d > 0.8 indicates a significant difference) between 5-year and 16-year liquors.

Appendix

1.1 Proof of equation

$$\begin{array}{*{20}c} I(X;Y)=\sum\limits_{{y \in Y}} {\sum\limits_{{x \in X}} {p(x,y)\log } } \frac{{p(x,y)}}{{p(x)p(y)}} \hfill \\ =\sum\limits_{{y \in Y}} {\sum\limits_{{x \in X}} {p(x,y)\log } } \frac{{p(y|x)}}{{p(y)}} \hfill \\ =\sum\limits_{{y \in Y}} {\sum\limits_{{x \in X}} {p(x,y)\log } } p(y|x) - \sum\limits_{{y \in Y}} {\sum\limits_{{x \in X}} {p(x,y)\log } } p(y) \hfill \\ = - ( - \sum\limits_{{x \in X}} {\sum\limits_{{y \in Y}} {p(x,y)\log } } p(y|x))+( - \sum\limits_{{y \in Y}} {p(y)\log } p(y)) \hfill \\ = - H(Y|X)+H(Y) \hfill \\ =IG(Y|X) \hfill \\\end{array}$$

1.2 A brief introduction of Gujing Tribute Liquor

With a long history, Gujing Tribute Liquor with fragrant taste is one of the eight most famous liquors in China. In 196AD, Cao Cao presented the “Jiuyun Spring Liquor” produced in his hometown as the royal liquor and its brewing methods to the Emperor Xian of the Han Dynasty. During the Wan Li Reign of the Ming Dynasty, it was presented to the royal court as a “tribute” until Qing Dynasty. Hence the liquor is named “Gujing Tribute Liquor.“ Based on traditional processes, it has scientific recipes and technological innovations. It features “crystal clear, sweet and mellow like orchid, velvety and lasting after tasting” and brings a unique taste known for its sweetness, aroma, and full flavor. It was awarded the gold medal of the national liquor-tasting conference four times and won “National Famous Liquor.“ In March 2003, it was incorporated into the system for protecting original products. In 2005, it became a national iconic geographical product, gained wide acclaim, and has been popular both at home and abroad. (source: http://english.bozhou.gov.cn/content/33.html)

1.3 Iris dataset case study

To further demonstrate the proposed method can be generalized to other domains, we also conducted a case study on the iris dataset. The iris dataset is a widely used classification dataset (created by R.A. Fisher). The dataset contains three classes (iris setosa, iris versicolour and iris virginica) and four features (sepal length, sepal width, petal length, and petal width). Each class has 50 samples. Figure 4 is the scatter plot after dimension reduction. The first class “setosa” is far from the others. The second and third classes (“versicolor” and “virginica”) are close to each other.

Fig. 4

Scatterplot of the iris dataset. The original 4 features are reduced to 2 by PCA

For this dataset, we choose use the same atom metric set and meta-learner model as before. After training, the meta-learner model “d = w0 + w1×BER + w2×ACC + w3×IG” has the following weights: w0 = -1.78, w1 = 1.37, w2 = 3.15, w3 = 0.10.

Table 5 shows the final classifiable scores given by the trained model. As the dataset is multi-class, we use the “ovo (one vs. one)” strategy. The between-class scores indicate all classes are highly classifiable. Table 6 shows the logistic regression models trained on the iris dataset.

Table 5 Classifiability analysis results on the iris dataset

Table 6 Classification result on the iris dataset with logistic regression models

Through this case study, the predicted classifiable scores are consistent with the dataset’s scatter plot (Fig. 4) and actual classification results (Table 6). For example, the score between versicolour and virginica is smaller than the other two pairs (1.41 vs. 1.59), which is also reflected in Fig. 4; Table 6 (classification accuracy: 0.97 vs. 1.0).