Abstract
This half-day tutorial introduces modern techniques for practical uncertainty quantification specifically in the context of multi-class and multi-label text classification. First, we explain the usefulness of estimating aleatoric uncertainty and epistemic uncertainty for text classification models. Then, we describe several state-of-the-art approaches to uncertainty quantification and analyze their scalability to big text data: Virtual Ensemble in GBDT, Bayesian Deep Learning (including Deep Ensemble, Monte-Carlo Dropout, Bayes by Backprop, and their generalization Epistemic Neural Networks), as well as Evidential Deep Learning (including Prior Networks and Posterior Networks). Next, we discuss typical application scenarios of uncertainty quantification in text classification (including in-domain calibration, cross-domain robustness, and novel class detection). Finally, we list popular performance metrics for the evaluation of uncertainty quantification effectiveness in text classification. Practical hands-on examples/exercises are provided to the attendees for them to experiment with different uncertainty quantification methods on a few real-world text classification datasets such as CLINC150.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
- 7.
- 8.
References
Abdar, M.: A review of uncertainty quantification in deep learning: techniques. Appli. Challen. Inf. Fusion 76, 243–297 (2021). https://doi.org/10.1016/j.inffus.2021.05.008
Arora, G., Jain, C., Chaturvedi, M., Modi, K.: HINT3: Raising the bar for Intent Detection in the Wild. In: Proceedings of the First Workshop on Insights from Negative Results in NLP, pp. 100–105. Association for Computational Linguistics, Online (Nov 2020). https://doi.org/10.18653/v1/2020.insights-1.16
Arora, U., Huang, W., He, H.: Types of out-of-distribution texts and how to detect them. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10687–10701. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic (Nov 2021). https://doi.org/10.18653/v1/2021.emnlp-main.835
Blundell, C., Cornebise, J., Kavukcuoglu, K., Wierstra, D.: Weight Uncertainty in neural network. In: Proceedings of the 32nd International Conference on Machine Learning, pp. 1613–1622. PMLR (Jun 2015)
Cerutti, F., Kaplan, L.M., Kimmig, A., Şensoy, M.: Handling epistemic and aleatory uncertainties in probabilistic circuits. Mach. Learn. 111(4), 1259–1301 (2022)
Cerutti, F., Kaplan, L.M., Sensoy, M.: Evidential reasoning and learning: a survey. In: Thirty-First International Joint Conference on Artificial Intelligence, vol. 6, pp. 5418–5425 (Jul 2022). https://doi.org/10.24963/ijcai.2022/760
Charpentier, B., Zügner, D., Günnemann, S.: Posterior network: uncertainty estimation without ood samples via density-based pseudo-counts. In: Advances in Neural Information Processing Systems, vol. 33, pp. 1356–1367. Curran Associates, Inc. (2020)
Foong, A.Y.K., Li, Y., Hernández-Lobato, J.M., Turner, R.E.: ’In-Between’ Uncertainty in Bayesian Neural Networks (Jun 2019). https://doi.org/10.48550/arXiv.1906.11537
Gal, Y., Ghahramani, Z.: Dropout as a bayesian approximation: representing model uncertainty in deep learning. In: Proceedings of The 33rd International Conference on Machine Learning, pp. 1050–1059. PMLR (Jun 2016)
GGawlikowski, J., et al.: A Survey of Uncertainty in Deep Neural Networks (Jan 2022). https://doi.org/10.48550/arXiv.2107.03342
Ghosh, S., et al.: Uncertainty quantification 360: a hands-on tutorial. In: 5th Joint International Conference on Data Science & Management of Data (9th ACM IKDD CODS and 27th COMAD), CODS-COMAD 2022, pp. 333–335. Association for Computing Machinery, New York (Jan 2022). https://doi.org/10.1145/3493700.3493767
Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: Proceedings of the 34th International Conference on Machine Learning, ICML 2017, vol. 70, pp. 1321–1330. JMLR.org, Sydney, NSW, Australia (Aug 2017)
Guo, Z., et al.: A Survey on Uncertainty Reasoning and Quantification for Decision Making: Belief Theory Meets Deep Learning (Jun 2022). https://doi.org/10.48550/arXiv.2206.05675
Hariri, R.H., Fredericks, E.M., Bowers, K.M.: Uncertainty in big data analytics: survey, opportunities, and challenges. J. Big Data 6(1), 1–16 (2019). https://doi.org/10.1186/s40537-019-0206-3
Hendrycks, D., Gimpel, K.: A baseline for detecting misclassified and out-of-distribution examples in neural networks. In: International Conference on Learning Representations (2017)
Hendrycks, D., Mazeika, M., Dietterich, T.G.: Deep anomaly detection with outlier exposure. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, 6–9 May 2019. OpenReview.net (2019)
Houlsby, N., Huszár, F., Ghahramani, Z., Lengyel, M.: Bayesian Active Learning for Classification and Preference Learning (Dec 2011). https://doi.org/10.48550/arXiv.1112.5745
Hu, Y., Khan, L.: Uncertainty-aware reliable text classification. In: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD 2021, pp. 628–636. Association for Computing Machinery, New York (Aug 2021). https://doi.org/10.1145/3447548.3467382
Hüllermeier, E., Waegeman, W.: Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods. Mach. Learn. 110(3), 457–506 (2021). https://doi.org/10.1007/s10994-021-05946-3
Kabir, H.M.D., Khosravi, A., Hosen, M.A., Nahavandi, S.: Neural network-based uncertainty quantification: a survey of methodologies and applications. IEEE Access 6, 36218–36234 (2018). https://doi.org/10.1109/ACCESS.2018.2836917
Kendall, A., Gal, Y.: What uncertainties do we need in bayesian deep learning for computer vision? In: Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017)
Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. In: Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017)
Larson, S., et al.: An evaluation dataset for intent classification and out-of-scope prediction. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1311–1316. Association for Computational Linguistics, Hong Kong, China (Nov 2019). https://doi.org/10.18653/v1/D19-1131
Leibig, C., Allken, V., Ayhan, M.S., Berens, P., Wahl, S.: Leveraging uncertainty information from deep neural networks for disease detection. Sci. Rep. 7(1), 17816 (2017). https://doi.org/10.1038/s41598-017-17876-z
Li, Y., Chen, J., Feng, L.: Dealing with uncertainty: a survey of theories and practices. IEEE Trans. Knowl. Data Eng. 25(11), 2463–2482 (2013). https://doi.org/10.1109/TKDE.2012.179
Lu, X., et al.: Robustness of Epinets against Distributional Shifts (Jun 2022). https://doi.org/10.48550/arXiv.2207.00137
Malinin, A., Gales, M.: Predictive uncertainty estimation via prior networks. In: Advances in Neural Information Processing Systems, vol. 31. Curran Associates, Inc. (2018)
Malinin, A., Prokhorenkova, L., Ustimenko, A.: Uncertainty in gradient boosting via ensembles. In: International Conference on Learning Representations (Mar 2021)
Mena, J., Pujol, O., Vitrià, J.: A survey on uncertainty estimation in deep learning classification systems from a bayesian perspective. ACM Comput. Sur. 54(9), 193:1–193:35 (2021). https://doi.org/10.1145/3477140
Osband, I., et al.: Epistemic Neural Networks (Oct 2022). https://doi.org/10.48550/arXiv.2107.08924
Ovadia, Y., et al.: Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. In: Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc. (2019)
Psaros, A.F., Meng, X., Zou, Z., Guo, L., Karniadakis, G.E.: Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons (Jan 2022). https://doi.org/10.48550/arXiv.2201.07766
Rawat, M., Hebbalaguppe, R., Vig, L.: PnPOOD : Out-Of-Distribution Detection for Text Classification via Plug and Play Data Augmentation (Oct 2021). https://doi.org/10.48550/arXiv.2111.00506
Sensoy, M., Kaplan, L., Cerutti, F., Saleki, M.: Uncertainty-Aware Deep Classifiers Using Generative Models. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34(04), pp. 5620–5627 (2020). https://doi.org/10.1609/aaai.v34i04.6015
Sensoy, M., Kaplan, L., Kandemir, M.: Evidential Deep Learning to Quantify Classification Uncertainty. In: Advances in Neural Information Processing Systems, vol. 31. Curran Associates, Inc. (2018)
Siddhant, A., Lipton, Z.C.: Deep bayesian active learning for natural language processing: results of a large-scale empirical study. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2904–2909. Association for Computational Linguistics, Brussels, Belgium (Oct 2018). https://doi.org/10.18653/v1/D18-1318
Ulmer, D.: A Survey on Evidential Deep Learning For Single-Pass Uncertainty Estimation (Dec 2021). https://doi.org/10.48550/arXiv.2110.03051
Van Landeghem, J., Blaschko, M., Anckaert, B., Moens, M.F.: Benchmarking scalable predictive uncertainty in text classification. IEEE Access 10, 43703–43737 (2022). https://doi.org/10.1109/ACCESS.2022.3168734
Widmann, D., Lindsten, F., Zachariah, D.: Calibration tests in multi-class classification: a unifying framework. In: Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc. (2019)
Widmann, D., Lindsten, F., Zachariah, D.: Calibration tests beyond classification. In: International Conference on Learning Representations (2021)
Zhang, H., Xu, H., Zhao, S., Zhou, Q.: Learning Discriminative Representations and Decision Boundaries for Open Intent Detection (Jul 2022). https://doi.org/10.48550/arXiv.2203.05823
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, D., Sensoy, M., Makrehchi, M., Taneva-Popova, B. (2023). Uncertainty Quantification for Text Classification. In: Kamps, J., et al. Advances in Information Retrieval. ECIR 2023. Lecture Notes in Computer Science, vol 13982. Springer, Cham. https://doi.org/10.1007/978-3-031-28241-6_38
Download citation
DOI: https://doi.org/10.1007/978-3-031-28241-6_38
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-28240-9
Online ISBN: 978-3-031-28241-6
eBook Packages: Computer ScienceComputer Science (R0)