tutorial

Uncertainty Quantification 360: A Hands-on Tutorial

Authors:

Karthikeyan Natesan Ramamurthy,

Prasanna Sattigeri,

Yunfeng ZhangAuthors Info & Claims

CODS-COMAD '22: Proceedings of the 5th Joint International Conference on Data Science & Management of Data (9th ACM IKDD CODS and 27th COMAD)

Pages 333 - 335

https://doi.org/10.1145/3493700.3493767

Published: 08 January 2022 Publication History

Abstract

This tutorial presents an open source Python package (https://github.com/IBM/UQ360) named Uncertainty Quantification 360 (UQ360), a toolkit that provides a broad range of capabilities for quantifying, evaluating, improving, and communicating uncertainty in the AI application development lifecycle. We will first introduce the concepts in uncertainty quantification through an interactive experience (http://uq360.mybluemix.net) followed by use cases with different quantification algorithms and evaluation metrics. The hands-on experience gained from tutorial will aid researchers and developers in producing and evaluating high-quality uncertainties from AI models in an efficient manner.

References

[1]

Umang Bhatt, Javier Antorán, Yunfeng Zhang, Q. Vera Liao, Prasanna Sattigeri, Riccardo Fogliato, Gabrielle Gauthier Melançon, Ranganath Krishnan, Jason Stanley, Omesh Tickoo, Lama Nachman, Rumi Chunara, Madhulika Srikumar, Adrian Weller, and Alice Xiang. 2021. Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty. In Proceedings of the AAAI/ACM Conference on Artificial Intelligence, Ethics, and Society.

Digital Library

[2]

Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. 2015. Weight Uncertainty in Neural Network. In Proceedings of the 32nd International Conference on Machine Learning(Proceedings of Machine Learning Research, Vol. 37), Francis Bach and David Blei (Eds.). PMLR, Lille, France, 1613–1622. http://proceedings.mlr.press/v37/blundell15.html

[3]

Tongfei Chen, Jirí Navrátil, Vijay Iyengar, and Karthikeyan Shanmugam. 2019. Confidence scoring using whitebox meta-models with linear classifier probes. In Proceedings of the International Conference on Artificial Intelligence and Statistics. 1467–1475.

[4]

Benjamin Elder, Matthew Arnold, Anupama Murthi, and Jiri Navratil. 2021. Learning Prediction Intervals for Model Performance. In Proceedings of the AAAI Conference on Artificial Intelligence.

[5]

Michael Fernandes, Logan Walls, Sean Munson, Jessica Hullman, and Matthew Kay. 2018. Uncertainty displays using quantile dotplots or cdfs improve transit decision-making. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 144.

Digital Library

[6]

Soumya Ghosh, William T. Stephenson, Tin D. Nguyen, Sameer Deshpande, and Tamara Broderick. 2020. Approximate Cross-Validation for Structured Models. In Advances in Neural Information Processing Systems, Vol. 33.

[7]

Soumya Ghosh, Jiayu Yao, and Finale Doshi-Velez. 2019. Model Selection in Bayesian Neural Networks via Horseshoe Priors. Journal of Machine Learning Research 20, 182 (2019), 1–46.

[8]

Alex Kendall and Yarin Gal. 2017. What uncertainties do we need in Bayesian deep learning for computer vision?. In Advances in Neural Information Processing Systems, Vol. 30. 5580–5590.

[9]

Roger Koenker and Gilbert Bassett, Jr.1978. Regression Quantiles. Econometrica 46, 1 (Jan. 1978), 33–50.

[10]

Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using Bayesian binning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 29.

[11]

Jiri Navratil, Matthew Arnold, and Benjamin Elder. 2020. Uncertainty Prediction for Deep Sequential Regression Using Meta Models. arxiv:2007.01350 [cs.LG]

[12]

Jiri Navratil, Benjamin Elder, Matthew Arnold, Soumya Ghosh, and Prasanna Sattigeri. 2021. Uncertainty Characteristics Curves: A Systematic Assessment of Prediction Intervals. arxiv:2106.00858 [cs.LG]

[13]

John C. Platt. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin Classifiers. MIT Press, Cambridge, MA, 61–74.

[14]

Carl Edward Rasmussen and Christopher K. I. Williams. 2006. Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA.

[15]

Jayaraman J Thiagarajan, Bindya Venkatesh, Prasanna Sattigeri, and Peer-Timo Bremer. 2020. Building calibrated deep models via uncertainty matching with auxiliary interval predictors. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 6005–6012.

[16]

Bianca Zadrozny and Charles Elkan. 2001. Obtaining calibrated probability estimates from decision trees and naive Bayesian classifiers. In Proceedings of the International Conference on Machine Learning. 609–616.

[17]

Yunfeng Zhang, Q Vera Liao, and Rachel KE Bellamy. 2020. Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency. 295–305.

Digital Library

Cited By

Sun PHolden PBirks H(2024)Can machine-learning algorithms improve upon classical palaeoenvironmental reconstruction models?Climate of the Past10.5194/cp-20-2373-202420:10(2373-2398)Online publication date: 24-Oct-2024
https://doi.org/10.5194/cp-20-2373-2024
Sander JYu CJalaian BBastian N(2024)Uncertainty-Quantified Neurosymbolic AI for Open Set Recognition in Network Intrusion DetectionMILCOM 2024 - 2024 IEEE Military Communications Conference (MILCOM)10.1109/MILCOM61039.2024.10773953(13-18)Online publication date: 28-Oct-2024
https://doi.org/10.1109/MILCOM61039.2024.10773953
Manzano MAyala CGómez C(2024)TrustML: A Python package for computing the trustworthiness of ML modelsSoftwareX10.1016/j.softx.2024.10174026(101740)Online publication date: May-2024
https://doi.org/10.1016/j.softx.2024.101740
Show More Cited By

Index Terms

Uncertainty Quantification 360: A Hands-on Tutorial

Index terms have been assigned to the content through auto-classification.

Recommendations

Numerical approach for quantification of epistemic uncertainty

In the field of uncertainty quantification, uncertainty in the governing equations may assume two forms: aleatory uncertainty and epistemic uncertainty. Aleatory uncertainty can be characterised by known probability distributions whilst epistemic ...
Uncertainty quantification methods for evolutionary optimization under uncertainty
GECCO '20: Proceedings of the 2020 Genetic and Evolutionary Computation Conference Companion

In this paper, we discuss the role of uncertainty quantification (UQ) in assisting optimization under uncertainty. UQ plays a significant role in quantifying the robustness of solutions so as to help the optimizer in achieving robust optimum solutions. ...
Quantification of model-form and predictive uncertainty for multi-physics simulation

Traditional uncertainty quantification in multi-physics design problems involves the propagation of parametric uncertainties in input variables such as structural or aerodynamic properties through a single, or series of models constructed to represent ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CODS-COMAD '22: Proceedings of the 5th Joint International Conference on Data Science & Management of Data (9th ACM IKDD CODS and 27th COMAD)

January 2022

357 pages

ISBN:9781450385824

DOI:10.1145/3493700

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 January 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Tutorial
Research
Refereed limited

Conference

CODS-COMAD 2022

Sponsor:

SIGGRAPH

CODS-COMAD 2022: 5th Joint International Conference on Data Science & Management of Data (9th ACM IKDD CODS and 27th COMAD)

January 8 - 10, 2022

Bangalore, India

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
214
Total Downloads

Downloads (Last 12 months)52
Downloads (Last 6 weeks)9

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sun PHolden PBirks H(2024)Can machine-learning algorithms improve upon classical palaeoenvironmental reconstruction models?Climate of the Past10.5194/cp-20-2373-202420:10(2373-2398)Online publication date: 24-Oct-2024
https://doi.org/10.5194/cp-20-2373-2024
Sander JYu CJalaian BBastian N(2024)Uncertainty-Quantified Neurosymbolic AI for Open Set Recognition in Network Intrusion DetectionMILCOM 2024 - 2024 IEEE Military Communications Conference (MILCOM)10.1109/MILCOM61039.2024.10773953(13-18)Online publication date: 28-Oct-2024
https://doi.org/10.1109/MILCOM61039.2024.10773953
Manzano MAyala CGómez C(2024)TrustML: A Python package for computing the trustworthiness of ML modelsSoftwareX10.1016/j.softx.2024.10174026(101740)Online publication date: May-2024
https://doi.org/10.1016/j.softx.2024.101740
Goldsteen AFarkash AHind M(2024)Assessing and implementing trustworthy AI across multiple dimensionsEthics in Online AI-based Systems10.1016/B978-0-443-18851-0.00001-9(229-257)Online publication date: 2024
https://doi.org/10.1016/B978-0-443-18851-0.00001-9
Tyralis HPapacharalampous G(2024)A review of predictive uncertainty estimation with machine learningArtificial Intelligence Review10.1007/s10462-023-10698-857:4Online publication date: 18-Mar-2024
https://doi.org/10.1007/s10462-023-10698-8
Upreti RLind PElmokashfi AYazidi A(2024)Trustworthy machine learning in the context of security and privacyInternational Journal of Information Security10.1007/s10207-024-00813-323:3(2287-2314)Online publication date: 3-Apr-2024
https://dl.acm.org/doi/10.1007/s10207-024-00813-3
Prabhudesai SHauth JGuo DRao ABanovic NHuan X(2023)Lowering the computational barrier: Partially Bayesian neural networks for transparency in medical imaging AIFrontiers in Computer Science10.3389/fcomp.2023.10711745Online publication date: 15-Feb-2023
https://doi.org/10.3389/fcomp.2023.1071174
Prabhudesai SYang LAsthana SHuan XLiao QBanovic N(2023)Understanding Uncertainty: How Lay Decision-makers Perceive and Interpret Uncertainty in Human-AI Decision MakingProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584033(379-396)Online publication date: 27-Mar-2023
https://dl.acm.org/doi/10.1145/3581641.3584033
Barandas MFolgado DSantos RSimão RGamboa H(2022)Uncertainty-Based Rejection in Machine Learning: Implications for Model Development and InterpretabilityElectronics10.3390/electronics1103039611:3(396)Online publication date: 28-Jan-2022
https://doi.org/10.3390/electronics11030396
Stitzlein CFielke SWaldner FSanderson T(2021)Reputational Risk Associated with Big Data Research and Development: An Interdisciplinary PerspectiveSustainability10.3390/su1316928013:16(9280)Online publication date: 18-Aug-2021
https://doi.org/10.3390/su13169280

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten