Anirban I Ghosh
ABSTRACT
Businesses are increasingly reliant on Machine Learning models to manage user experiences. It becomes important to not only focus on building robust and state-of-the-art models but also continuously monitor and evaluate them. Continuous monitoring enables the AI team to ensure the right frequency of model training and pro-actively investigate erroneous patterns and predictions, before it has a wider business impact. A robust and effective monitoring system is thus needed to ensure business and engineering teams are aware of model performance and any data anomalies which could impact downstream model accuracy. In this paper, we present our Health Assurance model monitoring solution. Currently, the system serves the health monitoring needs of more than 250 models across 11 AI verticals with an average anomaly detection precision of 60%.
- Samuel Ackerman, Parijat Dube, Eitan Farchi, Orna Raz, and Marcel Zalmanovici. 2021. Machine Learning Model Drift Detection Via Weak Data Slices. (2021). https://doi.org/10.48550/ARXIV.2108.05319Google Scholar
- McKinsey B. Cheatham. [n. d.]. Confronting the risks of artificial intelligence. https://www.mckinsey.com/business-functions/quantumblack/our-insights/confronting-the-risks-of-artificial-intelligenceGoogle Scholar
- Rosana Noronha Gemaque, Albert França Josuá Costa, Rafael Giusti, and Eulanda Miranda dos Santos. 2020. An overview of unsupervised drift detection methods. WIREs Data Mining and Knowledge Discovery 10, 6 (2020), e1381. https://doi.org/10.1002/widm.1381 arXiv:https://wires.onlinelibrary.wiley.com/doi/pdf/10.1002/widm.1381Google ScholarCross Ref
- Carlos A. Gomez-Uribe and Neil Hunt. 2016. The Netflix Recommender System: Algorithms, Business Value, and Innovation. ACM Trans. Manage. Inf. Syst. 6, 4, Article 13 (dec 2016), 19 pages. https://doi.org/10.1145/2843948Google ScholarDigital Library
- Marcia Henke, Eulanda Santos, Eduardo Souto, and Altair O Santin. 2021. Spam Detection Based on Feature Evolution to Deal with Concept Drift. JUCS - Journal of Universal Computer Science 27, 4 (2021), 364–386. https://doi.org/10.3897/jucs.66284 arXiv:https://doi.org/10.3897/jucs.66284Google ScholarCross Ref
- Grace A. Lewis, Sebastián Echeverría, Lena Pons, and Jeffrey Chrabaszcz. 2022. Augur: A Step Towards Realistic Drift Detection in Production ML Systems. In 2022 IEEE/ACM 1st International Workshop on Software Engineering for Responsible Artificial Intelligence (SE4RAI). 37–44.Google ScholarDigital Library
- Scott M Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. In Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Vol. 30. Curran Associates, Inc.https://proceedings.neurips.cc/paper/2017/file/8a20a8621978632d76c43dfd28b67767-Paper.pdfGoogle Scholar
- David Nigenda, Zohar S. Karnin, Muhammad Bilal Zafar, Raghu Ramesha, Alan Tan, Michele Donini, and Krishnaram Kenthapadi. 2021. Amazon SageMaker Model Monitor: A System for Real-Time Insights into Deployed Machine Learning Models. ArXiv abs/2111.13657(2021).Google Scholar
- Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. "Why Should I Trust You?": Explaining the Predictions of Any Classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (San Francisco, California, USA) (KDD ’16). Association for Computing Machinery, New York, NY, USA, 1135–1144. https://doi.org/10.1145/2939672.2939778Google ScholarDigital Library
- D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, and Michael Young. 2014. Machine Learning: The High Interest Credit Card of Technical Debt. In SE4ML: Software Engineering for Machine Learning (NIPS 2014 Workshop).Google Scholar
- Ruoying Wang, Kexin Nie, Tie Wang, Yang Yang, and Bo Long. 2020. Deep Learning for Anomaly Detection. In Proceedings of the 13th International Conference on Web Search and Data Mining. Association for Computing Machinery, New York, NY, USA, 894–896. https://doi.org/10.1145/3336191.3371876Google ScholarDigital Library
- WSJ. [n. d.]. Zillow Quits Home-Flipping Business, Cites Inability to Forecast Prices. https://www.wsj.com/articles/zillow-quits-home-flipping-business-cites-inability-to-forecast-prices-11635883500Google Scholar
Recommendations
Internet of Things-assisted intelligent monitoring model to analyze the physical health condition
Frontiers in Telemedicine and Internet of Things in Health MonitoringBACKGROUND:Physical health monitoring may take several forms, from individual quality changes to complex health checks carried out by health staff. Present health issues are detected with monitoring, and potential health ...
Construction of medical equipment-based doctor health monitoring system
The health status of doctors has been overlooked by the society and even the doctors themselves, especially those doctors who work long hours. Their attention is always on patients, so they are more likely to ignore their own health problems. Therefore, ...
A medical monitoring scheme and health‐medical service composition model in cloud‐based IoT platform
AbstractAdvanced technologies such as internet of things (IoT) and clouds have significantly influenced on modern medical monitoring systems. Analytical statistics derived from massive patients' medical data via different data analysis methods, contribute ...
image image The objective of this paper is to present an inclusive scheme for modern medical monitoring systems that apply emerging technologies such as IoT and clouds in which, the proper medical prescriptions are offered by the system based on the ...
Comments