Abstract
Machine Learning models often excel in controlled environments but may struggle with noisy, incomplete, or shifted real-world data. Ensuring that these models maintain high performance despite these imperfections is crucial for practical applications, such as medical diagnosis or autonomous driving. This paper introduces a novel framework to systematically analyse the robustness of Machine Learning models against noisy data. We propose two empirical methods: (1) Noise Tolerance Estimation, which calculates the noise level a model can withstand without significant degradation in performance, and (2) Robustness Ranking, which ranks Machine Learning models by their robustness at specific noise levels. Utilizing Cohen’s kappa statistic, we measure the consistency between a model’s predictions on original and perturbed datasets. Our methods are demonstrated using various datasets and Machine Learning techniques, identifying models that maintain reliability under noisy conditions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
For comparison purposes, we disregard the class label, acknowledging that noise may alter an instance’s true class.
References
Arslan, M., Guzel, M., Demirci, M., Ozdemir, S.: Smote and gaussian noise based sensor data augmentation. In: 2019 4th International Conference on Computer Science and Engineering (UBMK), pp. 1–5. IEEE (2019)
Braun, S., Neil, D., Liu, S.C.: A curriculum learning method for improved noise robustness in automatic speech recognition. In: 2017 25th EUSIPCO, pp. 548–552. IEEE (2017)
Fabra-Boluda, R., Ferri, C., Ramírez-Quintana, M., Martínez-Plumed, F.: Unveiling the robustness of machine learning families. Mach. Learn.: Sci. Technol. (2024). http://iopscience.iop.org/article/10.1088/2632-2153/ad62ab
Fernández-Delgado, M., Cernadas, E., Barro, S., Amorim, D.: Do we need hundreds of classifiers to solve real world classification problems? JMLR 15(1), 3133–3181 (2014)
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. In: 3rd Intenational Conference on Learning, Representations, ICLR (2015). http://arxiv.org/abs/1412.6572
Hendrycks, D., et al.: The many faces of robustness: a critical analysis of out-of-distribution generalization. In: Proceedings of the IEEE/CVF ICCV, pp. 8340–8349 (2021)
Kalapanidas, E., Avouris, N., Craciun, M., Neagu, D.: Machine learning algorithms: a study on noise sensitivity. In: 1st Balcan Conference in Informatics, pp. 356–365 (2003)
Landis, R., Koch, G.: An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics 363–374 (1977)
Li, J., Xiong, C., Hoi, S.C.: Learning from noisy data with robust representation learning. In: Proceedings of the IEEE/CVF ICCV, pp. 9485–9494 (2021)
Ljunggren, D., Ishii, S.: A comparative analysis of robustness to noise in machine learning classifiers (2021)
Martínez-Plumed, F., Prudêncio, R.B., Martínez-Usó, A., Hernández-Orallo, J.: Item response theory in AI: analysing machine learning classifiers at the instance level. Artif. Intell. 271, 18–42 (2019)
Molnar, C., Bischl, B., Casalicchio, G.: iml: an R package for interpretable machine learning. JOSS 3(26), 786 (2018)
Ramoni, M., Sebastiani, P.: Robust learning with missing data. Mach. Learn. 45, 147–170 (2001)
Rousseeuw, P.J.: Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20, 53–65 (1987)
Saberi, M., Azadeh, A., Nourmohammadzadeh, A., Pazhoheshfar, P.: Comparing performance and robustness of SVM and ANN for fault diagnosis in a centrifugal pump. In: 19th International Congress on Modelling and Simulation, pp. 433–439 (2011)
Sáez, J.A., Galar, M., Luengo, J., Herrera, F.: Analyzing the presence of noise in multi-class problems: alleviating its influence with the one-vs-one decomposition. Knowl. Inf. Syst. 38(1), 179–206 (2014)
Sáez, J.A., Luengo, J., Herrera, F.: Evaluating the classifier behavior with noisy data considering performance and robustness: the equalized loss of accuracy measure. Neurocomputing 176, 26–35 (2016)
Subbaswamy, A., Adams, R., Saria, S.: Evaluating model robustness and stability to dataset shift. In: AISTATS, pp. 2611–2619. PMLR (2021)
Vanschoren, J., Van Rijn, J.N., Bischl, B., Torgo, L.: OpenML: networked science in machine learning. ACM SIGKDD Explor. Newsl. 15(2), 49–60 (2014)
Viera, A.J., Garrett, J.M., et al.: Understanding interobserver agreement: the kappa statistic. Fam. Med. 37(5), 360–363 (2005)
Wu, X., Zhu, X.: Mining with noise knowledge: error-aware data mining. IEEE Trans. Syst. Man Cybern.-Part A: Syst. Hum. 38(4), 917–932 (2008)
Zhu, X., Wu, X.: Class noise vs. attribute noise: a quantitative study. Artif. Intell. Rev. 22(3), 177–210 (2004)
Acknowledgments
This work was funded by CIPROM/2022/6 (FASSLOW) funded by Generalitat Valenciana, FISCALTICS (I+D+i PID2022-140110OA-I00) granted by MICIU/AEI/10.13039/ 501100011033, and Spanish grant PID2021-122830OB-C42 (SFERA) funded by MCIN/AEI/10.13039/ 501100011033 and “ERDF A way of making Europe”. CPF is supported by UPV under FPI grant PAID-01-22.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Padró-Ferragut, C., Ramírez-Quintana, M.J., Martínez-Plumed, F. (2025). Noise Tolerance and Robustness Ranking in Machine Learning Models. In: Julian, V., et al. Intelligent Data Engineering and Automated Learning – IDEAL 2024. IDEAL 2024. Lecture Notes in Computer Science, vol 15347. Springer, Cham. https://doi.org/10.1007/978-3-031-77738-7_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-77738-7_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-77737-0
Online ISBN: 978-3-031-77738-7
eBook Packages: Computer ScienceComputer Science (R0)