Abstract
The performance of machine learning models deteriorates when the distribution of test data changes, which is called concept drift. One way to deal with concept drift is to continuously rebuild the model. If we want to minimize the frequency of rebuilding due to some constraints, however, it is important to detect concept drift as the timing when rebuilding is truly necessary. Taking advantage of ensemble models for concept drift detection may improve the detection accuracy. However, the behavior of ensemble model’s predictions and parameters in the presence of concept drift has not been fully investigated. In this study, we investigated how the ensemble models constructed by two different methods behave in the presence of concept drift. In the experiments, we monitored some metrics including the metrics that can be calculated only by the ensemble model and the metrics based on the model parameters. As a result, we found that the metrics show some behaviors that seem to be influenced by concept drift, suggesting that the detection accuracy of concept drift may be improved by using these metrics.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
A series of processes including preprocessing, model training, data and model validation, and inference using the model.
References
MLOps: Continuous delivery and automation pipelines in machine learning, https://cloud.google.com/architecture/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning. Accessed 29 July 2021
Korycki, Ł., Krawczyk, B.: Class-incremental experience replay for continual learning under concept drift. In: CVPR Workshops (2021)
Lindstrom, P., Mac Namee, B., Delany, S.J.: Drift detection using uncertainty distribution divergence. In: ICDM Workshops (2011)
Zhao, Y., et al.: SUOD: accelerating large-scale unsupervised heterogeneous outlier detection. In: MLSys (2021)
Fort, S., Hu, H., Lakshminarayanan, B.: Deep ensembles: a loss landscape perspective. arXiv preprint arXiv:1912.02757v2 (2020)
Lu, J., Liu, A., Dong, F., Gu, F., Gama, J., Zhang, G.: Learning under concept drift: a review. IEEE Trans. Knowl. Data Eng. 31(12), 2346–2363 (2020)
Lu, N., Zhang, G., Lu, J.: Concept drift detection via competence models. Artif. Intell. 209, 11–28 (2014)
Ditzler, G., Polikar, R.: Hellinger distance based drift detection for nonstationary environments. In: IEEE Symposium on Computational Intelligence in Dynamic and Uncertain Environments (2011)
Qahtan, A., Wang, S.: A pca-based change detection framework for multidimensional data streams categories and subject descriptors. In: KDD (2015)
Gama, J., Medas, P., Castillo, G., Rodrigues, P.: Learning with drift detection. In: Bazzan, A.L.C., Labidi, S. (eds.) SBIA 2004. LNCS (LNAI), vol. 3171, pp. 286–295. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-28645-5_29
Sethi, T.S., Kantardzic, M.: On the reliable detection of concept drift from streaming unlabeled data. Expert Syst. Appl. 82, 77–99 (2017)
Smutz, C., Stavrou, A.: When a tree falls: using diversity in ensemble classifiers to identify evasion in malware detectors. In: Proceedings 2016 Network and Distributed System Security (NDSS) Symposium, pp. 21–24. Internet Society, Reston (2016)
Haug, J., Kasneci, G.: Learning parameter distributions to detect concept drift in data streams. In: Proceedings of the 25th International Conference on Pattern Recognition (ICPR) (2020)
Tran, L., et al.: Hydra: preserving ensemble diversity for model distillation. In: ICML 2020 Workshop on Uncertainty and Robustness in Deep Learning (UDL) (2020)
Lakshminarayanan, B., Pritzel, A., Blundel, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. In: NeurIPS (2017)
UCI Machine Learning Repository. http://archive.ics.uci.edu/ml. Accessed 29 July 2021
Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: Proceedings of the International Conference on Machine Learning (ICML) (2013)
Stewart, G.W.: The efficient generation of random orthogonal matrices with an application to condition estimators. SIAM J. Numer. Anal. 17, 403–409 (1980)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of the 3rd International Conference on Learning Representations (ICLR) (2015)
Baier, L., Schlör, T., Schöffer, J., Kühl, N.: Detecting concept drift with neural network model uncertainty. arXiv preprint arXiv:2107.01873 (2021)
Montiel, J., Read, J., Bifet, A., Abdessalem, T.: Scikit-multiflow: a multi-output streaming framework. J. Mach. Learn. Res. 19(72), 1–5 (2018)
Kornblith, S., Norouzi, M., Lee, H., Hinton, G.: Similarity of neural network representations revisited. In: Proceedings of the 36th International Conference on Machine Learning (ICML) (2019)
Cortes, C., Mohri, M., Rostamizadeh, A.: Algorithms for learning kernels based on centered alignment. J. Mach. Learn. Res. 13, 795–828 (2012)
Ovadia, Y., et al.: Can you trust your model’s uncertainty? evaluating predictive uncertainty under dataset shift. In: NeurIPS (2019)
Acknowledgments
This research is partially supported by JST CREST Grant Number JPMJCR21F2.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
About this paper
Cite this paper
Yonekawa, K., Haruta, S., Konishi, T., Saito, K., Asoh, H., Kurokawa, M. (2022). A Study on Metrics for Concept Drift Detection Based on Predictions and Parameters of Ensemble Model. In: Hara, T., Yamaguchi, H. (eds) Mobile and Ubiquitous Systems: Computing, Networking and Services. MobiQuitous 2021. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 419. Springer, Cham. https://doi.org/10.1007/978-3-030-94822-1_37
Download citation
DOI: https://doi.org/10.1007/978-3-030-94822-1_37
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-94821-4
Online ISBN: 978-3-030-94822-1
eBook Packages: Computer ScienceComputer Science (R0)