Abstract
Data stream applications in highly dynamic environments often face concept drift problems, a phenomenon in which the statistical properties of the variables change over time, which can degrade the performance of Machine Learning models. This work presents a new model monitoring tool through the use of Meta Learning. The algorithm was conceived for data streams with concept drift and large target arrival delay. Additionally, a new set of Meta Features is proposed based on the use of unsupervised concept drift metrics. Unlike related Meta Learning approaches, a regressor was used at the meta level to predict the predictive performance of the base model. These predictions can be used to generate concept drift alerts before the arriving objects are labelled. Experimental results show that the proposed approach obtains, on average, a classification error reduction of 12.8%, when compared to the traditional Meta Learning approaches, and 38%, when compared to the baseline, the last known performance, in predicting the performance of the base model.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bashir, S.A., Petrovski, A., Doolan, D.: A framework for unsupervised change detection in activity recognition. Int. J. Pervasive Comput. Commun. 13(2), 157–175 (2017)
Brazdil, P.: Metalearning: Applications to Automated Machine Learning and Data Mining. Springer, Cham (2022). https://doi.org/10.1007/978-3-030-67024-5. Cognitive Technologies
Brazdil, P., Giraud-Carrier, C.: Metalearning and algorithm selection: progress, state of the art and introduction to the 2018 special issue. Mach. Learn. 107, 12 (2017)
Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Measur. 20(1), 37–46 (1960). April
Jáder M. C. de Sá, Andre, L.D., Rossi, Gustavo E. A. P. A. Batista, and Luís P. F. Garcia. Algorithm recommendation for data streams. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 6073–6080, 2021
Gama, J.: A survey on learning from data streams: current and future trends. Prog. Artif. Intell. 1(1), 45–55 (2012). January
Gama, J., Medas, P., Castillo, G., Rodrigues, P.: Learning with drift detection. In: Bazzan, A.L.C., Labidi, S. (eds.) SBIA 2004. LNCS (LNAI), vol. 3171, pp. 286–295. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-28645-5_29
Gemaque, R.N., Costa, A.F.J., Giusti, R., Dos Santos, E.M.: An overview of unsupervised drift detection methods. Wiley Interdisc. Rev. Data Min. Knowl. Discov. 10, e1381 (2020)
Gemaque, R.N., Costa, A.F.J., Giusti, R., Dos Santos, E.M.: An overview of unsupervised drift detection methods. WIREs Data Min. Knowl. Discov. 10(6), e1381 (2020)
Janardan and Shikha Mehta: Concept drift in streaming data classification: algorithms, platforms and issues. Procedia Comput. Sci. 122, 804–811 (2017)
Kadam, S.V.: A Survey on Classification of Concept Drift with Stream Data. working paper or preprint (2019)
Ke, G., et al.: Lightgbm: a highly efficient gradient boosting decision tree. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS 2017, pp. 3149–3157, Red Hook, NY, USA, Curran Associates Inc (2017)
Maletzke, A.G., dos Reis, D.M., Batista, G.E.: Combining instance selection and self-training to improve data stream quantification. J. Braz. Comput. Soc. 24(1), 1–17 (2018)
Mouss, H., Mouss, D., Mouss, N., Sefouhi, L.: Test of page-hinckley, an approach for fault detection in an agro-alimentary production system, vol. 2, no. 08, pp. 815–818 (2004)
Nguyen, H.-L., Woon, Y.-K., Ng, W.-K.: A survey on data stream clustering and classification. Knowl. Inf. Syst. 45(3), 535–569 (2014). December
du Pisanie, J., Allison, J.S., Visagie, J.: A proposed simulation technique for population stability testing in credit risk scorecards. Mathematics 11(2), 492 (2022)
Rice, J.R.: The algorithm selection problem**this work was partially supported by the national science foundation through grant gp-32940x. this chapter was presented as the george e. forsythe memorial lecture at the computer science conference, february 19, 1975, washington, d. c. volume 15 of Advances in Computers, pp. 65–118. Elsevier (1976)
Rivolli, A., Garcia, L.P., Soares, C., Vanschoren, J., de Carvalho, A.C.: Meta-features for meta-learning. Knowl. Based Syst. 240, 108101 (2022)
Rossi, A.L.D., de Leon Ferreira, A.C.P., Soares, C., De Souza, B.F.: MetaStream: a meta-learning based method for periodic algorithm selection in time-changing data. Neurocomputing, 127, 52–64 (2014)
Rossi, A.L.D., De Souza, B.F., Soares, C., de Leon Ferreira de Carvalho, A., Ponce, C.: A guidance of data stream characterization for meta-learning. Intell. Data Anal. 21(4), 1015–1035 2017
Rossi, A.L.D., Soares, C., de Souza, B.F., de Leon Ferreira, A.C.P.: Algorithm selection for time-changing data. Inf. Sci. 565, 262–277 (2021)
Tegjyot Singh Sethi and Mehmed Kantardzic: Don’t pay for validation: detecting drifts from unlabeled data using margin density. Procedia Comput. Sci. 53, 103–112 (2015)
Siddiqi, N.: Credit Risk Scorecards: Developing and Implementing Intelligent Credit Scoring. Wiley, Hopboken (2005)
Maggio, D.P.S.: A Primer on Data Drift & Drift Detection Techniques. Dataiku (2020)
Smith-Miles, K.A.: Cross-disciplinary perspectives on meta-learning for algorithm selection. ACM Comput. Surv. 41(1), 1–25 (2009)
Souza, V.M.A., Reis, D.M., Maletzke, A.G., Batista, G.E.A.P.A.: Challenges in benchmarking stream learning algorithms with real-world data. Data Min. Knowl. Disc. 34, 1805–1858 (2020)
Tsymbal, A.: The problem of concept drift: definitions and related work (2004)
Vanschoren, J., Soares, C., Brazdil, P., Kotthoff, L.: Meta-Learning and Algorithm Selection (2014)
Verma, D., Nashine, R.: Data mining: next generation challenges and futureDirections. Int. J. Model. Optim. 2(5), 603 (2012)
Vilalta, R., Giraud-Carrier, C., Brazdil, P.: Meta-learning - concepts and techniques. In: Maimon, O., Rokach, L. (eds.) Data Mining and Knowledge Discovery Handbook, pp. 717–731. Springer, Boston (2009). https://doi.org/10.1007/978-0-387-09823-4_36
Wolpert, D.H., Macready, W.G.: No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1(1), 67–82 (1997)
Žliobaitė, I., Pechenizkiy, M., Gama, J.: An overview of concept drift applications. In: Japkowicz, N., Stefanowski, J. (eds.) Big Data Analysis: New Algorithms for a New Society. SBD, vol. 16, pp. 91–114. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-26989-4_4
Acknowledgment
This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brasil (CAPES) - Finance Code 001.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Melo, F.A., de Carvalho, A.C.P.L.F., Lorena, A.C., Garcia, L.P.F. (2023). Model Performance Prediction: A Meta-Learning Approach for Concept Drift Detection. In: García Bringas, P., et al. Hybrid Artificial Intelligent Systems. HAIS 2023. Lecture Notes in Computer Science(), vol 14001. Springer, Cham. https://doi.org/10.1007/978-3-031-40725-3_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-40725-3_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-40724-6
Online ISBN: 978-3-031-40725-3
eBook Packages: Computer ScienceComputer Science (R0)