Abstract
While batch machine learning algorithms typically assume that all the concepts are available at training, the reality is often different when dealing with continuous streams of data, where new concepts can emerge and existing ones change over time. The task of novelty detection is an increasingly popular field that tackles this problem by trying to recognize these formerly unidentified concepts that fall outside the decision boundary of the models. Although there have been numerous works discussing the implementation of such algorithms, studies covering their adequate performance evaluation are still scarce. In this paper, we present an evaluation framework that aims to streamline the evaluation of novelty detection algorithms. This framework irons out the shortcomings we identified in the domain, allowing us to obtain a more robust assessment of the performance. Specifically, we propose novel metrics to complement the existing ones, and we incorporate the temporal aspect of data streams within the evaluation. We empirically test the impact of intrinsic data streams’ characteristics when using our proposed framework. We show the added value of this novel framework with experiments carried out on both artificial and real-world data sets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Al-Khateeb, T., Masud, M.M., Khan, L., Aggarwal, C., Han, J., Thuraisingham, B.: Stream classification with recurring and novel class detection using class-based ensemble. In: 2012 IEEE 12th International Conference on Data Mining, pp. 31–40 (2012). https://doi.org/10.1109/ICDM.2012.125
Bifet, A., Holmes, G., Kirkby, R., Pfahringer, B.: MOA: massive online analysis. J. Mach. Learn. Res. 11, 1601–1604 (2010). https://doi.org/10.5555/1756006.1859903
Branco, P., Torgo, L., Ribeiro, R.P.: A survey of predictive modeling on imbalanced domains. ACM Comput. Surv. 49(2), 1–50 (2016). https://doi.org/10.1145/2907070
Dua, D., Graff, C.: UCI machine learning repository (2017). http://archive.ics.uci.edu/ml
Faria, E.R., Gonçalves, I.J.C.R., de Carvalho, A.C.P.L.F., Gama, J.: Novelty detection in data streams. Artif. Intell. Rev. 45(2), 235–269 (2015). https://doi.org/10.1007/s10462-015-9444-8
de Faria, E.R., Gonçalves, I.R., Gama, J., de Leon Ferreira Carvalho, A.C.P.: Evaluation of multiclass novelty detection algorithms for data streams. IEEE TKDE 27(11), 2961–2973 (2015). https://doi.org/10.1109/TKDE.2015.2441713
de Faria, E.R., Ponce de Leon Ferreira Carvalho, A.C., Gama, J.: MINAS: multiclass learning algorithm for novelty detection in data streams. Data Min. Knowl. Disc. 30(3), 640–680 (2015). https://doi.org/10.1007/s10618-015-0433-y
Gama, J.: Knowledge Discovery from Data Streams, 1st edn. Chapman & Hall/CRC (2010)
Haque, A., Khan, L., Baron, M.: SAND: semi-supervised adaptive novel class detection and classification over data stream. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 30, no. 1 (2016). https://doi.org/10.1609/aaai.v30i1.10283
Krawczyk, B., Woźniak, M.: Incremental learning and forgetting in one-class classifiers for data streams. In: Burduk, R., Jackowski, K., Kurzynski, M., Wozniak, M., Zolnierek, A. (eds.) CORES 2013, vol. 226, pp. 319–328. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-319-00969-8_31
Krawczyk, B., Woźniak, M.: One-class classifiers with incremental learning and forgetting for data streams with concept drift. Soft. Comput. 19(12), 3387–3400 (2014). https://doi.org/10.1007/s00500-014-1492-5
Masud, M., Gao, J., Khan, L., Han, J., Thuraisingham, B.M.: Classification and novel class detection in concept-drifting data streams under time constraints. IEEE TKDE 23(6), 859–874 (2011). https://doi.org/10.1109/TKDE.2010.61
Romano, S., Vinh, N.X., Bailey, J., Verspoor, K.: Adjusting for chance clustering comparison measures. JMLR 17(1), 4635–4666 (2016)
Stoica, P., Selen, Y.: Model-order selection: a review of information criterion rules. IEEE Sig. Process. Mag. 21(4), 36–47 (2004). https://doi.org/10.1109/MSP.2004.1311138
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Gaudreault, JG., Branco, P. (2023). Toward Streamlining the Evaluation of Novelty Detection in Data Streams. In: Bifet, A., Lorena, A.C., Ribeiro, R.P., Gama, J., Abreu, P.H. (eds) Discovery Science. DS 2023. Lecture Notes in Computer Science(), vol 14276. Springer, Cham. https://doi.org/10.1007/978-3-031-45275-8_47
Download citation
DOI: https://doi.org/10.1007/978-3-031-45275-8_47
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-45274-1
Online ISBN: 978-3-031-45275-8
eBook Packages: Computer ScienceComputer Science (R0)