Abstract
In this survey paper, we report the results of a comprehensive study involving the application of dynamic self-organizing neural networks (SONNs) to the problem of novelty detection in time series data. The study is comprised of three main parts. In the first part, we aim at evaluating how the performances of nonrecurrent dynamic SONNs are influenced by the introduction of different short-term memory kernels, such as Gamma, Gamma II and Laguerre, in the network input. In the second part, we analyze the performances of recurrent dynamic SONNs with the goal of inferring if they possess or not any competitive advantage over nonrecurrent architectures for novelty detection in time series. Finally, in the third part of the study, we introduce an alternative approach for dynamic SONN-based novelty detection by revisiting the operator map framework introduced by Lampinen and Oja (Proceedings of the 6th Scandinavian conference on image analysis (SCIA’89), pp 120–127, 1989) and Lehtimäki et al. (Proceedings of the joint international conference on artificial neural networks and neural information processing (ICANN/ICONIP’2003), pp 622–629, 2003). This framework allows the design of dynamic SONNs whose neurons are regarded as adaptive local linear models. In this case, novel/abnormal patterns are detected based on the statistics of prediction errors of the local models. A comprehensive performance comparison involving several nonrecurrent and recurrent dynamic SONN architectures is carried out using both synthetic and real-world time series data.
Similar content being viewed by others
Notes
Also known as Shewhart charts or process-behavior charts.
Difference between the observed and the predicted value of the variable of interest.
By definition, the (unilateral) Z-transform of a discrete-time signal \(\{x(n)\}_{n=0}^{\infty }\) is defined as \(X(z) = \mathcal {Z}\{x(n)\} = \sum _{n=0}^{\infty } x(n)z^{-n}\), where n is an integer and \(z=Ae^{j\phi }=A(\cos \phi + j\sin \phi )\). In words, the Z-transform converts a time series, which is a sequence of real (or complex) numbers, into a complex frequency domain representation.
Roots of the polynomial in the denominator of \(G^0(z)\) and G(z). For Eq. (37), the pole is found by doing \(z-(1-\lambda )=0\).
For the remaining parts of this paper, the label NORMAL will be used for known or ordinary patterns; and ABNORMAL for the novel, anomalous or unknown ones.
The percentile of a distribution of values is a number \(N_{\alpha }\) such that a percentage \(100(1-\alpha )\) of the sample values are less than or equal to \(N_{\alpha }\).
References
Alhoniemi E, Hollmén J, Simula O, Vesanto J (1999) Process monitoring and modeling using the self-organizing map. Integr Comput Aided Eng 6(1):3–14
Alippi C, Roveri M, Trovò F (2014) A self-building and cluster-based cognitive fault diagnosis system for sensor networks. IEEE Trans Neural Netw Learn Syst 25(6):1021–1032
Barreto G, Mota J, Souza L, Frota R (2004) Nonstationary time series prediction using local models based on competitive neural networks. In: Orchard B, Yang C, Ali M (eds) Innovations in applied artificial intelligence. Lecture notes in computer science, vol 3029. Springer, Berlin, pp 1146–1155
Barreto GA, Aguayo L (2009) Time series clustering for anomaly detection using competitive neural networks. In: Principe JC, Miikkulainen R (eds) Advances in self-organizing maps, vol LNCS-5629. Springer, New York, pp 28–36
Barreto GA, Frota RA (2013) A unifying methodology for the evaluation of neural network models on novelty detection tasks. Pattern Anal Appl 16(1):83–97
Barreto GA, Mota JCM, Souza LGM, Frota RA, Aguayo L (2005) Condition monitoring of 3G cellular networks through competitive neural models. IEEE Trans Neural Netw 16(5):1064–1075
Brighenti C, Sanz-Bobi MA (2011) Auto-regressive processes explained by self-organized maps: application to the detection of abnormal behavior in industrial processes. IEEE Trans Neural Netw 22(12):2078–2090
Carpenter GA, Grossberg S (2003) Adaptive resonance theory. In: Arbib MA (ed) The handbook of brain theory and neural networks. MIT Press, Cambridge, pp 87–90
Carpenter GA, Grossberg S, Rosen DB (1991) Fuzzy ART: fast stable learning and categorization of analog patterns by an adaptive resonance system. Neural Netw 4(6):759–771
Carpinteiro OAS, Lima I, Moreira EM, Pinheiro CAM, Seraphim E, Pinto JVL (2009) A hierarchical hybrid neural model with time integrators in long-term load forecasting. Neural Comput Appl 18(8):1057–1063
Carvalho AX, Tanner MA (2005) Mixtures-of-experts of autoregressive time series: asymptotic normality and model specification. IEEE Trans Neural Netw 16(1):39–56
Chappell GJ, Taylor JG (1993) The temporal Kohonen map. Neural Netw 6(3):441–445
Chawla NV (2005) Data mining for imbalanced datasets: an overview. In: Maimon O, Rokach L (eds) Data mining and knowledge discovery handbook. Springer, New York, pp 853–867
Clifton DA, Clifton LA, Bannister PR, Tarassenko L (2008) Automated novelty detection in industrial systems. In: Liu Y, Sun A, Loh HT, Lu WF, Lim EP (eds) Advances of computational intelligence in industrial systems. Studies in computational intelligence, vol 116. Springer, Berlin, pp 269–296
Coelho DN, Barreto GA, Medeiros CMS, Santos JDA (2014) Performance comparison of classifiers in the detection of short circuit incipient fault in a three-phase induction motor. In: Proceedings of the 2014 IEEE symposium on computational intelligence for engineering solutions (CIES’2014). Orlando, pp 42–48
Côme E, Cottrell M, Verleysen M, Lacaille J (2010) Aircraft engine health monitoring using self-organizing maps. In: Advances in data mining—applications and theoretical aspects. Springer, vol LNCS-6171, pp 405–417
Dablemont S, Simon G, Lendasse A, Ruttiens A, Blayo F, Verleysen M (2003) Time series forecasting with SOM and local non-linear models—application to the DAX30 index prediction. In: Proceedings of the 4th workshop on self-organizing maps (WSOM’2003), pp 340–345
Estevez PA, Hernandez R (2009) Gamma SOM for temporal sequence processing. In: Proceedings of 7th workshop on self-organizing maps (WSOM’09). Springer, vol LNCS-5629, pp 63–71
Estevez PA, Hernandez R, Perez C, Held C (2011) Gamma-filter self-organising neural networks for unsupervised sequence processing. Electron Lett 47(8):494–496
Fawcett T (2006) An introduction to ROC analysis. Pattern Recogn Lett 27(8):861–874
Fink O, Zio E, Weidmann U (2015) Novelty detection by multivariate kernel density estimation and growing neural gas algorithm. Mech Syst Signal Process 50–51:427–436
Fritzke B (1994) A growing neural gas network learns topologies. In: Tesauro G, Touretzky DS, Leen TK (eds) Advances in neural information processing systems (NIPS), Denver, Colorado. MIT Press, Cambridge, MA, pp 625–632
Fu TC (2011) A review on time series data mining. Eng Appl Artif Intell 37:164–181
Galar M, Fernández A, Barrenechea E, Bustince H, Herrera F (2011) A review on ensembles for the class imbalance problem: bagging-, boosting-, and hybrid-based approaches. IEEE Trans Syst Man Cybern C-42(4):463–484
Gan M, Peng H, Peng X, Chen X, Inoussa G (2010) A locally linear RBF network-based state-dependent AR model for nonlinear time series modeling. Inf Sci 180(22):4370–4383
Gollamudi S, Nagaray S, Kapoor S, Huang Y (1998) Set-membership filtering and a set-membership normalized LMS algorithm with an adaptive step size. IEEE Signal Process Lett 5:111–114
Granger E, Grossberg S, Rubin MA, Streilein WW (1999) Familiarity discrimination of radar pulses. In: Kearns MJ, Solla SA, Cohn DA (eds) Advances in neural information processing systems, vol 11. MIT Press, Cambridge, pp 875–881
Hampshire JB II, Weibel A (1992) The Meta-Pi network: building distributed knowledge representations for robust multisource pattern recognition. IEEE Trans Mach Intell Pattern Anal 14(7):751–769
Hodge VJ, Austin J (2004) A survey of outlier detection methodologies. Artif Intell Rev 22(2):85–126
Jacobs RA, Jordan MI, Nowlan SJ, Hinton GE (1991) Adaptive mixtures of local experts. Neural Comput 3(1):79–87
Jamsa-Jounela SL, Vermasvuori M, Enden P, Haavisto S (2003) A process monitoring system based on the kohonen self-organizing maps. Control Eng Pract 11(1):83–92
Kangas JA, Kohonen TK, Laaksonen J (1990) Variants of self-organizing maps. IEEE Trans Neural Netw 1(1):93–99
Kantz H, Schreiber T (1997) Nonlinear time series analysis. Cambridge University Press, Cambridge
Kohonen T (2013) Essentials of the self-organizing map. Neural Netw 37:52–65
Kohonen TK (1990) The self-organizing map. Proc IEEE 78(9):1464–1480
Koskela T, Varsta M, Heikkonen J, Kaski K (1998) Time series prediction using recurrent SOM with local linear models. Int J Knowl Based Intell Eng Syst 2(1):60–68
Lampinen J, Oja E (1989) Self-organizing maps for spatial and temporal AR models. In: Proceedings of the 6th Scandinavian conference on image analysis (SCIA’89). Helsinki, Finland, pp 120–127
Lee HJ, Cho S (2005) SOM-based novelty detection using novel data. In: Gallagher M, Hogan JP, Maire F (eds) Intelligent data engineering and automated learning (IDEAL’2005). Lecture notes in computer science, vol 3578. Springer, New York, pp 359–366
Lehtimäki P, Raivio K, Simula O (2003) Self-organizing operator maps in complex system analysis. In: Proceedings of the joint international conference on artificial neural networks and neural information processing (ICANN/ICONIP’2003). Istanbul, Turkey, pp 622–629
Liao TW (2007) A clustering procedure for exploratory mining of vector time series. Pattern Recogn 40:2250–2562
Liao Y, Vemuri R, Pasos A (2007) Adaptive anomaly detection with evolving connectionist systems. J Netw Comput Appl 30:60–80
Markou M, Singh S (2003) Novelty detection: a review—part 1: statistical approaches. Sig Process 83:2481–2497
Martínez-Rego D, Fontenla-Romero O, Alonso-Betanzos A (2011) Efficiency of local models ensembles for time series prediction. Expert Syst Appl 38(6):6884–6894
Mounce SR, Mounce RB, Boxall J (2011) Novelty detection for time series data analysis in water distribution systems using support vector machines. J HydroInform 13(4):672–686
Nelson LS (1984) Technical aids. J Qual Technol 16(4):238–239
Neme A, Lugo B, Cervera A (2011) Authorship attribution as a case of anomaly detection: a neural network model. Int J Hybrid Intell Syst 8(4):225–235
Neme A, Pulido J, Muñoz A, Hernández S, Dey T (2015) Stylistics analysis and authorship attribution algorithms based on self-organizing maps. Neurocomputing 147:147–159
Ni H, Yin H (2009) A self-organising mixture autoregressive network for FX time series modelling and prediction. Neurocomputing 72:3529–3537
Ni H, Yin H (2014) A neural gas mixture autoregressive network for modelling and forecasting FX time series. Neurocomputing 135:171–179
Oakland JS (2007) Statistical process control, 6th edn. Routledge, London
Oliveira ALI, Meira SRL (2006) Detecting novelties in time series through neural networks forecasting with robust confidence intervals. Neurocomputing 70(1–3):79–92
Oliveira ALI, Lima Neto FB, Meira SRL (2003) Novelty detection for short time series with neural networks. In: Abraham A, Köppen M, Franke K (eds) Design and application of hybrid intelligent systems. IOS Press, pp 66–75
Pimentel MAF, Clifton DA, Clifton L, Tarassenko L (2014) A review of novelty detection. Sig Process 99:215–249
Principe JC, de Vries B, de Oliveira PG (1994) An analysis of the gamma memory in dynamic neural networks. IEEE Trans Neural Netw 5(2):331–337
Principe JC, de Vries B, Oliveira PG (1993) The gamma filter—a new class of adaptive IIR filters with restricted feedback. IEEE Trans Signal Process 41(2):649–656
Sarasamma ST, Zhu QA (2006) Min-max hyperellipsoidal clustering for anomaly detection in network security. IEEE Trans Syst Man Cybern 36(4):887–901
Singh S, Markou M (2004) An approach to novelty detection applied to the classification of image regions. IEEE Trans Knowl Data Eng 16(4):1041–4347
Strickert M, Hammer B (2005) Merge SOM for temporal data. Neurocomputing 64:39–71
Voegtlin T (2002) Recursive self-organizing maps. Neural Netw 15:979–991
Wälchli M, Braun T (2009) Efficient signal processing and anomaly detection in wireless sensor networks. In: Giacobini M, Brabazon A, Cagnoni S, Di Caro GA, Ekárt A, Esparcia-Alcázar AI, Farooq M, Fink A, Machado P (eds) Applications of evolutionary computing. Lecture notes in computer science, vol 5484. Springer, Berlin, pp 81–86
Weigend AS, Gershenfeld NA (1994) Time series prediction: forecasting the future and understanding the past. Perseus Book, New York
Weigend AS, Mangeas M, Srivastana AN (1995) Nonlinear gated experts for time series: discovering regimes and avoiding overfitting. Int J Neural Syst 6(4):373–399
Werner S, Diniz PSR (2001) Set-membership affine projection algorithm. IEEE Signal Process Lett 8(8):231–235
Wong CS, Li WK (2001) On a logistic mixture autoregressive model. Biometrika 88(3):833–846
Wong M, Jack L, Nandi A (2006) Modified self-organising map for automated novelty detection applied to vibration signal monitoring. Mech Syst Signal Process 20(3):593–610
Yu Y, Zhu Y, Li S, Wan D (2014) Time series outlier detection based on sliding window prediction. Math Probl Eng 2014(Article ID 879736):1–14. doi:10.1155/2014/879736
Acknowledgements
The first author thanks the support from CAPES for this work via a PNPD (National Program of Post-Doctorate) grant. The second author thanks CNPq for the Grant Number 309451/2015-9.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Aguayo, L., Barreto, G.A. Novelty Detection in Time Series Using Self-Organizing Neural Networks: A Comprehensive Evaluation. Neural Process Lett 47, 717–744 (2018). https://doi.org/10.1007/s11063-017-9679-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11063-017-9679-2