Skip to main content

Advertisement

SpringerLink
Log in

Variational restricted Boltzmann machines to automated anomaly detection

  • S.I.: LSNC & OUAI
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Data-driven methods are implemented using particularly complex scenarios that reflect in-depth perennial knowledge and research. Hence, the available intelligent algorithms are completely dependent on the quality of the available data. This is not possible for real-time applications, due to the nature of the data and the computational cost that is required. This work introduces an Automatic Differentiation Variational Inference (ADVI) Restricted Boltzmann Machine (RBM) to perform real-time anomaly detection of industrial infrastructure. Using the ADVI methodology, local variables are automatically transformed into real coordinate space. This is an innovative algorithm that optimizes its parameters with mathematical methods by choosing an approach that is a function of the transformed variables. The ADVI RBM approach proposed herein identifies anomalies without the need for prior training and without the need to find a detailed solution, thus making the whole task computationally feasible.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Boubekeur M (2017) Industrial applications for cyber-physical systems. In: 2017 first international conference on embedded distributed systems (EDiS), pp 59–59. https://doi.org/10.1109/EDIS.2017.8284020

  2. Banafa A (2018) 2 The Industrial Internet of Things (IIoT): challenges, requirements and benefits. In: Secure and smart internet of things (IoT): using blockchain and AI, river publishers, pp 7–12. [Online]. https://ieeexplore.ieee.org/document/9226906. Accessed 19 Jan 2021

  3. Radanliev P et al (2020) Cyber risk at the edge: current and future trends on cyber risk analytics and artificial intelligence in the industrial internet of things and industry 4.0 supply chains. [Online]. https://www.preprints.org/manuscript/201903.0123/v2. Accessed 19 Jan 2021

  4. Demertzis K, Iliadis L, Tziritas N, Kikiras P (2020) Anomaly detection via blockchained deep learning smart contracts in industry 4.0. Neural Comput Appl 32(23):17361–17378. https://doi.org/10.1007/s00521-020-05189-8

    Article  Google Scholar 

  5. Chalapathy R, Chawla S (2019) Deep learning for anomaly detection: a survey. [Online]. Preprint http://arxiv.org/abs/1901.03407. Accessed 08 Sep 2021

  6. Tsiknas K, Taketzis D, Demertzis K, Skianis C (2021) Cyber threats to industrial IoT: a survey on attacks and countermeasures. IoT. https://doi.org/10.3390/iot2010009

    Article  Google Scholar 

  7. Deorankar AV, Thakare SS (2020) Survey on anomaly detection of (IoT)- Internet of Things cyberattacks using machine learning. In: 2020 fourth international conference on computing methodologies and communication (ICCMC), pp 115–117. https://doi.org/10.1109/ICCMC48092.2020.ICCMC-00023

  8. Pang G, Shen C, Cao L, van den Hengel A (2021) Deep learning for anomaly detection: a review. ACM Comput Surv 54(2):1–38. https://doi.org/10.1145/3439950

    Article  Google Scholar 

  9. Elmrabit N, Zhou F, Li F, Zhou H (2020) Evaluation of machine learning algorithms for anomaly detection. In: 2020 international conference on cyber security and protection of digital services (cyber security), pp 1–8. https://doi.org/10.1109/CyberSecurity49315.2020.9138871

  10. Al Jallad K, Aljnidi M, Desouki MS (2020) Anomaly detection optimization using big data and deep learning to reduce false-positive. J Big Data 7(1):68. https://doi.org/10.1186/s40537-020-00346-1

    Article  Google Scholar 

  11. Falco G, Caldera C, Shrobe H (2018) IIoT cybersecurity risk modeling for SCADA systems. IEEE Internet Things J 5(6):4486–4495. https://doi.org/10.1109/JIOT.2018.2822842

    Article  Google Scholar 

  12. Gawlikowski J, et al. (2021) A Survey of uncertainty in deep neural Networks. [Online]. Preprint http://arxiv.org/abs/2107.03342. Accessed 06 Nov 2021

  13. Demertzis K, Iliadis L, Kikiras P (2021) A lipschitz-shapley explainable defense methodology against adversarial attacks. In: Artificial intelligence applications and innovations. AIAI 2021 IFIP WG 12.5 international workshops, Cham, pp 211–227. https://doi.org/10.1007/978-3-030-79157-5_18

  14. Samrin R, Vasumathi D (2017) Review on anomaly based network intrusion detection system. In: 2017 international conference on electrical, electronics, communication, computer, and optimization techniques (ICEECCOT), pp 141–147. https://doi.org/10.1109/ICEECCOT.2017.8284655

  15. Dias R, Alexandre L, Mauricio F, Poggi M (2020) Toward an efficient real-time anomaly detection system for cloud datacenters. In: 2020 IFIP networking conference (networking), pp 529–533

  16. Feng T, Du Z, Sun Y, Wei J, Bi J, Liu J (2017) Real-time anomaly detection of short-time-scale GWAC survey light curves. In: 2017 IEEE international congress on big data (bigdata congress), pp 224–231. https://doi.org/10.1109/BigDataCongress.2017.38

  17. Demertzis K, Iliadis L, Spartalis S (2017) A spiking one-class anomaly detection framework for cyber-security on industrial control systems. In: Engineering applications of neural networks, Cham, pp 122–134. https://doi.org/10.1007/978-3-319-65172-9_11

  18. Kumar M, Mathur R (2014) Unsupervised outlier detection technique for intrusion detection in cloud computing. In: International conference for convergence for technology-2014, pp 1–4. https://doi.org/10.1109/I2CT.2014.7092027

  19. MohanaPriya P, Shalinie SM (2017) Restricted Boltzmann machine based detection system for ddos attack in software defined networks. In: 2017 fourth international conference on signal processing, communication and networking (ICSCN), pp 1–6. https://doi.org/10.1109/ICSCN.2017.8085731

  20. Xu X, Li J, Yang Y, Shen F (2021) Toward effective intrusion detection using log-cosh conditional variational autoencoder. IEEE Internet Things J 8(8):6187–6196. https://doi.org/10.1109/JIOT.2020.3034621

    Article  Google Scholar 

  21. Zavrak S, İskefiyeli M (2020) Anomaly-based intrusion detection from network flow features using variational autoencoder. IEEE Access 8:108346–108358. https://doi.org/10.1109/ACCESS.2020.3001350

    Article  Google Scholar 

  22. Doersch C (2021) Tutorial on variational autoencoders, [Online]. Preprint http://arxiv.org/abs/1606.05908. Accessed 09 Sep 2021

  23. Kim J, Sim A, Kim J, Wu K (2020) Botnet detection using recurrent variational autoencoder. In: GLOBECOM 2020-2020 IEEE global communications conference, pp 1–6. https://doi.org/10.1109/GLOBECOM42002.2020.9348169

  24. Gu Z, Yang Y (2021) Detecting malicious model updates from federated learning on conditional variational autoencoder. In: 2021 IEEE international parallel and distributed processing symposium (IPDPS), pp 671–680. https://doi.org/10.1109/IPDPS49936.2021.00075

  25. Farooq MJ, Zhu Q (2019) IoT supply chain security: overview, challenges, and the road ahead, [Online]. Preprint http://arxiv.org/abs/1908.07828. Accessed 19 Jan 2021

  26. Li H, Zhou S, Yuan W, Li J, Leung H (2020) Adversarial-Example attacks toward android malware detection system. IEEE Syst J 14(1):653–656. https://doi.org/10.1109/JSYST.2019.2906120

    Article  Google Scholar 

  27. Liu Y, Mao S, Mei X, Yang T, Zhao X (2019) Sensitivity of adversarial perturbation in fast gradient sign method. In: 2019 IEEE symposium series on computational intelligence (SSCI), pp 433–436. https://doi.org/10.1109/SSCI44817.2019.9002856

  28. Hwang W-S, Yun J-H, Kim J, Kim HC (2019) Time-series aware precision and recall for anomaly detection: considering variety of detection result and addressing ambiguous labeling. In: Proceedings of the 28th ACM international conference on information and knowledge management, New York, pp 2241–2244. https://doi.org/10.1145/3357384.3358118

  29. Tang P, Wang W, Lou J, Xiong L (2021) Generating adversarial examples with distance constrained adversarial imitation networks. IEEE Trans Dependable Secure Comput. https://doi.org/10.1109/TDSC.2021.3123586

    Article  Google Scholar 

  30. Yu P, Song K, Lu J (2018) Generating adversarial examples with conditional generative adversarial net. In: 2018 24th international conference on pattern recognition (ICPR), pp 676–681. https://doi.org/10.1109/ICPR.2018.8545152

  31. Han K, Li Y, Xia B (2021) A cascade model-aware generative adversarial example detection method. Tsinghua Sci Technol 26(6):800–812. https://doi.org/10.26599/TST.2020.9010038

    Article  Google Scholar 

  32. Hwang U, Park J, Jang H, Yoon S, Cho NI (2019) PuVAE: a variational autoencoder to purify adversarial examples. IEEE Access 7:126582–126593. https://doi.org/10.1109/ACCESS.2019.2939352

    Article  Google Scholar 

  33. Austin J, Kennedy J, Lees K (1995) A neural architecture for fast rule matching. In: Proceedings 1995 second new zealand international two-stream conference on artificial neural networks and expert systems, pp 255–260. https://doi.org/10.1109/ANNES.1995.499484

  34. Xing L, Demertzis K, Yang J (2020) Identifying data streams anomalies by evolving spiking restricted Boltzmann machines. Neural Comput Appl 32(11):6699–6713. https://doi.org/10.1007/s00521-019-04288-5

    Article  Google Scholar 

  35. Mrad AB, Delcroix V, Piechowiak S, Leicester P, Abid M (2015) An explication of uncertain evidence in bayesian networks: likelihood evidence and probabilistic evidence. Appl Intell 43(4):802–824. https://doi.org/10.1007/s10489-015-0678-6

    Article  Google Scholar 

  36. De La Rosa E, Yu W (2015) Restricted Boltzmann machine for nonlinear system modeling. In: 2015 IEEE 14th International conference on machine learning and applications (ICMLA), pp 443–446. https://doi.org/10.1109/ICMLA.2015.24

  37. Marlin B, Swersky K, Chen B, Freitas N (2010) Inductive principles for restricted Boltzmann machine learning. In: Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp 509–516. [Online]. https://proceedings.mlr.press/v9/marlin10a.html. Accessed 10 Sep 2021

  38. van de Schoot R et al (2021) Bayesian statistics and modelling. Nat Rev Methods Primer. https://doi.org/10.1038/s43586-020-00001-2

    Article  Google Scholar 

  39. Kingma DP, Welling M (2014) Auto-encoding variational bayes. [Online]. Preprint http://arxiv.org/abs/1312.6114. Accessed 09 Sep 2021

  40. Bengio Y, Courville A, Vincent P (2013) Representation learning: a review and new perspectives. IEEE Trans Pattern Anal Mach Intell 35(8):1798–1828. https://doi.org/10.1109/TPAMI.2013.50

    Article  Google Scholar 

  41. Ahmadlou M, Adeli H (2010) Enhanced probabilistic neural network with local decision circles: a robust classifier. Integr Comput Aided Eng 17(3):197–210

    Article  Google Scholar 

  42. Tversky A, Kahneman D (1973) Availability: a heuristic for judging frequency and probability. Cognit Psychol 5(2):207–232. https://doi.org/10.1016/0010-0285(73)90033-9

    Article  Google Scholar 

  43. Hastie T, Tibshirani R, Friedman J (2016) The elements of statistical learning: data mining, inference, and prediction, second edition, 2nd edn. Springer, New York

    MATH  Google Scholar 

  44. Barshan E, Fieguth P (2014) Scalable learning for restricted Boltzmann machines. In: 2014 IEEE international conference on image processing (ICIP), pp 2754–2758. https://doi.org/10.1109/ICIP.2014.7025557

  45. Chen L, Zou W (2018) Improvement of restricted Boltzmann machine by sparse representation based on lorentz function. In: 2018 7th international congress on advanced applied informatics (IIAI-AAI), pp 968–969. https://doi.org/10.1109/IIAI-AAI.2018.00205

  46. Yu J, Gwak J, Lee S, Jeon M (2015) An incremental learning approach for restricted Boltzmann machines. In 2015 International conference on control, automation and information sciences (ICCAIS), pp 113–117. https://doi.org/10.1109/ICCAIS.2015.7338643

  47. He T, Luo X, Liu Z (2017) A probabilistic indoor localization algorithm based on Restricted Boltzmann Machine. In 2017 IEEE 2nd advanced information technology, electronic and automation control conference (IAEAC), pp 1364–1368. https://doi.org/10.1109/IAEAC.2017.8054237

  48. Alam KMdR, Siddique N, Adeli H (2020) A dynamic ensemble learning algorithm for neural networks. Neural Comput Appl 32(12):8675–8690. https://doi.org/10.1007/s00521-019-04359-7

    Article  Google Scholar 

  49. Bishop CM (2006) Pattern recognition and machine learning. Springer, New York

    MATH  Google Scholar 

  50. Fortunato S (2010) Community detection in graphs. Phys Rep 486(3–5):75–174. https://doi.org/10.1016/j.physrep.2009.11.002

    Article  MathSciNet  Google Scholar 

  51. Lacasa L, Luque B, Ballesteros F, Luque J, Nuño JC (2008) From time series to complex networks: the visibility graph. Proc Natl Acad Sci 105(13):4972–4975. https://doi.org/10.1073/pnas.0709247105

    Article  MathSciNet  MATH  Google Scholar 

  52. Pollock JL (2007) Reasoning and probability. Law Probab Risk 6(1–4):43–58. https://doi.org/10.1093/lpr/mgm014

    Article  Google Scholar 

  53. Hamer D (2012) Probability, anti-resilience, and the weight of expectation. Law Probab Risk 11(2–3):135–158. https://doi.org/10.1093/lpr/mgs004

    Article  Google Scholar 

  54. Bengio Y, Courville A, Vincent P (2014) Representation Learning: a review and new perspectives. [Online]. Preprint http://arxiv.org/abs/1206.5538. Accessed 09 Sep 2021

  55. Kerschke P, Hoos HH, Neumann F, Trautmann H (2019) Automated algorithm selection: survey and perspectives. Evol Comput 27(1):3–45. https://doi.org/10.1162/evco_a_00242

    Article  Google Scholar 

  56. Calvayrac F (2015) Kullback-Leibler divergence as an estimate of reproducibility of numerical results. In: 2015 7th international conference on new technologies, mobility and security (NTMS), pp 1–5. https://doi.org/10.1109/NTMS.2015.7266501

  57. Liu J-W, Chi G-H, Luo X-L (2013) Contrastive divergence learning for the restricted Boltzmann machine. In: 2013 Ninth international conference on natural computation (ICNC), pp 18–22. https://doi.org/10.1109/ICNC.2013.6817936

  58. Yildirim I, Bayesian inference: gibbs sampling, p 6

  59. Zhang C, Butepage J, Kjellstrom H, Mandt S (2018) Advances in variational inference [Online]. Preprint http://arxiv.org/abs/1711.05597. Accessed 08 Sep 2021

  60. Gregor K, Papamakarios G, Besse F, Buesing L, Weber T (2019) Temporal difference variational auto-encoder. [Online]. Preprint http://arxiv.org/abs/1806.03107. Accessed 08 Sep 2021

  61. Shekhar S, Xiong H, Zhou X (eds) (2017) Euclidean distance. In: Encyclopedia of GIS, Springer, Cham, p 556. https://doi.org/10.1007/978-3-319-17885-1_100372

  62. Xue Y. Zhang L, Wang B, Li F (2019) feature selection based on the kullback-leibler distance and its application on fault diagnosis. In: 2019 seventh international conference on advanced cloud and big data (CBD), pp 246–251. https://doi.org/10.1109/CBD.2019.00052

  63. Kim DH, Baddar WJ, Jang J, Ro YM (2019) Multi-objective based spatio-temporal feature representation learning robust to expression intensity variations for facial expression recognition. IEEE Trans Affect Comput 10(2):223–236. https://doi.org/10.1109/TAFFC.2017.2695999

    Article  Google Scholar 

  64. Kucukelbir A, Tran D, Ranganath R, Gelman A, Blei DM (2016) Automatic differentiation variational inference. [Online]. Preprint http://arxiv.org/abs/1603.00788. Accessed 10 Sep 2021

  65. Shin H-K, Lee W, Yun J-H, Kim H (2020) {HAI} 1.0: HIL-based augmented {ICS} security dataset, presented at the 13th {USENIX} workshop on cyber security experimentation and test ({CSET} 20), [Online]. https://www.usenix.org/conference/cset20/presentation/shin. Accessed 10 Sep 2021

  66. Shin H-K, Lee W, Yun J-H, Min B-G (2021) Two ICS security datasets and anomaly detection contest on the HIL-based augmented ICS testbed. In: Cyber security experimentation and test workshop, Virtual CA USA, pp 36–40. https://doi.org/10.1145/3474718.3474719

  67. Choi S, Yun J-H, Kim S-K (2019) A comparison of ICS datasets for security research based on attack paths. In: Critical information infrastructures security, Springer, Cham, pp 154–166. https://doi.org/10.1007/978-3-030-05849-4_12

  68. Wood SN (2015) Core statistics. In: Cambridge core, https://www.cambridge.org/core/books/core-statistics/F303F4463E162C6534641616AE38C0A6 Accessed 29 Jun 2021

Download references

Author information

Authors and Affiliations

  1. School of Science and Technology, Informatics Studies, Hellenic Open University, Thermi, Greece

    Konstantinos Demertzis & Panagiotis Kikiras

  2. Computer Science and Creative Technologies, University of the West of England, Bristol, UK

    Lazaros Iliadis & Panagiotis Kikiras

  3. Department of Computer Science, University of Thessaly, 35100, Lamia, Greece

    Elias Pimenidis & Panagiotis Kikiras

Authors
  1. Konstantinos Demertzis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lazaros Iliadis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Elias Pimenidis
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Panagiotis Kikiras
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Konstantinos Demertzis.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Demertzis, K., Iliadis, L., Pimenidis, E. et al. Variational restricted Boltzmann machines to automated anomaly detection. Neural Comput & Applic 34, 15207–15220 (2022). https://doi.org/10.1007/s00521-022-07060-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07060-4

Keywords

Search

Navigation