Skip to main content
Springer Nature Link
Log in

Cyberattack detection model using deep learning in a network log system with data visualization

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Network log data is significant for network administrators, since it contains information on every event that occurs in a network, including system errors, alerts, and packets sending statuses. Effectively analyzing large volumes of diverse log data brings opportunities to identify issues before they become problems and to prevent future cyberattacks; however, processing of the diverse NetFlow data poses challenges such as volume, velocity, and veracity of log data. In this study, by means of Elasticsearch, Logstash, and Kibana, i.e., the ELK Stack, we construct an analysis and management system for network log data, which provides functions to filter, analyze, and display network log data for further applications and creates data visualization on a Web browser. In addition, an advanced cyberattack detection model is facilitated using deep neural network (DNN), recurrent neural networks (RNN), and long short-term memory (LSTM) approaches. By knowing cyberattack behaviors and cross-validating with the log analysis system, one can learn from this model the characteristics of a variety of cyberattacks. Finally, we also implement Grafana to perform metrics monitoring.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23

Similar content being viewed by others

References

  1. Appleyard RH, Adams J (2016) Using the ELK Stack for CASTOR Application Logging at RAL. In: Conference: International Symposium on Grids and Clouds 2015, (p 027). https://doi.org/10.22323/1.239.0027

  2. Betke E, Kunkel J (2017) Real-Time i/o-Monitoring of Hpc Applications with Siox, Elasticsearch, Grafana and Fuse. In: International Conference on High Performance Computing, Springer, pp 174–186

  3. Carela-Español V, Barlet-Ros P, Cabellos-Aparicio A, Solé-Pareta J (2011) Analysis of the impact of sampling on NetFlow traffic classification. Comput Netw 55:1083–1099

    Article  Google Scholar 

  4. Grafana Labs (2020) The analytics platform for all your metrics. https://grafana.com/, online; accessed 20 June 2019

  5. Kalech M (2019) Cyberattack detection in SCADA systems using temporal pattern recognition techniques. Comput Secur 84:225–238

    Article  Google Scholar 

  6. Kim TY, Cho SB (2018) Web traffic anomaly detection using C-LSTM neural networks. Expert Syst Appl 106:66–76

    Article  Google Scholar 

  7. Kiran M, Chhabra A (2019) Understanding flows in high-speed scientific networks: a netflow data study. Future Gener Comput Syst 94:72–79

    Article  Google Scholar 

  8. Kozik R (2018) Distributing extreme learning machines with apache spark for netflow-based malware activity detection. Pattern Recogn Lett 101:14–20

    Article  Google Scholar 

  9. Kozik R, Choraá M, Ficco M, Palmieri F (2018) A scalable distributed machine learning approach for attack detection in edge computing environments. J Parallel Distrib Comput 119:18–26

    Article  Google Scholar 

  10. Kristiani E, Yang CT, Huang CY, Ko PC, Fathoni H (2020) On construction of sensors, edge, and cloud (ISEC) framework for smart system integration and applications. IEEE Internet Things J 8(1):309–319

    Article  Google Scholar 

  11. Langi PP, Najib W, Aji TB, et al (2015) An Evaluation of Twitter River and Logstash Performances as Elasticsearch Inputs for Social Media Analysis of Twitter. In: 2015 International Conference on Information & Communication Technology and Systems (ICTS), IEEE, pp 181–186

  12. Lee S, Huh JH (2019) An effective security measures for nuclear power plant using big data analysis approach. J Supercomput 75(8):4267–4294

    Article  Google Scholar 

  13. Liu H, Lang B, Liu M, Yan H (2019) CNN and RNN based payload classification methods for attack detection. Knowl-Based Syst 163:332–341

    Article  Google Scholar 

  14. Mahmoud MS, Hamdan MM, Baroudi UA (2019) Modeling and control of cyber-physical systems subject to cyber attacks: a survey of recent advances and challenges. Neurocomputing 338:101–115

    Article  Google Scholar 

  15. Moh M, Pininti S, Doddapaneni S, Moh T (2016) Detecting Web Attacks Using Multi-Stage Log Analysis. In: 2016 IEEE 6th International Conference on Advanced Computing (IACC), pp 733–738, https://doi.org/10.1109/IACC.2016.141

  16. Navarro J, Deruyver A, Parrend P (2018) A systematic survey on multi-step attack detection. Comput Secur 76:214–249

    Article  Google Scholar 

  17. Perry I, Li L, Sweet C, Su SH, Cheng FY, Yang SJ, Okutan A (2018) Differentiating and Predicting Cyberattack Behaviors Using lstm. In: 2018 IEEE Conference on Dependable and Secure Computing (DSC), IEEE, pp 1–8

  18. Prakash T, Kakkar M, Patel K (2016) Geo-Identification of Web Users Through Logs Using Elk Stack. In: 2016 6th International Conference-Cloud System and Big Data Engineering (Confluence), IEEE, pp 606–610

  19. Rastogi R, Akash S, Shobha G, Poonam G, Pratiba D, Singh A (2016) Design and development of generic web based framework for log analysis. In: 2016 IEEE Region 10 Conference (TENCON), IEEE, pp 232–236

  20. Sahingoz OK, Buber E, Demir O, Diri B (2019) Machine learning based phishing detection from URLs. Expert Syst Appl 117:345–357

    Article  Google Scholar 

  21. Sahoo KS, Panda SK, Sahoo S, Sahoo B, Dash R (2019) Toward secure software-defined networks against distributed denial of service attack. J Supercomput 75(8):4829–4874

    Article  Google Scholar 

  22. Taylor A, Leblanc S, Japkowicz N (2018) Probing the limits of anomaly detectors for automobiles with a cyberattack framework. IEEE Intell Syst 33(2):54–62

    Article  Google Scholar 

  23. Tsung CK, Hsieh HY, Yang CT (2019) An implementation of scalable high throughput data platform for logging semiconductor testing results. IEEE Access 7:26,497-26,506

    Article  Google Scholar 

  24. Vinayakumar R, Alazab M, Soman K, Poornachandran P, Al-Nemrat A, Venkatraman S (2019) Deep learning approach for intelligent intrusion detection system. IEEE Access 7:41525–41550

    Article  Google Scholar 

  25. Wang CY, Ou CL, Zhang YE, Cho FM, Chen PH, Chang JB, Shieh CK (2018) BotCluster: a session-based P2P botnet clustering system on NetFlow. Comput Netw 145:175–189

    Article  Google Scholar 

  26. Wu P, Lu Z, Zhou Q, Lei Z, Li X, Qiu M, Hung PC (2019) Bigdata logs analysis based on seq2seq networks for cognitive internet of things. Future Gener Comput Syst 90:477–488

    Article  Google Scholar 

  27. Xue Q, Chuah MC (2018) New attacks on RNN based healthcare learning system and their detections. Smart Health 9–10:144–157

    Article  Google Scholar 

  28. Yang CT, Chen ST, Cheng WH, Chan YW, Kristiani E (2019a) A heterogeneous cloud storage platform with uniform data distribution by software-defined storage technologies. IEEE Access 7:147,672-147,682

    Article  Google Scholar 

  29. Yang CT, Chen ST, Liu JC, Yang YY, Mitra K, Ranjan R (2019b) Implementation of a real-time network traffic monitoring service with network functions virtualization. Future Gener Comput Syst 93:687–701

    Article  Google Scholar 

  30. Yang CT, Jiang WJ, Kristiani E, Chan YW, Liu JC (2019c) The Implementation of a Network Log System Using Rnn on Cyberattack Detection with Data Visualization. In: International Conference on Frontier Computing, Springer, pp 321–329

  31. Yang CT, Kristiani E, Wang YT, Min G, Lai CH, Jiang WJ (2020a) On construction of a network log management system using ELK Stack with Ceph. J Supercomput 76:6344–6360

    Article  Google Scholar 

  32. Yang CT, Liu JC, Kristiani E, Liu ML, You I, Pau G (2020b) Netflow monitoring and cyberattack detection using deep learning with Ceph. IEEE Access 8:7842–7850

    Article  Google Scholar 

  33. Yang Y, Zheng K, Wu C, Yang Y (2019d) Improving the classification effectiveness of intrusion detection by using improved conditional variational autoencoder and deep neural network. Sensors 19(11):2528

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the Ministry of Science and Technology, Taiwan, under Grant MOST 108-2622-E-029-007-CC3, 109-2625-M-029-001 and 109-2221-E-029-020.

Author information

Authors and Affiliations

  1. Department of Computer Science, Tunghai University, Taichung City, 407224, Taiwan, R.O.C.

    Jung-Chun Liu, Chao-Tung Yang & Wei-Je Jiang

  2. Research Center for Smart Sustainable Circular Economy, Tunghai University, No. 1727, Sec.4, Taiwan Boulevard, Taichung City, 407224, Taiwan, R.O.C.

    Chao-Tung Yang

  3. Research Center for Nanotechnology, Tunghai University, No. 1727, Sec.4, Taiwan Boulevard, Taichung City, 407224, Taiwan, R.O.C.

    Chao-Tung Yang

  4. College of Computing and Informatics, Providence University, Taichung City, 43301, Taiwan, R.O.C.

    Yu-Wei Chan

  5. Department of Industrial Engineering and Enterprise Information, Tunghai University, Taichung City, 407224, Taiwan, R.O.C.

    Endah Kristiani

  6. Department of Informatics, Krida Wacana Christian University, Jakarta, 11470, Indonesia

    Endah Kristiani

Authors
  1. Jung-Chun Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Chao-Tung Yang
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Yu-Wei Chan
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Endah Kristiani
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Wei-Je Jiang
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Chao-Tung Yang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, JC., Yang, CT., Chan, YW. et al. Cyberattack detection model using deep learning in a network log system with data visualization. J Supercomput 77, 10984–11003 (2021). https://doi.org/10.1007/s11227-021-03715-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-021-03715-6

Keywords