Skip to main content
Springer Nature Link
Log in

On construction of a network log management system using ELK Stack with Ceph

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

A log management system is essential for the networks administrator. With a log management tool, we can collect, store, analyze, archive, and finally dispose of the log information. In this paper, we propose the architecture model of a log management system using ELK Stack with Ceph to provide a safe network, good Wi-Fi signal strength, and adequate backup data mechanism. In this case, we use our campus data of Wi-Fi log and NetFlow log. First, we collect and store data of our Wi-Fi log using Filebeats tool, and then, we use Elasticsearch, Logstash, and Kibana Stack to visualize the Wi-Fi log data. Second, we collect and store our NetFlow log using NFDUMP, and then, we also use ELK Stack to visualize the NetFlow log data. Third, we integrate the Wi-Fi log and NetFlow log data in one architecture using a distributed storage Ceph file system (CephFS). Moreover, we also compare the performance of RADOS Gateway and CephFS for better storage mechanism.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

References

  1. Rudd J, Sullivan P, King M, Bouchard F, Turner K, Olson M, Schroeder K, Kaplan A (2009) Education for a smarter planet: the future of learning. 2012-09-09. http://www.redbooks.ibm.com/redpapers/pdfs/redp4564.pdf. Accessed 29 Nov 2018

  2. Raghav R, Akash S, Shobha G, Poonam G, Pratiba D, Singh A (2016) Design and development of generic web based framework for log analysis. In: 2016 IEEE Region 10 Conference (TENCON). IEEE, pp 232–236

  3. Awad M, Menasc DA (2015) Automatic workload characterization using system log analysis. In: Computer Measurement Group Conference on Performance and Capacity, San Antonio, TX

  4. Kononenko O, Baysal O, Holmes R, Godfrey MW (2014) Mining modern repositories with elasticsearch. In: Proceedings of the 11th Working Conference on Mining Software Repositories. ACM, pp 328–331

  5. Yang C-T, Chen S-T, Den W, Wang Y-T, Kristiani E (2019) Implementation of an intelligent indoor environmental monitoring and management system in cloud. Fut Gener Comput Syst 96:731–749

    Article  Google Scholar 

  6. ELK Stack (2018) https://www.elastic.co/elk-stack

  7. The Complete Guide to the ELK Stack 2018, https://logz.io/learn/complete-guide-elk-stack/intro

  8. Gupta P, Nair S (2014) Survey paper on elastic search. Int J Sci Res (IJSR) 5(1):4

    Google Scholar 

  9. Sanjappa S, Ahmed M (2017) Analysis of logs by using logstash. In: Proceedings of the 5th International Conference on Frontiers in Intelligent Computing: Theory and Applications. Springer, Singapore, pp 579–585

  10. Chuvakin A, Schmidt K, Phillips C (2012) Logging and log management: the authoritative guide to understanding the concepts surrounding logging and log management. Newnes

  11. Gormley C, Tong Z (2015) Elasticsearch: the definitive guide: a distributed real-time search and analytics engine. O’Reilly Media Inc, Sebastopol

    Google Scholar 

  12. Nginx introduction (2018) http://www.webopedia.com/TERM/N/nginx.html

  13. https://www.elastic.co/products/beats/filebeat (2018)

  14. http://nfdump.sourceforge.net/ (2018)

  15. CEPH (2018) https://ceph.com/

  16. Yang C-T, Chen C-J, Chen T-Y (2017) Implementation of Ceph storage with big data for performance comparison. In: International Conference on Information Science and Applications. Springer, Singapore, pp 625–633

  17. Zhan K, Piao AH (2016) Optimization of Ceph reads/writes based on multi-threaded algorithms. In: 2016 IEEE 18th International Conference on High Performance Computing and Communications; IEEE 14th International Conference on Smart City; IEEE 2nd International Conference on Data Science and Systems (HPCC/SmartCity/DSS). IEEE, pp 719–725

  18. https://www.alluxio.org/ (2018)

  19. Nguyen VN, Tran VC (2016) An efficient log management system. VNU J Comput Sci Commun Eng 32(2):43–48

    Google Scholar 

  20. Kumar A, Bandyopadhyay A, Bhoomika H, Singhania I, Shah K (2018) Analysis of network traffic and security through log aggregation. Int J Comput Sci Inf Secur 16(6)

  21. Anastopoulos V, Katsikas S (2018) Design of a dynamic log management infrastructure using risk and affiliation network analysis. In: Proceedings of the 22nd Pan-Hellenic Conference on Informatics. ACM, pp 52–57

  22. Miao C, Wang J, Wang H, Zhang J, Zhou W, Liu S (2018) A multi-dimension measurement study of a large scale campus Wi-Fi network. In: 2018 IEEE 43rd Conference on Local Computer Networks (LCN). IEEE, pp 351–359

  23. Qu Z, Xie C, Liu C (2018) The study of mixed storage scheme of private cloud platform based on Ceph. In: 2018 International Conference on Network, Communication, Computer Engineering (NCCE 2018). Atlantis Press

  24. Dubey S, Balaii B, Rao D, Rao D (2018) Data visualization on GitHub repository parameters using elastic search and Kibana. In: 2018 2nd International Conference on Trends in Electronics and Informatics (ICOEI). IEEE, pp 554–558

  25. Kumar P, Kumar P, Zaidi N, Vijay SR (2018) Analysis and comparative exploration of elastic search, Mongodb and Hadoop big data processing. In: Soft computing: theories and applications. Springer, Singapore, pp 605–615

  26. Almohannadi H, Awan I, Al Hamar J, Cullen A, Disso JP, Armitage L (2018) Cyber threat intelligence from honeypot data using elasticsearch. In: 2018 IEEE 32nd International Conference on Advanced Information Networking and Applications (AINA). IEEE, pp 900–906

  27. Prakash T, Kakkar M, Patel K (2016) Geo-identification of web users through logs using ELK Stack. In: 2016 6th International Conference on Cloud System and Big Data Engineering (Confluence). IEEE, pp 606–610

  28. Elasticsearch guidance page, https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started.html (2018)

Download references

Acknowledgements

This work was supported in part by the Ministry of Science and Technology, Taiwan (ROC), under Grants Number 107-2221-E-029-008, 107-2218-E-029-003 and 106-3114-E-029-003.

Author information

Authors and Affiliations

  1. Department of Computer Science, Tunghai University, No. 1727, Sec. 4, Taiwan Boulevard, Xitun District, Taichung, 40704, Taiwan, ROC

    Chao-Tung Yang, Endah Kristiani, Ching-Han Lai & Wei-Je Jiang

  2. Department of Industrial Engineering and Enterprise Information, Tunghai University, No. 1727, Sec. 4, Taiwan Boulevard, Xitun District, Taichung, 40704, Taiwan, ROC

    Endah Kristiani

  3. Department of Informatics, Faculty of Engineering and Computer Science, Krida Wacana Christian University, Jakarta, 11470, Indonesia

    Endah Kristiani

  4. Cloud Computing Laboratory, Chunghwa Telecom Laboratories, No. 99, Dianyan Rd., Yangmei Dist., Taoyuan City, 326, Taiwan, ROC

    Yuan-Ting Wang

  5. Department of Mathematics and Computer Science, College of Engineering, Mathematics and Physical Science, University of Exeter, Exeter, EX4 4QF, UK

    Geyong Min

Authors
  1. Chao-Tung Yang
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Endah Kristiani
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Yuan-Ting Wang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Geyong Min
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Ching-Han Lai
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Wei-Je Jiang
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Chao-Tung Yang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, CT., Kristiani, E., Wang, YT. et al. On construction of a network log management system using ELK Stack with Ceph. J Supercomput 76, 6344–6360 (2020). https://doi.org/10.1007/s11227-019-02853-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-019-02853-2

Keywords