Skip to main content
SpringerLink
Log in

Dlog: diagnosing router events with syslogs for anomaly detection

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Router systems are notoriously difficult to understand or diagnose for their closure and heterogeneity. A common way of gaining insight into the router system and detecting the anomaly behaviors is to inspect the router syslogs. Unfortunately, syslogs are difficult to inspect because they are large-scale, unstructured and various in different vendors and services. Besides, they are too low-level to be directly used in anomaly detection. Prevalent approaches to understanding syslogs focus on simple keyword search (such as error and exception) of logs that may be associated with the failures. Such an approach is time consuming and error prone. In this paper, we present Dlog which can automatically transform and compress such low-level and minimally structured syslog messages into meaningful and prioritized high-level network events that can be used in anomaly detection. Dlog has two main steps: the first is the training process that learns the features of the normal and abnormal events; the second is anomaly detection and classification which can detect the anomalous events and provide the network operators with specific attack modes. We have applied our approach in a university network which contains Cisco, Huawei and Dlink routers for 5 months. We aligned our experiment with a former work as a baseline for comparison. Dlog is 23% faster in log template extraction and has improved the accuracy rate in template extraction 2 times higher than the former work. Besides, we can achieve 96% precision rate in anomaly detection and provide users with the attack modes in seven clusters.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Sipos R, Fradkin D, Moerchen F, Wang Z (2014) Log-based predictive maintenance. In: Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, pp 1867–1876

  2. Qiu T, Ge Z, Pei D, Wang J, Xu J (2010) What happened in my network: mining network events from router syslogs. In: Proceedings of the 10th ACM SIGCOMM Conference on Internet Measurement. ACM, pp 472–484

  3. Lin Q, Zhang H, Lou JG, Zhang Y, Chen X (2016) Log clustering based problem identification for online service systems. In: Proceedings of the 38th International Conference on Software Engineering Companion. ACM, pp 102–111

  4. Lou JG, Fu Q, Yang S, Xu Y, Li J (2010) Mining invariants from console logs for system problem detection. In: USENIX Annual Technical Conference

  5. Turner D, Levchenko K, Snoeren AC, Savage S (2010) California fault lines: understanding the causes and impact of network failures. In: ACM SIGCOMM Computer Communication Review, vol 40, no 4. ACM, pp 315–326

  6. Jiang W, Hu C, Pasupathy S, Kanevsky A, Li Z, Zhou Y (2009) Understanding customer problem troubleshooting from storage system logs. In: FAST, vol 9, pp 43–56

  7. Shang W, Jiang ZM, Hemmati H, Adams B, Hassan AE, Martin P (2013) Assisting developers of big data analytics applications when deploying on hadoop clouds. In: Proceedings of the 2013 International Conference on Software Engineering. IEEE Press, pp 402–411

  8. Wu Y, Zhao M, Haeberlen A, Zhou W, Loo BT (2014) Diagnosing missing events in distributed systems with negative provenance. In: ACM SIGCOMM Computer Communication Review, vol 44, no 4. ACM, pp 383–394

  9. Xu W, Huang L, Fox A, Patterson D, Jordan MI (2009) Detecting large-scale system problems by mining console logs. In: Proceedings of the 22nd ACM SIGOPS. ACM, pp 117–132

  10. Zhao X, Zhang Y, Lion D, Ullah MF, Luo Y, Yuan D, Stumm M (2014) lprof: a non-intrusive request flow profiler for distributed systems. In: OSDI, vol 14, pp 629–644

  11. Lin Z, Jiang X, Xu D, Zhang X (2008) Automatic protocol format reverse engineering through context-aware monitored execution. In: NDSS, vol 8, pp 1–15

  12. Wondracek G, Comparetti PM, Kruegel C, Kirda E, Anna SSS (2008) Automatic network protocol analysis. In: NDSS, vol 8, pp 1–14

  13. Potharaju R, Jain N, Nita-Rotaru C (2013) Juggling the jigsaw: towards automated problem inference from network trouble tickets. In: NSDI, pp 127–141

  14. Fu Q, Lou JG, Wang Y, Li J (2009) Execution anomaly detection in distributed systems through unstructured log analysis. In: Ninth IEEE International Conference on ICDM’09. IEEE, pp 149–158

  15. Beschastnikh I, Brun Y, Ernst MD, Krishnamurthy A (2014) Inferring models of concurrent systems from logs of their behavior with csight. In: Proceedings of the 36th International Conference on Software Engineering. ACM, pp 468–479

  16. Ya J, Liu T, Zhang H, Shi J, Guo L (2015) An automatic approach to extract the formats of network and security log messages. In: Military Communications Conference, MILCOM’15. IEEE, pp 1542–1547

  17. Liang C, Benson T, Kanuparthy P, He Y (2016) Finding needles in the haystack: harnessing syslogs for data center management. arXiv preprint arXiv:1605.06150

  18. Medem A, Akodjenou MI, Teixeira R (2009) Troubleminer: mining network trouble tickets. In: IFIP/IEEE International Symposium on Integrated Network Management-Workshops, IM’09. IEEE, pp. 113–119

  19. Wold S, Esbensen K, Geladi P (1987) Principal component analysis. Chemom Intell Lab Syst 2(1–3):37–52

    Article  Google Scholar 

  20. Xu W, Huang L, Fox A, Patterson D, Jordan M (2009) Online system problem detection by mining patterns of console logs. In: Ninth IEEE International Conference on ICDM’09. IEEE, pp 588–597

  21. Kimura T, Watanabe A, Toyono T, Ishibashi K (2015) Proactive failure detection learning generation patterns of large-scale network logs. In: Network and Service Management (CNSM’15). IEEE, pp 8–14

  22. Gerhards R (2009) The syslog protocol. http://www.rfc-base.org/rfc-5424.html

  23. Velmurugan T, Santhanam T (2010) Computational complexity between k-means and k-medoids clustering algorithms for normal and uniform distributions of data points. J Comput Sci 6(3):363

    Article  Google Scholar 

  24. Dlink, Firmware of dir-100. ftp://ftp.dlink.eu/Products/dir/dir-100/driver_software/DIR-100_fw_reva_113_ALL_en_20110915.zip

  25. Binwalk, Binwalk software. https://github.com/devttys0/binwalk

  26. Yamanishi K, Maruyama Y (2005) Dynamic syslog mining for network failure monitoring. In: Proceedings of the Eleventh ACM SIGKDD. ACM, pp 499–508

  27. Kimura T, Ishibashi K, Mori T, Sawada H, Toyono T, Nishimatsu K, Watanabe A, Shimoda A, Shiomoto K (2014) Spatio-temporal factorization of log data for understanding network events. In: INFOCOM, 2014 Proceedings IEEE. IEEE, pp 610–618

  28. rizhiyi, rizhiyi software. https://www.rizhiyi.com/

  29. OSSEC (2008) Ossec software. http://www.ossec.net

  30. netcool. http://www-01.ibm.com/software/tivoli/welcome/netcool

  31. lonix. https://www.emc.com/zh-cn/index.htm

  32. Haeberlen A, Kouznetsov P, Druschel P (2007) Peerreview: practical accountability for distributed systems. In: ACM SIGOPS, vol 41, no 6. ACM, pp 175–188

  33. Haeberlen A, Avramopoulos IC, Rexford J, Druschel P (2009) Netreview: detecting when interdomain routing goes wrong. In: NSDI, pp 437–452

  34. Wu Y, Haeberlen A, Zhou W, Loo BT (2013) Answering why-not queries in software-defined networks with negative provenance. In: Proceedings of the Twelfth ACM Workshop on Hot Topics in Networks. ACM, p 3

  35. Fukuda K (2011) On the use of weighted syslog time series for anomaly detection. In: 2011 IFIP/IEEE International Symposium on Integrated Network Management (IM). IEEE, pp 393–398

  36. Tan T, Gao S, Yang W, Song Y, Lin C (2016) Two new term weighting methods for router syslogs anomaly detection. In: HPCC’16/SmartCity’16/DSS’16. IEEE, pp 1454–1460

  37. Chuah E, Kuo SH, Hiew P, Tjhi WC, Lee G, Hammond J, Michalewicz MT, Hung T, Browne JC (2010) Diagnosing the root-causes of failures from cluster log files. In: High Performance Computing (HiPC). IEEE, pp 1–10

Download references

Acknowledgements

This work is supported by the National High Technology Research and Development Program (863 Program) of China (No. 2015AA017203), the National Natural Science Foundation of China (Nos. 61303033, 61502368, 61602537), the Key Program of NSFC (No. U1405255), the Natural Science Basis Research Plan in Shaanxi Province of China (No. 2016JM6034), China 111 Project (No. B16037), and the Special Research Foundation of MIIT (No. MJ-2014-S-37).

Author information

Authors and Affiliations

  1. School of Cyber Engineering, Xidian University, Xi’an, 710071, China

    Teng Li, JianFeng Ma & Cong Sun

Authors
  1. Teng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. JianFeng Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cong Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Teng Li.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, T., Ma, J. & Sun, C. Dlog: diagnosing router events with syslogs for anomaly detection. J Supercomput 74, 845–867 (2018). https://doi.org/10.1007/s11227-017-2165-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-017-2165-9

Keywords

Search

Navigation