Article

Dynamic syslog mining for network failure monitoring

Authors:

Kenji Yamanishi,

Yuko MaruyamaAuthors Info & Claims

KDD '05: Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining

Pages 499 - 508

https://doi.org/10.1145/1081870.1081927

Published: 21 August 2005 Publication History

Abstract

Syslog monitoring technologies have recently received vast attentions in the areas of network management and network monitoring. They are used to address a wide range of important issues including network failure symptom detection and event correlation discovery. Syslogs are intrinsically dynamic in the sense that they form a time series and that their behavior may change over time. This paper proposes a new methodology of dynamic syslog mining in order to detect failure symptoms with higher confidence and to discover sequential alarm patterns among computer devices. The key ideas of dynamic syslog mining are 1) to represent syslog behavior using a mixture of Hidden Markov Models, 2) to adaptively learn the model using an on-line discounting learning algorithm in combination with dynamic selection of the optimal number of mixture components, and 3) to give anomaly scores using universal test statistics with a dynamically optimized threshold. Using real syslog data we demonstrate the validity of our methodology in the scenarios of failure symptom detection, emerging pattern identification, and correlation discovery.

References

[1]

R. Agrawal and R. Srikant. Mining sequential patterns. In Proc. of the Eleventh International Conference on Data Engineering (ICDE95), pages 3--14, 1995.

Digital Library

[2]

L. E. Baum and T. Petrie and G. Soules and N. Weiss. A maximization technique occurring in the statistical analysis of probabilistic functions of Markov chains. The Annals of Statistics, 41(1):164--171,1970.

[3]

L. Burns and J. L. Hellerstein and S. Ma and C. S. Perng and D. A. Rabenhorst and D. Taylor. A systematic approach to discovering correlation rules for event management. In Proc. of IEEE/IFIP International Sysmposium on Integrated Network Management, 2001.

[4]

G. Jakobson and M. D. Weissman. Alarm correlation. IEEE Networks, 37:52--59, 1993.

Digital Library

[5]

S. E. Hansen and E. T. Atkins. Automated system monitoring and notification with swatch. In Proc. of USENIX Seventh System Administration Conference (LISA93), 1993.

Digital Library

[6]

M. Klemettinen and H. Mannila and H. Toivonen. Rule discovery in telecommunication alarm data. Journal of Network and Systems Management, 7(4): 395--423, 1999.

Digital Library

[7]

R. E. Krichevsky and V. K. Trofimov. The performance of universal encoding. IEEE Trans. on Inform. Theory, 27:199--207, 1981.

Digital Library

[8]

C. Lonvick. The BSD syslog protocol, RFC, 3164, 2001.

Digital Library

[9]

H. Mannila and H. Toivonen and A. I. Vernamo. Discovery of frequent episodes in event sequences. Data Mining and Knowledge Discovery, 1:259--289, 1997.

Digital Library

[10]

Y. Maruyama and K. Yamanishi. Dynamic model selection with its applications to computer security. In Proc. of 2004 IEEE International Workshop on Information Theory, 2004.

[11]

R. M. Neal and G. E. Hinton. A view of the EM algorithm that justifies incremental, sparse, and other variants. Learning in Graphical Models, M. Jordan (editor), MIT Press, Cambridge, MA, USA, pages 355--368, 1999.

Digital Library

[12]

C-S. Perng and D. Thoenen and G. Grabarnik and S. Ma and J. Hellerstein. Data-driven validation, completion and construction of event relationship networks. In Proc. of the Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining(KDD2003), pages 729--734, 2003.

Digital Library

[13]

J. Rissanen. Universal coding, information, prediction, and estimation. IEEE Trans. on Inform. Theory, 30:629--636, 1984.

Digital Library

[14]

P. Smyth. Markov monitoring with unknown states. IEEE Journal on Selected Areas in Communications (JSAC), Special Issue on Intelligent Signal Processing for Communications, 1994.

Digital Library

[15]

M. Steinder and A. Sethi. The present and future of event correlation: A need for end-to-end service fault localization. In Proc. of 2001 World Multi-Conference on Systemics, Cybernetics and Informatics, 2001.

[16]

M. Steinder and A. Sethi. Probabilistic fault localization in communication systems using belief networks. IEEE Trans. on Networking, 12(5):809--822, 2004.

Digital Library

[17]

R. Vaarandi. A data clustering algorithm for mining patterns from event logs. In Proc. of 2003 IEEE Workshop on IP Operations & Management (IPOM2003), 2003.

[18]

R. Vaarandi. Sec - a lightweight event correlation tool. In Proc. of 2002 IEEE Workshop on IP Operations & Management (IPOM2002), 2002.

[19]

A. J. Viterbi. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Trans. on Inform. Theory, IT-13:260--267, 1967.

Digital Library

[20]

K. Yamanishi and J. Takeuchi and G. Williams and P. Milne. On-line unsupervised oultlier detection using finite mixtures with discounting learning algorithms. In Proc. of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining(KDD2000), pages 320--324, ACM Press, 2000.

Digital Library

[21]

K. Yamanishi and J. Takeuchi. A unifying framework for detecting outliers and change-points from non-stationary time series data. In Proc. of the ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining(KDD2002), pages 676--681, ACM Press, 2002.

Digital Library

[22]

S. A. Yemini and S. Kliger and E. Mozes and Y. Yemini and D. Ohsie. High speed and robust event correlation. IEEE Communications Magazine, 34(5):82--90, 1996.

Digital Library

[23]

J. Ziv and A. Lempel. Compression of individual sequences via variable-rate coding. IEEE Trans. on Inform. Theory, IT-24:530--536, 1978.

Digital Library

[24]

J. Ziv. On classification with empirically observed statistics and universal data compression. IEEE Trans. on Inform. Theory, IT-34:278--286, 1988.

Digital Library

Cited By

Yang LChen JGao SGong ZZhang HKang YLi H(2024)Try with Simpler - An Evaluation of Improved Principal Component Analysis in Log-based Anomaly DetectionACM Transactions on Software Engineering and Methodology10.1145/364438633:5(1-27)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3644386
Xie YYang K(2024)Log Anomaly Detection by Adversarial Autoencoders With Graph Feature FusionIEEE Transactions on Reliability10.1109/TR.2023.330537673:1(637-649)Online publication date: Mar-2024
https://doi.org/10.1109/TR.2023.3305376
Xue KHan QHan SShi ZQiao Y(2024)A Review of Software Testing Process Log Parsing and Mining2024 IEEE International Conference on Software Services Engineering (SSE)10.1109/SSE62657.2024.00055(334-343)Online publication date: 7-Jul-2024
https://doi.org/10.1109/SSE62657.2024.00055
Show More Cited By

Index Terms

Dynamic syslog mining for network failure monitoring
1. Computing methodologies
  1. Artificial intelligence
    1. Philosophical/theoretical foundations of artificial intelligence

Recommendations

Failure Detection in Large-Scale Internet Services by Principal Subspace Mapping

Fast and accurate failure detection is becoming essential in managing large scale Internet services. This paper proposes a novel detection approach based on the subspace mapping between system inputs and internal measurements. By exploring these ...
Failure Detection and Randomization: A Hybrid Approach to Solve Consensus

We present a consensus algorithm that combines unreliable failure detection and randomization, two well-known techniques for solving consensus in asynchronous systems with crash failures. This hybrid algorithm combines advantages from both approaches:...
A Dynamic Failure Detector for P2P Storage System
NISS '09: Proceedings of the 2009 International Conference on New Trends in Information and Service Science

P2P storage systems have a lot of attractive advantages, such as self-organizing, scalability, robust and fault-tolerant. Unfortunately, there are some problems in real network conditions, especially, when considering asynchronous wide-area networks, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

KDD '05: Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining

August 2005

844 pages

ISBN:159593135X

DOI:10.1145/1081870

General Chair:
Robert Grossman
University of Illinois at Chicago & Open Data Partners, USA
,
Program Chairs:
Roberto Bayardo
IBM Almaden Research, USA
,
Kristin Bennett
RPI, USA

Copyright © 2005 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 August 2005

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

KDD05

Sponsor:

KDD05: The Eleventh ACM SIGKDD International Conference on Knowledge Discovery and Data Mining

August 21 - 24, 2005

Illinois, Chicago, USA

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '25

Sponsor:
sigkdd
sigkdd

The 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 3 - 7, 2025

Toronto , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

144
Total Citations
View Citations
1,758
Total Downloads

Downloads (Last 12 months)60
Downloads (Last 6 weeks)5

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yang LChen JGao SGong ZZhang HKang YLi H(2024)Try with Simpler - An Evaluation of Improved Principal Component Analysis in Log-based Anomaly DetectionACM Transactions on Software Engineering and Methodology10.1145/364438633:5(1-27)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3644386
Xie YYang K(2024)Log Anomaly Detection by Adversarial Autoencoders With Graph Feature FusionIEEE Transactions on Reliability10.1109/TR.2023.330537673:1(637-649)Online publication date: Mar-2024
https://doi.org/10.1109/TR.2023.3305376
Xue KHan QHan SShi ZQiao Y(2024)A Review of Software Testing Process Log Parsing and Mining2024 IEEE International Conference on Software Services Engineering (SSE)10.1109/SSE62657.2024.00055(334-343)Online publication date: 7-Jul-2024
https://doi.org/10.1109/SSE62657.2024.00055
Zhang YZhang WJiang XSun YFeng BXiong NWo T(2024)An Intelligent Secure Fault Classification and Identification Scheme for Mining Valuable Information in IIoTIEEE Systems Journal10.1109/JSYST.2024.343718518:3(1705-1716)Online publication date: Sep-2024
https://doi.org/10.1109/JSYST.2024.3437185
Wei Z(2024)Simulation of Artificial Intelligence Algorithm Based on Network Anomaly Detection and Wireless Sensor Network in Sports Cardiopulmonary Monitoring SystemMobile Networks and Applications10.1007/s11036-024-02409-6Online publication date: 28-Aug-2024
https://doi.org/10.1007/s11036-024-02409-6
Hadadi FDawes JShin DBianculli DBriand L(2024)Systematic Evaluation of Deep Learning Models for Log-based Failure PredictionEmpirical Software Engineering10.1007/s10664-024-10501-429:5Online publication date: 20-Jun-2024
https://dl.acm.org/doi/10.1007/s10664-024-10501-4
Chen XWang PChen JWang W(2023)AS-Parser: Log Parsing Based on Adaptive SegmentationProceedings of the ACM on Management of Data10.1145/36267191:4(1-26)Online publication date: 12-Dec-2023
https://dl.acm.org/doi/10.1145/3626719
Wijesinghe NHemmati H(2023)Log-based Anomaly Detection of Enterprise Software: An Empirical Study2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS)10.1109/QRS60937.2023.00012(12-23)Online publication date: 22-Oct-2023
https://doi.org/10.1109/QRS60937.2023.00012
Zhang DEgersdoerfer CMahmud TZheng MDai D(2023)Drill: Log-based Anomaly Detection for Large-scale Storage Systems Using Source Code Analysis2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS54959.2023.00028(189-199)Online publication date: May-2023
https://doi.org/10.1109/IPDPS54959.2023.00028
García BRicca Fdel Alamo JLeotta M(2023)Enhancing Web Applications Observability through Instrumented Automated BrowsersJournal of Systems and Software10.1016/j.jss.2023.111723203:COnline publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1016/j.jss.2023.111723
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten