research-article

OLMPT: Research on Online Log Parsing Method Based on Prefix Tree

Authors:

Pengyu Wen,

Zhizhong Zhang,

Bingguang DengAuthors Info & Claims

ICITEE '20: Proceedings of the 3rd International Conference on Information Technologies and Electrical Engineering

Pages 55 - 59

https://doi.org/10.1145/3452940.3452951

Published: 17 May 2021 Publication History

Get Access

Abstract

With the formal commercial use of the fifth generation mobile communication technology (5G) in China, the Internet of Things (IoT) industry has entered a period of innovation and development. The Internet of Things provides an open sharing platform for a large number of terminal devices. However, these devices will generate a large number of semi-structured log, it is urgent to automatic parsing. Parsing unstructured logs in different formats into structural templates is the first for later parsing. An online log parsing method based on prefix tree is proposed in this paper. Firstly, we extract the first character of the log template to form a string. Secondly, we structure the parse tree to obtain the string of the new log message. Thirdly, we compare with the string in the log template to dynamical update the similarity threshold. Finally, we update the parse tree in real time.

References

[1]

IEEE Draft Standard for an Architectural Framework for the Internet of Things (IoT)." IEEE P2413/D0.4.5, 2018, 12(24), 1--264.

Google Scholar

[2]

Hoffmann J. B, Heimes P, Senel S. IoT Platforms for the Internet of Production. IEEE Internet of Things Journal, 2019, 6(3): 4098--4105.

Crossref

Google Scholar

[3]

Cui L, Yang S, Chen Z, et al. A Decentralized and Trusted Edge Computing Platform for Internet of Things. IEEE Internet of Things Journal, 2020, 7(5): 3910--392

Crossref

Google Scholar

[4]

He P, Zhu J, He S, et al. Towards Automated Log Parsing for Large-Scale Log Data Analysis. IEEE Transactions on Dependable and Secure Computing, 2018, 15(6): 931--944.

Crossref

Google Scholar

[5]

He P, Zhu J, Zheng Z. Drain: An Online Log Parsing Approach with Fixed Depth Tree. 2017 IEEE International Conference on Web Services (ICWS), Honolulu, HI, 2017, 33--40.

Crossref

Google Scholar

[6]

Makanju A, Zincir-Heywood A. N, Milios E E. A Lightweight Algorithm for Message Type Extraction in System Application Logs. IEEE Transactions on Knowledge and Data Engineering, 2012, 24(11): 1921--1936.

Digital Library

Google Scholar

[7]

Du M, Li F. Spell: Streaming Parsing of System Event Logs. 2016 IEEE 16th International Conference on Data Mining (ICDM), Barcelona, 2016, 859--864.

Crossref

Google Scholar

[8]

Nagappan M, Vouk M A. Abstracting log lines to log event types for mining software system logs. 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010), Cape Town, 2010, 114--117.

Crossref

Google Scholar

[9]

Tang L, Li T, Perng C S. LogSig: Generating system events from raw textual logs. The 20th ACM Conference on Information and Knowledge Management (CIKM 2011), Glasgow, United Kingdom, 2011.

Digital Library

Google Scholar

[10]

Xu W, Huang L, Fox A, et al. Detecting Large-Scale System Problems by Mining Console Logs. The 27th International Conference on Machine Learning (ICML 10), Haifa, Israel, 2010.

Digital Library

Google Scholar

[11]

Loghub: A collection of system log datasets for intelligent log analysis. [Online]. Available: https://github.com/logpai/loghub.

Google Scholar

[12]

Analysis L. Introduction to Information Retrieval. 2008.

Google Scholar

[13]

Evaluation of clustering. [Online]. Available: http://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-clustering-1.html.

Google Scholar

Cited By

View all

Raeiszadeh MEstrada-Solano FGlitho REker JMini R(2024)A Data-Driven Approach for Adaptive Real-Time Log Parsing in Cloud Environments2024 IEEE International Mediterranean Conference on Communications and Networking (MeditCom)10.1109/MeditCom61057.2024.10621416(173-178)Online publication date: 8-Jul-2024
https://doi.org/10.1109/MeditCom61057.2024.10621416
Lupton SWashizaki HYoshioka NFukazawa Y(2024)Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection MethodsIEEE Access10.1109/ACCESS.2024.338728712(78193-78218)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3387287
Batoun MSayagh MAghili ROuni ALi H(2024)A literature review and existing challenges on software logging practicesEmpirical Software Engineering10.1007/s10664-024-10452-w29:4Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1007/s10664-024-10452-w
Show More Cited By

Index Terms

OLMPT: Research on Online Log Parsing Method Based on Prefix Tree
1. Information systems

Index terms have been assigned to the content through auto-classification.

Recommendations

XDrain: Effective log parsing in log streams using fixed-depth forest
Abstract
Logs record rich information that can help operators diagnose system failure [1]. Analyzing logs in log streams can expedite the diagnostic process and effectively mitigate the impact of failures. Log parsing is a prerequisite for automated log ...
Tools and benchmarks for automated log parsing
ICSE-SEIP '19: Proceedings of the 41st International Conference on Software Engineering: Software Engineering in Practice

Logs are imperative in the development and maintenance process of many software systems. They record detailed runtime information that allows developers and support engineers to monitor their systems and dissect anomalous behaviors and errors. The ...
A Syntactic-based Word Re-ordering for English-Vietnamese Statistical Machine Translation System
PRICAI '08: Proceedings of the 10th Pacific Rim International Conference on Artificial Intelligence: Trends in Artificial Intelligence

In machine translation, the re-ordering of word from source to target language is one of the major steps that affect mainly the performance of the system. Among many approaches for this type of problem, syntactic is an effective method for handling word-...

Comments

Information & Contributors

Information

Published In

ICITEE '20: Proceedings of the 3rd International Conference on Information Technologies and Electrical Engineering

December 2020

687 pages

ISBN:9781450388665

DOI:10.1145/3452940

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 May 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

The National Major Project of China

Conference

ICITEE2020

ICITEE2020: The 3rd International Conference on Information Technologies and Electrical Engineering

December 3 - 5, 2020

Hunan, Changde City, China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
76
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)2

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Raeiszadeh MEstrada-Solano FGlitho REker JMini R(2024)A Data-Driven Approach for Adaptive Real-Time Log Parsing in Cloud Environments2024 IEEE International Mediterranean Conference on Communications and Networking (MeditCom)10.1109/MeditCom61057.2024.10621416(173-178)Online publication date: 8-Jul-2024
https://doi.org/10.1109/MeditCom61057.2024.10621416
Lupton SWashizaki HYoshioka NFukazawa Y(2024)Landscape and Taxonomy of Online Parser-Supported Log Anomaly Detection MethodsIEEE Access10.1109/ACCESS.2024.338728712(78193-78218)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3387287
Batoun MSayagh MAghili ROuni ALi H(2024)A literature review and existing challenges on software logging practicesEmpirical Software Engineering10.1007/s10664-024-10452-w29:4Online publication date: 18-Jun-2024
https://dl.acm.org/doi/10.1007/s10664-024-10452-w
Zhang TQiu HCastellano GRifai MChen CPianese F(2023)System Log Parsing: A SurveyIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2022.322241735:8(8596-8614)Online publication date: 1-Aug-2023
https://dl.acm.org/doi/10.1109/TKDE.2022.3222417
Lupton SWashizaki HYoshioka NFukazawa Y(2021)Online Log Parsing: Preliminary Literature Review2021 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)10.1109/ISSREW53611.2021.00091(304-305)Online publication date: Oct-2021
https://doi.org/10.1109/ISSREW53611.2021.00091
Lupton SWashizaki HYoshioka NFukazawa Y(2021)Literature Review on Log Anomaly Detection Approaches Utilizing Online Parsing Methodology2021 28th Asia-Pacific Software Engineering Conference (APSEC)10.1109/APSEC53868.2021.00068(559-563)Online publication date: Dec-2021
https://doi.org/10.1109/APSEC53868.2021.00068

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

XDrain: Effective log parsing in log streams using fixed-depth forest

Tools and benchmarks for automated log parsing

A Syntactic-based Word Re-ordering for English-Vietnamese Statistical Machine Translation System

Comments

Information

Published In

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Funding Sources

Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations