skip to main content
10.1145/3534678.3539155acmconferencesArticle/Chapter ViewAbstractPublication PageskddConference Proceedingsconference-collections
research-article
Public Access

CAT: Beyond Efficient Transformer for Content-Aware Anomaly Detection in Event Sequences

Published: 14 August 2022 Publication History

Abstract

It is critical and important to detect anomalies in event sequences, which becomes widely available in many application domains. Indeed, various efforts have been made to capture abnormal patterns from event sequences through sequential pattern analysis or event representation learning. However, existing approaches usually ignore the semantic information of event content. To this end, in this paper, we propose a self-attentive encoder-decoder transformer framework, Content-Aware Transformer CAT, for anomaly detection in event sequences. In CAT, the encoder learns preamble event sequence representations with content awareness, and the decoder embeds sequences under detection into a latent space, where anomalies are distinguishable. Specifically, the event content is first fed to a content-awareness layer, generating representations of each event. The encoder accepts preamble event representation sequence, generating feature maps. In the decoder, an additional token is added at the beginning of the sequence under detection, denoting the sequence status. A one-class objective together with sequence reconstruction loss is collectively applied to train our framework under the label efficiency scheme. Furthermore, CAT is optimized under a scalable and efficient setting. Finally, extensive experiments on three real-world datasets demonstrate the superiority of CAT.

Supplemental Material

MP4 File
Presentation video for paper "CAT: Beyond Efficient Transformer for Content-Aware Anomaly Detection in Event Sequences"

References

[1]
Seyed Mojtaba Hosseini Bamakan, Huadong Wang, and Yong Shi. 2017. Ramp loss K-Support Vector Classification-Regression; a robust and sparse multi-class approach to the intrusion detection problem. Knowledge-Based Systems 126 (2017), 113--126.
[2]
Peter Bodik, Moises Goldszmidt, Armando Fox, Dawn B Woodard, and Hans Andersen. 2010. Fingerprinting the datacenter: automated classification of performance crises. In 5th European conference on Computer systems. 111--124.
[3]
Shyam Boriah, Varun Chandola, and Vipin Kumar. 2008. Similarity measures for categorical data: A comparative evaluation. In Proceedings of the 2008 SIAM international conference on data mining. SIAM, 243--254.
[4]
Markus M Breunig, Hans-Peter Kriegel, Raymond T Ng, and Jörg Sander. 2000. LOF: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD international conference on Management of data. 93--104.
[5]
Suratna Budalakoti, Ashok N Srivastava, and Matthew Eric Otey. 2008. Anomaly detection and diagnosis algorithms for discrete symbol sequences with applications to airline safety. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 39, 1 (2008), 101--113.
[6]
Varun Chandola, Arindam Banerjee, and Vipin Kumar. 2010. Anomaly detection for discrete sequences: A survey. IEEE transactions on knowledge and data engineering 24, 5 (2010), 823--839.
[7]
Rui Chen, Shenglin Zhang, Dongwen Li, Yuzhe Zhang, Fangrui Guo, Weibin Meng, Dan Pei, Yuzhi Zhang, Xu Chen, and Yuqing Liu. 2020. LogTransfer: Cross-system log anomaly detection for software systems with transfer learning. In ISSRE. IEEE, 37--47.
[8]
Sung-Bae Cho and Hyuk-Jang Park. 2003. Efficient anomaly detection by modeling privilege flows using hidden Markov model. computers & security 22, 1 (2003), 45--55.
[9]
Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. 2015. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint arXiv:1511.07289 (2015).
[10]
Christophe Combet, Christophe Blanchet, Christophe Geourjon, and Gilbert Deleage. 2000. NPS@: network protein sequence analysis. Trends in biochemical sciences 25, 3 (2000), 147--150.
[11]
Jeffrey Delmerico, Titus Cieslewski, Henri Rebecq, Matthias Faessler, and Davide Scaramuzza. 2019. Are we ready for autonomous drone racing? the UZH-FPV drone racing dataset. In 2019 International Conference on Robotics and Automation (ICRA). IEEE, 6713--6719.
[12]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
[13]
Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar. 2017. Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. 1285--1298.
[14]
Amir Farzad and T Aaron Gulliver. 2020. Unsupervised log message anomaly detection. ICT Express 6, 3 (2020), 229--237.
[15]
Gilberto Fernandes, Joel JPC Rodrigues, Luiz Fernando Carvalho, Jalal F Al- Muhtadi, and Mario Lemes Proença. 2019. A comprehensive survey on network anomaly detection. Telecommunication Systems 70, 3 (2019), 447--489.
[16]
Robin Gandhi, Anup Sharma, William Mahoney, William Sousan, Qiuming Zhu, and Phillip Laplante. 2011. Dimensions of cyber-attacks: Cultural, social, economic, and political. IEEE Technology and Society Magazine 30, 1 (2011), 28--38.
[17]
Haixuan Guo, Shuhan Yuan, and Xintao Wu. 2021. Logbert: Log anomaly detection via bert. In 2021 International Joint Conference on Neural Networks. IEEE.
[18]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735--1780.
[19]
Shaohan Huang, Yi Liu, Carol Fung, Rong He, Yining Zhao, Hailong Yang, and Zhongzhi Luan. 2020. Hitanomaly: Hierarchical transformers for anomaly detection in system log. IEEE Transactions on Network and Service Management 17, 4 (2020), 2064--2076.
[20]
Wenke Lee and Salvatore Stolfo. 1998. Data mining approaches for intrusion detection. (1998).
[21]
Yinglung Liang, Yanyong Zhang, Hui Xiong, and Ramendra Sahoo. 2007. Failure prediction in ibm bluegene/l event logs. In Seventh IEEE International Conference on Data Mining (ICDM 2007). IEEE, 583--588.
[22]
Qingwei Lin, Hongyu Zhang, Jian-Guang Lou, Yu Zhang, and Xuewei Chen. 2016. Log clustering based problem identification for online service systems. In 2016 IEEE/ACM 38th International Conference on Software Engineering Companion (ICSE-C). IEEE, 102--111.
[23]
Fei Tony Liu, Kai Ming Ting, and Zhi-Hua Zhou. 2008. Isolation forest. In 2008 eighth ieee international conference on data mining. IEEE, 413--422.
[24]
Zhen Liu, Nathalie Japkowicz, Ruoyu Wang, Yongming Cai, Deyu Tang, and Xianfa Cai. 2020. A statistical pattern based feature extraction method on system call traces for anomaly detection. Information and Software Technology 126 (2020).
[25]
Jian-Guang Lou, Qiang Fu, Shengqi Yang, Ye Xu, and Jiang Li. 2010. Mining Invariants from Console Logs for System Problem Detection. In USENIX Annual Technical Conference. 1--14.
[26]
Song-song Lu, Xiao-feng Wang, and Li Mao. 2014. Network security situation awareness based on network simulation. In 2014 IEEE workshop on electronics, computer and applications. IEEE, 512--517.
[27]
Weibin Meng, Ying Liu, Yuheng Huang, Shenglin Zhang, Federico Zaiter, Bingjin Chen, and Dan Pei. 2020. A semantic-aware representation framework for online log analysis. In 2020 29th International Conference on Computer Communications and Networks (ICCCN). IEEE, 1--7.
[28]
Weibin Meng, Ying Liu, Yichen Zhu, Shenglin Zhang, Dan Pei, Yuqing Liu, Yihao Chen, Ruizhi Zhang, Shimin Tao, Pei Sun, et al. 2019. LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs. In IJCAI, Vol. 19. 4739--4745.
[29]
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013).
[30]
Sasho Nedelkoski, Jasmin Bogatinovski, Alexander Acker, Jorge Cardoso, and Odej Kao. 2020. Self-attentive classification-based anomaly detection in unstructured logs. In ICDM. IEEE, 1196--1201.
[31]
Adam Oliner and Jon Stearley. 2007. What supercomputers say: A study of five system logs. In 37th annual IEEE/IFIP international conference on dependable systems and networks (DSN'07). IEEE, 575--584.
[32]
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 1532--1543.
[33]
Bernhard Schölkopf, John C Platt, John Shawe-Taylor, Alex J Smola, and Robert C Williamson. 2001. Estimating the support of a high-dimensional distribution. Neural computation 13, 7 (2001), 1443--1471.
[34]
David MJ Tax and Robert PW Duin. 2004. Support vector data description. Machine learning 54, 1 (2004), 45--66.
[35]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems. 5998--6008.
[36]
ZhiweiWang, Zhengzhang Chen, Jingchao Ni, Hui Liu, Haifeng Chen, and Jiliang Tang. 2021. Multi-scale one-class recurrent neural networks for discrete event sequence anomaly detection. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 3726--3734.
[37]
Christina Warrender, Stephanie Forrest, and Barak Pearlmutter. 1999. Detecting intrusions using system calls: Alternative data models. In Proceedings of the 1999 IEEE symposium on security and privacy (Cat. No. 99CB36344). IEEE, 133--145.
[38]
Bryan Watkins. 2014. The impact of cyber attacks on the private sector. Briefing Paper, Association for International Affair 12 (2014), 1--11.
[39]
David J Weller-Fahy, Brett J Borghetti, and Angela A Sodemann. 2014. A survey of distance and similarity measures used within network intrusion anomaly detection. IEEE Communications Surveys & Tutorials 17, 1 (2014), 70--91.
[40]
Wei Xu, Ling Huang, Armando Fox, David Patterson, and Michael I Jordan. 2009. Detecting large-scale system problems by mining console logs. In Proceedings of the ACM SIGOPS 22nd symposium on Operating systems principles. 117--132.
[41]
Lin Yang, Junjie Chen, Zan Wang, Weijing Wang, Jiajun Jiang, Xuyuan Dong, and Wenbin Zhang. 2021. Semi-supervised log-based anomaly detection via probabilistic label estimation. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 1448--1460.
[42]
Chunkai Zhang, XinyuWang, Hongye Zhang, Hanyu Zhang, and Peiyi Han. 2021. Log Sequence Anomaly Detection Based on Local Information Extraction and Globally Sparse Transformer Model. IEEE Transactions on Network and Service Management 18, 4 (2021), 4119--4133.
[43]
Meng J Zhao, Anne R Driscoll, Srijan Sengupta, Ronald D Fricker Jr, Dan J Spitzner, and William HWoodall. 2018. Performance evaluation of social network anomaly detection using a moving window--based scan method. Quality and Reliability Engineering International 34, 8 (2018), 1699--1716.
[44]
Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021. Informer: Beyond efficient transformer for long sequence time-series forecasting. In Proceedings of AAAI.

Cited By

View all
  • (2024)Outlier Detection in Streaming Data for Telecommunications and Industrial Applications: A SurveyElectronics10.3390/electronics1316333913:16(3339)Online publication date: 22-Aug-2024
  • (2024)Pluto: Sample Selection for Robust Anomaly Detection on Polluted Log DataProceedings of the ACM on Management of Data10.1145/36771392:4(1-25)Online publication date: 30-Sep-2024
  • (2024)Try with Simpler - An Evaluation of Improved Principal Component Analysis in Log-based Anomaly DetectionACM Transactions on Software Engineering and Methodology10.1145/364438633:5(1-27)Online publication date: 3-Jun-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
KDD '22: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
August 2022
5033 pages
ISBN:9781450393850
DOI:10.1145/3534678
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 August 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. anomaly detection
  2. content-aware sequential pattern mining
  3. event sequence modeling
  4. one-class classification

Qualifiers

  • Research-article

Funding Sources

Conference

KDD '22
Sponsor:

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)744
  • Downloads (Last 6 weeks)64
Reflects downloads up to 17 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Outlier Detection in Streaming Data for Telecommunications and Industrial Applications: A SurveyElectronics10.3390/electronics1316333913:16(3339)Online publication date: 22-Aug-2024
  • (2024)Pluto: Sample Selection for Robust Anomaly Detection on Polluted Log DataProceedings of the ACM on Management of Data10.1145/36771392:4(1-25)Online publication date: 30-Sep-2024
  • (2024)Try with Simpler - An Evaluation of Improved Principal Component Analysis in Log-based Anomaly DetectionACM Transactions on Software Engineering and Methodology10.1145/364438633:5(1-27)Online publication date: 3-Jun-2024
  • (2024)CauseFormer: Interpretable Anomaly Detection With Stepwise Attention for Cloud ServiceIEEE Transactions on Network and Service Management10.1109/TNSM.2023.329984621:1(637-652)Online publication date: Feb-2024
  • (2024)GADformer: A Transparent Transformer Model for Group Anomaly Detection on Trajectories2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10650999(1-8)Online publication date: 30-Jun-2024
  • (2024)HEDGE: Heterogeneous Semantic Dynamic Graph Framework for Log Anomaly Detection in Digital Service Network2024 IEEE International Conference on Web Services (ICWS)10.1109/ICWS62655.2024.00041(208-216)Online publication date: 7-Jul-2024
  • (2024)DualAttlog: Context aware dual attention networks for log-based anomaly detectionNeural Networks10.1016/j.neunet.2024.106680(106680)Online publication date: Aug-2024
  • (2024)AFMFKnowledge-Based Systems10.1016/j.knosys.2024.111912296:COnline publication date: 19-Jul-2024
  • (2024)Log anomaly detection based on BERTSignal, Image and Video Processing10.1007/s11760-024-03327-618:8-9(6431-6441)Online publication date: 13-Jun-2024
  • (2024)Enhancing multivariate time-series anomaly detection with positional encoding mechanisms in transformersThe Journal of Supercomputing10.1007/s11227-024-06694-681:1Online publication date: 11-Dec-2024
  • Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Login options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media