poster

A distributed data streaming algorithm for network-wide traffic anomaly detection

Authors:

Yang Liu,

Linfeng Zhang,

Yong GuanAuthors Info & Claims

ACM SIGMETRICS Performance Evaluation Review, Volume 37, Issue 2

Pages 81 - 82

https://doi.org/10.1145/1639562.1639596

Published: 16 October 2009 Publication History

Get Access

Abstract

Nowadays, Internet has serious security problems and network failures that are hard to resolve, for example, botnet attacks, polymorphic worm/virus spreading, DDoS, and flash crowds. To address many of these problems, we need to have a network-wide view of the traffic dynamics, and more importantly, be able to detect traffic anomaly in a timely manner. To our knowledge, Principle Component Analysis (PCA)is the best-known spatial detection method for the network-wide traffic anomaly. However, existing PCA-based solutions have scalability problems in that they require O(m² n)running time and O(mn)space to analyze traffic measurements from m aggregated traffic flows within a sliding window of the length n. We propose a novel data streaming algorithm for PCA-based network-wide traffic anomaly detection in a distributed fashion. Our algorithm can archive O(wn log n)running time and O(wn)space at local monitors,and O(m² log n)running time and O(m log n) space at Network Operation Center (NOC), where w denotes the maximum number of traffic flows at a local monitor.

References

[1]

Abilene observatory data collections. www.internet2.edu/observatory/

Google Scholar

[2]

N. Alon, P.B. Gibbons, Y. Matias, and M. Szegedy. Tracking join and self-join sizes in limited storage. PODS '99 pages 10--20, 1999.

Digital Library

Google Scholar

[3]

L. Huang, X.L. Nguyen, M. Garofalakis, J. Hellerstein, M. Jordan, A. Joseph, and N. Taft. Communication-efficient online detection of network-wide anomalies. INFOCOM '07 pages 134--142, 2007.

Digital Library

Google Scholar

[4]

J.E. Jackson and G.S. Mudholkar. Control procedures for residuals associated with principal de-component analysis. Thechnometrics pages 341--349, 1979.

Google Scholar

[5]

A. Lakhina, M. Crovella, and C. Diot. Diagnosing network-wide traffic anomalies. SIGCOMM Comput. Commun. Rev.34(4):219--230, 2004.

Digital Library

Google Scholar

[6]

A. Lakhina,M. Crovella,and C. Diot. Mining anomalies using traffic feature distributions. SIGCOMM '05 pages 217--228, 2005.

Digital Library

Google Scholar

[7]

X. Li, F. Bian, M. Crovella, C. Diot, R. Govindan, G. Iannaccone, and A. Lakhina. Detection and identification of network anomalies using sketch subspaces. IMC '06 pages 147--152, 2006.

Digital Library

Google Scholar

[8]

Y. Liu, L. Zhang, and Y. Guan. Sketch-based network-wide traffic anomaly detection. Technical Report, ECpE Department, Iowa State University (http://home.eng.iastate.edu/?yangl/), April 2009.

Google Scholar

[9]

H. Ringberg, A. Soule, J. Rexford, and C. Diot. Sensitivity of pca for traffic anomaly detection. SIGMETRICS '07 pages 109--120, 2007.

Digital Library

Google Scholar

[10]

G. Stewart and J.-G. Sun. Matrix perturbation theory Academic Press, Boston, 1990.

Google Scholar

[11]

S.S. Vempala. The Random Projection Method American Mathematical Society, Rhode Island, 2004.

Google Scholar

Cited By

View all

Talapula DKumar ARavulakollu KKumar M(2023)Anomaly Detection in Online Data Streams Using Deep Belief Neural NetworksProceedings of Fourth Doctoral Symposium on Computational Intelligence10.1007/978-981-99-3716-5_59(729-749)Online publication date: 17-Sep-2023
https://doi.org/10.1007/978-981-99-3716-5_59
Ma MLin WPan DWang P(2022)Self-Adaptive Root Cause Diagnosis for Large-Scale Microservice ArchitectureIEEE Transactions on Services Computing10.1109/TSC.2020.299325115:3(1399-1410)Online publication date: 1-May-2022
https://doi.org/10.1109/TSC.2020.2993251
Ma MLin WPan DWang P(2022)ServiceRank: Root Cause Identification of Anomaly in Large-Scale Microservice ArchitecturesIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2021.308367119:5(3087-3100)Online publication date: 1-Sep-2022
https://doi.org/10.1109/TDSC.2021.3083671
Show More Cited By

Index Terms

A distributed data streaming algorithm for network-wide traffic anomaly detection

Recommendations

Sketch-Based Streaming PCA Algorithm for Network-Wide Traffic Anomaly Detection
ICDCS '10: Proceedings of the 2010 IEEE 30th International Conference on Distributed Computing Systems

Internet has become an essential part of the daily life for billions of users worldwide, who are using a large variety of network services and applications everyday. However, there have been serious security problems and network failures that are hard ...
Detecting distributed network traffic anomaly with network-wide correlation analysis
Special issue on signal processing applications in network intrusion detection systems

Distributed network traffic anomaly refers to a traffic abnormal behavior involving many links of a network and caused by the same source (e.g., DDoS attack, worm propagation). The anomaly transiting in a single link might be unnoticeable and hard to ...
Network-wide traffic analysis: methods and applications

Comments

Information & Contributors

Information

Published In

cover image ACM SIGMETRICS Performance Evaluation Review

ACM SIGMETRICS Performance Evaluation Review Volume 37, Issue 2

September 2009

89 pages

ISSN:0163-5999

DOI:10.1145/1639562

Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 October 2009

Published in SIGMETRICS Volume 37, Issue 2

Check for updates

Qualifiers

Poster

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
363
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)1

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Talapula DKumar ARavulakollu KKumar M(2023)Anomaly Detection in Online Data Streams Using Deep Belief Neural NetworksProceedings of Fourth Doctoral Symposium on Computational Intelligence10.1007/978-981-99-3716-5_59(729-749)Online publication date: 17-Sep-2023
https://doi.org/10.1007/978-981-99-3716-5_59
Ma MLin WPan DWang P(2022)Self-Adaptive Root Cause Diagnosis for Large-Scale Microservice ArchitectureIEEE Transactions on Services Computing10.1109/TSC.2020.299325115:3(1399-1410)Online publication date: 1-May-2022
https://doi.org/10.1109/TSC.2020.2993251
Ma MLin WPan DWang P(2022)ServiceRank: Root Cause Identification of Anomaly in Large-Scale Microservice ArchitecturesIEEE Transactions on Dependable and Secure Computing10.1109/TDSC.2021.308367119:5(3087-3100)Online publication date: 1-Sep-2022
https://doi.org/10.1109/TDSC.2021.3083671
Ma MXu JWang YChen PZhang ZWang P(2020)AutoMAP: Diagnose Your Microservice-based Web Applications AutomaticallyProceedings of The Web Conference 202010.1145/3366423.3380111(246-258)Online publication date: 20-Apr-2020
https://dl.acm.org/doi/10.1145/3366423.3380111
Laufenberg DLi LShahriar HHan M(2020)Developing a Blockchain-Enabled Collaborative Intrusion Detection System: An Exploratory StudyAdvances in Information and Communication10.1007/978-3-030-39445-5_14(172-183)Online publication date: 25-Feb-2020
https://doi.org/10.1007/978-3-030-39445-5_14
Liu PChen YNie XZhu JZhang SSui KZhang MPei D(2019)FluxRank: A Widely-Deployable Framework to Automatically Localizing Root Cause Machines for Software Service Failure Mitigation2019 IEEE 30th International Symposium on Software Reliability Engineering (ISSRE)10.1109/ISSRE.2019.00014(35-46)Online publication date: Oct-2019
https://doi.org/10.1109/ISSRE.2019.00014
Ma MLin WPan DWang P(2019)MS-Rank: Multi-Metric and Self-Adaptive Root Cause Diagnosis for Microservice Applications2019 IEEE International Conference on Web Services (ICWS)10.1109/ICWS.2019.00022(60-67)Online publication date: Jul-2019
https://doi.org/10.1109/ICWS.2019.00022
Malaiya RKwon DSuh SKim HKim IKim J(2019)An Empirical Evaluation of Deep Learning for Network Anomaly DetectionIEEE Access10.1109/ACCESS.2019.29432497(140806-140817)Online publication date: 2019
https://doi.org/10.1109/ACCESS.2019.2943249
Lin WMa MPan DWang P(2018)FacGraph: Frequent Anomaly Correlation Graph Mining for Root Cause Diagnose in Micro-Service Architecture2018 IEEE 37th International Performance Computing and Communications Conference (IPCCC)10.1109/PCCC.2018.8711092(1-8)Online publication date: Nov-2018
https://doi.org/10.1109/PCCC.2018.8711092
Malaiya RKwon DKim JSuh SKim HKim I(2018)An Empirical Evaluation of Deep Learning for Network Anomaly Detection2018 International Conference on Computing, Networking and Communications (ICNC)10.1109/ICCNC.2018.8390278(893-898)Online publication date: Mar-2018
https://doi.org/10.1109/ICCNC.2018.8390278
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Sketch-Based Streaming PCA Algorithm for Network-Wide Traffic Anomaly Detection

Detecting distributed network traffic anomaly with network-wide correlation analysis

Network-wide traffic analysis: methods and applications

Comments

Information

Published In

Publisher

Publication History

Check for updates

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations