research-article

Recursive lattice search: hierarchical heavy hitters revisited

Author:

Kenjiro ChoAuthors Info & Claims

IMC '17: Proceedings of the 2017 Internet Measurement Conference

Pages 283 - 289

https://doi.org/10.1145/3131365.3131377

Published: 01 November 2017 Publication History

Abstract

The multidimensional Hierarchical Heavy Hitter (HHH) problem identifies significant clusters in traffic across multiple planes such as source and destination addresses, and has been widely studied in the literature. A compact summary of HHHs provides an overview on complex traffic behavior and is a powerful means for traffic monitoring and anomaly detection. In this paper, we present a new efficient HHH algorithm which fits operational needs. Our key insight is to revisit the commonly accepted definition of HHH, and apply the Z-ordering to make use of a recursive partitioning algorithm. The proposed algorithm produces summary outputs comparable to or even better in practice than the existing algorithms, and runs orders of magnitude faster for bitwise aggregation. We have implemented the algorithm into our open-source tool and have made longitudinal datasets of backbone traffic openly available.

References

[1]

Ran Ben Basat, Gil Einziger, Roy Friedman, Marcelo C. Luizelli, and Erez Waisbard. 2017. Constant Time Updates in Hierarchical Heavy Hitters. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication (SIGCOMM '17). ACM, New York, NY, USA, 127--140.

Digital Library

[2]

Kevin Beyer and Raghu Ramakrishnan. 1999. Bottom-up Computation of Sparse and Iceberg CUBE. In Proceedings of the 1999 ACM SIGMOD International Conference on Management of Data (SIGMOD '99). ACM, New York, NY, USA, 359--370.

Digital Library

[3]

Kenjiro Cho, Ryo Kaizaki, and Akira Kato. 2001. Aguri: An Aggregation-Based Traffic Profiler. In Proceedings of the Second International Workshop on Quality of Future Internet Services (COST 263). Springer-Verlag, London, UK, UK, 222--242. http://dl.acm.org/citation.cfm?id=646462.693721

Digital Library

[4]

Kenjiro Cho, Koushirou Mitsuya, and Akira Kato. 2000. Traffic Data Repository at the WIDE Project. In Proceedings of the Annual Conference on USENIX Annual Technical Conference (ATEC '00). USENIX Association, Berkeley, CA, USA, 51--51. http://dl.acm.org/citation.cfm?id=1267724.1267775

Digital Library

[5]

Graham Cormode and Marios Hadjieleftheriou. 2008. Finding Frequent Items in Data Streams. Proc. VLDB Endow. 1, 2 (Aug. 2008), 1530--1541.

Digital Library

[6]

Graham Cormode, Flip Korn, S. Muthukrishnan, and Divesh Srivastava. 2003. Finding Hierarchical Heavy Hitters in Data Streams. In Proceedings of the 29th International Conference on Very Large Data Bases - Volume 29 (VLDB '03). VLDB Endowment, 464--475. http://dl.acm.org/citation.cfm?id=1315451.1315492

Digital Library

[7]

Graham Cormode, Flip Korn, S. Muthukrishnan, and Divesh Srivastava. 2004. Diamond in the Rough: Finding Hierarchical Heavy Hitters in Multi-dimensional Data. In Proceedings of the 2004 ACM SIGMOD International Conference on Management of Data (SIGMOD '04). ACM, New York, NY, USA, 155--166.

Digital Library

[8]

Cristian Estan, Stefan Savage, and George Varghese. 2003. Automatically Inferring Patterns of Resource Consumption in Network Traffic. In Proceedings of the 2003 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications (SIGCOMM '03). ACM, New York, NY, USA, 137--148.

Digital Library

[9]

R. A. Finkel and J. L. Bentley. 1974. Quad Trees a Data Structure for Retrieval on Composite Keys. Acta Inf. 4, 1 (March 1974), 1--9.

Digital Library

[10]

John Hershberger, Nisheeth Shrivastava, Subhash Suri, and Csaba D. Tóth. 2005. Space Complexity of Hierarchical Heavy Hitters in Multi-dimensional Data Streams. In Proceedings of the Twenty-fourth ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems (PODS '05). ACM, New York, NY, USA, 338--347.

Digital Library

[11]

Lavanya Jose, Minlan Yu, and Jennifer Rexford. 2011. Online Measurement of Large Traffic Aggregates on Commodity Switches. In Proceedings of the 11th USENIX Conference on Hot Topics in Management of Internet, Cloud, and Enterprise Networks and Services (Hot-ICE'11). USENIX Association, Berkeley, CA, USA, 13--13. http://dl.acm.org/citation.cfm?id=1972422.1972439

Digital Library

[12]

Midori Kato, Kenjiro Cho, Michio Honda, and Hideyuki Tokuda. 2012. Monitoring the Dynamics of Network Traffic by Recursive Multi-Dimensional Aggregation. In Presented as part of the 2012 Workshop on Managing Systems Automatically and Dynamically. USENIX, Hollywood, CA. https://www.usenix.org/conference/mad12/workshop-program/presentation/Kato

[13]

Yunqi Li, Jiahai Yang, Changqing An, and Hui Zhang. 2007. Finding Hierarchical Heavy Hitters in Network Measurement System. In Proceedings of the 2007 ACM Symposium on Applied Computing (SAC '07). ACM, New York, NY, USA, 232--236.

Digital Library

[14]

Donald Meagher. 1982. Geometric Modeling Using Octree Encoding. Computer Graphics and Image Processing 19 (1982), 249--270.

[15]

M. Mitzenmacher, T. Steinke, and J. Thaler. 2012. Hierarchical Heavy Hitters with the Space Saving Algorithm. In Proceedings of the Meeting on Algorithm Engineering & Expermiments (ALENEX '12). Society for Industrial and Applied Mathematics, Philadelphia, PA, USA, 160--174. http://dl.acm.org/citation.cfm?id=2790265.2790281

Digital Library

[16]

G. M. Morton. 1966. A Computer Oriented Geodetic Data Base and a New Technique in File Sequencing. Technical Report. IBM Ltd.

[17]

Masoud Moshref, Minlan Yu, Ramesh Govindan, and Amin Vahdat. 2014. DREAM: Dynamic Resource Allocation for Software-defined Measurement. In Proceedings of the 2014 ACM Conference on SIGCOMM (SIGCOMM '14). ACM, New York, NY, USA, 419--430.

Digital Library

[18]

Diana Andreea Popescu, Gianni Antichi, and Andrew W. Moore. 2017. Enabling Fast Hierarchical Heavy Hitter Detection Using Programmable Data Planes. In Proceedings of the Symposium on SDN Research (SOSR '17). ACM, New York, NY, USA, 191--192.

Digital Library

[19]

Hanan Samet. 1984. The Quadtree and Related Hierarchical Data Structures. ACM Comput. Surv. 16, 2 (June 1984), 187--260.

Digital Library

[20]

V. Srinivasan, G. Varghese, S. Suri, and M. Waldvogel. 1998. Fast and Scalable Layer Four Switching. In Proceedings of the ACM SIGCOMM '98 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication (SIGCOMM '98). ACM, New York, NY, USA, 191--202.

Digital Library

[21]

Da Tong and Viktor Prasanna. 2015. High Throughput Hierarchical Heavy Hitter Detection in Data Streams. In Proceedings of the 2015 IEEE 22Nd International Conference on High Performance Computing (HiPC) (HIPC '15). IEEE Computer Society, Washington, DC, USA, 224--233.

Digital Library

[22]

WIDE Project 2017. WIDE Project web page. (2017). Retrieved September 28, 2017 from http://www.wide.ad.jp/

[23]

Lihua Yuan, Chen-Nee Chuah, and Prasant Mohapatra. 2007. ProgME: Towards Programmable Network Measurement. In Proceedings of the 2007 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications (SIGCOMM '07). ACM, New York, NY, USA, 97--108.

Digital Library

[24]

Yin Zhang, Sumeet Singh, Subhabrata Sen, Nick Duffield, and Carsten Lund. 2004. Online Identification of Hierarchical Heavy Hitters: Algorithms, Evaluation, and Applications. In Proceedings of the 4th ACM SIGCOMM Conference on Internet Measurement (IMC '04). ACM, New York, NY, USA, 101--114.

Digital Library

Cited By

Li FLv YYan YGao CWang XCao J(2024)Learning-Based Sketch for Adaptive and High-Performance Network MeasurementIEEE/ACM Transactions on Networking10.1109/TNET.2024.336417632:3(2571-2585)Online publication date: Jun-2024
https://doi.org/10.1109/TNET.2024.3364176
Su HXiao Q(2023)Online Detection of 1D and 2D Hierarchical Super-Spreaders in High-Speed NetworksProceedings of the 7th Asia-Pacific Workshop on Networking10.1145/3600061.3600080(109-115)Online publication date: 29-Jun-2023
https://dl.acm.org/doi/10.1145/3600061.3600080
Miano SChen XBasat RAntichi G(2023)Fast In-kernel Traffic Sketching in eBPFACM SIGCOMM Computer Communication Review10.1145/3594255.359425653:1(3-13)Online publication date: 20-Apr-2023
https://dl.acm.org/doi/10.1145/3594255.3594256
Show More Cited By

Index Terms

Recursive lattice search: hierarchical heavy hitters revisited
1. Networks
  1. Network algorithms
    1. Data path algorithms
      1. Packet classification
  2. Network services
    1. Network monitoring

Recommendations

Computing discounted multidimensional hierarchical aggregates using modified Misra Gries algorithm

Finding the "Top k" list or heavy hitters is an important function in many computing applications, including database joins, data warehousing (e.g., OLAP), web caching and hits, network usage monitoring, and detecting DDoS attacks. While most ...
Online identification of hierarchical heavy hitters: algorithms, evaluation, and applications
IMC '04: Proceedings of the 4th ACM SIGCOMM conference on Internet measurement

In traffic monitoring, accounting, and network anomaly detection, it is often important to be able to detect high-volume traffic clusters in near real-time. Such heavy-hitter traffic clusters are often hierarchical (<i>ie</i>, they may occur at ...
Differentially Private Hierarchical Heavy Hitters
PODS

The task of finding Hierarchical Heavy Hitters (HHH) was introduced by Cormode et al. [12] as a generalisation of the heavy hitter problem. While finding HHH in data streams has been studied extensively, the question of releasing HHH when the underlying ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

IMC '17: Proceedings of the 2017 Internet Measurement Conference

November 2017

509 pages

ISBN:9781450351188

DOI:10.1145/3131365

General Chairs:
Steve Uhlig
Queen Mary University of London
,
Olaf Maennel
Tallinn University of Technology

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

In-Cooperation

USENIX Assoc: USENIX Assoc

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

IMC '17

Sponsor:

IMC '17: Internet Measurement Conference

November 1 - 3, 2017

London, United Kingdom

Acceptance Rates

Overall Acceptance Rate 277 of 1,083 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
546
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li FLv YYan YGao CWang XCao J(2024)Learning-Based Sketch for Adaptive and High-Performance Network MeasurementIEEE/ACM Transactions on Networking10.1109/TNET.2024.336417632:3(2571-2585)Online publication date: Jun-2024
https://doi.org/10.1109/TNET.2024.3364176
Su HXiao Q(2023)Online Detection of 1D and 2D Hierarchical Super-Spreaders in High-Speed NetworksProceedings of the 7th Asia-Pacific Workshop on Networking10.1145/3600061.3600080(109-115)Online publication date: 29-Jun-2023
https://dl.acm.org/doi/10.1145/3600061.3600080
Miano SChen XBasat RAntichi G(2023)Fast In-kernel Traffic Sketching in eBPFACM SIGCOMM Computer Communication Review10.1145/3594255.359425653:1(3-13)Online publication date: 20-Apr-2023
https://dl.acm.org/doi/10.1145/3594255.3594256
Tang LHuang QLee P(2023)MVPipe: Enabling Lightweight Updates and Fast Convergence in Hierarchical Heavy Hitter DetectionIEEE/ACM Transactions on Networking10.1109/TNET.2023.327330731:6(3207-3221)Online publication date: Dec-2023
https://doi.org/10.1109/TNET.2023.3273307
Meng YZhang QTang XZhang WWang J(2023)FTM-RCA: A Fast Two-Stage Multi-dimensional Root-Cause Analysis of Network Anomalies2023 IEEE/ACM 31st International Symposium on Quality of Service (IWQoS)10.1109/IWQoS57198.2023.10188732(01-10)Online publication date: 19-Jun-2023
https://doi.org/10.1109/IWQoS57198.2023.10188732
Alfasi EEinziger G(2022)Botnet Mapping Based on Intersections of TracesProceedings of the 23rd International Conference on Distributed Computing and Networking10.1145/3491003.3491025(198-207)Online publication date: 4-Jan-2022
https://dl.acm.org/doi/10.1145/3491003.3491025
Ben Basat REinziger GKeslassy IOrda AVargaftik SWaisbard E(2022)Memento: Making Sliding Windows Efficient for Heavy HittersIEEE/ACM Transactions on Networking10.1109/TNET.2021.313238530:4(1440-1453)Online publication date: Aug-2022
https://doi.org/10.1109/TNET.2021.3132385
Yan YLi FWang WWang X(2022)TalentSketch: LSTM-based Sketch for Adaptive and High-Precision Network Measurement2022 IEEE 30th International Conference on Network Protocols (ICNP)10.1109/ICNP55882.2022.9940396(1-12)Online publication date: 30-Oct-2022
https://doi.org/10.1109/ICNP55882.2022.9940396
Li M(2021)Generalized fractional Gaussian noise and its application to traffic modelingPhysica A: Statistical Mechanics and its Applications10.1016/j.physa.2021.126138579(126138)Online publication date: Oct-2021
https://doi.org/10.1016/j.physa.2021.126138
Zhu SLi SWang ZChen XQian ZKrishnamurthy SChan KSwami AHan DFeldmann A(2020)You do (not) belong hereProceedings of the 16th International Conference on emerging Networking EXperiments and Technologies10.1145/3386367.3431311(183-197)Online publication date: 23-Nov-2020
https://dl.acm.org/doi/10.1145/3386367.3431311
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten