Scalable Coordination Techniques for Distributed Network Monitoring

Abstract

Emerging network monitoring infrastructures capture packet-level traces or keep per-flow statistics at a set of distributed vantage points. Today, distributed monitors in such an infrastructure do not coordinate monitoring effort, which both can lead to duplication of effort and can complicate subsequent data analysis. We argue that nodes in such a monitoring infrastructure, whether across the wide-area Internet, or across a sensor network, should coordinate effort to minimize resource consumption. We propose space-efficient data structures for use in gossip-based protocols to approximately summarize sets of monitored flows. With some fine-tuning of our methods, we can ensure that all flows observed by at least one monitor are monitored, and only a tiny fraction are monitored redundantly. Our preliminary results over a realistic ISP topology demonstrate the effectiveness of our techniques on monitoring tens of thousands of point-of-presence (PoP) level network flows. Our methods are competitive with optimal off-line coordination, but require significantly less space and network overhead than naive approaches.