Skip to main content
SpringerLink
Log in

Lightweight and Informative Traffic Metrics for Data Center Monitoring

  • Published:
Journal of Network and Systems Management Aims and scope Submit manuscript

Abstract

In recent years, thousands of commodity servers have been deployed in Internet data centers to run large scale Internet applications or cloud computing services. Given the sheer volume of data communications between servers and millions of end users, it becomes a daunting task to continuously monitor the availability, performance and security of data centers in real-time operational environments. In this paper, we propose and evaluate a lightweight and informative traffic metric, streaming frequency, for network monitoring in Internet data centers. The power-series based metric that is extracted from the aggregated IP traffic streams, not only carries temporal characteristics of data center servers, but also helps uncover traffic patterns of these servers. We show the convergence and reconstructability properties of this metric through theoretical proof and algorithm analysis. Using real data-sets collected from multiple data centers of a large Internet content provider, we demonstrate its applications in detecting unwanted traffic towards data center servers. To the best of our knowledge, this paper is the first to introduce a streaming metric with a unique reconstruction capability that could aid data center operators in network management and security monitoring.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. We do not include the variance in calculating the distance, since calculating the variance requires reconstructions at each time unit.

References

  1. Dreger, H., Feldmann, A., Paxson V., Sommer, R.: In: Proceedings of International Symposium on Recent Advances in Intrusion Detection (2008)

  2. Bhatia, S., Kumar, A., Fiuczynski, M., Peterson, L.: In: Proceedings of USENIX Symposium on Operating Systems Design and Implementation (2008)

  3. MRTG. Multi Router Traffic Grapher. http://oss.oetiker.ch/mrtg/

  4. Estan, C., Savage, S., Varghese, G.: In: Proceedings of ACM SIGCOMM (2003)

  5. Kornexl, S., Paxson, V., Dreger, H., Feldmann, A., Sommer R.: In: Proceedings of ACM SIGCOMM Conference on Internet Measurement (2005)

  6. Krishnamurthy, B., Sen, S., Zhang, Y., Chen, Y.: In: Proceedings of ACM/USENIX IMC (2003)

  7. Karagiannis, T., Papagiannaki K., Faloutsos, M.: In: Proceedings of ACM SIGCOMM (2005)

  8. Yahoo! Labs: Datasets. http://labs.yahoo.com/

  9. Wall Street Journal. http://online.wsj.com/article/BT-CO-20100112-716798.html (2009)

  10. Al-Fares, M., Loukissas, A., Vahdat, A.: In: Proceedings of ACM SIGCOMM (2008)

  11. Guo, C., Wu, H., Tan, K., Shi, L., Zhang, Y., Lu, S.: In: Proceedings of ACM SIGCOMM (2008)

  12. Joseph, D., Tavakoli, A., Stoica, I.: In: Proceedings of ACM SIGCOMM (2008)

  13. PC World. Microsoft: Datacenter Growth Defies Moore’s Law. http://www.pcworld.com/article/130921/article.html

  14. Greenberg, A., Hamilton, J., Maltz, D., Patel, P.: ACM SIGCOMM Comput. Commun. Rev. 39(1), 68 (2009)

    Google Scholar 

  15. Nedevschi, S., Padhye, J., Ratnasamy, S.: In: Proceedings of USENIX OSDI Workshop on Power Aware Computing and System (2008)

  16. Benson, T., Anand, A., Akella, A., Zhang, M.: In: Proceedings of SIGCOMM WREN Workshop (2009)

  17. Amazon. Amazon Web Services. http://aws.amazon.com/

  18. Zhou M., Lang, S.-D.: Systemics, Cybernetics and Informatics, vol. 2 (2004)

  19. Barford, P., Kline, J., Plonka, D., Ron, A.: In: Proceedings of ACM SIGCOMM IMW (2002)

  20. Lu, W., Ghorbani, A.: Network anomaly detection based on wavelet analysis, EURASIP. J. Adv. Signal Proc. (2009)

  21. Iliofotou, M., Pappu, P., Faloutsos, M., Mitzenmacher, M., Singh, S., Varghese, G. In: Proceedings of ACM SIGCOMM Internet Measurement Conference (2007)

  22. Cormode, G., Korn, F., Muthukrishnan, S., Srivastava, D.: In: Proceedings of VLDB (2003)

  23. Kumar, A., Xu, J., Li, L., Wang, J.: In: Proceedings of ACM SIGCOMM Conference on internet Measurement (2003)

  24. Muthukrishnan, S.: Data Streams: Algorithms and Applications. Now Publishers (2005)

  25. Cohen, E., Duffield, N., Kaplan, H., Lund, C., Thorup, M.: In: Proceedings of ACM SIGCOMM Internet Measurement Conference (2007)

  26. Ramachandran, A., Seetharaman, S., Feamster N., Vazirani, V.: In: Proceedings of ACM SIGCOMM Internet Measurement Conference (2008)

  27. Armbrust, M., Fox, A., Armando, G.R., Joseph, A.D., Katz, R.R., Konwinski, A., Lee, G., Patterson, D., Rabkin, A., Stoica, I., Zaharia, M.: Above the clouds: a berkeley view of cloud computing. Commun. ACM 53(4), 50 (2010)

    Google Scholar 

  28. Ristenpart, T., Tromer, E., Shacham, H., Savage, S.: In: Proceedings of ACM Conference on Computer and Communication Security (CCS) (2009)

  29. Mcqueen, J.: In: Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability (1967)

  30. Jin, Y., Sharafuddin, E., Zhang, Z.-L.: In: Proceedings of International Conference on emerging Networking EXperiments and Technologies (CoNext) (2007)

  31. Jin, Y., Simon, G., Xu, K., Zhang, Z.-L., Kumar, V.: In: Proceedings of USENIX NSDI Workshop on Tackling Computer Systems Problems with Machine Learning Techniques (SysML) (2007)

  32. Jin, Y., Zhang, Z.-L., Xu, K., Cao, F., Sahu, S.: In: Proceedings of ACM SIGMETRICS MineNet Workshop (2007)

  33. Yegneswaran, V., Barford, P., Ullrich, J.: In: Proceedings of ACM SIGMETRICS (2003)

  34. Pang, R., Yegneswaran, V., Barford, P., Paxson V., Peterson, L.: In: Proceedings of ACM SIGCOMM IMC (2004)

  35. Xu, K., Zhang, Z.L., Bhattacharyya, S.: In: Proceedings of Steps to Reducing Unwanted Traffic on the Internet Workshop (SRUTI) (2005)

  36. Xu, K., Zhang, Z.-L., Bhattacharyya, S.: Internet traffic behavior profiling for network security monitoring. IEEE/ACM Trans. Network. 16, 1241 (2008)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Arizona State University, 4701 W. Thunderbird Road, Glendale, AZ, 85306, USA

    Kuai Xu, Feng Wang & Haiyan Wang

Authors
  1. Kuai Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Feng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haiyan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kuai Xu.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Xu, K., Wang, F. & Wang, H. Lightweight and Informative Traffic Metrics for Data Center Monitoring. J Netw Syst Manage 20, 226–243 (2012). https://doi.org/10.1007/s10922-011-9200-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10922-011-9200-6

Keywords

Search

Navigation