Skip to main content
SpringerLink
Log in

Declarative platform for high-performance network traffic analytics

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

This paper presents Scalanytics, a declarative platform that supports high-performance application layer analysis of network traffic. Scalanytics uses (1) stateful network packet processing techniques for extracting application layer data from network packets, (2) a declarative rule-based language called Analog for compactly specifying analysis pipelines from reusable modules, and (3) a task-stealing architecture for processing network packets at high throughput within these pipelines, by leveraging multi-core processing capabilities in a load-balanced manner without the need for explicit performance profiling. In a cluster of machines, Scalanytics further improves throughput through the use of a consistent-hashing based load partitioning strategy. Our evaluation on a 16-core machine demonstrate that Scalanytics achieves up to 11.4\(\times \) improvement in throughput compared with the best uniprocessor implementation. Moreover, Scalanytics outperforms the Bro intrusion detection system by an order of magnitude when used for analyzing SMTP traffic. We further observed increased throughput when running Scalanytics pipelines across multiple machines.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. The FLOW_ID attribute is a concatenation of the source and destination IP addresses and port numbers, and used to uniquely identify a TCP or SMTP session.

References

  1. Dobrescu, M., Egi, N., Argyraki, K., Chun, B.G., Fall, K., et al.: RouteBricks: exploiting parallelism to scale software routers. In: SOSP (2009).

  2. Wolf, T., Weng, N., Tai, C.H.: Runtime support for multicore packet processing systems. Netw. IEEE 21(4), 29–37 (2007)

    Article  Google Scholar 

  3. Kohler, E., Morris, R., Chen, B., Jannotti, J., Kaashoek, M.F.: The click modular router. ACM Trans. Comput. Syst. 18(3), 263–297 (2000)

    Google Scholar 

  4. Loo, B.T., Condie, T., Garofalakis, M., Gay, D.E., Hellerstein, J.M., Maniatis, P., Ramakrishnan, R., Roscoe, T., Stoica, I.: Declarative networking. CACM (2009).

  5. Sahami, M., Dumais, S., Heckerman, D., Horvitz, E.: A bayesian approach to filtering junk e-mail. In: Learning for Text Categorization: Papers from the 1998 workshop, vol. 62, pp. 98–105. Madison, Wisconsin: AAAI Technical, Report WS-98-05 (1998).

  6. SIP: Session Initiation Protocol: http://www.ietf.org/rfc/rfc3261.txt, Accessed 25 Sept 2013

  7. Mukkamala, S., Sung, A.H.: Detecting denial of service attacks using support vector machines. In: IEEE International Conference on Fuzzy Systems (IEEE FUZZ) (2003).

  8. Chang, C.C., Lin, C.J.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2(27), 1–27 (2011)

    Google Scholar 

  9. DARPA Intrusion Detection Dataset: http://www.ll.mit.edu/mission/communications/ist/corpora/ideval/data/index.html, Accessed 25 Sept 2013

  10. Reinders, J.: Intel thread building blocks. In: OReilly Associates (2007).

  11. RapidNet: a declarative toolkit for rapid network simulation and experimentation: http://netdb.cis.upenn.edu/rapidnet/, Accessed 25 Sept 2013

  12. Paxson, V.: Bro: a system for detecting network intruders in real-time. Comput. Netw. 31(23–24), 2435–2463 (1999)

    Article  Google Scholar 

  13. Gill, H., Lin, D., Han, X., Nguyen, C., Gill, T., Loo, B.T.: Scalanytics: a declarative multi-core platform for scalable composable traffic analytics. In: 22nd International ACM Symposium on High Performance and Distributed Computing (HPDC) (2013).

  14. Frigo, M., Leiserson, C.E., Randall, K.H.: The implementation of the Cilk-5 multithreaded language. In: PLDI (1998).

  15. Ramakrishnan, R., Ullman, J.D.: A survey of research on deductive database systems. J. Logic Program. 23, 125–149 (1993)

    Google Scholar 

  16. Przymusinski, T.C.: On the declarative semantics of deductive databases and logic programs, pp. 193–216. Morgan Kaufmann Publishers Inc., San Francisco, CA (1988)

    Book  Google Scholar 

  17. Stoica, I., Morris, R., Karger, D., Kaashoek, M.F., Balakrishnan, H.: Chord: a scalable peer-to-peer lookup service for internet applications (2001).

  18. Libnids: http://libnids.sourceforge.net/, Accessed 25 Sept 2013

  19. Bro Intrusion Detection System. http://bro-ids.org, Accessed 25 Sept 2013

  20. GNU oSIP library. http://www.gnu.org/software/osip, Accessed 25 Sept 2013

  21. SpamAssassin Dataset: http://spamassassin.apache.org/publiccorpus, Accessed 25 Sept 2013

  22. SIPp Open Source test tool / traffic generator for the SIP protocol.: http://sipp.sourceforge.net/, Accessed 25 Sept 2013

  23. Intel VTune Amplifier XE 2013. http://software.intel.com/en-us/intel-vtune-amplifier-xe/, Accessed 25 Sept 2013

  24. Chen, B., Morris, R.: Flexible control of parallelism in a multiprocessor pc router. In: USENIX ATC (2001).

  25. Chen, X., Wu, Y., Xu, L., Xue, Y., Li, J.: Para-snort: a multi-thread snort on multi-core ia platform. In: Proceedings of Parallel and Distributed Computing and Systems (PDCS) (2009).

  26. Vasiliadis, G., Antonatos, S., Polychronakis, M., Markatos, E.P., Ioannidis, S.: Gnort: High performance network intrusion detection using graphics processors. In: RAID (2008).

  27. Vasiliadis, G., Polychronakis, M., Loannidis, S.: Midea: a multi-parallel intrusion detection architecture. In: Proceedings of the 18th ACM conference on Computer and communications security, CCS’11, pp. 297–308. ACM, New York, NY (2011). doi:10.1145/2046707.2046741.

  28. Sommer, R., Paxson, V., Weaver, N.: An architecture for exploiting multi-core processors to parallelize network intrusion prevention. Practice and Experience, Concurrency and Computation (2009)

  29. OpenFlow. http://www.openflow.org/

Download references

Acknowledgments

This project is supported in part by NSF Grants CNS-1218066, CNS-1117185, CNS-1117052, CAREER CNS-0845552, DARPA SAFER award N66001-C-4020, and a AFOSR Young Investigator Award FA9550-12-1-0327.

Author information

Authors and Affiliations

  1. Computer and Information Science Department, University of Pennsylvania, 3330 Walnut Street, Philadelphia, PA, 19104, USA

    Harjot Gill, Dong Lin, Cam Nguyen, Tanveer Gill & Boon Thau Loo

Authors
  1. Harjot Gill
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dong Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cam Nguyen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tanveer Gill
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Boon Thau Loo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Boon Thau Loo.

Appendices

Appendix 1: SMTP example in Analog

figure i

Appendix 2: SIP example in Analog

figure j

Appendix 3: DoS Example in Analog

figure k

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gill, H., Lin, D., Nguyen, C. et al. Declarative platform for high-performance network traffic analytics. Cluster Comput 17, 1121–1137 (2014). https://doi.org/10.1007/s10586-014-0363-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-014-0363-2

Keywords

Search

Navigation