Abstract
We introduce Agilis—a lightweight collaborative event processing platform that can be deployed in a Semantic Room to facilitate sharing and correlating event data generated in real time by multiple widely distributed sources. Agilis aims to balance simplicity of use and robustness on the one hand, and scalable performance in large-scale settings on the other. To this end, Agilis is built upon the open source Hadoop’s MapReduce infrastructure augmented with a RAM-based data store and several locality-oriented optimizations to improve responsiveness and reduce overhead. The processing logic is specified in a flexible high-level language, called Jaql, which supports data flows and SQL-like query constructs. We demonstrate the versatility of the Agilis framework as well as its utility for collaborative attack detection by showing how it can be leveraged in the following two attack scenarios: stealthy inter-domain port scanning, and a botnet-driven HTTP session hijacking attack. We evaluate the performance of Agilis in both these scenarios and, in the case of inter-domain port scanning, compare it to Semantic Room, which deploys the centralized high-end event processing system called Esper. Our results show that while Agilis is slower than Esper in a local area network, its relative performance improves substantially as we move toward larger scale distributed deployments.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
2000 DARPA intrusion detection scenario specific data sets. http://www.ll.mit.edu/mission/communications/ist/corpora/ideval/data/2000data.html
Hadoop. http://hadoop.apache.org/
ITOC Research CDX Datasets. http://www.itoc.usma.edu/research/dataset/index.html
Jaql. http://www.jaql.org/
JavaScript Object Notation (JSON). http://www.json.org/
DShield: Cooperative Network Security Community—Internet Security. http://www.dshield.org/indexd.html/ (2009)
Hadoop-HDFS architecture. http://hadoop.apache.org/common/docs/current/hdfs_design.html (2009)
IBM WebSphere eXtreme scale. http://www-01.ibm.com/software/webservers/appserv/extremescale/ (2009)
Where complex event processing meets open source: Esper and NEsper. http://esper.codehaus.org/ (2011)
WANem the wide area network emulator. http://wanem.sourceforge.net/ (2011)
Amini, L., Jain, N., Sehgal, A., Silber, J., Verscheure, O.: Adaptive control of extreme-scale stream processing systems. In: ICDCS’06: Proceedings of the 26th IEEE International Conference on Distributed Computing Systems, p. 71. IEEE Computer Society, Washington (2006)
Daswani, N., Stoppelman, M.: The anatomy of Clickbot.A. In: Proc. of Workshop on Hot Topics in Understanding Botnets, Berkeley, CA, USA (2007). USENIX Association
Florêncio, D.A.F., Herley, C.: A large-scale study of web password habits. In: Proceedings of the 16th International Conference on World Wide Web, WWW’07, Banff, Alberta, Canada, pp. 657–666. ACM, New York (2007)
Goodin, D.: Server-based botnet floods net with brutish SSH attacks. http://www.theregister.co.uk/2010/08/12/server_based_botnet/ (accessed on 01/24/11) (2010)
Gross, G., McMillian, R.: Five Arrested in Scam Involving Theft via Botnet, vol. 10 (2010)
Jeffrey, D., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)
Kanich, C., Kreibich, C., Levchenko, K., Enright, B., Voelker, G.M., Paxson, V., Savage, S.: Spamalytics: an empirical analysis of spam marketing conversion. In: Proc. of ACM CCS’08, pp. 3–14. ACM, New York (2008)
Jin, T., Arlitt, M.: 1998 World Cup Web Site Access Logs, August (1998)
DefenseIntelligence Matt Thompson. Mariposa Botnet Analysis, October 2009. Updated February (2010)
Microsoft. Battling Botnets for Control of Computers. Microsoft: Security Intelligence Report 9 (2010)
Weber, J.E., Paulson, R.A.: Cyberextortion: an overview of distributed denial of service attacks against online gaming companies. Issues Inf. Syst. 7, 52–56 (2006)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Esteves Verssimo, P. et al. (2012). Distributed Attack Detection Using Agilis. In: Baldoni, R., Chockler, G. (eds) Collaborative Financial Infrastructure Protection. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-20420-3_8
Download citation
DOI: https://doi.org/10.1007/978-3-642-20420-3_8
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-20419-7
Online ISBN: 978-3-642-20420-3
eBook Packages: Computer ScienceComputer Science (R0)