Abstract
In this paper we propose a novel approach to train Multi-Agent Reinforcement Learning (MARL) agents to cooperate to detect intrusions in the form of normal and abnormal states in the network. We present an architecture of distributed sensor and decision agents that learn how to identify normal and abnormal states of the network using Reinforcement Learning (RL). Sensor agents extract network-state information using tile-coding as a function approximation technique and send communication signals in the form of actions to decision agents. By means of an on line process, sensor and decision agents learn the semantics of the communication actions. In this paper we detail the learning process and the operation of the agent architecture. We also present tests and results of our research work in an intrusion detection case study, using a realistic network simulation where sensor and decision agents learn to identify normal and abnormal states of the network.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Awerbuch, B., Holmer, D., Rubens, H.: Provably Secure Competitive Routing against Proactive Byzantine Adversaries via Reinforcement Learning. John Hopkins University, Tech. Rep. (May 2003)
Barford, P., Jha, S., Yegneswaran, V.: Fusion and filtering in distributed intrusion detection systems. In: Proceedings of the 42nd Annual Allerton Conference on Communication, Control and Computing (September 2004)
Boyan, J., Littman, M.: Packet routing in dynamically changing networks: A reinforcement learning approach. Advances in Neural Information Processing Systems 6, 671–678 (1994)
Cannady, J.: Applying CMAC-based on-line learning to intrusion detection. In: Proceedings of the International Joint Conference on Neural Networks, vol. 5, pp. 405–410 (2000)
Cannady, J.: Next Generation Intrusion Detection: Autonomous Reinforcement Learning of Network Attacks. In: Proc. 23rd National Information Systems Security Conference (2000)
CheckPoint. CheckPoint, N.G.X.: Firewall SmartDefense (June 2008), http://www.checkpoint.com/products/ips-1/index.html
Cisco. Configuring Anomaly Detections (June 2008), http://www.cisco.com/en/US/docs/security/ips/6.1/configuration/guide/cli/cli_anomaly_detection.html
Dowling, J., Curran, E., Cunningham, R., Cahill, V.: Using feedback in collaborative reinforcement learning to adaptively optimize MANET routing. Systems, Man and Cybernetics, Part A, IEEE Transactions on 35(3), 360–372 (2005)
Gelenbe, E., Lent, M., Su, R.: Autonomous smart routing for network QoS. In: Proceedings of International Conference on Autonomic Computing 2004, pp. 232–239 (2004)
Ghavamzadeh, M., Mahadevan, S., Makar, R.: Hierarchical multi-agent reinforcement learning. Autonomous Agents and Multi-Agent Systems 13(2), 197–229 (2006)
Hwang, K., Tan, S., Hsiao, M., Wu, C.: Cooperative Multiagent Congestion Control for High-Speed Networks. Systems, Man and Cybernetics, Part B, IEEE Transactions on 35(2), 255–268 (2005)
Institute, S.: Sans top-20 2007 security risks, 2007 annual update (2008)
Katja Verbeeck1, P.V., Nowe, A.: Networks of learning automata and limiting games. In: Adaptive Learning Agents and Multi Agent Systems 2007, pp. 171–182 (2007)
Mirkovic, J., Reiher, P.: D WARD, A Source-End Defense against Flooding Denial of Service Attacks. Dependable and Secure Computing, IEEE Transactions on 2(3), 216–232 (2005)
Panait, L., Luke, S.: Cooperative multi-agent learning: The state of the art. Autonomous Agents and Multi-Agent Systems 11(3), 387–434 (2005)
Powers, R., Shoham, Y.: New criteria and a new algorithm for learning in multi-agent systems. Advances in Neural Information Processing Systems 17, 1089–1096 (2005)
Servin, A.L., Kudenko, D.: Multi-agent Reinforcement Learning for Intrusion Detection. In: Tuyls, K., Nowe, A., Guessoum, Z., Kudenko, D. (eds.) ALAMAS 2005, ALAMAS 2006, and ALAMAS 2007. LNCS (LNAI), vol. 4865, pp. 211–223. Springer, Heidelberg (2008)
Shoham, Y., Powers, R., Grenager, T.: If multi-agent learning is the answer, what is the question? Artificial Intelligence 171(7), 365–377 (2007)
Siaterlis, C., Maglaris, B.: Towards multisensor data fusion for dos detection. In: Proc. of the 19th ACM Symposium on Applied Computing, Nicosia, Cyprus, pp. 439–446 (2004)
N. Simulator. 2 (NS2) (January 2008), http://www.isi.edu/nsnam/
I. SourceFire. Snort (June 2008), http://www.snort.org/
Sutton, R.: Tile Coding Software, Version 2.0 (2007)
Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)
Watkins, C., Dayan, P.: Q-learning. Machine Learning 8(3), 279–292 (1992)
Xu, X., Sun, Y., Huang, Z.: Defending DDoS Attacks Using Hidden Markov Models and Cooperative Reinforcement Learning. In: Yang, C.C., Zeng, D., Chau, M., Chang, K., Yang, Q., Cheng, X., Wang, J., Wang, F.-Y., Chen, H. (eds.) PAISI 2007. LNCS, vol. 4430, p. 196. Springer, Heidelberg (2007)
Xu, X., Xie, T.: A Reinforcement Learning Approach for Host-Based Intrusion Detection Using Sequences of System Calls. In: Proceedings of the International Conference on Intelligent Computing (2005)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Servin, A., Kudenko, D. (2008). Multi-Agent Reinforcement Learning for Intrusion Detection: A Case Study and Evaluation. In: Bergmann, R., Lindemann, G., Kirn, S., Pěchouček, M. (eds) Multiagent System Technologies. MATES 2008. Lecture Notes in Computer Science(), vol 5244. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-87805-6_15
Download citation
DOI: https://doi.org/10.1007/978-3-540-87805-6_15
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-87804-9
Online ISBN: 978-3-540-87805-6
eBook Packages: Computer ScienceComputer Science (R0)