Abstract
This work presents several scenarios used to identify security incidents based on the analysis of web server log files. The main goal of this work is to identify security events triggered by web robots which can be considered as dangerous or unwelcome. Analysis of all security incidents was based on archived web server log files which were collected from 03.03.2014 to 31.01.2015 and came from the real and fully functional environment, available at www.darmowe-obrazki.pl. All data were obtained automatically on a daily basis and analyzed using Advanced Web Statistics software.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
HTML 4.01 Specification—Appendix B: Performance, Implementation, and Design Notes—B.4.1 Search robots, W3C (1999). http://www.w3.org/TR/html4/appendix/notes.html#h-B.4.1.1
Josh, U.A.: Googlebot is Chrome (2011). http://ipullrank.com/googlebot-is-chrome
Koster, M.: A Method for Web Robots Control, Network Working Group, Internet draft (1996). http://www.robotstxt.org/norobots-rfc.txt
Koster, M.: A Standard for Robot Exclusion, Internet draft (1994). http://www.robotstxt.org/orig.html
LaMacchia, B.A.: Internet fish. Ph.D. thesis, Artificial Intelligence Laboratory and Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology (1996). http://www.farcaster.com/papers/ifish/ifish-tr.pdf
Majestic-12: DSearch: MJ12bot—How can I block MJ12bot? (2014). http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
Martijn Koster—Wikipedia, the free encyclopedia. http://en.wikipedia.org/wiki/Martijn_Koster. Accessed July 2014
Robots exclusion standard—Nonstandard extensions—Wikipedia, the free encyclopedia. http://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directive. Accessed Feb 2015
Scrapy 0.24.4 Documentation—Settings (ROBOTSTXT_OBEY). http://doc.scrapy.org/en/latest/topics/settings.html. Accessed Jan 2015
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing Switzerland
About this paper
Cite this paper
Orzeł, M.J., Kołaczek, G. (2017). Detection of Security Incidents in a Context of Unwelcome or Dangerous Activity of Web Robots. In: Zgrzywa, A., Choroś, K., Siemiński, A. (eds) Multimedia and Network Information Systems. Advances in Intelligent Systems and Computing, vol 506. Springer, Cham. https://doi.org/10.1007/978-3-319-43982-2_19
Download citation
DOI: https://doi.org/10.1007/978-3-319-43982-2_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-43981-5
Online ISBN: 978-3-319-43982-2
eBook Packages: EngineeringEngineering (R0)