Abstract
Responding to or anticipating a sequence of events caused by adversarial human actors, such as crimes, can be a difficult task. Reinforcement learning has not been highly utilized as a method for positioning agents to respond to such events. In our earlier work, which was applied to positioning naval vessel agents to respond to Somali maritime piracy attacks, we developed a method to synthetically augment the information in the events’ environment with digital pheromones and other information augmenters, used the resulting augmenter signatures as states that agents could react to, and applied reinforcement learning to exploit regularities in the timing and location of events to position agents in spatio-temporal proximity of anticipated events. This work extends that methodology with a new learning boosting method wherein learning is improved as partial augmenter signatures are reinforced, which is not possible when learning is based only on the aggregated state. The enhanced methodology is applied to positioning police patrols in response to a sequence of business robberies in Denver, Colorado and its effectiveness is analyzed.
Access this article
Rent this article via DeepDyve
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Barbosa, S.E., Petty, M.D.: Reinforcement learning in an environment synthetically augmented with digital pheromones. Adv. Artif. Intell. 2014, 1–23 (2014). doi:10.1155/2014/932485
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)
Watkins, C.H., Dayan, P.: Q-learning. Mach. Learn. 8(3), 279–292 (1992)
da Silva, B.C., Basso, E.W., Bazzan, A.L.C., Engel, P.M.: Dealing with nonstationary environments using context detection. In: Proceedings of the 23rd International Conference on Machine Learning, pp 217–224 (2006)
Gordon, D.M.: Ants at Work: How an Insect Society is Organized. The Free Press, New York (1999)
Dorigo, M., Stützle, T.: Ant Colony Optimization. MIT Press, Cambridge (2004)
Huang, K.L., Liao, C.J.: Ant colony optimization combined with taboo search for the job shop scheduling problem. Comput. Oper. Res. 35(4), 1030–1046 (2008)
Lee, Z.J., Lee, C.Y., Su, S.F.: An immunity-based ant colony optimization algorithm for solving weapon-target assignment problem. Appl. Soft Comput. 2(1), 39–47 (2002)
Bautista, J., Pereira, J.: Ant algorithms for a time and space constrained assembly line balancing problem. Eur. J. Oper. Res. 177(3), 2016–2032 (2007)
Gosnell, M., O’Hara, S., Simon, M.: Spatially decomposed searching by heterogeneous unmanned systems. In: Proceedings of the International Conference on Integration of Knowledge Intensive Multi-Agent Systems (2007)
Fu, J.G.M., Ang, M.H.: Probabilistic ants (PAnts) in multi-agent patrolling. In: Proceedings of the International Conference on Advanced Intelligent Mechatronics, pp. 1371–1376 (2009)
Chu, H., Glad, A., Simonin, O., Sempe, F., Drogoul, A., Charpillet, F.: Swarm approaches for the patrolling problem, information propagation vs. pheromone evaporation. In: ICTAI’07 IEEE International Conference on Tools with Artificial Intelligence, pp 442–449 (2007)
Sauter, J.A., Matthews, R., Parunak, H.V.D., Brueckner, S.: Performance of digital pheromones for swarming vehicle control. In: Proceedings of the Conference on Autonomous Agents and Multiagent Systems, pp. 903–910 (2005)
Monekosso, N., Remagnino, P.: An analysis of the pheromone Q-learning algorithm. In: Proceedings of the Eighth Ibero-American Conference on Artificial Intelligence, pp 224–232 (2002)
Furtado, V., Melo, A., Coelho, A., Menezes, R., Perrone, R.: A bio-inspired crime simulation model. Decis. Support Syst. 48(1), 282–292 (2009)
Bowers, K.J., Johnson, S.D., Pease, K.: Prospective hot-spotting the future of crime mapping? Br. J. Criminol. 44(5), 641–658 (2004)
Li, L., Jiang, Z., Duan, N., Dong, W., Hu, K., Sun, W.: Police patrol service optimization based on the spatial pattern of hotspots. In: Service Operations, Logistics, and Informatics (SOLI), 2011 IEEE International Conference, pp. 45–50 (2011)
Jones, P.A., Brantingham, P.J., Chayes, L.R.: Statistical models of criminal behavior: the effects of law enforcement actions. Math. Models Methods Appl. Sci. 20(supp01), 1397–1423 (2010)
Mohler, G.O., Short, M.B., Brantingham, P.J., Schoenberg, F.P., Tita, G.E.: Self-exciting point process modeling of crime. J. Am. Stat. Assoc. 106(493), 100–108 (2011)
Denver Open Data Catalog, Crime Data. http://data.denvergov.org/dataset/city-and-county-of-denver-crime (2014). Accessed Feb 2014
Denver Police Department. In: Wikipedia, The Free Encyclopedia. Retrieved 13:47, 18 February 2014. http://en.wikipedia.org/w/index.php?title=Denver_Police_Department&oldid=586785112 (2013). Accessed 19 Dec 2013
Denver Police Department. http://www.denvergov.org/police (2014). Accessed Feb 2014
Johnson, S.D., Bernasco, W., Bowers, K.J., Elffers, H., Ratcliffe, J., Rengert, G., Townsley, M.: Space-time patterns of risk: a cross national assessment of residential burglary victimization. J. Quant. Criminol. 23(3), 201–219 (2007)
Bolstad, W.M.: Introduction to Bayesian Statistics. Wiley, Hoboken, New Jersey (2007)
Stone, J.V.: Bayes’ Rule: A Tutorial Introduction to Bayesian Analysis (2013)
Denver Street System. In: Wikipedia, The Free Encyclopedia. Retrieved 13:49, February 18, 2014. http://en.wikipedia.org/w/index.php?title=Street_system_of_Denver&oldid=594739065 (2014). Accessed 9 Feb 2014
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Barbosa, S.E., Petty, M.D. Exploiting spatio-temporal patterns using partial-state reinforcement learning in a synthetically augmented environment. Prog Artif Intell 3, 55–71 (2015). https://doi.org/10.1007/s13748-014-0057-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13748-014-0057-2