Elsevier

Expert Systems with Applications

Volume 71, 1 April 2017, Pages 370-382
Expert Systems with Applications

Hybrid Multi-Agent Strategy Discovering Algorithm for human behavior

https://doi.org/10.1016/j.eswa.2016.11.036Get rights and content

Highlights

  • An algorithm for analyzing behavior of security teams during training is proposed.

  • Cognitive and emotional properties of agents are considered during analysis.

  • HMASDA extracts strategies in the form of physical and mental behavioral patterns.

  • Human-understandable descriptions of the strategies used are constructed.

Abstract

Training in simulators through serious games is widely used in domains where it is too dangerous to train in a real environment. Simulations can help to model complex social and psychological aspects and can enable repetitiveness during game-based learning, which is especially important when opposing or cooperating humans can get hurt. When a trainee team interacts with other humans or software agents with human-like performance, cognitive and psychological properties and interactions that arise in various situations play an important role in serious game training. Therefore, special tools and methods that integrate physical and cognitive activities need to be developed in order to analyze the way trainees tackle the scenario. We have addressed these problems with the Hybrid Multi-Agent Strategy Discovering Algorithm (HMASDA), which builds upon an existing algorithm for physical strategy identification (MASDA) by adding the ability to process and consider cognitive models. To include the cognitive behavior of trainees, and to identify integrated policies based on their overall behavior, we introduced additional features that take into account the trainees’ cognitive state, their well-being, and their emotional reactions. Using a predefined asymmetric conflict scenario, we demonstrate that it is possible to obtain physical and cognitive descriptions of the behavior that trainees display.

Introduction

Extensive training of personnel in simulated environments is becoming increasingly important in several domains. This kind of training offers several advantages over conventional training. It enables personnel to achieve the required skills faster and train in a safe environment for a wide set of initial conditions, scenarios, and events. For example, in military operations in urban terrain, mission-analysis simulations (Page & Smith, 1998) are employed to prepare trainees in accordance with specific military doctrines in difficult environments, and facing diverse enemies with different cultural and social backgrounds. In a similar way, flight simulations are used extensively in the aviation industry to train civil and military pilots, as well as other crew members, for various possible scenarios or airport traffic simulations (Davidrajuha & Lin, 2011) to verify the air traffic management capabilities of airports and their staff. Modern variations of training simulations involve the use of serious games, which are essentially interactive high-fidelity video games that reproduce the environment, the physics of objects and the definition of the scenario. Although such form of training are gaining importance, the literature includes few serious game simulations for various events and situations involving interactions of groups of antagonistic cognitive agents (Chittaro, Sioni, 2015, Ribeiro, Almeida, Rossetti, Coelho, Coelho, 2012; Silva, Almeida, Pereira, Rossetti and Coelho, 2013; Smith, Trendholme, 2009, Wei, Chen, Cruz, Hayes, Kruger, Blasch, 2008, van der Zee, Holkenborg, Robinson, 2012). Furthermore, in confrontations between groups of humans, several factors must be taken into consideration, including physical and cognitive components. After all, beliefs, feelings, motivations and psychology play an important role in any human group conflict (Fisher, 2000, Power, 2011).

In this paper we address the issue of the efficient and human-transparent analysis of human training sessions in serious games using artificial intelligence (AI) programs to learn and describe human performance, where cognitive properties play an important role; that is, they influence both the effectiveness of learning and the preparedness of the trainees (Worley, Wahlman, & Gleeson, 2000). When humans are involved in game-based learning, it is also important to consider the social interactions and cognitive processes that lead to the adoption of specific behavior during the performance evaluation of the trainees. Different authors have adopted various mechanisms to analyze the trainees’ performance. Most of these mechanisms evaluate key metrics after the simulation training (Chittaro, Sioni, 2015, Patel, Gallagher, Nicholson, Cates, 2006, Qudrat-Ullah, 2014, Rosen, Weaver, Lazzara, Salas, Wu, Silvestri, et al., 2010, Tavcar, Kaluza, Kvassay, Schneider, Gams, 2014), and provide them with questionnaires (Papadopoulos, Pentzou, Louloudiadis, & Tsiatsos, 2013); however, these methods do not directly address the cognitive interactions between trainees. We propose a different approach that analyses the entire course of the simulation and provides insights regarding the behavior of the observed trainees. This makes it possible to assess their performance over the entire progress of the simulation, to analyze the constructed behavior strategies on their own, and to provide descriptions for the specific physical and cognitive behavior traits identified during the evaluation phase. Experts can use the provided information to understand the conditions that trigger a specific behavior by the trainees, to learn how they perform under psychological strain, to follow their learning curve, and to create new training scenarios based on their progress. The idea is to treat human performance as consisting of various types (such as physical, cognitive, social, etc.), and to treat each of these aspects as one of the equally important parts of the integrated behavior.

For the effective analysis of simulation runs, we propose the Hybrid Multi-Agent Strategy-Discovering Algorithm (HMASDA), which exploits hierarchically ordered domain knowledge to generate graphical and symbolic descriptions of strategic behavior. The algorithm builds upon the existing Multi-Agent Strategy-Discovering Algorithm (MASDA) (Bezek, Gams, 2005, Bezek, Gams, Bratko, 2006), which is for physical strategy identification only. We have introduced several modifications that enable the algorithm to integrate cognitive and psychological components with operational ones, thereby allowing a more efficient capture of their behavior.

The HMASDA method was evaluated on a MOUT (military operations on urban terrain) domain, where the goal is to perform mission analyses and training for squads (for example, a squad leader plus four security personnel) facing threats such as riots or looting of shops that can evolve in asymmetric scenarios in urban environments. The scenario is implemented in a serious game that enables real security personnel to train and learn how to resolve the problem. The training system allows the exportation of the traces (movements, parameters, properties, and actions of all types performed), which are in turn used to analyze integrated behavior during the simulation.

The rest of the paper is structured as follows: first, we provide a formal definition of the problem. In the next section, we briefly describe the domain and the scenarios used to evaluate the algorithm. A review of the related literature is provided in Section 4; and the HMASDA algorithm is presented in detail in 5 Hybrid Multi-Agent Strategy Discovering Algorithm, 6 Experimental setup and 7 are dedicated to experimentation and validation activities; finally, we summarize the findings and present some ideas for future work.

Section snippets

Problem definition

The task is to discover and highlight the predominant behavior patterns and learned strategic trajectories, and to provide human-understandable explanations just from analyzing low-level traces of physical, cognitive and social actions of agents. These traces are obtained from the interaction of cognitive agents – either from observing real humans, or from simulated agents. The formal definition of an agent is taken from Lettmann, Baumann, Eberling, and Kemmerich (2011), with extensions related

Motivating domain

In military and security applications, an important part of the squad members‘ training is engagement in the MOUT (Military Operations on Urban Terrain) domains. Exhaustive training is necessary due to the practically impossible constraints; therefore, MOUT multi-agent simulations are often introduced where a human is assigned the role of squad leader, while the roles of peace keepers are assigned to other humans or agents that interact with hostile humans, represented as cognitive agents (

Related work

This work is also closely linked to agent-behavior modeling, which has been studied in various domains. For example, the RoboCop simulation soccer game spurred many successful efforts towards the use of learned-opponent models to adapt the team for adversarial strategies. Given a set of segments from simulation logs, agglomerative clustering was applied to classify the team’s behavior, and to generate counter tactics (Erdogan & Veloso, 2011). Support vector machine have been used to model

Hybrid Multi-Agent Strategy Discovering Algorithm

The discovery of informative strategic integrated behavior is performed using the HMASDA. The major improvements to the basic algorithm involve the introduction of cognitive ontologies, and features, an upgrade of the algorithm to combine physical and cognitive aspects during the behavior analysis, and the ability to cope with large action graphs (AG). The central task is to transform raw multi-agent physical and mental action traces and states to a sequence of strategic action concepts that

Experimental setup

The goal of the experimental evaluation was to discover if a computer algorithm is able to grasp and meaningfully present the strategic behavior of real human experts solely from observations of their physical actions and the provided cognitive features of the civilian opponents. The task was made more complicated by introducing two distinct overall strategies. In this way, some low-level patterns, for example, how to deal with a single rioter, were the same in both strategies, while there were

Evaluation of discovered patterns

Our intention was to demonstrate that HMASDA is able to capture complex integrated behavior displayed by interacting groups of humans at the mid- and strategic levels, and to present it in a transparent way. Under the given conditions, HMASDA constructed around 100 middle-level and five semi-strategic patterns. In the next sections, we present a subset of the discovered patterns.

Conclusion and future work

The HMASDA algorithm for the analysis of the behavior of trainees from simulation runs in serious games was initially introduced as a cognitive extension of previous algorithms such as MASDA. This is because interactions between two opposing groups of humans are largely based on psychological, cognitive, and emotional factors. However, it soon became clear that instead of upgrading the previous algorithm, a novel algorithm had to be designed and implemented from scratch using only knowledge

Acknowledgment

This work was partially supported by the EDA project A-0938-RT-GC EUSAS.

References (43)

  • WeiM. et al.

    Game-theoretic modeling and control of military operations with partially emotional civilian players

    Decision Support Systems

    (2008)
  • D. Avrahami-Zilberbrand et al.

    Fast and complete symbolic plan recognition

    Proceedings of the international joint conference on artificial intelligence

    (2005)
  • A. Bezek et al.

    From basic agent behaviour to strategic patterns in a robotic soccer domain

    Informatica

    (2005)
  • A. Bezek et al.

    Multi-agent strategic modeling in a robotic soccer domain

    Proceedings of the fifth international joint conference on autonomous agents and multi-agent systems, AAMAS

    (2006)
  • H. Camacho-Colladosa et al.

    A decision support system for predictive police patrolling

    Decision Support Systems

    (2015)
  • O. Champalle et al.

    Facilitate sharing of training experience by exploring behavior discovery in trainees traces

    Design for teaching and learning in a networked world

    (2015)
  • W.W. Cohen et al.

    A simple, fast, and effective rule learner

    Proceedings of the sixteenth national conference on artificial intelligence

    (1999)
  • T.M. Connollya et al.

    A systematic literature review of empirical evidence on computer games and serious games

    Computers and Education

    (2012)
  • S. Cresswell et al.

    Generalised domain model acquisition from action traces

    Proceedings of the international conference on automated planning and scheduling

    (2011)
  • R. Davidrajuha et al.

    Exploring airport traffic capability using petri net based model

    Expert Systems with Applications

    (2011)
  • C. Erdogan et al.

    Action selection via learning behavior patterns in multi-robot systems

    Proceedings of the twenty second international joint conference on artificial intelligence

    (2011)
  • Cited by (8)

    • Principles for small-unit sUAS tactical deployment from a combat-simulating agent-based model analysis

      2022, Expert Systems with Applications
      Citation Excerpt :

      Of particular relevance to this work are ABMs that explore tactical combat situations and sUAS combat applications. Such models emphasize the actions of individual warfighters and small units of warfighters (Cil & Mala, 2010) and explore the impacts of the environment and developing scenario on warfighter behavior and decision making (Tavcar et al., 2017). They also explore possible applications for sUAS when deployed in coordination with ground forces (Muratore et al., 2014).

    • Surrogate-agent modeling for improved training

      2018, Engineering Applications of Artificial Intelligence
      Citation Excerpt :

      This paper proposes a novel approach called PATDS (Performance Analysis of Training through Data-Farming Simulations) that addresses the above-mentioned challenges using the following three key ideas. First, the high-level behavior patterns of the trainee are extracted from simulation logs using the Hybrid Multi-Agent Strategy Discovering Algorithm, HMASDA (Tavcar et al., 2017), which exploits hierarchically ordered domain knowledge to generate graphical and symbolic descriptions of strategic behavior. Second, the identified behavior patterns are transferred into a surrogate agent (SA) that is able to reproduce the behavior of the trainee.

    • Secure agent-based diagnosis and classification using optimal kernel support vector machine

      2021, International Journal of Biomedical Engineering and Technology
    • Micro-behavior Analysis of Pilots Based on Human-Aircraft-Environment Interaction

      2020, Transactions of Nanjing University of Aeronautics and Astronautics
    View all citing articles on Scopus
    View full text