Deep Q-network-based multi-criteria decision-making framework for virtual simulation environment

Abstract

Deep learning improves the realistic expression of virtual simulations specifically to solve multi-criteria decision-making problems, which are generally rely on high-performance artificial intelligence. This study was inspired by the motivation theory and natural life observations. Recently, motivation-based control has been actively studied for realistic expression, but it presents various problems. For instance, it is hard to define the relation among multiple motivations and to select goals based on multiple motivations. Behaviors should generally be practiced to take into account motivations and goals. This paper proposes a deep Q-network (DQN)-based multi-criteria decision-making framework for virtual agents in real time to automatically select goals based on motivations in virtual simulation environments and to plan relevant behaviors to achieve those goals. All motivations are classified according to the five-level Maslow’s hierarchy of needs, and the virtual agents train a double DQN by big social data, select optimal goals depending on motivations, and perform behaviors relying on a predefined hierarchical task networks (HTNs). Compared to the state-of-the-art method, the proposed framework is efficient and reduced the average loss from 0.1239 to 0.0491 and increased accuracy from 63.24 to 80.15%. For behavioral performance using predefined HTNs, the number of methods has increased from 35 in the Q network to 1511 in the proposed framework, and the computation time of 10,000 behavior plans reduced from 0.118 to 0.1079 s.

Appendix

The virtual simulation environment for this study, shown in Fig. 15, was implemented using the commercial game engine Unity 2017 (Unity Technologies ApS, San Francisco, CA, USA). The virtual environment demonstrates a small city. In the virtual city, we simulate actions of humans and animals on the road areas. In addition, there are eight junctions including six T junctions and two cross-junctions. Each T junction contains nine traffic lights, and each cross junction contains twelve. Therefore, a total of 78 traffic lights are utilized in the simulator.

Fig. 15

Virtual simulation environment

Table 19 lists screens of behaviors, results, motivations, goals, and behavior outcomes when the virtual human agent was activated in the simulation after learning using the proposed framework.

Table 19 Motivation, goal, and behavior result of the virtual human gent in the proposed framework

For instance, Fig. 16a shows virtual human agents commuting to offices or schools in the morning. Figure 16b–d illustrate the virtual human agents having meals at noon, returning home in the evening, and going to sleep later in the evening, respectively.

Fig. 16

Virtual human agents selecting goals and performing behaviors

Table 20 lists the behaviors, goals, and top-priority motivations of the virtual animal during simulation after learning using the proposed framework.

Table 20 Motivation–goal–behavior results of the virtual animal in the proposed framework

Figure 17a, b illustrate that multiple virtual animal agents tend to move in groups using the proposed framework. An interaction between virtual animal and human agents is shown in Fig. 17c, and the behavior of only one virtual animal agent is illustrated in Fig. 17d.

Fig. 17

Goal selection and behavior performance of the virtual animal