A closed-loop healthcare processing approach based on deep reinforcement learning

Abstract

In healthcare, the human body is a controlled input-output system, which generates different observations with the variations of external interventions. The intervention acts as the input, and the output is the phenotype observation that reflects the latent health state of the body system. The objective of healthcare is to determine effective intervention strategies that can nurse an unhealthy human body to a healthy state. With the advances of Internet-of-Things (IoT) and body sensor networks, it becomes convenient to observe the multimedia data of the human body anywhere and anytime. To aid healthcare decision making, we put forward to construct the human body simulators based on deep neural networks (DNNs) for healthcare research. At first, we formulate the model of the human body system based on DNNs. During our analysis, we realize that DNN-based models could simulate practical situations, e.g. some health states are unreachable. Then, we combine deep reinforcement learning (DRL) with conceptual embedding techniques to explore effective healthcare strategies for simulated human bodies. We implement a virtual human body simulator, which can take interventions and represent its hidden states by high-dimensional images, and a DRL-based treatment module, which can diagnose latent health state through the image observations and choose interventions to nurse the simulated body to a target state. By combining the body simulator and treatment module, we create a dynamic closed-loop for healthcare information processing. Experimental simulations are performed to validate the feasibility of the offered approach.

Appendix

1.1 The generated tongue images of decoding network

Given a 9-dimensional health state vector, the decoding network will generate a 32 × 32 × 3 tongue image with corresponding features. The typical tongue images of nine BC types are illustrated in Fig. 8. The representation space of the generated tongue images by the trained decoding network are illustrated in Fig. 9.

Fig. 8

Nine typical tongue images belong to different BC types. The leftmost tongue image is a balanced type, named Gentleness, and the other tongue images belong to unbalanced BC types

Fig. 9

Illustration of the generated tongue images of a trained decoding network. The images of the leftmost column are gentleness type h⁽¹⁾ corresponding to a state vector (1,0,0,0,0,0,0,0,0), and the images of the rightmost column are other sub-healthy BC types, corresponding to $\boldsymbol {h}^{(2)} \thicksim \boldsymbol {h}^{(9)}$. For each line, the images are generated from interpolated codes that change the representation vector of gentleness to vectors of the other BC type by a value of 0.1. E.g., in the first line, the latent state vector of the second tongue image from left is (0.9,0.1,0,0,0,0,0,0,0)

1.2 Different scales of regulating network

The scale of hidden layer in the regulating network can affect the complexity of the treatment. The intervention dimension also can affect the probability of finding a better healthcare strategy. We generate 5 random regulating networks for different scales respectively, and use the conceptual alignment DDPG to treat the same model five times. The best converged results are illustrated in Tables 9, 10 and 11.

Table 9 The best converged results of 5 random regulating networks with 20-dimensional input and 20 hidden neurons

Table 10 The best converged results of 5 random regulating networks with 20-dimensional input and 50 hidden neurons

Table 11 The best converged results of 5 random regulating networks with 20-dimensional input and 100 hidden neurons

Table 12 The best converged results of 5 random regulating networks with 10-dimensional input and 20 hidden neurons

Table 13 The best converged results of 5 random regulating networks with 50-dimensional input and 20 hidden neurons