Transfer physics informed neural network: a new framework for distributed physics informed neural networks via parameter sharing

Abstract

We introduce Transfer Physics Informed Neural Network (TPINN), a neural network-based approach for solving forward and inverse problems in nonlinear partial differential equations (PDEs). In TPINN, one or more layers of physics informed neural network (PINN) corresponding to each non-overlapping subdomains are changed using a unique set of parameters for each PINN. The remaining layers of individual PINNs are the same through parameter sharing. The subdomains can be those obtained by partitioning the global computational domain or subdomains part of the problem definition, which adds to the total computational domain. Solutions from different subdomains are connected while training using problem-specific interface conditions incorporated into the loss function. The proposed method handles forward and inverse problems where PDE formulation changes or when there is a discontinuity in PDE parameters across different subdomains efficiently. Parameter sharing reduces parameter space dimension, memory requirements, computational burden and increases accuracy. The efficacy of the proposed approach is demonstrated by solving various forward and inverse problems, including classical benchmark problems and problems involving parameter heterogeneity from the heat transfer domain. In inverse parameter estimation problems, statistical analysis of estimated parameters is performed by solving the problem independently six times. Noise analysis by varying the noise level in the input data is performed for all inverse problems.

Appendices

Appendices

Appendix 1: Conjugate heat transfer in a 2D counter-flow heat exchanger

Conjugate heat transfer problems come across in many practical applications, where heat conduction in a solid region is tightly coupled with convection heat transfer in an adjacent fluid region. Common examples include heat transfer enhancement with a finned surface, cooling of microelectronic chips, heat transfer in a cavity with thermally conducting walls or internal baffle, heat transfer in the workpiece for arc welding etc.

We try to solve a conjugate heat transfer problem in a two-dimensional counter-flow heat exchanger using our approach. This problem is discussed and solved using a SIMPLE-like algorithm in Chen et al. [46]. The problem’s geometry is shown in Fig. 26a. The following assumptions are made for the present problem; steady-state condition, no surface heat source, phase change, or fluid dissipation (or suction) at the solid-fluid interface. Solid and fluid properties are assumed to be constant, although the solid material properties may differ from those of the liquid. Both fluid flows are assumed to be 2D laminar flow through a straight channel occurring in the x direction. The governing equations are Eqs. (14–16, 18). For solid region these equations are still valid by putting u = v = 0 as convection is absent there. Also, there is no heat generation anywhere, so Q = 0. In Fig. 26a bottom fluid is cold, and the top fluid is hot. The upper and lower parallel flow passages with widths $\Delta _1$ and $\Delta _3$ are separated by a metallic plate with a thickness of $\Delta _2$.

Fig. 26

Geometry, temperature comparison, and TPINN solution for conjugate heat transfer problem

The outer walls of the heat exchanger are assumed to be adiabatic. It is assumed that hot and cold fluids possess the same properties. Uniform velocity and temperature inlet conditions, zero outlet pressure conditions are applied for both fluids. It should be noted that the direction of velocities should be considered while applying boundary conditions by changing the signs. The values of the variables for the current problem are as follows: $\Delta _1$ = $\Delta _2$ = $\Delta _3$ = 0.1 m and l = 1 m; fluid parameters are $U_1$ = 0.2 m/s, $T_1$ = 800 K, $U_3$ = 0.1 m/s, $T_3$ = 300 K, $\rho _{\text {f}}$ = 1000 kg/m$^3$, $k_{\text {f}}$ = 10 W/(m K), $C_\mathrm{pf}$ = 25 J/(kg K) and $\mu _{\text {f}}$ = 0.15 kg/(ms); solid properties $\rho _s$ = 8000 kg/m$^3$, $k_{\text {s}}$ = 50 W/(m K) and $C_\mathrm{ps}$ = 500 J/(kg K). All the physical dimensions, velocity, temperature values are scaled in the range of 0 to 1 using Min–Max scaling, and corresponding scaled continuity, x momentum, y momentum, and energy equations are used. The cold fluid, solid region, and hot fluid are taken as subdomain 1,2,3, respectively. The neural network of TPINN with the first hidden layer change contains 3 hidden layers with 15 neurons in each of them. Input layer takes x, y co-ordinates and output layer gives x component velocity u, pressure p and temperature T. y component velocity v for this problem is zero everywhere. We used 1444 random interior data points and 40 random data points along each boundary and interface per subdomain for training the neural networks. Penalty factors are $\lambda _{\text {pde}}$ = 1, $\lambda _{\text {bc}}$ = 2, $\lambda _{\text {if}}$ = 1. Optimization is carried out initially using Adam optimizer with a learning rate of 0.001 for 30000 iterations, followed by the SLSQP optimizer. Figure 26b shows the comparison of temperature profile at x = l/2. The predicted temperature profile is in good agreement with those reported in [46]. Fig. 26c shows the contour plot of predicted temperature across the whole domain. The contour plot is matching with that reported in [46]. We can observe that the temperature is not bounded at the inlet of both hot and cold fluid. It is because uniform inlet temperature boundary condition enforces a zero temperature slope condition. On the other hand, the conservation of heat flux condition at the interface enforces a non-zero temperature slope condition. So we have conflicting boundary conditions and interface flux conditions.

Hence, the neural network cannot strictly apply the uniform temperature inlet condition, and because of that, the temperature goes out of bounds at the inlets.

Appendix 2: Estimating parameter of temperature dependent thermal conductivity in a 1D steady-state heterogeneous heat conduction

We shall consider the inverse problem of estimating the parameter of temperature-dependent thermal conductivity in one-dimensional steady-state heat conduction through a two-layered composite slab with no heat generation. Figure 23 can be considered as the geometry for the present problem with a = 0.5. The non dimensional temperatures are $T_l$ = 0 to $T_r$ = 1. The interface is at x=0.5. A perfect thermal contact at the interface is assumed. For subdomains 1 and 2, the thermal conductivity functions are assumed to be of the form $k_1(T) = e^{a_1T}$ and $k_2(T) = e^{a_2T}$. True values are $a_1$ = 0.28 and $a_2$ = 1.2. In this problem, estimating the values of $a_1$ and $a_2$ using noisy temperature and heat flux measurements from the interior of the subdomains is considered. Input data are synthetic observations obtained by adding error to true heat flux values and uncorrelated Gaussian noise to the exact temperature values. Each slab is taken as a subdomain here also. TPINN with first layer change is adopted here. The architecture is depth = 3 and width = 16. For subdomains 1 and 2, heat flux and temperature values are available at x = $\{$0.02, 0.25, 0.48$\}$ and x = $\{$0.52, 0.75, 0.98$\}$ respectively. $\therefore$ $N_{\text {domain}}^j$ = 3 in Eq. (11), $N_{\text {flux}}^j$ = 3 in Eq. (12) $\forall$ j. Along with the locations where temperature and heat flux observations are available, $J_{\text {pde}}$ in Eq. (6) is calculated at 20 more random x data points from each subdomain. So $N_{jpde}^j$=23 in Eq. (6) $\forall$ j. The parameters $a_1$, $a_2$ to be estimated are set as trainable parameter of corresponding neural networks and are initialized with a value of 3.0. [1,1.5,1] are the penalty factors used for $J_{\text {pde}}$, $J_{\text {domain}}$, and $J_{\text {flux}}$ respectively. Optimization is carried out using Adam optimizer with a learning rate of 0.001 up to 7000 iterations, followed by the SLSQP optimizer. The weights are initialized using Xavier initialization approach. The present problem is solved independently for 6 times and the statistics of the estimated parameters for a confidence level of 95% are shown in Table 7. The relative error of mean is calculated w.r.t the corresponding true parameter value. Average MSE reported are calculated using $T_{\text {prediction}}$ & $k_{\text {prediction}}$ on a test data set consisting of 20 uniformly spaced x values from each subdomain. We can notice that TPINN has predicted the parameters with good accuracy when the noise level is 0%. When the input data is corrupted with error, the accuracy falls. The confidence interval width appears to be increasing as the noise level increases, which is expected. The relative error of the mean value is high for the first subdomain than the second subdomain for all noise levels.

Table 7 Statistics of parameters estimated by TPINN for a confidence level of 95% for Appendix 2. Training was done on Google Colab instance: Intel(R) Xeon(R) CPU with two cores @2.3GHz and 13GB RAM

However, the MSE of $k_{\text {prediction}}$ is comparatively high for subdomain 2, which can be due to the fact that thermal conductivity is more sensitive to the corresponding parameter in the second subdomain than the first subdomain.

Solution using conventional approach

The conventional approach for solving inverse parameter estimation problems involves a forward model to solve the corresponding forward problem. An inverse model is there, which updates the unknown parameter through an optimization algorithm. A stopping criterion is set to terminate the iterative procedure. The conventional approach can be described as:

• Step 1: Obtain the heat flux and temperature values at the measurement locations by running the forward model using the available values or initial values of parameters.

• Step 2: Calculate the objective function value representing the error in the heat flux and temperature values estimated from the forward model w.r.t the available measurements.

• Step 3: Update the values of parameters using the inverse model to minimize the objective function. Sensitivity coefficients calculation may be required depending on the inverse model.

• Step 4: Using the updated values of parameters solve the forward problem and calculate the objective function value.

• Step 5: Check the satisfaction of stopping criteria. If no, go to step 3. If yes, take the latest values of parameters as estimated values, and terminate the iterative procedure.

Conventional techniques involve solving the forward and inverse models iteratively till convergence which is time-consuming. The present problem is solved using a conventional way for comparison purposes. TPINN itself is taken as the forward model, and the Genetic algorithm is taken as the optimization algorithm for the inverse model. Results obtained are compared with that of TPINN for computational time and accuracy.

TPINN forward model: TPINN with first layer change is used. The architecture is depth = 3 and width = 10. All penalty factors have a value of 1. $N_{pde}^j \forall j$ = 10 $\forall j$ in Eq. (5). The weights of the neural networks are initialized using the Xavier initialization approach. Loss minimization is accomplished using Adam optimizer (learning rate = 0.001) for the first 200 iterations followed by the SLSQP optimizer. The validation of TPINN forward model is shown in Fig. 27 for different combination of $a_1$ and $a_2$ values. The temperature and thermal conductivity values predicted by TPINN are in a close match with the exact solution. Hence TPINN can be used as a reliable forward model for the conventional approach.

Genetic Algorithm: In the Genetic algorithm (GA), the parameter to be estimated is represented as a string of fixed string length containing zeros and ones. The decimal equivalent of the binary string is obtained using the upper and lower bounds of the parameters given. The fitness function or objective function value is calculated using the decimal form. The optimization begins with a finite population of random strings generated corresponding to each parameter to be estimated. Population size is fixed throughout the optimization procedure and is decided before the procedure starts. The population is passed through three phases; reproduction, crossover, and mutation to obtain a new population. The fitness values of the new population are then calculated, and convergence criteria are checked. A generation of GA consists of one cycle of reproduction, crossover, mutation, and further evaluation of fitness function. The procedure is terminated when the convergence criteria are satisfied; otherwise, the new population is passed to the next generation. The three phases of GA for a maximization problem can be described as:

Reproduction: The reproduction phase chooses the best candidates(strings) from the population to create a mating pool according to a probabilistic procedure. The candidates are chosen with a probability proportional to their fitness function value. Zero to multiple copies of each candidate in the population are copied to the mating pool. For the present problem Roulette wheel selection [62] approach is employed. By this approach, probabilistically, more copies of candidates with high fitness values are placed into the mating pool. No new candidates are generated in the reproduction stage. This phase ensures that highly fit candidates survive and reproduce, while less fit ones are rejected from the process. At the end of this phase, a mating pool with size equals population size is obtained.

Crossover: The mating pool created at the end of the reproduction phase is passed to the crossover phase. A crossover operation starts by randomly selecting two candidates (parents). Along the string length, a site for a single crossover is decided then. A number between 0 and 1 is generated randomly. If this number is less than crossover probability ($P_{\text {c}}$)—a predefined parameter, the portion of the strings to the right of the crossover site of both the parents are swapped with each other forming two new child candidates. Otherwise, no changes are performed on the parents. The child candidates obtained may or may not be fitter than the parents depending on the crossover site. In this phase, new candidates are generated.

Fig. 27

Validation of TPINN forward model for Appendix 2. $(a_1,a_2)$ values for different cases are, $C_1$:(0.02,2), $C_2$:(0.8,1.5), $C_3$:(1.5,0.8), $C_4$:(2,0.02)

Mutation: After the crossover stage, the population is passed to the final phase called mutation. A bit-wise mutation is used here. In bit-wise mutation, a candidate is selected, and each bit (0 or 1) of its string is considered at a time in sequence. A number between 0 and 1 is generated randomly. If the random number is less than mutation probability ($P_{\text {m}}$)—a predefined value, the bit considered is changed from 0 to 1 or 1 to 0. Else no change is done. Mutation operation makes alterations in the candidate locally and helps in maintaining the diversity of the population.

For the present problem, the objective function to maximize by GA is given by

$$\begin{aligned} f = 1/(1+f_1+f_2+f_3+f_4){\text {where}} \end{aligned}$$

$$\begin{aligned} f_1 = \frac{1}{N_{\text {meas}}^1}\sum _{i=1}^{N_{\text {meas}}^1}(T_{1i,{\text {meas}}}-T_{1i,{\text {est}}})^2, \end{aligned}$$

$$\begin{aligned} f_2 =\frac{1}{N_{\text {meas}}^2}\sum _{i=1}^{N_{\text {meas}}^2}(T_{2i,{\text {meas}}}-T_{2i,{\text {est}}})^2, \end{aligned}$$

$$\begin{aligned} f_3 =\frac{1}{N_{\text {meas}}^3}\sum _{i=1}^{N_{\text {meas}}^3}(q_{1i,{\text {meas}}}-q_{1i,{\text {est}}})^2, \end{aligned}$$

$$\begin{aligned} f_4 =\frac{1}{N_{\text {meas}}^4}\sum _{i=1}^{N_{\text {meas}}^4}(q_{2i,{\text {meas}}}-q_{2i,{\text {est}}})^2 \end{aligned}$$

where $T_{lm,{\text {est}}}$, $T_{lm,{\text {meas}}}$ are the temperature estimated from the forward model, the temperature measured at mth measurement location in lth subdomain. Similarly $q_{lm,{\text {est}}}$, $q_{lm,{\text {meas}}}$ are the heat flux estimated from the forward model, the heat flux measured at mth measurement location in lth subdomain. For the present problem, ${N_{\text {meas}}^1} = {N_{\text {meas}}^2} = {N_{\text {meas}}^3} = {N_{\text {meas}}^4} =$ 3. The details of input variables for GA are as follow: String length = 7, $P_{\text {m}}$ = 0.01, $P_{\text {c}}$ = 0.6, population size = 60. For both parameters, 0.02 and 2 are given as lower and upper bounds to search the solution. We ran GA for three generations. After each generation, the values of parameters corresponding to the highest objective function among the population are stored along with the objective function value. After three generations, the values of the parameters corresponding to the highest objective function among the stored ones are considered as estimated values of the parameters $a_1$ and $a_2$.

The results obtained on input data with 0% noise by executing four independent runs of the conventional approach are shown in Table 8. In each run, GA runs for three generations. By comparing the results of Table 8 with results obtained for TPINN shown in Table 7, we can observe that TPINN’s performance is much better in terms of both accuracy and computational time. TPINN estimates the parameters with high accuracy and less variance than the conventional method on clean data. The conventional approach estimates the parameters with a considerable amount of variance across different runs. The maximum relative error in the estimated values of $a_1$ and $a_2$ are 63.21% and 12.6%. The computational time taken by the conventional approach is nearly 15x more than that of TPINN. These initial results show that TPINN could be a promising method for parameter estimation problems by providing a significant amount of savings on computational time and cost.

Appendix 3: Number of learnable parameters of TPINN and DPINN for a given depth and width of the network

Let D = Depth of the neural network (No. of hidden layers plus output layer)

W = Width of the neural network (No. of neurons in each hidden layer. Same for all hidden layers)

${N_1}$ = Number of neurons in the input layer

${N_n}$ = Number of neurons in the output layer

${N_{\text {SD}}}$ = Number of subdomains

${P_{\text {D}}}$ = Total number of learnable parameters of DPINN

${P_{\text {T}}}$ = Total number of learnable parameters of TPINN.

As an example, TPINN with first hidden layer change is considered. Then,

${P_{\text {D}}}$ = ${(N_1W+W+(W^2+W)(D-2)+WN_n+N_n)N_{\text {SD}}}$

${P_{\text {T}}}$ = ${(N_1W+W)N_{\text {SD}}+(W^2+W)(D-2)+WN_n+N_n}$

${P_{\text {D}}} - {P_{\text {T}}}$ = ${((W^2+W)(D-2)+WN_n+N_n)(N_{\text {SD}}-1)}$

When ${N_{\text {SD}}} > 1$, ${P_{\text {D}} > P_{\text {T}}}$. Hence for same depth and width, the total number of learnable parameters of TPINN is less than DPINN.

Table 8 Genetic algorithm results obtained for Appendix 2. Computation was done on Google Colab instance: Intel(R) Xeon(R) CPU with two cores @2.3GHz and 13GB RAM. RE stands for relative error in %