A CVAE-within-Gibbs sampler for Bayesian linear inverse problems with hyperparameters

Abstract

We propose a conditional variational auto-encoder within Gibbs sampling (CVAE-within-Gibbs) for Bayesian linear inverse problems where the prior or the likelihood function depends on ambiguous hyperparameters. The method builds on ideas from classical sampling theory and recent advances in deep generative models to approximate complicated probability distributions. Specifically, we use a CVAE model which is trained with a large amount of data to learn the conditional density of hyperparameters in the original Gibbs sampler. The learned property of the conditional posterior provides more flexibility than classical Gibbs sampling because it avoids manually or experimentally determining the hyperpriors and their hyperparameters. We demonstrate the performance of the proposed method for three linear inverse problems, i.e., image deblurring, signal denoising, and boundary heat flux identification in a heat conduction problem.

Appendices

Appendix A: Datasets

The details of datasets for the three experiments are summarized in Table 8. The unknown parameters ${\varvec{u}}$, the measurable data ${\varvec{y}}$, and the hyperparameters $\varvec{\theta }$ are included in these datasets. In the image deblurring experiment, the variance of the prior distribution contains the hyperparameters $\gamma$ and d, as shown in Eq. (11), and the mean of the prior distribution contains $\mu$, and s is included in the variance of the noise term. In the signal denoising experiment and the IHCP experiment, the only hyperparameter $\sigma _\mathrm{{obs}}$ is drawn from the variance of the noise term $\varvec{\Sigma }_\mathrm{{obs}}=\sigma ^2_\mathrm{{obs}}{\textbf{I}}$. The intervals in the second column of Table 8 indicate the range of values of hyperparameters determined empirically. In order to generate dataset, P discrete points are uniformly selected from each interval to form different combinations of hyperparameters values, and corresponding $\{y^i,u^i\},i=1,\ldots ,{\bar{N}}$ are generated based on each combination. The last three columns of the table represent the size of the synthetic datasets, the dimension of the attribute ${\varvec{u}}$, and the dimension of ${\varvec{y}}$. For example, in the first experiment, there are a total of $200,000=50\times 20\times 20\times 10$ combinations of hyperparameters, and 10 samples are randomly generated based on each combination, yielding a total of 2,000,000 samples.

Table 8 Details of datasets

Appendix B: Network architectures

1.1 Image deblurring

The CVAE model used for the image deblurring consists of 2 fully connected hidden layers for the encoder and 3 for the decoder, with 5 Gaussian latent variables. We selected batch size as 128, 10 epochs, and the Adam optimizer with learning rate $1.7 \times 10^{-6}$. The ReLU activation functions are used in the encoder and decoder. The number of neurons contained in each layer is shown in Table 9.

1.2 Inverse heat conduction problem

In the IHCP experiment, we employed the reclassification strategy discussed in Sect. 5.1.1, i.e., a classification network for ${\varvec{y}}$ is added to CVAE. The CVAE used for IHCP consists of 3 fully connected hidden layers for the encoder, 5 for the re-classification network, and 3 for the decoder, with 5 Gaussian latent variables. We chose Xavier Initialization, batch size of 128, 40 epochs, the Adam optimizer with learning rate $2\times 10^{-5}$ for the re-classification network and learning rate $5.5\times 10^{-4}$ for the VAE. The Leaky ReLU activation functions are used in the encoder and decoder and the ReLU activation functions are used in the re-classification network. The number of neurons contained in each layer is shown in Table 9.

1.3 Signal denoising

In the signal denoising experiment, we also employed the reclassification strategy discussed in Sect. 5.1.1. The CVAE model used for signal denoising consists of 5 fully connected hidden layers for the encoder and 7 for the decoder, with 5 Gaussian latent variables. We used a ResNet with 3 residual blocks and a linear layer for the re-classification network. The residual block has 2 fully connected layers with the dropout probability of p = 0.2. We utilize Xavier Initialization and the Leaky ReLU activation functions in all three networks. We chose Xavier Initialization, batch size as 64, 40 epochs, the RMSprop optimizer with learning rate $1\times 10^{-5}$ for the re-classification network, and the NAdam optimizer with learning rate $1\times 10^{-4}$ for the CVAE. The number of neurons contained in each layer is shown in Table 9.

Table 9 Details of network architecture