Keywords

1 Introduction

Compared with traditional 2-D images, which integrate the intensity of the light rays from all directions at a pixel location, LF images separately record the light ray intensity from different directions, thus providing additional information on the 3-D scene geometry. Such information is proportional to the angular resolution, i.e. the number of directions of the light rays, captured by the LF image. Densely sampled LF, with high resolution in the angular domain, contains sufficient information for accurate depth inference [1,2,3,4], post-capture refocusing [5] and 3D display [6, 7].

LF images [8, 9] can be acquired in a single shot using camera arrays [10] and consumer hand-held LF cameras such as Lytro [11] and Raytrix [12]. The former, due to the large number of sensors, can capture LF with higher spatial resolution while being expensive and bulky. Through multiplexing the angular domain into the spatial domain, the later is able to capture LF images with a single sensor, and thus are cheaper and portable. However, due to the limited sensor resolution, there is a trade-off between spatial and angular resolution. As a result, these cameras cannot densely sample in both the spatial and angular domains.

Reconstruction of a densely-sampled LF from a sparsely-sampled LF input is an on-going problem. Recent development in deep learning based LF reconstruction models [13, 14] have achieved far superior performance over the traditional approaches [1,2,3,4]. Most notably, Kalantari et al. [13] proposed a sequential convolutional neural network (CNN) with disparity estimation and Wu et al. [14] proposed to use a blur-deblur scheme to counter the problem of information asymmetry between angular and spatial domain and a single CNN is used to map the blurred epipolar-plane images (EPIs) from low to high resolution. However, both approaches require heavy pre- or post-processing steps and long runtime, making them impractical to be applied in consumer LF imaging system.

In this paper, we propose a novel learning based model for fast reconstruction of a densely-sampled LF from a very sparsely-sampled LF. Our model, an end-to-end CNN, is composed of two phases, i.e., view synthesis and refinement, which are realized by computationally efficient convolutions to deeply characterize the spatial-angular clues in a coarse-to-fine manner. Specifically, the view synthesis network is designed to synthesize a set of intermediate novel sub-aperture images (SAIs) based on the input sparsely-sampled LF and the view refinement network is deployed for further exploiting the intrinsic LF structure among the synthesized novel SAIs. Our model does not require disparity warping nor any computationally intensive pre- and post-processing steps. Moreover, reconstruction of all novel SAIs are performed in one forward pass during which the intrinsic LF structural information among them is fully explored. Hence, our model fully preserves the intrinsic structure of reconstructed densely-sampled LF, leading to better EPI quality that can contribute to more accurate depth estimation.

Experimental results show that our model provides over 3 dB improvement in the average reconstruction quality while requiring less than 20 s on CPU, achieving over 30\(\times \) speed up, compared with the state-of-the-art methods in synthesizing a densely-sampled LF from a sparsely-sampled LF. Experiment also shows that the proposed model can perform well on large baseline LF inputs and provides substantial quality improvement of over 3 dB with extrapolation. Our algorithm not only increases the number of samples for depth inference and post-capture refocusing, it can also enable LF to be captured with higher spatial resolution from hand-held LF cameras and potentially be applied in compression of LF images.

2 Related Work

Early works on LF reconstruction are based on the idea of warping the given SAIs to novel SAIs guided by an estimated disparity map. Wanner and Goldluecke [1] formulated the SAI synthesis problem as an energy minimization problem with a total variation prior, where the disparity map is obtained through global optimisation with a structure tensor computed on the 2-D EPI slices. Their approach considers disparity estimation as a separate step from SAI synthesis, which makes the reconstructed LF heavily dependent on the quality of the estimated disparity maps. Although subsequent research [2,3,4] has shown significantly better disparity estimations, ghosting and tearing effects are still present when the input SAIs are sparse.

Kalantari et al. [13] alleviated the drawback of Wanner and Goldluecke [1] by synthesizing the novel SAIs with two sequential CNNs that are jointly trained end-to-end. The first CNN performs disparity estimation based on a set of depth features pre-computed from the given input SAIs. The estimated disparities are then used to warp the given SAIs to the novel SAIs for the second CNN to perform color estimation. This approach is accurate but slow due to the computation intensive depth features extraction. Furthermore, each novel SAI is estimated at a separate forward pass, hence the intrinsic LF structure among the novel SAIs is neglected. Moreover, the reconstruction quality depends heavily upon the intermediate disparity warping step, and thus the synthesized SAIs are prone to occlusions.

Advancement in single image super-resolution (SISR) is recently made possible by the adoption of deep CNN models [15,16,17,18]. Following this, Yoon et al. [19, 20], developed a CNN model that jointly super-resolves the LF in both the spatial and angular domain. This model concatenates at the channel dimension a subset of the spatially super-resolved SAIs from a CNN that closely resembles the model proposed in [15]. The concatenated SAIs are then passed into a second CNN for angular super-resolution. Their approach is designed specificity for scale 2 angular super-resolution and can not flexibly adapt to perform on very sparsely-sampled LF input.

Recently, Wu et al. [14] developed a CNN model that inherits the basic architecture of [15] with an addition residual learning component as in [16]. Using the idea of SISR, their model focuses on recovering the high frequency details of the bicubic upsampled EPI while a blur-deblur scheme is proposed to counter the information asymmetry problem caused by sparse angular sampling. Their model is adaptable to different devices. Since each EPI is a 2-D slice in both the spatial and angular domains of the 4-D LF, EPI based model can only utilize SAIs from the same horizontal or vertical angular coordinate of the sparsely-sampled LF to recover the novel SAIs in between, thus severely restricting the accessible information of the model. For the novel SAIs that do not fall within the same horizontal or vertical angular coordinate as the input SAIs, they are reconstructed based on the previously estimated SAIs. As a result, these SAIs are biased due to input errors. Moreover, due to the limitation in the blurring kernel size and bicubic interpolation, this method cannot be applied to sparsely-sampled LF with only \(2\times 2\) SAIs or with disparity larger than 5 pixels.

Fig. 1.
figure 1

LF captured with a single sensor device. The angular information of an LF is captured via the separation of light rays by the micro-lens array. The resulting LF can be parameterized by the spatial coordinates and the angular coordinates, i.e. the position of the SAI.

Full size image

3 The Proposed Approach

3.1 4-D Light Field and Problem Formulation

4-D LF can be represented using the two-plane parameterization structure, as illustrated in Fig. 1, where the light ray travels and intersects the angular plane (s, t) and the spatial plane (x, y) [21]. Let \(\mathcal {I}\in \mathbb {R}^{W \times H\times M\times N \times 3}\) denote an LF with \(M \times N\) SAIs of spatial dimension \(W\times H\times 3\), and \(\mathcal {I}(:, :, s, t, :)\in \mathbb {R}^{W\times H\times 3}\) be the (s, t)-th SAI (\(1\le s\le M\), \(1\le t\le N\)).

Densely-sampled LF reconstruction aims to construct an LF \(\mathcal {I'}\in \mathbb {R}^{W \times H\times M' \times N' \times 3}\) including a large number of SAIs, from an LF \(\mathcal {I}\) containing a small number of SAIs , where \(M'>M\) and \(N' > N\). Since the densely-sampled LF \(\mathcal {I'}\) also contains the set of input SAIs, denoted as \(\mathcal {K}\), the SAIs to be estimated is therefore reduced to the set of (\( M' \times N' - M \times N\)) novel SAIs, denoted as \(\mathcal {N}\).

Efficient modelling of the intrinsic structure of LF , i.e. photo-consistency, defined as the relationship of pixels from different SAIs that represent the same scene point, is crucial for synthesising high quality LF SAIs. However, real-world scenes usually contain factors such as occlusions, specularities and non-Lambertian lighting, making it challenging to characterize this structure accurately. In this paper, we propose a CNN based approach for efficient characterisation of spatial-angular clues for high quality reconstruction of densely sampled LFs.

Fig. 2.
figure 2

The workflow of reconstructing a densely-sampled LF with \(8 \times 8\) SAIs from a sparsely-sampled LF with \(2 \times 2\) SAIs. Our proposed model focuses on reconstructing the luma components (Y) of the novel SAIs, while angular bilinear interpolation recovers the other two chrominance components (Cb and Cr). Note that the reshape operations in the view synthesis network are included for understanding of the data flow and are not required in actual implementation.

Full size image

3.2 Overview of Network Architecture

As illustrated in Fig. 2, we propose a novel CNN model to provide direct end-to-end mapping between the luma component of the input SAIs, denoted as \(\mathcal {K}_Y\), and that of the novel SAIs, denoted as \(\widehat{\mathcal {N}}_Y\). Our proposed network consists of two phases: view synthesis and view refinement. The view synthesis network, denoted as \(f_S(.)\), first synthesizes the whole set of intermediate novel SAIs based on all input SAIs. The synthesized novel SAIs are then combined with the input SAIs to form a 4-D LF structure using a customised reshape-concat layer. This intermediate LF is then fed into the refinement network, denoted as \(f_R(.)\), for recovering the fine details. At the end, the estimated fine details are added to the intermediate synthesized SAIs in an pixel-wise manner to give the final prediction of the novel SAIs \(\widehat{\mathcal {N}}_Y\). The relations between the inputs and outputs of our model is represented as:

$$\begin{aligned} \widehat{\mathcal {N}}_Y = f_S(\mathcal {K}_Y) + f_R(f_S(\mathcal {K}_Y), \mathcal {K}_Y). \end{aligned}$$
(1)

Note that the full color novel SAIs \(\widehat{\mathcal {N}}\) are obtained from combining \(\widehat{\mathcal {N}}_Y\) with angular bilinear interpolation of the other two chrominance components, i.e., Cb and Cr. Contrary to the previous approaches that synthesize a particular novel SAI at a each forward pass [13], and an EPI of a row or column of novel SAIs at each forward pass [14], our approach is capable of jointly producing all novel SAIs at one pass to preserve the intrinsic LF structure among them. Our network is full 4-D convolutional and uses Leaky Relu with the parameter of 0.2 for activation. Table 1 provides a summary of the network architecture.

3.3 View Synthesis Network

The view synthesis network estimates a set of intermediate novel SAIs by uncovering the coarse spatial-angular clues carried by the limited number of SAIs of the input sparsely-sampled LF. This step takes in all input SAIs from the given LF for the estimation of novel SAIs, and thus it can make full use of available information on the structural relationship among SAIs. For achieving this, it is necessary to perform convolution on all both the spatial and the angular dimensions of the input LF.

4-D convolution is a straightforward choice for this task. However, for this particular problem, the computational cost required by 4-D convolution makes training such a model impossible in a reasonable amount of time. Pseudo filters or separable filters, which reduce model complexity by approximating a high dimensional filter with filters of lower dimension, have been applied to solve different computer vision problems, such as image structure extraction [22], 3-D rendering [23] and video frame interpolation [24]. This is recently adopted in [25] for LF material classification, which verifies that the pseudo 4-D filters can achieve similar performance as 4-D filters.

For preventing potential overfitting and long training time from the use of full 4-D filter while characterizing 4-D information of LF, we adopt the pseudo 4-D filter which approximates a single 4-D filtering step with two 2-D filters that perform convolution on the spatial and the angular dimensions of the LF in an alternating manner. Such a design requires only the computation of \(2/n^2\) of a 4-D convolution while still utilizing all available information from the input SAIs.

In the synthesis network, spatial-angular alternating convolutions are adopted only for intermediate feature extraction. For the initial feature extraction step and the novel SAIs synthesis step, 4-D convolution is applied since the computational complexity is less. Such a design obtains a significant reduction in parameter size as well as computational cost. Moreover, the low computational cost also benefits from that feature extraction is performed at the coarse angular resolution of \( M \times N\) as opposed to [14] at the fine level of \( M' \times N'\).

Table 1. Model specification for reconstructing a densely-sampled LF with \(8 \times 8\) SAIs from a sparsely-sampled LF with \(2 \times 2\) SAIs on the luma component. The first two dimensions of the filters, input and output data tensor correspond to the spatial dimension whereas the third and the forth dimension correspond to the angular dimension. The fifth dimension of the output tensor denotes the number of feature maps in the intermediate convolutional layers while representing the number of novel SAIs at the final layer. Stride and Paddings are given in the form of (Spatial/Angular). All convolutional layers contain biases. Note that the intermediate LF reconstruction step is performed with reshape and concatenation operations to enable back-propagation of loss from the view refinement network to the view synthesis network.
Full size table

3.4 View Refinement Network

In the view synthesis phase, novel SAIs are independently synthesized, and the relationship among them is not taken into account. Therefore, a view refinement network is designed to further exploit the relationship among the synthesized novel SAIs from the intermediate LF, which is expected to contribute positively to the reconstruction quality of the densely-sampled LF. This can be considered as a regularizer that imposes the LF structure on the synthesized SAIs.

Inspired by the success of residual learning on image reconstruction [14, 16,17,18], we equip our view refinement network with guided residual learning that is specifically designed for the LF data structure. Typical residual learning attempts to learn a transformation \({R(\cdot )}\) to recover the residual \({R(\mathcal {I}')}\) for the input data \({\mathcal {I}'}\), i.e. the intermediate LF, as shown in Eq. (2). However, the input to the refinement network consists of a set of SAIs \({\mathcal {K}_Y\subset \mathcal {I}'}\) from the given sparsely-sampled LF, which is absolutely correct, i.e. \({R(\mathcal {K}_Y) = 0}\), and a set of synthesized SAIs \({\mathcal {N}'_Y = f_S(\mathcal {K}_Y})\subset \mathcal {I}'\), which is erroneous. Hence, residual learning on \({\mathcal {K}_Y}\) is unnecessary. Guided residual learning can be formulated as a typical residual learning on \({\mathcal {N}'_Y}\) with the guidance from the additional input, \({\mathcal {K}_Y}\), as shown in Eq. (3).

$$\begin{aligned} \widehat{\mathcal {I}}_Y = \mathcal {I}' + R(\mathcal {I}') \end{aligned}$$
(2)
$$\begin{aligned} \widehat{\mathcal {N}}_Y = \mathcal {N}'_Y + R( \mathcal {N}'_Y | \mathcal {K}_Y) \end{aligned}$$
(3)

Guided residual learning has the following benefits: (i) \({\mathcal {K}_Y}\), as a set of ground-truth SAIs, offers correct complementary information of the scene; (ii) learning 0 residuals for \({\mathcal {K}_Y}\) is not performed; and (iii) By placing \({\mathcal {K}_Y}\) and \(\mathcal {N}'_Y\) in the form of \(\mathcal {I}'\), a densely sampled intermediate LF, for input to the second stage refinement network, it encourages the first stage, i.e., view synthesis network, to generate SAIs that preserve the LF structure exhibiting in the EPI shown in Fig. 1(d).

Since the angular dimension increases significantly from \(M \times N\) to \(M' \times N'\) after the view synthesis processes, alternating convolution will incur a substantially higher computation cost that increases linearly in angular dimension. For reducing the computation to a manageable level, stride-2 4-D convolution is used for efficient angular dimension reduction while the feature map number is set to increase gradually. Note that to allow back-propagation, an intermediate 4-D LF is reconstructed from the previously synthesized novel SAIs and the input SAIs via a customised reshape-concat layer. The refinement details of all novel SAIs are independently estimated at the final 4-D convolution layer and are added to the previously synthesized intermediate novel SAIs to give the final reconstructed novel SAIs.

3.5 Training Details

The training objective is to minimise the \(\mathcal {L}_{2}\) distance between all reconstructed novel SAIs \(\widehat{\mathcal {N}}_Y\) and their respective ground-truth \(\mathcal {N}_Y\):

$$\begin{aligned} \mathcal {L}_{2}(\mathcal {N}_Y, \widehat{\mathcal {N}}_Y) = \sum _{x}\sum _{y}\sum _{s}\sum _{t}\left( \widehat{\mathcal {N}}_Y(x, y, s, t) - \mathcal {N}_Y(x, y, s, t)\right) ^2. \end{aligned}$$

We trained a model for each task on the training set with 100 scenes provided by Kalantari et al. [13]Footnote 1. All images were taken with a Lytro Illum camera and were decoded to \(14 \times 14\) SAIs with spatial resolution \( 376 \times 541\). Since the three SAIs from each side are usually black, we only adopted the middle \(8\times 8\) SAIs for training and testing as done in [13].

Training LFs were spatially cropped to \(64 \times 64\) patches with stride 1, giving a maximum of approximately 15,000,000 training samples. Moreover, we adopted stochastic gradient descent to optimize the model, and the batch size was set to 1. The spatial resolution of the model output is kept unchanged at \( 64 \times 64 \) with padding of zeros.

We implemented the model with the MatConvNet toolbox [26] in MATLAB and trained it with the GTX 1080 Ti GPU. Random filter weights under the MSRA method [27] were used to initialize our model, while biases were initialized to 0. Throughout training, momentum parameter was set to 0.9. Depending on model depth, a learning rate between to was applied without weight decay, and epoch number was set between 8000 to 12000 each with 1000 iterations. Training time increases linearly with the number of alternating convolutions, ranging from around 1 day for model with 1 alternating convolution and 10 days for model with 16 alternating convolutions.

4 Experimental Results

Our model was compared with two state-of-the-art CNN based methods that are specifically designed for densely-sampled LF reconstruction, i.e., Kalantari et al. [13] and Wu et al. [14]. Comparisons were performed over three different tasks, shown in Fig. 3: \(3 \times 3\) – \(7 \times 7\), \(3 \times 3\) – \(9 \times 9\) and \(2 \times 2\) – \(8 \times 8\). Task \(M \times N - M' \times N'\) stands for reconstructing densely-sampled LFs with \(M' \times N'\) SAIs from sparsely-sampled LFs with \(M \times N\) SAIs. Moreover, we investigated the effect of the positions of SAIs involved in the sparsely-sampled LF input on the reconstruction quality via task \(2 \times 2\) – \(8 \times 8\).

Fig. 3.
figure 3

Illustration of inputs (red blocks) and outputs (yellow blocks) for different tasks. From left to right: (a) \(3 \times 3\) – \(7 \times 7\), (b) \(3 \times 3\) – \(9 \times 9\), (c) \(2 \times 2\) – \(8 \times 8\), (d) \(2 \times 2\) – \(8 \times 8\) extrapolation-1, (e) \(2 \times 2\) – \(8 \times 8\) extrapolation-2. (Color figure online)

Full size image

Both quantitative and qualitative results will be shown in the following subsections. Reconstruction quality is measured with PSNR and SSIM, averaged over all synthesised novel SAIs. Due to limited space, we only report the average result for all data entries in each dataset. The (5, 5)-th SAI of the reconstructed densely-sampled LF is chosen for display. Both training and testing codes are publicly availableFootnote 2.

4.1 \(3\times 3\) – \(7 \times 7\) Light Field Reconstruction

For the task \(3 \times 3\) – \(7 \times 7\), we compared with Kalantari et al. [13] and Wu et al. [14]. We set the number of spatial-angular alternating convolutional layers to 4. Comparisons were performed on the 30 Scenes dataset [13], the reflective-29 and occlusion-16 LFs from the Stanford Lytro Lightfield Archive [28] and Neurons 20\(\times \) from the Stanford Light Field microscope dataset [29]. The reconstruction quality measured in PSNR and SSIM is shown in Table 2. For each LF, the results are the average of the luma component of all 40 novel SAIs. Our proposed model performs better for all datasets than the two comparing methods: with 0.88 dB and 3.31 dB reconstruction advantage over Kalantari et al. [13] and Wu et al. [14], respectively. A 2.3 dB advantage for the Neurons 20\(\times \) dataset shows that the proposed LF reconstruction model generalizes well to different LF capturing devices.

Table 2. Quantitative comparisons of the reconstruction quality of the proposed model and the state-of-the-art methods under the task \(3 \times 3\) – \(7 \times 7\).
Full size table
Table 3. Quantitative comparisons of reconstruction quality of the proposed model, Kalantari et al. and Wu et al. over Buddha and Mona from the HCI dataset.
Full size table

4.2 \(3 \times 3\) – \(9 \times 9\) Reconstruction on Large Disparity Light Field

To demonstrate that our model can work on LFs with larger disparities, the proposed model was modified for task \(3 \times 3\) – \(9 \times 9\) and was trained with LFs from the HCI dataset [30], which are created with Blender software [31], with larger disparities compared with Lytro Illum captures. The LFs Budda and Mona are used for testing and the rest are used for training. For this task, we set the number of spatial-angular alternating convolution layers to 8. Due to limited number of training images, data augmentation was applied for obtaining more data training samples.

Comparison results with [14] are reported in Table 3. Using only 7 training LFs, our proposed method provides superior reconstruction quality on the luma component, averaged across all 72 novel SAIs.

4.3 \(2\times 2\) – \(8 \times 8\) Light Field Reconstruction

We carried out comparison with the method by Kalantari et al. [13] retrained with the same training dataset as ours. The method by Wu et al. [14] cannot be compared since their approach requires 3 views in each angular dimension to provide enough information for the bicubic interpolation step. Our testing dataset contains 30 test scenes from [13] (See footnote 1) and 118 LFs from the EPFL dataset [32]Footnote 3 with diversified real-world scenes. To further evaluate the robustness of the algorithms, we also included the Refractive and Reflective Surfaces and the Occlusions categories from the Stanford Lytro Lightfield Archive [28], which contain 31 and 43 LFs, respectively. Note that the 8 LFs from the Occlusions category and 1 LF from the Refractive and Reflective Surfaces category were removed from testing as they were used for training. This test set contains 222 LFs which is sufficient to provide objective evaluation of model performance.

Table 4. Quantitative comparisons of reconstruction quality of the proposed model and Kalantari et al. under task \(2 \times 2\) – \(8 \times 8\) over 222 real-world LFIs.
Full size table

Reconstruction quality is measured with PSNR and SSIM averaged over the RGB channels, and over all 60 novel SAIs. As shown in Table 4, our proposed model with 16 alternating convolutions in the synthesis network obtains an average of 37.76 dB, 0.86 higher than that of Kalantari et al. [13].

Figure 4 further visually demonstrates that our algorithm is able to obtain better reconstruction quality compare with the state-of-the-art. As shown in the error maps, Kalantari et al. produces artifacts near the boundaries of the foreground objects. In most cases, thin edges cannot be reconstructed correctly, leaving blurred and overlapped regions between occluders and the background. Moreover, since our method fully explores the relationship among all SAIs in the reconstruction process, the LF structure is well preserved, leading to better EPI quality that can contribute to more accurate depth estimation.

Fig. 4.
figure 4

Visual comparison of our proposed approach with Kalantari et al. [13] on the (5, 5)-th synthesised novel SAI for the task \(2 \times 2\) – \(8 \times 8\). Selected regions have been zoomed on for better comparison. Digital zoom-in is recommended for more visual details.

Full size image

4.4 \(2 \times 2\) – \(8 \times 8\) Light Field Reconstruction with Extrapolation

Figures 5(a) and (b) show the average quality of each novel SAIs by Kalantari et al. [13] and the proposed approach under the task \(2 \times 2\) – \(8 \times 8\), where it can be observed that reconstruction quality of the center SAIs has significantly worse quality compared with the novel SAIs near the input SAIs. The central view is furthest away from any of the input SAIs, therefore it poses greatest challenge to correctly infer the details. Based on this analysis, we investigated the possibility of combing interpolation and extrapolation for the LF reconstruction, which can make the average distances from all novel SAIs shorter to the input SAIs.

We trained two models with the exact same network architecture as Ours 8L, however, with different input view position configurations as shown in Fig. 3(d) and (e), which we name as Ours Extra. 1 and Ours Extra. 2, respectively. Note that for the first model, 1 row and column of SAIs are extrapolated while for the second model, 2 rows and columns of SAIs are extrapolated.

As shown in Table 5, when our model combines interpolation and extrapolation, an average of 2.5 dB improvement can be achieved for all novel SAIs on the 222 LFs dataset. Figures 5(c) and (d) also show the average quality of each novel SAIs by Ours Extra. 1 and Ours Extra. 2, respectively. The significant gain in reconstruction quality indicates the potential for the proposed algorithm to be applied on LF compression [33, 34].

4.5 Depth Estimation

To verify that the densely-sampled LF generated from our proposed model not only produces high PSNR for each SAIs, but also well preserves the 3-D geometric structures among the SAIs, we further applied the depth estimation algorithm [3] on the reconstructed densely-sampled LF with \(8 \times 8\) SAIs generated from a sparsely-sampled LF with \(2 \times 2\) SAIs. Figure 6 shows in each row the depth maps based on the sparsely-sampled LFs, the densely-sampled LFs from Kalantari et al., the densely-sampled LFs from our model and the ground-truth densely-sampled LFs. It can be observed that the depth maps from Ours Extra. 1 are more accurate than those by Kalantari et al.

Table 5. Quantitative comparisons of reconstruction quality of Ours, Ours Extra. 1, Ours Extra. 2 and Kalantari et al. over 222 real-world LFs. For the proposed models, the number of spatial-angular alternating convolutions is set to 8.
Full size table
Fig. 5.
figure 5

Each subfigure displays the average reconstruction quality measured as PSNR at different SAI position under the task \(2 \times 2\) – \(8 \times 8\) of different models. The white blocks indicate the input SAIs. From left to right: (a) Kalantari et al. [13], (b) Ours, (c) Ours Extra. 1 and (d) Ours Extra. 2.

Full size image
Fig. 6.
figure 6

Visual comparison of the depth estimation results from a sparsely-sampled LF, reconstructed densely-sampled LF from our proposed approach and Kalantari et al. [13] and a ground-truth densely-sampled LF.

Full size image

4.6 Runtime and Reconstruction Quality vs. Model Depth

The runtime and performance trade-off of our proposed model with different numbers of alternating convolutions are shown in Fig. 7. We can observe that the reconstruction quality by our model increases rapidly with the number of alternating convolutions increasing. Furthermore, the adoption of extrapolation leads to a significant improvement in reconstruction with a runtime of around 11 s, over 30\(\times \) speed up compared with Kalantari et al. [13], on an Intel i7-6700K CPU @ 4.00 GHz without GPU acceleration. Moreover, the scalable structure in the synthesis network enables a trade-off between the reconstruction quality and speed. For task \(2 \times 2\) – \(8 \times 8\), our model with 16 alternating convolutions needs approximately 20 s. If speed is of priority, at similar reconstruction quality to Kalantari et al., our model with 1 alternating convolution can provide over 130\(\times \) speed up, taking only 3.15 s to process an LF.

Fig. 7.
figure 7

The trade-off between runtime and reconstruction quality at different model depth. Execution time in seconds were calculated as the average of 50 tests performed on an Intel i7-6700K CPU @ 4.00 GHz without GPU acceleration.

Full size image

5 Conclusion and Future Work

We have presented a novel learning based framework for densely-sampled LF reconstruction. To characterize the high-dimensional spatial-angular clues within LF data accurately and efficiently, we have designed an end-to-end trained CNN that extensively employs spatial-angular alternating convolutions for fast feature transformation and stride-2 4-D convolutions for rapid angular dimension reduction. Moreover, our network synthesizes novel SAIs in a coarse-to-fine manner by first reconstructing a set of intermediate novel SAIs synthesized at the coarse angular dimension, then applying guided residual learning to refine the intermediate views at a finer level.

Extensive evaluations on real-world and synthetic LF scenes show that our proposed model is able to provide over 3 dB reconstruction quality in average than the state-of-the-art methods while being over 30\(\times \) faster. Especially, our model can handle complex scenes with serious occlusions well. Moreover, our model is able to perform well under LFs with larger disparities, and more accurate depth can be inferred from the reconstructed densely-sampled LFs by our method. Considering the efficiency and effectiveness of the proposed CNN model in processing LF data, we believe such a design has great potential on LF compression, as well as a wide range of LF image processing tasks, including but not limited to LF spatial super-resolution, temporal super-resolution and depth estimation.