STDIN: Spatio-temporal distilled interpolation for electron microscope images

Abstract

Recently, flow-based approaches have shown considerable success in interpolating video images. However, in contrast to video images, electron microscope (EM) images are further complex due to noise and severe deformation between consecutive sections. Consequently, conventional flow-based interpolation algorithms, which assume a single offset per position, are not able to robustly model the movement of such complicated data. To address the aforementioned problems, this study propose a novel EM image interpolation framework that accommodates a range of offsets per location and further distills the intermediate features. First, a spatio-temporal ensemble (STE) interpolation module for capturing the missing middle features is presented. The STE is subdivided into two modules: temporal interpolation and residual spatial-correlated block (RSCB). The former predicts the intermediate features in two directions with several offsets at each location. Moreover, the RSCB uses the correlation coefficients for aggregated sampling. Thus, even if intermediate features are severely deformed, the STE effectively improves their accuracy. Second, a stackable feedback distillation block (SFDB) is introduced, which enhances the quality of intermediate features by distilling them from the input, and interpolated images, using a feedback mechanism. Extensive experiments demonstrate that the proposed method presents a superior performance compared with previous studies, both quantitatively and qualitatively.

Introduction

The temporal information contained in consecutive images allows an increase in z-axis resolution and thus an improvement in the quality of volume reconstruction (see Fig. 1), when imaging with an electron microscope (EM). For example, images with a 4 nm z-axis resolution are interpolated to achieve a 2 nm resolution. High-resolution serial slices containing fine z-axis motion dynamics [1] build superiorly refined biological structures, which facilitates intelligent data analysis tasks in biology. Furthermore, significant surface imperfections are restored through EM image interpolation. The aforementioned illustrates the social importance of EM interpolation in microscopic imaging.

Series section imaging records the continuity of biological tissues along the z-axis, while video streams reflect the temporal progression of events. Compared with general video streams, biological serial slices have fundamental differences in appearance, including grayscale, more prominent edges, and content patterns. These distinctions stem from the SEM imaging system and impact the application of common motion estimation and compensation algorithms. Fig. 2 illustrates the distinctions between EM and natural images. First, neither grayscale nor sRGB images in natural scenes reveal the edges of objects. In contrast, EM images in grayscale hold visible tissue edges. Second, the content patterns between natural and EM images are dramatically different. EM images are dominated by simple patterns, such as membrane structures, mitochondria, and vesicles, whereas the former contains a more diverse range of textures and substances. Lastly, as highlighted in colorful boxes, the motion trend in natural images is traceable and observable. For EM images, a z-axis resolution of just 10 nm for EM images causes massive and chaotic deformations. In addition, the EM imaging system is more complex and less stable than the optical CCD, resulting in a low signal-to-noise ratio and unstable image quality. In summary, EM images exhibit more complex deformation patterns. Consequently, a single offset per pixel in the flow estimator [5], [6], [7], [8] is inadequate for describing complicated motion in EM images.

In recent years, the use of deep convolutional neural networks to interpolate video images has shown promising results. In this setting, early researches [9], [10] estimate the spatial adaptive convolution kernel per pixel, and employ separable strategies to reduce the model’s capacity. However, the motion construction with adaptive kernels is as well one of the reasons they perform inadequately in complex scenarios. The deep optical flow [5], [6], [7], [8] computes the motion relationships between frames with high accuracy. The following studies [11], [12], [13], [14], [3] interpolate video images using deep optical flow estimation, resulting in visually convenient results. However, the deep optical flow predicts a single offset for each location and warps the pixel at that point to conform to the predicted offset. Due to their limited modeling capabilities, flow-based interpolation algorithms perform inconveniently on complicated EM images, resulting in severe smoothing and artifacts. For motion estimation, deformable convolution [15], [16] provides a novel solution, which calculates multiple offsets for each position. Numerous researches [17], [18] employ deformable convolution rather than deep flow estimation to achieve implicit temporal alignment. Recently, [4] integrated deformable convolution and pyramid features [19], [20] into frame interpolation and proposed a module for interpolating pyramid temporal features. However, the feature temporal interpolation module only captures the temporal context without incorporating joint spatial correlation, leading to the degradation of intermediate features. Furthermore, the lack of spatial rectification accentuates the temporal mismatch when large motions occur. The ConvLSTM [21], used for large movements, presents significant model parameters as well as a slow run time.

To address the aforementioned shortcomings, this study proposes an efficient spatio-temporal distilled interpolation network (STDIN) for EM images. The feature spatio-temporal ensemble (STE) module handles the dynamic background and predicts the interpolated features accurately. Specifically, the STE captures pyramid temporal features and calculates the spatial correlation coefficients. Afterward, the final interpolated features are synthesized using region sampling. Although the intermediate features with dual-domain embedding show promising performance in motion processing, the interpolated features still exhibit multiple mismatches when encountering severe anisotropy and large motions. Moreover, when using interpolated features as a reference and subsequently extracting from the input features the relevant ones, the incorrect prediction of intermediate features is exacerbated due to the interference of background noise and image quality fluctuations. Therefore, this paper proposes a lightweight, stackable feedback distillation block (SFDB) to purify intermediate features and minimize the temporal mismatch caused by large deformations. The SFDB module adapts the feedback distillation in response to input features. Moreover, this study discovers that the feedback distillation correction is stackable. Hence, the resulting intermediate features become more precise as the number of stacked modules increases.

The contributions of this paper are summarized as follows:

• The work of video frame interpolation is extended to electron microscopy and proposes a simple but effective framework for interpolating EM images. This approach incorporates spatio-temporal ensemble sampling and feedback distillation. Subsequently, the interpolated frames generated from EM images are more precise than those generated by previous interpolation algorithms.

• A spatio-temporal ensemble module that includes temporal context and spatial correlated information is presented. Moreover, a novel feedback distillation module is introduced, which enables the acquirement of the best aligned intermediate features under the supervision of input images.

• Extensive experiments demonstrate that this approach achieves state-of-the-art performance on the EM benchmark datasets and outperforms the recent best frame interpolation algorithms.