Accurate MR image super-resolution via lightweight lateral inhibition network
Graphical abstract
Introduction
Magnetic resonance imaging (MRI) is a commonly-used and versatile non-invasive imaging modality with the advantages of multi-contrast and no ionizing radiation etc. Spatial resolution is one of the most important imaging parameters in most MRI experiments. In general, high-resolution (HR) images usually provide rich structural details and benefit more accurate image postprocessing, hence promoting effective subsequent analysis and early clinic diagnosis (Greenspan et al., 2001, Greenspan et al., 2002, Reeth et al., 2012, Shi et al., 2019). However, the spatial resolution of magnetic resonance (MR) images are typically constrained by various physical and physiological limitations, e.g., hardware device, imaging time, signal-to-noise ratio (SNR), and motion artifacts etc. (Reeth et al., 2012, Plenge et al., 2012). Increasing the spatial resolution of MR images typically reduces SNR, and increase imaging time and thus patient discomfort, indicating that these imaging parameters are highly interdependent to each other (Plenge et al., 2012).
Image super-resolution (SR) provides an effective alternative to enhance the resolution of MR images from the perspective of postprocessing (Zhao et al., 2018c), which aims at recovering a HR image from one or more low-resolution (LR) images. As a postprocessing method, image SR is an active research field that can substantially break through the limitations of hardware device and improve image resolution (Park et al., 2003, Plenge et al., 2012). In recent years, deep learning techniques (LeCun et al., 2015), especially convolutional neural networks (CNNs) (LeCun et al., 1989), have greatly promoted the development of this field, resulting in the emergence of many advanced SR methods, such as SRCNN (Dong et al., 2016a), DRCN (Kim et al., 2016b), DRRN (Tai et al., 2017a), MemNet (Tai et al., 2017b), VDSR (Kim et al., 2016a), EDSR/MDSR (Lim et al.), RDN (Zhang et al., 2018b), RCAN (Zhang et al., 2018a) and CSN (Zhao et al., 2018c) etc. Although these models have excellent performance, most of them are mainly aimed at the SR tasks on natural images, instead of MR images.
In medical image processing community, there are also some deep CNN-based medical image SR methods, e.g., Pham et al., Chen et al., 2018b, Chen et al., 2018a, Zhao et al., 2018a, Zhao et al., 2018c etc. The primary intention of these methods, to some extent, is to improve the performance of MR image SR tasks. However, a fundamental consideration in many MRI experiments is how to reduce imaging time to improve patient comfort and avoid motion artifacts as much as possible. Therefore, high-efficiency HR image reconstruction is also of significance in practical applications. On the other hand, an important problem in medical image processing with deep learning techniques is the degradation of training samples (Litjens et al., 2017, Zhao et al., 2018c). As the model scale (e.g., model parameters, network depth/width etc.) increases, it will be more difficult to train larger models with these degraded medical training samples (Zhao et al., 2018c, Zhao et al., 2019), and more tricks are needed for successful model training (Li et al., 2018). In this regard, lightweight models may be more appropriate for practical applications of medical image SR tasks.
With these considerations, we aim at efficient MR image SR reconstruction by introducing a lightweight CNN model in this paper. The proposed model, which we term as lateral inhibition network (LIN), is well-motivated and inspired by the biological lateral inhibition mechanism that assumes there exists explicit inhibitory regulation between adjacent neurons. The building module of our LIN network is local inhibition unit (LIU) that takes residual block (Lim et al., Zhang et al., 2018b, Zhang et al., 2018a) as the backbone and a inhibition tail (IT) is attached to integrate lateral inhibition mechanism into feature mapping. A series of cascaded LIUs construct a local nonlinear mapping block, i.e., lateral inhibition block (LIB), as shown in Fig. 2. Then multiple LIBs are stacked together to build the nonlinear subnet of the proposed LIN model. Besides, to extract shallow features with different receptive field sizes, we use a group of 3 × 3 dilated conv layers with different dilation rates in the feature extraction subnet, as shown in Fig. 1. Like Lim et al., Zhang et al., 2018b, Zhao et al., 2018c, we only apply one 3 × 3 conv layer to reconstruct the final output.
Deep CNN models are generally built upon the convolution operation that extracts informative local features by integrating spatial and channel information together within local receptive fields. In fact, a lot of work demonstrates that careful structural design helps to improve the representational capacity of deep models substantially (Simonyan and Zisserman, 2014, He et al., 2016a, He et al., 2016b, Kim et al., 2016a, Huang et al., 2017, Hu et al., 2017, Zhao et al., 2018b). The proposed LIU follows this point and serves as a feature regulator that simulates the Hartline–Ratliff equation and explicitly adjusts the hierarchical features of deep models. The explicit adjustment of the hierarchical features is considered beneficial to alleviate the representational burden of deep models and therefore improve SR performance (Hu et al., 2017, Zhao et al., 2018c). The main contributions of this work are as following:
A lightweight CNN model, LIN, is proposed for efficient and accurate MR image SR tasks. With moderate model parameters and computational overhead, our LIN achieves high-precision and fast SR reconstruction.
Motivated by the lateral inhibition mechanism, we design a local inhibition unit (LIU) to explicitly impose inhibitory regulation on feature maps, alleviating the representational burden of the model.
We propose to integrate the shallow features with different receptive field sizes to boost model performance. Through this strategy, we can increase the diversity of the extracted features and provide more effective evidence for nonlinear inference and image reconstruction.
We experimentally and analytically verify that combining the lateral inhibition mechanism with our shallow feature extraction strategy favors to improving the performance of deep models.
Extensive experiments on various MR images show that our model achieves competitive SR performance with much less model parameters and higher efficiency. The remainder of this paper is organized as follows: in Section 2, we introduce some previous work related to this work. The proposed LIN model is illustrated in Section 3. The experimental results and analyses are given in Section 4, and the conclusion is in Section 5.
Section snippets
MR image super-resolution
High-resolution medical images provide rich structural and textural details that are critical for accurate postprocessing and early diagnoses. However, HR acquisition based on hardware devices typically decreases image SNR and increases scanning time (Greenspan et al., 2002, Plenge et al., 2012, Shi et al., 2015, Chen et al., 2018b). As an alternative, image SR methods are widely used to enhance the resolution of MR images, and many SR techniques for MR images are studied and proposed in the
Motivation: Visual inhibition
In neurobiology, lateral inhibition refers to the phenomenon where the excitation of a neuron in a neural network inhibits the response of its neighbors, thus creating a competition between neurons (Rodieck and Stone, 1965, Arkachar and Wagh, 2007, Bakshi and Ghosh, 2017). It mainly occurs in visual processes and makes visual neurons more sensitive to nonlinear features in the scene (Bakshi and Ghosh, 2017). A famous computing model for simulating this visual inhibition is the Hartline–Ratliff
Dataset
In this paper, the dataset used in the experiments is the same as that used in CSN (Zhao et al., 2018c), which is derived from the IXI dataset.1 It contains three typical MR image types: proton density (PD) images, T1-weighted images and T2-weighted images. For each image type, there are 500, 70 and 6 volumes of size 240 × 240 × 96 (heightwidthdepth) for training, testing and validation, respectively. Besides, two image degradations are also included
Conclusion
We demonstrate a novel CNN model for single MR image SR tasks in this paper, which is motivated by the lateral inhibition mechanism in neurobiology. An inhibition tail that explicitly adjusts the activation of hidden neurons is designed to simulate the Hartline–Ratliff Equation (Hartline and Ratliff, 1974) and used as a regulator of hierarchical features. When the model is lightweight in scale, explicitly imposing inhibitory adjustment on features is considered to help alleviate the
CRediT authorship contribution statement
Xiaole Zhao: Writing - original draft, Supervision, Investigation, Software, Writing - review & editing. Xiafei Hu: Investigation, Software. Ying Liao: Writing - original draft, Investigation. Tian He: Visualization, Investigation. Tao Zhang: Conceptualization, Methodology, Supervision. Xueming Zou: Conceptualization, Supervision. Jinsha Tian: Formal analysis, Validation, Writing - review & editing.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
The work is supported in part by Sichuan Science and Technology Program under Grant 2019YJ0181, and National Key Research and Development Program of China under Grant No. 2016YFC0100800 and 2016YFC0100802.
References (62)
- et al.
Criticality of lateral inhibition for edge enhancement in neural systems
Neurocomputing
(2007) - et al.
A neural model of attention and feedback for computing perceived brightness in vision
Handbook of Neural Computation
(2017) - et al.
A cortical model of winner-take-all competition via lateral inhibition
Neural Netw.
(1992) - et al.
Segmentation from motion of non-rigid objects by neuronal lateral interaction
Pattern Recognit. Lett.
(2001) - et al.
Image enhancement via lateral inhibition: An analysis under illumination changes
Optik
(2016) - et al.
A survey on deep learning in medical image analysis
Med. Image Anal.
(2017) - et al.
Non-local MRI upsampling
Med. Image Anal.
(2010) - et al.
Single-image super-resolution of brain MR images using over complete dictionaries
MIA
(2013) - et al.
Fast, accurate, and lightweight super-resolution with cascading residual network
- et al.
Lateral inhibition-inspired convolutional neural network for visual attention and saliency detection
Efficient and accurate MRI super-resolution using a generative adversarial network and 3D multi-level densely connected network
Edge enhancement using adaptive lateral inhibition
Image super-resolution using deep convolutional networks
IEEE TPAMI
Accelerating the super-resolution convolutional neural network
Lateral inhibition pyramidal neural network for image classification
IEEE Trans. Cybern.
Color video segmentation by lateral inhibition in accumulative computation
Signal Image Video Process.
Understanding the difficulty of training deep feedforward neural networks
Super-resolution in MRI
MRI inter-slice reconstruction using super-resolution
Studies on Excitation and Inhibition in the Retina
Deep residual learning for image recognition
Identity mappings in deep residual networks
Squeeze-and-excitation networks
Densely connected convolutional networks
Fast and accurate single image super-resolution via information distillation network
Deep learning for undersampled MRI reconstruction
Accurate image super-resolution using very deep convolutional networks
Deeply-recursive convolutional network for image super-resolution
Adam: A method for stochastic optimization
Imagenet classification with deep convolutional neural networks
Cited by (10)
Plug-and-Play video super-resolution using edge-preserving filtering
2022, Computer Vision and Image UnderstandingCitation Excerpt :Such technology overcomes the inherent resolution limitations in videos (Elwarfalli and Hardie, 2021). The mentioned task considers the inverse problem of original high resolution video frame recovery using prior information and reasonable suppositions (Singh et al., 2020; Liu et al., 2021; Zhao et al., 2020). The existing video super resolution techniques can be categorized into single frame and multi frame super resolution methods (Yang et al., 2018).
Acne Vulgaris Detection and Classification: A Dual Integrated Deep CNN Model
2023, Informatica (Slovenia)Cross-resolution feature attention network for image super-resolution
2023, Visual ComputerBoosting Single Image Super-Resolution via Partial Channel Shifting
2023, Proceedings of the IEEE International Conference on Computer VisionDenoising Supervision Based Generative Adversarial Networks for MRI Super-Resolution Reconstruction
2023, Lecture Notes in Electrical Engineering