Feedback attention network for cardiac magnetic resonance imaging super-resolution

https://doi.org/10.1016/j.cmpb.2022.107313Get rights and content

Highlights

  • Proposed the feedback attention network structure.

  • Developed the multi-scale residual group mudule.

  • Designed a mixed attention mechanism.

Abstract

Background and objective

Atrial fibrillation (AF) is a common clinical arrhythmia with a high disability and mortality rate. Improving the resolution of atrial structure and its changes in patients with AF is very important for understanding and treating AF.

Methods

Aiming at the problems of previous deep learning-based image super-resolution (SR) reconstruction methods simply deepening the network, loss of upsampling information, and difficulty in the reconstruction of high-frequency information, we propose the Feedback Attention Network (FBAN) for cardiac magnetic resonance imaging (CMRI) super-resolution. The network comprises a preprocessing module, a multi-scale residual group module, an upsampling module, and a reconstruction module. The preprocessing module uses a convolutional layer to extract shallow features and dilate the number of channels of the feature map. The multi-scale residual group module adds a multi-channel network, a mixed attention mechanism, and a long and short skip connection to expand the receptive field of the feature map, improve the multiplexing of multi-scale features and strengthen the reconstruction of high-frequency information. The upsampling module adopts the sub-pixel method to upsample the feature map to the target image size. The reconstruction module consists of a convolutional layer, which is used to restore the number of channels of the feature map to the original number to obtain the reconstructed high-resolution (HR) image.

Results

Furthermore, the test results on the public dataset of CMRI show that the HR images reconstructed by the FBAN method not only have a good improvement in reconstructed edge and texture information but also have a good improvement in the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) objective evaluation indicators.

Conclusion

Compared with the local magnified image, the edge information of the FBAN method reconstructed image has been enhanced, more high-frequency information of the CMRI is restored, the texture details are less lost, and the reconstructed image is less blurry. Overall, the reconstructed image has a lighter feeling of smearing, and the visual experience is more apparent and sharper.

Introduction

Atrial fibrillation (AF) is the most common persistent heart rate disorder caused by a fast and irregular beating of the heart. The odds of developing AF increase significantly with age. Approximately two percent of people under 65 years of age have AF, and nine percent of those 65 years or older have AF [1,2]. The AF predisposes the body to blood clots, which block blood vessels and significantly increase heart failure and strokes, resulting in higher morbidity and mortality. According to the survey, among people over 60 years old, strokes due to AF accounts for one-fifth [3].

Recurrent episodes of AF also lead to further changes in structural features, namely structural remodeling of the atrium (dilation, muscle fiber changes, and fibrosis). Therefore, the direct study of the atrial structure and its changes in patients with AF is crucial for understanding and treating AF [4,5]. To date, gadolinium-based contrast agents are used in one-third of cardiac magnetic resonance imaging (CMRI) scans to enhance the clarity of images of the anatomy of a patient's internal organs by improving the visibility of common disease-related structures (e.g., fibrosis, inflammation), such as tumors and blood vessels [5]. Clinical studies of AF patients have shown that atrial fibrosis's extent and distribution are reliable predictors of catheter ablation success and can be used for patient stratification in medical management. However, experts cannot make an accurate diagnosis due to the low-resolution (LR) of CMRI caused by medical equipment and improper operation. The use of deep learning technology to improve the resolution is of great significance to assist experts in the medical treatment of AF [6].

In recent years, the problem of image super-resolution (SR) reconstruction has received extensive attention from scholars at home and abroad. Currently, the most studied is single-image super-resolution reconstruction (SISR). The task of SISR is to reconstruct corresponding high-resolution (HR) images from degraded LR images, but this is an ill-conditioned inverse problem because one LR image corresponds to multiple HR images, and different methods rarely construct the HR images are also different [7,8].

With the development of deep learning, more and more image SR methods based on deep convolutional neural networks have been proposed. In 2014, Dong et al. [9] introduced depthwise convolution to the field of SISR and proposed a deep neural network with three layers of convolutional neural network (CNN), namely the SRCNN method. Compared with traditional sparse coding methods, SRCNN has more advanced performance, which has triggered the upsurge of deep learning in the field of SR. On this basis, Dong et al. [10] proposed the FSRCNN method. Shi et al. [11] proposed the ESPCN method, which directly inputs the LR image without interpolation and amplification to accelerate the network's performance.

Existing studies have shown that more profound and broader networks can usually bring better performance, but deeper and broader networks often get difficulties in network training [12]. He et al. [13] proposed a deep residual network (ResNet). The proposal of ResNet not only enables the movement of deeper networks but also improves network performance to a certain extent. Therefore, Kim et al. [14] proposed the VDSR method based on previous research scholars, further increasing the depth of the network to twenty layers and introducing global residual learning to ease the difficulty of training so that the network performance has been dramatically improved. Furthermore, Kim et al. [15] proposed the DRCN method, which uses the recursive idea to simplify the network and achieves a similar effect to VDSR. Inspired by VDSR and DRCN, Tai et al. [16] proposed the DRRN method, which increased the network depth and improved performance by combining local and global residuals.

Although the above algorithm achieved good results, it gave the same weight to all channels with different amounts of information in network training. Therefore, Zhang et al. [17] used the channel attention mechanism in the SR field to treat other channels differently and proposed residual channel attention networks (RCAN). However, the RCAN method only performs upsampling once and cannot fully use high-frequency information. Later, Harris et al. [18] proposed the DBPN method based on the back-projection network. In an iterative process, the error feedback mechanism is used to adjust the error, so that the neural network can better learn the mapping relationship between LR and HR.

Section snippets

Residual learning

The classic convolutional or fully connected layers will have problems such as information loss and loss during information transmission. Residual networks solve this problem to some extent, protecting the integrity of the information by directly detouring the input information to the output. The entire network only needs to learn the art of the difference between input and output, simplifying the learning objective and difficulty [19]. The residual block is constructed based on a feedforward

Feedback residual attention network

The network structure of our proposed Feedback Attention Network (FBAN) method is shown in Fig. 3, which consists of a shallow initial feature layer, a recurrent feedback attention module, and a reconstruction module. Where ILR0 represents the input LR image, ISR0 represents the original SR image after cyclic the FBAB module reconstruction, and ISR1 represents the final SR image.

Dataset and training details

The DIV2K dataset [23] is used as a training and validation dataset, a high-quality image dataset containing 800 training images and 100 validation images. The training data is rotated, scaled, and flipped to enhance. In the training dataset, LR images and HR images exist in pairs. To obtain the corresponding LR images in the DIV2K training dataset, the HR images are scaled down using Bicubic interpolation with scaling factors in Matlab R2021b. The test dataset is Set5 [24], Set14 [25],

Conclusion

In this paper, we propose the Feedback Attention Network (FBAN) super-resolution method for the rich texture of CMRI. The FBAN method realizes the multiplexing of network parameters by iterating the feedback attention module many times, improves the utilization of high-frequency information, recognizes the multiple utilization of LR image, and then obtains the final HR image through the sub-pixel convolution layer. To a certain extent, the problem of high-frequency information loss is

Ethical approval

No ethics approval was required.

Data availability statement

The datasets [GENERATED/ANALYZED] for this study can be found in the [Cardiac MRI dataset] [https://digital-heart.org/], [AMRG Cardiac Atlas] [http://www.cardiacatlas.org/studies/amrg-cardiac-atlas/].

Declaration of Competing Interest

The authors declare that they have no conflicts of interest.

Acknowledgment

This work was supported by the National Natural Science Foundation of China (Grant Nos. 71673143 and 18ZDA327).

References (32)

  • J. Yang et al.

    Image super-resolution via sparse representation

    IEEE Trans. Image Process.

    (2010)
  • C. Dong et al.

    Learning a deep convolutional network for image super-resolution

  • C. Dong et al.

    Accelerating the super-resolution convolutional neural network

  • W. Shi et al.

    Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network

  • K. He et al.

    Deep residual learning for image recognition

  • J. Kim et al.

    Accurate image super-resolution using very deep convolutional networks

  • Cited by (4)

    • PET Imaging Super-Resolution Using Attention-Enhanced Global Residual Dense Network

      2023, 2023 IEEE 3rd International Conference on Computer Systems, ICCS 2023
    View full text