Skip to main content
SpringerLink
Log in

CCOCSA-based multi-frame sparse coding super-resolution via mutual information-based weighted image fusion

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Image super-resolution (SR) is one of the most urgent requirements in many applications in computer vision. Though many techniques have been proposed to date, they are not suitable in real-life applications because of their lack of performance when the low resolution (LR) images are noisy. In the single-frame SR mechanism, the output image can not reproduce the information which is lost due to sub-sampling at the time of image formation. Multi-frame image SR technique can solve this problem to a reasonable extent. However, most of the multi-frame image SR techniques are based on image registration which is noise sensitive. The registration errors in noisy LR images deteriorate the performance of the registration process and hence the performance of the multi-frame SR algorithms. This paper addresses this problem and proposes a novel mutual information-based multi-frame sparse coding super-resolution via Chaotic Centroid-Oppositional Crow Search Optimization Algorithm (CCOCSA)-based image registration. The proposed SR algorithm is accomplished in two steps: (1) CCOCSA-based image registration and (2) Mutual information-based weighted mean multi-frame super-resolution reconstruction through sparse representation. The proposed CCOCSA is a noise-robust optimization that provides better accuracy in estimating registration parameters. We achieve this by enhancing the exploration by dynamically varying the control parameters and introducing chaos within the algorithm. The weighted mean of the registered LR frames based on mutual information provides a good noise reduction capability. Also, it gathers the information from all the LR frames on the registered weighted mean frame, which enriches the information in the reconstructed SR image. The performance of the proposed algorithm is tested through the experiments on four benchmark datasets (Vid4, McMaster, Milanfar, and Kodak) for videos and still images. We verify the performance of the proposed technique by computing four well-known quality metrics, such as Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), Edge Keeping Index (EKI), blur-metric, and Perceptual SIMilarity (PSIM) measure. The experimental results show that the proposed SR algorithm outperforms the others quantitatively and qualitatively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Algorithm 2
Algorithm 3
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

References

  1. An T, Zhang X, Huo C, Xue B, Wang L, Pan C (2022) TR-MISR: multiimage super-resolution based on feature fusion with transformers. IEEE J Select Top Appl Earth Observ Remote Sens 15:1373–1388. https://doi.org/10.1109/JSTARS.2022.3143532

    Article  Google Scholar 

  2. Anter AM, Ali M (2020) Feature selection strategy based on hybrid crow search optimization algorithm integrated with chaos theory and fuzzy c-means algorithm for medical diagnosis problems. Soft Comput 24:1565–1584

    Article  Google Scholar 

  3. Anter AM, Hassenian AE, Oliva D (2019) An improved fast fuzzy c-means using crow search optimization algorithm for crop identification in agricultural. Expert Syst Appl 118:340–354, ISSN 0957–4174. https://doi.org/10.1016/j.eswa.2018.10.009

    Article  Google Scholar 

  4. Askarzadeh A (2016) A novel metaheuristic method for solving constrained engineering optimization problems: crow search algorithm. Comput Struct 169:1–12, ISSN 0045-7949. https://doi.org/10.1016/j.compstruc.2016.03.001

    Article  Google Scholar 

  5. Chakraborty F, Roy PK, Nandi D (2021) A novel chaotic symbiotic organisms search optimization in multilevel image segmentation. Soft Comput 25:6973–6998. https://doi.org/10.1007/s00500-021-05611-w

    Article  Google Scholar 

  6. Chang H, Yeung D-Y, Xiong Y (2004) Super-resolution through neighbor embedding, Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004, pp I-I, https://doi.org/10.1109/CVPR.2004.1315043

  7. Chaudhuri A, Sahu TP (2021) Feature selection using Binary Crow Search Algorithm with time varying flight length. Exp Syst Appl 168:114288, ISSN 0957-4174. https://doi.org/10.1016/j.eswa.2020.114288

    Article  Google Scholar 

  8. Chen X, Wu H (2018) Random sequence generation Algorithm for multi-chaotic systems. In: Meng L, Zhang Y (eds) Machine learning and intelligent communications. MLICOM 2018. Lecture notes of the Institute for Computer Sciences, social informatics and telecommunications engineering, vol 251. Springer, Cham. https://doi.org/10.1007/978-3-030-00557-3_14

    Chapter  Google Scholar 

  9. Cheng M, Wang C, Li J (2015) Single-image super-resolution in RGB space via group sparse representation. IET Image Process 9:461–467. https://doi.org/10.1049/iet-ipr.2014.0313

    Article  Google Scholar 

  10. Dong C, Loy CC, He K, Tang X (2016) Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 38(2):295–307. https://doi.org/10.1109/TPAMI.2015.2439281

    Article  Google Scholar 

  11. dos Santos Coelho L, Richter C, Mariani VC, Askarzadeh A (2016) Modified crow search approach applied to electromagnetic optimization, 2016 IEEE Conference on Electromagnetic Field Computation (CEFC), pp 1–1, https://doi.org/10.1109/CEFC.2016.7815927

  12. Farsiu S, Robinson MD, Elad M, Milanfar P (2004) Fast and robust multiframe super resolution. IEEE Trans Image Process 13(10):1327–1344. https://doi.org/10.1109/TIP.2004.834669

    Article  Google Scholar 

  13. Glasner D, Bagon S, Irani M (2009) Super-resolution from a single image, 2009 IEEE 12th International Conference on Computer Vision, pp 349–356, https://doi.org/10.1109/ICCV.2009.5459271

  14. Gu K, Li L, Lu H, Min X, Lin W (2017) A fast reliable image quality predictor by fusing Micro- and macro-structures. IEEE Trans Ind Electron 64(5):3903–3912. https://doi.org/10.1109/TIE.2017.2652339

    Article  Google Scholar 

  15. Gupta D, Rodrigues JJPC, Sundaram S et al (2020) Usability feature extraction using modified crow search algorithm: a novel approach. Neural Comput & Applic 32:10915–10925. https://doi.org/10.1007/s00521-018-3688-6

    Article  Google Scholar 

  16. Han X, Xu Q, Yue L, Dong Y, Xie G, Xu X (2020) An improved crow search Algorithm based on spiral search mechanism for solving numerical and engineering optimization problems. IEEE Access 8:92363–92382. https://doi.org/10.1109/ACCESS.2020.2980300

    Article  Google Scholar 

  17. Irani M, Peleg S (1991) Improving resolution by image registration. CVGIP: Graph Models Image Process 53(3):231–239, ISSN 1049-9652. https://doi.org/10.1016/1049-9652(91)90045-L

    Article  Google Scholar 

  18. Karaboga D, Basturk B (2007) A powerful and efficient algorithm for numerical function optimization: artificial bee colony (ABC) algorithm. J Glob Optim 39:459–471

    Article  MathSciNet  Google Scholar 

  19. Karthikumar K, Senthil Kumar V (2021) A new opposition crow search optimizer-based two-step approach for controlled intentional islanding in microgrids. Soft Comput 25:2575–2588. https://doi.org/10.1007/s00500-020-05280-1

    Article  Google Scholar 

  20. Kato T, Hino H, Murata N (2017) Double sparsity for multi-frame super resolution. Neurocomput 240:115–126, ISSN 0925-2312. https://doi.org/10.1016/j.neucom.2017.02.043

    Article  Google Scholar 

  21. Kim KI, Kwon Y (2010) Single-image super-resolution using sparse regression and natural image prior. IEEE Trans Pattern Anal Mach Intell 32(6):1127–1133. https://doi.org/10.1109/TPAMI.2010.25

    Article  Google Scholar 

  22. Kodak E (n.d.) Kodak lossless true color image suite (Photo PCD 0992). https://r0k.us/graphics/kodak/

  23. Liu C, Sun D (2011) A Bayesian approach to adaptive video super resolution. CVPR 2011:209–216. https://doi.org/10.1109/CVPR.2011.5995614

    Article  Google Scholar 

  24. Mahdavi S, Rahnamayan S, Deb K (2018) Opposition based learning: a literature review. Swarm Evol Comput 39:1–23, ISSN 2210-6502. https://doi.org/10.1016/j.swevo.2017.09.010

    Article  Google Scholar 

  25. Majhi SK, Sahoo M, Pradhan R (2021) Oppositional crow search Algorithm with mutation operator for global optimization and application in designing FOPID controller. Evol Syst 12:463–488. https://doi.org/10.1007/s12530-019-09305-5

    Article  Google Scholar 

  26. Mirjalili S (2015) Moth-flame optimization algorithm: a novel nature-inspired heuristic paradigm. Knowl Based Syst 89:228–249, ISSN 0950-7051. https://doi.org/10.1016/j.knosys.2015.07.006

    Article  Google Scholar 

  27. Mirjalili S, Gandomi AH, Mirjalili SZ, Saremi S, Faris H, Mirjalili SM (2017) Salp Swarm Algorithm: A bio-inspired optimizer for engineering design problems. Adv Eng Softw 114:163–191, ISSN 0965-9978. https://doi.org/10.1016/j.advengsoft.2017.07.002

    Article  Google Scholar 

  28. Mohammadi F, Abdi H (2018) A modified crow search algorithm (MCSA) for solving economic load dispatch problem. Appl Soft Comput 71:51–65, ISSN 1568-4946. https://doi.org/10.1016/j.asoc.2018.06.040

    Article  Google Scholar 

  29. Morales-Castañeda B, Zaldívar D, Cuevas E, Fausto F, Alma Rodríguez A (2020) Better balance in metaheuristic algorithms: Does it exist? Swarm Evol Comput 54:100671, ISSN 2210-6502. https://doi.org/10.1016/j.swevo.2020.100671

    Article  Google Scholar 

  30. Mousavi HS, Monga V (2017) Sparsity-based color image super resolution via exploiting Cross Channel constraints. IEEE Trans Image Process 26(11):5094–5106. https://doi.org/10.1109/TIP.2017.2704443

    Article  MathSciNet  Google Scholar 

  31. Nandi D, Karmakar J, Kumar A, Mandal MK (2019) Sparse representation based multi-frame image super-resolution reconstruction using adaptive weighted features. IET Image Process 13(4):663–672. https://doi.org/10.1049/iet-ipr.2018.5139

    Article  Google Scholar 

  32. Oliva D, Hinojosa S, Cuevas E, Pajares G, Avalos O, Gálvez J (2017) Cross entropy based thresholding for magnetic resonance brain images using crow search Algorithm. Exp Syst Appl 79:164–180, ISSN 0957-4174. https://doi.org/10.1016/j.eswa.2017.02.042

    Article  Google Scholar 

  33. Ouadfel S, Elaziz MA (2020) Enhanced Crow Search Algorithm for Feature Selection. Exp Syst Appl 159:113572, ISSN 0957-4174. https://doi.org/10.1016/j.eswa.2020.113572

    Article  Google Scholar 

  34. Qingzheng X, Wang L, Wang N, Hei X, Li Zhao A (2014) Review of opposition-based learning from 2005 To 2012. Eng Appl Artif Intell 29:1–12, ISSN 0952–1976. https://doi.org/10.1016/j.engappai.2013.12.004

    Article  Google Scholar 

  35. Rahnamayan S, Tizhoosh HR, Salama MMA (2008) Opposition-based differential evolution. IEEE Trans Evol Comput 12(1):64–79. https://doi.org/10.1109/TEVC.2007.894200

    Article  Google Scholar 

  36. Sayed GI, Hassanien AE, Azar AT (2019) Feature selection via a novel chaotic crow search algorithm. Neural Comput & Applic 31:171–188. https://doi.org/10.1007/s00521-017-2988-6

    Article  Google Scholar 

  37. Seyedali Mirjalili SCA (2016) A sine cosine Algorithm for solving optimization problems. Knowl-Based Syst 96:120–133, ISSN 0950-7051. https://doi.org/10.1016/j.knosys.2015.12.022

    Article  Google Scholar 

  38. Sharma M, Kaur P (2021) A comprehensive analysis of nature-inspired Meta-heuristic techniques for feature selection problem. Arch Computat Methods Eng 28:1103–1127. https://doi.org/10.1007/s11831-020-09412-6

    Article  MathSciNet  Google Scholar 

  39. Shekhawat S, Saxena A (2020) Development and applications of an intelligent crow search algorithm based on opposition based learning. ISA Trans 99:210–230, ISSN 0019-0578. https://doi.org/10.1016/j.isatra.2019.09.004

    Article  Google Scholar 

  40. Sun J, Zongben X, Shum H-Y (2008) Image super-resolution using gradient profile prior. IEEE Conf Comput Vis Pattern Recogn 2008:1–8. https://doi.org/10.1109/CVPR.2008.4587659

    Article  Google Scholar 

  41. Trinh D, Luong M, Dibos F, Rocchisani J, Pham C, Nguyen TQ (2014) Novel example-based method for super-resolution and Denoising of medical images. IEEE Trans Image Process 23(4):1882–1895. https://doi.org/10.1109/TIP.2014.2308422

    Article  MathSciNet  Google Scholar 

  42. Vella M, Mota JFC (2021) Robust single-image super-resolution via CNNs and TV-TV minimization. IEEE Trans Image Process 30:7830–7841. https://doi.org/10.1109/TIP.2021.3108907

    Article  Google Scholar 

  43. Wachowiak MP, Smolikova R, Zheng Y, Zurada JM, Elmaghraby AS (2004) An approach to multimodal biomedical image registration utilizing particle swarm optimization. IEEE Trans Evol Comput 8(3):289–301. https://doi.org/10.1109/TEVC.2004.826068

    Article  Google Scholar 

  44. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  45. Wang Y, Fevig R, Schultz RR (2008) Super-resolution mosaicking of UAV surveillance video, 2008 15th IEEE International Conference on Image Processing, pp 345–348, https://doi.org/10.1109/ICIP.2008.4711762

  46. Wang P, Hu X, Xuan B, Mu J, Peng S (2011) Super resolution reconstruction via multiple frames joint learning. Int Conf Multimed Signal Process 2011:357–361. https://doi.org/10.1109/CMSP.2011.79

    Article  Google Scholar 

  47. Wang Z et al (2019) Multi-memory convolutional neural network for video super-resolution. IEEE Trans Image Process 28(5):2530–2544. https://doi.org/10.1109/TIP.2018.2887017

    Article  MathSciNet  Google Scholar 

  48. Wang X et al (2019) ESRGAN: enhanced super-resolution generative adversarial networks. In: Leal-Taixé L, Roth S (eds) Computer vision – ECCV 2018 workshops. ECCV 2018. Lecture notes in computer science(), vol 11133. Springer, Cham. https://doi.org/10.1007/978-3-030-11021-5_5

    Chapter  Google Scholar 

  49. Wei W, Liu Z, Gueaieb W, He X (2011) Single-image super-resolution based on Markov random field and contourlet transform. J Electron Imaging 20(2):023005. https://doi.org/10.1117/1.3580750

    Article  Google Scholar 

  50. Wronski B, Garcia-Dorado I, Ernst M, Kelly D, Krainin M, Liang CK, Levoy M, Milanfar P (2019) Handheld multi-frame super-resolution. ACM Trans Graph (TOG) 38(4):1–18

    Article  Google Scholar 

  51. Xu Z, Ma Q, Yuan F (2020) Single color image super-resolution using sparse representation and color constraint. J Syst Eng Electron 31(2):266–271. https://doi.org/10.23919/JSEE.2020.000004

    Article  Google Scholar 

  52. Yang XS, Deb S (2014) Cuckoo search: recent advances and applications. Neural Comput & Applic 24:169–174. https://doi.org/10.1007/s00521-013-1367-1

    Article  Google Scholar 

  53. Yang J, Wright J, Huang TS, Ma Y (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873. https://doi.org/10.1109/TIP.2010.2050625

    Article  MathSciNet  Google Scholar 

  54. Zamani H, Nadimi-Shahraki MH, Gandomi AH (2019) CCSA: Conscious Neighborhood-based Crow Search Algorithm for Solving Global Optimization Problems. Appl Soft Comput 85:105583, ISSN 1568-4946. https://doi.org/10.1016/j.asoc.2019.105583

    Article  Google Scholar 

  55. Zhang K, Wang B, Zuo W, Zhang H, Zhang L (2016) Joint learning of multiple Regressors for single image super-resolution. IEEE Signal Process Lett 23(1):102–106. https://doi.org/10.1109/LSP.2015.2504121

    Article  Google Scholar 

  56. Zhang Q et al (2020) TV-sparse super-resolution method for radar forward-looking imaging. IEEE Trans Geosci Remote Sens 58(9):6534–6549. https://doi.org/10.1109/TGRS.2020.2977719

    Article  Google Scholar 

  57. Zolghadr-Asli B, Bozorg-Haddad O, Chu X (2018) Crow search Algorithm (CSA). In: Bozorg-Haddad O (ed) Advanced optimization by nature-inspired algorithms. Studies in computational intelligence, vol 720. Springer, Singapore. https://doi.org/10.1007/978-981-10-5221-7_14

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, National Institute of Technology, Durgapur, 713209, India

    Amit Mukhopadhyay, Debashis Nandi & Baisakhi Chakraborty

  2. Indian Statistical Institute, CVPR Unit, Kolkata, 700009, India

    Umapada Pal

Authors
  1. Amit Mukhopadhyay
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Debashis Nandi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Umapada Pal
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Baisakhi Chakraborty
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Debashis Nandi.

Ethics declarations

This research is a part of the doctoral programme in NIT, Durgapur. No special funds, grants, or any other support was received.

The authors have no competing interests to declare that are relevant to the content of this article.

Data sharing is not applicable to this article as no separate benchmark dataset was generated or analyzed during the current study. However, we captured few real-life scene images for the purpose of testing the algorithm and are uploaded with the paper. The benchmark datasets used in this study are publicly available [12, 22, 23, 49].

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mukhopadhyay, A., Nandi, D., Pal, U. et al. CCOCSA-based multi-frame sparse coding super-resolution via mutual information-based weighted image fusion. Multimed Tools Appl 83, 2427–2471 (2024). https://doi.org/10.1007/s11042-023-15647-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-15647-9

Keywords

Search

Navigation