Skip to main content
SpringerLink
Log in

A new content-aware image resizing based on Rényi entropy and deep learning

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

One of the most popular techniques for changing the purpose of an image or resizing a digital image with content awareness is the seam-carving method. The performance of image resizing algorithms based on seam machining shows that these algorithms are highly dependent on the extraction of importance map techniques and the detection of salient objects. So far, various algorithms have been proposed to extract the importance map. In this paper, a new method based on Rényi entropy is proposed to extract the importance map. Also, a deep learning network has been used to detect salient objects. The simulator results showed that combining Rényi’s importance map with a deep network of salient object detection performed better than classical seam-carving and other extended seam-carving algorithms based on deep learning.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

The data that support the findings of this study are available from: MSRA-10k, DUTS, ECSSD, HKU-IS, and RetargetMe.

Notes

  1. https://people.csail.mit.edu/mrub/retargetme/download.html.

References

  1. Avidan S, Shamir A (2007) Seam carving for content-aware image resizing. ACM Trans Graph (TOG) 26:10

    Article  Google Scholar 

  2. Guo D, Ding J, Tang J, Xu M, Zhao C (2015) NIF-based seam carving for image resizing. Multimed Syst 21(6):603–613

    Article  Google Scholar 

  3. Hashemzadeh M, Asheghi B, Farajzadeh N (2019) Content-aware image resizing: an improved and shadow-preserving seam carving method. Signal Process 155:233–246

    Article  Google Scholar 

  4. Rubinstein M, Shamir A, Avidan S (2008) Improved seam carving for video retargeting. ACM Trans Graph (TOG) 27:16

    Article  Google Scholar 

  5. Ayubi P, Setayeshi S, Rahmani AM (2020) Deterministic chaos game: a new fractal based pseudo-random number generator and its cryptographic application. J Inf Secur Appl 52:102472

    Google Scholar 

  6. Ayubi P, Jafari Barani M, Yousefi Valandar M, Yosefnezhad Irani B, Sedagheh Maskan Sadigh R (2021) A new chaotic complex map for robust video watermarking. Artif Intell Rev 54(2):1237–1280

    Article  Google Scholar 

  7. Lopes R, Betrouni N (2009) Fractal and multifractal analysis: a review. Med Image Anal 13(4):634–649

    Article  Google Scholar 

  8. Fisher Y (2012) Fractal image compression: theory and application. Springer, Cham

    Google Scholar 

  9. Itti L, Koch C, Niebur E (1998) A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Pattern Anal Mach Intell 11:1254–1259

    Article  Google Scholar 

  10. Suh B, Ling H, Bederson BB, Jacobs DW (2003) Automatic thumbnail cropping and its effectiveness. In: Proceedings of the 16th annual ACM symposium on User interface software and technology. ACM, pp 95–104

  11. Chen L-Q, Xie X, Fan X, Ma W-Y, Zhang H-J, Zhou H-Q (2003) A visual attention model for adapting images on small displays. Multimed Syst 9(4):353–364

    Article  Google Scholar 

  12. Zhang M, Zhang L, Sun Y, Feng L, Ma W (2005) Auto cropping for digital photographs. In: 2005 IEEE international conference on multimedia and expo. IEEE, p 4

  13. Santella A, Agrawala M, DeCarlo D, Salesin D, Cohen M (2006) Gaze-based interaction for semi-automatic photo cropping. In: Proceedings of the SIGCHI conference on human factors in computing systems. ACM, pp 771–780

  14. Li X, Ling H (2009) Learning based thumbnail cropping. In: 2009 IEEE international conference on multimedia and expo. IEEE, pp 558–561

  15. Nishiyama M, Okabe T, Sato Y, Sato I (2009) Sensation-based photo cropping. In: Proceedings of the 17th ACM international conference on multimedia. ACM, pp 669–672

  16. Luo Y, Yuan J, Xue P, Tian Q (2011) Saliency density maximization for efficient visual objects discovery. IEEE Trans Circuits Syst Video Technol 21(12):1822–1834

    Article  Google Scholar 

  17. Achanta R, Süsstrunk S (2009) Saliency detection for content-aware image resizing. In: 2009 16th IEEE international conference on image processing (ICIP). IEEE, pp 1005–1008

  18. Choi J, Kim C (2016) Sparse seam-carving for structure preserving image retargeting. J Signal Process Syst 85(2):275–283

    Article  Google Scholar 

  19. Han D, Sonka M, Bayouth J, Wu X (2010) Optimal multiple-seams search for image resizing with smoothness and shape prior. Vis Comput 26(6–8):749–759

    Article  Google Scholar 

  20. Shafieyan F, Karimi N, Mirmahboub B, Samavi S, Shirani S (2017) Image retargeting using depth assisted saliency map. Signal Process Image Commun 50:34–43

    Article  Google Scholar 

  21. Zhang L, Li K, Ou Z, Wang F (2017) Seam warping: a new approach for image retargeting for small displays. Soft Comput 21(2):447–457

    Article  Google Scholar 

  22. Goferman S, Zelnik-Manor L, Tal A (2011) Context-aware saliency detection. IEEE Trans Pattern Anal Mach Intell 34(10):1915–1926

    Article  Google Scholar 

  23. Ito I (2016) Gradient-based global features for seam carving. EURASIP J Image Video Process 2016(1):1–9

    Article  Google Scholar 

  24. Yin T, Yang G, Li L, Zhang D, Sun X (2015) Detecting seam carving based image resizing using local binary patterns. Comput Secur 55:130–141

    Article  Google Scholar 

  25. Zhang D, Li Q, Yang G, Li L, Sun X (2017) Detection of image seam carving by using weber local descriptor and local binary patterns. J Inf Secur Appl 36:135–144

    Google Scholar 

  26. Ye J, Shi Y-Q (2017) An effective method to detect seam carving. J Inf Secur Appl 35:13–22

    Google Scholar 

  27. Li Y, Xia M, Liu X, Yang G (2020) Identification of various image retargeting techniques using hybrid features. J Inf Secur Appl 51:102459

    Google Scholar 

  28. Liu F, Gleicher M (2005) Automatic image retargeting with fisheye-view warping. In: Proceedings of the 18th annual ACM symposium on user interface software and technology. ACM, pp 153–162

  29. Gal R, Sorkine O, Cohen-Or D (2006) Feature-aware texturing. Render Tech 2006(17th):2

    Google Scholar 

  30. Wang Y-S, Tai C-L, Sorkine O, Lee T-Y (2008) Optimized scale-and-stretch for image resizing. ACM Trans Graph (TOG) 27:118

    Article  Google Scholar 

  31. Zhang G-X, Cheng M-M, Hu S-M, Martin RR (2009) A shape-preserving approach to image resizing. In: Computer graphics forum, vol 28. Wiley Online Library, pp 1897–1906

  32. Jin Y, Liu L, Wu Q (2010) Nonhomogeneous scaling optimization for realtime image resizing. Vis Comput 26(6–8):769–778

    Article  Google Scholar 

  33. Niu Y, Liu F, Li X, Gleicher M (2012) Image resizing via non-homogeneous warping. Multimed Tools Appl 56(3):485–508

    Article  Google Scholar 

  34. Guo Y, Liu F, Shi J, Zhou Z-H, Gleicher M (2009) Image retargeting using mesh parametrization. IEEE Trans Multimed 11(5):856–867

    Article  Google Scholar 

  35. Cui J, Cai Q, Lu H, Jia Z, Tang M (2020) Distortion-aware image retargeting based on continuous seam carving model. Signal Process 166:107242

    Article  Google Scholar 

  36. Xu L, Ren JS, Liu C, Jia J (2014) Deep convolutional neural network for image deconvolution. In: Advances in neural information processing systems, pp 1790–1798

  37. Cheng Z, Yang Q, Sheng B (2015) Deep colorization. In: Proceedings of the IEEE international conference on computer vision, pp 415–423

  38. Xie J, Xu L, Chen E (2012) Image denoising and inpainting with deep neural networks. In: Advances in neural information processing systems, pp 341–349

  39. Xu N, Price B, Cohen S, Huang T (2017) Deep image matting. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2970–2979

  40. Cho D, Park J, Oh T-H, Tai Y-W, So Kweon I (2017) Weakly-and self-supervised learning for content-aware deep image retargeting. In: Proceedings of the IEEE international conference on computer vision, pp 4558–4567

  41. Arar M, Danon D, Cohen-Or D, Shamir A. Image resizing by reconstruction from deep features. arXiv preprint arXiv:1904.08475

  42. Song E, Lee M, Lee S (2018) Carvingnet: content-guided seam carving using deep convolution neural network. IEEE Access 7:284–292

    Article  Google Scholar 

  43. Abebe MA, Hardeberg JY (2018) Application of radial basis function interpolation for content aware image retargeting. In: 2018 14th international conference on signal-image technology & internet-based systems (SITIS). IEEE, pp 174–183

  44. Zhang Y, He X, Tian Z, Jeong JJ, Lei Y, Wang T, Zeng Q, Jani AB, Curran WJ, Patel P et al (2020) Multi-needle detection in 3d ultrasound images using unsupervised order-graph regularized sparse dictionary learning. IEEE Trans Med Imaging 39(7):2302–2315

    Article  Google Scholar 

  45. Zhang Y, Liu S, Qu X, Shang X (2022) Multi-instance discriminative contrastive learning for brain image representation. Neural Comput Appl. https://doi.org/10.1007/s00521-022-07524-7

    Article  Google Scholar 

  46. Battiato S, Farinella GM, Puglisi G, Ravi D (2014) Saliency-based selection of gradient vector flow paths for content aware image resizing. IEEE Trans Image Process 23(5):2081–2095

    Article  MathSciNet  Google Scholar 

  47. Shamir A, Avidan S (2009) Seam carving for media retargeting. Commun ACM 52(1):77–85

    Article  Google Scholar 

  48. Chen L-C, Papandreou G, Schroff F, Adam H. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587

  49. Shannon CE (1949) Communication theory of secrecy systems. Bell Syst Tech J 28(4):656–715

    Article  MathSciNet  Google Scholar 

  50. Rényi A (1961) On measures of information and entropy. In: Proceedings of the 4th Berkeley symposium on mathematics, statistics and probability, p 1

  51. Wang L, Lu H, Wang Y, Feng M, Wang D, Yin B, Ruan X (2017) Learning to detect salient objects with image-level supervision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 136–145

  52. Shi J, Yan Q, Xu L, Jia J (2015) Hierarchical image saliency detection on extended CSSD. IEEE Trans Pattern Anal Mach Intell 38(4):717–729

    Article  Google Scholar 

  53. Li G, Yu Y (2015) Visual saliency based on multiscale deep features. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5455–5463

  54. Borji A, Cheng M-M, Jiang H, Li J (2015) Salient object detection: a benchmark. IEEE TIP 24(12):5706–5722

    MathSciNet  Google Scholar 

  55. Rubinstein M, Gutierrez D, Sorkine O, Shamir A (2010) A comparative study of image retargeting. In: ACM SIGGRAPH Asia 2010 papers, pp 1–10

  56. Rubinstein M, Shamir A, Avidan S (2009) Multi-operator media retargeting. ACM Trans Graph (TOG) 28(3):1–11

    Article  Google Scholar 

  57. Pritch Y, Kav-Venaki E, Peleg S (2009) Shift-map image editing. In: IEEE 12th international conference on computer vision. IEEE, pp 151–158

  58. Krähenbühl P, Lang M, Hornung A, Gross M (2009) A system for retargeting of streaming video. In: ACM SIGGRAPH Asia 2009 papers, pp 1–10

  59. Zhang Y, Lin W, Zhang X, Fang Y, Li L (2016) Aspect ratio similarity (ARS) for image retargeting quality assessment. In: 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, pp 1080–1084

  60. Lin J, Zhou T, Chen Z (2019) Deepir: a deep semantics driven framework for image retargeting. In: 2019 IEEE international conference on multimedia & expo workshops (ICMEW). IEEE, pp 54–59

  61. Tan W, Yan B, Lin C, Niu X (2019) Cycle-IR: deep cyclic image retargeting. IEEE Trans Multimed 22(7):1730–1743

    Article  Google Scholar 

Download references

Acknowledgements

This article is dedicated to Imam Hassan, Karim Ahl al-Bayt.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, Urmia University, Urmia, Iran

    Jila Ayubi, Mehdi Chehel Amirani & Morteza Valizadeh

Authors
  1. Jila Ayubi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mehdi Chehel Amirani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Morteza Valizadeh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Morteza Valizadeh.

Ethics declarations

Conflict of interest

The authors of this paper confirm that any part of this work was not published or submitted for publication elsewhere, and the authors do not have any conflict of interest with anybody else.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ayubi, J., Chehel Amirani, M. & Valizadeh, M. A new content-aware image resizing based on Rényi entropy and deep learning. Neural Comput & Applic 36, 8885–8899 (2024). https://doi.org/10.1007/s00521-024-09517-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-024-09517-0

Keywords

Search

Navigation