Skip to main content
SpringerLink
Log in

IRNet-RS: image retargeting network via relative saliency

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Image retargeting (IR) aims to fit different display terminals by changing the image aspect ratio while retaining its important content. The IR methods utilize a saliency map to predict the importance of pixels; however, existing methods assume that multiple objects within the saliency map have the same importance, which does not mimic the human attention mechanism and thus results in inadequate protection of the main content during the retargeting operation. To solve these problems, we propose an image retargeting network via relative saliency (IRNet-RS), which has three key steps: relative saliency detection, edge detection, and IR operation. By using a relative saliency stratified supervision module and saliency rank guidance for the refinement module, IRNet-RS detects the salient objects in an image, allocates different saliency weights to them to obtain a relative saliency map, and then fuses it with the edge detection map to generate an importance map. An adaptive duplicate convolution module is also designed to help the network learn the pixel-by-pixel shift map from the source to the target grid, followed by retargeting the image using pixel shifting. Our experimental results show that the proposed IRNet-RS achieves better performance than alternative methods in terms of saliency rank prediction and retargeting results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Algorithm 1
Algorithm 2
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data availability

The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Avidan S, Shamir A (2007) Seam carving for content-aware image resizing. ACM Trans Graph 26(3):10

    Article  Google Scholar 

  2. Garg A, Negi A, Jindal P (2021) Structure preservation of image using an efficient content-aware image retargeting technique. Signal Image Video Process 15(1):185–193

    Article  Google Scholar 

  3. Naderi MR, Givkashi MH, Karimi N, Shirani S, Samavi S (2022) Oair: object-aware image retargeting using pso and aesthetic quality assessment. arXiv preprint arXiv:2209.04804

  4. Tang Z, Yao J, Zhang Q (2022) Multi-operator image retargeting in compressed domain by preserving aspect ratio of important contents. Multimed Tools Appl 81(1):1501–1522

    Article  Google Scholar 

  5. Asheghi B, Salehpour P, Khiavi AM, Hashemzadeh M (2022) A comprehensive review on content-aware image retargeting: from classical to state-of-the-art methods. Signal Process 108496

  6. Alswaidan Z, Shullar MH, Chikhaoui K, Alfarraj M (2022) Object-aware adaptive image retargeting via importance map fusion. Available at SSRN 4206874

  7. Guo YC, Liang YH, Yu M, Zhang TT (2018) An improved seam carving algorithm based on image blocking and optimized cumulative energy map. J Electron Inf Technol 40(2):331–337

    Google Scholar 

  8. Cui J, Cai Q, Lu H, Jia Z, Tang M (2020) Distortion-aware image retargeting based on continuous seam carving model. Signal Process 166:107242

    Article  Google Scholar 

  9. Garg A, Nayyar A, Singh AK (2022) Improved seam carving for structure preservation using efficient energy function. Multimed Tools Appl 81(9):12883–12924

    Article  Google Scholar 

  10. Wolf L, Guttmann M, Cohen Or D (2007) Non-homogeneous content-driven video-retargeting. In: 2007 IEEE 11th international conference on computer vision. IEEE, pp 1–6

  11. Kim Y, Jung S, Jung C, Kim C (2018) A structure-aware axis-aligned grid deformation approach for robust image retargeting. Multimed Tools Appl 77(6):7717–7739

    Article  Google Scholar 

  12. Rubinstein M, Shamir A, Avidan S (2009) Multi-operator media retargeting. ACM Trans Graph (TOG) 28(3):1–11

    Article  Google Scholar 

  13. Dong WM, Bao GB, Zhang XP, Paul JC (2012) Fast multi-operator image resizing and evaluation. J Comput Sci Technol 27(1):121–134

    Article  Google Scholar 

  14. Zhao T, Wu X (2019) Pyramid feature attention network for saliency detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3085–3094

  15. Qin X, Zhang Z, Huang C, Gao C, Dehghan M, Jagersand M (2019) Basnet: boundary-aware salient object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 7479–7489

  16. Wu Z, Su L, Huang Q (2019) Cascaded partial decoder for fast and accurate salient object detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 3907–3916

  17. Wu Z, Su L, Huang Q (2019) Stacked cross refinement network for edge-aware salient object detection. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 7264–7273

  18. Chen T, Xiao J, Hu X, Zhang G, Wang S (2022) Spatiotemporal context-aware network for video salient object detection. Neural Comput Appl 1–17

  19. Islam MA, Kalash M, Bruce ND (2018) Revisiting salient object detection: simultaneous detection, ranking, and subitizing of multiple salient objects. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7142–7150

  20. Siris A, Jiao J, Tam GK, Xie X, Lau RW (2020) Inferring attention shift ranks of objects for image saliency. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 12133–12143

  21. Song E, Lee M, Lee S (2018) Carvingnet: content-guided seam carving using deep convolution neural network. IEEE Access 7:284–292

    Article  Google Scholar 

  22. Liu S, Wei Z, Sun Y, Ou X, Lin J, Liu B, Yang M-H (2018) Composing semantic collage for image retargeting. IEEE Trans Image Process 27(10):5032–5043

    Article  ADS  MathSciNet  Google Scholar 

  23. Yan B, Niu X, Bare B, Tan W (2019) Semantic segmentation guided pixel fusion for image retargeting. IEEE Trans Multimed 22(3):676–687

    Article  Google Scholar 

  24. Ahmadi M, Karimi N, Samavi S (2021) Context-aware saliency detection for image retargeting using convolutional neural networks. Multimed Tools Appl 80(8):11917–11941

    Article  Google Scholar 

  25. Wu J, Xie R, Song L, Liu B (2019) Deep feature guided image retargeting. In: 2019 IEEE visual communications and image processing (VCIP). IEEE, pp 1–4

  26. Shocher A, Bagon S, Isola P, Irani M (2019) Ingan: Capturing and retargeting the" dna" of a natural image. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 4492–4501

  27. Mastan ID, Raman S (2020) Dcil: deep contextual internal learning for image restoration and image retargeting. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp 2366–2375

  28. Zhou Y, Chen Z, Li W (2020) Weakly supervised reinforced multi-operator image retargeting. IEEE Trans Circuits Syst Video Technol 31(1):126–139

    Article  Google Scholar 

  29. Kajiura N, Kosugi S, Wang X, Yamasaki T (2020) Self-play reinforcement learning for fast image retargeting. In: Proceedings of the 28th ACM international conference on multimedia, pp 1755–1763

  30. Tan W, Yan B, Lin C, Niu X (2019) Cycle-ir: deep cyclic image retargeting. IEEE Trans Multimed 22(7):1730–1743

    Article  Google Scholar 

  31. Fan X, Lei J, Liang J, Fang Y, Cao X, Ling N (2021) Unsupervised stereoscopic image retargeting via view synthesis and stereo cycle consistency losses. Neurocomputing 447:161–171

    Article  Google Scholar 

  32. Cho D, Park J, Oh T-H, Tai Y-W, So Kweon I (2017) Weakly-and self-supervised learning for content-aware deep image retargeting. In: Proceedings of the IEEE international conference on computer vision, pp 4558–4567

  33. Kompella A, Kulkarni RV (2020) Weakly supervised multi-scale recurrent convolutional neural network for co-saliency detection and co-segmentation. Neural Comput Appl 32(21):16571–16588

    Article  Google Scholar 

  34. Mu N, Xu X, Zhang X, Zhang H (2018) Salient object detection using a covariance-based cnn model in low-contrast images. Neural Comput Appl 29(8):181–192

    Article  Google Scholar 

  35. Sun Y, Xia C, Gao X, Yan H, Ge B, Li K-C (2022) Aggregating dense and attentional multi-scale feature network for salient object detection. Digit Signal Process 130:103747

    Article  Google Scholar 

  36. Chen T, Hu X, Xiao J, Zhang G, Wang S (2022) Cfidnet: cascaded feature interaction decoder for rgb-d salient object detection. Neural Comput Appl 34(10):7547–7563

    Article  Google Scholar 

  37. Zhou H, Xie X, Lai J-H, Chen Z, Yang L (2020) Interactive two-stream decoder for accurate and fast saliency detection. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 9141–9150

  38. Wang Z, Zhang Y, Liu Y, Wang Z, Coleman S, Kerr D (2022) Tf-sod: a novel transformer framework for salient object detection. Neural Comput Appl 1–18

  39. Liu N, Li L, Zhao W, Han J, Shao L (2021) Instance-level relative saliency ranking with graph reasoning. IEEE Trans Pattern Anal Mach Intell

  40. Lin J, Zhou T, Chen Z (2019) Deepir: A deep semantics driven framework for image retargeting. In: 2019 IEEE international conference on multimedia & expo workshops (ICMEW). IEEE, pp 54–59

  41. Danon D, Arar M, Cohen-Or D, Shamir A (2021) Image resizing by reconstruction from deep features. Comput Vis Media 1–14

  42. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  43. Fan D-P, Zhai Y, Borji A, Yang J, Shao L (2020) Bbs-net: Rgb-d salient object detection with a bifurcated backbone strategy network. In: European conference on computer vision. Springer, pp 275–292

  44. Cheng B, Chen L-C, Wei Y, Zhu Y, Huang Z, Xiong J, Huang TS, Hwu W-M, Shi H (2019) Spgnet: semantic prediction guidance for scene parsing. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 5218–5228

  45. Liu J-J, Hou Q, Cheng M-M (2020) Dynamic feature integration for simultaneous detection of salient object, edge, and skeleton. IEEE Trans Image Process 29:8652–8667

    Article  ADS  Google Scholar 

  46. Rubinstein M, Gutierrez D, Sorkine O, Shamir A (2010) A comparative study of image retargeting. ACM Trans Graph 29(6):1–10

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by National Natural Science Foundation of China (61806071 and 62102129) and Natural Science Foundation of Hebei Province (F2019202381 and F2019202464).

Author information

Authors and Affiliations

  1. School of Artificial Intelligence, Hebei University of Technology, No.5340, Xiping Road, Tianjin, 300401, China

    Yingchun Guo, Xiaoke Hao & Gang Yan

  2. Purification Equipment Research Institute of CSIC, No.17, Zhanlan Road, Handan, 056027, China

    Meng Zhang

Authors
  1. Yingchun Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Meng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaoke Hao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gang Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Yingchun Guo or Xiaoke Hao.

Ethics declarations

Conflict of interest

The authors declare that there is no any conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Guo, Y., Zhang, M., Hao, X. et al. IRNet-RS: image retargeting network via relative saliency. Neural Comput & Applic 36, 4133–4149 (2024). https://doi.org/10.1007/s00521-023-09258-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-09258-6

Keywords

Search

Navigation