Abstract
Content-aware image retargeting have received extensively research attentions. However, most of exiting retargeting approaches perform resizing on raw image data in the pixel domain. Since images in the actual world are mostly stored and transmitted in the compressed domain, e.g. discrete cosine transformation (DCT) domain, the complete decompression and recompression are almost inevitable by using the pixel domain based retargeting methods, causing extra overheads with high computation complexity. To address this issue, we propose a novel multi-operator image retargeting method in the DCT domain, in which three techniques including indirect seam carving, similarity transformation, and direct seam carving based on gradient vector flow (GVF), are utilized to perform resizing. To eliminate the zigzag effects in the retargeted images, we also present a novel similarity transformation algorithm in the DCT domain by which the DCT coefficients instead of a whole block are rescaled during resizing. In addition, we develop two decoding schemes to solve the issue that the traditional inverse DCT cannot be directly applied to the decoding the retargeted images. Extensive results demonstrate that the presented multi-operator image retargeting method in the DCT domain can preserve the aspect ratio of visual important contents well and obtain the resized images of better quality than the existing methods.
Similar content being viewed by others
References
Avidan S, Shamir A (2007) Seam carving for content-aware image resizing. ACM Trans Graph 26(3):1–10
Battiato S, Farinella G -M, Puglisi G, Ravì D (2014) Saliency-based selection of Gradient Vector Flow paths for content aware image resizing. IEEE Trans Image Process 23(5):2081–2095
Chang H, Shih T-K, Chang C-K (2019) CMAIR:Content and mask-aware image retargeting. Multimed Tools Appl 78:21731–21758
Chang H -S, Kang K (2005) A compressed domain scheme for classifying block edge patterns. IEEE Trans Image Process 8(3):145–151
Cheng M, Mitra N-J, Huang X, Torr P-H-S, Hu S (2015) Global contrast based salient region detection. IEEE Trans Pattern Anal Machine Intell 37(3):569–582
Cho D, Park J, Oh T, Tai Y, Kweon I-S (2017) Weakly-and self-supervised learning for content-aware deep image retargeting. In: Proceedings of IEEE international conference on computer vision, pp 4568–4577
Choi K-S, Ko S-J (2009) Fast Content-Aware image resizing scheme in the compressed domain. IEEE Trans Consum Electron 55(3):1514–1521
Chou Y-C, Su P-C (2016) Efficient content-based cropping using visual saliency and SLIC for image retargeting. In: Proceedings of IEEE international conference on consumer electronics, pp 1–2
Cui J, Cai Q, Lu H, Jia Z, Tang M (2020) Distortion-aware image retargeting based on continuous seam carving model. Signal Process 166 (Jan):107242.1-107242.10
Dong W, Zhou N, Paul J-C, Zhang X (2009) Optimized image resizing using seam carving and scaling. ACM Trans Graph 28(5):1–10
Fang Y, Chen Z, Lin W, Lin C (2012) Saliency detection in the compressed domain for adaptive image retargeting. IEEE Trans Image Process 21 (9):3888–3901
Guo Y, Liu F, Shi J, Zhou Z -H, Gleicher M (2009) Image retargeting using mesh parametrization. IEEE Trans Multimed 11(5):856–867
Hsu C-C, Lin C-W, Fang Y, Lin W (2014) Objective quality assessment for image retargeting based on perceptual geometric distortion and information loss. IEEE J Select Topics Signal Process 8(3):377–389
Itti L, Baldi P (2009) Bayesian surprise attracts human attention. Vis Res 49(10):1295–1306
Itti L, Koch C, Niebur E (1998) A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Pattern Anal Mach Intell 20(11):1254–1259
Kim J-S, Jeong S-G, Joo Y, Kim C-S (2011) Content-aware image and video resizing based on frequency domain analysis. IEEE Trans Consum Electron 57(2):615–622
Kim Y, Jung S, Jung C, Kim C (2018) A structure-aware axis-aligned grid deformation approach for robust image retargeting. Multimed Tools Appl 77(6):7717–7739
Krahenbuhl P, Lang M, Hornung A, Gross M (2009) A system for retargeting of streaming video. ACM Trans Graph 126
Li K, Yan B, Liu L, Sun K (2014) Efficient DCT-based image retargeting in compressed domain. Proc IEEE Int Conf Multimed Expo 3(1):1–6
Lin S-S, Yeh I-C, Lin C-H, Lee T-Y (2013) Patch-Based Image warping for Content-Aware retargeting. IEEE Trans Multimed 15(2):359–368
Liu T, Sun J, Zheng N, Tang X, Shum H-Y (2007) Learning to detect a salient object. IEEE Computer Vision and Pattern Recognition 1–8
Luo S, Zhang J, Zhang Q, Yuan X (2012) Multi-operator image retargeting with automatic integration of direct and indirect seam carving. Image Vis Comput 655–667
Mishra A, Scharfenberger C, Siva P, Li F, Wong A, Clausi D (2015) Desire:discontinuous energy seam carving for image retargeting via structural and textural energy functionals. In: Proceedings of IEEE international conference on image processing, pp 3695–3699
Mukherjee P, Lall B (2020) Conditional random field based salient proposal set generation and its application in content aware seam carving. Signal Process Image Commun 87(2020):115890
Murthy O-V-R, Muthuswamy K, Rajan D (2010) Image retargeting in compressed domain. In: Proceedings of IEEE international conference on pattern recognition, pp 4424–4427
Nguyen T-V, Gao G (2017) Novel evaluation metrics for seam carving based image retargeting. In: Proceedings of IEEE international conference on image processing, pp 450–454
Pritch Y, Kavvenaki E, Peleg S (2009) Shift-map image editing. Proc IEEE Int Conf Comput Vision 30(2):151–158
Qi S, Chi Y-J, Peter A-M, Ho J (2016) CASAIR:Content and Shape-aware image retargeting and its applications. IEEE Trans Image Process 25 (5):2222–2232
Rubinstein M, Gutierrez D, Sorkine O, Shamir A (2010) A comparative study of image retargeting. ACM Trans Graph 29(6):1–9
Rubinstein M, Shamir A, Avidan S (2008) Improved seam carving for video retargeting. ACM Trans Graph 27(3):23–31
Rubinstein M, Shamir A, Avidan S (2009) Multi-operator media retargeting. ACM Trans Graph 28(3):23–31
Shafieyan F, Karimi N, Mirmahboub B, Samavi S, Shirani S (2014) Image seam carving using depth assisted saliency map. In: Proceedings of IEEE international conference on image processing , pp 1155–1159
Shao F, Lin W, Lin W, Jiang Q, Jiang G (2017) QoE-guided warping for stereoscopic image retargeting. IEEE Trans Image Process 26 (10):4790–4805
Tan W, Yan B, Lin C, Niu X (2020) Cycle-IR:deep Cyclic Image Retargeting. IEEE Trans Multimed 22(7):1730–1743
Wang Y-S, Tai C-L, Sorking O, Lee T-Y (2008) Optimized scaleand-stretch for image resizing. ACM Trans Graph 27(5):1–8
Wolf L, Guttmann M, Cohenor D (2007) Non-homogeneous content-driven video-retargeting. In: Proceedings of IEEE international conference on computer vision, pp 1–6
Yan B, Niu X, Bare B, Tan W (2020) Semantic segmentation guided pixel fusion for image retargeting. IEEE Trans Multimed 22(3):676–687
Yan J, Lin S, Kang S-B, Tang X (2013) Learning the change for automatic image cropping. In: Proceedings of IEEE conference on computer vision and pattern recognition, pp 971–978
Zhang D, Yang G, Li F (2020) Detecting seam carved images using uniform local binary patterns. Multimed Tools Appl 79:8415–8430
Zhang J, Li S (2014) Compressed-domain video retargeting. IEEE Trans Image Process 23(2):797–809
Zhang Y, Sun Z, Jiang P, Huang Y, Peng J-L (2017) Hybrid image retargeting using optimized seam carving and scaling. Multimed Tools Appl 76(6):1–19
Zhang Y-B, Fang Y-M, Lin W-S, Zhang X-F, Li L-D (2016) Backward Registration-Based aspect ratio similarity for image retargeting quality assessment. IEEE Trans Image Process 25(9):4286–429
Zhou Y, Zhang L, Zhang C, Li P, Li X (2018) Perceptually aware image retargeting for mobile devices. IEEE Trans Image Process 27(5):2301–2313
Zhu L, Chen Z, Chen X, Liao N (2016) Saliency and structure preserving multi-operator image retargeting. In: Proceedings of IEEE international conference on acoustics, speech and signal processing, pp 1706–1710
Acknowledgements
This work was supported by the National Natural Science Foundation of China (Grant No. 61461006, 61563004). Zhenhua Tang is the corresponding author.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Tang, Z., Yao, J. & Zhang, Q. Multi-operator image retargeting in compressed domain by preserving aspect ratio of important contents. Multimed Tools Appl 81, 1501–1522 (2022). https://doi.org/10.1007/s11042-021-11376-z
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-021-11376-z