Abstract
The numerous works on media retargeting call for a thorough and comprehensive survey for reviewing and categorizing existing works and providing insights that can help future design of retargeting approaches and its applications. First, we present the basic problem of media retargeting and detail state-of-the-art retargeting methods devised to solve it. Second, we review recent works on objective quality assessment of media retargeting, where we find that although these works are designed to make the objective assessment result in accordance with the subjective evaluation, they are only suitable for certain situations. Considering the subjective nature of aesthetics, designing objective assessment metric for media retargeting could be a promising area for investigation. Third, we elaborate on other applications extended from retargeting techniques. We show how to apply the retargeting techniques in other fields to solve their challenging problems, and reveal that retargeting technique is not just a simple scaling algorithm, but a thought or concept, which has great flexibility and is quite useful.We believe this review can help researchers and practitioners to solve the existing problems of media retargeting and bring new ideas in their works.
Similar content being viewed by others
References
Grundmann M, Kwatra V, Han M, Essa I. Discontinuous seam-carving for video retargeting. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2010, 569–576
Avidan S, Shamir A. Seam carving for content-aware image resizing. ACM Transactions on Graphics (TOG), 2007, 26(3): 10
Panozzo D, Weber O, Sorkine O. Robust image retargeting via axisaligned deformation. Computer Graphics Forum, 2012, 31(2pt1): 229–236
Wang Y S, Tai C L, Sorkine O, Lee T Y. Optimized scale-and-stretch for image resizing. ACM Transactions on Graphics (TOG), 2008, 27(5): 118
Yan B, Li K, Yang X C, Hu T X. Seam searching based pixel fusion for image retargeting. IEEE Transactions on Circuits and Systems for Video Technology, 2015, 25(1): 15–23
Fang Y M, Chen Z Z, Lin W S, Lin C W. Saliency-based image retargeting in the compressed domain. In: Proceedings of the 19th ACM international conference on Multimedia. 2011, 1049–1052
Mansfield A, Gehler P, Van Gool L, Rother C. Scene carving: scene consistent image retargeting. In: Daniilidis K, Maragos P, Paragios N, eds. Computer Vision–ECCV 2010. Springer Berlin Heidelberg, 2010, 143–156
Qi S Y, Ho J. Seam segment carving: retargeting images to irregularlyshaped image domains. In: Fitzgibbon A, Lazebnik S, Perona P, et al, eds. Computer Vision–ECCV 2012, Springer Berlin Heidelberg, 2012, 314–326
Shen J B, Wang D P, Li X L. Depth-aware image seam carving. IEEE Transactions on Cybernetics, 2013, 43(5): 1453–1461
Noh H, Han B. Seam carving with forward gradient difference maps. In: Proceedings of the 20th ACM international conference on Multimedia. 2012, 709–712
Battiato S, Farinella G M, Puglisi G, Ravi D. Saliency-based selection of gradient vector flow paths for content aware image resizing. IEEE Transactions on Image Processing, 2014, 23(5): 2081–2095
Dong W M, Zhou N, Lee T Y, Wu F Z, Kong Y, Zhang X P. Summarization-based image resizing by intelligent object carving. IEEE Transactions on Visualization and Computer Graphics, 2014, 20(1): 1
Santella A, Agrawala M, DeCarlo D, Salesin D, Cohen M. Gaze-based interaction for semi-automatic photo cropping. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2006, 771–780
Zhang L M, Wang M, Nie L Q, Hong L, Rui Y, Tian Q. Retargeting semantically-rich photos. IEEE Transactions on Multimedia (TMM), 2015, 17(9): 1538–1549
Chang C H, Chuang Y Y. A line-structure-preserving approach to image resizing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2012, 1075–1082
Lin S S, Yeh I C, Lin C H, Lee T Y. Patch-based image warping for content-aware retargeting. IEEE Transactions on Multimedia (TMM), 2013, 15(2): 359–368
Felzenszwalb P F, Huttenlocher D P. Efficient graph-based image segmentation. International Journal of Computer Vision, 2004, 59(2): 167–181
Wu Y C, Liu X T, Liu S X, Ma K L. ViSizer: a visualization resizing framework. IEEE Transactions on Visualization and Computer Graphics, 2013, 19(2): 278–290
Gallea R, Ardizzone E, Pirrone R. Physical metaphor for streaming media retargeting. IEEE Transactions on Multimedia, 2014, 16(4): 971–979
Yan B, Yang X C, Li K. Efficient image retargeting via adaptive pixel fusion. In: Proceedings of the 22nd ACM International Conference on Multimedia. 2014, 929–932
Rubinstein M, Shamir A, Avidan S. Multi-operator media retargeting. ACM Transactions on Graphics, 2009, 28(3): 23
Dong W M, Zhou N, Paul J C, Zhang X P. Optimized image resizing using seam carving and scaling. ACM Transactions on Graphics, 2009, 28(5): 125
Liu Z, Yan H B, Shen L Q, Ngan K N, Zhang Z Y. Adaptive image retargeting using saliency-based continuous seam carving. Optical Engineering, 2010, 49(1)
Zhang G X, Cheng M M, Hu S M, Martin R R. A shape-preserving approach to image resizing. Computer Graphics Forum, 2009, 28(7): 1897–1906
Liu Y, Sun L F, Yang S Q. A retargeting method for stereoscopic 3D video. Computational Visual Media, 2015, 1(2): 119–127
Dong WM, Wu F Z, Kong Y, Mei X, Lee T Y, Zhang X P. Image retargeting by texture-aware synthesis. IEEE Transactions on Visualization and Computer Graphics (TVCG), 2016, 22(2): 1088–1101
Dong W M, Bao G B, Zhang X P, Paul J C. Fast multi-operator image resizing and evaluation. Journal of Computer Science and Technology, 2012, 27(1): 121–134
Wu H, Wang Y S, Feng K C, Wong T T, Lee T Y, Heng P A. Resizing by symmetry-summarization. ACM Transactions on Graphics, 2010, 29(6): 159
Itti L, Koch C, Niebur E. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998 (11): 1254–1259
Hu S M, Chen T, Xu K, Cheng MM, Martin R R. Internet visual media processing: a survey with graphics and vision applications. The Visual Computer, 2013, 29(5): 393–405
Kraevoy V, Sheffer A, Shamir A, Cohenor D. Non-homogeneous resizing of complex models. ACM Transactions on Graphics, 2008, 27(5): 111
Wang K P, Zhang C M. Content-aware model resizing based on surface deformation. Computers & Graphics, 2009, 33(3): 433–438
Xiao C X, Jin L Q, Nie YW, Wang R F, Sun H Q, Ma K L. Contentaware model resizing with symmetry-preservation. The Visual Computer, 2015, 31(2): 155–167
Chen L, Meng X X. Anisotropic resizing of model with geometric textures. In: Proceedings of the 2009 SIAM/ACM Joint Conference on Geometric and Physical Modeling. 2009, 289–294
Lin J J, Cohen-Or D, Zhang H, Liang C, Sharf A, Deussen O, Chen B Q. Structure-preserving retargeting of irregular 3D architecture. ACM Transactions on Graphics, 2011, 30(6): 183
Shamir A, Sorkine O. Visual media retargeting. ACM SIGGRAPH ASIA 2009 Courses, 2009
Rubinstein M, Shamir A, Avidan S. Improved seam carving for video retargeting. ACM Transactions on Graphics, 2008, 27(3): 1–9
Chiang C K, Wang S F, Chen Y L, Lai S H. Fast JND-based video carving with GPU acceleration for real-time video retargeting. IEEE Transactions on Circuits and Systems for Video Technology, 2009, 19(11): 1588–1597
Chao W L, Su H H, Chien S Y, Hsu W, Ding J J. Coarse-to-fine temporal optimization for video retargeting based on seam carving. In: Proceedings of the 2011 IEEE International Conference on Multimedia and Expo. 2011, 1–6
Deselaers T, Dreuw P, Ney H. Pan, zoom, scan–time-coherent, trained automatic video cropping. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2008, 1–8
Fan X, Xie X, Zhou H Q, Ma WY. Looking into video frames on small displays. In: Proceedings of the 11th ACM international conference on Multimedia. 2003, 247–250
Liu F, Gleicher M. Video retargeting: automating pan and scan. In: Proceedings of the 14th Annual ACM International Conference on Multimedia. 2006, 241–250
Kopf S, Haenselmann T, Farin D, Effelsberg W. Automatic generation of summaries for the Web. In: Yeung M M, Lienhart R W, Li C S, eds. Storage and Retrieval for Image and Video Databases, 2004, 417–428
Wolf L, Guttmann M, Cohen-Or D. Non-homogeneous content-driven video-retargeting. In: Proceedings of the 11th IEEE International Conference on Computer Vision. 2007, 1–6
Zhang Y F, Hu S M, Martin R R. Shrinkability maps for content-aware video resizing. Computer Graphics Forum, 2008, 27(7): 1797–1804
Wang Y S, Fu H, Sorkine O, Lee T Y, Seidel H P. Motion-aware temporal coherence for video resizing. ACMTransactions on Graphics, 2009, 28(5): 127
Krähenbühl P, Lang M, Hornung A, Gross M. A system for retargeting of streaming video. ACM Transactions on Graphics, 2009, 28(5): 126
Kim J S, Kim J H, Kim C S. Adaptive image and video retargeting technique based on Fourier analysis. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. 2009, 1730–1737
Wang S F, Lai S H. Compressibility-aware media retargeting with structure preserving. IEEE Transactions on Image Processing, 2011, 20(3): 855–865
Shi L, Wang J Q, Duan L Y, Lu H Q. Consumer video retargeting: context assisted spatial-temporal grid optimization. In: Proceedings of the 17th ACM International Conference on Multimedia. 2009, 301–310
Wang Y S, Lin H C, Sorkine O, Lee T Y. Motion-based video retargeting with optimized crop-and-warp. ACM Transactions on Graphics, 2010, 29(4): 90
Wang Y S, Hsiao J H, Sorkine O, Lee T Y. Scalable and coherent video resizing with per-frame optimization. ACM Transactions on Graphics, 2011, 30(4): 88
Yen T C, Tsai C M, Lin CW. Maintaining temporal coherence in video retargeting using mosaic-guided scaling. IEEE Transactions on Image Processing, 2011, 20(8): 2339–2351
Khan S, Shah M. Object based segmentation of video using color, motion and spatial information. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2001
Paris S. Edge-preserving smoothing and mean-shift segmentation of video streams. In: Forsyth D, Torr P, Zisserman A, eds. Computer Vision–ECCV 2008. Springer Berlin Heidelberg, 2008, 460–473
Wang J, Thiesson B, Xu Y Q, Cohen M. Image and video segmentation by anisotropic kernel mean shift. In: Proceedings of the 10th European Conference on Computer Vision. 2004, 238–249
Hu Y Q, Rajan D. Hybrid shift map for video retargeting. In: Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition. 2010, 577–584
Yan B, Sun K R, Liu L. Matching area based seam carving for video retargeting. IEEE Transactions on Circuits and Systems for Video Technology. 2013, 23(2): 302–310
Lin S S, Lin C H, Yeh I C, Chang S H, Yeh C K, Lee T Y. Contentaware video retargeting using object-preserving warping. IEEE Transactions on Visualization and Computer Graphics, 2013, 19(10): 1677–1686
Qu Z, Wang J Q, Xu M, Lu H Q. Context-aware video retargeting via graph model. IEEE Transactions on Multimedia, 2013, 15(7): 1677–1687
Yuan Z, Lu T R, Huang Y, Wu D P, Yu H. Addressing visual consistency in video retargeting: a refined homogeneous approach. IEEE Transactions on Circuits and Systems for Video Technology, 2012, 22(6): 890–903
Li B, Duan L Y, Wang J, Ji R, Lin C W, Gao W. Spatiotemporal grid flow for video retargeting. IEEE Transactions on Image Processing, 2014, 23(4): 1615–1628
Nie Y W, Zhang Q, Wang R F, Xiao C X. Video retargeting combining warping and summarizing optimization. The Visual Computer, 2013, 29(6–8): 785–794
Wang Z, Bovik A C, Sheikh H R, Simoncelli E P. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600–612
Hsu C C, Lin CW, Fang Y, Lin W. Objective quality assessment for image retargeting based on perceptual geometric distortion and information loss. IEEE Journal of Selected Topics in Signal Processing, 2014, 8(3): 377–389
Bare B, Li K, Wang W Y, Yan B. Learning to assess image retargeting. In: Proceedings of the 22nd ACM International Conference on Multimedia. 2014, 925–928
Rubinstein M, Gutierrez D, Sorkine O, Shamir A. A comparative study of image retargeting. ACM Transactions on Graphics, 2010, 29(6): 160
Pele O, Werman M. Fast and robust earth mover’s distances. In: Proceedings of the 12th IEEE international conference on Computer vision. 2009, 460–467
Liu C, Yuen J, Torralba A, Sivic J, Freeman W T. Sift flow: dense correspondence across different scenes. In: Proceedings of the 10th European Conference on Computer Vision. 2008, 28–42
Liu Y J, Luo X, Xuan Y M, Chen W F, Fu X L. Image retargeting quality assessment. Computer Graphics Forum, 2011, 30(2): 583–592
Zhang J, Kuo C C J. An objective quality of experience (QoE) assessment index for retargeted images. In: Proceedings of the ACM International Conference on Multimedia. 2014, 257–266
Fang Y M, Zeng K, Wang Z, Lin W S, Fang Z J, Lin C W. Objective quality assessment for image retargeting based on structural similarity. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 2014, 4(1): 95–105
Barnes C, Shechtman E, Finkelstein A, Goldman D. Patchmatch: a randomized correspondence algorithm for structural image editing. ACM Transactions on Graphics, 2009, 28(3): 24
Manjunath B S, Ohm J R, Vasudevan V V, Yamada A. Color and texture descriptors. IEEE Transactions on Circuits and Systems for Video Technology, 2001, 11(6): 703–715
Simakov D, Caspi Y, Shechtman E, Irani M. Summarizing visual data using bidirectional similarity. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2008, 1–8
Kasutani E, Yamada A. The MPEG-7 color layout descriptor: a compact image feature description for high-speed image/video segment retrieval. In: Proceedings of the 2001 International Conference on Image Processing. 2001, 674–677
Yan B, Yuan B H, Yang B. Effective video retargeting with jittery assessment. IEEE Transactions on Multimedia, 2014, 16(1): 272–277
Tsai S S, Chen D, Takacs G, Chandrasekhar V, Singh J P, Girod B. Location coding for mobile image retrieval. In: Proceedings of the 5th International ICST Mobile Multimedia Communications Conference. 2009
Chandrasekhar V, Takacs G, Chen D, Tsai S, Grzeszczuk R, Girod B. Chog: compressed histogram of gradients a low bit-rate feature descriptor. In: Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. 2009, 2504–2511
Lowe D G. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 2004, 60(2): 91–110
Yang X Y, Liu L L, Qian X M, Mei T, Shen J L, Tian Q. Mobile visual search via hievarchical sparse coding. In: Proceedings of the 2014 IEEE International Conference on Multimedia and Expo. 2014, 1–6
Tan WM, Yan B, Li K, Tian Q. Image retargeting for preserving robust local feature: application to mobile visual search. IEEE Transactions on Multimedia, 2016, 18(1): 128–137
Ke Y, Sukthankar R. PCA-SIFT: a more distinctive representation for local image descriptors. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. 2004, 506–513
Seber G A F. Multivariate observations. John Wiley & Sons, 2009
Spath H. The cluster dissection and analysis theory FORTRAN programs examples. Prentice-Hall, Inc., 1985
Philbin J, Chum O, Isard M, Sivic J, Zisserman A. Object retrieval with large vocabularies and fast spatial matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2007, 1–8
Nie L Q, Wang M, Gao Y, Zha Z J, Chua T S. Beyond text QA: multimedia answer generation by harvesting Web information. IEEE Transactions on Multimedia, 2013, 15(2): 426–441
Nie L Q, Yan S C, Wang M, Hong R C, Chua T S. Harvesting visual concepts for image search with complex queries. In: Proceedings of the 20th ACM international conference on Multimedia. 2012, 59–68
Nie L Q, Wang M, Zha Z J, Chua T S. Oracle in image search: a content-based approach to performance prediction. ACM Transactions on Information Systems, 2012, 30(2): 13
Hong R C, Li G D, Nie L Q, Tang J H, Chua T S. Exploring large scale data for multimedia QA: an initial study. In: proceedings of the ACM International Conference on Image and Video Retrieval. 2010, 74–81
Lu S P, Dauphin G, Lafruit G, Munteanu A. Color retargeting: interactive time-varying color image composition from time-lapse sequences. Computational Visual Media, 2015, 1(4): 321–330
Guo Y W, Liu M, Gu T T, Wang W P. Improving photo composition elegantly: considering image similarity during composition optimization. Computer Graphics Forum, 2012, 31(7): 2193–2202
Zhang F L, Wang M, Hu S M. Aesthetic image enhancement by dependence-aware object recomposition. IEEE Transactions on Multimedia, 2013, 15(7): 1480–1490
Li K, Yan B, Li J, Majumder A. Seam carving based aesthetics enhancement for photos. Signal Processing: Image Communication, 2015, 39: 509–516
Bertalmio M, Sapiro G, Caselles V, Ballester C. Image in-painting. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. 2000, 417–424
Yeung M M, Yeo B L. Video visualization for compact presentation and fast browsing of pictorial content. IEEE Transactions on Circuits and Systems for Video Technology, 1997, 7(5): 771–785
Oh J, Wen Q, Lee J, Hwang S, Video abstraction. Hershey, PA: Idea Group Inc. and IRM Press, 2004
Liu T M, Zhang H J, Qi F H. A novel video key-frame-extraction algorithm based on perceived motion energy model. IEEE Transactions on Circuits and Systems for Video Technology, 2003, 13(10): 1006–1013
Hanjalic A, Zhang H J. An integrated scheme for automated video abstraction based on unsupervised cluster-validity analysis. IEEE Transactions on Circuits and Systems for Video Technology, 1999, 9(8): 1280–1289
You J Y, Liu G Z, Sun L, Li H L. A multiple visual models based perceptive analysis framework for multilevel video summarization. IEEE Transactions on Circuits and Systems for Video Technology, 2007, 17(3): 273–285
Qu W, Zhang Y F, Wang D L, Feng S, Yu G. Semantic movie summarization based on string of IE-RoleNets. Computational Visual Media, 2015, 1(2): 129–141
Pritch Y, Rav-Acha A, Peleg S. Nonchronological video synopsis and indexing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008, 30(11): 1971–1984
Lu S P, Zhang S H, Wei J, Hu S M, Martin R R. Timeline editing of objects in video. IEEE Transactions on Visualization and Computer Graphics, 2013, 19(7): 1218–1227
Nie Y W, Sun H Q, Li P, Xiao C X, Ma K L. Object movements synopsis via part assembling and stitching. IEEE Transactions on Visualization and Computer Graphics, 2014, 20(9): 1303–1315
Nie YW, Xiao C X, Sun H Q, Li P. Compact video synopsis via global spatiotemporal optimization. IEEE Transactions on Visualization and Computer Graphics, 2013, 19(10): 1664–1676
Li K, Yan B, Wang W, Gharavi H. An effective video synopsis approach with seam carving. IEEE Signal Processing Letters, 2016, 23(1): 11–14
Lee D S. Effective Gaussian mixture learning for video background subtraction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2005, 27(5): 827–832
Li Z, Ishwar P, Konrad J. Video condensation by ribbon carving. IEEE Transactions on Image Processing, 2009, 18(11): 2572–2583
Author information
Authors and Affiliations
Corresponding author
Additional information
Weimin Tan received his master’s degree at the College of Communication Engineering, Chongqing University, China. He is currently pursuing the doctoral degree with the School of Computer Science at Fudan University, China. His research interests include digital image and video processing.
Bo Yan received his PhD degree in computer science and engineering from the Chinese University of Hong Kong (CUHK), China in 2004. Before that, he received his degrees of BE and ME in communication engineering both from Xi’an Jiaotong University (XJTU), China in 1998 and 2001 respectively. From 2004 to 2006, he worked in the National Institute of Standards and Technology US (NIST) as a Postdoctoral Guest Researcher. Dr. Yan is currently a professor in School of Computer Science at Fudan University, China. He has served as the Associate Editor for IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), and the Guest Editor of Special Issue on “Content-aware Visual Systems: Analysis, Streaming and Retargeting” for IEEE Journal on Emerging and Selected topics in Circuits and Systems (JETCAS). He is the awardee of the NSFC Excellent Young Scholars Program in 2015. His research interests include video processing, computer vision and multimedia communications.
Electronic supplementary material
Rights and permissions
About this article
Cite this article
Tan, W., Yan, B. A survey on high coherence visual media retargeting: recent advances and applications. Front. Comput. Sci. 10, 778–796 (2016). https://doi.org/10.1007/s11704-016-6084-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11704-016-6084-3