Skip to main content

Retargeting Visual Data with Deformation Fields

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Abstract

Seam carving is an image editing method that enables content-aware resizing, including operations like removing objects. However, the seam-finding strategy based on dynamic programming or graph-cut limits its applications to broader visual data formats and degrees of freedom for editing. Our observation is that describing the editing and retargeting of images more generally by a deformation field yields a generalisation of content-aware deformations. We propose to learn a deformation with a neural network that keeps the output plausible while trying to deform it only in places with low information content. This technique applies to different kinds of visual data, including images, 3D scenes given as neural radiance fields, or even polygon meshes. Experiments conducted on different visual data show that our method achieves better content-aware retargeting compared to previous methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Ardizzone, L., et al.: Analyzing inverse problems with invertible neural networks. CoRR abs/1808.04730 (2018). http://arxiv.org/abs/1808.04730

  2. Avidan, S., Shamir, A.: Seam carving for content-aware image resizing. ACM Trans. Graph. 26(3), 10 (2007)

    Article  Google Scholar 

  3. Basha, T.D., Moses, Y., Avidan, S.: Stereo seam carving a geometrically consistent approach. IEEE Trans. Pattern Anal. Mach. Intell. 35(10), 2513–2525 (2013)

    Article  Google Scholar 

  4. Berkiten, S., Halber, M., Solomon, J., Ma, C., Li, H., Rusinkiewicz, S.: Learning detail transfer based on geometric features. Comput. Graph. Forum 36(2), 361–373 (2017)

    Article  Google Scholar 

  5. Cai, H., Feng, W., Feng, X., Wang, Y., Zhang, J.: Neural surface reconstruction of dynamic scenes with monocular RGB-D camera. In: Neural Information Processing Systems (NeurIPS) (2022)

    Google Scholar 

  6. Chen, Z., Zhang, H.: Learning implicit fields for generative shape modeling. In: CVPR, pp. 5939–5948 (2019)

    Google Scholar 

  7. Dong, W., Bao, G., Zhang, X., Paul, J.: Fast multi-operator image resizing and evaluation. J. Comput. Sci. Technol. 27(1), 121–134 (2012)

    Article  Google Scholar 

  8. Dong, W., Zhou, N., Lee, T., Wu, F., Kong, Y., Zhang, X.: Summarization-based image resizing by intelligent object carving. IEEE Trans. Vis. Comput. Graph. 20(1), 1 (2014). https://doi.org/10.1109/TVCG.2013.103

    Article  Google Scholar 

  9. Dong, W., Zhou, N., Paul, J., Zhang, X.: Optimized image resizing using seam carving and scaling. ACM Trans. Graph. 28(5), 125 (2009)

    Article  Google Scholar 

  10. Efros, A.A., Freeman, W.T.: Image quilting for texture synthesis and transfer. In: Pocock, L. (ed.) Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH, pp. 341–346. ACM (2001)

    Google Scholar 

  11. Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: ICCV, pp. 1033–1038 (1999)

    Google Scholar 

  12. Newton2 at English Wikipedia: Broadway tower (2007). https://commons.wikimedia.org/wiki/File:Broadway_tower.jpg

  13. Garbin, S.Jet al.: VolTeMorph: realtime, controllable and generalisable animation of volumetric representations. arXiv:2208.00949 (2022)

  14. Gatys, L.A., Ecker, A.S., Bethge, M.: Texture synthesis using convolutional neural networks. In: Neural Information Processing Systems, pp. 262–270 (2015)

    Google Scholar 

  15. Gu, J., Zhai, S., Zhang, Y., Susskind, J., Jaitly, N.: Matryoshka diffusion models. arXiv preprint arXiv:2310.15111 (2023)

  16. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015). http://arxiv.org/abs/1512.03385

  17. Henzler, P., Deschaintre, V., Mitra, N.J., Ritschel, T.: Generative modelling of BRDF textures from flash images. ACM Trans. Graph. 40(6), 284:1–284:13 (2021)

    Google Scholar 

  18. Henzler, P., Mitra, N.J., , Ritschel, T.: Learning a neural 3D texture space from 2D exemplars. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Google Scholar 

  19. Hertz, A., Hanocka, R., Giryes, R., Cohen-Or, D.: Deep geometric texture synthesis. ACM Trans. Graph. 39(4), 108:1–108:11 (2020)

    Google Scholar 

  20. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local nash equilibrium. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  21. Huang, Y., Cao, Y., Lai, Y., Shan, Y., Gao, L.: NeRF-texture: texture synthesis with neural radiance fields. In: ACM SIGGRAPH 2023 Conference Proceedings, pp. 43:1–43:10 (2023)

    Google Scholar 

  22. Kajiura, N., Kosugi, S., Wang, X., Yamasaki, T.: Self-play reinforcement learning for fast image retargeting. CoRR abs/2010.00909 (2020). https://arxiv.org/abs/2010.00909

  23. Kawar, B., et al.: Imagic: text-based real image editing with diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6007–6017 (2023)

    Google Scholar 

  24. Kopf, J., Fu, C.W., Cohen-Or, D., Deussen, O., Lischinski, D., Wong, T.T.: Solid texture synthesis from 2d exemplars. ACM Trans. Graph. 26(3), 2:1–2:9 (2007)

    Google Scholar 

  25. Lai, Y., Hu, S., Gu, X., Martin, R.R.: Geometric texture synthesis and transfer via geometry images. In: Proceedings of the Tenth ACM Symposium on Solid and Physical Modeling, pp. 15–26. ACM (2005)

    Google Scholar 

  26. Liu, F., Gleicher, M.: Automatic image retargeting with fisheye-view warping. In: Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology, pp. 153–162. ACM (2005)

    Google Scholar 

  27. Liu, F., Gleicher, M.: Video retargeting: automating pan and scan. In: Nahrstedt, K., Turk, M.A., Rui, Y., Klas, W., Mayer-Patel, K. (eds.) Proceedings of the 14th ACM International Conference on Multimedia, pp. 241–250. ACM (2006)

    Google Scholar 

  28. Luiten, J., Kopanas, G., Leibe, B., Ramanan, D.: Dynamic 3D gaussians: tracking by persistent dynamic view synthesis. In: 3DV (2024)

    Google Scholar 

  29. Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., Geiger, A.: Occupancy networks: learning 3D reconstruction in function space. In: CVPR, pp. 4460–4470 (2019)

    Google Scholar 

  30. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: NeRF: representing scenes as neural radiance fields for view synthesis. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 405–421. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_24

    Chapter  Google Scholar 

  31. Moreira, T.P., Santana, M.C.S., Passos, L.A., Papa, J.P., da Costa, K.A.P.: An end-to-end approach for seam carving detection using deep neural networks. In: Pinho, A.J., Georgieva, P., Teixeira, L.F., Sánchez, J.A. (eds.) IbPRIA 2022. LNCS, vol. 13256, pp. 447–457. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-04881-4_35

    Chapter  Google Scholar 

  32. Nam, S., Ahn, W., Yu, I., Kwon, M., Son, M., Lee, H.: Deep convolutional neural network for identifying seam-carving forgery. IEEE Trans. Circuits Syst. Video Technol. 31(8), 3308–3326 (2021)

    Article  Google Scholar 

  33. Nataraj, L., Gudavalli, C., Mohammed, T.M., Chandrasekaran, S., Manjunath, B.S.: Seam carving detection and localization using two-stage deep neural networks. CoRR abs/2109.01764 (2021)

    Google Scholar 

  34. North, R.: Grand theft auto v. Steam (2015). https://www.rockstargames.com/V/

  35. Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: DeepSDF: learning continuous signed distance functions for shape representation. In: CVPR, pp. 165–174 (2019)

    Google Scholar 

  36. Park, K., Sinha, U., Barron, J.T., Bouaziz, S., Goldman, D.B., Seitz, S.M., Martin-Brualla, R.: NeRFies: deformable neural radiance fields. In: ICCV, pp. 5845–5854 (2021)

    Google Scholar 

  37. Park, K., et al.: HyperNeRF: a higher-dimensional representation for topologically varying neural radiance fields. ACM Trans. Graph. 40(6), 238:1–238:12 (2021)

    Google Scholar 

  38. Peebles, W., Xie, S.: Scalable diffusion models with transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195–4205 (2023)

    Google Scholar 

  39. Peng, Y., et al.: CageNeRF: cage-based neural radiance fields for generalized 3D deformation and animation. In: Advances in Neural Information Processing Systems (2022)

    Google Scholar 

  40. Pumarola, A., Corona, E., Pons-Moll, G., Moreno-Noguer, F.: D-NeRF: neural radiance fields for dynamic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2021)

    Google Scholar 

  41. Richter, S.R., AlHaija, H.A., Koltun, V.: Enhancing photorealism enhancement. IEEE Trans. Pattern Anal. Mach. Intell. 45(2), 1700–1715 (2022)

    Article  Google Scholar 

  42. Rubinstein, M., Gutierrez, D., Sorkine, O., Shamir, A.: A comparative study of image retargeting. ACM Trans. Graph. (Proc. SIGGRAPH ASIA) 29(6), 160:1–160:10 (2010)

    Google Scholar 

  43. Rubinstein, M., Shamir, A., Avidan, S.: Improved seam carving for video retargeting. ACM Trans. Graph. 27(3), 16 (2008)

    Article  Google Scholar 

  44. Rubinstein, M., Shamir, A., Avidan, S.: Multi-operator media retargeting. ACM Trans. Graph. 28(3), 23 (2009). https://doi.org/10.1145/1531326.1531329

    Article  Google Scholar 

  45. Shaham, T.R., Dekel, T., Michaeli, T.: SinGAN: learning a generative model from a single natural image. In: ICCV, pp. 4569–4579 (2019)

    Google Scholar 

  46. Song, E., Lee, M., Lee, S.: CarvingNet: content-guided seam carving using deep convolution neural network. IEEE Access 7, 284–292 (2019). https://doi.org/10.1109/ACCESS.2018.2885347

    Article  Google Scholar 

  47. Sorkine-Hornung, O., Alexa, M.: As-rigid-as-possible surface modeling. In: Symposium on Geometry Processing (2007)

    Google Scholar 

  48. Srinivas, S., Fleuret, F.: Full-gradient representation for neural network visualization. In: Advances in neural information processing systems, vol. 32 (2019)

    Google Scholar 

  49. Tan, W., Yan, B., Lin, C., Niu, X.: Cycle-IR: deep cyclic image retargeting. IEEE Trans. Multimedia 22(7), 1730–1743 (2019)

    Article  Google Scholar 

  50. Wang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., Wang, W.: NeuS: learning neural implicit surfaces by volume rendering for multi-view reconstruction. In: Advances in Neural Information Processing Systems, pp. 27171–27183 (2021)

    Google Scholar 

  51. Wei, L., Levoy, M.: Fast texture synthesis using tree-structured vector quantization. In: Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH, pp. 479–488. ACM (2000)

    Google Scholar 

  52. Wu, G., et al.: 4D Gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528 (2023)

  53. Wu, H., Wang, Y., Feng, K., Wong, T., Lee, T., Heng, P.: Resizing by symmetry-summarization. ACM Trans. Graph. 29(6), 159 (2010)

    Article  Google Scholar 

  54. Wu, R., Liu, R., Vondrick, C., Zheng, C.: Sin3dm: Learning a diffusion model from a single 3d textured shape. CoRR abs/2305.15399 (2023)

    Google Scholar 

  55. Wu, R., Zheng, C.: Learning to generate 3D shapes from a single example. ACM Trans. Graphics (TOG) 41(6) (2022)

    Google Scholar 

  56. Xian, W., et al.: TextureGAN: controlling deep image synthesis with texture patches. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)

    Google Scholar 

  57. Xu, T., Harada, T.: Deforming radiance fields with cages. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13693, pp. 159–175. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19827-4_10

    Chapter  Google Scholar 

  58. Yang, Z., Gao, X., Zhou, W., Jiao, S., Zhang, Y., Jin, X.: Deformable 3D gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309.13101 (2023)

  59. Yuan, Y.J., Sun, Y.T., Lai, Y.K., Ma, Y., Jia, R., Gao, L.: NeRF-editing: geometry editing of neural radiance fields. In: CVPR, pp. 18332–18343 (2022)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Tim Elsner .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (zip 61019 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Elsner, T., Berger, J., Wu, T., Czech, V., Gao, L., Kobbelt, L. (2025). Retargeting Visual Data with Deformation Fields. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15112. Springer, Cham. https://doi.org/10.1007/978-3-031-72949-2_16

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72949-2_16

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72948-5

  • Online ISBN: 978-3-031-72949-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics