Skip to main content
SpringerLink
Log in

SimFormer: Real-to-Sim Transfer with Recurrent Restoration

  • Conference paper
  • First Online:
Smart Multimedia (ICSM 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13497))

Included in the following conference series:

  • 505 Accesses

  • The original version of this chapter was revised: For all authors, a second, incorrect affiliation had been stated. This has been removed. The correction to this chapter is available at https://doi.org/10.1007/978-3-031-22061-6_35

Abstract

Real-to-Sim transfer is a popular research topic in robotics. Utilizing a simulated environment, the development processes can achieve lower costs and make the testing process easier. In addition, after the Real-to-Sim transfer, the simulated environment can lower the texture effect and light effect, which can be further applied to other computer vision tasks, such as robot grasping. Differing from artistic style transfer, Real-to-Sim transfer has higher accuracy requirements for content preservation. In this paper, we utilize the transformer to solve the Real-to-Sim transfer. We creatively design the restoration stage to preserve the content information. We also propose the restoration loss function. After these improvements, our architecture can achieve better performance on light removal, content preservation, and feature embedding.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Change history

  • 03 February 2023

    In the originally published version of chapter 33, erroneously, a second, incorrect affiliation had been added for all of the authors. This has been corrected.

References

  1. Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2414–2423 (2016)

    Google Scholar 

  2. Li, Y., et al.: Universal style transfer via feature transforms. In: Advances in Neural Information Processing Systems, 30 (2017)

    Google Scholar 

  3. Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1501–1510 (2017)

    Google Scholar 

  4. Csurka, G.: Domain adaptation for visual applications: a comprehensive survey. arXiv preprint arXiv:1702.05374 (2017)

  5. Bousmalis, K., et al.: Using simulation and domain adaptation to improve efficiency of deep robotic grasping. In: 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 4243–4250. IEEE, May 2018

    Google Scholar 

  6. Rao, K., Harris, C., Irpan, A., Levine, S., Ibarz, J., Khansari, M.: Rl-CycleGAN: reinforcement learning aware simulation-to-real. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11157–11166 (2020)

    Google Scholar 

  7. Prakash, A., Debnath, S., Lafleche, J.F., Cameracci, E., Birchfield, S., Law, M.T.: Self-supervised real-to-sim scene generation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16044–16054 (2021)

    Google Scholar 

  8. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, 30 (2017)

    Google Scholar 

  9. Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12346, pp. 213–229. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58452-8_13

    Chapter  Google Scholar 

  10. Dai, Z., Cai, B., Lin, Y., Chen, J.: UP-DETR: unsupervised pre-training for object detection with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1601–1610 (2021)

    Google Scholar 

  11. Zheng, S., et al.: Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6881–6890 (2021)

    Google Scholar 

  12. Wang, Y., et al.: End-to-end video instance segmentation with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8741–8750 (2021)

    Google Scholar 

  13. Deng, Y., Tang, F., Pan, X., Dong, W., Ma, C., Xu, C.: StyTr\(^ 2\): unbiased image style transfer with transformers. arXiv preprint arXiv:2105.14576 (2021)

  14. Chen, H., et al.: Artistic style transfer with internal-external learning and contrastive learning. In: Advances in Neural Information Processing Systems, 34 (2021)

    Google Scholar 

  15. Qian, K., Zhou, J., Xiong, F., Zhou, H., Du, J.: Object tracking in hyperspectral videos with convolutional features and kernelized correlation filter. In: Basu, A., Berretti, S. (eds.) ICSM 2018. LNCS, vol. 11010, pp. 308–319. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-04375-9_26

    Chapter  Google Scholar 

  16. Mukherjee, S., Valenzise, G., Cheng, I.: Potential of deep features for opinion-unaware, distortion-unaware, no-reference image quality assessment. In: McDaniel, T., Berretti, S., Curcio, I.D.D., Basu, A. (eds.) ICSM 2019. LNCS, vol. 12015, pp. 87–95. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-54407-2_8

    Chapter  Google Scholar 

  17. Liu, C., Cheng, I., Basu, A.: Synthetic vision assisted real-time runway detection for infrared aerial images. In: Basu, A., Berretti, S. (eds.) ICSM 2018. LNCS, vol. 11010, pp. 274–281. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-04375-9_23

    Chapter  Google Scholar 

  18. Avila, M., Ponce, P., Molina, A., Romo, K.: Simulation framework for load management and behavioral energy efficiency analysis in smart homes. In: McDaniel, T., Berretti, S., Curcio, I.D.D., Basu, A. (eds.) ICSM 2019. LNCS, vol. 12015, pp. 497–508. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-54407-2_42

    Chapter  Google Scholar 

  19. Lugo, G., Hajari, N., Reddy, A., Cheng, I.: Textureless object recognition using an RGB-D sensor. In: McDaniel, T., Berretti, S., Curcio, I.D.D., Basu, A. (eds.) ICSM 2019. LNCS, vol. 12015, pp. 13–27. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-54407-2_2

    Chapter  Google Scholar 

  20. Park, D.Y., Lee, K.H.: Arbitrary style transfer with style-attentional networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5880–5888 (2019)

    Google Scholar 

  21. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)

  22. Chen, H., et al.: Artistic style transfer with internal-external learning and contrastive learning. In: Advances in Neural Information Processing Systems, 34 (2021)

    Google Scholar 

  23. Kang, M., Park, J.: ContraGAN: contrastive learning for conditional image generation. In: Advances in Neural Information Processing Systems, 33, 21357–21369 (2020)

    Google Scholar 

  24. Arjovsky, M., Chintala, S., Bottou, L.: Wasserstein generative adversarial networks. In: International conference on machine learning, pp. 214–223. PMLR, July 2017

    Google Scholar 

  25. Lin, T.-Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  26. Li, X., Liu, S., Kautz, J., Yang, M.H.: Learning linear transformations for fast image and video style transfer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3809–3817 (2019)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computing Science, University of Alberta, Edmonton, Canada

    Yingnan Ma, Fan Yang, Xudong Li, Chen Jiang & Anup Basu

Authors
  1. Yingnan Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xudong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chen Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Anup Basu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yingnan Ma .

Editor information

Editors and Affiliations

  1. University of Florence, Florence, Italy

    Stefano Berretti

  2. Dolby Labs, California, CA, USA

    Guan-Ming Su

Rights and permissions

Reprints and permissions

Copyright information

Âİ 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ma, Y., Yang, F., Li, X., Jiang, C., Basu, A. (2022). SimFormer: Real-to-Sim Transfer with Recurrent Restoration. In: Berretti, S., Su, GM. (eds) Smart Multimedia. ICSM 2022. Lecture Notes in Computer Science, vol 13497. Springer, Cham. https://doi.org/10.1007/978-3-031-22061-6_33

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-22061-6_33

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-22060-9

  • Online ISBN: 978-3-031-22061-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation