Skip to main content

Towards Hybrid Camera Sensor Simulation for Autonomous Vehicles

  • Conference paper
  • First Online:
Advances on P2P, Parallel, Grid, Cloud and Internet Computing (3PGCIC 2020)

Abstract

To accurately test and validate algorithms used in autonomous vehicles, numerous test vehicles and very large data sets are required, resulting in safety constraints and increased financial cost. For this reason, it is desired to train the algorithms at least partly in simulation. In this work we lay the focus on the camera sensor and propose a novel methodology for injecting the instance of simulated vehicles into the camera data of real vehicles. To get qualitative results and improve generalization capabilities, the simulated data must sufficiently correspond to real-world sensor data in order to prevent loss of performance when moving the model to a real environment after training. The realism of the output is evaluated by object detection systems and a realism score produced by a CNN. Results show the potential of this approach for improving hybrid simulators for the validation of autonomous vehicles.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. de Hoog, J., Pepermans, M., Mercelis, S., Hellinckx, P.: Towards a scalable distributed real-time hybrid simulator for autonomous vehicles. In: Advances on P2P, Parallel, Grid, Cloud and Internet Computing, vol. 24, pp. 447–456. Springer International Publishing, Cham (2019)

    Google Scholar 

  2. Tsai, Y., Shen, X., Lin, Z., Sunkavalli, K.,  Lu, X., Yang, M.: Deep image harmonization. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2799–2807. IEEE Computer Society, Los Alamitos, July 2017

    Google Scholar 

  3. Wu, H., Zheng, S., Zhang, J., Huang, K.: GP-GAN: towards Realistic High-Resolution Image Blending. In: Proceedings of the 27th ACM International Conference on Multimedia, ser. MM 2019, pp. 2487–2495. Association for Computing Machinery, New York (2019)

    Google Scholar 

  4. Gechter, F., Dafflon, B., Gruer, P., Koukam, A.: Towards a hybrid real/virtual simulation of autonomous vehicles for critical scenarios. In: The Sixth International Conference on Advances in System Simulation (SIMUL 2014), pp. 14–17 (2014)

    Google Scholar 

  5. Reinhard, E., Adhikhmin, M., Gooch, B., Shirley, P.: Color transfer between images. IEEE Comput. Graphics Appl. 21(4), 34–41 (2001)

    Article  Google Scholar 

  6. Sunkavalli, K., Johnson, M.K., Matusik, W., Pfister, H.: Multi-scale image harmonization. ACM Trans. Graph. 29(4), 1–10 (2010)

    Article  Google Scholar 

  7. Tsai, Y.-H., Shen, X., Lin, Z., Sunkavalli, K., Yang, M.-H.: Sky is not the limit. ACM Trans. Graph. 35(4), 1–11 (2016)

    Article  Google Scholar 

  8. Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. ACM Trans. Graph. 22(3), 313–318 (2003)

    Article  Google Scholar 

  9. Xue, S., Agarwala, A., Dorsey, J., Rushmeier, H.: Understanding and improving the realism of image composites. ACM Trans. Graph. 31(4), 1–10 (2012)

    Article  Google Scholar 

  10. Zhu, J.-Y., Krahenbuhl, P., Shechtman, E., Efros, A.A.: Learning a discriminative model for the perception of realism in composite images. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 3943–3951. IEEE, December 2015

    Google Scholar 

  11. Zhang, L., Wen, T., Shi, J.: Deep image blending. In: 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 231–240. IEEE Computer Society, Los Alamitos, March 2020

    Google Scholar 

  12. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of StyleGAN. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110–8119 (2020)

    Google Scholar 

  13. Blender Foundation: Cycles Open Source Production Rendering (2018). https://www.cycles-renderer.org/. Accessed 30 June 2020

  14. Burt, P., Adelson, E.: The Laplacian pyramid as a compact image code. IEEE Trans. Commun. 31(4), 532–540 (1983)

    Article  Google Scholar 

  15. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3354–3361. IEEE, June 2012

    Google Scholar 

  16. Davis, J., Goadrich, M.: The relationship between precision-recall and roc curves. In: Proceedings of the 23rd International Conference on Machine Learning, ser. ICML 2006, pp. 233–240. Association for Computing Machinery, New York (2006)

    Google Scholar 

  17. Redmon, J., Farhadi, A.: Yolov3: an incremental improvement. arXiv preprint arXiv:1804.02767 (2018)

  18. Dai, J., Li, Y., He, K., Sun, J.: R-FCN: object detection via region-based fully convolutional networks. In: Proceedings of the 30th International Conference on Neural Information Processing Systems, ser. NIPS 2016, pp. 379–387. Curran Associates Inc., Red Hook (2016)

    Google Scholar 

  19. Everingham, M., Gool, L.V., Williams, C.K.I., Winn, J., Zisserman, A.: The pascal visual object classes (VOC) challenge. Int. J. Comput. Vision 88(2), 303–338 (2009)

    Article  Google Scholar 

  20. Blender Foundation: Eevee rendering introduction (2018). https://docs.blender.org/manual/en/latest/render/eevee/introduction.html. Accessed 30 June 2020

Download references

Acknowledgements

This research received funding from the Flemish Government under the “Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen” programme.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Dieter Balemans .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Balemans, D., De Boeck, Y., de Hoog, J., Anwar, A., Mercelis, S., Hellinckx, P. (2021). Towards Hybrid Camera Sensor Simulation for Autonomous Vehicles. In: Barolli, L., Takizawa, M., Yoshihisa, T., Amato, F., Ikeda, M. (eds) Advances on P2P, Parallel, Grid, Cloud and Internet Computing. 3PGCIC 2020. Lecture Notes in Networks and Systems, vol 158. Springer, Cham. https://doi.org/10.1007/978-3-030-61105-7_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-61105-7_29

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-61104-0

  • Online ISBN: 978-3-030-61105-7

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics