Skip to main content
Springer Nature Link
Log in

Feature semantic space-based sim2real decision model

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

At present, the intelligent decision model of unmanned systems can only be applied to virtual scenes, which makes it difficult to migrate to real scenes because the image gap between virtual scenes and real scenes is relatively large. The main solutions are domain randomization, domain adaptation, and image translation. However, these methods simply add noise and transform the perceptual information and do not consider the semantic information of the agent’s perceptual space. This causes the problem of low accuracy in the migration of virtual scene decision models to real scenes. Considering the above problems, we propose a feature semantic space-based sim2real decision model, which includes an environment representation module, policy optimization module and intelligent decision module. The model framework can narrow the image gap between real-world scenes and virtual scenes. First, using the environment representation module, the virtual scene and real scene are simultaneously mapped to the feature semantic space through semantic segmentation. Then, in the policy optimization module, we propose an AMDDPG policy optimization algorithm. The algorithm obtains the local and global experience in the learning process through the global and local network architecture. It also solves the problem of the slow learning rate of sim2real. Finally, in the intelligent decision module, the data in the semantic space integrating virtual scene and real scene features are used as the training data of the agent autonomous decision model. Experimental results confirm that our method has more effective generalization and robustness of the model in the real scene and can be better migrated to the real scene.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Kiran BR, Sobh I, Talpaert V, et al. (2021) Deep reinforcement learning for autonomous driving: A survey. IEEE Trans Intell Transp Syst 16(1):1–18

    Google Scholar 

  2. Sallab AE, Abdou M, Perot E, Yogamani S (2016) End-to-end deep reinforcement learning for lane keeping assist. arXivpreprint arXiv:1612.04340

  3. Sallab AE, Abdou M, Perot E, Yogamani S (2017) Deep reinforcement learning framework for autonomous driving. Electron Imaging 36(2):70–76

    Article  Google Scholar 

  4. Zong X, Xu G, Yu G, et al. (2018) Obstacle avoidance for self-driving vehicle with reinforcement learning. SAE Int J Passenger Cars Electron Electr Syst 11(1):28–37

    Google Scholar 

  5. Spryn M, Sharma A, Parkar D, et al. (2018) Distributed deep reinforcement learning on the cloud for autonomous driving. 2018 IEEE ACM 1st International Workshop on Software Engineering for AI in Autonomous Systems (SEFAIAS).IEEE Computer Society

  6. Wang S, Jia D, Weng X (2016) Deep reinforcement learning for autonomous driving. Computer Vision and Pattern Recognition

  7. Xu N, Tan B, Kong B (2017) Autonomous driving in reality with reinforcement learning and image translation. arXiv:1801.05299

  8. Xiao C, Lu P, He Q (2021) Flying through a narrow gap using end-to-end deep reinforcement learning augmented with curriculum learning and sim2real. IEEE Transactions on Neural Networks and Learning Systems

  9. Zhang T, Zhang K, Lin J, Louie W-YG, Huang H (2021) Sim2real learning of obstacle avoidance for robotic manipulators in uncertain environments. IEEE Robot Autom Lett 7(1):65–72

    Article  Google Scholar 

  10. Kaspar M, Osorio JDM (2020) Sim2real transfer for reinforcement learning without dynamics randomization. In: 2020 IEEE/RSJ international conference on intelligent robots and systems (IROS), IEEE, pp 4383–4388

  11. Zhao W, Queralta JP, Qingqing L, Westerlund T (2020) Towards closing the sim-to-real gap in collaborative multi-robot deep reinforcement learning. In: 2020 5th international conference on robotics and automation engineering (ICRAE), IEEE, pp 7–12

  12. Liu J, Shen H, Wang D, Kang Y, Tian Q (2021) Unsupervised domain adaptation with dynamics-aware rewards in reinforcement learning. arXiv preprint arXiv:2110.12997

  13. Jaunet T, Bono G, Vuillemot R, Wolf C (2021) Sim2realviz: Visualizing the sim2real gap in robot ego-pose estimation. arXiv preprint arXiv:2109.11801

  14. Gao H, Yang Z, Su X, Tan T, Chen F (2020) Adaptability preserving domain decomposition for stabilizing sim2real reinforcement learning. In: 2020 IEEE/RSJ international conference on intelligent robots and systems (IROS), IEEE, pp 4403– 4410

  15. Blum T, Paillet G, Laine M, Yoshida K (2020) Rl star platform: Reinforcement learning for simulation based training of robots. arXiv preprint arXiv:2009.09595

  16. Li G, Yang Y, Li S, Qu X, Lyu N, Li SE (2021) Decision making of autonomous vehicles in lane change scenarios: Deep reinforcement learning approaches with risk awareness. Transportation Research Part C: Emerging Technologies, 103452

  17. Chen J, Li SE, Tomizuka M (2021) Interpretable end-to-end urban autonomous driving with latent deep reinforcement learning. IEEE Transactions on Intelligent Transportation Systems

  18. Zhang Q, Pan W, Reppa V (2021) Model-reference reinforcement learning for collision-free tracking control of autonomous surface vehicles. IEEE Transactions on Intelligent Transportation Systems

  19. Wang G, Niu H, Zhu D, Hu J, Zhan X, Zhou G (2021) Model: A modularized end-to-end reinforcement learning framework for autonomous driving. arXiv preprint arXiv:2110.11573

  20. Mnih V, Kavukcuoglu K, Silver D, et al. (2013) Playing atari with deep reinforcement learning. Comput Sci 42(3):1556–1568

    Google Scholar 

  21. Mnih V, Badia AP, Mirza M, et al. (2017) Asynchronous methods for deep reinforcement learning

  22. Lillicrap TP, Hunt JJ, Pritzel A, et al. (2013) Continuous control with deep reinforcement learning. computer science-learning statistics. Comput Sci Learn Stat Mach Learn 16(1):32–42

    Google Scholar 

  23. Schulman J, Levine S, Abbeel P, Jordan M, Moritz P (2015) Trust region policy optimization. In: International conference on machine learning, PMLR, pp 1889–1897

  24. Haarnoja T, Zhou A, Abbeel P, Levine S (2018) Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: International conference on machine learning, PMLR, pp 1861–1870

  25. Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

  26. Vilas SC, KS, Vilas ST (2021) Autonomous racing using a hybrid imitation-reinforcement learning architecture. arXiveprints, 2110

  27. Niu H, Hu J, Cui Z, Zhang Y (2021) Dr2l: Surfacing corner cases to robustify autonomous driving via domain randomization reinforcement learning. In: The 5th international conference on computer science and application engineering, pp 1–8

  28. Pan X, You Y, Wang Z, et al. (2017) Virtual to real reinforcement learning for autonomous driving. https://arxiv.org/pdf/1704.03952.pdf

  29. Liu Q, Zhai JW, Zhang ZZ, et al. (2018) A survey on deep reinforcement learning. Chin J Comput 41(1):1–27

    Google Scholar 

  30. Brostow GJ, Fauqueur J, Cipolla R (2009) Semantic object classes in video: A high-definition ground truth database. Pattern Recogn Lett 30(2):88–97

    Article  Google Scholar 

  31. Geiger A, Lenz P, Stiller C, Urtasun R (2013) Vision meets robotics: The kitti dataset. The International Journal of Robotics Research 32(11):1231–1237

    Article  Google Scholar 

  32. Cordts M, Omran M, Ramos S, Rehfeld T, Enzweiler M, Benenson R, Franke U, Roth S, Schiele B (2016) The cityscapes dataset for semantic urban scene understanding. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3213–3223

  33. Müller M, Dosovitskiy A, Ghanem B, Koltun V (2018) Driving policy transfer via modularity and abstraction. arXiv preprint arXiv:1804.09364

  34. Dosovitskiy A, Ros G, Codevilla F, Lopez A, Koltun V (2017) Carla: An open urban driving simulator. In: Conference on robot learning, PMLR, pp 1–16

  35. Xu H, Gao Y, Yu F, Darrell T (2017) End-to-end learning of driving models from large-scale video datasets. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2174–2182

  36. Zhao H, Shi J, Qi X, Wang X, Jia J (2017) Pyramid scene parsing network. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2881–2890

  37. Bewley A, Rigley J, Liu Y, Hawke J, Shen R, Lam V-D, Kendall A (2019) Learning to drive from simulation without real world labels. In: 2019 international conference on robotics and automation (ICRA), IEEE, pp 4818–4824

  38. Kingma DP, Welling M (2013) Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114

  39. Liu M. -Y., Breuel T, Kautz J (2017) Unsupervised image-to-image translation networks. In: Advances in neural information processing systems, pp 700–708

  40. Yang L, Liang X, Wang T, Xing E (2018) Real-to-virtual domain unification for end-to-end autonomous driving. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 530–545

  41. Loiacono D, Cardamone L, Lanzi PL (2013) Simulated car racing championship: Competition software manual Computer Science

  42. Lillicrap TP, Hunt JJ, Pritzel A, Heess N, Erez T, Tassa Y, Silver D, Wierstra D (2015) Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971

Download references

Funding

The work of this paper is a part of the project of National Natural Science Foundation of China. National Natural Science Foundation of China : 61991415.

Author information

Author notes

    Authors and Affiliations

    1. School of computer engineering and Science, Shanghai University, Shanghai, 200444, China

      Wenwen Xiao, Xiangfeng Luo & Shaorong Xie

    Authors
    1. Wenwen Xiao
      View author publications

      You can also search for this author inPubMed Google Scholar

    2. Xiangfeng Luo
      View author publications

      You can also search for this author inPubMed Google Scholar

    3. Shaorong Xie
      View author publications

      You can also search for this author inPubMed Google Scholar

    Corresponding author

    Correspondence to Wenwen Xiao.

    Additional information

    Publisher’s note

    Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

    Xiangfeng Luo and Shaorong Xie contributed equally to this work.

    Rights and permissions

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Xiao, W., Luo, X. & Xie, S. Feature semantic space-based sim2real decision model. Appl Intell 53, 4890–4906 (2023). https://doi.org/10.1007/s10489-022-03566-5

    Download citation

    • Accepted:

    • Published:

    • Issue Date:

    • DOI: https://doi.org/10.1007/s10489-022-03566-5

    Keywords