Skip to main content
Springer Nature Link
Log in

Leveraging Synthetic Data for DNN-Based Visual Analysis of Passenger Seats

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

Deep neural network (DNN)-based vision systems could improve passenger transportation safety by automating processes such as verifying the correct positioning of luggage, seat occupancy, etc. Abundant and well-distributed data are essential to make DNNs learn appropriate pattern recognition features and have enough generalization ability. The use of synthetic data can reduce the effort of generating varied and annotated data. However, synthetic data usually present a domain gap with real-world samples, that can be reduced with domain adaptation techniques. This paper proposes a methodology to build simulated environments to generate balanced and varied synthetic data and avoid including redundant samples to train classification DNNs for passenger seat analysis. We show a practical implementation for detecting whether luggage is correctly placed or not in an aircraft cabin. Experimental results show the contribution of the synthetic samples and the importance of correctly discarding redundant data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Hou Q, Cheng M, Hu X, Borji A, Tu Z, Torr PHS. Deeply supervised salient object detection with short connections. IEEE Trans Pattern Anal Mach Intell. 2019;41(4):815–28.

    Article  Google Scholar 

  2. Maninis KK, Caelles S, Chen Y, Pont-Tuset J, Leal-Taix´e L, Cremers D, Van Gool L. Video object segmentation without temporal information. IEEE TPAMI. 2019;41(6):1515–30.

    Article  Google Scholar 

  3. Xi Y, Zhang Y, Ding S, Wan S. Visual question answering model based on visual relationship detection. Signal Process Image Commun. 2020;80: 115648.

    Article  Google Scholar 

  4. Zhang P, Lan C, Xing J, Zeng W, Xue J, Zheng N. View adaptive neural networks for high performance skeleton-based human action recognition. IEEE TPAMI. 2019;41(8):1963–78.

    Article  Google Scholar 

  5. Mujika A, Fanlo AD, Tamayo I, Senderos O, Barandiaran J, Aranjuelo N, Nieto M, Otaegui O. “Web-based video-assisted point cloud annotation for ADAS validation.” In Proceedings International Conference on 3D Web Technology. 2019. pp. 1–9.

  6. Shorten C, Khoshgoftaar T. A survey on image data augmentation for deep learning. J Big Data. 2019;6:1–48.

    Article  Google Scholar 

  7. Singh R, Vatsa M, Patel VM, Ratha N, editors. Domain adaptation for visual understanding. Cham: Springer; 2020.

    Google Scholar 

  8. Nikolenko SI. “Synthetic data for deep learning.” In Springer Optimization and Its Applications. 2019;174.

  9. Seib V, Lange B, Wirtz S. Mixing real and synthetic data to enhance neural network training—a review of current approaches. 2020. arXiv preprint arXiv:2007.08781. Accessed 28 July 2022.

  10. Tremblay J, Prakash A, Acuna D, Brophy M, Jampani V, Anil C, To T, Cameracci E, Boochoon S, Birchfield S. “Training deep networks with synthetic data: Bridging the reality gap by domain randomization,” In Proceedings IEEE CVPR workshops. 2019. pp. 969–977.

  11. Aranjuelo N, Garc´ıa J, Unzueta L, Garc´ıa S, Elordi U, Otaegui O. “Building synthetic simulated environments for configuring and training multi-camera systems for surveillance applications.” In Proceedings VISIGRAPP (5: VISAPP). 2021. pp. 80–91.

  12. Hurl B, Czarnecki K, Waslander SL. “Precise synthetic image and lidar (presil) dataset for autonomous vehicle perception.” In IEEE IV. 2019. pp. 2522–2529.

  13. Saleh FS, Aliakbarian MS, Salzmann M, Petersson L, Alvarez JM. “Effective use of synthetic data for urban scene semantic segmentation.” In Proceedings ECCV, vol. 11206 of LNCS. 2018. pp. 86–103.

  14. Scheck T, Seidel R, Hirtz G. “Learning from theodore: A synthetic omnidirectional top-view indoor dataset for deep transfer learning.” In Proceeding IEEE WACV. 2020. pp. 932–941.

  15. Lai K-T, Lin C-C, Kang C-Y, Liao M-E, Chen M-S. “VIVID: virtual environment for visual deep learning.” In Proc. ACM MM. 2018. pp. 1356–1359.

  16. Shah S, Dey D, Lovett C, Kapoor A. “AirSim: High-fidelity visual and physical simulation for autonomous vehicles.” Field and Service Robotics. Cham: Springer; 2017. p. 621–35.

    Google Scholar 

  17. Khan S, Phan B, Salay R, Czarnecki K. “ProcSy: Procedural synthetic dataset generation towards influence factor studies of semantic segmentation networks.” In Proceedings CVPR Workshops. 2019. pp. 88–96.

  18. Rajpura PS, Bojinov H, Hegde RS. Object detection using deep CNNs trained on synthetic images. 2017. arXiv preprint arXiv:1706.06782. Accessed 28 July 2022.

  19. Loper M, Mahmood N, Romero J, Pons-Moll J, Black MJ. “SMPL: a skinned multi-person linear model.” ACM Transactions on Graphics (Proc. SIGGRAPH Asia). 2015. 34:248:1–248:16.

  20. Bourke P, Felinto D. “Blender and immersive gaming in a hemispherical dome.” In International Conference on Computer Games, Multimedia and Allied Technology. 2010;1: 280–284.

  21. Hernandez-Leal P, Kartal B, Taylor M. A survey and critique of multiagent deep reinforcement learning. Auton Agents MultiAgent Syst. 2019;33:750–97.

    Article  Google Scholar 

  22. Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L. “Imagenet: a large-scale hierarchical image database.” In Proceedings IEEE/CVF CVPR. 2009. pp. 248–255.

  23. Kuznetsova A, Rom H, Alldrin N, Uijlings J, Krasin I, Pont-Tuset J, Kamali S, Popov S, Malloci M, Kolesnikov A, et al. The open images dataset v4. Int J Comput Vision. 2020;128(7):1956–81.

    Article  Google Scholar 

  24. Kolesnikov A, Beyer L, Zhai X, Puigcerver J, Yung J, Gelly S, Houlsby N. “Big transfer (bit): General visual representation learning.” In Proceedings ECCV. Springer. 2020 pp. 491–507.

  25. Raghu M, Zhang C, Kleinberg JM, Bengio S. “Transfusion: understanding transfer learning for medical imaging.” In advances in neural information processing systems 32: Annual Conference on Neural Information Processing Systems, NeurIPS Wallach HM, Larochelle H, Beygelzimer A, d’Alch´e-Buc F, Fox EB, Garnett R (eds). 2019. pp. 3342–3352.

  26. Niu S, Liu M, Liu Y, Wang J, Song H. Distant domain transfer learning for medical imaging. IEEE J Biomed Health Inform. 2021;25(10):3784–93.

    Article  Google Scholar 

  27. Reed CJ, Yue X, Nrusimha A, Ebrahimi S, Vijaykumar V, Mao R, Li B, Zhang S, Guillory D, Metzger S, Keutzer K, Darrell T. “Self-supervised pretraining improves self-supervised pretraining.” In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV.2022. pp. 1050–1060.

  28. Sun Q, Liu Y, Chua T-S, Schiele B. “Meta-transfer learning for few-shot learning.” In Proceedings IEEE/CVF CVPR. 2019. pp. 403–412.

  29. Shafaei A, Little JJ, Schmidt N. “Play and learn: using video games to train computer vision models,” In Proceedings BMVC. 2016.

  30. Ros G, Sellart L, Materzynska J, Vazquez D, Lopez AM. “The synthia dataset: a large collection of synthetic images for semantic segmentation of urban scenes.” In Proceedings IEEE CVPR. 2016. pp. 3234–3243.

  31. Aranjuelo N, Garc´ıa S, Loyo E, Unzueta L, Otaegui O. Key strategies for synthetic data generation for training intelligent systems based on people detection from omnidirectional cameras. Comput Electr Eng. 2021;92:107105.

    Article  Google Scholar 

  32. Cort´es A, Rodr´ıguez C, V´elez G, Barandiar´an J, Nieto M. “Analysis of classifier training on synthetic data for cross-domain datasets.” IEEE Trans. on Intelligent Transportation Systems. 2020.

  33. Zhu J-Y, Park T, Isola P, Efros AA. “Unpaired image-to-image translation using cycle-consistent adversarial networks.” In Proceedings IEEE international conference on computer vision. 2017. pp. 2223–2232.

  34. Tonutti M, Ruffaldi E, Cattaneo A, Avizzano CA. Robust and subject-independent driving manoeuvre anticipation through domainadversarial recurrent neural networks. Robot Auton Syst. 2019;115:162–73.

    Article  Google Scholar 

  35. Vicomtech. VCD—video content description. 2020. https://vcd.vicomtech.org/. Accessed 28 July 2022.

  36. ASAM. OpenLABEL. 2020. https://www.asam.net/project-detail/scenario-storage-and-labelling/. Accessed 28 July 2022.

  37. Tan M, Le QV. “Efficientnet: rethinking model scaling for convolutional neural networks.” In Proceedings ICML Chaudhuri K, Salakhutdinov R (eds.) 97 of PMLR. 2019. pp. 6105–6114.

  38. Ganin Y, Ustinova E, Ajakan H, Germain P, Larochelle H, Laviolette F, Marchand M, Lempitsky V. Domain-adversarial training of neural networks. J Mach Learn Res. 2016;17(1):2096–2030.

    MathSciNet  MATH  Google Scholar 

  39. Dauphin YN, de Vries H, Bengio Y. “Equilibrated adaptive learning rates for non-convex optimization,” In Advances in Neural Information Processings Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7–12, 2015, Montreal, Quebec, Canada. 2015. pp. 1504–1512.

Download references

Acknowledgements

This work has received funding from the Clean Sky 2 Joint Undertaking under the European Union’s Horizon 2020 research and innovation program under grant agreement No. 865162, SmaCS (https://www.smacs.eu/).

Author information

Authors and Affiliations

  1. Vicomtech, Basque Research and Technology Alliance (BRTA), San Sebastian, Spain

    Nerea Aranjuelo, Jose Luis Apellaniz, Luis Unzueta, Jorge Garcia, Sara Garcia, Unai Elordi & Oihana Otaegui

  2. Basque Country University (UPV/EHU), San Sebastian, Spain

    Nerea Aranjuelo & Unai Elordi

Authors
  1. Nerea Aranjuelo
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Jose Luis Apellaniz
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Luis Unzueta
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Jorge Garcia
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Sara Garcia
    View author publications

    You can also search for this author inPubMed Google Scholar

  6. Unai Elordi
    View author publications

    You can also search for this author inPubMed Google Scholar

  7. Oihana Otaegui
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Nerea Aranjuelo.

Ethics declarations

Conflict of interest

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the topical collection “Computer Vision, Imaging and Computer Graphics Theory and Applications” guest edited by Jose Braz, A. Augusto Sousa, Alexis Paljic, Christophe Hurter and Giovanni Maria Farinella.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aranjuelo, N., Apellaniz, J.L., Unzueta, L. et al. Leveraging Synthetic Data for DNN-Based Visual Analysis of Passenger Seats. SN COMPUT. SCI. 4, 40 (2023). https://doi.org/10.1007/s42979-022-01453-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-022-01453-x

Keywords