Skip to main content
SpringerLink
Log in

Application in multimedia: from camera to VR

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This work describes a framework that allows children and domestic users to create architectural structures like mazes, houses etc., and navigate them in virtual reality (VR). The user can draw a 2D map of a maze etc. using a simple paper, pen and ruler. The application works by taking as input such a hand drawn image via gallery or camera, and then building a 3D model in VR using Unity3D. The 3D model completely follows the design of the floor plan including the placement of doors and windows. Detecting and then constructing walls, doors and windows on the fly in VR is a challenge. The user can also customize walls, windows, doors and floors. This is done by looking at the object and using the VR controller to select various options. The purpose of the framework is purely entertainment and is specifically designed for children and domestic users. From hand-drawn images on a paper to ready-to-navigate 3-D model in VR in just one click is the novelty of our work. User is completely isolated from the complexities and intricacies of programming and usage of different tools and platforms. We also extend our work from hand drawn designs to professional home blueprints and floor plans. We used fully convolutional network to extract walls from the professional floor plans with mean pixel accuracy of 97.3%. For hand-drawn designs, we were able to detect doors and windows with MAE of 0. Our wall segmentation method reported an MAE of only 1.2.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Algorithm 1
Algorithm 2
Algorithm 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Alias Systems Corporation (2020) Maya, [Online]. Available: https://www.autodesk.com/products/maya. Accessed 15 Dec 2020

  2. Ahmed S, Liwicki M, Weber M, Dengel A (2011) Improved automatic analysis of architectural floor plans. In: 2011 International conference on document analysis and recognition. IEEE, pp 864–869

  3. Autodesk, Inc. (2020) 3ds max, [Online]. Available: https://www.autodesk.com/products/3ds-max. Accessed 15 Dec 2020

  4. Autodeskcom (2022) Inventor software. https://www.autodesk.com/products/inventor/overview. Accessed 19 Feb 2022

  5. Bukowski R (1995) The walkthru editor: towards realistic and e ective interaction with virtual building environments. University of California, Berkeley

  6. Chung TM (2018) Strategies for vr prototyping, [Online]. Available: https://medium.com/inborn-experience/strategies-for-vr-prototyping-810e0d3aa21d. Accessed 15 Dec 2020

  7. Dalal S, Vishwakarma VP, Kumar S (2020) Feature-based sketch-photo matching for face recognition. Procedia Comput Sci 167:562–570

    Article  Google Scholar 

  8. de las Heras L-P, Terrades OR, Robles S, Sánchez G (2015) Cvc-fp and sgt: a new database for structural floor plan analysis and its groundtruthing tool. Int J Doc Anal Recogn (IJDAR) 18(1):15–30

    Article  Google Scholar 

  9. Dodge S, Xu J, Stenger B (2017) . Parsing floor plan images 05:358–361

    Google Scholar 

  10. Dosch P, Tombre K, Ah-Soon C, Masini G (2000) A complete system for the analysis of architectural drawings. IJDAR 3:102–116

    Article  Google Scholar 

  11. Exxar (2022) Experience cad to reality. https://exxar.cloud/cad-to-vr/. Accessed 19 Feb 2022

  12. Fritz M (2018) Ar/vr prototyping: take designs from paper to glasses, [Online]. Available: https://medium.com/inborn-experience/ar-vr-prototyping-cd765bad650f. Accessed 15 Dec 2020

  13. Gai W, Yang C, Bian Y, Dong M, Liu J, Dong Y, Niu C, Lin C, Meng X, Shen C (2016) Ubimaze: a new genre of virtual reality game based on mobile devices. In: Proceedings of the 18th international conference on human-computer interaction with mobile devices and services adjunct, pp 591–593

  14. Gai W, Yang C, Bian Y, Shen C, Meng X, Wang L, Liu J, Dong M, Niu C, Lin C (2017) Supporting easy physical-to-virtual creation of mobile vr maze games: a new genre. In: Proceedings of the 2017 CHI conference on human factors in computing systems, pp 5016–5028

  15. Gimenez L, Robert S, Suard F, Zreik K (2016) Automatic reconstruction of 3d building models from scanned 2d floor plans. Autom Constr 63:48–56

    Article  Google Scholar 

  16. Google Cloud Vision API (2019) Vision ai derive image insights via ml cloud vision api. https://cloud.google.com/vision. Accessed 16 May 2020

  17. Halim Z, Baig AR, Mujtaba H (2010) Measuring entertainment and automatic generation of entertaining games. Int J Inf Technol Commun Converg 1 (1):92–107

    Google Scholar 

  18. Horna S, Damiand G, Meneveaux D, Bertrand Y (2007) Building 3d indoor scenes topology from 2d architectural plans. In: GRAPP, pp 37–44

  19. Kamińska D, Sapiński T, Wiak S, Tikk T, Haamer RE, Avots E, Helmi A, Ozcinar C, Anbarjafari G (2019) Virtual reality and its applications in education: survey. Information 10(10):318

    Article  Google Scholar 

  20. Kamppari-Miller S (2018) Vr paper prototyping, [Online]. Available: https://blog.prototypr.io/vr-paper-prototyping-9e1cab6a75f3. Accessed 15 Dec 2020

  21. Kamppari-Miller S (2018) Vr sketch sheets & paper prototyping, [Online]. Available: https://medium.com/designer-geeking/vr-sketch-sheets-paper-prototyping-859d3b7300e4. Accessed 15 Dec 2020

  22. Kurbatov V (2017) Draw sketches for virtual reality like a pro, [Online]. Available: https://medium.com/inborn-experience/vr-sketches-56599f99b357. Accessed 15 Dec 2020

  23. Kurbatov V (2019) Templates for ar/vr sketches, [Online]. Available: https://medium.com/inborn-experience/templates-for-ar-vr-sketches-e424dfb60e54. Accessed 15 Dec 2020

  24. Lewis R, Séquin C (1998) Generation of 3d building models from 2d architectural plans. Comput Aided Des 30(10):765–779

    Article  MATH  Google Scholar 

  25. Liu C, Wu J, Kohli P, Furukawa Y (2017) Raster-to-vector: revisiting floorplan transformation, pp 2214–2222

  26. Long J, Shelhamer E, Darrell T (2015) Fully convolutional models for semantic segmentation. In: CVPR, vol 3, p 4

  27. Maxon Computer (2018) Sketchup, [Online]. Available: https://www.maxon.net/en/cinema-4d

  28. McIntosh J, Rodgers M, Marques B, Gibbard A (2019) The use of vr for creating therapeutic environments for the health and wellbeing of military personnel, their families and their communities. J Dig Landsc Archit 1(4):185–195. Accessed 15 Dec 2020

    Google Scholar 

  29. Nebeling M, Madier K (2019) 60proto: making interactive virtual reality & augmented reality prototypes from paper. In: Proceedings of the 2019 CHI conference on human factors in computing systems, pp 1–13

  30. Noor M, Nazir M, Rehman S, Tariq J (2021) Sketch-recognition using pre-trained model

  31. OpenCV (2017) Open contour. https://docs.opencv.org/3.4/df/d0d/tutorialfindcontours.html

  32. OpenCV (2017) Convex hull. https://docs.opencv.org/3.4/d7/d1d/tutorialhull.html. Accessed 15 Dec 2020

  33. OpenCV (2017) Approx polydp. https://docs.opencv.org/3.4/jscontourfeaturesapproxPolyDP.html. Accessed 15 Dec 2020

  34. OpenCV (2017) Bounding rect. https://docs.opencv.org/3.4/da/d0c/tutorialboundingrect.html. Accessed 15 Dec 2020

  35. Planner5D (2016) Planner 5d - home and interior design creator. https://planner5d.com/. Accessed 15 Dec 2020

  36. Racz A, Zilizi G (2018) Vr aided architecture and interior design. In: 2018 International conference on advances in computing and communication engineering (ICACCE). IEEE, pp 11–16

  37. Rieffe C, Oosterveld P, Terwogt MM (2006) An alexithymia questionnaire for children: factorial and concurrent validation results. Personal Individ Differ 40(1):123–133

    Article  Google Scholar 

  38. Ryan RM, Mims V, Koestner R (1983) Relation of reward contingency and interpersonal context to intrinsic motivation: a review and test using cognitive evaluation theory. J Person Soc Psychol 45(4):736

    Article  Google Scholar 

  39. Software SDI (2022) Nx virtual reality | siemens software. https://www.plm.automation.siemens.com/global/en/products/mechanical-design/nx-virtual-reality.html. Accessed 19 Feb 2022

  40. Swaileh W, Kotzinos D, Ghosh S, Jordan M, Vu NS, Qian Y (2021) Versailles-fp dataset: wall detection in ancient floor plans. In: International conference on document analysis and recognition. Springer, pp 34–49

  41. Sweetser P, Rogalewicz Z, Li Q (2019) Understanding enjoyment in vr games with gameflow. In: 25th ACM symposium on virtual reality software and technology, pp 1–2

  42. Trimble, Inc. (2020) Sketchup, [Online]. Available: https://www.sketchup.com/. Accessed 15 Dec 2020

  43. Trimble, Inc. (2000) Sketchup, [Online]. Available: https://www.autodesk.com/products/revit. Accessed 15 Dec 2020

  44. Ton Roosendaal (2017) Blender, [Online]. Available: https://www.blender.org/. Accessed 15 Dec 2020

  45. Vidil F, Damiand C (2003) Moka, [Online]. Available: www.sic.sp2mi.univpoitiers.fr/moka/. Accessed 15 Dec 2020

  46. Wang W, Dong S, Zou K, Li W (2020) Room classification in floor plan recognition. In: 2020 4th International conference on advances in image processing, pp 48–54

  47. Wang W, Dong S, Zou K, Li WS (2020) Room classification in floor plan recognition; room classification in floor plan recognition 2020. https://doi.org/10.1145/3441250.3441265

  48. Xu Z, Yang C, Alheejawi S, Jha N, Mehadi S, Mandal M (2021) Floor plan semantic segmentation using deep learning with boundary attention aggregated mechanism, pp 346–353

  49. Yan R, Masood A, Li P, Ali SG, Sheng B, Ren J (2018) 3d simulation of interior house design in vr using vr3id method. In: 2018 IEEE International conference on progress in informatics and computing (PIC). IEEE, pp 236–240

  50. Yamasaki T, Zhang J, Takada Y (2018) Apartment structure estimation using fully convolutional networks and graph model, pp 1–6

  51. Zhu J, Zhang H, Wen Y (2014) A new reconstruction method for 3d buildings from 2d vector floor plan. Comput-aided Des Applic 11(6):704–714

    Article  Google Scholar 

  52. Zhang X, Huang Y, Zou Q, Pei Y, Zhang R, Wang S (2020) A hybrid convolutional neural network for sketch recognition. Pattern Recogn Lett 130:73–82

    Article  Google Scholar 

Download references

Acknowledgements

We would like to thank our undergraduate students Hamza Khan, Hafsah Shehzad, Khadija Asim, Arham Fatima for searching and installing 3D modeling software discussed in Table 1 and assisting in their analysis.

Funding

No funding was received for conducting this study.

Author information

Authors and Affiliations

  1. Department of Computer Science, National University of Computer and Emerging Sciences, Lahore, Pakistan

    Aamir Wali, Aliza Lisan, Hammad Ather, Muhammad Qasim & Muhammad Uzair Abid

Authors
  1. Aamir Wali
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aliza Lisan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hammad Ather
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Muhammad Qasim
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Muhammad Uzair Abid
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aamir Wali.

Ethics declarations

Ethics approval and consent to participate

This statement is to certify that the author list is correct. The Authors also confirm that this research has not been published previously and that it is not under consideration for publication elsewhere. On behalf of all Co-Authors, the Corresponding Author shall bear full responsibility for the submission.

All procedures performed in studies involving human participants were in accordance with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. Informed consent was obtained from the parents of all individual participants involved in the study. All subjects gave written informed consent.

Competing interests

The authors declare that they have no competing financial interests or personal relationships that could influence the work reported in this paper.

Conflict of Interests

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix: Snapshot of customization options for walls, doors and windows

Appendix: Snapshot of customization options for walls, doors and windows

Fig. 18
figure 18

Customization menu for walls and floor

Full size image
Fig. 19
figure 19

Customization menu for windows

Full size image
Fig. 20
figure 20

Customization menu for doors

Full size image

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wali, A., Lisan, A., Ather, H. et al. Application in multimedia: from camera to VR. Multimed Tools Appl 82, 11721–11751 (2023). https://doi.org/10.1007/s11042-022-13687-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-13687-1

Keywords

Search

Navigation