Skip to main content

Advertisement

Springer Nature Link
Log in

Augmented reality instructions for construction toys enabled by accurate model registration and realistic object/hand occlusions

  • Original Article
  • Published:
Virtual Reality Aims and scope Submit manuscript

Abstract

BRICKxAR is a novel augmented reality (AR) instruction method for construction toys such as LEGO®. With BRICKxAR, physical LEGO construction is guided by virtual bricks. Compared with the state of the art, accuracy of the virtual–physical model alignment is significantly improved through a new design of marker-based registration, which can achieve an average error less than 1 mm throughout the model. Realistic object occlusion is accomplished to reveal the true spatial relationship between physical and virtual bricks. LEGO players’ hand detection and occlusion are realized to visualize the correct spatial relationship between real hands and virtual bricks, and allow virtual bricks to be “grasped” by real hands. The major finding of the research is that the integration of these features makes AR instructions possible for small parts assembly, validated through a working AR prototype for constructing LEGO Arc de Triomphe and quantitative measures of the accuracies of registration and occlusions. In addition, a heuristic evaluation of BRICKxAR’s features has led to findings that the present method could advance AR instructions in terms of enhancing part visibility, match between mental models and visualization, alignment of physical and virtual parts in perspective views and spatial transformations, tangible user interface, consolidated structural diagrams, virtual cutaway views, among other benefits for guiding construction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

Video is available at https://www.youtube.com/watch?v=7JDW_lDv7FU.

References

  • Abate AF, Narducci F, and Ricciardi S (2014) An image based approach to hand occlusions in mixed reality environments. In: International conference on virtual, augmented and mixed reality, 319–28. Springer

  • Abe U, Hotta K, Hotta A, Takami Y, Ikeda H, and Ikeda Y (2017) Digital construction - demonstration of interactive assembly using smart discrete papers with RFID and AR codes. In: Janssen P, Loh P, Raonic A, Schnabel MA (Eds.), Protocols, Flows, and Glitches - Proceedings of the 22nd CAADRIA Conference, 75–84. Xi’an Jiaotong-Liverpool University, Suzhou, China

  • Agrawala M, Phan D, Heiser J, Haymaker J, Klingner J, Hanrahan P, Tversky B (2003) Designing effective step-by-step assembly instructions. ACM Trans Graph (TOG) 22(3):828–837

    Article  Google Scholar 

  • AlNajdi SM, Alrashidi MQ, Almohamadi KS (2020) The effectiveness of using augmented reality (AR) on assembling and exploring educational mobile robot in pedagogical virtual machine (PVM). Interact Learn Environ 28(8):964–990

    Article  Google Scholar 

  • Alonso L, Zhang YR, Grignard A, Noyman A, Sakai Y, ElKatsha M, Doorley, R and Larson K (2018) Cityscope: a data-driven interactive simulation tool for urban design. Use Case volpe. In: International Conference on Complex Systems, 253–61. Springer

  • Ashtari N, Bunt A, McGrenere J, Nebeling M, and Chilana PK (2020) Creating augmented and virtual reality applications: current practices, challenges, and opportunities. In: Proceedings of the 2020 CHI conference on human factors in computing systems, 1–13

  • Bailey T, Durrant-Whyte H (2006) Simultaneous localization and mapping (SLAM): part II. IEEE Robot Autom Mag 13(3):108–117

    Article  Google Scholar 

  • Bottani E, Vignali G (2019) Augmented reality technology in the manufacturing industry: a review of the last decade. IISE Trans 51(3):284–310

    Article  Google Scholar 

  • Büttner S, Prilla M, and Röcker C (2020) Augmented reality training for industrial assembly work-are projection-based AR assistive systems an appropriate tool for assembly training?. In: Proceedings of the 2020 CHI conference on human factors in computing systems, 1–12

  • Cao Y, Qian X, Wang T, Lee R, Huo K, and Ramani K (2020) An exploratory study of augmented reality presence for tutoring machine tasks. In: Proceedings of the 2020 CHI conference on human factors in computing systems, 1–13

  • Chen ZR, Liao CJ, Chu CH (2018) An assembly guidance system of tou kung based on augmented reality. Proc Caadria 2018:349–358

    Article  Google Scholar 

  • Christensen AL, Biskjaer MM (2018) Designing digital interactive instructions for children’s construction play. In: Proceedings of the 36th European conference on cognitive ergonomics, 5. ACM

  • Du C, Chen Y-L, Ye M, and Ren L (2016) Edge snapping-based depth enhancement for dynamic occlusion handling in augmented reality. In: 2016 IEEE international symposium on mixed and augmented reality (ISMAR), 54–62. https://doi.org/10.1109/ISMAR.2016.17

  • Durrant-Whyte H, Bailey T (2006) Simultaneous localization and mapping: part I. IEEE Robot Autom Mag 13(2):99–110

    Article  Google Scholar 

  • Eisenberg M (1998) Middle Tech: blurring the division between high and low tech in education. In: The design of children’s technology, 244–73

  • Endsley TC, Sprehn KA, Brill RM, Ryan KJ, Vincent EC, and Martin JM (2017) Augmented reality design heuristics: designing for dynamic interactions. In: Proceedings of the human factors and ergonomics society annual meeting, 61:2100–2104. SAGE Publications Sage CA: Los Angeles, CA

  • Funk M, Kosch T, Greenwald SW, and Schmidt A (2015) A benchmark for interactive augmented reality instructions for assembly tasks. In: Proceedings of the 14th international conference on mobile and ubiquitous multimedia, 253–57

  • Hahm S, Maciel A, Sumitiomo E, and Rodriguez AL (2019) FlowMorph - exploring the human-material interaction in digitally augmented craftsmanship. In: Haeusler M, Schnabel MA, Fukuda T (Eds.), Intelligent & informed - proceedings of the 24th CAADRIA conference, 1:553–62. Victoria University of Wellington, Wellington, New Zealand

  • Heiser J, Phan D, Agrawala M, Tversky B, and Hanrahan P (2004) Identification and validation of cognitive design principles for automated generation of assembly instructions. In: Proceedings of the working conference on advanced visual interfaces, 311–19. ACM

  • Henderson SJ, and Feiner SK (2011) Augmented reality in the psychomotor phase of a procedural task. In: 2011 10th IEEE international symposium on mixed and augmented reality, 191–200. IEEE

  • Hoover M, Miller J, Gilbert S, and Winer E (2020) Measuring the performance impact of using the microsoft hololens 1 to provide guided assembly work instructions. J Comput Inform Sci Eng 20 (6)

  • Ibáñez M-B, Delgado-Kloos C (2018) Augmented reality for STEM learning: a systematic review. Comput Educ 123:109–123

    Article  Google Scholar 

  • Jahn G, Newnham C, and Beanland M (2018) Making in mixed reality. Holographic design, fabrication, assembly and analysis of woven steel structures. In: ACADIA // 2018: recalibration. on imprecisionand infidelity. Proceedings of the 38th annual conference of the association for computer aided design in architecture (ACADIA), 88–97. Mexico City, Mexico

  • Jirout JJ, Newcombe NS (2015) Building blocks for developing spatial skills: evidence from a large, representative US sample. Psychol Sci 26(3):302–310

    Article  Google Scholar 

  • Kasperi J, Edwardsson MP, and Romero M (2017) Occlusion in outdoor augmented reality using geospatial building data. In: Proceedings of the 23rd ACM symposium on virtual reality software and technology, 1–10

  • Kobie N (2017) LEGO AR-studio clicks together virtual and physical blocks. Wired UK, December 1, 2017. https://www.wired.co.uk/article/lego-ar-app

  • Legolizer (2018) LEGO 21036 Arc de Triomphe digital model. https://www.turbosquid.com/3d-models/paris-lego-arc-triomphe-3d-1263341

  • Lindstrom M (2016) Small data: the tiny clues that uncover huge trends. Martin’s Press, St

    Google Scholar 

  • MacAllister A, Hoover M, Gilbert S, Oliver J, Radkowski R, Garrett T, Holub J, Winer E, Terry S, and Davies P (2017) Comparing visual assembly aids for augmented reality work instructions. Mechanical Engineering Conference Presentations, Papers, and Proceedings. 197

  • Martin CV (2007a) Toward More Usable Pictorial Assembly Instructions for Children. In: Proceedings of the human factors and ergonomics society annual meeting, 51:1025–28. SAGE Publications Sage CA: Los Angeles, CA

  • Martin CV (2007b) Usability of pictorial toy assembly instructions for young children. Unpublished doctoral dissertation, virginia polytechnic Institute and State University, Blacksburg

  • Martin CV, Smith-Jackson TL (2008) Evaluation of pictorial assembly instructions for young children. Hum Factors 50(4):652–662

    Article  Google Scholar 

  • Nielsen J, and Molich R (1990) Heuristic evaluation of user interfaces. In: Proceedings of the SIGCHI conference on human factors in computing systems, 249–56

  • Olson E (2011) AprilTag: a robust and flexible visual fiducial system. In: 2011 IEEE international conference on robotics and automation, 3400–3407. IEEE

  • Qian K (2019) Building simplification - a fabrication method based on augmented reality. In: Intelligent & informed, proceedings of the 24th international conference of the association for computer-aided architectural design research in Asia (CAADRIA), 495–504. Hong Kong

  • Richardson T, Gilbert SB, Holub J, Thompson F, MacAllister A, Radkowski R, Winer E, Davies P, and Terry S (2014) Fusing self-reported and sensor data from mixed-reality training. In: Industrial and Manufacturing Systems Engineering Conference Proceedings and Posters. 92

  • Schwald B, and De Laval B (2003) An augmented reality system for training and assistance to maintenance in the industrial context. Journal of WSCG, Vol. 11, No. 1, ISSN 1213-6972. WSCG’2003, February 3–7, 2003, Plzen, Czech Republic

  • Shaik KB, Ganesan P, Kalist V, Sathish BS, J. Merlin Mary Jenitha. (2015) Comparative study of skin color detection and segmentation in HSV and YCbCr color space. Proc Comput Sci 57(12):41–48

    Article  Google Scholar 

  • Smith TP, Singer JP, Balliro GM, Lerner ND (2003) Developing consumer product instructions. US Consumer Product Safety Commission, Washington, DC

    Google Scholar 

  • Son K, Chun H, Park S, and Hyun KH (2020) C-Space: an interactive prototyping platform for collaborative spatial design exploration. In: Proceedings of the 2020 CHI conference on human factors in computing systems, 1–13

  • Tang A, Owen C, Biocca F, and Mou W (2003) Comparative effectiveness of augmented reality in object assembly. In: Proceedings of the SIGCHI conference on human factors in computing systems, 73–80

  • Tatić D, Tešić B (2017) The application of augmented reality technologies for the improvement of occupational safety in an industrial environment. Comput Ind 85:1–10

    Article  Google Scholar 

  • Tracy DM (1987) Toys, spatial ability, and science and mathematics achievement: Are they Related? Sex Roles 17(3–4):115–138

    Article  Google Scholar 

  • Wang X, Ong SK, Nee AYC (2016) A comprehensive survey of augmented reality assembly research. Adv Manuf 4(1):1–22

    Article  Google Scholar 

  • Westerfield G, Mitrovic A, and Billinghurst M (2013) Intelligent augmented reality training for assembly tasks. In: International conference on artificial intelligence in education, 542–51. Springer

  • Wood H, and Ashton P (2010) The factors of project complexity. In: TG62-special track 18th CIB world building congress

Download references

Author information

Authors and Affiliations

  1. Texas A&M University, College Station, Texas, 77843, USA

    Wei Yan

Authors
  1. Wei Yan
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Wei Yan.

Ethics declarations

Conflict of interest

There are no conflicts of interest/competing interests.

Ethical statement

The manuscript complies with the Ethical Rules applicable for this journal as stated in the Instructions for Authors of the journal Virtual Reality.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (MP4 96045 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yan, W. Augmented reality instructions for construction toys enabled by accurate model registration and realistic object/hand occlusions. Virtual Reality 26, 465–478 (2022). https://doi.org/10.1007/s10055-021-00582-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10055-021-00582-7

Keywords