Skip to main content
SpringerLink
Log in

A novel interactive assembly teaching aid using multi-template augmented reality

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Augmented Reality (AR) technology is a fast growing field in the academics and industry. This paper presents a novel design of an interactive assembly teaching aid based on a multi-template AR system, which consists of three units: a multi-template AR unit, an online three-dimension (3D) model assembly unit, a hand-gesture interaction unit. The design of the multi-template AR unit employs an efficient multi-template pose tracking method to detect and track multiple template images simultaneously. The online 3D model assembly unit is enabled, when the pose of each target template is tracked and computed by the multi-template pose tracking method. This method measures the distance between the two templates to determine the 3D rendering mode of the virtual object. The third unit aims to realize a vision-based human-computer interactive system, which combines the AR rendering system with the real-time hand-gesture recognition method to decide the status of AR rendering of a 3D dynamic animation according to the user’s hand gesture. Based on the feedback of twenty-one users of the AR-based interactive teaching-aid, it can be concluded that the system creates an interactive experiences that engages the users and facilitates them to increase their learning interest significantly. In future, we plan to port the proposed AR system to Android and iOS mobile devices while improving the functionality, interactivity, and entertaining qualities of the teaching aid system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

References

  1. 3ds Max. Available online: https://www.autodesk.com.tw/products/3ds-max/overview (accessed on 17/03/2019).

  2. Alsmirat MA, AI-Alem F, AI-Ayyoub M, Jararweh Y, Gupta B (2019) Impact of digital fingerprint image quality on the fingerprint recognition accuracy. Multimed Tools Appl 78(3):3649–3688

    Article  Google Scholar 

  3. Alvarado LAR, Domínguez EL, Velázquez YH, Isidro SD, Toledo CBE (2018) Layered software architecture for the development of mobile learning objects with augmented reality. IEEE Access 6(2018):57897–57909

    Article  Google Scholar 

  4. Andújar JM, Mejías A, Márquez MA (2011) Augmented reality for the improvement of remote laboratories: an augmented remote laboratory. IEEE Trans Educ 54(3):492–500

    Article  Google Scholar 

  5. ARCore. Available online: https://developers.google.com/ar/ (accessed on 09/08/2019)

  6. Argyros AA; Lourakis MIA (2006) Vision-based interpretation of hand gestures for remote control of a computer mouse. In Proceedings of ECCV 2006: Computer vision in human-computer interaction, Graz, Austria, pp 40-51

  7. ARToolKit. Available online: https://github.com/artoolkit (accessed on 13/03/2019).

  8. ARToolKit camera calibration. Available online: http://www.hitl.washington.edu/artoolkit/documentation/usercalibration.htm (accessed on 21/03/2019).

  9. Azuman R (1997) A survey of augmented reality. Presence Teleop Virt 6(4):355–385

    Article  Google Scholar 

  10. Bimber O; Raskar R (2005) Spatial augmented reality: Merging real and virtual worlds, 1st ed.; Publisher: A K Peters Ltd, Wellesley, Massachusetts, USA, pp. 71–90

  11. Bin Abdul Rahman NA; WeiKC; See J (2006) RGB-H-CbCr skin colour model for human face detection. In Proceedings of the MMU International Symposium on Information and Communications Technologies, Petaling Jaya, Malaysia, pp. 1–6

  12. Bocevska A, Kotevski Z (2017) Implementation of interactive augmented reality in 3D assembly design presentation. International Journal of Computer Science & Information Technology 9(2):141–149

    Article  Google Scholar 

  13. Buenaposada JM; Baumela L (2002) Real-time tracking and estimation of plane pose. In Proceedings of IEEE International Conference on Pattern Recognition, Quebec, Canada, pp. 697–700

  14. Camera calibration toolbox for Matlab. Available online: http://www.vision.caltech.edu/bouguetj/calib_doc/ (accessed on 21/03/2019).

  15. Camera calibration with OpenCV. Available online: https://docs.opencv.org/2.4.13.7/doc/tutorials/calib3d/camera_calibration/camera_calibration.html (accessed on 21/03/2019).

  16. Casseb do Carmo RM; Meiguins BS; Meiguins ASG; Pinheiro SCV; Almeida LH; Godinho PIA (2007) Coordinated and multiple views in augmented reality environment. In Proceedings of 11th International Conference Information Visualization, Zurich, Switzerland, pp. 156–162

  17. Chen H; Feng K; Mo C; Cheng S; Guo Z; Huang Y (2011) Application of augmented reality in engineering graphics education. In Proceedings of the IEEE International Symposium on IT in Medicine and Education, Cuangzhou, China, pp. 362–365

  18. Chiu C-W (2010) The study of architecture design and research about multiuser augmented reality system. Master Thesis, Department of Digital Content and Technology, National Taichung University of Education Institutional Repository

  19. Educational App Store, Augmented Reality Apps. Available online: https://www.educationalappstore.com/app/category/augmented-reality-apps (accessed on 30/08/2019).

  20. EE Times Taiwan, Embedded Technologies. Available online: http://archive.eettaiwan.com/www.eettaiwan.com/ART_8800658971_676964_NP_bb0a194c.HTM (accessed on 05/07/2019).

  21. Experimental result of the proposed multi-template markless AR system. Available online: https://www.youtube.com/watch?v=EzjkaDbzsiE (accessed on 21/03/2019)

  22. Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–395

    Article  MathSciNet  Google Scholar 

  23. Free3D. Available online: https://free3d.com (accessed on 17/03/2019).

  24. Freeman WT; Weissman CD (1995) Television control by hand gestures. In Proceedings of International Workshop on Automatic Face and Gesture Recognition, Zurich, Switzerland, pp. 179–183

  25. Ghoneim A, Muhammad G, Amin SU, Gupta B (2018) Medical image forgery detection for smart healthcare. IEEE Commun Mag 56(4):33–37

    Article  Google Scholar 

  26. Goléa NE-H, Melkemi KE (2019) ROI-based fragile watermarking for medical image tamper detection. International Journal of High Performance Computing and Networking 13(2):199–210

    Article  Google Scholar 

  27. Ha H, Rameau F, Kweon IS (2016) 6-DOF direct homography tracking with extended Kalman filter. Lect Notes Comput Sci 9431:447–460

    Article  Google Scholar 

  28. Hartley R, Zisserman A (2004) Multiple view geometry in computer vision, 2nd edn. Publisher, Cambridge University Press

    Book  Google Scholar 

  29. Holzer, S.; Ilic, S.; Tan, D.; Pollefeys, M.; Navab, N.: Efficient learning of linear predictors for template tracking. Int J Comput Vis, 111(1), 12–28 (2015)

  30. Ibánez M-B, Di-Serio Á, Villarán-Molina D, Delgado-Kloos C (2015) Augmented reality-based simulators as discovery learning tools: an empirical study. IEEE Trans Educ 58(3):208–213

    Article  Google Scholar 

  31. Ibánez M-B, Di-Serio Á, Villarán-Molina D, Delgado-Kloos C (2016) Support for augmented reality simulation systems: the effects of scaffolding on learning outcomes and behavior patterns. IEEE Trans Learn Technol 9(1):46–56

    Article  Google Scholar 

  32. Imbert N, Vignat F, Kaewrat C, Boonbrahm P (2013) Adding physical properties to 3D models in augmented reality for realistic interactions experiments. Procedia Computer Science 25(2013):364–369

    Article  Google Scholar 

  33. Kovac J; Peer P; Solina F (2003) Human skin colour clustering for face detection. In Proceedings of the IEEE Region 8 EUROCON, Ljubljana, Slovenia, pp. 144–148

  34. Kumar A (2019) Design of secure image fusion technique using cloud for privacy-preserving and copyright protection. International Journal of Cloud Applications and Computing 9(3):22–36

    Article  Google Scholar 

  35. Lei T, Liu X-F, Cai G-P, Liu Y-M, Liu P (2019) Pose estimation of a noncooperative target based on monocular visual slam. International Journal of Aerospace Engineering 9086891:1–14

    Article  Google Scholar 

  36. Li Y, Shi H, Chen L, Jiang F (2019) Convolutional approach also benefits traditional face pattern recognition algorithm. International Journal of Software Science and Computational Intelligence 11(4):1–16

    Article  Google Scholar 

  37. Liu W, Wu S, Wu X (2018) Pose estimation method for planar mirror based on one-dimensional target. Opt Eng 57(7):073101

    Google Scholar 

  38. Meena S (2011) A study on hand gesture recognition technique. Master of Technology, Department of Electronics and Communication Engineering, National Institute of Technology, Rourkela, India

  39. Milgram P; Takemura H; Utsumi A; Kishino F (1995) Augmented reality: a class of displays on the reality-virtuality continuum. In Proceedings of SPIE 2351, Telemanipulator and Telepresence Technologies, Boston, USA

  40. Nugraha IE; Sen TW; Wahyu RB; Sulistyo B (2017) Rosalina.: Assembly instruction with augmented reality on android application “Assembly with AR”. In Proceedings of 4th International Conference on New Media Studies, Yogyakarta, Indonesia, pp.32–37

  41. Rekimoto J (2002) SmartSkin: An infrastructure for freehand manipulation on interactive surfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Minneapolis, USA, pp. 113–120

  42. Rentzos L; Papanastasiou S; Papakostas N; Chryssolouris G (2013) Augmented reality for human-based assembly: using product and process semantics. In Proceedings of 12th IFAC Symposium on Analysis, Design, and Evaluation of Human-Machine Systems, Las Vegas, USA, 46(15), pp. 98–101

  43. Salman FH; Riley DR (2016) Augmented reality crossover gamified design for sustainable engineering education. In Proceedings of Future Technologies Conference, San Francisco, USA, pp. 1353–1356

  44. Santos CME, Chen A, Taketomi T (2014) Augmented reality learning experiences: survey of prototypes design and evaluation. IEEE Transactions on Learning Technologies 7(1):38–56

    Article  Google Scholar 

  45. SiftGPU (A GPU Implementation of SIFT). Available online: https://github.com/pitzer/SiftGPU (accessed on 05/07/2019).

  46. Sklansky J (1972) Measuring concavity on a rectangular mosaic. IEEE Trans Comput 21(12):1355–1364

    Article  MathSciNet  Google Scholar 

  47. Su J-H, Chin C-Y, Li J-Y, Tseng VS (2019) Parallel big image data retrieval by conceptualised clustering and un-conceptualised clustering. International Journal of High Performance Computing and Networking 15(1–2):22–30

    Article  Google Scholar 

  48. Sun C, Li C, Zhu Y (2019) A novel convolutional neural network based localization system for monocular images. International Journal of Software Science and Computational Intelligence 11(2):38–50

    Article  Google Scholar 

  49. Tian K, Urata M, Endo M, Mouri K, Yasuda T, Kato J (2019) Real-world oriented smartphone AR supported learning system based on planetarium contents for seasonal constellation observation. Appl Sci 9(3508):1–18

    Google Scholar 

  50. Tsai C-Y, Huang C-H, Tsao A-H (2016) Graphics processing unit accelerated multi-resolution exhaustive search algorithm for real-time keypoint descriptor matching in high-dimensional spaces. IET Comput Vis 10(3):212–219

    Article  Google Scholar 

  51. Tsai C-Y, Hsu K-J, Nisar H (2018) Efficient model-based object pose estimation based on multi-template tracking and PnP algorithms. Algorithms 11(8:122):1–14

    MATH  Google Scholar 

  52. TurboSquid. https://www.turbosquid.com (accessed on 17/03/2019).

  53. Urbano D; de Fátima Chouzel M; Restivo MT (2015) How students and teachers react to an AR free puzzle game: preliminary tests. In Proceedings of IEEE Global Engineering Education Conference, Tallinn, Estonia, pp. 852–855

  54. Usage of Augmented Reality in Education Process in 2017. Available online: https://thinkmobiles.com/blog/augmented-reality-education/ (accessed on 13/03/2019).

  55. Vuforia engine. Available online: https://developer.vuforia.com/ (accessed on 09/08/2019)

  56. Yu C, Li J, Li X, Ren X, Gupta BB (2018) Four-image encryption scheme based on quaternion Fresnel transform, chaos and computer generated hologram. Multimed Tools Appl 77(4):4585–4608

    Article  Google Scholar 

  57. Zhou F; Duh HBL; Billinghurst M (2008) Trends in augmented reality tracking, interaction and display: A review of ten years of ISMAR. In Proceedings of 7th IEEE/ACM International Symposium on Mixed and Augmented Reality, Cambridge, UK, pp. 193–202

Download references

Acknowledgments

This work was supported by the Ministry of Science and Technology of Taiwan, ROC under grant MOST 108-2221-E-032-046 and MOST 109-2221-E-032-039 and was supported in part by NUWA Robotics Co., Ltd., Nanjing East Road, Zhongshan District, Taipei City 104, Taiwan.

Author information

Authors and Affiliations

  1. Department of Electrical and Computer Engineering, TamKang University, 151 Yingzhuan Road, Tamsui District, New Taipei City, 251, Taiwan, Republic of China

    Chi-Yi Tsai, Ting-Yuan Liu & Yun-Han Lu

  2. Department of Electronic Engineering, Universiti Tunku Abdul Rahman, Jalan Universiti, Bandar Barat, 31900, Kampar, Perak, Malaysia

    Humaira Nisar

Authors
  1. Chi-Yi Tsai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ting-Yuan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yun-Han Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Humaira Nisar
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization, Chi-Yi Tsai; Methodology, Chi-Yi Tsai and Ting-Yuan Liu; Software, Chi-Yi Tsai and Ting-Yuan Liu; Validation, Chi-Yi Tsai and Ting-Yuan Liu; Formal Analysis, Chi-Yi Tsai; Investigation, Chi-Yi Tsai; Resources, Chi-Yi Tsai; Experimental Data Collection, Ting-Yuan Liu and Yun-Han Lu; Writing-Original Draft Preparation, Chi-Yi Tsai, Ting-Yuan Liu and Humaira Nisar; Writing-Review & Editing, Chi-Yi Tsai and Humaira Nisar; Visualization, Ting-Yuan Liu and Yun-Han Lu; Supervision, Chi-Yi Tsai; Project Administration, Chi-Yi Tsai; Funding Acquisition, Chi-Yi Tsai.

Corresponding author

Correspondence to Chi-Yi Tsai.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tsai, CY., Liu, TY., Lu, YH. et al. A novel interactive assembly teaching aid using multi-template augmented reality. Multimed Tools Appl 79, 31981–32009 (2020). https://doi.org/10.1007/s11042-020-09584-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-09584-0

Keywords

Search

Navigation