Skip to main content
SpringerLink
Log in

3D model-based tracking combining edges, keypoints and fiducial markers

  • Original Article
  • Published:
Virtual Reality Aims and scope Submit manuscript

Abstract

Model-based tracking is an essential task in fields such as Augmented Reality. State-of-the-art approaches rely on the model’s edges, sometimes combined with image keypoints and color. Nevertheless, these image features are not considered part of the model but as temporary information discarded every time the tracking process is restarted. This paper proposes a novel approach that employs an enhanced model that combines edges, keypoints, and fiducial markers for robust and real-time tracking. The experiments conducted show that our method outperforms state-of-the-art model-based approaches and suggest that fiducial markers are a good choice for texturing models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data availability

The datasets generated during and/or analysed during the current study are not publicly available due to property rights protection agreed with the partnering company Seabery.

Notes

  1. https://www.seaberyat.com/.

  2. https://www.soldamatic.com/.

  3. https://www.optitrack.com/software/.

  4. https://ViSP.inria.fr/.

References

  • Bouthemy P (1989) A maximum likelihood framework for determining moving edges. IEEE Trans Pattern Anal Mach Intell 11(5):499–511

    Article  Google Scholar 

  • Cavallaro R (1997) The foxtrax hockey puck tracking system. IEEE Comput Graphics Appl 17(2):6–12

    Article  Google Scholar 

  • Chen Y, Li W, Sakaridis C, Dai D, Van Gool L (June 2018) Domain adaptive faster r-cnn for object detection in the wild. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)

  • Choi C, Christensen HI (2012) Robust 3d visual tracking using particle filtering on the special euclidean group: a combined approach of keypoint and edge features. Int J Robot Res 31(4):498–519

    Article  Google Scholar 

  • Choi C, Christensen Henrik I (2010) Real-time 3d model-based tracking using edge and keypoint features for robotic manipulation. In: 2010 IEEE international conference on robotics and automation, p 4048–4055

  • Collet A, Berenson D, Srinivasa Siddhartha S, Ferguson D (2009) Object recognition and full pose registration from a single image for robotic manipulation. In: 2009 IEEE international conference on robotics and automation, p 48–55. IEEE

  • Comport AI, Marchand E, Pressigout M, Chaumette F (2006) Real-time markerless tracking for augmented reality: the virtual visual servoing framework. IEEE Trans Visual Comput Graphics 12(4):615–628

    Article  Google Scholar 

  • Drummond T, Cipolla R (1999) Real-time tracking of complex structures with on-line camera calibration. In: BMVC, p 1–10. Citeseer

  • Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–395

    Article  MathSciNet  Google Scholar 

  • Foxlin E, Altshuler Y, Naimark L, Harrington M (2004) Flighttracker: a novel optical/inertial tracker for cockpit enhanced vision. In: Third IEEE and ACM international symposium on mixed and augmented reality, p 212–221. IEEE

  • Gao X, Wang R, Demmel N (2018) and Daniel Cremers. Direct sparse odometry with loop closure, Ldso

  • Garrido-Jurado S, Muñoz-Salinas R, Madrid-Cuevas FJ, Marín-Jiménez MJ (2014) Automatic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recogn 47(6):2280–2292

    Article  Google Scholar 

  • Gidaris S, Komodakis N (December 2015) Object detection via a multi-region and semantic segmentation-aware cnn model. In: Proceedings of the IEEE international conference on computer vision (ICCV)

  • Gordon Iryna, Lowe David G (2006) What and where: 3d object recognition with accurate pose. In: Toward category-level object recognition, Springer, p 67–82

  • Han P, Zhao G (2019) A review of edge-based 3d tracking of rigid objects. Virtual Real Intell Hardw 1(6):580–596

    Article  Google Scholar 

  • Harris C, Stennett C (1990) Rapid-a video rate object tracker. In: BMVC, pages 1–6

  • Horn BKP (1987) Closed-form solution of absolute orientation using unit quaternions. J Opt Soc Am A 4(4):629–642

    Article  Google Scholar 

  • Issac J, Wüthrich M, Cifuentes C, Bohg J, Trimpe S, Schaal S (2016) Depth-based object tracking using a robust gaussian filter. In 2016 IEEE international conference on robotics and automation (ICRA), p 608–615. IEEE

  • Jiao L, Zhang F, Liu F, Yang S, Li L, Feng Z, Rong Q (2019) A survey of deep learning-based object detection. IEEE Access 7:128837–128868

    Article  Google Scholar 

  • Kang K, Li H, Yan J, Zeng X, Yang B, Xiao T, Zhang C, Wang Z, Wang R, Wang X, Ouyang W (2018) T-cnn: Tubelets with convolutional neural networks for object detection from videos. IEEE Trans Circuits Syst Video Technol 28(10):2896–2907

    Article  Google Scholar 

  • Kyrki V, Kragic D (2005) Integration of model-based and model-free cues for visual object tracking in 3d. In: Proceedings of the 2005 IEEE international conference on robotics and automation, p 1554–1560. IEEE

  • Lee G, Billinghurst M, Clark A (2015) A survey of augmented reality. Foundations and Trends® in Human-Computer Interaction 8(2-3):73–272

  • Marchand E, Spindler F, Chaumette F (2005) Visp for visual servoing: a generic software platform with a wide class of robot control skills. IEEE Robot Automation Mag 12(4):40–52

    Article  Google Scholar 

  • Munoz Fernando I Ireta, Comport Andrew I (2016) Point-to-hyperplane rgb-d pose estimation: Fusing photometric and geometric measurements. In: 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS), p 24–29, IEEE

  • Muñoz-Salinas R, Marín-Jimenez MJ, Medina-Carnicer R (2019) Spm-slam: Simultaneous localization and mapping with squared planar markers. Pattern Recogn 86:156–171

    Article  Google Scholar 

  • Muñoz-Salinas R, Medina-Carnicer R (2020) Ucoslam: simultaneous localization and mapping by fusion of keypoints and squared planar markers. Pattern Recognition, 107193

  • Mur-Artal R, Tardós JD (2017) Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE Trans Rob 33(5):1255–1262

    Article  Google Scholar 

  • Nee AYC, Ong SK, Chryssolouris G, Mourtzis D (2012) Augmented reality applications in design and manufacturing. CIRP Ann 61(2):657–679

    Article  Google Scholar 

  • Olson E (2011) Apriltag: a robust and flexible visual fiducial system. In: 2011 IEEE international conference on robotics and automation, p 3400–3407. IEEE

  • Paterson MS, Yao FF (1990) Efficient binary space partitions for hidden-surface removal and solid modeling. Discret Comput Geom 5(5):485–503

    Article  MathSciNet  MATH  Google Scholar 

  • Petit A, Marchand E, Kanani K (2014) Combining complementary edge, keypoint and color features in model-based tracking for highly dynamic scenes. In: 2014 IEEE international conference on robotics and automation (ICRA), p 4115–4120. IEEE

  • Pfrommer B, Daniilidis K (2019) Tagslam: robust slam with fiducial markers. arXiv preprint arXiv:1910.00679

  • Pressigout M, Marchand E (2007) Real-time hybrid tracking using edge and texture information. Int J Robot Res 26(7):689–713

    Article  Google Scholar 

  • Pressigout M, Marchand E (2006) Real-time 3d model-based tracking: combining edge and texture information. In: Proceedings 2006 IEEE international conference on robotics and automation, 2006. ICRA 2006., p 2726–2731. IEEE

  • Ren S, He K, Girshick R, Sun J (2015) Faster r-cnn: Towards real-time object detection with region proposal networks. In: C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds) Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.,

  • Ren C, Prisacariu V, Kaehler O, Reid I, Murray D (2014) 3d tracking of multiple objects with identical appearance using rgb-d input. In: 2014 2nd International conference on 3D vision, volume 1, p 47–54. IEEE

  • Rey D, Neuhäuser M (2011) Wilcoxon-Signed-Rank Test, p 1658–1659. Springer Berlin Heidelberg, Berlin, Heidelberg

  • Romero-Ramirez FJ, Muñoz-Salinas R, Medina-Carnicer R (2018) Speeded up detection of squared fiducial markers. Image Vis Comput 76:38–47

    Article  Google Scholar 

  • Rublee E, Rabaud V, Konolige K, Bradski G (Nov 2011) Orb: an efficient alternative to sift or surf. In 2011 International conference on computer vision, p 2564–2571

  • Sun Y, Page DL, Paik JK, Koschan A, Abidi MA (2002) Triangle mesh-based edge detection and its application to surface segmentation and adaptive surface smoothing. In: Proceedings. international conference on image processing, volume 3, pages 825–828

  • Sutherland Ivan E (1968) A head-mounted three dimensional display. In: Proceedings of the December 9-11, 1968, fall joint computer conference, Part I, AFIPS ’68 (Fall, part I), pp 757-764, New York, NY, USA. Association for Computing Machinery

  • Triggs B, McLauchlan P, Hartley R, Fitzgibbon A (1999) Bundle adjustment – a modern synthesis. In ICCV ’99: Proceedings of the International Workshop on Vision Algorithms., p 298–372. Springer-Verlag

  • Trinh S, Spindler F, Marchand E, Chaumette F (2018) A modular framework for model-based visual tracking using edge, texture and depth features. In 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS), p 89–96. IEEE

  • Vacchetti L, Lepetit V, Fua P (2004) Combining edge and texture information for real-time accurate 3d camera tracking. In: Third IEEE and ACM international symposium on mixed and augmented reality, p 48–56. IEEE

  • Ye E, Althoff M (2019) Model-based offline vehicle tracking in automotive applications using a precise 3d model. In: 2019 IEEE intelligent transportation systems conference (ITSC), p 1128–1135

  • Yovcheva Z, Buhalis D, Gatzidis C (2012) Smartphone augmented reality applications for tourism. E-Rev Tour Res (ertr) 10(2):63–66

    Google Scholar 

Download references

Funding

This research was funded by the Industrial PhD Program of Córdoba University with Seabery R &D and Project PID2019-103871GB-I00 of the Spanish Ministry of Economy, Industry and Competitiveness, and FEDER and Project 1380047-F UCOFEDER-2021 of Andalusia.

Author information

Authors and Affiliations

  1. Departamento de Informática y Análisis Numérico, Edificio Einstein. Campus de Rabanales, Universidad de Coŕdoba, 14071, Córdoba, Spain

    David Jurado-Rodriguez, Rafael Muñoz-Salinas, Francisco J. Romero-Ramirez & Rafael Medina-Carnicer

  2. Seabery R &D, Doctor Emilio Haya Prats 13, 21005, Huelva, Spain

    David Jurado-Rodriguez & Sergio Garrido-Jurado

  3. Instituto Maimónides de Investigación en Biomedicina (IMIBIC), Avenida Menéndez Pidal s/n, 14004, Córdoba, Spain

    Rafael Muñoz-Salinas & Rafael Medina-Carnicer

Authors
  1. David Jurado-Rodriguez
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rafael Muñoz-Salinas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sergio Garrido-Jurado
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Francisco J. Romero-Ramirez
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Rafael Medina-Carnicer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rafael Muñoz-Salinas.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jurado-Rodriguez, D., Muñoz-Salinas, R., Garrido-Jurado, S. et al. 3D model-based tracking combining edges, keypoints and fiducial markers. Virtual Reality 27, 3051–3065 (2023). https://doi.org/10.1007/s10055-023-00853-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10055-023-00853-5

Keywords

Search

Navigation