Abstract
This paper addresses robust and ultrafast pose tracking on mobile devices, such as smartphones and small drones. Existing methods, relying on either vision analysis or inertial sensing, are either too computational heavy to achieve real-time performance on a mobile platform, or not sufficiently robust to address unique challenges in mobile scenarios, including rapid camera motions, long exposure time of mobile cameras, etc. This paper presents a novel hybrid tracking system which utilizes on-device inertial sensors to greatly accelerate the visual feature tracking process and improve its robustness. In particular, our system adaptively resizes each video frame based on inertial sensor data and applies a highly efficient binary feature matching method to track the object pose in each resized frame with little accuracy degradation. This tracking result is revised periodically by a model-based feature tracking method (Hare et al. 2012) to reduce accumulated errors. Furthermore, an inertial tracking method and a solution of fusing its results with the feature tracking results are employed to further improve the robustness and efficiency. We first evaluate our hybrid system using a dataset consisting of 16 video clips with synchronized inertial sensing data and then assess its performance in a mobile augmented reality application. Experimental results demonstrated our method’s superior performance to a state-of-the-art feature tracking method (Hare et al. 2012), a direct tracking method (Engel et al. 2014) and the Vuforia SDK (Ibañez and Figueras 2013), and can run at more than 40 Hz on a standard smartphone. We will release the source code with the pubilication of this paper.
Similar content being viewed by others
References
Chliveros G, Pateraki M, Trahanias P (2013) Robust multi-hypothesis 3d object pose tracking International conference on computer vision systems. Springer, pp 234–243
Chum O, Matas J (2005) Matching with prosac-progressive sample consensus 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), vol 1. IEEE, pp 220–226
Crivellaro A, Rad M, Verdie Y, Moo Yi K, Fua P, Lepetit V (2015) A novel representation of parts for accurate 3d object detection and tracking in monocular images Proceedings of the IEEE international conference on computer vision, pp 4391–4399
Daniel W, Dieter S (2007) Artoolkitplus for pose tracking on mobile devices Proceedings of 12th computer vision winter workshop
DiVerdi S, Hollerer T (2007) Groundcam: a tracking modality for mobile mixed reality 2007 IEEE virtual reality conference. IEEE, pp 75–82
Engel J, Schöps T, Cremers D (2014) Lsd-slam: large-scale direct monocular slam European conference on computer vision. Springer, pp 834–849
Engel J, Sturm J, Cremers D (2013) Semi-dense visual odometry for a monocular camera Proceedings of the IEEE international conference on computer vision, pp 1449–1456
Fiala M (2010) Designing highly reliable fiducial markers. IEEE Trans Pattern Anal Mach Intell 32(7):1317–1324
Hallaway D, Feiner S, Höllerer T (2004) Bridging the gaps: hybrid tracking for adaptive mobile augmented reality. Appl Artif Intell 18(6):477–500
Hare S, Saffari A, Torr PH (2012) Efficient online structured output learning for keypoint-based object tracking IEEE conference on computer vision and pattern recognition (CVPR), 2012. IEEE, pp 1894–1901
Ibañez AS, Figueras JP (2013) Vuforia v1. 5 sdk: analysis and evaluation of capabilities. Ph.D. thesis, Master Thesis, Universitat Politecnica De Catalunya
Kato H, Billinghurst M (1999) Marker tracking and hmd calibration for a video-based augmented reality conferencing system 2nd IEEE and ACM international workshop on augmented reality, 1999.(IWAR’99) proceedings. IEEE, pp 85–94
Klein G, Murray D (2007) Parallel tracking and mapping for small ar workspaces 6th IEEE and ACM international symposium on mixed and augmented reality, 2007. ISMAR 2007. IEEE, pp 225–234
Klein G, Murray D (2009) Parallel tracking and mapping on a camera phone 8th IEEE international symposium on mixed and augmented reality, 2009. ISMAR 2009. IEEE, pp 83–86
Klein G, Murray DW (2006) Full-3d edge tracking with a particle filter BMVC, pp 1119–1128
Lepetit V, Fua P (2006) Keypoint recognition using randomized trees. IEEE Trans Pattern Anal Mach Intell 28(9):1465–1479
Li M, Kim BH, Mourikis AI (2013) Real-time motion tracking on a cellphone using inertial sensing and a rolling-shutter camera Robotics and automation (ICRA), 2013 IEEE international conference on. IEEE, pp 4712–4719
Ligorio G, Sabatini AM (2013) Extended kalman filter-based methods for pose estimation using visual, inertial and magnetic sensors: comparative analysis and performance evaluation. Sensors 13(2):1919–1941
Morgan DL (2014) Improved orientation estimation for smart phone indoor localization. Ph.D. thesis, UNIVERSITY OF CALIFORNIA SANTA BARBARA
Obeidy WK, Arshad H, Chowdhury SA, Parhizkar B, Huang J (2013) Increasing the tracking efficiency of mobile augmented reality using a hybrid tracking technique International visual informatics conference. Springer, pp 447–457
Ozuysal M, Calonder M, Lepetit V, Fua P (2010) Fast keypoint recognition using random ferns. IEEE Trans Pattern Anal Mach Intell 32(3):448–461
Payet N, Todorovic S (2011) From contours to 3d object detection and pose estimation 2011 International conference on computer vision. IEEE, pp 983–990
Prisacariu VA, Reid ID (2012) Pwp3d: real-time segmentation and tracking of 3d objects. Int J Comput Vis 98(3):335–354
Prisacariu VA, Segal AV, Reid I (2012) Simultaneous monocular 2d segmentation, 3d pose recovery and 3d reconstruction Asian conference on computer vision. Springer, pp 593–606
Rolland JP, Davis L, Baillot Y (2001) A survey of tracking technology for virtual environments. Fundamentals of Wearable Computers and Augmented Reality 1 (1):67–112
Rosten E, Porter R, Drummond T (2010) Faster and better: a machine learning approach to corner detection. IEEE Trans Pattern Anal Mach Intell 32(1):105–119
Rublee E, Rabaud V, Konolige K, Bradski G (2011) Orb: an efficient alternative to sift or surf 2011 International conference on computer vision. IEEE, pp 2564–2571
Schöps T, Engel J, Cremers D (2014) Semi-dense visual odometry for ar on a smartphone IEEE international symposium on mixed and augmented reality (ISMAR), 2014. IEEE, pp 145–150
Wagner D, Langlotz T, Schmalstieg D (2008) Robust and unobtrusive marker tracking on mobile phones 7th IEEE/ACM international symposium on mixed and augmented reality, 2008. ISMAR 2008. IEEE, pp 121–124
Wagner D, Reitmayr G, Mulloni A, Drummond T, Schmalstieg D (2008) Pose tracking from natural features on mobile phones Proceedings of the 7th IEEE/ACM international symposium on mixed and augmented reality. IEEE Computer Society, pp 125–134
Wagner D, Reitmayr G, Mulloni A, Drummond T, Schmalstieg D (2010) Real-time detection and tracking for augmented reality on mobile phones. IEEE Trans Vis Comput Graph 16(3):355–368
White S, Feiner S, Kopylec J (2006) Virtual vouchers: prototyping a mobile augmented reality user interface for botanical species identification 3D user interfaces (3DUI’06). IEEE, pp 119– 126
Yang X, Cheng KTT (2014) Local difference binary for ultrafast and distinctive feature description. IEEE Trans Pattern Anal Mach Intell 36(1):188–194
Yang X, Huang C, Cheng KTT (2014) libldb: a library for extracting ultrafast and distinctive binary feature description Proceedings of the 22nd ACM international conference on multimedia. ACM, pp 671–674
Yang X, Si X, Xue T, Cheng KTT (2015) [Poster] Fusion of vision and inertial sensing for accurate and efficient pose tracking on smartphones IEEE international symposium on mixed and augmented reality (ISMAR), 2015. IEEE, pp 68–71
Yang X, Si X, Xue T, Zhang L, Cheng KTT (2015) Vision-inertial hybrid tracking for robust and efficient augmented reality on smartphones Proceedings of the 23rd ACM international conference on multimedia. ACM, pp 1039–1042
Acknowledgements
This work is funded by the National Science Foundation of China (grant no. 61502188).
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
(MP4 76.2 MB)
Rights and permissions
About this article
Cite this article
Yang, X., Guo, J., Xue, T. et al. Robust and real-time pose tracking for augmented reality on mobile devices. Multimed Tools Appl 77, 6607–6628 (2018). https://doi.org/10.1007/s11042-017-4575-3
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-017-4575-3