Skip to main content
SpringerLink
Log in

3D Object Tracking with Adaptively Weighted Local Bundles

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

The 3D object tracking from a monocular RGB image is a challenging task. Although popular color and edge-based methods have been well studied, they are only applicable to certain cases and new solutions to the challenges in real environment must be developed. In this paper, we propose a robust 3D object tracking method with adaptively weighted local bundles called AWLB tracker to handle more complicated cases. Each bundle represents a local region containing a set of local features. To alleviate the negative effect of the features in low-confidence regions, the bundles are adaptively weighted using a spatially-variant weighting function based on the confidence values of the involved energy terms. Therefore, in each frame, the weights of the energy items in each bundle are adapted to different situations and different regions of the same frame. Experiments show that the proposed method can improve the overall accuracy in challenging cases. We then verify the effectiveness of the proposed confidence-based adaptive weighting method using ablation studies and show that the proposed method overperforms the existing single-feature methods and multi-feature methods without adaptive weighting.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Lepetit V, Fua P. Monocular model-based 3D tracking of rigid objects: A survey. Found. Trends® in Comput. Graph. Vis., 2005, 1(1): 1-89. https://doi.org/10.1561/0600000001

  2. Vacchetti L, Lepetit V, Fua P. Stable real-time 3D tracking using online and offline information. IEEE Trans. Pattern Anal. Mach. Intell., 2004, 26(10): 1385-1391. https://doi.org/10.1109/TPAMI.2004.92.

    Article  Google Scholar 

  3. Lourakis M I A, Zabulis X. Model-based pose estimation for rigid objects. In Proc. the 9th International Conference on Computer Vision Systems, July 2013, pp.83-92. https://doi.org/10.1007/978-3-642-39402-7_9.

  4. Tan D J, Tombari F, Ilic S, Navab N. A versatile learning-based 3D temporal tracker: Scalable, robust, online. In Proc. the 2015 IEEE International Conference on Computer Vision, December 2015, pp.693-701. https://doi.org/10.1109/ICCV.2015.86.

  5. Besl P J, McKay N D. A method for registration of 3-D shapes. IEEE Trans. Pattern Anal. Mach. Intell., 1992, 14(2): 239-256. https://doi.org/10.1109/34.121791.

    Article  Google Scholar 

  6. Peng S, Liu Y, Huang Q, Zhou X, Bao H. PVNet: Pixel-wise voting network for 6DoF pose estimation. In Proc. the 2019 IEEE Conference on Computer Vision and Pattern Recognition, June 2019, pp.4561-4570. https://doi.org/10.1109/CVPR.2019.00469.

  7. Ye Y, Zhang C, Hao X. ARPNET: Attention region proposal network for 3D object detection. Sci. China Inf. Sci., 2019, 62(12): Article No. 220104. https://doi.org/10.1007/s11432-019-2636-x.

  8. Garon M, Lalonde J. Deep 6-DOF tracking. IEEE Trans. Vis. Comput. Graph., 2017, 23(11): 2410-2418. https://doi.org/10.1109/TVCG.2017.2734599.

    Article  Google Scholar 

  9. Li Y, Wang G, Ji X, Xiang Y, Fox D. DeepIM: Deep iterative matching for 6D pose estimation. Int. J. Comput. Vis., 2020, 128(3): 657-678. https://doi.org/10.1007/s11263-019-01250-9.

    Article  Google Scholar 

  10. Harris C, Stennett C. RAPID—A video rate object tracker. In Proc. the 1990 British Machine Vision Conference, September 1990, pp.73-77. https://doi.org/10.5244/C.4.15.

  11. Seo B, Park H, Park J, Hinterstoisser S, Ilic S. Optimal local searching for fast and robust textureless 3D object tracking in highly cluttered backgrounds. IEEE Trans. Vis. Comput. Graph., 2014, 20(1): 99-110. https://doi.org/10.1109/TVCG.2013.94.

    Article  Google Scholar 

  12. Wang G, Wang B, Zhong F, Qin X, Chen B. Global optimal searching for textureless 3D object tracking. The Visual Computer, 2015, 31(6/7/8): 979-988. https://doi.org/10.1007/s00371-015-1098-7.

  13. Wang B, Zhong F, Qin X. Robust edge-based 3D object tracking with direction-based pose validation. Multimedia Tools Appl., 2019, 78(9): 12307-12331. https://doi.org/10.1007/s11042-018-6727-5.

    Article  Google Scholar 

  14. Zhang Y, Li X, Liu H, Shang Y. Comparative study of visual tracking method: A probabilistic approach for pose estimation using lines. IEEE Trans. Circuits Syst. Video Technol., 2017, 27(6): 1222-1234. https://doi.org/10.1109/TCSVT.2016.2527219.

    Article  Google Scholar 

  15. Prisacariu V A, Reid I D. PWP3D: Real-time segmentation and tracking of 3D objects. Int. J. Comput. Vis., 2012, 98(3): 335-354. https://doi.org/10.1007/s11263-011-0514-3.

    Article  MathSciNet  Google Scholar 

  16. Tjaden H, Schwanecke U, Schömer E. Real-time monocular segmentation and pose tracking of multiple objects. In Proc. the 14th European Conference on Computer Vision, October 2016, pp.423-438. https://doi.org/10.1007/978-3-319-46493-0_26.

  17. Hexner J, Hagege R R. 2D-3D pose estimation of heterogeneous objects using a region based approach. Int. J. Comput. Vis., 2016, 118(1): 95-112. https://doi.org/10.1007/s11263-015-0873-2.

    Article  MathSciNet  Google Scholar 

  18. Tjaden H, Schwanecke U, Schömer E. Real-time monocular pose estimation of 3D objects using temporally consistent local color histograms. In Proc. the 2017 IEEE International Conference on Computer Vision, October 2017, pp.124-132. https://doi.org/10.1109/ICCV.2017.23.

  19. Tjaden H, Schwanecke U, Schömer E, Cremers D. A region-based gauss-newton approach to real-time monocular multiple object tracking. IEEE Trans. Pattern Anal. Mach. Intell., 2019, 41(8): 1797-1812. https://doi.org/10.1109/TPAMI.2018.2884990.

  20. Marchand É, Bouthemy P, Chaumette F. A 2D-3D model-based approach to real-time visual tracking. Image Vis. Comput., 2001, 19(13): 941-955. https://doi.org/10.1016/S0262-8856(01)00054-3.

    Article  Google Scholar 

  21. Drummond T, Cipolla R. Real-time visual tracking of complex structures. IEEE Trans. Pattern Anal. Mach. Intell., 2002, 24(7): 932-946. https://doi.org/10.1109/TPAMI.2002.1017620.

    Article  Google Scholar 

  22. Wuest H, Vial F, Stricker D. Adaptive line tracking with multiple hypotheses for augmented reality. In Proc. the 4th IEEE/ACM International Symposium on Mixed and Augmented Reality, October 2005, pp.62-69. https://doi.org/10.1109/ISMAR.2005.8.

  23. Choi C, Christensen H I. Robust 3D visual tracking using particle filtering on the special Euclidean group: A combined approach of keypoint and edge features. The International Journal of Robotics Research, 2012, 31(4): 498-519. https://doi.org/10.1177/0278364912437213.

    Article  Google Scholar 

  24. Wang B, Zhong F, Qin X. Pose optimization in edge distance field for textureless 3D object tracking. In Proc. the 2017 Computer Graphics International Conference, June 2017, Article No. 32. https://doi.org/10.1145/3095140.3095172.

  25. Osher S, Sethian J A. Fronts propagating with curvature-dependent speed: Algorithms based on Hamilton-Jacobi formulations. Journal of Computational Physics, 1988, 79(1): 12-49. https://doi.org/10.1016/0021-9991(88)90002-2.

    Article  MathSciNet  MATH  Google Scholar 

  26. Zhong L, Zhao X, Zhang Y, Zhang S, Zhang L. Occlusion-aware region-based 3D pose tracking of objects with temporally consistent polar-based local partitioning. IEEE Trans. Image Process., 2020, 29: 5065-5078. https://doi.org/10.1109/TIP.2020.2973512.

    Article  Google Scholar 

  27. Lecun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86(11): 2278-2324. https://doi.org/10.1109/5.726791.

    Article  Google Scholar 

  28. Crivellaro A, Rad M, Verdie Y, Yi K M, Fua P, Lepetit V. Robust 3D object tracking from monocular images using stable parts. IEEE Trans. Pattern Anal. Mach. Intell., 2018, 40(6): 1465-1479. https://doi.org/10.1109/TPAMI.2017.2708711.

    Article  Google Scholar 

  29. Zhong L, Zhang L. A robust monocular 3D object tracking method combining statistical and photometric constraints. Int. J. Comput. Vis., 2019, 127(8): 973-992. https://doi.org/10.1007/s11263-018-1119-x.

    Article  Google Scholar 

  30. Ma Y, Soatto S, Košecká J, Sastry S S. An Invitation to 3-D Vision: From Images to Geometric Models (1st edition). Springer-Verlag New York Publishers, 2004.

  31. Zhong F, Qin X, Chen J, Hua W, Peng Q. Confidence-based color modeling for online video segmentation. In Proc. the 9th Asian Conference on Computer Vision, September 2009, pp.697-706. https://doi.org/10.1007/978-3-642-12304-7_66.

  32. Wu P, Lee Y, Tseng H, Ho H, Yang M, Chien S. A benchmark dataset for 6DoF object pose tracking. In Proc. the 2017 IEEE International Symposium on Mixed and Augmented Reality Adjunct, October 2017, pp.186-191. https://doi.org/10.1109/ISMAR-Adjunct.2017.62.

  33. Brachmann E, Michel F, Krull A, Yang M Y, Gumhold S, Rother C. Uncertainty-driven 6D pose estimation of objects and scenes from a single RGB image. In Proc. the 2016 IEEE Conference on Computer Vision and Pattern Recognition, June 2016, pp.3364-3372. https://doi.org/10.1109/CVPR.2016.366.

  34. Whelan T, Leutenegger S, Salas-Moreno R F, Glocker B, Davison A J. ElasticFusion: Dense SLAM without a pose graph. In Proc. the 2015 Robotics: Science and Systems, July 2015. https://doi.org/10.15607/RSS.2015.XI.001.

  35. Mur-Artal R, Tardós J D. ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras. IEEE Trans. Robotics, 2017, 33(5): 1255-1262. https://doi.org/10.1109/TRO.2017.2705103.

    Article  Google Scholar 

  36. Marchand É, Uchiyama H, Spindler F. Pose estimation for augmented reality: A hands-on survey. IEEE Trans. Vis. Comput. Graph., 2016, 22(12): 2633-2651. https://doi.org/10.1109/TVCG.2015.2513408.

    Article  Google Scholar 

  37. Cheng M, Liu Y, Lin W, Zhang Z, Rosin P L, Torr P H S. BING: Binarized normed gradients for objectness estimation at 300fps. Comput. Vis. Media, 2019, 5(1): 3-20. https://doi.org/10.1007/s41095-018-0120-1.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Software, Shandong University, Jinan, 250101, China

    Jia-Chen Li & Xue-Ying Qin

  2. School of Computer Science and Technology, Shandong University, Qingdao, 266237, China

    Fan Zhong

  3. College of Engineering and Computing, University of South Carolina, Columbia, 29208, U.S.A.

    Song-Hua Xu

Authors
  1. Jia-Chen Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fan Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Song-Hua Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xue-Ying Qin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Fan Zhong or Xue-Ying Qin.

Additional information

Xue-Ying Qin and Fan Zhong both supervised this work and provided funding support

Supplementary Information

ESM 1

(PDF 150 kb)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, JC., Zhong, F., Xu, SH. et al. 3D Object Tracking with Adaptively Weighted Local Bundles. J. Comput. Sci. Technol. 36, 555–571 (2021). https://doi.org/10.1007/s11390-021-1272-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-021-1272-5

Keywords

Search

Navigation