Skip to main content
SpringerLink
Log in

Multi-view 2D–3D alignment with hybrid bundle adjustment for visual metrology

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

High-precision measurement based on multi-view geometry benefits from aligning a template CAD model to the multi-view observations of a target object. It not only improves the multi-camera calibration but also assists the measurement by serving as a “scaffold.” A straightforward approach is to reconstruct the target object and perform a 3D registration with the CAD model. However, the accuracy of such 3D alignment cannot meet the high-precision requirement. We formulate the problem as a bundle adjustment where we jointly optimize the 6DoF poses of both the template model and the multiple cameras. To accommodate the manufacturing error of products, we propose a simple and robust solution based on a discrete–continuous optimization which interleaves between correspondence selection and pose optimization. In the discrete step, a robust RANSAC-based selection process selects well-matched 2D–3D feature points according to the current model/camera poses. In the continuous step, it jointly optimizes the 6D poses of the CAD model and the multiple cameras. This interleaving optimization constitutes a novel hybrid bundle adjustment (HBA). In HBA, the reprojection error of a feature point is measured either with the 2D–3D correspondence between the observation image and the CAD model or with the cross-view correspondence between multiple images, whichever is more reliable according to the discrete selection step. Through extensive evaluation on real-world data, we demonstrate that HBA achieves high-precision multi-camera calibration, outperforming alternative approaches significantly.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Kalibr. https://github.com/ethz-asl/kalibr (2020)

  2. Agarwal, S., Snavely, N., Seitz, S.M., Szeliski, R.: Bundle adjustment in the large. In European conference on computer vision, pp. 29–42. Springer, Berlin (2010)

  3. Bas, G., Stoev, L., Durakbasa, N.M.: Assessment of the production quality in machining by integrating a system of high precision measurement. Proc. Eng. 100, 1616–1624 (2015)

    Article  Google Scholar 

  4. Bellekens, B., Spruyt, V., Berkvens, R., Penne, R., Weyn, M.: A benchmark survey of rigid 3d point cloud registration algorithms. Int. J. Adv. Intell. Syst 8, 118–127 (2015)

    Google Scholar 

  5. Bergström, P., Fergusson, M., Sjödahl, M.: Virtual projective shape matching in targetless cad-based close-range photogrammetry for efficient estimation of specific deviations. Opt. Eng. 57(5), 053110 (2018)

    Article  Google Scholar 

  6. Bian, J., Lin, W.Y., Matsushita, Y., Yeung, S.K., Nguyen, T.D., Cheng, M.M.: Gms: Grid-based motion statistics for fast, ultra-robust feature correspondence. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4181–4190 (2017)

  7. Birdal, T., Bala, E., Eren, T., Ilic, S.: Online inspection of 3d parts via a locally overlapping camera network. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1–10. IEEE (2016)

  8. Brandner, M.: Bayesian uncertainty evaluation in vision-based metrology. Vision Sensors and Edge Detection (1993)

  9. Brenner, C., Böhm, J., Gühring, J.: Cad-based object recognition for a sensor/actor measurement robot. Int. Arch. Photogramme. Remote Sens. 32, 209–216 (1998)

    Google Scholar 

  10. Byröd, M., Åström, K.: Conjugate gradient bundle adjustment. In Proceedings of the ECCV, pp. 114–127. Springer, Berlin (2010)

  11. Chen, Z., Pears, N., Liang, B.: A method of visual metrology from uncalibrated images. Pattern Recogn. Lett. 27(13), 1447–1456 (2006)

    Article  Google Scholar 

  12. Criminisi, A.: Single-view metrology: Algorithms and applications. In Joint Pattern Recognition Symposium, pp. 224–239. Springer, Berlin (2002)

  13. Criminisi, A.: Accurate visual metrology from single and multiple uncalibrated images. Springer, Berlin (2012)

    MATH  Google Scholar 

  14. Debevec, P.E., Taylor, C.J., Malik, J.: Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pp. 11–20 (1996)

  15. Delaunoy, A., Pollefeys, M.: Photometric bundle adjustment for dense multi-view 3d modeling. In Proceedings of the CVPR, pp. 1486–1493 (2014)

  16. Deng, J., Trigeorgis, G., Zhou, Y., Zafeiriou, S.: Joint multi-view face alignment in the wild. IEEE Trans. Image Process. 28(7), 3636–3648 (2019)

    Article  MathSciNet  Google Scholar 

  17. Fua, P.: Using model-driven bundle-adjustment to model heads from raw video sequences. In: Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 1, pp. 46–53. IEEE (1999)

  18. Hartley, R., Zisserman, A.: Multiple view geometry in computer vision. Cambridge University Press, Cambridge (2003)

    MATH  Google Scholar 

  19. van den Heuvel, F.: Trends in cad-based photogrammetric measurement. In International Archives of Photogrammetry and Remote Sensing. CUMINCAD (2000)

  20. Hinterstoisser, S., Holzer, S., Cagniart, C., Ilic, S., Konolige, K., Navab, N., Lepetit, V.: Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes. In 2011 international conference on computer vision, pp. 858–865. IEEE (2011)

  21. Hinterstoisser, S., Lepetit, V., Ilic, S., Holzer, S., Bradski, G., Konolige, K., Navab, N.: Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes. In Asian conference on computer vision, pp. 548–562. Springer, Berlin (2012)

  22. Huber, P.J.: Robust estimation of a location parameter. In Breakthroughs in statistics, pp. 492–518. Springer, Berlin (1992)

  23. Liang, B., Chen, Z., Pears, N.: Uncalibrated two-view metrology. In Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004., vol. 1, pp. 96–99. IEEE (2004)

  24. Lim, J.J., Pirsiavash, H., Torralba, A.: Parsing ikea objects: Fine pose estimation. In Proceedings ICCV, pp. 2992–2999 (2013)

  25. Lin, C.H., Wang, O., Russell, B.C., Shechtman, E., Kim, V.G., Fisher, M., Lucey, S.: Photometric mesh optimization for video-aligned 3d object reconstruction. In Proceedings of the CVPR, pp. 969–978 (2019)

  26. Long, C., Zhu, J., Yi, W.: Portable visual metrology without traditional self-calibration measurement model. Measurement 90, 424–437 (2016)

    Article  Google Scholar 

  27. Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision 60(2), 91–110 (2004)

    Article  Google Scholar 

  28. McMurtry, D.R.: Coordinate measuring machine (1982). US Patent 4,333,238

  29. Oakland, R.J., Oakland, J.S.: Statistical process control. Routledge, London (2018)

    Book  Google Scholar 

  30. Olson, E.: AprilTag: A robust and flexible visual fiducial system. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pp. 3400–3407. IEEE (2011)

  31. Pilu, M.: A direct method for stereo correspondence based on singular value decomposition. In Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 261–266. IEEE (1997)

  32. Porikli, F., Divakaran, A.: Multi-camera calibration, object tracking and query generation. In 2003 International Conference on Multimedia and Expo. ICME’03. Proceedings (Cat. No. 03TH8698), vol. 1, pp. I–653. IEEE (2003)

  33. Ren, Z., Liao, J., Cai, L.: Three-dimensional measurement of small mechanical parts under a complicated background based on stereo vision. Appl. Opt. 49(10), 1789–1801 (2010)

    Article  Google Scholar 

  34. Salas-Moreno, R.F., Newcombe, R.A., Strasdat, H., Kelly, P.H., Davison, A.J.: Slam++: Simultaneous localisation and mapping at the level of objects. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1352–1359 (2013)

  35. Salaün, Y., Marlet, R., Monasse, P.: Robust and accurate line-and/or point-based pose estimation without manhattan assumptions. In European Conference on Computer Vision, pp. 801–818. Springer, Berlin (2016)

  36. Schönberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In Conference on Computer Vision and Pattern Recognition (CVPR) (2016)

  37. Schönberger, J.L., Zheng, E., Frahm, J.M., Pollefeys, M.: Pixelwise view selection for unstructured multi-view stereo. In Proceedings of the ECCV, pp. 501–518. Springer, Berlin (2016)

  38. Schops, T., Schonberger, J.L., Galliani, S., Sattler, T., Schindler, K., Pollefeys, M., Geiger, A.: A multi-view stereo benchmark with high-resolution images and multi-camera videos. In: Proc. CVPR, pp. 3260–3269 (2017)

  39. Schwenke, H., Siebert, B., Wäldele, F., Kunzmann, H.: Assessment of uncertainties in dimensional metrology by monte carlo simulation: proposal of a modular and visual software. CIRP Ann. 49(1), 395–398 (2000)

    Article  Google Scholar 

  40. Scott, W.R., Roth, G., Rivest, J.F.: View planning for automated three-dimensional object reconstruction and inspection. ACM Comput. Surv. (CSUR) 35(1), 64–96 (2003)

    Article  Google Scholar 

  41. Shan, Y., Liu, Z., Zhang, Z.: Model-based bundle adjustment with application to face modeling. In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, vol. 2, pp. 644–651. IEEE (2001)

  42. Sorkine-Hornung, O., Rabinovich, M.: Least-squares rigid motion using svd. Computing 1(1), 1–5 (2017)

    Google Scholar 

  43. Su, Y., Ai, H., Lao, S.: Multi-view face alignment using 3d shape model for view estimation. In International Conference on Biometrics, pp. 179–188. Springer, Berlin (2009)

  44. Sun, J., Zheng, N.N., Shum, H.Y.: Stereo matching using belief propagation. IEEE Trans. Pattern Anal. Mach. Intell. 25(7), 787–800 (2003)

    Article  Google Scholar 

  45. Szeliski, R., Torr, P.H.: Geometrically constrained structure from motion: Points on planes. In European Workshop on 3D Structure from Multiple Images of Large-Scale Environments, pp. 171–186. Springer, Berlin (1998)

  46. Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W.: Bundle adjustment–a modern synthesis. In International workshop on vision algorithms, pp. 298–372. Springer, Berlin (1999)

  47. Wolfson, H.J.: Model-based object recognition by geometric hashing. In European conference on computer vision, pp. 526–536. Springer, Berlin (1990)

  48. Wulf, O., Wagner, B.: Fast 3d scanning methods for laser measurement systems. In International conference on control systems and computer science (CSCS14), pp. 2–5. Citeseer (2003)

  49. Xia, R., Su, R., Zhao, J., Chen, Y., Fu, S., Tao, L., Xia, Z.: An accurate and robust method for the measurement of circular holes based on binocular vision. Meas. Sci. Technol. 31(2), 025006 (2019)

    Article  Google Scholar 

  50. Zach, C.: Robust bundle adjustment revisited. In Proceedings of the ECCV, pp. 772–787. Springer, Berlin (2014)

  51. Zhou, Q.Y., Koltun, V.: Color map optimization for 3d reconstruction with consumer depth cameras. ACM Trans. Graph. (SIGGRAPH) 33(4), 1–10 (2014)

    Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Key Research and Development Program of China (2018AAA-0102200) and the National Natural Science Foundation of China (61572507, 61532003, 61622212, 61902419).

Author information

Authors and Affiliations

  1. National University of Defense Technology, Changsha, China

    Yang Yu, Chengjie Niu, Jun Li & Kai Xu

Authors
  1. Yang Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chengjie Niu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kai Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Jun Li or Kai Xu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Yang Yu and Chengjie Niu are joint first authors.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yu, Y., Niu, C., Li, J. et al. Multi-view 2D–3D alignment with hybrid bundle adjustment for visual metrology. Vis Comput 38, 1483–1494 (2022). https://doi.org/10.1007/s00371-021-02082-w

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-021-02082-w

Keywords

Search

Navigation