Skip to main content
SpringerLink
Log in

MultiCol Bundle Adjustment: A Generic Method for Pose Estimation, Simultaneous Self-Calibration and Reconstruction for Arbitrary Multi-Camera Systems

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

In this paper, we present a generic, modular bundle adjustment method for pose estimation, simultaneous self-calibration and reconstruction for multi-camera systems. In contrast to other approaches that use bearing vectors (camera rays) as observations, we extend the common collinearity equations with a general camera model and include the relative orientation of each camera w.r.t to the fixed multi-camera system frame yielding the extended collinearity equations that directly express all image observations as functions of all unknowns. Hence, we can either calibrate the camera system, the cameras, reconstruct the observed scene, and/or simply estimate the pose of the system by including the corresponding parameter block into the Jacobian matrix. Apart from evaluating the implementation with comprehensive simulations, we benchmark our method against recently published methods for pose estimation and bundle adjustment for multi-camera systems. Finally, all methods are evaluated using a 6 degree of freedom ground truth data set, that was recorded with a lasertracker.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  • Agarwal, S., Furukawa, Y., Snavely, N., Simon, I., Curless, B., Seitz, S. M., et al. (2011). Building Rome in a day. Communications of the ACM, 54(10), 105–112.

    Article  Google Scholar 

  • Agarwal, S., & Mierle, K. (2012). Ceres solver: Tutorial & reference. Mountain View: Google Inc.

    Google Scholar 

  • Baker, P., & Aloimonos, Y. (2000). Complete calibration of a multi-camera network. In Proceedings IEEE Workshop on Omnidirectional Vision (pp. 134–141). IEEE.

  • Barreto, J., & Daniilidis, K. (2004). Wide area multiple camera calibration and estimation of radial distortion. In Omnivis-2004, ECCV-2004 Workshop.

  • Blanco-Claraco, J. L., Moreno-Dueñas, F. Á., & González-Jiménez, J. (2014). The Málaga urban dataset: High-rate stereo and LiDAR in a realistic urban scenario. The International Journal of Robotics Research, 33(2), 207–214.

    Article  Google Scholar 

  • Bouguet, J. (2014). Camera calibration toolbox for Matlab. Retrieved February, 2014, from http://www.vision.caltech.edu/bouguetj/calib_doc/index.html

  • Brown, D. C. (1966). Decentering distortion of lenses. Photometric Engineering, 32(3), 444–462.

    Google Scholar 

  • Brückner, M., Bajramovic, F., & Denzler, J. (2014). Intrinsic and extrinsic active self-calibration of multi-camera systems. Machine Vision and Applications, 25(2), 389–403.

    Article  Google Scholar 

  • Carrera, G., Angeli, A., & Davison, A.J. (2011). SLAM-based automatic extrinsic calibration of a multi-camera rig. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 2652–2659). IEEE.

  • Ceriani, S., Fontana, G., Giusti, A., Marzorati, D., Matteucci, M., Migliore, D., et al. (2009). Rawseeds ground truth collection systems for indoor self-localization and mapping. Autonomous Robots, 27(4), 353–371.

    Article  Google Scholar 

  • Clipp, B., Kim, J.H., Frahm, J.M., Pollefeys, M., & Hartley, R. (2008). Robust 6dof motion estimation for non-overlapping, multi-camera systems. In IEEE Workshop on Applications of Computer Vision. WACV 2008 (pp. 1–8). IEEE.

  • Davison, A.J., Cid, Y.G., Kita, N. (2004). Real-time 3d SLAM with wide-angle vision. In Proceedings of the IFAC/EURON Symposium on Intelligent Autonomous Vehicles.

  • Duane, C. B. (1971). Close-range camera calibration. Photogrammetric Engineering, 37, 855–866.

    Google Scholar 

  • Ess, A., Neubeck, A., Van Gool, L.J. (2007). Generalised linear pose estimation. In BMVC (pp. 1–10).

  • Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6), 381–395.

    Article  MathSciNet  Google Scholar 

  • Förstner, W. (2010). Minimal representations for uncertainty and estimation in projective spaces. In Computer Vision–ACCV 2010 (pp. 619–632). Springer, Berlin.

  • Frahm, J.M., Köser, K., & Koch, R. (2004). Pose estimation for multi-camera systems. In Pattern Recognition (pp. 286–293). Springer, Berlin.

  • Frank, O., Katz, R., Tisse, C.L., & Durrant-Whyte, H.F. (2007). Camera calibration for miniature, low-cost, wide-angle imaging systems. In BMVC (pp. 1–10). Citeseer.

  • Gao, X. S., Hou, X. R., Tang, J., & Cheng, H. F. (2003). Complete solution classification for the perspective-three-point problem. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(8), 930–943.

    Article  Google Scholar 

  • Geiger, A., Lenz, P., & Urtasun, R. (2012). Are we ready for autonomous driving? The KITTI vision benchmark suite. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012 (pp. 3354–3361). IEEE

  • Geyer, C., Daniilidis, K. (2000). A unifying theory for central panoramic systems and practical implications. In Computer VisionECCV 2000 (pp. 445–461). Springer, Berlin.

  • Hartley, R., & Zisserman, A. (2003). Multiple view geometry in computer vision. Cambridge: Cambridge university press.

    MATH  Google Scholar 

  • Hesch, J.A., Roumeliotis, S.I. (2011). A direct least-squares (DLS) method for PnP. In IEEE International Conference on Computer Vision (ICCV) (pp. 383–390). IEEE.

  • Kannala, J., & Brandt, S. S. (2006). A generic camera model and calibration method for conventional, wide-angle, and fish-eye lenses. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(8), 1335–1340.

    Article  Google Scholar 

  • Kneip, L., & Furgale, P. (2014). OpenGV: A unified and generalized approach to real-time calibrated geometric vision. InIEEE International Conference on Robotics and Automation (ICRA) (pp. 1–8). IEEE.

  • Kneip, L., Furgale, P., & Siegwart, R. (2013) Using multi-camera systems in robotics: Efficient solutions to the NPnP problem. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 3770–3776). IEEE.

  • Kneip, L., Li, H., & Seo, Y. (2014). UPnP: An optimal O(n) solution to the absolute pose problem with universal applicability. In Computer Vision–ECCV (pp. 127–142). Springer, Berlin.

  • Kneip, L., Scaramuzza, D., & Siegwart, R. (2011). A novel parametrization of the perspective-three-point problem for a direct computation of absolute camera position and orientation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 2969–2976). IEEE.

  • Kumar, R.K., Ilie, A., Frahm. J.M., & Pollefeys, M. (2008). Simple calibration of non-overlapping cameras with a mirror. In IEEE Conference on, Computer Vision and Pattern Recognition. CVPR 2008 (pp. 1–7). IEEE

  • Kummerle, R., Grisetti, G., Strasdat, H., Konolige, K., & Burgard, W. (2011). \(g^2o\): A general framework for graph optimization. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 3607–3613). IEEE

  • Kurillo, G., Li, Z., & Bajcsy, R. (2008). Wide-area external multi-camera calibration using vision graphs and virtual calibration object. In Second ACM/IEEE International Conference on In: Distributed Smart Cameras, ICDSC 2008 (pp. 1–9). IEEE

  • Lepetit, V., Moreno-Noguer, F., & Fua, P. (2009). EPnP: An accurate O(n) solution to the PnP problem. International Journal of Computer Vision, 81(2), 155–166.

    Article  Google Scholar 

  • Li, B., Heng, L., Koser, K., & Pollefeys M. (2013). A multiple-camera system calibration toolbox using a feature descriptor-based calibration pattern. iN IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2013 (pp. 1301–1307). IEEE

  • Lourakis, M. I., & Argyros, A. A. (2009). SBA: A software package for generic sparse bundle adjustment. ACM Transactions on Mathematical Software (TOMS), 36(1), 2.

    Article  MathSciNet  Google Scholar 

  • Ly, D. S., Demonceaux, C., Vasseur, P., & Pégard, C. (2014). Extrinsic calibration of heterogeneous cameras by line images. Machine Vision and Applications, 25(6), 1601–1614.

    Article  Google Scholar 

  • Maas, H. G. (1999). Image sequence based automatic multi-camera system calibration techniques. ISPRS Journal of Photogrammetry and Remote Sensing, 54(5), 352–359.

    Article  Google Scholar 

  • Mei, C., & Rives, P. (2007). Single view point omnidirectional camera calibration from planar grids. In IEEE International Conference on Robotics and Automation

  • Micušık, B. (2004). Two-view geometry of omnidirectional cameras. Ph.D. Thesis, Czech Technical University.

  • Muhle, D., Abraham, S., Heipke, C., & Wiggenhagen, M. (2011). Estimating the mutual orientation in a multi-camera system with a non overlapping field of view. In Photogrammetric Image Analysis (pp. 13–24). Springer, Berlin

  • Puig, L., Bermúdez, J., Sturm, P., & Guerrero, J. (2012). Calibration of omnidirectional cameras in practice: A comparison of methods. Computer Vision and Image Understanding, 116(1), 120–137. doi:10.1016/j.cviu.2011.08.003. http://www.sciencedirect.com/science/article/pii/S1077314211001858.

  • Scaramuzza, D. (2014). OCamCalib: Omnidirectional camera calibration toolbox for matlab. Retrieved February, 2014, from https://sites.google.com/site/scarabotix/ocamcalib-toolbox

  • Scaramuzza, D., Martinelli, A., & Siegwart, R. (2006a). A flexible technique for accurate omnidirectional camera calibration and structure from motion. In IEEE International Conference on Computer Vision Systems, 2006 ICVS’06 (pp. 45–45). IEEE

  • Scaramuzza, D., Martinelli, A., & Siegwart, R. (2006b). A toolbox for easily calibrating omnidirectional cameras. In IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 5695–5701). IEEE

  • Schönbein, M., & Geiger, A. (2014) Omnidirectional 3d reconstruction in augmented manhattan worlds. In International Conference on Intelligent Robots and Systems (IROS).

  • Schneider, J., & Förstner, W. (2013). Bundle adjustment and system calibration with points at infinity for omnidirectional camera systems. Z. f. Photogrammetrie, Fernerkundung und Geoinformation, 4, 309–321. doi:10.1127/1432-8364/2013/0179.

    Article  Google Scholar 

  • Schneider, J., Schindler, F., Läbe, T., & Förstner, W. (2012). Bundle adjustment for multi-camera systems with points at infinity. In XXII ISPRS congress–technical commission III. ISPRS annals of the photogrammetry, remote sensing and spatial information sciences, vol. I-3 (pp. 75–80). Melbourne.

  • Schönbein, M., Straus, T., & Geiger, A. (2014). Calibrating and centering quasi-central catadioptric cameras. In IEEE International Conference on Robotics and Automation (ICRA), 2014 (pp. 4443–4450). IEEE.

  • Schweighofer, G., & Pinz, A. (2008). Globally optimal O(n) solution to the PnP problem for general camera models. In: BMVC (pp. 1–10).

  • Sturm, J., Engelhard, N., Endres, F., Burgard, W., & Cremers, D. (2012). A benchmark for the evaluation of RGB-D SLAM systems. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2012 (pp. 573–580). IEEE.

  • Svoboda, T. (2003). Quick guide to multi-camera self-calibration. ETH, Swiss Federal Institute of Technology, Zurich. Technical Report BiWi-TR-263, http://www.vision.ee.ethz.ch/svoboda/SelfCal.

  • Triggs, B., McLauchlan, P.F., Hartley, R.I., & Fitzgibbon, A.W. (2000). Bundle adjustment-a modern synthesis. In: Vision Algorithms: Theory and Practice (pp. 298–372). Springer, Berlin.

  • Urban, S., Leitloff, J., & Hinz, S. (2015). Improved wide-angle, fisheye and omnidirectional camera calibration. ISPRS Journal of Photogrammetry and Remote Sensing, 108, 72–79.

    Article  Google Scholar 

  • Wu, C. (2011). VisualSFM: A visual structure from motion system (Vol. 9). http://homes.cs.washington.edu/~ccwu/vsfm.

  • Zhang, Z. (1999). Flexible camera calibration by viewing a plane from unknown orientations. In The Proceedings of the Seventh IEEE International Conference on Computer Vision, vol. 1 (pp. 666–673). IEEE.

  • Zheng, Y., Kuang, Y., Sugimoto, S., Astrom, K., & Okutomi, M. (2013). Revisiting the PnP problem: A fast, general and optimal solution. In IEEE International Conference on Computer Vision (ICCV) (pp. 2344–2351). IEEE.

Download references

Acknowledgments

This project was partially funded by the DFG research group FG 1546 “Computer-Aided Collaborative Subway Track Planning in Multi-Scale 3D City and Building Models”.

Author information

Authors and Affiliations

  1. KIT - Karlsruhe Institute of Technology, Institute of Photogrammetry and Remote Sensing, Karlsruhe, Germany

    Steffen Urban, Sven Wursthorn, Jens Leitloff & Stefan Hinz

Authors
  1. Steffen Urban
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sven Wursthorn
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jens Leitloff
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Stefan Hinz
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Steffen Urban.

Additional information

Communicated by Long Quan.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Urban, S., Wursthorn, S., Leitloff, J. et al. MultiCol Bundle Adjustment: A Generic Method for Pose Estimation, Simultaneous Self-Calibration and Reconstruction for Arbitrary Multi-Camera Systems. Int J Comput Vis 121, 234–252 (2017). https://doi.org/10.1007/s11263-016-0935-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-016-0935-0

Keywords

Search

Navigation