Skip to main content
SpringerLink
Log in

Extraction of 3D freeform surfaces as visual landmarks for real-time tracking

  • Special Issue
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

This work presents a system for the generation of a free-form surface model from video sequences. Although any single centered camera can be applied in the proposed system the approach is demonstrated using fish-eye lenses because of their good properties for tracking. The system is designed to function automatically and to be flexible with respect to size and shape of the reconstructed scene. To minimize geometric assumptions a statistic fusion of dense depth maps is utilized. Special attention is payed to the necessary rectification of the spherical images and the resulting iso-disparity surfaces, which can be exploited in the fusion approach. Before dense depth estimation can be performed the cameras’ pose parameters are extracted by means of a structure-from-motion (SfM) scheme. In this respect automation of the system is achieved by thorough decision model based on robust statistics and error propagation of projective measurement uncertainties. This leads to a scene-independent set of only a few parameters. All system components are formulated in a general way, making it possible to cope with any single centered projection model, in particular with spherical cameras. In using wide field-of-view cameras the presented system is able to reliably retrieve poses and consistently reconstruct large scenes. A textured triangle mesh constructed on basis of the scene’s reconstructed depth, makes the system’s results suitable to function as reference models in a GPU driven analysis-by-synthesis framework for real-time tracking.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. [53] is solving for a polynomial function \(f:{{\mathbb{R}}} \to {{\mathbb{R}}}\) which is used to add a third component to a normalized point yielding an optical ray in Euclidean coordinates. The angle Θ between the optical axis and this optical ray can then be calculated by means of \(\Uptheta(r)=\arccos(f(r)/\sqrt{r^2+1}).\)

  2. for the presented reconstruction system it was found that the average projection error of several hundred 3D points is usually in the range of up to one pixel, therefore λ = 1.0 is chosen.

  3. The expression epipolar curve is chosen because the spherical projection of an epipolar plane and the following transformation into a planar image does not generally produce a straight line.

  4. In this work X 0 = tan(75°) was chosen.

  5. Due to matching constrains this does not have to be the best similarity in the search range.

References

  1. Adelson, E.H., Bergen, J.: The plenoptic function and the elements of early vision. In: Landy, M., Movshon, J.A. (eds.) Computation Models of Visual Processing, pp. 3–20. MIT Press, Cambridge (1991)

  2. Arican, Z., Frossard, P.: Dense disparity estimation from omnidirectional images. In: Proceedings of the 2007 IEEE International Conference on Advanced Video and Signal based Surveillance (2007)

  3. Ayache, N., Hansen, C.: Rectification of images for binocular and trinocular stereovision. In: Proceedings International Conference on Pattern Recognition, pp. 11–16 (1988)

  4. Badri, J., Tilmant, C., Lavest, J.M., Pham, Q.C., Sayd, P.: Camera-to-camera mapping for hybrid pan-tilt-zoom sensors calibration, pp. 132–141 (2007)

  5. Baker, S., Matthews, I.: Lucas-Kanade 20 Years On: A Unifying Framework. Int. J. Comput. Vis. 56(3), 221–255 (2004)

    Article  Google Scholar 

  6. Peter, B., Sven, F., Wolfgang, S.: The Wägele: a mobile platform for acquisition of 3D models of indoor outdoor environments. In: 9th Tübingen Perception Conference (TWK 2006), Tübingen (2006)

  7. Birchfield, S., Tomasi, C.: Depth discontinuities by pixel-to-pixel stereo. Int. J. Comput. Vis. (1999)

  8. Beder, C., Steffen, R.: Determining an initial image pair for fixing the scale of a 3D reconstruction from an image sequence. In: Franke, K., Müller, K-R., Nickolay, B., Schäfer, R. (eds.) Pattern Recognition, number 4174 in LNCS, pp. 657–666. Springer, Heidelberg (2006)

  9. Cornelius, H., Sara, R., Martinec, D., Pajdla, T., Chum, O., Matas, J.: Towards complete free-form reconstruction of complex 3D scenes from an unordered set of uncalibrated images. In: Statistical Methods in Video Processing, ECCV 2004 Workshop, pp. 1–12. Springer, Heidelberg (2004)

  10. Falkenhagen, L.: Hierarchical block-based disparity estimation considering neighbourhood constraints. In: International Workshop on SNHC and 3D Imaging. Rhodes, Greece (1997)

  11. Fischler, M., Bolles, R.: RANdom SAmpling Consensus: a paradigm for model fitting with application to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)

    Article  MathSciNet  Google Scholar 

  12. Fleck, M.: Perspective projection: the wrong imaging model. Technical Report TR 95-01, Computer Science, University of Iowa (1995)

  13. Förstner, W.: Uncertainty and projective geometry. In: Handbook of Computational Geometry for Pattern Recognition, Computer Vision, Neurocomputing and Robotics, pp. 493–534. Springer, Heidelberg (2005)

  14. Fusiello, A.: Improving feature tracking with robust statistics. In: Pattern Analysis and Application. Springer, London 2:312–320 (1999)

  15. Geyer, C., Daniilidis, K.: Catadioptric projective geometry. Int. J. Comput. Vis. 43, 223–243 (2001)

    Article  Google Scholar 

  16. Geyer C., Daniilidis, K.: (2003) Conformal rectification of omnidirectional stereo Pairs. In: Proceedings of Omnivis 2003: Workshop on Omnidirectional Vision and Camera Networks

  17. Gonzalez-Barbosa, J-J., Lacroix, S.: (2005) Fast dense panoramic stereovision. In: Proceedings of the 2005 IEEE International Conference on Robotics and Automation, pp. 1210–1215

  18. Hampel, F.R., Ronchetti, E.M., Rousseeuw, P.J., Stahel, W.A.: Robust statistics: the approach based on influence functions (Wiley Series in Probability and Statistics), revised edition. Wiley, New York (2005)

  19. Hartley, R.: Theory and practice of projective rectification. Int. J. Comput. Vis. 35(2), 115–127 (1999)

    Article  MathSciNet  Google Scholar 

  20. Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000)

  21. Hartley, R., Zissermann, A.: Multiple View Geometry in Computer Vision, second edition. Cambridge University Press, Cambridge (2004)

  22. Aloimonos, Y., Neumann, J., Fermuller, C.: Eyes from eyes: new cameras for structure from motion. In: In IEEE Workshop on Omnidirectional Vision, pp. 19–26 (2002)

  23. McGlone, J.C.: Manual of Photogrammetry, 5th edition. ASPRS (2004)

  24. Julier, S.J., Uhlmann, J.K., Durrant-Whyte, H.F.: New Approach for Filtering Nonlinear Systems. In: Proceedings of the American Control Conference (ACC), pp. 1628–1632. Seattle WA, USA (1995)

  25. Julier, S.J., Uhlmann, J.K.: Reduced sigma point filters for the propagation of means and covariances through nonlinear transformations. In: Proceedings of the IEEE American Control Conference, pp. 887–892, Anchorage AK, USA, 8–10 May (2002). IEEE (2002)

  26. Julier, S.J., Uhlmann, J.K.: The scaled unscented transformation. In: Proceedings of the IEEE American Control Conference, pp. 4555–4559. Anchorage AK, USA, 8–10 May (2002). IEEE (2002)

  27. Kenichi K.: Statistical optimization for geometric computation: theory and practice. Dover, New York (2005)

  28. Kang, S.B., Szeliski, R.: 3-D scene data recovery using omnidirectional multibaseline stereo. In: CVPR ’96: Proceedings of the 1996 Conference on Computer Vision and Pattern Recognition (CVPR ’96), pp. 364. IEEE Computer Society, Washington, DC, USA (1996)

  29. Koch, R., Koeser, K., Streckel, B., Evers-Senne, J-F.: Markerless Image-based 3D tracking for real-time augmented reality applications. In: WIAMIS 2005. Montreux, Switzerland (2005)

  30. Koeser, K., Bartczak, B., Koch, R.: Drift-free pose estimation with hemispherical cameras. In: Proceedings of CVMP 2006, pp. 20–28. London (2006)

  31. Koeser, K., Bartczak, B., Koch, R.: Robust GPU-assisted camera tracking using free-form surface models. J Real-Time Image Proc (submitted to same issue) (2007)

  32. Lepetit, V., Fua, P.: Monocular model-based 3D tracking of rigid objects: a survey. Found. Trends Comput. Graph. Vis. 1(1):1–89 (2005)

    Article  Google Scholar 

  33. Lepetit, V., Lagger, P., Fua, P.: Randomized trees for real-time keypoint recognition. In: Conference on Computer Vision and Pattern Recognition 2, 775–781 (2005)

  34. Li, S.: Real-time spherical stereo. In: ICPR ’06: Proceedings of the 18th International Conference on Pattern Recognition, pp. 1046–1049. IEEE Computer Society, Washington, DC, USA (2006)

  35. Loop, C., Zhang, Z.: Computing rectifying homographies for stereo vision. Proc. Comput. Vis. Pattern Recogn. 125–131 (1999)

  36. Lucas, B.D., Kanade, T.: An iterative image registration technique with an application to stereo vision. In: IJCAI81, pp. 674–679 (1999)

  37. Van Meerbergen, G., Vergauwen, M., Pollefeys, M., Van Gool, L.: A hierarchical symmetric stereo algorithm using dynamic programming. Int. J. Comput. Vis. 47(1–3), 275–285 (2002)

    Article  MATH  Google Scholar 

  38. Meguro, J., Takiguchi, J., Amano, Y., Hashizume, T.: 3D reconstruction using multibaseline omnidirectional motion stereo based on gps/dead-reckoning compound navigation system. Int. J. Robot. Res. 26(6), 625–636 (2007)

    Article  Google Scholar 

  39. Micusik, B.: Two-view geometry of omnidirectional cameras. PhD thesis, Czech Technical University in Prague, Technicka 2, 166 27 Prague 6, Czech Republic (2004)

  40. Micusik, B., Pajdla, P.: Estimation of omnidirectional camera model from epipolar geometry. In: Proceedings of Conference on Computer Vision and Pattern Recognition (CVPR’03), Vol. 1, p. 485. IEEE Computer Society (2003)

  41. Micusik, B., Pajdla, T.: Structure from Motion with Wide Circular Field of View Cameras. IEEE Trans. Pattern Anal. Mach. Intell. 28(7), 1135–1149 (2006)

    Article  Google Scholar 

  42. Mikhail, E.M., Ackermann, F.E.: Observations and least squares. IEP, New York (1976)

    Google Scholar 

  43. Mordohai, P., Frahm, J-M., Akbarzadeh, A., Clipp, B., Engels, C., Gallup, D., Merrell, P., Salmi, C., Sinha, S., Talton, B., Wang, L., Yang, Q., StewTnius, H., Towles, H., Welch, G., Yang, R., Pollefeys, M., NistTr, D.: Real-time video-based reconstruction of urban environments. In: 3D-ARCH 2007: 3D Virtual Reconstruction and Visualization of Complex Architectures, Zurich, Switzerland (2007)

  44. Nistér, D.: Automatic dense reconstruction from uncalibrated video sequences. PhD thesis, Kungl Tekniska Hogskolen (2001)

  45. Nistér, D.: Preemptive RANSAC for live structure and motion estimation. In: Proceedings of the ICCV, pp. 199–206 (2003)

  46. Nistér, D.: An efficient solution to the five-point relative pose problem. TPAMI (2004)

  47. Pajarola, R., Meng, Y., Sainz, M.: Fast depth-image meshing and warping. Technical Report UCI-ECE-TR-02-02, The Henry Samueli School of Engineering, University of California, Irvine (2002)

  48. Perwass, C., Sommer, G.: The inversion camera model. In: 28. Symposium für Mustererkennung, DAGM 2006, Berlin, 12–14.09.2006, number 4174 in LNCS, pp. 647–656. Springer, Heidelberg (2006)

  49. Pollefeys, M., Koch, R., Van Gool, L.: A simple and efficient rectification method for general motion. In: ICCV (1), pp. 496–501 (1999)

  50. Pollefeys, M., Sinha, S.: Iso-disparity Surfaces for general Stereo Configurations. In: Pajdla, T., Matas, J. (eds.) Computer vision - ECCV 2004, Vol. 3023, pp. 509–520. Springer, Heidelberg (2004)

  51. Pollefeys, M., van Gool, L., Vergauwen, M., Verbiest, F., Cornelis, K., Tops, J., Reinhard, K.: Visual modeling with a hand-held camera. Int. J. Comput. Vis. 59(3), 207–232 (2004)

    Article  Google Scholar 

  52. Roy, S., Meunier, J., Cox, I.: Cylindrical rectification to minimize epipolar distortion (1997)

  53. Scaramuzza, D., Martinelli, A., Siegwart, R.: A flexible technique for accurate omnidirectional camera calibration and structure from motion. In: Proceedings of IEEE International Conference of Vision Systems. IEEE (2006)

  54. Scharstein, D., Szeliski, R., Zabih, R.: A Taxonomy and evaluation of dense two-frame stereo correspondence algorithms. In: Proceedings of IEEE Workshop on Stereo and Multi-BaselineVision. Kauai, HI (2001)

  55. Schroeder, W.J., Zarge, J.A., Lorensen, W.E.: Decimation of triangle meshes. Comput. Graph. 26(2), 65–70 (1992)

    Article  Google Scholar 

  56. Seitz, S., Curless, B., Diebel, J., Scharstein, D., Szeliski, R.: A comparison and evaluation of multi-view stereo reconstruction algorithms (2006)

  57. Shi, J., Tomasi, C.: Good features to track. In: Conference on computer vision and pattern recognition. Seattle, IEEE, pp. 593–600 (1994)

  58. Skrypnyk, I., Lowe, D.G.: Scene modelling, recognition and tracking with invariant image features. In: IEEE and ACM International Symposium on Mixed and Augmented Reality, pp. 110–119 (2004)

  59. Streckel, B., Koch, R.: Lens model selection for visual tracking. In: Lecture Notes in Computer Science 3663 (DAGM 2005), Vienna, Austria (2005)

  60. Takiguchi, J.-I., Yoshida, M., Takeya, A., Eino, J-I., Hashizume, T.: High precision range estimation from an omnidirectional stereo system. In: Jacques, G.V. (eds.) Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems 1, 263–268 (2002)

  61. Thomas, G.A., Jin, J., Niblett, T., Urquhart, C.: A versatile camera position measurement system for virtual reality. In: Proceedings of International Broadcasting Convention, pp. 284–289 (1997)

  62. Triggs, B., McLauchlan, P., Hartley, R., Fitzgibbon, A.: Bundle adjustment—a modern synthesis. In: Triggs, B, Zisserman, A, Szeliski, R (eds.) Vision Algorithms: Theory and Practice, Vol. 1883 of Lecture Notes in Computer Science. Springer, Heidelberg, pp. 298–372 (2000)

  63. Zhang, Z.Y.: Parameter-estimation techniques: a tutorial with application to conic fitting. IVC 15(1), 59–76 (1997)

    Article  Google Scholar 

Download references

Acknowledgments

This work has been partially funded by the European Union in project MATRIS IST-002013.

Author information

Authors and Affiliations

  1. Institute of Computer Science, Christian-Albrechts-Universität Kiel, Hermann-Rodewald-Strasse 3, 24098, Kiel, Germany

    Bogumil Bartczak, Kevin Koeser, Felix Woelk & Reinhard Koch

Authors
  1. Bogumil Bartczak
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kevin Koeser
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Felix Woelk
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Reinhard Koch
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bogumil Bartczak.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bartczak, B., Koeser, K., Woelk, F. et al. Extraction of 3D freeform surfaces as visual landmarks for real-time tracking. J Real-Time Image Proc 2, 81–101 (2007). https://doi.org/10.1007/s11554-007-0042-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-007-0042-0

Keywords

Search

Navigation