Multi perspective panoramic imaging

https://doi.org/10.1016/j.imavis.2008.06.008Get rights and content

Abstract

Panoramic images have only been feasible if all contributing image patches share a common center of projection. Then, they can be consolidated into a single image using perspective transforms. In contrast to that, we propose a novel non-linear warping scheme which allows the merging of multi-perspective images, thus taking advantage of scattered cameras. Therefore, a polygonal cut is defined in two source images to be merged. Usually, the layout of the cuts does not allow a user to stitch both images together naively. Thus, two convex combinations of a warped and a canonic coordinate system are applied so that both source images fit together at the cutting edge while the inevitable distortion decreases towards the borders of the image to obtain a natural appearance.

Introduction

Image sensors have recently become almost as inexpensive and available as scalar sensors which are used for temperature or light measurements. The Stanford Multi-Camera Array project is an early example for the simultaneous usage of more than 100 inexpensive CCD cameras [1], [2], [3], [4]. Other projects are currently emerging in the field of sensor networks. The ESB sensor node platform by the FU-Berlin is one instance of a small, wireless and video-enabled device [5].

Other than scalar values which can be displayed on a virtual map or which can simply be aggregated, it is not obvious how to display a massive amount of (possibly uncalibrated) images, particularly in a way that makes sense for a human observer.

Consolidating all images into a single one could be a possible solution. Similar attempts have been made in the field of panoramic images, in which a series of pictures are stitched to one another to produce a continuous view. For a long time, panoramic images have been considered feasible only if all images have the same focal point, respectively, if the camera does not alter its location. In this paper, we devise a novel method for creating panoramic views from images with varying focal points. The specific problems which arise here are described in the following section, along with descriptions of prior attempts to create panoramic images from movies. Section 3 suggests a basic warping scheme as a solution. In Section 4, we identify some shortcomings of the basic scheme which are solved by an extension. The evaluation in Section 5 shows some examples and analyzes the new degree of freedom, but also fundamental limitations, of multi-perspective still imaging and video in general. The outlook in Section 6 sketches future improvements to reduce the amount of human interaction.

Section snippets

Related work

In the context of our paper, we will distinguish between monoperspective panoramic images and multi-perspective imaging.

Warping for panoramic stitching

In this section, we will show that panoramic images are possible, even if the focal point of the camera changes significantly. Of course, the resulting image will imply several changes in perspective and, unlike existing approaches, these changes will by no means be continuous. But, as we will see, this does not necessarily result in an unnatural output.

Fig. 4 shows a building from two different perspectives with a certain overlap. In conventional panoramic image generation, semi-transparent

Extended warping

A typical result of the above-described simple warping scheme can be seen in Fig. 8. We identified four different kinds of artifacts, namely expansions, contractions, undefined areas, and reflections (see Fig. 9).

The emergence of expansions and contractions are most obvious. If two neighboring abscissas exhibit a large opening angle in the source image, but a smaller angle in the target image, this means that a large image patch in the source area will be squeezed into a small patch within the

Evaluation and limitations

Fig. 12, Fig. 13 show examples of panoramic images with changing perspectives. Intuitively speaking, warping and stitching images work best if the object being shown can be unfolded, theoretically. This is often true for buildings. In particular, stitching works without problems if the camera follows a straight path. The reason is that succeeding images can be concatenated to existing ones if the above-mentioned trajectory exists and if it is not occluded from two neighboring viewpoints. We can

Conclusion and outlook

A new approach for producing panoramic images from photos with varying centers of projection is proposed. A trajectory has to be found in neighboring images which serves as a cut. A vertical warping scheme distorts neighboring images such that they fit together. Another horizontal warping process makes the distorted parts of the image near the cut converge against the natural image near the left and right borders of the panorama. Finally, artifacts originating from the warping and from the lens

Acknowledgements

We want to thanks Mr. Allen Wrenn and Mrs. Weyerer for their suggestions and improvements which contributed significantly to the quality of this work.

References (27)

  • B. Wilburn, N. Joshi, V. Vaish, E. Talvala, E. Antunez, A. Barth, A. Adams, M. Levoy, M. Horowitz, High performance...
  • B. Wilburn, N. Joshi, V. Vaish, M. Levoy, M. Horowitz, High speed video using a dense camera array, in: Proc. of the...
  • V. Vaish, B. Wilburn, N. Joshi, M. Levoy, Using plane and parallax for calibrating dense camera arrays, in: Proc. of...
  • B. Wilburn, M. Smulski, H.K. Lee, M. Horowitz, The light field video camera, in: Proc. of Media Processors 2002, SPIE...
  • E. Kappe, A. Liers, H. Ritter, J. Schiller, Low-power image transmission in wireless sensor networks using scatterweb...
  • G.N. Barnard

    Barnard’s Photographic Views of the Sherman Campaign

    (1866)
  • D. Farin

    Automatic video segmentation employing object/camera modeling techniques

    (2005)
  • R. Szeliski

    Video mosaics for virtual environments

    IEEE Computer Graphics and Applications

    (1996)
  • E.A. Maxwell

    Methods of Plane Projective Geometry Based on the Use of General Homogeneous Coordinates

    (1946)
  • E.A. Maxwell

    General Homogeneous Coordinates in Space of Three Dimensions

    (1951)
  • L.G. Roberts, Homogeneous matrix representations and manipulations of n-dimensional constructs, Tech. Rep. Document MS...
  • R. Szeliski, H.-Y. Shum, Creating full view panoramic image mosaics and environment maps, in: Proc. of the ACM...
  • R. Szeliski, Image mosaicing for tele-reality applications, in: Proc. of the Second IEEE Workshop on Applications of...
  • Cited by (17)

    View all citing articles on Scopus
    View full text