Stabilization of panoramic videos from mobile multi-camera platforms

https://doi.org/10.1016/j.imavis.2015.02.002Get rights and content

Highlights

  • Stabilization of panoramic and stabilization of single camera videos are separate problems.

  • Panoramic videos suffer from global and inter-camera vibrations.

  • Blend-masks are useful for dealing with inter-camera vibrations.

  • Our survey suggests that viewers prefer this scheme over prior works.

Abstract

Wide field of view panoramic videos have recently become popular due to the availability of high resolution displays. These panoramic videos are generated by stitching video frames captured from a panoramic video acquisition system, typically comprising of multiple video cameras arranged on a static or mobile platform. A mobile panoramic video acquisition system may suffer from global mechanical vibrations as well as independent inter-camera vibrations resulting in a jittery panoramic video. While existing stabilization schemes generally tackle single-camera vibrations, they do not account for these inter-camera vibrations. In this paper, we propose a video stabilization technique for multi-camera panoramic videos under the consideration that independent jitter may be exhibited by content of each camera. The proposed method comprises of three steps; the first step removes the global jitter in the video by estimating collective motion and subsequently removing the high frequency component from it. The second step removes the independent i.e. local jitter of each camera by estimating motion of each camera content separately. Pixels that are located in the overlapping regions of panoramic video are contributed by neighboring cameras, therefore, the estimated camera motion for these pixels is weighted using the blend masks generated by the stitching process. The final step applies local geometric warping to the stitched frames and removes any residual jitter induced due to parallax. Experimental results prove that proposed scheme performs better than existing panoramic stabilization schemes.

Introduction

Multimedia technology has seen a rapid evolution in recent years in terms of both quality and quantity of the information delivered through multimedia displays. Wide Field of View (FOV) panoramic images and videos provide ultra-high definition content that when displayed on large high definition displays, provide an immersive experience to the viewers. Panoramic videos are generated either by using fish-eye lens or by stitching together synchronized video frames coming from multiple cameras arranged on a rig [1], [2]. The use of stitching-based panorama generation and display systems once limited to geological surveys and surveillance applications [3] has already been extended for entertainment purposes [4], [5], [6]. With decreasing cost of commodity cameras, this trend is expected to be more ubiquitous [7] in the entertainment industry, closely followed by household consumer market in the future. Thus, instead of static panorama acquisition systems [3] that arrange closely coupled sensors in a dedicated closed unit [2], affordable panoramic acquisition systems are emerging that employ commodity cameras mounted on a platform. When set on a mobile platform, such a panoramic video acquisition system may suffer from mechanical vibrations that are global to the camera rig or independent to a particular constituent camera. The global vibrations appear as global jitter in the panoramic video frames while the inter-camera vibrations result in jitter in the spatial region of panoramic video contributed by the particular affected cameras. This results in an unpleasant experience for viewers. This is illustrated conceptually in Fig. 1 which shows a single frame of a panoramic video (‘Man video’) that was formed by stitching frames acquired using three cameras. Please refer to the accompanying video to watch this panorama sequence. The tail of the arrows in Fig. 1 represents the location of a few salient features tracked using Kanade–Lucas–Tomasi (KLT) feature tracker [8], [9]. The yellow, red and green vectors represent the direction of motion of these features at a particular instance in time and are color coded to signify the independent direction of motion for the contents captured from camera 1, camera 2 and camera 3 respectively. Fig. 1 illustrates an example where camera 3 is experiencing a different vibration and hence its motion vectors (green) appear to have a different direction as compared to camera 1 and camera 2. In the panoramic video (Man video), this effect appears as jitter in the region of panoramic video frames that is contributed by camera 3. Since this jitter appears in a sub-section of panoramic frame, we term it as sub-frame jitter. The ‘Man video’ demonstrates this effect for an actual setup and thus supports our motivation for the requirement of a stabilizing scheme particularly aimed at panoramic videos.

The problem of single camera video stabilization has been extensively researched and has reached a certain level of maturity [10], [11], [12], [13], [14]. However, for stabilization of panoramic videos captured from multi-camera platforms, little work has been reported. Furthermore, the existing schemes for such systems [15], [16] do not account for the inter-camera vibrations experienced in mobile platforms. In this paper, we treat stabilization of panoramic multi-camera videos as a distinct problem from that of a single camera video. We achieve this by classifying the effects of these vibrations as global, sub-frame and local jitter, and by proposing a method to deal with each one of these in a systematic fashion. In summary, global stabilization is achieved by estimating 2D motion models using the tracked feature trajectories over the complete panoramic frame. Sub-frame stabilization is achieved by making use of the information available in the blend masks that are generated by the stitching application [1]. Blend masks are the intensity weights that are used to blend together the images acquired from multiple cameras to generate a seamless panorama. Finally, the local stabilization tackles the residual jitter that might appear in parts of the panoramic scene in the video due to differences in the scene depth. To the best of our knowledge, this is the first scheme that acknowledges that panoramic videos need to be stabilized temporally as a sequence of stitched frames, as well as spatially to handle the sub-frame jitter.

This paper is organized as follows. In Section 2, we provide a brief account of predominant approaches for video stabilization. In Section 3, we provide the necessary background for panoramic video stitching process followed by a description of the proposed stabilization scheme. In Section 4, we discuss the comparison of the results of the proposed scheme with that of two recent stabilization schemes [14], [15] for a number of videos. Finally in Section 5, we present our conclusion.

Section snippets

Literature review

Single camera video stabilization problem has been thoroughly addressed in previous works over the last decade. Most video stabilization methods comprise of three main steps: Estimating a 2D motion model between subsequent video frames, computing a smooth motion model that removes the unpleasant jitter and finally applying stabilizing geometric transforms to the video frames [11], [12], [14], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29]. These methods differ in

Proposed methodology

In this section we present a brief overview of panoramic video generation followed by a detailed description of the proposed video stabilization technique.

Results and discussion

To evaluate our proposed scheme for panoramic videos, we developed a custom panoramic video acquisition system that consists of 5 Sony FS100 cameras, mounted on a wheeled metal rig in a mirror based setting as shown in Fig. 2a. Each camera captured 1920 × 1080 resolution frames at 30 frames per second. For stitching the videos, we used the method described in [1] using OpenCV. In the following, we provide a thorough discussion and experimental analysis of our three step stabilization scheme

Conclusions

In this paper we propose a video stabilization scheme specifically for stitched panoramic videos generated using mobile multi-camera rigs. The proposed scheme acknowledges that each camera in the panoramic video acquisition system may exhibit independent motion due to which, parts of the panorama contents coming from a camera may jitter differently from the rest. It was demonstrated that by classifying the vibrations for such systems into three categories: global, sub-frame and local

Acknowledgement

We would like to thank IT R&D program of MKE/ETRI (14ZR1110, HCI based UHD Panorama Technology Development) for their generous funding.

References (40)

  • D. Gledhill et al.

    Panoramic imaging—a review

    Comput. Graph.

    (2003)
  • S. Ertürk

    Real-time digital image stabilization using Kalman filters

    Real-Time Imaging

    (2002)
  • C. Vogler et al.

    Outlier rejection in high-dimensional deformable models

    Image Vis. Comput.

    (2007)
  • M. Brown et al.

    Automatic panoramic image stitching using invariant features

    Int. J. Comput. Vis.

    (2007)
  • A. Majumder et al.

    Immersive teleconferencing: a new algorithm to generate seamless panoramic video imagery

  • D. Kimber et al.

    Flyabout: spatially indexed panoramic video

  • C. Fehn et al.

    Creation of high-resolution video panoramas for sport events

    Int. J. Semant. Comput.

    (2007)
  • K. Pulli et al.

    Real-time computer vision with OpenCV

    Commun. ACM

    (2012)
  • C. Tomasi et al.

    Detection and tracking of point features

    (1991)
  • J. Shi et al.

    Good features to track

  • C. Buehler et al.

    Non-metric image-based rendering for video stabilization

  • H.C. Chang et al.

    A robust and efficient video stabilization algorithm

  • S. Battiato et al.

    SIFT features tracking for video stabilization

  • F. Liu et al.

    Subspace video stabilization

    ACM Trans. Graph.

    (2011)
  • G.R. Yeon et al.

    Robust online digital image stabilization based on point-feature trajectory without accumulative global motion estimation

    IEEE Signal Process Lett.

    (2012)
  • Anwar-ul-Haq et al.

    VSAMS: video stabilization approach for multiple sensors

  • M. Kamali et al.

    Stabilizing omnidirectional videos using 3D structure and spherical image warping

  • S. Erturk

    Digital image stabilization with sub-image phase correlation based global motion estimation

    IEEE Trans. Consum. Electron.

    (2003)
  • S. Wu et al.

    Image based hand tracking via interacting multiple model and probabilistic data association (IMM-PDA) algorithm

  • Cited by (0)

    This paper has been recommended for acceptance by Michael Goesele.

    View full text