Stabilization of panoramic videos from mobile multi-camera platforms☆
Graphical abstract
Introduction
Multimedia technology has seen a rapid evolution in recent years in terms of both quality and quantity of the information delivered through multimedia displays. Wide Field of View (FOV) panoramic images and videos provide ultra-high definition content that when displayed on large high definition displays, provide an immersive experience to the viewers. Panoramic videos are generated either by using fish-eye lens or by stitching together synchronized video frames coming from multiple cameras arranged on a rig [1], [2]. The use of stitching-based panorama generation and display systems once limited to geological surveys and surveillance applications [3] has already been extended for entertainment purposes [4], [5], [6]. With decreasing cost of commodity cameras, this trend is expected to be more ubiquitous [7] in the entertainment industry, closely followed by household consumer market in the future. Thus, instead of static panorama acquisition systems [3] that arrange closely coupled sensors in a dedicated closed unit [2], affordable panoramic acquisition systems are emerging that employ commodity cameras mounted on a platform. When set on a mobile platform, such a panoramic video acquisition system may suffer from mechanical vibrations that are global to the camera rig or independent to a particular constituent camera. The global vibrations appear as global jitter in the panoramic video frames while the inter-camera vibrations result in jitter in the spatial region of panoramic video contributed by the particular affected cameras. This results in an unpleasant experience for viewers. This is illustrated conceptually in Fig. 1 which shows a single frame of a panoramic video (‘Man video’) that was formed by stitching frames acquired using three cameras. Please refer to the accompanying video to watch this panorama sequence. The tail of the arrows in Fig. 1 represents the location of a few salient features tracked using Kanade–Lucas–Tomasi (KLT) feature tracker [8], [9]. The yellow, red and green vectors represent the direction of motion of these features at a particular instance in time and are color coded to signify the independent direction of motion for the contents captured from camera 1, camera 2 and camera 3 respectively. Fig. 1 illustrates an example where camera 3 is experiencing a different vibration and hence its motion vectors (green) appear to have a different direction as compared to camera 1 and camera 2. In the panoramic video (Man video), this effect appears as jitter in the region of panoramic video frames that is contributed by camera 3. Since this jitter appears in a sub-section of panoramic frame, we term it as sub-frame jitter. The ‘Man video’ demonstrates this effect for an actual setup and thus supports our motivation for the requirement of a stabilizing scheme particularly aimed at panoramic videos.
The problem of single camera video stabilization has been extensively researched and has reached a certain level of maturity [10], [11], [12], [13], [14]. However, for stabilization of panoramic videos captured from multi-camera platforms, little work has been reported. Furthermore, the existing schemes for such systems [15], [16] do not account for the inter-camera vibrations experienced in mobile platforms. In this paper, we treat stabilization of panoramic multi-camera videos as a distinct problem from that of a single camera video. We achieve this by classifying the effects of these vibrations as global, sub-frame and local jitter, and by proposing a method to deal with each one of these in a systematic fashion. In summary, global stabilization is achieved by estimating 2D motion models using the tracked feature trajectories over the complete panoramic frame. Sub-frame stabilization is achieved by making use of the information available in the blend masks that are generated by the stitching application [1]. Blend masks are the intensity weights that are used to blend together the images acquired from multiple cameras to generate a seamless panorama. Finally, the local stabilization tackles the residual jitter that might appear in parts of the panoramic scene in the video due to differences in the scene depth. To the best of our knowledge, this is the first scheme that acknowledges that panoramic videos need to be stabilized temporally as a sequence of stitched frames, as well as spatially to handle the sub-frame jitter.
This paper is organized as follows. In Section 2, we provide a brief account of predominant approaches for video stabilization. In Section 3, we provide the necessary background for panoramic video stitching process followed by a description of the proposed stabilization scheme. In Section 4, we discuss the comparison of the results of the proposed scheme with that of two recent stabilization schemes [14], [15] for a number of videos. Finally in Section 5, we present our conclusion.
Section snippets
Literature review
Single camera video stabilization problem has been thoroughly addressed in previous works over the last decade. Most video stabilization methods comprise of three main steps: Estimating a 2D motion model between subsequent video frames, computing a smooth motion model that removes the unpleasant jitter and finally applying stabilizing geometric transforms to the video frames [11], [12], [14], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29]. These methods differ in
Proposed methodology
In this section we present a brief overview of panoramic video generation followed by a detailed description of the proposed video stabilization technique.
Results and discussion
To evaluate our proposed scheme for panoramic videos, we developed a custom panoramic video acquisition system that consists of 5 Sony FS100 cameras, mounted on a wheeled metal rig in a mirror based setting as shown in Fig. 2a. Each camera captured 1920 × 1080 resolution frames at 30 frames per second. For stitching the videos, we used the method described in [1] using OpenCV. In the following, we provide a thorough discussion and experimental analysis of our three step stabilization scheme
Conclusions
In this paper we propose a video stabilization scheme specifically for stitched panoramic videos generated using mobile multi-camera rigs. The proposed scheme acknowledges that each camera in the panoramic video acquisition system may exhibit independent motion due to which, parts of the panorama contents coming from a camera may jitter differently from the rest. It was demonstrated that by classifying the vibrations for such systems into three categories: global, sub-frame and local
Acknowledgement
We would like to thank IT R&D program of MKE/ETRI (14ZR1110, HCI based UHD Panorama Technology Development) for their generous funding.
References (40)
- et al.
Panoramic imaging—a review
Comput. Graph.
(2003) Real-time digital image stabilization using Kalman filters
Real-Time Imaging
(2002)- et al.
Outlier rejection in high-dimensional deformable models
Image Vis. Comput.
(2007) - et al.
Automatic panoramic image stitching using invariant features
Int. J. Comput. Vis.
(2007) - et al.
Immersive teleconferencing: a new algorithm to generate seamless panoramic video imagery
- et al.
Flyabout: spatially indexed panoramic video
- et al.
Creation of high-resolution video panoramas for sport events
Int. J. Semant. Comput.
(2007) - et al.
Real-time computer vision with OpenCV
Commun. ACM
(2012) - et al.
Detection and tracking of point features
(1991)
Good features to track
Non-metric image-based rendering for video stabilization
A robust and efficient video stabilization algorithm
SIFT features tracking for video stabilization
Subspace video stabilization
ACM Trans. Graph.
Robust online digital image stabilization based on point-feature trajectory without accumulative global motion estimation
IEEE Signal Process Lett.
VSAMS: video stabilization approach for multiple sensors
Stabilizing omnidirectional videos using 3D structure and spherical image warping
Digital image stabilization with sub-image phase correlation based global motion estimation
IEEE Trans. Consum. Electron.
Image based hand tracking via interacting multiple model and probabilistic data association (IMM-PDA) algorithm
Cited by (0)
- ☆
This paper has been recommended for acceptance by Michael Goesele.