High dynamic range video reconstruction from a stereo camera setup
Introduction
The high dynamic range of a real scene cannot be fully captured by regular CCD or CMOS camera sensors. The resulting images are therefore called low dynamic range (LDR) images. However, since the human visual system (HVS) is sensitive to a far wider dynamic range compared to LDR images, techniques for creating HDR content are useful to capture more information of a real scene, and thus, improve image quality. Several methods exist to generate HDR images which can be categorized into three classes according to [1]: Direct acquisition using particular HDR cameras [2], HDR rendering for synthetically created images, and exposure bracketing. The first class has two major drawbacks. The special hardware is very expensive and it cannot capture a dynamic range as large as when using an exposure bracketing technique. The second class is only relevant for computer-generated content, and hence, is not applicable for improving images of a real scene. The third class of HDR acquisition is a low-budget solution. In recent years, many consumer cameras started to offer exposure bracketing as a method to create HDR images of static scenes. For exposure bracketing, a series of LDR pictures is taken with different exposure times one after the other in order to estimate the camera response, convert the pixel values into radiance values and then merge them together to generate the HDR output. Two well-known approaches are the algorithm by Debevec and Malik [3] and the method by Robertson et al. [4]. The different exposures can also be realized by using different apertures or neutral density (ND) filters instead of exposure times.
However, this approach is not applicable for capturing dynamic scenes or even HDR video due to the temporal fashion of the exposure bracketing since the scene is changing over time. An example for a static scene without and with camera motion is illustrated in Fig. 1. It can be seen that temporal exposure bracketing completely fails when there is motion. There are approaches which compensate for the motion [5], but they have the drawback not being able to easily apply ND-filters, and hence, yielding artifacts from different motion blur or depth of field.
The goal of this work is therefore to allow for HDR image acquisition of dynamic scenes and HDR video. The proposed framework utilizes a stereo camera setup to realize a spatial exposure bracketing where the differently exposed images are distributed among the two views. Due to this, all necessary information to create an HDR image is available at the same time instant. This stereo setup leads to two challenges. On the one hand, HDR image reconstruction requires perfect alignment of the views after image warping. Therefore, a robust disparity estimation is needed. On the other hand, typical stereo matching algorithms rely on the brightness constancy assumption which does not hold for differently exposed views. Each stage of the proposed framework is built to properly handle the just mentioned challenges and also under-/oversaturated regions. Instead of applying a stereo matching approach, it would also be possible to utilize time-of-flight cameras in order to obtain the disparity maps. However, they can only generate low resolution disparity maps, increase the complexity of the setup, and are still considerably expensive. For testing purposes, datasets from the Middlebury Stereo Vision Page [6] and two self-recorded datasets are used. Besides visual quality evaluation, a suitable quality metric is applied with respect to a single-view HDR reference image.
The key novel contributions are the new one-stage framework compared to the two-stage frameworks in the literature [7], [8] and the introduction of a saturation map to reduce errors in the final HDR image. Beyond this, an objective quality assessment, which is directly computed on the HDR images, is done for the first time.
The remainder of this article is structured as follows. The upcoming section presents previous work, whereas Section 3 covers the layout and the details of the framework and its modules. Experimental results are then given in Section 4. Finally, Section 5 concludes this contribution.
Section snippets
Previous work
This contribution is mainly inspired by the work in [7], [8] where stereo matching is done in two stages and the camera responses are computed from the matches found in the first stage. These camera responses are then used for converting the input images to radiance space and a second stereo matching for disparity refinement is performed. In this work, however, unlike the existing algorithms, the camera responses are not estimated from the stereo images. This step is done only once in an
Proposed stereo high dynamic range imaging framework
Before the particular stages of the proposed stereo high dynamic range (Stereo-HDR) imaging framework are discussed in detail, an overview of the framework is given. A schematic layout in the form of a block diagram is provided in Fig. 2. Basically, the framework can be split into the main workflow and an offline step.
The new offline step is responsible for calculating the camera response curves and weighting functions (see Section 3.4), which are both necessary for the conversion of the LDR
Experimental results
This section is divided into three parts. The first part gives an overview of the simulation setup. The second part is about visual quality evaluation and the last one states some objective quality results.
Conclusion
In this contribution, a stereo high dynamic range imaging framework was presented, which is capable of handling dynamic scenes and video. Beyond that the proposed framework is easily extendable to an arbitrary number of cameras. Nevertheless, this stereo case can be regarded as an important case as it is a good representative for off the shelf low cost solutions. The main idea of this framework is to apply a spatial instead of a temporal exposure bracketing. However, this leads to two
References (32)
- et al.
High Dynamic Range ImagingAcquisition, Display, and Image-Based Lighting (The Morgan Kaufmann Series in Computer Graphics)
(2005) - et al.
A high dynamic range CMOS image sensor for scientific imaging applications
IEEE Sensors Journal
(2009) - P.E. Debevec, J. Malik, Recovering high dynamic range radiance maps from photographs, in: Proceedings of the 24th...
- M. Robertson, S. Borman, R. Stevenson, Dynamic range improvement through multiple exposures, in: Proceedings of...
- et al.
High dynamic range video
ACM Transactions on Graphics
(2003) - Middlebury Stereo Vision Page,...
- N. Sun, H. Mansour, R. Ward, HDR image construction from multi-exposed stereo LDR images, in: Proceedings of the 17th...
- A. Troccoli, S.B. Kang, S. Seitz, Multi-view multi-exposure stereo, in: Proceedings of the Third International...
- X. Mei, X. Sun, M. Zhou, S. Jiao, H. Wang, X. Zhang, On building an accurate stereo matching system on graphics...
- F. Kainz, R. Bogart, Technical Introduction to OpenEXR, Industrial Light & Magic (February...
Handbook of Image and Video Processing
Evaluation of stereo matching costs on images with radiometric differences
IEEE Transactions on Pattern Analysis and Machine Intelligence
Robust stereo matching using adaptive normalized cross-correlation
IEEE Transactions on Pattern Analysis and Machine Intelligence
Cited by (35)
Robust HDR reconstruction using 3D patch based on two-scale decomposition
2024, Signal ProcessingAutomata design for honeybee search algorithm and its applications to 3D scene reconstruction and video tracking
2021, Swarm and Evolutionary ComputationCitation Excerpt :There are several different proposals to perform video tracking. One of the most straightforward proposals is ZNCC, which measures the correlation between two image templates [43,63,64]. The grade of similarity between a template (searched object) and a suspected location in the image frame is obtained as a scalar number, the higher it is, the more correlated these templates are.
HDR multiview image sequence generation: Toward 3D HDR video
2017, High Dynamic Range Video: Concepts, Technologies and ApplicationsGhosting in HDR video
2017, High Dynamic Range Video: Concepts, Technologies and ApplicationsHDR Image Construction from Trifocal Multiexposure Images
2021, IEEE 23rd International Workshop on Multimedia Signal Processing, MMSP 2021