Keywords

1 Introduction

There is no doubt about the versatility of human vision. Our eyes sense the optical information of a scene with over 5 orders of magnitude in real time and up to 8 orders with long time adaption [1]. This high dynamic range (HDR) of intensity sensing of the optical information has been a model for our achievements in making new image sensory devices. The HDR sensing generally but necessarily corresponds to having rich number of tonal levels (nTLs) in which each tonal level represents a distinguishable intensity of luminance. The current standard image sensors are not able to perform as what the human eye can sense through adaption of the iris; i.e. the nTLs of the standard image sensors are much lower than human eye. However, it is possible to merge multiple low or standard dynamic range images to obtain a HDR image [2,3,4]. Pursuing real time eye performance modeling; i.e. without long time adaption, has caused developments of hardware and software solutions to obtain enriched nTLs images. Recently our group has shown such a software solution by utilizing a virtually flexible fill factor parameter [5, 6].

The performance evaluation of HDR techniques from their achieved enriched nTLs images is not only an obvious issue but rather a challenging task. There are few No-reference image quality assessment (IQA) methods which can work directly on the enriched nTLs images. Also, there are few display systems with rich nTLs capability for eventual subjective evaluation of the enriched nTLs image. Therefore, such images are generally mapped by tone mapping methods to standard low dynamic range (SDR) images which have at most 256 tonal levels. Then subjective evaluation, using standard display systems, or objective evaluation by implementing reference, reduced reference, and No-reference IQAs are possible to be performed which are highly dependable on the chosen tone mapping method.

In this paper, we show our investigation in finding an evaluation frame work, in general, for HDR and enriched nTLs images, specifically for the generated enriched nTLs images utilizing a virtually flexible fill factor parameter [5, 6]. The paper is organized as follows. In Sect. 2, the related works are explained. Then the generation of the enriched nTLs Images is presented in Sect. 3. Section 4 presents two evaluation methods including our new proposed method for evaluating the enriched nTLs Images. Then the results are shown and discussed in Sect. 5. Finally, we summarized our work in this paper in Sect. 6.

2 Related Works

The image quality is not affected only by the pixel size or quantum efficiency of a sensor [7]. An effective method to improve the performance of an image sensor is to increase the sensor fill factor, e.g., by arranging an array of microlenses on the sensor array [8, 9]. However, due to the physical limitation in practical development and manufacturing, the fill factor of an image sensor cannot be 100% [10]. Recently a new approach was presented [5] in which the fill factor was increased virtually, resulting in enriched nTLs image and widening of the dynamic range. In the approach, the original fill factor is assumed to be known. In [5] a method was proposed to estimate the fill factor using a single arbitrary image.

An HDR or enriched nTLs image is transformed to SDR image utilizing tone mapping. There is a huge amount of tone-mapping work which to review of such methods more information is referred to (e.g. [11, 12]). In this paper, we implement four state-of-the-art tone mapping operators (TMOs) which are a simple logarithmic mapping, Banterle TMO, Krawczyk TMO, and Fattal TMO. Banterle TMO [13] reconstructs data that are not recorded by the image sensor; i.e. the lost information in saturated areas is recovered. This is achieved by using expansion maps to represent the low frequency version of an image in areas of high luminance. Krawczyk TMO [14] utilizes perceptually effect by building a Gaussian pyramid for deriving local adaptation levels in the tone-mapping, and additional effects e.g. visual acuity and veiling luminance are added based on this construction. Furthermore, the loss of color perception in the range of scotopic vision is modeled and temporal adaptation is performed using an exponentially decaying function; i.e. for filtering the adaptation level over time. Fattal TMO [15] is a gradient domain tone mapping method in which the gradient field of the luminance image is manipulated by attenuating the magnitudes of large gradients. A new SDR image is then obtained by solving a Poisson equation on the modified gradient field.

There are few no-reference HDR IQA methods and to best of our knowledge they are still under evaluation using different databases; e.g. [16, 17]. The available full reference and no reference IQAs are used for SDR images; for further information, the interested reader is referred to (e.g. [18, 19]). In this paper, we implement two state-of-the-art IQAs which are fast image sharpness (FISH) and multi scale structural similarity measure (MS-SSIM). They are used to measure the quality of SDR images which are tone mapped from their respective HDR images. The no reference FISH [20] is an effective wavelet-based algorithm for estimating both global and local image sharpness where the image’s overall sharpness is computed via a weighted average of the log-energies of the three-level separable discrete wavelet transform subbands. The full reference MS-SSIM [21] compares two images using information about luminous, contrast and structure in multi scale levels; i.e. utilizing Gaussian pyramid.

3 Generation of the Enriched nTLs Images

The CID2013 database [22] is used in which there are 480 color jpg images captured from 79 different cameras. For the experiments, 14 Images of the database are chosen which have the same content and resolution of 1600 by 1200 in jpeg format. The images are captured by 14 different cameras, including the cell phone cameras, digital compact cameras and DSLR camera. From each image, the following steps are applied, the luminance channel is extracted, a grid of sub pixel is generated, the value of sub pixel is calculated and deformed image with the luminance is combined and evaluated.

  1. 1.

    Each image is either directly down sampled (DDS) or by Gaussian pyramid down sampled (PDS) which generates DDS image and PDS image.

  2. 2.

    For each DDS or PDS image the RGB color space is converted to YUV color space. The mathematical relationship between RGB and YUV can be found in [23,24,25,26].The luminance component of the YUV space is chosen for further processing due to its significant quality impact on the image.

  3. 3.

    The grid pixel of the luminance image is extended to sub pixel level. By using the concept of fill factor FF, every pixel is extended to 30 by 30 square sub pixel, in which the active area S is deduced from \( {\text{S}} = 30 \times \sqrt {FF} \) [5]. The intensity value of every pixel in the luminance image is dispatched to the active area and the value of non-sensitive area is zero. For instance, the former is composed by 24 \( \times \) 24 sub pixels, hereby the value of S is 24 with the FF equals to 0.64.

  4. 4.

    Next processing step is the computation of the intensity values of the extended sub pixels. Due to statistical fluctuation of the incident photons and their conversion to electrons on the sensor, a Bayesian inference statistical model is applied in intensity estimating of each extended subpixel. In realization of the model, the initiate seed has a Gaussian distribution characteristic which is propagated through the extended sub pixels. The subpixels are projected back to the original grid and new enriched nTLs luminance image is obtained.

4 Evaluation of the Enriched nTLs Images

The 28 generated nTLs luminance images are evaluated by following methods.

4.1 Method-1

The nTLs luminance images are tone mapped by normalization, a simple logarithmic mapping, Banterle TMO, Krawczyk TMO, and Fattal TMO which generates 5 × 28 SDR images. The IQAs of FISH and MS-SSIM are used to measure the quality of the SDR images without any reference and with comparison to the original DDS or PDS image respectively.

4.2 Method-2

The enriched nTLs luminance images are compared to the original DDS or PDS image respectively by computing

$$ \rho = \frac{{var\left( {I_{i} J_{i} } \right)}}{var\left( I \right)var\left( J \right)} for\;i = 1\;\;to\;S $$
(1)

where \( I \) is the original image, \( J \) is the respective enriched nTLs image, the \( i \) is image index, \( var \) is variance, \( S \) is the size of image. Then the original images are tone mapped by normalization, a simple logarithmic mapping, Banterle TMO, Krawczyk TMO, and Fattal TMO to generate SDR images. Each of generated SDR images is compared to the respective enriched nTLs images according to Eq. 1. Finally, by computing

$$ min\left( {\sum\nolimits_{k = 1}^{N} {\left( {\rho_{(org - hdr)} - \rho_{j} } \right)} } \right) \,for \,j = 1 \,to\;5 $$
(2)

the ton mapping method among the five methods which is closest to the HDR computation is found; where in Eq. 2, \( \rho_{(org - hdr)} \) is the comparison of the respective enriched nTLs image to the original image according to Eq. 1, \( \rho_{j} \) is the comparison of the respective enriched nTLs image to one of five tone mapping methods according to Eq. 1, \( j \) is index of tone mapping method. The IQA of MS-SSIM is used to measure the quality of the PDS and DDS images in comparison to the reference SDR image which is found from Eq. 2. The quality of a HDR image in comparison to the respective PDS or DDS image is also measured by

$$ Improvement\,factor = \rho_{(org - hdr)} U $$
(3)

where \( U \) is the average of unsatisfactory factor of subjects’ opinion score which is highest possible score (i.e. it is 5 in 5 levels of inquiry) mins mean opinion score (MOS) of the subjects in subjective test.

5 Result and Conclusion

Evaluation results are presented and discussed in this section.

5.1 Result of Evaluation Method-1

The 14 images of the CID2013 database; shown in Fig. 1, are down sampled in two ways and generated DDS and PDS images as it is described in Sect. 4.1. Then the nTLs luminance images of these 28 images are generated; see Sect. 3 for details of the generation process. Each nTLs image (HDR) is tone mapped by normalization (Nor), a simple logarithmic mapping (L-TMO), Banterle TMO (B-TMO), Krawczyk TMO (K-TMO), and Fattal TMO (F-TMO) which generate 5 SDR images; Totally 5 × 28 SDR images are generated. The result of no reference IQA of FISH for original PDS (Org) and related HDR, Nor, L-TMO, B-TMO, K-TMO, and F-TMO are presented in Table 1. The Table 2 shows the same evaluation process as Table 1 but for DDS images.

Fig. 1.
figure 1

Original 14 images captured by different cameras with same comment.

Full size image
Table 1. FISH scores on PDS related images.
Full size table
Table 2. FISH scores on DDS related images.
Full size table

The results in Tables 1 and 2 indicate that the B-TMO (Banterle tone mapping) is more successful to represent the enriched nTLs image in low dynamic rang. However, by simple observation of the B-TMO images we considered that the images are worse than any other generated SDR images. Thus, our conclusion is that the no reference IQA of FISH is not suitable for evaluation of our HDR images. However, the IQA method is used for comparison between DDS and PDS images, the results show consistency to the expectation, see Fig. 2.

Fig. 2.
figure 2

Comparison between DDS and PDS image using IQA of FISH

Full size image

The result of full reference IQA of MS-SSIM for PDS images (i.e. the reference images) and related HDR, Nor, L-TMO, B-TMO, K-TMO, and F-TMO are presented in Table 3. The Table 4 shows the same evaluation process as Table 3 but for DDS images.

Table 3. The result of full reference IQA of MS-SSIM for PDS images. The bold and red values show the maximum and minimum similarity respectively.
Full size table
Table 4. The result of full reference IQA of MS-SSIM for DDS images. The bold and red values show the maximum and minimum similarity respectively.
Full size table

The results in Tables 3 and 4 show inconsistency to draw any conclusion from them. The Normalized (Nor) in both tables is most similar image to the original respective image which makes doubt if the Nor is appropriate method for the tone mapping of the HDR images. The red values in the tables show dissimilarity of the images to the respective reference image. Both tables show inconsistency in dissimilarity results. Thus, our conclusion is that no reference IQA of MS-SSIM is not suitable for evaluation of our HDR images when the reference images are DDS or PDS.

5.2 Result of Evaluation Method-2

The same as evaluation method-1, 5 × 28 SDR images are generated from the original PDS and DDs images. For each set of 5 × 14 images related to PDS or DDS, the \( \varvec{\rho} \) (see Eq. 1) of each two images in a set is computed. The results of average of the \( \varvec{\rho} \) (i.e. for 14 image) for original PDS (Org) and related HDR, Nor, L-TMO, B-TMO, K-TMO, and F-TMO are presented in Table 5. The Table 6 shows the same evaluation process as Table 5 but for DDS images.

Table 5. The mean values of \( \rho \) for PDS related images.
Full size table
Table 6. The mean values of \( \rho \) for DDS related images.
Full size table

The results in Tables 5 and 6 show that the HDR image has average \( \rho \) values of 25.21 and 23.97 to the original image of PDS and DDS respectively. The tone mapping method which has almost the same average \( \rho \) values; blue values, to the original images is K-TMO (Krawczyk tone mapping). This is also verified by computing of Eq. 2 for each set of 14 images which is shown in Table 7.

Table 7. Computation of Eq. 2 for each set of DDS and PDS images.
Full size table

The above results show that the SDR images by Krawczyk tone mapping is good representation of the HDR images; in low dynamic range. Figure 3 shows the normalized \( \rho \) values between HDR images and original images ((org, hdr), blue lines) and between K-TMO and original images ((org, K-TMO), red lines) for set of DDS (in the left) and PDS (in the right) images respectively.

Fig. 3.
figure 3

The normalized \( \rho_{(org - hdr)} \) and \( \rho_{{(org - \left( {K - TMO} \right))}} \) of each set of DDS and PDS are shown in left and right respectively. (Color figure online)

Full size image

The full reference IQA of MS-SSIM for PDS and DDS images is used when the reference images are the respective K-TMO images. The Tables 8 and 9 show the score results.

Table 8. The result of full reference IQA of MS-SSIM for PDS images when the reference images are the respective K-TMO images. The bold and red values show the maximum and minimum similarity respectively.
Full size table
Table 9. The result of full reference IQA of MS-SSIM for DDS images when the reference images are the respective K-TMO images. The bold and red values show the maximum and minimum similarity respectively.
Full size table

The results in Tables 8 and 9 show more consistency to observation in comparison to the results from Tables 3 and 4. The bold and red values show the maximum similarity and dissimilarity respectively. These results are more consistent than the similar results in Tables 3 and 4. However, the scores show the similarity/dissimilarity between K-TMO and original SDR images of DDS or PDS. Our conclusion is that it is ambiguous to know the relation of an HDR image and its original SDR image when we use the representative of the HDR image (i.e. the respective K_TMO image) and utilizing MS-SSIM. Thus, we used the result of subjective tests in relation to the original images which are available from the CID2013 database. The MOS values of the subjective tests are between 0 and 100 in the database. We changed the value to the original 5 levels inquiry; i.e. between 0 and 5. Then the highest value of 5 is subtracted from actual MOS value for each image; i.e. to find average unsatisfactory of subjects for each image. Then the \( \rho \) value between each HDR image and the respective original PDS or DDS is computed according to Eq. 1. Each \( \rho \) value is weighted by the average of unsatisfactory of subjects; i.e. see Eq. 3. Accordingly, the result of the normalized improvement factor for each set of DDS and PDS images is shown in Fig. 4 in the left and right respectively.

Fig. 4.
figure 4

The normalized improvement factors of the DDS and PDS images are shown in the left and right respectively.

Full size image

To this end we have shown that the image quality of the enriched nTLs images in comparison to the respective original SDR images (i.e. set of the PDS and DDS images) are measurable in term of variation measurement by MS-SSIM and quantity of improvement by implementing Eq. 3. To visualize the improvement, we argue to show SDR images by Krawczyk tone mapping in comparison to the respective original SDR images of DDS and PDS sets. This is due to that the SDR images by Krawczyk tone mapping are the most representative of the enriched nTLs image, see Tables 5 and 6. The Figs. 5 and 6 show the comparison of K-TMO images to the respective DDS and PDS set of images respectively. In the figures the normalized improvement factors; shown in Fig. 4, are shown in percentage values.

Fig. 5.
figure 5

The comparison of K-TMO images to the respective DDS set of images. The percentage values show the normalized improvement.

Full size image
Fig. 6.
figure 6

The comparison of K-TMO images to the respective PDS set of images. The percentage values show the normalized improvement.

Full size image

6 Conclusion

The quality assessment of a nTLS or a HDR image is a challenging task. The few available no reference IQA methods for HDR images are generally in evaluation stage. The most available IQA methods; i.e. full reference, reduced reference, and no reference methods, are designed to assess SDR images. In the paper, we show the difficulty of assessment process when the generated nTLs images are tone mapped to low dynamic range. We show that the no reference IQAs of FISH and MS-SSIMM; two popular IQAs, are not adequate tools for assessment of SDRs tone mapped images. We propose a new method to compare the generated nTLs image with its original one; seeing Sect. 5.2. We show how to find the best tone mapping method (K-TMO) among chosen TMO methods. Implementing this strategy makes the IQA of MS-SSIM more useful in which instead of original image, the K-TMO image is used as the reference image. The subjective test data is used to find the amount of improvement of an nTLs image in comparison to its original image.

The results show that the generated nTLs images not only have more number of tonal levels in comparison to original ones but also the dynamic range of images have significantly increased due to improvement factors.