Image noise detection in global illumination methods based on FRVM
Introduction
The main objective of global illumination methods is to produce synthetic images with photo-realistic quality. These methods are generally based on path tracing theory in which stochastic paths are generated from the camera point of view through each pixel toward the 3D scene [1]. The Monte Carlo theory ensures that this process will converge to the correct image when the number of paths grows [2]. However, there is no information about the number of required paths for the image to be visually converged. In order to solve this problem, various perceptual models have been proposed in the literature [3], [4], [5]. They used the visible differences predictor (VDP) algorithm to provide the quantitative measures of perceptual convergence by predicting and estimating the perceivable differences between the intermediate and the reference images. A similar approach has been proposed by Takouachet et al. [6]. The VDP was used for estimating the differences between the first very noisy image and the successive images of the progressive rendering process. In both approaches, the VDP operates only on the achromatic channel and needs high computation time. Various visual models have also been adapted in order to accelerate the global illumination computation in dynamic environments [7], [8]. These models are used to indicate where computational effort should be spent during the lighting solution. Rendering system then spent more time to calculate the observers׳ regions of interest. However, the proposed models were originally validated by neuro-biological and psycho-physical studies but their simplifications have not been validated yet. Through experimental results, it is shown that the VDP, which needs the reference image, does not always give an accurate response [9]. It is well known that images are very noisy at the beginning of the generation of image synthesis in global illumination algorithms, and they become more and more photo-realistic as the number of paths increases. Consequently this process needs either an image quality measure or its counterpart noise level measure to decide the necessary number of paths. Image quality measure, concerning image synthesis, is hitherto made using human observers (providing experimental psycho-visual scores) that is very time consuming. It is obvious that the automatic measure of the image׳s quality is very important to characterize its visual quality. It is of great interest in image compression (JPEG models) and in image synthesis. It encompasses three models in the literature: full reference, no-reference, and reduced-reference models [10], [11], [12]. First the full reference models that use the original version of the image to indicate the quality assessment of the processed version as the signal to noise ratio SNR, structural similarity index measure SSIM [11], [13] and mathematical metrics [14], [15]. These models are the most used methods to evaluate image quality. Unfortunately, the SNR approach gives a global measure of the difference between two images without considering the perceptibility of each pixel in the image, while the SSIM model and the mathematical metrics need the full original image which is not available in all cases (particularly in the case of image synthesis).
Second the no-reference models which evaluate the quality of the image without access to reference images [12], [16]. Some recent papers proposed no-reference quality assessment of JPEG images, although the authors obtained good results but these reported quality measures have their limitations because they are based on theoretical models of noise.
Finally in the reduced-reference models, the processed image is analyzed using some relevant information to calculate the quality of the result image [10], [17], [18]. These models will be used in our study as shown in this paper. However, the proposed models, which are based on theoretical models of noise, present sensitivity limits in global illuminations. The human visual system (HVS) carries out a fascinating strategy of compression and sensitivity thresholds. In fact HVS cannot perceive equally all the components of our environment. For this system, some parts of the environment are very important while other parts are automatically ignored. As a consequence of these limits and the high computation cost of global illumination algorithms, perception approaches have been proposed in the literature. The main idea of such approaches is to replace the human observer by a vision model [19], [8]. These approaches provide interesting results but are complex and still incomplete due to the internal system complexity and its partial knowledge. They need long computation times and are often difficult to be used and parametrized. Another possible way is to use interesting capacities of machine learning to automatically compute image quality measures. A first attempt was made using SVMs [20]. Unfortunately SVMs are very efficient to learn perceptual features but are less efficient to learn noise. This drawback leads us to study a possible more adapted method.
So this paper focuses on the use of a new learning model to detect and to quantify stochastic noise present in a synthetic image. We propose a reduced image quality approach based on feature generation and fast relevance vector machine. In the context of machine learning, relevance vector machine (RVM) has been studied by Tipping [21]. Tipping introduced the principle of Bayesian inference in a machine learning with a particular emphasis on the importance of marginalization for dealing with uncertainty. The RVM model conveys a number of advantages over the very popular support vector machine (SVM) because it is probabilistic and it uses a small number of kernel functions which do not necessarily satisfy the needed Mercer׳s condition. However, the learning algorithm is typically much slower than the SVM. The fast relevance vector machine (FRVM) learning algorithm, also proposed by Tipping, is an accelerated version which exploits the properties of the marginal likelihood function to enable maximization via efficient sequential addition and deletion of candidate basis functions [22]. The advantage of our application is that it embodies sparse Bayesian learning that makes it possible to treat complete images and to benefit from probabilistic predictions and automatic estimation of nuisance parameters [23], [24]. By mimicking the HVS, such model can provide important improvement for rendering.
The paper is structured as follows: Section 2 describes the experimental database we use and Section 3 describes the fast relevance vector machine theory. Section 4 introduces the FRVM design for image quality evaluation, Section 5 explains how to generate features for image quality evaluation while Section 6 shows the experimental results obtained by the learning models. Finally the paper is summarized with some conclusions in Section 7.
Section snippets
Image quality database
The model is built on data corresponding to images of globally illuminated scenes. The path tracing algorithm was used in order to reduce noise [1]. This algorithm generates stochastic paths from the camera to the 3D scene. For each intersection of a path with the surface, a direction of reflection or refraction is randomly extracted. The luminance at a point x in direction w is defined by [1]where S is the scene surface, Le is the emitted
Fast relevance vector machine
The RVM is based on a probabilistic Bayesian learning framework as well as a good generalization capability. It acquires relevance vectors and weights by maximizing a marginal likelihood function. The structure of the RVM is described by the sum of the product of weights and kernel functions. A kernel function means a set of basis functions projecting the input data into a high dimensional feature space.
Given a data set of input-target pairs , we write the targets as a vector
FRVM design for image quality evaluation
In this paper, perceptual noises are quantified using classic denoising algorithms. The goal of denoising algorithms is to remove the noise from image and to highlight the important image features as much as possible. There are two basic approaches to image denoising: the spatial filtering method and the transform domain filtering one. There are different spatial filtering techniques such as low-pass smoothing filters, median filter, Wiener filter, and Bilateral filter. The low-pass smoothing
Noise features from denoising algorithms
In this section, we will discuss how to extract image noise features from denoising algorithms in order to achieve better performance for the training algorithms in time and space complexities. First we apply image denoising algorithms to an image L in order to obtain its denoised version LD. Then, the estimated image noise at the pixel location is obtained by a pixel-wise subtraction between the current image pixel and the denoised one:The mean and the
Learning and evaluation
In order to test the performance of the proposed technique, the scene named Bar is used for learning and evaluation whereas the scene Class is only used for the testing process (Fig. 6). In fact, considering one scene is definitely not enough to train the model. In our study the worst case is tried using only one scene in order to test the ability of the learning models to generalize on the testing scenes by using a small number of learning examples as well as to reduce the computation learning
Conclusion
The main idea of this paper is to introduce the application of the FRVM to take into account the uncertainty (noise) present in global illumination applications. Path tracing methods provide unbiased and realistic images, but they converge slowly and exhibit perceptual noise during their convergence process. They should be stopped only when noise is not visually perceptible. In addition to offering a good prediction on the testing base, the introduced FRVM approach uses fewer basis functions
Acknowledgments
The research described in the paper has been funded in part by the Lebanese University program through grant number ER26. Special thanks to Miss Ferial Srour-Nemr for her excellent proof reading.
Joseph Constantin obtained the M.S. degree in Software Engineering and Systems Modeling from the Lebanese University in 1997 and the Ph.D. degree in Automatic and Robotic control from the Picardie Jules Verne University, France, in 2000. Since 2001, he has been an associate professor at the Lebanese University, Faculty of Sciences and a researcher in the Applied Physics Laboratory of the Doctoral School of Sciences and Technology at the Lebanese University. His current research interests are in
References (38)
- et al.
Kurtosis based no-reference quality assessment of JPEG2000 images
Signal Process.: Image Commun.
(2011) - et al.
Objective quality assessment of color images based on a generic perceptual reduced reference
Signal Process.: Image Commun.
(2008) - I. Wald, T. Kollig, C. Benthin, A. Keller, P. Slusalleki, Interactive global illumination using fast ray tracing, in:...
- T. Kollig, A. Keller, Efficient bidirectional path tracing by randomized quasi-monte carlo integration, in: H....
- K. Myszkowski, Visible differences predictor: applications to global illumination problems, in: Eurographics Rendering...
A perceptual metric for production testing
J. Graph. Tools
(2004)- M. Ramasubramanian, S. Pattanaik, D. Greenberg, A perceptually based physical error metric for realistic image...
- N. Takouachet, S. Delepoulle, C. Renaud, A perceptual stopping condition for global illumination computations, in:...
- H. Yee, S. Pattanaik, D. Greenberg, Spatio temporal sensitivity and visual attention for efficient rendering of dynamic...
- P. Longhurst, K. Debattista, A. Chalmers, A Gpu based saliency map for high-fidelity selective rendering, in: AFRIGRAPH...
Selection low-level features for image quality assessment by statistical methods
J. Comput. Inf. Technol.
Estimation of optimal PDE-Based denoising in the SNR sense
IEEE Trans. Image Process.
Intrinsic sensor noise features for forensic analysis on scanners and scanned images
IEEE Trans. Inf. Forensics Secur.
A no-reference objective image sharpness metric based on the notion of just noticeable blur (JNB)
IEEE Trans. Image Process.
Reduced-reference image quality assessment using divisive normalization based image representation
IEEE J. Sel. Top. Signal Process.
Cited by (15)
Soft sensor development based on kernel dynamic time warping and a relevant vector machine for unequal-length batch processes
2021, Expert Systems with ApplicationsDe-noising of digital image correlation based on stationary wavelet transform
2017, Optics and Lasers in EngineeringCitation Excerpt :With respect to the method for image noise removal, a variety of discriminative methods and noise filtering methods have been researched in optics [11,12] (e.g. nonlinear filter algorithm [2,13], iterative reconstruction technique for image restoration [14–16], non-blind deblurring method for partly-textured blurred images with Poisson noise [17], multiscale denoising algorithm adapted to the unknown noise model [18]). The first category is noise removal for the given noise, as proposed in [4,19–22]. However, compared with the given noise, the light noise is a random noise in the image.
Perception-JND-driven path tracing for reducing sample budget
2024, Visual ComputerHeterogeneous Regularization for Fast Rendering using Deep Spike Neural Network
2023, Research SquareGuided-Generative Network for noise detection in Monte-Carlo rendering
2021, Proceedings - 20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021
Joseph Constantin obtained the M.S. degree in Software Engineering and Systems Modeling from the Lebanese University in 1997 and the Ph.D. degree in Automatic and Robotic control from the Picardie Jules Verne University, France, in 2000. Since 2001, he has been an associate professor at the Lebanese University, Faculty of Sciences and a researcher in the Applied Physics Laboratory of the Doctoral School of Sciences and Technology at the Lebanese University. His current research interests are in the fields of machine learning, image processing, robot dynamics and control, diagnosis systems and biomedical engineering.
André Bigand (IEEE Member) received the Ph.D. degree in 1993 from the University Paris 6 and the “HDR” degree in 2001 from the Université du Littoral of Calais (ULCO, France). He is currently senior associate professor in ULCO since 1993. His current research interest include uncertainty modeling and machine learning with applications to image processing and synthesis (particularly noise modeling and filtering). He is currently with the LISIC Laboratory (ULCO).
Ibtissam Constantin received the Dipl.-Ing. degree in electrical engineering and the M.S. degree in industrial control from the Faculty of Engineering, Lebanese University, in 2000 and 2002, respectively, and the Ph.D. degree from Troyes University of Technology, France, in 2007. Since 2007, she has been an associate professor at the Lebanese University, Faculty of Sciences and a researcher in the Applied Physics Laboratory of the Doctoral School of Sciences and Technology at the Lebanese University. Her current research interests are in the field of machine learning and kernel methods.
Denis Hamad is a professor at the University of Littoral Côte d׳Opale since 2002. He obtained a HDR (Habilitation à Diriger la Recherche) degree in neural networks for unsupervised pattern classification and a Ph.D. degree in supervision of complex systems from the Lille 1 University, in 1997 resp. in 1986. Between 1998 and 2002, he was a professor at the University of Picardie Jules Vernes, Amiens-France. His main research interests are in machines learning, image and signal processing. Actually, his research is in the area of machine learning for monitoring of marine ecosystems.