3D shape reconstruction from multifocus image fusion using a multidirectional modified Laplacian operator
Introduction
Digital imaging of real scenes is usually a two-dimensional (2D) discretization process of three-dimensional (3D) scenes. Since the depth information is lost in this process, mining 3D information from 2D images is crucial in computer vision. In recent years, various 3D information recovery techniques have been proposed, and they are roughly divided into three categories: multiview stereo vision, depth camera and monocular vision. Three-dimensional reconstruction based on multiview stereo vision tends to fuse features from different views of the same scene. Some representative examples are multiview information fusion [1] and convolutional neural network [2]. Depth cameras provide explicit 3D information that can significantly reduce the difficulty of depth information acquisition. Lately, some advanced techniques such as multitask learning [3], [4] and 3D convolutional neural networks [5] have been introduced into 3D scene recognition. In addition, depth maps can be recovered by capturing one or several images with a monocular vision system. These methods are referred to as shape from x [6], where x denotes a crucial factor of the depth information acquisition, such as shading [7], contours [8] and motion [9]. The focus level of the image is another important element for depth information recovery, on which a passive method named shape from focus (SFF) [10] has been developed for 3D shape reconstruction from 2D images.
SFF methods obtain the depth map from a sequence of images captured by changing the focus setting of the single-view camera. Because various focus settings correspond to different depth of field, only a part of the area is focused during the imaging process of the object in the 3D scene, while the rest is defocused. If the image sequence is sufficient to cover the entire depth range in the 3D scene by adjusting the distance between the camera and the object, the depth image can be obtained by the position in the image sequence in which each pixel has the highest focus level. Most SFF methods are run by mapping image sequences through an appropriate focus measure (FM) function followed by a reconstruction scheme to obtain 3D reconstruction results. FM is a kind of function evaluating the focus level of each pixel in an image. Some FM functions, such as 3D Laplacian [11], curvelet [12] and optical-based [13] have been proposed in relevant studies. The initial depth map can be obtained from maximizing these focus measure values. After that, some reconstruction schemes, such as the Gaussian model [14] and genetic programming [15], will make use of the focus information in the image sequence to find the accurate depth map.
Optical microscopy is an important tool for high precision analysis and measurement of microscale objects due to its high magnification. However, as the resolution increases, the depth of field becomes shallower, leaving more regions out of focus, subsequently leading to inefficiencies when capturing the 3D structural information of the observed objects. Therefore, it is very necessary to develop a 3D reconstruction method based on optical microscopy. Multiview stereo vision has a complex image registration process, which may have higher complexity and increase the probability of incorrect reconstruction results in 3D reconstruction of optical microscopy [16]. Depth cameras are susceptible to the interference of ambient light in the process of microscopic imaging, leading to noise, occlusion and loss of depth information [17]. Fortunately, no special hardware assistance is needed in SFF. Low hardware complexity makes this algorithm easy to apply to 3D shape reconstruction in optical microscopy [18]. The SFF methods are undeniably efficient, but the following issues deserve further investigation.
- (1)
The effect of noises in the real scene on the reconstruction results. Typically, experiments examine the effectiveness of the SFF algorithms on simulated objects, and then these algorithms are transferred onto real cases. However, the reconstruction results always suffer from performance deterioration due to noises that are difficult to reproduce in a simulation. For example, a large number of highly reflective areas in the images are inevitably produced in the optical imaging process of reflective objects.
- (2)
The effect of weak contrast regions on reconstruction results. When imaging a concave object with a great depth, it is difficult for light to enter the inside of the object, resulting in low-contrast and low-texture areas. Fig. 1 shows image sequences of a concave object with different lens settings. It is clear that the details are difficult to detect and that the contrast declines in regions with small red rectangular windows due to the weak light. The traditional SFF methods might fail to recognize changes in focus in low-contrast areas, leading to a significant deviation in estimating the depth information [12].
- (3)
Additionally, in many application scenarios, for example in microscopy for printed circuit board defects detection, depth information of the object obtained by SFF methods cannot accurately judge the type of defect, and auxiliary gray information is needed as well. Unfortunately, capturing gray information from source images leads to fusion discreteness and then produces more discontinuous regions in fused images, ultimately resulting in low-quality fusion results.
Mitigation of the noises in the real scene will be achieved by an image acquisition device in Section 2. In this paper, for amending the shortcomings of (2) and (3), multifocus image fusion techniques are considered.
Like in SFF, the key step in multifocus image fusion is to select an effective focus measure [19]. Therefore, we expect multifocus image fusion algorithms to provide more potential solutions for SFF. There are two types of multifocus image fusion algorithms: spatial domain methods and transform domain methods. Spatial domain methods directly pick pixels, blocks or regions to construct fused images linearly or nonlinearly. Nevertheless, these methods rely heavily on the accuracy of the pixels, blocks and regions [20]. Therefore, when these spatial domain methods are applied to SFF, they might cause spatial distortions and ghosting artifacts, which result in shape inconsistency in reconstruction results. To overcome the above disadvantages of spatial domain fusion methods, transform domain fusion methods should be considered.
Transform domain fusion methods decompose images into high and low frequency coefficients and employ the fusion rules to handle these coefficients. In recent years, decomposition tools, such as wavelet transform [21] and discrete cosine harmonic wavelet transform [22], have been applied in multifocus image fusion. However, poor directionality and shift variance that lead to unsatisfactory solutions is a common deficiency in these tools. Nonsubsampled contourlet transform (NSCT) [23] is regarded as an expedient remedy to these problems, but it suffers from large computational burdens. The recently proposed nonsubsampled shearlet transform (NSST) overcomes the above mentioned shortcomings [24]. By avoiding downsampling operations, NSST has better performance in terms of shift invariance. It yields decomposition subimages all having the same size of source images and facilitates tracking the depth information during the decomposition process. In addition, compared to other transform domain methods, NSST not only has excellent properties such as anisotropy and multiscale but also provides better direction sensitivity to capture the intrinsic geometrical structure of images. The abundant detailed information obtained by using these properties can provide a more accurate basis for depth information evaluation. Therefore, it is feasible to capture depth information in the process of NSST-based multifocus image fusion.
The main contributions of this paper include three aspects. (1) This paper proposes a new framework for embedding SFF method into multifocus image fusion algorithm, in which the fused image and depth map of an object in the 3D scene can be obtained simultaneously. (2) A new multidirectional modified Laplacian (MDML) as a focus measure to realize the mapping from high-frequency subbands to depth maps is analyzed and discussed. (3) An iterative edge repair method that can automatically detect and repair the error areas in the depth maps is proposed, and in the end, this method can effectively improve reconstruction accuracy of the low contrast regions.
The structure of this paper is as follows. Section 2 describes the problems inherent in applying the SFF technique to the 3D reconstruction of microscales and designs an image capture device. Section 3 shows the proposed algorithm in detail, and Section 4 presents several experimental settings, comparative analysis. Conclusions and future work are discussed in Section 5.
Section snippets
Background
To generate the depth image of an object, it is necessary to estimate the distance between every point of the object and the camera. Fig. 2 illustrates the schematic illustration of microscopic imaging, where f1 and f2 indicate the focal lengths of the lenses L1 and L2, u is the distance of the object from the lens, and v is the distance of the magnified image from the lens. The relationship between f1, u and v is given by the Gaussian lens law: . The object AB is located near the
The proposed approach
Most established multifocus image fusion methods provide better visual perception and quality. However, these methods usually use a large number of image processing techniques, which are not conducive to accurately extract the depth information of image sequences. In this paper, a new 3D shape reconstruction scheme is proposed to search the depth maps that represent the best-focused image during the image fusion process. Briefly, it takes three steps to implement the proposed method. First, we
Datasets
Both simulated and real objects are used to verify the effectiveness of the proposed 3D shape reconstruction method. The first simulated sequence consists of 100 images of 360 × 360 pixels, and the corresponding image generation algorithm of this sequence can be found in the previous study [16]. There are three other image sequences with respect to microscopic objects with varying textures and structures. The second and third sequences, which have concave structures, come from the intaglio
Conclusion
The NSST-based microscopic multifocus fusion method for 3D shape reconstruction has been introduced in this paper. It is based on the existence of depth information in the fused image. Due to the combination of depth estimates and fusion processes by using a novel multidirectional modified Laplacian operator, the proposed method can directly identify the depth map and fused image simultaneously. Future work on this topic will focus on three aspects:
- (1)
SFF scheme based on multifactor collaborative
Acknowledgements
This work is supported by National Key R&D Program of China (No. 2018YFB1004300), the National Natural Science Foundation of China (Nos. 61672332 and 61872226), the Key R&D Program of Shanxi Province, China (No. 201803D421012), the Scientific and Technological Innovation Programs of Higher Education Institutions in Shanxi, China (No. 2019L0100)
Tao Yan (corresponding author) was received the Ph.D. degree from Chengdu Institute of Computer Applications, Chinese Academy of Science. He is now a lecturer at Shanxi University. His research interests include image processing and evolutionary computation.
References (44)
- et al.
Multi-view ensemble manifold regularization for 3D object recognition
Inf. Sci.
(2015) - et al.
Contactless and partial 3D fingerprint recognition using multi-view deep representation
Pattern Recognit.
(2018) - et al.
Sparse transfer for facial shape-from-shading
Pattern Recognit.
(2017) - et al.
3D modeling of multiple-object scenes from sets of images
Pattern Recognit.
(2014) - et al.
3D shape recovery from image focus using kernel regression in eigenspace
Image Vis. Comput.
(2010) - et al.
Shape from focus using fast discrete curvelet transform
Pattern Recognit.
(2011) - et al.
A novel algorithm for estimation of depth map using image focus for 3D shape recovery in the presence of noise
Pattern Recognit.
(2008) - et al.
Analysis of focus measure operators for shape-from-focus
Pattern Recognit.
(2013) - et al.
Optimal depth estimation by combining focus measures using genetic programming
Inf. Sci.
(2011) - et al.
Reliability measure for shape-from-focus
Image Vis. Comput.
(2013)
Surface area-based focus criterion for multi-focus image fusion
Inf. Fusion
Pixel-level image fusion: a survey of the state of the art
Inf. Fusion
A wavelet-based image fusion tutorial
Pattern Recognit.
Sparse directional image representations using the discrete shearlet transform
Appl. Comput. Harmon. Anal.
Measure of image sharpness using eigenvalues
Inf. Sci.
3D shape from focus using LULU operators and discrete pulse transform in the presence of noise
J. Vis. Commun. Image Represent.
Evaluation of focus measures in multi-focus image fusion
Pattern Recognit. Lett.
Multi-focus image fusion with a deep convolutional neural network
Inf. Fusion
Multi-scale weighted gradient-based fusion for multi-focus images
Inf. Fusion
Multifocus image fusion using the nonsubsampled contourlet transform
Signal Process.
A general framework for image fusion based on multi-scale transform and sparse representation
Inf. Fusion
Latent max-margin multitask learning with skelets for 3D action recognition
IEEE Trans. Cybern.
Cited by (35)
A novel algorithm for three-dimensional shape reconstruction for microscopic objects based on shape from focus
2024, Optics and Laser TechnologyInfrared and visible image fusion based on Multi-State contextual hidden Markov Model
2023, Pattern RecognitionAn efficient point-set registration algorithm with dual terms based on total least squares
2023, Pattern RecognitionShape from focus using gradient of focus measure curve
2023, Optics and Lasers in EngineeringCitation Excerpt :Based on the imaging principle of applied optics, shape from focus (SFF) analyzes the relationship among image distance, focal length, object distance and focused images to recovery the 3D shape of the object. The SFF technique has been successfully utilized in many applications, 3D camera [1], micro manufacturing [2] and interfacial toughness [3], etc. It is also commonly used in obtaining the all-in-focus image and the surface feature recognition as a common tool of the optical microscope [4].
Frequency-domain segmentation algorithm for three-dimensional light-field display based on pixel deviation threshold
2022, Optics and Laser TechnologyCitation Excerpt :By eliminating pixel deviation in the high-frequency part (severe pixel deviation) of the image and maintaining sharpness in the low-frequency part (slight pixel deviation), pixel deviation and sharpness degradation issues could be solved simultaneously. Currently, image processing technology in the frequency-domain has been extensively applied in defect inspection [16], image deblurring [17], and 3D image reconstruction [18], among several others. To our knowledge, image frequency-domain processing techniques have not been used to overcome the problems associated with the pixel deviation in light-field 3D display.
Tao Yan (corresponding author) was received the Ph.D. degree from Chengdu Institute of Computer Applications, Chinese Academy of Science. He is now a lecturer at Shanxi University. His research interests include image processing and evolutionary computation.