Elsevier

Pattern Recognition

Volume 98, February 2020, 107065
Pattern Recognition

3D shape reconstruction from multifocus image fusion using a multidirectional modified Laplacian operator

https://doi.org/10.1016/j.patcog.2019.107065Get rights and content

Highlights

  • The feasibility of combining multifocus image fusion and shape-from-focus methods to obtain real 3D reconstruction results is introduced.

  • The advantages of 3D shape reconstruction using nonsubsampled shearlet transform and a multidirectional modified Laplacian are discussed.

  • The major challenges faced by conventional shape-from-focus algorithms in reconstruction of low-contrast regions are analyzed.

  • The difficulties of existing image fusion methods in mining depth information are summarized.

  • Future applications of the proposed 3D shape reconstructions in the field of intaglio printing are presented.

Abstract

Multifocus image fusion techniques primarily emphasize human vision and machine perception to evaluate an image, which often ignore depth information contained in the focus regions. In this paper, a novel 3D shape reconstruction algorithm based on nonsubsampled shearlet transform (NSST) microscopic multifocus image fusion method is proposed to mine 3D depth information from the fusion process. The shift-invariant property of NSST guarantees the spatial corresponding relationship between the image sequence and its high-frequency subbands. Since the high-frequency components of an image represent the focus level of the image, a new multidirectional modified Laplacian (MDML) as the focus measure maps the high-frequency subbands to images of various levels of depth. Next, the initial 3D reconstruction result is obtained by using an optimal level selection strategy based on the summation of the multiscale Laplace responses to exploit these depth maps. Finally, an iterative edge repair method is implemented to refine the reconstruction result. The experimental results show that the proposed method has better performance, especially when the source images have low-contrast regions.

Introduction

Digital imaging of real scenes is usually a two-dimensional (2D) discretization process of three-dimensional (3D) scenes. Since the depth information is lost in this process, mining 3D information from 2D images is crucial in computer vision. In recent years, various 3D information recovery techniques have been proposed, and they are roughly divided into three categories: multiview stereo vision, depth camera and monocular vision. Three-dimensional reconstruction based on multiview stereo vision tends to fuse features from different views of the same scene. Some representative examples are multiview information fusion [1] and convolutional neural network [2]. Depth cameras provide explicit 3D information that can significantly reduce the difficulty of depth information acquisition. Lately, some advanced techniques such as multitask learning [3], [4] and 3D convolutional neural networks [5] have been introduced into 3D scene recognition. In addition, depth maps can be recovered by capturing one or several images with a monocular vision system. These methods are referred to as shape from x [6], where x denotes a crucial factor of the depth information acquisition, such as shading [7], contours [8] and motion [9]. The focus level of the image is another important element for depth information recovery, on which a passive method named shape from focus (SFF) [10] has been developed for 3D shape reconstruction from 2D images.

SFF methods obtain the depth map from a sequence of images captured by changing the focus setting of the single-view camera. Because various focus settings correspond to different depth of field, only a part of the area is focused during the imaging process of the object in the 3D scene, while the rest is defocused. If the image sequence is sufficient to cover the entire depth range in the 3D scene by adjusting the distance between the camera and the object, the depth image can be obtained by the position in the image sequence in which each pixel has the highest focus level. Most SFF methods are run by mapping image sequences through an appropriate focus measure (FM) function followed by a reconstruction scheme to obtain 3D reconstruction results. FM is a kind of function evaluating the focus level of each pixel in an image. Some FM functions, such as 3D Laplacian [11], curvelet [12] and optical-based [13] have been proposed in relevant studies. The initial depth map can be obtained from maximizing these focus measure values. After that, some reconstruction schemes, such as the Gaussian model [14] and genetic programming [15], will make use of the focus information in the image sequence to find the accurate depth map.

Optical microscopy is an important tool for high precision analysis and measurement of microscale objects due to its high magnification. However, as the resolution increases, the depth of field becomes shallower, leaving more regions out of focus, subsequently leading to inefficiencies when capturing the 3D structural information of the observed objects. Therefore, it is very necessary to develop a 3D reconstruction method based on optical microscopy. Multiview stereo vision has a complex image registration process, which may have higher complexity and increase the probability of incorrect reconstruction results in 3D reconstruction of optical microscopy [16]. Depth cameras are susceptible to the interference of ambient light in the process of microscopic imaging, leading to noise, occlusion and loss of depth information [17]. Fortunately, no special hardware assistance is needed in SFF. Low hardware complexity makes this algorithm easy to apply to 3D shape reconstruction in optical microscopy [18]. The SFF methods are undeniably efficient, but the following issues deserve further investigation.

  • (1)

    The effect of noises in the real scene on the reconstruction results. Typically, experiments examine the effectiveness of the SFF algorithms on simulated objects, and then these algorithms are transferred onto real cases. However, the reconstruction results always suffer from performance deterioration due to noises that are difficult to reproduce in a simulation. For example, a large number of highly reflective areas in the images are inevitably produced in the optical imaging process of reflective objects.

  • (2)

    The effect of weak contrast regions on reconstruction results. When imaging a concave object with a great depth, it is difficult for light to enter the inside of the object, resulting in low-contrast and low-texture areas. Fig. 1 shows image sequences of a concave object with different lens settings. It is clear that the details are difficult to detect and that the contrast declines in regions with small red rectangular windows due to the weak light. The traditional SFF methods might fail to recognize changes in focus in low-contrast areas, leading to a significant deviation in estimating the depth information [12].

  • (3)

    Additionally, in many application scenarios, for example in microscopy for printed circuit board defects detection, depth information of the object obtained by SFF methods cannot accurately judge the type of defect, and auxiliary gray information is needed as well. Unfortunately, capturing gray information from source images leads to fusion discreteness and then produces more discontinuous regions in fused images, ultimately resulting in low-quality fusion results.

Mitigation of the noises in the real scene will be achieved by an image acquisition device in Section 2. In this paper, for amending the shortcomings of (2) and (3), multifocus image fusion techniques are considered.

Like in SFF, the key step in multifocus image fusion is to select an effective focus measure [19]. Therefore, we expect multifocus image fusion algorithms to provide more potential solutions for SFF. There are two types of multifocus image fusion algorithms: spatial domain methods and transform domain methods. Spatial domain methods directly pick pixels, blocks or regions to construct fused images linearly or nonlinearly. Nevertheless, these methods rely heavily on the accuracy of the pixels, blocks and regions [20]. Therefore, when these spatial domain methods are applied to SFF, they might cause spatial distortions and ghosting artifacts, which result in shape inconsistency in reconstruction results. To overcome the above disadvantages of spatial domain fusion methods, transform domain fusion methods should be considered.

Transform domain fusion methods decompose images into high and low frequency coefficients and employ the fusion rules to handle these coefficients. In recent years, decomposition tools, such as wavelet transform [21] and discrete cosine harmonic wavelet transform [22], have been applied in multifocus image fusion. However, poor directionality and shift variance that lead to unsatisfactory solutions is a common deficiency in these tools. Nonsubsampled contourlet transform (NSCT) [23] is regarded as an expedient remedy to these problems, but it suffers from large computational burdens. The recently proposed nonsubsampled shearlet transform (NSST) overcomes the above mentioned shortcomings [24]. By avoiding downsampling operations, NSST has better performance in terms of shift invariance. It yields decomposition subimages all having the same size of source images and facilitates tracking the depth information during the decomposition process. In addition, compared to other transform domain methods, NSST not only has excellent properties such as anisotropy and multiscale but also provides better direction sensitivity to capture the intrinsic geometrical structure of images. The abundant detailed information obtained by using these properties can provide a more accurate basis for depth information evaluation. Therefore, it is feasible to capture depth information in the process of NSST-based multifocus image fusion.

The main contributions of this paper include three aspects. (1) This paper proposes a new framework for embedding SFF method into multifocus image fusion algorithm, in which the fused image and depth map of an object in the 3D scene can be obtained simultaneously. (2) A new multidirectional modified Laplacian (MDML) as a focus measure to realize the mapping from high-frequency subbands to depth maps is analyzed and discussed. (3) An iterative edge repair method that can automatically detect and repair the error areas in the depth maps is proposed, and in the end, this method can effectively improve reconstruction accuracy of the low contrast regions.

The structure of this paper is as follows. Section 2 describes the problems inherent in applying the SFF technique to the 3D reconstruction of microscales and designs an image capture device. Section 3 shows the proposed algorithm in detail, and Section 4 presents several experimental settings, comparative analysis. Conclusions and future work are discussed in Section 5.

Section snippets

Background

To generate the depth image of an object, it is necessary to estimate the distance between every point of the object and the camera. Fig. 2 illustrates the schematic illustration of microscopic imaging, where f1 and f2 indicate the focal lengths of the lenses L1 and L2, u is the distance of the object from the lens, and v is the distance of the magnified image from the lens. The relationship between f1, u and v is given by the Gaussian lens law: 1/f1=1/u+1/v. The object AB is located near the

The proposed approach

Most established multifocus image fusion methods provide better visual perception and quality. However, these methods usually use a large number of image processing techniques, which are not conducive to accurately extract the depth information of image sequences. In this paper, a new 3D shape reconstruction scheme is proposed to search the depth maps that represent the best-focused image during the image fusion process. Briefly, it takes three steps to implement the proposed method. First, we

Datasets

Both simulated and real objects are used to verify the effectiveness of the proposed 3D shape reconstruction method. The first simulated sequence consists of 100 images of 360 × 360 pixels, and the corresponding image generation algorithm of this sequence can be found in the previous study [16]. There are three other image sequences with respect to microscopic objects with varying textures and structures. The second and third sequences, which have concave structures, come from the intaglio

Conclusion

The NSST-based microscopic multifocus fusion method for 3D shape reconstruction has been introduced in this paper. It is based on the existence of depth information in the fused image. Due to the combination of depth estimates and fusion processes by using a novel multidirectional modified Laplacian operator, the proposed method can directly identify the depth map and fused image simultaneously. Future work on this topic will focus on three aspects:

  • (1)

    SFF scheme based on multifactor collaborative

Acknowledgements

This work is supported by National Key R&D Program of China (No. 2018YFB1004300), the National Natural Science Foundation of China (Nos. 61672332 and 61872226), the Key R&D Program of Shanxi Province, China (No. 201803D421012), the Scientific and Technological Innovation Programs of Higher Education Institutions in Shanxi, China (No. 2019L0100)

Tao Yan (corresponding author) was received the Ph.D. degree from Chengdu Institute of Computer Applications, Chinese Academy of Science. He is now a lecturer at Shanxi University. His research interests include image processing and evolutionary computation.

References (44)

  • M. Nejati et al.

    Surface area-based focus criterion for multi-focus image fusion

    Inf. Fusion

    (2017)
  • S. Li et al.

    Pixel-level image fusion: a survey of the state of the art

    Inf. Fusion

    (2017)
  • G. Pajares et al.

    A wavelet-based image fusion tutorial

    Pattern Recognit.

    (2004)
  • G. Easley et al.

    Sparse directional image representations using the discrete shearlet transform

    Appl. Comput. Harmon. Anal.

    (2008)
  • C.Y. Wee et al.

    Measure of image sharpness using eigenvalues

    Inf. Sci.

    (2007)
  • R. Rahmat et al.

    3D shape from focus using LULU operators and discrete pulse transform in the presence of noise

    J. Vis. Commun. Image Represent.

    (2013)
  • W. Huang et al.

    Evaluation of focus measures in multi-focus image fusion

    Pattern Recognit. Lett.

    (2007)
  • Y. Liu et al.

    Multi-focus image fusion with a deep convolutional neural network

    Inf. Fusion

    (2017)
  • Z. Zhou et al.

    Multi-scale weighted gradient-based fusion for multi-focus images

    Inf. Fusion

    (2014)
  • Q. Zhang et al.

    Multifocus image fusion using the nonsubsampled contourlet transform

    Signal Process.

    (2009)
  • Y. Liu et al.

    A general framework for image fusion based on multi-scale transform and sparse representation

    Inf. Fusion

    (2015)
  • Y. Yang et al.

    Latent max-margin multitask learning with skelets for 3D action recognition

    IEEE Trans. Cybern.

    (2017)
  • Cited by (35)

    • Shape from focus using gradient of focus measure curve

      2023, Optics and Lasers in Engineering
      Citation Excerpt :

      Based on the imaging principle of applied optics, shape from focus (SFF) analyzes the relationship among image distance, focal length, object distance and focused images to recovery the 3D shape of the object. The SFF technique has been successfully utilized in many applications, 3D camera [1], micro manufacturing [2] and interfacial toughness [3], etc. It is also commonly used in obtaining the all-in-focus image and the surface feature recognition as a common tool of the optical microscope [4].

    • Frequency-domain segmentation algorithm for three-dimensional light-field display based on pixel deviation threshold

      2022, Optics and Laser Technology
      Citation Excerpt :

      By eliminating pixel deviation in the high-frequency part (severe pixel deviation) of the image and maintaining sharpness in the low-frequency part (slight pixel deviation), pixel deviation and sharpness degradation issues could be solved simultaneously. Currently, image processing technology in the frequency-domain has been extensively applied in defect inspection [16], image deblurring [17], and 3D image reconstruction [18], among several others. To our knowledge, image frequency-domain processing techniques have not been used to overcome the problems associated with the pixel deviation in light-field 3D display.

    View all citing articles on Scopus

    Tao Yan (corresponding author) was received the Ph.D. degree from Chengdu Institute of Computer Applications, Chinese Academy of Science. He is now a lecturer at Shanxi University. His research interests include image processing and evolutionary computation.

    View full text