Separation of Transmitted Light and Scattering Components in Transmitted Microscopy

Abstract

In transmitted light microscopy, a specimen tends to be observed as unclear. This is caused by a phenomenon that an image sensor captures the sum of these scattered light rays traveled from different paths due to scattering. To cope with this problem, we propose a novel computational photography approach for separating directly transmitted light from the scattering light in a transmitted light microscope by using high-frequency lighting. We first investigated light paths and clarified what types of light overlap in transmitted light microscopy. The scattered light can be simply represented and removed by using the difference in observations between focused and unfocused conditions, where the high-frequency illumination becomes homogeneous. Our method makes a novel spatial multiple-spectral absorption analysis possible, which requires absorption coefficients to be measured in each spectrum at each position. Experiments on real biological tissues demonstrated the effectiveness of our method.

1 Introduction

Transmitted light microscope is widely used in biomedical, food safety, and many other applications. When a specimen is observed through a microscope, light rays hit tiny particles in the specimen and change direction before they reach the image sensor. Consequently, the image sensor captures the sum of these scattered light rays that have traveled along different paths. This scattering of light makes the observed image unclear. Figure 1(b) shows an observed image in transmitted light microscopy, in which scattering media was put on the micrometer (Fig. 1(a)). The scattering makes the image unclear.

This unclearness of an object due to scattering is a critical concern in biomedical imaging. For example, in spatial spectral analysis of biological tissues, the absorption coefficient must be able to be measured accurately at a specific point. The measured signal includes scattered light coming from different points and this causes mixing absorptions at different points in a tissue. To accurately measure the brightness at a specific point in a tissue, it is essential to separate the directly transmitted light coming from that point and the scattered light coming from other points. In this paper, we propose a novel computational photography approach using high-frequency lighting for separating directly transmitted light from scattering components in transmitted light microscopy.

Fig. 1.

Examples of transmitted light microscope image affected by strong scattering.

In the field of computational photography, several methods have been proposed for removing scattering components from reflected [1] and transmitted light through an object [2]. Nayar et al. [1] proposed to use of high-frequency illumination, where a high-frequency spatial pattern is projected onto a scene for separating the directly reflected light, i.e., the direct component, and scattered light including inter-reflections, i.e., the global components, of the scene. This high-frequency illumination method has been extended [3,4,5, 7]. In particular, Tanaka et al. [2] extended Nayar et al.’ s approach so that it can handle transmitted light by using a parallel high-frequency illumination setup. This extension is effective, but making the transmitted light parallel as is required in [2] is difficult in typical transmitted light microscopes; the lens setup makes it impossible for the optical paths of the microscope to be parallel to each other.

In this paper, we propose a feasible approach for separating directly transmitted light and other scattering components of images captured by typical transmitted light microscopes. Since the scattered light paths in the case of transmitted microscopy are different from those in the case of reflectance, we first investigated these light paths and clarified what types of light overlap in transmitted light microscopy. To separate them, we use the difference in observations between focused and unfocused conditions of the camera and illumination, where the high-frequency illumination becomes blurry and quickly homogeneous in an unfocused plane. Based on this fact, the global component can be simply represented and removed. To the best of our knowledge, this is the first attempt to obtain a direct transmitted image in transmitted light microscopy by using high-frequency illumination.

We applied this method to a novel spatial multiple-spectral absorption analysis that requires the absorption coefficients to be measured in each spectrum at each position. Note that while the current multiple-spectral absorption analysis cannot measure the spatial information of absorptions, our method makes this analysis possible (see Sect. 3.2). The results of experiments on real biological tissues demonstrate the effectiveness of our method; we successfully measured the absorption coefficient of each spectrum by separating the direct and scattering components and found that the measured spectral signals were suitable for biomedical research. We believe our method opens new avenues of research towards more precise analyses of biological specimens, such as spatial multiple-spectral absorption analysis.

Fig. 2.

Illustration of (a) microcopy (b) overlapping rays, in the cases of (c) a red point lit, and (d) a red point unlit. The white patch is lit, and the black is unlit.

2 High-Frequency Illumination Microscopy Method

2.1 Overlapping Light Rays Leading to Unclearness

As discussed in the introduction, one reason for unclear images captured in transmitted light microscopy is that the measured signal includes scattered light coming from different points and resultant mixing absorption at different points in a tissue. In this section, we first investigate these light rays passing through an object in transmitted microscopy and clarify what types of light ray overlap and reach the image sensor together.

To investigate all of the light rays that overlap at a sensor, we consider the case of measuring the intensity at a point of interest in transmitted microscopy. The light rays that reach the same sensor can be classified into three types: directly transmitted light, single scattered light from a different depth, and multiply scattered light. Figure 2(b) shows an illustration of the light paths, where the objective point is the red point. First, the red arrow is the directly transmitted light that is only scattered at the point of interest and thereafter reaches the sensor without being scattered. This is the objective to be measured. Second, the green arrow is single scattered light from a different depth that is scattered at the green point located a different depth from the point of interest. This light overlaps the directly transmitted light and reaches the same sensor. Third, the blue arrow is a multiply scattered light that is scattered more than once and consequently overlaps the above two types. The sum of these three types of light is captured at the same image sensor. In this paper, we call directly transmitted light the direct components, and the other two types of scattered light the global components. The global components make images unclear and prevent the direct component from being measured. Therefore, we need to separate these overlapping rays into direct and global components.

2.2 Separation of Direct and Global Component

We separate the direct transmitted light and global components in transmitted light microscopy by using the high-frequency illumination concept. We take advantage of the difference in observations between in focused and unfocused conditions of the camera and illumination. Based on the fact that an image in an unfocused slice appears as a Gaussian blurred version of the in-focus image for both the camera and illumination, we assume that the high-frequency pattern is blurry and becomes homogeneous in unfocused slices, as shown in Fig. 3. This is a key assumption of our method.

Here, we will use the above assumption to describe the concept of the original high-frequency illumination method [1], which separates direct and global components in the case of reflectance observation, in transmitted light microscopy.

First, let us consider the case in which there is no illumination pattern, and the point of interest is in focus in both the camera and illumination, and in the case where illumination is uniform. The image intensity captured at pixel $\varvec{p}$, which corresponds to the point of interest, is the sum of the direct component $D[\varvec{p}]$ and the global component $G[\varvec{p}]$:

$$\begin{aligned} L[\varvec{p}]= & {} D[\varvec{p}] + G[\varvec{p}]. \end{aligned}$$

(1)

We set a high-frequency binary checkered pattern between the light source and the illumination focus lens, where the white patch is lit, and the black is unlit in Fig. 2(c) and (d). The target slice is also in focus in both the camera and the high-frequency checkered pattern. Then, we capture images when the point of interest is lit or unlit in focus by moving the checker pattern. When the point of interest (red point) is lit, the intensity corresponding it in the target slice contains both the direct (red arrow) and global components (green and blue arrows), as shown in Fig. 2(c). On the other hand, when the point of interest is unlit, the light contains only global components (green and blue arrows), since there are no light rays that are scattered at the point, and it directly reaches to the sensor as shown in Fig. 2(d).

Fig. 3.

Examples of (a) an image in camera focus, (b) image out of camera focus, (c) illuminated pattern in illumination focus, and (d) illuminated pattern out of illumination focus. (e) Illustration of two focuses of a camera and illumination in transmitted light microscopy.

Assuming that the illumination light is uniform in the unfocused slices (Fig. 3), when the ratio of the lit area in the checker pattern is 0.5, the light intensity of a point in the unfocused slices is half of that when the illumination is uniform. Thus, the total amount of single scattered light from a different depth and multiply scattered light also becomes half in this case. Using this fact, the intensity $L_{+}[\varvec{p}]$ at a pixel $\varvec{p}$ corresponding with the interesting point in the case when it is lit and the intensity $L_{-}[\varvec{p}]$ when it is unlit can be computed as follows:

$$\begin{aligned} L_{+}[\varvec{p}]= & {} \sum _{\varvec{x}\in \varOmega (\varvec{p},d)} S[\varvec{p},\varvec{x}] + \left( \frac{1}{2}\sum _{\varvec{x} \in \varPsi (\varvec{p},d) - \varOmega (\varvec{p},d)} S[\varvec{p},\varvec{x}] + \frac{1}{2}\sum _{\varvec{x} \in \varPsi (\varvec{p},d) } M[\varvec{p},\varvec{x}] \right) \nonumber \\\end{aligned}$$

(2)

$$\begin{aligned} L_{-}[\varvec{p}]= & {} \frac{1}{2}\sum _{\varvec{x}\in \varPsi (\varvec{p},d)- \varOmega (\varvec{p},d)} S[\varvec{p},\varvec{x}] + \frac{1}{2}\sum _{\varvec{x} \in \varPsi (\varvec{p},d) } M[\varvec{p},\varvec{x}], \end{aligned}$$

(3)

where $S[\varvec{p},\varvec{x}]$ is the intensity of single scattered light at point $\varvec{x}=[x, y, z]^t$, and $M[\varvec{p},\varvec{x}]$ is the multiply scattered light that the last scattering point is $\varvec{x}$. Since the focused depth d has a range, a pixel corresponds to a set of points. We denote that $\varOmega (\varvec{p},d)$ is the area that corresponds with the pixel $\varvec{p}$ and d, and $\varPsi (\varvec{p},d)$ indicates all of the points that correspond with the pixel $\varvec{p}$ in the specimen.

As illustrated in Fig. 2(c), the first term of Eq. 3 is the directly transmitted light at the target slice ($z=d$) illuminated in focus (red point). The second term is single scattered light at different slices illuminated out of focus (green point), and the third term is multiply scattered light (blue point) in the global components. The transmitted light from the target slice and other global component can be calculated as

$$\begin{aligned} D[\varvec{p}]= & {} \sum _{\varvec{x}\in \varOmega (\varvec{p},d)} S[\varvec{p},\varvec{x}] = L_{+}[\varvec{p}] - L_{-}[\varvec{p}] \nonumber \\ G[\varvec{p}]= & {} \sum _{\varvec{x}\in \varPsi (\varvec{p},d) - \varOmega (\varvec{p},d)} S[\varvec{p},\varvec{x}] + \sum _{\varvec{x} \in \varPsi (\varvec{p},d) } M[\varvec{p},\varvec{x}] = 2 L_{-}[\varvec{p}]. \end{aligned}$$

(4)

To obtain the direct and global components for all pixels, a set of intensities at each pixel is recorded by shifting the high-frequency illumination pattern and selecting the minimum and maximum intensity for each point, denoted as $L_{max}$ and $L_{min}$, respectively. In the above case of two images, $L_{+} = L_{max}$, and $L_{-} = L_{min}$. Here, we should note that since a high-frequency checker patch has a size in a real setup, the direct transmitted light, which is only scattered at the point of interest, may be scattered several times in the same small patch. In our method, this light that is scattered a few times in a small area is considered to be in the direct component. The number of scatterings in a small patch depends on the properties of the object and the spatial filter frequency. Thus, we should select the frequency depending on the properties of the object.

Fig. 4.

(a) Transmitted microscopy system, (b) input image with mesh pattern illumination, (c) direct components, and (d) global components.

3 Experimental Results

3.1 Measuring System

We constructed a transmitted microscopy system that uses high-frequency illumination. As shown in Fig. 4, our experimental setup consisted of a microscope, a light source, a high-frequency checker pattern filter, and a camera. We used a microscope (Olympus BX53) with a halogen lamp. For the high-frequency pattern filter, we used a photomask with various binary checker patterns, as shown in Fig. 3(c), where the ratio of the lit and unlit area is 0.5. The sizes of the patterns ranged from $1\times 1$ to $16\times 16$ $\upmu$m and they could be easily changed depending on the target tissue. The illumination lens was set above the filter, which could change the depth of focus of the illumination. To capture all the points in both lit and unlit cases, the high-frequency pattern filter was repeatedly moved on an XYZ-axes motorized stage (Suruga Seiki KXT04015-LC), as shown in Fig. 4(a). The stage shifted the high-frequency pattern the required distance quickly, precisely, and repeatedly. Figure 4(b) shows a image when the high-frequency pattern is lit on a human skin, and Fig. 4(c) and (d) shows the separation results.

3.2 Spatial Multi-spectral Absorption Analysis

In the experiments, our method was applied to a real biological research that measures the absorption coefficients for each wavelength of small parts of melanin tissues in a human skin. These experiments require to spatially measure the absorptions since a skin includes several tissues, such as fibers. This spatial multi-spectral absorption analysis is a novel analysis, and our high-frequency illumination method for transmitted light microscopy makes it possible.¹

To obtain multiple spectral images, we used a multispectral camera (EBA Japan NH-8) whose spatial resolution was 1280$\,\times \,$1024 pixels and whose spectral resolution was 5 nm in the range from 400 nm to 950 nm. The camera was calibrated radiometrically. Note that the camera setup can be changed depending on the application. The specimens were hydrated with water and sealed between a microscopic slide and a coverslip. The depth of a focused slice is 3 $\upmu$m, and the thickness of the tissue is 10 $\upmu$m. For each field of view, 16 images were captured by randomly shifting the high-frequency filter, where the patch size was 16 $\upmu$m. Then, the direct $D(\varvec{p},\lambda )$ and global $G(\varvec{p},\lambda )$ components were calculated for each pixel $\varvec{p}$ and wavelength $\lambda$ by using Eq. 4. We then computed absorption and scattering coefficients for each wavelength on the basis of the Beer-Lambert law [6], in which the relation between incident light intensity $I_0$ and outgoing direct transmitted intensity. The absorption and scattering coefficients can be directly calculated for each wavelength and each pixel.

Fig. 5.

Examples of absorption and scattering coefficients (blue and orange curves, respectively). (a) The solution of food dye and scattering media intralipid (green curve is absorbance of food dye), (b) melanin of human cheek skin, (c) image captured at wavelength 535 nm (the arrow points to the measurement point of (b)), (d) directly transmitted light image and (e) global scattering components image captured at 535 nm.

To evaluate our method, we first examined objects of known absorptions in scattering media intralipid, which have been widely used in light propagation analysis. Figure 5(a) shows the absorption and scattering coefficients of the solution of food dye and scattering media intralipid. The green curve is the scaled optical absorbance of food dye that was measured by a multi-detection microplate reader, which can be considered as groundtruth. The blue curve is measured by our system, which is similar with the green curve. This experiment demonstrated that our method could correctly measure the absorption coefficients by separating the transmitted light and scattering components.

Next, to demonstrate the efficacy of our method in a real biological research, we measured the spatial multi-spectral absorption coefficients of melanin in a human cheek tissue, which are unknown. Figure 5(d) and (e) show the direct and global components at a wavelength of 535 nm, which were separated by our method. Figure 5(b) shows the spectral distribution of melanin in tissue. The point was selected by professional biological researcher as shown in Fig. 5(c). Since there are no other measuring methods with which spatial multi-spectral analysis in transmitted light microscopy, our results were qualitatively compared with research [8] that measured the absorption coefficient of a mass of tissues by using an integrating sphere spectrophotometer and a Monte Carlo simulation. Our results are comparable with these results in that our absorption and scattering coefficients are on a similar order to theirs. Moreover, as in measurements [8], the scattering coefficients of each tissue that we calculated were also larger than the absorption coefficients. Our results indicate that absorption of melanin gradually decreased as the wavelength increased, as shown in Fig. 5(b). These observations correspond to the known properties of melanin. Using our results, we can conduct a further investigation analyzing the spatial distribution of tissues since our method can measure the optical properties at each point in tissues, something which can not be done by the other methods.

4 Conclusion

We proposed a new method for separating of transmitted and scattered light for spectral analysis of translucent objects. We incorporated this method in a high-frequency illumination microscopy system for separating the directly transmitted light from other components. Experimental results using biological tissues validated the accuracy of our method by measuring real absorption properties. We believe our method has the potential to be applied to many other microscopy investigations.