Keywords

1 Introduction

Spectral reflectance or transmittance of objects tends to have unique features that originate from chemical composition or micro-structure of the surface, and thus serve strong clues to identifying the objects. In order to obtain spectral information as images, we need to acquire images for each wavelength of light sampled over the visible range. The resultant data are called spectral images, which are also regarded as a two-dimensional array of spectral data measured at each pixel. The biggest problem in the acquisition of spectral images [1,2,3,4] is that it is inevitable to sacrifice either temporal or spatial resolution in exchange for increasing spectral resolution. A high-speed camera, although it can maintain both temporal and spatial resolution, suffers another problem of low signal-to-noise ratio because the light intensity, which is already weakened by narrow-band imaging, gets even weaker by shortened exposure time.

One of the solutions to this problem is the spectral matching imager [5,6,7], which performs spectral matching in a pixel-parallel manner by use of a correlation camera [8] to produce the correlation between the reference spectral reflectance of a target object and the spectral reflectance of the object in the scene. This system compresses multidimensional spectral information into just a few spectral correlation outputs on the focal plane. It realizes high spectral resolution by performing spectral correlation in the wavelength domain, while maintaining the same temporal and spatial resolution as that of an ordinary video camera, though it cannot measure the whole waveform of spectral reflectance of the object itself.

In this paper, as another solution to the problem with spectral imaging, we propose a system for acquiring the images of complex Fourier coefficients of the spectral reflectance of an object up to the second order at an ordinary frame rate. This remarkable feature is realized by two key components—a correlation camera and a special illumination called sinusoidally-modulated phase-shift spectral illumination (SMPSSI). The correlation camera, previously employed in the spectral matching imager [7] as mentioned above, is comprised of a time-domain correlation image sensor, which produces the temporal correlation between the intensity signal of incident light and external global reference signals pixel by pixel in every frame. The system proposed here is similar to the spectral matching imager to some extent, because it uses a correlation camera and the Fourier coefficients of spectral reflectance obtained from the correlation camera can be regarded as the correlations with sinusoidal reference spectra. The difference, however, lies in the spectral illumination used. The SMPSSI of the proposed system consists of a sum of spectral illuminations sinusoidally modulated at different frequencies with wavelength-linear phase shifts, while the spectral matching imager used a monochromatic illumination with a time-linear peak wavelength. The proposed system thus has an advantage of higher signal-to-noise ratio than that of the spectral matching imager because of much broader spectral power distribution of the SMPSSI and the temporal correlation with sinusoidal reference signals, for which the correlation camera achieves the highest accuracy. It can also be applied to spectral transmittance, of course, though the primary target of the current study is spectral reflectance.

Fourier representation of spectral reflectance has been mainly discussed in the context of color constancy models in human vision [9] and linear models for estimating spectral reflectance [10]. It has hardly been used, however, in most practical methods of spectral image processing. Meanwhile, Jia et al. recently proposed Fabry-Perot-type narrow-band filters with sinusoidal spectral transmittance [11], and developed a spectral imaging system with a mosaic array of the sinusoidal filters [12]. The use of filters with sinusoidal spectral transmittance can be regarded as a way of realizing Fourier transform during image acquisition. Their method, however, is inspired by Fourier transform spectroscopy and thus different from our system in three points. Firstly, they aim at the spectral power distribution of incident light itself coming directly from light sources or indirectly from reflective objects, not at the spectral reflectance of objects as in our method. Secondly, they do not use an active spectral illumination such as the SMPSSI used in our system. Thirdly, Fourier transform is not performed in the wavelength domain as in our method, but in the wavenumber domain.

The rest of this paper describes the principle of the proposed method, depicts the constructed imaging system, and shows experimental results obtained on the system.

2 Principle

2.1 Correlation Camera

The correlation camera has an input of external reference signals, which are supplied to all of the pixels. Let the frame time and its associated angular frame frequency denoted by T and \(\omega _0=2\pi /T\), respectively, and the intensity signal of incident light at a pixel (x, y) and a time t denoted by f(x, y, t). In this study, we supply a pair of sinusoidal reference signals \(\cos k\omega _0t\) and \(-\sin k\omega _0t\) (\(k=1,2,\ldots \)) to the correlation camera. The camera then produces the temporal correlation \(g_k(x,y)\) between f(x, y, t) and the complex sinusoidal reference signal \(e^{-jk\omega _0t}=\cos k\omega _0t-j\sin k\omega _0t\),

$$\begin{aligned} g_k(x,y) = \int _{-T/2}^{T/2} f(x,y,t) e^{-jk\omega _0t} dt, \end{aligned}$$
(1)

as well as the average intensity \(g_0(x,y)\) (the same output as from an ordinary video camera),

$$\begin{aligned} g_0(x,y) = \int _{-T/2}^{T/2} f(x,y,t) dt, \end{aligned}$$
(2)

at each pixel (x, y), and outputs them as images in every frame T. Note that Eq. (1) treats the correlations with the pair of reference signals as a single complex number. In the following, we will omit the pixel coordinates (x, y) because no spatial operations are involved in Eqs. (1) and (2).

2.2 Sinusoidally-Modulated Phase-Shift Spectral Illumination

The object is illuminated by sinusoidally-modulated phase-shift spectral illumination (SMPSSI). SMPSSI is defined with its spectral power distribution \(E(\lambda ,t)\) as

$$\begin{aligned} E(\lambda ,t) = E_0(\lambda ) \left[ C + \sum _{n=1}^N \cos ( n\omega _0t - \phi _n(\lambda ) ), \right] \end{aligned}$$
(3)

where \(\phi _n(\lambda )\) denotes the phase shift linear to wavelength \(\lambda \)

$$\begin{aligned} \phi _n(\lambda ) = 2\pi n\frac{\lambda -\lambda _{\min }}{\lambda _{\max }-\lambda _{\min }}, \end{aligned}$$
(4)

\([\lambda _{\min },\lambda _{\max }]\) the effective wavelength range, \(E_0(\lambda )\) the maximal spectral power distribution in the absence of modulation, and C a constant satisfying

$$\begin{aligned} C \ge -\min _{t,\lambda } \sum _{n=1}^N \cos ( n\omega _0t - \phi _n(\lambda ) ) \end{aligned}$$

to guarantee the physical constraint \(E(\lambda ,t)\ge 0\). Equation (3) implies that each spectral component at wavelength \(\lambda \) is modulated by a sum of sinusoidal carriers with integer multiples of the angular frame frequency \(\omega _0\). Note that, in the limit of \(N\rightarrow \infty \) in Eq. (3), we have

$$\begin{aligned} E(\lambda ,t) \sim E_0(\lambda ) \left[ C - \frac{1}{2} + \frac{T}{2} \sum _{m=-\infty }^{\infty } \delta \left( t-\frac{\lambda -\lambda _{\min }}{\lambda _{\max }-\lambda _{\min }}T-mT \right) \right] , \end{aligned}$$
(5)

where \(\delta (t)\) denotes Dirac’s \(\delta \)-function. Equation (5) implies that \(E(\lambda ,t)\) approaches to the monochromatic illumination used in the spectral matching imager [5, 7], the peak wavelength of which is swept linearly to time t from \(\lambda _{\min }\) to \(\lambda _{\max }\) in every frame.

2.3 Fourier Coefficient Imaging for Spectral Reflectance

Consider imaging an object with spectral reflectance \(R(\lambda )\) under the SMPSSI in Eq. (3) by the correlation camera. We set the spectral power distribution \(E_0(\lambda )\) inversely proportional to the spectral sensitivity of the correlation camera \(S(\lambda )\), i.e. \(E_0(\lambda )=K/S(\lambda )\) with a constant K. The intensity signal f(t) of the incident light received by the correlation camera via \(S(\lambda )\) is then expressed as

$$\begin{aligned} f(t)= & {} \int _{\lambda _{\min }}^{\lambda _{\max }} E(\lambda ,t) R(\lambda ) S(\lambda ) d\lambda \nonumber \\= & {} K \int _{\lambda _{\min }}^{\lambda _{\max }} d\lambda R(\lambda ) \left[ C + \sum _{n=1}^N \cos ( n\omega _0t - \phi _n(\lambda ) ) \right] . \end{aligned}$$
(6)

Note that the spectral sensitivity \(S(\lambda )\) is canceled out by \(E_0(\lambda )\). For f(t), each pixel of the correlation camera produces the temporal correlation \(g_k\) with a complex sinusoidal reference signal \(e^{-jk\omega _0t}\) (\(k=1,2,\ldots \)) as well as the average intensity \(g_0\) over a frame time T as

$$\begin{aligned} g_0= & {} \int _{-T/2}^{T/2} f(t) dt \nonumber \\= & {} \int _{-T/2}^{T/2} dt \cdot K \int _{\lambda _{\min }}^{\lambda _{\max }} d\lambda R(\lambda ) \left[ C + \sum _{n=1}^N \cos \left( n\omega _0t - \phi _n(\lambda ) \right) \right] \nonumber \\= & {} KCT \int _{\lambda _{\min }}^{\lambda _{\max }} d\lambda R(\lambda ) \nonumber \\\equiv & {} KCT(\lambda _{\max }-\lambda _{\min }) \mathcal{R}_0, \end{aligned}$$
(7)
$$\begin{aligned} g_k= & {} \int _{-T/2}^{T/2} f(t) e^{-jk\omega _0t} dt \nonumber \\= & {} \int _{-T/2}^{T/2} dt e^{-jk\omega _0t} \cdot K \int _{\lambda _{\min }}^{\lambda _{\max }} d\lambda R(\lambda ) \left[ C + \sum _{n=1}^N \cos \left( n\omega _0t - \phi _n(\lambda ) \right) \right] \nonumber \\= & {} \frac{KT}{2} \int _{\lambda _{\min }}^{\lambda _{\max }} d\lambda R(\lambda ) \sum _{n=1}^N (e^{-j\phi _n(\lambda )}\delta _{kn} + e^{j\phi _n(\lambda )}\delta _{k,-n}) \nonumber \\= & {} \frac{KT}{2} \int _{\lambda _{\min }}^{\lambda _{\max }} d\lambda R(\lambda ) \exp \left[ -j2\pi k\frac{\lambda -\lambda _{\min }}{\lambda _{\max }-\lambda _{\min }} \right] \nonumber \\\equiv & {} \frac{KT}{2}(\lambda _{\max }-\lambda _{\min }) \mathcal{R}_k, \end{aligned}$$
(8)

where \(\mathcal{R}_k\) (\(k=0,1,2,\ldots ,N\)) denotes the Fourier coefficients of \(R(\lambda )\) over \(\lambda _{\min }\le \lambda \le \lambda _{\max }\),

$$\begin{aligned} \mathcal{R}_k= & {} \frac{1}{\lambda _{\max }-\lambda _{\min }} \int _{\lambda _{\min }}^{\lambda _{\max }} R(\lambda ) \exp \left[ -j2\pi k\frac{\lambda -\lambda _{\min }}{\lambda _{\max }-\lambda _{\min }} \right] d\lambda . \end{aligned}$$
(9)

Equations (7) and (8) imply that \(g_0\) and \(g_k\) are respectively equivalent to the 0-th and k-th Fourier coefficients of the spectral reflectance of the object, and are output from each individual pixel in every frame.

Note that it is also possible to obtain Fourier coefficients of a spectral reflectance \(\rho (\kappa )=R(\lambda )\frac{d\lambda }{d\kappa }\) represented as a function of wavenumber \(\kappa =2\pi /\lambda \) by shifting the modulation phase, instead of \(\phi _n(\lambda )\) in Eq. (3), by \(\psi _n(\kappa )=2\pi (\kappa -\kappa _{\min })/(\kappa _{\max }-\kappa _{\min })\), where \(\kappa _{\max }=2\pi /\lambda _{\min }\), \(\kappa _{\min }=2\pi /\lambda _{\max }\). The spectrum of the SMPSSI under the wavenumber-linear phase shift \(\psi _n(\kappa )\), however, is less easy to understand than that under the wavelength-linear phase shift \(\phi _n(\lambda )\). Spectra in the visible range are usually measured as functions of wavelength \(\lambda \), instead of wavenumber \(\kappa \), by a spectrometer. The spectrum of the SMPSSI \(E(\lambda ,t)\) in Eq. (3) is observed as a sum of traveling sinusoidal waves with the same velocity toward longer wavelengths with an envelope \(E_0(\lambda )\). The observed \(E(\lambda ,t)\) is not sinusoidal under the wavenumber-linear phase shift \(\psi _n(\kappa )\). Moreover, it is difficult to relate Fourier coefficients defined for a wavenumber-domain spectrum \(\rho (\kappa )\) to its wavelength-domain waveform \(R(\lambda )\).

Fig. 1.
figure 1

A constructed system for imaging Fourier coefficients of spectral reflectance.

Full size image
Table 1. Specifications of the digital correlation camera.
Full size table

3 System Construction

3.1 Digital Correlation Camera

We constructed an imaging system for Fourier coefficients of spectral reflectance as shown in Fig. 1. We used a correlation camera of the digital accumulation type (DCI1 by Reglus Co., Ltd.), the specifications of which are listed in Table 1. DCI1 first captures a consecutive sequence of 32 image frames at 1,000 fps with the embedded image sensor, then produces the average intensity image \(g_0(x,y)\) by simply accumulating them, and the temporal correlation image \(g_k(x,y)\) by accumulating them with weights of the complex sinusoidal reference signals \(\cos k\omega _0t\) and \(\sin k\omega _0t\) for the real and imaginary parts, respectively, and finally outputs \(g_0(x,y)\) and \(g_k(x,y)\) at the frame rate of 31.25 fps. DCI1 can distribute reference signals of one to four different frequencies to every \(2\times 2\) pixels as a unit. This implies that, for example, we can acquire four complex correlation images \(g_k(x,y)\) (\(k=1,2,3,4\)) to the maximum, but suffer the lowest resolution (half of the full resolution). In Sect. 4, as a compromise, we acquire up to the second-order complex correlation images \(g_1(x,y)\), \(g_2(x,y)\) in a checkerboard manner with \(1/\sqrt{2}\) of the full resolution. Each low-resolution correlation image is interpolated by discrete Fourier transform. The camera also provides the frame synchronization signal as a trigger for the SMPSSI.

3.2 Programmable Spectral Light Source for SMPSSI

We implemented the SMPSSI on a programmable spectral light source (Optronic Laboratories Inc. OL490) [13], which we employed in the spectral matching imager [7]. In OL490, the light from a Xe lamp is diffracted onto the DMD (digital micromirror device), which consists of a two-dimensional array of micromirror pixels to switch the direction of reflected light in a binary manner, and the reflected light is collected and output through optics. We can change the spectral power distribution of the output light at a high speed by writing a sequence of binary images of “wavelength” and “intensity” dimensions on the DMD. The output light illuminates the object in a \(5\times 5\) cm\(^2\) area with almost uniform spectral irradiance.

Table 2. Specifications of the programmable spectral light source.
Full size table

Table 2 shows the specifications of OL490. We limited the wavelength range to 450–700 nm from the original 380–780 nm, because the output light intensity is very low in shorter and longer wavelength regions and the sensitivity of the correlation camera is also low at shorter wavelengths. The switching rate for spectral power distribution amounts to 250 times for every trigger input from the correlation camera (\(31.25 \text{ fps }\times 250=7812.5\approx 7830 \text{ fps }\)). The number of modulation frequencies was chosen to \(N=2\). These settings delay the modulation phase by \(2\pi /250\) at \(\omega _0\) and by \(4\pi /250\) at \(2\omega _0\) for each 1-nm increment of wavelength, making the spectral power distribution appear to travel toward longer wavelengths in every frame.

The maximal spectral power distribution in the absence of modulation, \(E_0(\lambda )\), must be adjusted to satisfy \(E_0(\lambda )=K/S(\lambda )\). To realize this, we acquired a sequence of images of a white reflectance standard with a flat spectral reflectance under monochromatic illumination from OL490 while its peak wavelength was swept. We attained the condition \(E_0(\lambda )=K/S(\lambda )\) after adjusting three times the intensity of the monochromatic illumination to equalize the image intensity of the reflectance standard, which is proportional to \(E_0(\lambda )S(\lambda )\), for all peak wavelengths.

The spectral power distribution of the SMPSSI after adjusting \(E_0(\lambda )\) is plotted in Fig. 2. It is confirmed that the spectral waveform travels toward longer wavelengths over time. It is also observed that the spectral envelope monotonically decreases toward longer wavelength, which is the direct result of adjusting \(E_0(\lambda )\).

Fig. 2.
figure 2

Measured spectral power distribution of the SMPSSI.

Full size image
Fig. 3.
figure 3

Experimental objects consisting of DIC color guide chips.

Full size image
Fig. 4.
figure 4

Captured images of the objects in Fig. 3. (a) Average intensity \(g_0(x,y)\). (b) Temporal correlation \(g_1(x,y)\) with \(e^{-j\omega _0t}\). (c) Temporal correlation \(g_2(x,y)\) with \(e^{-j2\omega _0t}\). (d) Hue representation of argument.

Full size image
Fig. 5.
figure 5

(a) Magnitude and (b) argument of \(g_1(x,y)\) and \(g_2(x,y)\) averaged within each color chip region, along with the Fourier coefficients \(\mathcal{R}_1\) and \(\mathcal{R}_2\) computed from the spectral reflectances of the color chips. (c) Magnitude error and (d) argument error of the averages of \(g_1(x,y)\) and \(g_2(x,y)\) from \(\mathcal{R}_1\) and \(\mathcal{R}_2\). (Color figure online)

Full size image
Fig. 6.
figure 6

Spectral reflectances of color chips #14, 31, 39, 52, 59 and 90, measured (black), reconstructed from \(g_1(x,y)\) only (red), reconstructed from \(g_1(x,y)\) and \(g_2(x,y)\) (blue), approximated from \(\mathcal{R}_0\) and \(\mathcal{R}_1\) (green), and approximated from \(\mathcal{R}_0\), \(\mathcal{R}_1\) and \(\mathcal{R}_2\) (magenta). (Color figure online)

Full size image

4 Experimental Results

4.1 Image Acquisition

We conducted experiments for the objects shown in Fig. 3. They are three discs in each of which 30 wedge-shaped color chips, selected out of DIC Color Guide Part 1 [14] with high Munsell chroma, are circularly placed side by side with their hue varying gradually. Figure 4 shows the images of the objects illuminated in the central \(5\times 5\) cm\(^2\) region by the SMPSSI and captured from an oblique direction by the correlation camera. Figure 4(a), (b) and (c) respectively correspond to the average intensity image \(g_0(x,y)\), and the temporal correlation images \(g_1(x,y)\) and \(g_2(x,y)\) with complex sinusoidal reference signals \(e^{-j\omega _0t}\) and \(e^{-j2\omega _0t}\). \(g_1(x,y)\) and \(g_2(x,y)\), which are complex-valued, have their magnitude represented in brightness, and their argument in hue according to the chart in Fig. 4(d). The horizontal order of the images in Fig. 4(a)–(c) coincides with that in Fig. 3(a)–(c).

4.2 Error Analysis

Figure 5(a) and (b) respectively plot the magnitude and argument of \(g_1(x,y)\) and \(g_2(x,y)\) averaged within each color chip region of the objects, along with the Fourier coefficients \(\mathcal{R}_1\) and \(\mathcal{R}_2\) computed from the spectral reflectances of the color chips. Figure 5(c) and (d) respectively plot the errors of the averaged magnitude and argument of \(g_1(x,y)\) and \(g_2(x,y)\) from those of \(\mathcal{R}_1\) and \(\mathcal{R}_2\). The abscissas represent the index of the color chips. The magnitudes \(|g_1(x,y)|\) and \(|g_2(x,y)|\) are scaled to the Fourier coefficients by a factor \(\mathcal{R}_0/g_0(x,y)\). The Fourier coefficients \(\mathcal{R}_0\), \(\mathcal{R}_1\) and \(\mathcal{R}_2\) are computed in the same range of 450–700 nm as that of the SMPSSI.

From Fig. 5(b) and (d), we observe that \(\angle g_1(x,y)\) agrees well to \(\angle \mathcal{R}_1\), while the errors of \(\angle g_2(x,y)\) from \(\angle \mathcal{R}_2\) are larger than those of \(\angle g_1(x,y)\) from \(\angle \mathcal{R}_1\), especially for the color chips with very small magnitude \(|g_2(x,y)|\). In this case, the argument \(\angle g_2(x,y)\) is not well-defined and thus vulnerable to noise. We also find that \(\angle g_1(x,y)\) and \(\angle \mathcal{R}_1\) monotonically increase in Fig. 5(b). This agrees well to the gradual change in hue of the color chips as well as to the peak shift in the spectral reflectance of the color chips toward shorter wavelengths.

4.3 Reconstruction of Spectral Reflectance

Figure 6 plots the spectral reflectances of color chips #14, 31, 39, 52, 59 and 90, which all have large errors in either magnitude or argument of \(g_1(x,y)\) and \(g_2(x,y)\) in Fig. 5(c), (d). The black, red, blue, green and magenta curves respectively represent the spectrum that was measured, reconstructed from \(g_1(x,y)\) only, reconstructed from \(g_1(x,y)\) and \(g_2(x,y)\), approximated from \(\mathcal{R}_0\) and \(\mathcal{R}_1\), or approximated from \(\mathcal{R}_0\), \(\mathcal{R}_1\) and \(\mathcal{R}_2\). The reconstruction and approximation were done in the same range of 450–700 nm as that of the SMPSSI. In all of the plots, we see that the reconstructed spectra agree well to the approximates, and that the whole waveform of the reconstructed spectra gets closer to that of the measured spectra when both \(g_1(x,y)\) and \(g_2(x,y)\) were incorporated than when only \(g_1(x,y)\) was used.

5 Conclusion

We proposed a system for acquiring the images of complex Fourier coefficients of the spectral reflectance of an object up to the second order at an ordinary frame rate. We developed an experimental system with a digital correlation camera and a programmable spectral light source to realize the sinusoidally-modulated phase-shift spectral illumination. Experimental results on color guide chips confirm that the proposed system extracted the Fourier coefficients up to the second order accurately, and that the spectra reconstructed from the output images approximated well the measured spectra of the color chips.