A soft-computing-based approach to artificial visual attention using human eye-fixation paradigm: toward a human-like skill in robot vision

Abstract

Fitting the skills of the natural vision is an appealing perspective for artificial vision systems, especially in robotics applications dealing with visual perception of the complex surrounding environment where robots and humans mutually evolve and/or cooperate, or in a more general way, those prospecting human–robot interaction. Focusing the visual attention dilemma through human eye-fixation paradigm, in this work we propose a model for artificial visual attention combining a statistical foundation of visual saliency and a genetic tuning of the related parameters for robots’ visual perception. The computational issue of our model relies on the one hand on center-surround statistical features’ calculations with a nonlinear fusion of different resulting maps, and on the other hand on an evolutionary tuning of human’s gazing way resulting in emergence of a kind of artificial eye-fixation-based visual attention. Statistical foundation and bottom-up nature of the proposed model provide as well the advantage to make it usable without needing prior information as a comprehensive solid theoretical basement. The eye-fixation paradigm has been considered as a keystone of the human-like gazing attribute, molding the robot’s visual behavior toward the human’s one. The same paradigm, providing MIT1003 and TORONTO image datasets, has served as evaluation benchmark for experimental validation of the proposed system. The reported experimental results show viability of the incorporated genetic tuning process in shoving the conduct of the artificial system toward the human-like gazing mechanism. In addition, a comparison to currently best algorithms used in the aforementioned field is reported through MIT300 dataset. While not being designed for eye-fixation prediction task, the proposed system remains comparable to most of algorithms within this leading group of currently best state-of-the-art algorithms used in the aforementioned field. Moreover, about ten times faster than the currently best state-of-the-art algorithms, the promising execution speed of our approach makes it suitable for an effective implementation fitting requirement for real-time robotics applications.

Appendices

Appendix A-1

Let us suppose the image $I_{YCC}$, represented by its pixels $I_{YCC} \left( x \right)$, in YCrCb color space, where $x\in \mathrm{N}^{2}$ denotes 2D pixel position. Let $I_Y \left( x \right)$, $I_{Cr} \left( x \right)$ and $I_{Cb} \left( x \right)$ be the colors values in channels Y, Cr and Cb, respectively. Similarly, let $I_{RGB} \left( x \right)$ be the same image in RGB color space and $I_R \left( x \right)$, $I_G \left( x \right)$ and $I_B \left( x \right)$ be its colors values in channels R, G and B, respectively. Finally, let $\overline{I_Y }$, $\overline{I_{Cr} }$ and $\overline{I_{Cb} }$ be median values for each channel throughout the whole image.

As a reminder, we recall that GSM is resulted from nonlinear fusion of two elementary maps denoted $M_Y \left( x \right)$ and $M_{CrCb} \left( x \right)$, relating luminance and chromaticity separately (Moreno et al. 2012 and Ramik 2012). Equations (A-1), (A-2) and (A-3) detail the calculation of each elementary map as well as the resulting GSM.

$$\begin{aligned}&M_Y \left( x \right) =\left\| {\overline{I_Y } -I_Y \left( x \right) } \right\| \end{aligned}$$

(A-1)

$$\begin{aligned}&M_{CrCb} \left( x \right) =\sqrt{\left( {\overline{I_{Cr} } -I_{Cr} \left( x \right) } \right) ^{2}+\left( {\overline{I_{Cb} } -I_{Cb} \left( x \right) } \right) ^{2}} \end{aligned}$$

(A-2)

$$\begin{aligned}&M_G \left( x \right) =\frac{1}{1-e^{-C\left( x \right) }} M_{CrCb} \left( x \right) \nonumber \\&\quad +\left( {1-\frac{1}{1-e^{-C\left( x \right) }}} \right) M_Y \left( x \right) \end{aligned}$$

(A-3)

$C\left( x \right)$ is a coefficient, computed accordingly to the equation (A-4) which depends on saturation of each pixel in RGB color space expressed by equation (A-5).

$$\begin{aligned} C\left( x \right)= & {} 10-0.5 C_C \left( x \right) \end{aligned}$$

(A-4)

$$\begin{aligned} C_C \left( x \right)= & {} Max \left( {I_R \left( x \right) , I_G \left( x \right) , I_B \left( x \right) } \right) \nonumber \\&-Min \left( {I_R \left( x \right) , I_G \left( x \right) , I_B \left( x \right) } \right) \end{aligned}$$

(A-5)

In the same way, based on center-surround histograms’ concept involving statistical properties of two centered windows (over each pixel) sliding alongside whole the image (initially proposed in Liu et al. 2007), LSM is also obtained from a nonlinear fusion of two statistical properties-based maps relating luminance and chromaticity of the image (Moreno et al. 2012 and Ramik 2012). Referring to Fig. 12, let $P\left( x \right)$ be a sliding window of sizeP, centered over a pixel located at position x, and let $Q\left( x \right)$ be another surrounding area around the same pixel of size Q, with $Q-P=p^{2}$. Let consider the “center-histogram” $H_C \left( x \right)$ as a histogram of pixels’ intensities in window $P\left( x \right)$ with $h_C \left( {x , k} \right)$ representing the value of kth bin of this histogram. Let consider the “surrounding-histogram” $H_S \left( x \right)$ as a histogram of pixels’ intensities in the surrounding window $Q\left( x \right)$ with $h_S \left( {x , k} \right)$ representing its kth bin’s value. Let define the “center-surround feature” $d_{ch} \left( x \right)$ as a sum of differences between normalized center and surrounding histograms over all 256 histograms’ bins computed for each channel “ch” ($ch\in \left\{ {Y , Cr , Cb} \right\} )$.

Fig. 12

Idea of centered windows sliding alongside an image of size nm

Equation (A-6) details the computation of the so-called “center-surround feature,” where $\left| {H_C \left( x \right) } \right|$ and $\left| {H_S \left( x \right) } \right|$ represent sums over all histogram bins. Concerning pixels near the image’s borders, the areas $P\left( x \right)$ and $Q\left( x \right)$ are limited to the image itself (e.g., not exceeding the image’s borders).

$$\begin{aligned} d_{ch} \left( x \right) =\sum _{k=1}^{256} {\left| {\frac{h_C \left( {x , k} \right) }{\left| {H_C \left( x \right) } \right| }-\frac{h_S \left( {x , k} \right) }{\left| {H_S \left( x \right) } \right| }} \right| } \end{aligned}$$

(A-6)

The LSM, resulting from a nonlinear fusion of the so-called center-surround features, is obtained accordingly to the equation (A-7), where the coefficient $C_\mu \left( x \right)$ represents the average color saturation over the window $P\left( x \right)$, computed as average of saturations $C\left( x \right)$ (please refer to equation (A-6)) using the equation (A-8).

$$\begin{aligned}&M_L \left( x \right) =\frac{1}{1-e^{-C_\mu \left( x \right) }} d_Y \left( x \right) \nonumber \\&\qquad \qquad \quad +\left( {1-\frac{1}{1-e^{-C_\mu \left( x \right) }}} \right) Max \left( { d_{Cr} \left( x \right) , d_{Cb} \left( x \right) } \right) \nonumber \\\end{aligned}$$

(A-7)

$$\begin{aligned}&C_\mu \left( x \right) =\frac{\sum _{l=1}^P {C\left( l \right) } }{P} \end{aligned}$$

(A-8)

Appendix A-2

Table 9 summarizes notations and symbols used in this articles.

Table 9 Notations and symbols used in the article