Keywords

1 Introduction

Image classification and retrieval for color images are two hotspots in pattern recognition. How to extract effective features, which are robust to color variations caused by the changes in the outdoor environment and geometric deformations caused by viewpoint changes, is the key issue. A classical approach is to construct invariant features for color images. Moment invariant is one of them.

Moment invariant was first proposed by Hu in [1]. In 1962, He defined geometric moment and constructed 7 geometric moment invariants which were invariant to the similarity transform (rotation, scaling and translation). Researchers applied them to many fields in pattern recognition and achieved good results [2, 3]. Nearly 30 years later, Flusser et al. constructed affine moment invariants (AMIs) in [4] which are invariant to affine transform. The geometric deformation of an object, which is caused by the viewpoint change, can be represented by projective transform. However, general projective transform is a kind of complex nonlinear transform. So, it’s difficult to construct projective moment invariants. When the distance between the camera and the object is much larger than the size of the object itself, the geometric deformation can be approximated by the affine transform. AMIs have been used in many practical applications [5, 6]. In order to get more AMIs, researchers designed various methods. Suk et al. presented the graph method in [7], which can be used to construct AMIs with arbitrary orders and degrees. Xu et al. proposed the concept of geometric primitives in [8], including distance, area and volume. AMIs can be constructed by using various geometric primitives. This method made the construction of moment invariants has geometric meaning.

The above-mentioned moment invariants are all designed for gray images. With the popularity of color images, invariants for color images began to appear gradually. Researchers tried to construct features which have invariance for geometric deformations and the changes in color space (Fig. 1).

Fig. 1.
figure 1

The changes in color resulting from changes in the outdoor environment. (Color figure online)

Full size image

Geusebroek et al. [9] proved that the affine transform model was the best linear model to simulate changes in color resulting from changes in the outdoor environment. Mindru et al. constructed the moment invariants in [10], which were invariant to shape affine and color diagonal-offset transforms. These invariants were obtained by using the related concepts of Lie group. Some complex partial differential equations had to be solved. Thus, the number of them was limited and difficult to be generalized. Suk et al. [11] put forward affine moment invariants for color images by combining all color channels. But this approach was not intuitive and did not work well for the changes in color space. To solve these problems, Gong et al. [12,13,14] constructed the color primitive by using the concept of geometric primitive proposed in [8]. Combining the color primitive with some shape primitives, moment invariants that were invariant to shape affine and color affine transforms can be obtained easily, which were named shape-color affine moment invariants (SCAMIs). In [14], they obtained 25 SCAMIs which satisfied the independency of functions. However, we find that a large number of SCAMIs with simple structures and good properties are missed in [14].

In this paper, we propose a general structural formula of shape-color primitives by using partial derivatives of each color channel. Then, two special cases of shape-color primitives are used to construct shape-color differential moment invariants (SCDMIs), which are invariant to shape affine and color affine transforms. We find that SCAMIs proposed in [14] is a special case of our method. Finally, commonly used image descriptors and SCDMIs are used to image classification and retrieval for color images, respectively. By comparing the results, we find that SCDMIs get better results.

2 Related Work

In order to construct image features which are robust to color variations and geometric deformations, researchers have made various attempts. Among them, SCAMIs proposed in [14] are worthy of special attention, which have invariance for shape affine and color affine transforms. Two affine transforms are defined by

$$\begin{aligned} \left( \begin{array}{c} x^{'}\\ y^{'}\\ \end{array} \right) =SA \cdot \left( \begin{array}{c} x\\ y\\ \end{array} \right) +ST = \left( \begin{array}{cc} \alpha _{1}&{} \alpha _{2}\\ \beta _{1}&{} \beta _{2}\\ \end{array} \right) \cdot \left( \begin{array}{c} x\\ y\\ \end{array} \right) + \left( \begin{array}{c} O_{x}\\ O_{y}\\ \end{array} \right) \end{aligned}$$
(1)
$$\begin{aligned} \left( \begin{array}{c} R^{'}(x,y)\\ G^{'}(x,y)\\ B^{'}(x,y)\\ \end{array} \right) =CA \cdot \left( \begin{array}{c} R(x,y)\\ G(x,y)\\ B(x,y)\\ \end{array} \right) +CT = \left( \begin{array}{ccc} \ a_{1}&{}a_{2}&{}a_{2}\\ \ b_{1}&{}b_{2}&{}b_{2}\\ \ c_{1}&{}c_{2}&{}c_{2}\\ \end{array} \right) \cdot \left( \begin{array}{c} R(x,y)\\ G(x,y)\\ B(x,y)\\ \end{array} \right) + \left( \begin{array}{c} O_{R}\\ O_{G}\\ O_{B}\\ \end{array} \right) \end{aligned}$$
(2)

where SA and CA are nonsingular matrices.

For the color image I(R(x, y), G(x, y), B(x, y)), let \((x_{p},y_{p}),(x_{q},y_{q}),(x_{r},y_{r})\) be three arbitrary points in the domain of I. The shape primitive and the color primitive are defined by

$$\begin{aligned} S(p,q)= \left| \begin{array}{cc} (x_{p}-\bar{x})&{}(x_{q}-\bar{x})\\ (y_{p}-\bar{y})&{}(y_{q}-\bar{y})\\ \end{array} \right| \end{aligned}$$
(3)
$$\begin{aligned} \begin{aligned}&C(p,q,r)= \left| \begin{array}{ccc} (R(x_{p},y_{p})-\bar{R})&{}(R(x_{q},y_{q})-\bar{R})&{}(R(x_{r},y_{r})-\bar{R})\\ (G(x_{p},y_{p})-\bar{G})&{}(G(x_{q},y_{q})-\bar{G})&{}(G(x_{r},y_{r})-\bar{G})\\ (B(x_{p},y_{p})-\bar{B})&{}(B(x_{q},y_{q})-\bar{B})&{}(B(x_{r},y_{r})-\bar{B})\\ \end{array} \right| \end{aligned} \end{aligned}$$
(4)

where \(\bar{X}\) represents the mean value of X, \(X \in \{x,y,R,G,B\}\). Then, using (3), the shape core (sCore) can be defined by

$$\begin{aligned} sCore(n,m;d_{1},d_{2},...,d_{n})=\underbrace{S(1,2)S(k,l)...S(r,n)}_m\ \end{aligned}$$
(5)

where n and m represent that the sCore is the product of m shape primitives which are constructed by n points \((x_{1},y_{1}),(x_{2},y_{2}),...,(x_{n},y_{n})\). \(k<l\), \(r<n\), \(k,l,r \in \left\{ 1,2,...n\right\} \). \(d_{i}\) represents the number of times that the point \((x_{i},y_{i})\) occurs in all shape primitives, \(i=1,2,...,n\).

Similarly, the color core (cCore) can be defined by

$$\begin{aligned} cCore(N,M;D_{1},D_{2},...,D_{N})=\underbrace{C(1,2,3)C(G,K,L)...C(P,Q,N)}_M\ \end{aligned}$$
(6)

Where N and M represent that the cCore is the product of M color primitives which are constructed by N points \((x_{1},y_{1}),(x_{2},y_{2}),...,(x_{N},y_{N})\). \(G<K<L\), \(P<Q<N\), \(G,K,L,P,Q \in \left\{ 1,2,...N\right\} \). \(D_{i}\) represents the number of times that the point \((x_{i},y_{i})\) occurs in all color primitives, \(i=1,2,...,N\).

Suppose the color image I(R(x, y), G(x, y), B(x, y)) is transformed into the image \(I^{'}(R^{'}(x^{'},y^{'}),G^{'}(x^{'},y^{'}),B^{'}(x^{'},y^{'}))\) by two transforms defined by (1) and (2). \((x^{'}_{p},y^{'}_{p}),(x^{'}_{q},y^{'}_{q})\) and \((x^{'}_{r},y^{'}_{r})\) in \(I^{'}\) are the corresponding points of \((x_{p},y_{p}),(x_{q},y_{q})\) and \((x_{r},y_{r})\) in I. Gong et al. [14] have proved

$$\begin{aligned} S^{'}(p,q)=|SA| \cdot S(p,q) \end{aligned}$$
(7)
$$\begin{aligned} C^{'}(p,q,r)=|CA| \cdot C(p,q,r) \end{aligned}$$
(8)

Further results can be concluded that

$$\begin{aligned} sCore^{'}(n,m;d_{1},d_{2},...,d_{n})=|SA|^{m}\cdot sCore(n,m;d_{1},d_{2},...,d_{n}) \end{aligned}$$
(9)
$$\begin{aligned} cCore^{'}(N,M;D_{1},D_{2},...,D_{N})=|CA|^{M}\cdot cCore(N,M;D_{1},D_{2},...,D_{N}) \end{aligned}$$
(10)

Therefore, the SCAMIs can be constructed by

$$\begin{aligned} \begin{aligned}&SCAMIs(n,m,N,M;d_{1},...,d_{n};D_{1},...,D_{N})\\ {}&= \frac{In(sCore(n,m,d_{1},...,d_{n})\cdot cCore(N,M,D_{1},...,D_{N}))}{In(sCore(1,0))^{max(n+N)+m-\frac{3M}{2}}\cdot In(cCore(3,2;2,2,2))^{\frac{M}{2}}}\\ \end{aligned} \end{aligned}$$
(11)

where In(X) means multiple integrals. Then there is a relation

$$\begin{aligned} \begin{aligned} SCAMIs^{'}(n,m,N,M;d_{1},...,d_{n};D_{1},...,D_{N})&\\=SCAMIs(n,m,N,M;d_{1},...,d_{n};D_{1},...,D_{N}) \end{aligned} \end{aligned}$$
(12)

It must be said that \(\max \{n,N\}\), \(\max \limits _{i}\{d_{i}\}\) and \(\max \limits _{i}\{D_{i}\}\) are named the degree, the shape order and the color order of SCAMIs, respectively. In fact, (11) can be expressed as the polynomial of shape-color moments. This kind of moments was first proposed in [16] and defined by

$$\begin{aligned} SCM_{pq\alpha \beta \gamma }=\iint (x-\bar{x})^{p}(y-\bar{y})^{q}(R(x,y)-\bar{R})^{\alpha }(G(x,y)-\bar{G})^{\beta }(B(x,y)-\bar{B})^{\gamma }dxdy \end{aligned}$$
(13)

Gong et al. [14] pointed out that they constructed all SCAMIs of which degrees \(\leqslant 4\), shape orders \(\leqslant 4\) and color orders \(\leqslant 2\). They obtained 24 SCAMIs which are functional independencies using the method proposed by Brown [17]. However, we will point out in Sect. 3 that they omitted many simple and well-behaved SCAMIs.

3 The Structural Framework of SCDMIs

In this section, we introduce the general definitions of shape-color differential moment and shape-color primitive, firstly. Then, using the shape-color primitive, the shape-color core can be constructed. Finally, according to (11) and the shape-color core, we obtain the general structural formula of SCDMIs. Also, 50 instances of SCDMIs are given for experiments in Sect. 4.

3.1 The Definition of General Shape-Color Moment

Definition 1

Suppose the color image I(R(x, y), G(x, y), B(x, y)) has the k-order partial derivatives \((k=0,1,2,...)\). The general shape-color differential moment is defined by

$$\begin{aligned} \begin{aligned} SCM^{k}_{pq\alpha \beta \gamma }=\iint&(x-\bar{x})^{p}(y-\bar{y})^{q}(R^{(k)}(x,y)-\bar{R}^{\delta (k)})^{\alpha }(G^{(k)}(x,y)-\bar{G}^{\delta (k)})^{\beta }\\ {}&(B^{(k)}(x,y)-\bar{B}^{\delta (k)})^{\gamma }dxdy\\ \end{aligned} \end{aligned}$$
(14)

where \((R^{(k)}(x,y)\), \(G^{(k)}(x,y), B^{(k)}(x,y))\) represent the k-order partial derivatives of (R(x, y), G(x, y), B(x, y)). \(\bar{R}, \bar{G}, \bar{B}\) represent the mean values of R, G, B. \(\delta (k)\) is the impact function.

$$\begin{aligned} \delta (k)=\left\{ \begin{aligned} 1&~&(k=0) \\ 0&~&(k\ne 0)\\ \end{aligned} \right. \end{aligned}$$
(15)

We can find that (13) and (14) are identical, when \(k=0\). Therefore, the shape-color moment is a special case of the general shape-color differential moment.

3.2 The Construction of General Shape-Color Primitive

Definition 2

Suppose the color image I(R(x, y), G(x, y), B(x, y)) has the k-order partial derivatives \((k=0,1,2,...)\). \((x_{p},y_{p}),(x_{q},y_{q}),(x_{r},y_{r})\) are three arbitrary points in the domain of I. The general shape-color primitive is defined by

$$\begin{aligned} \begin{aligned} SCP_{k}(p,q,r)= \left| \begin{array}{ccc} F^{k}_{R}(x_{p},y_{p})&{}F^{k}_{R}(x_{q},y_{q})&{}F^{k}_{R}(x_{r},y_{r})\\ F^{k}_{G}(x_{p},y_{p})&{}F^{k}_{G}(x_{q},y_{q})&{}F^{k}_{G}(x_{r},y_{r})\\ F^{k}_{B}(x_{p},y_{p})&{}F^{k}_{B}(x_{q},y_{q})&{}F^{k}_{B}(x_{r},y_{r})\\ \end{array} \right| \end{aligned} \end{aligned}$$
(16)

where

$$\begin{aligned} F^{k}_{C}(x,y)=\sum _{i=0}^{k}\left( {\begin{array}{c}k\\ i\end{array}}\right) (x-\bar{x})^{i}(y-\bar{y})^{k-i}\frac{\partial ^{k}C(x,y)}{\partial x^{i}\partial y^{k-i}}, ~~~C \in \{R,G,B\}. \end{aligned}$$
(17)

We can find that C(p, q, r) defined by (4) is a special case of \(SCP_{k}(p,q,r)\), when \(k=0\).

3.3 The Construction of General Shape-Color Core

Definition 3

Using Definition 2, the general shape-color core (scCore) is defined by

$$\begin{aligned} scCore_{k}(N,M;D_{1},D_{2},...,D_{N})=\underbrace{SCP_{k}(1,2,3)SCP_{k}(G,K,L)...SCP_{k}(P,Q,N)}_M\ \end{aligned}$$
(18)

Where \(k=1,2,...\), N and M represent that the \(scCore_{k}\) is the product of M shape-color primitives constructed by N points \((x_{1},y_{1}),(x_{2},y_{2}),...,(x_{N},y_{N})\). \(G<K<L\), \(P<Q<N\), \(G,K,L,P,Q \in \left\{ 1,2,...N\right\} \). \(D_{i}\) represents the number of times that the point \((x_{i},y_{i})\) occurs in all shape-color primitives, \(i=1,2,...,N\).

Obviously, \(cCore(N,M;D_{1},D_{2},...,D_{N})\) defined by (6) is a special case of \(scCore_{k}(N,M;D_{1},D_{2},...,D_{N})\), when \(k=0\).

3.4 The Construction of SCDMIs

Theorem 1

Let the color image I(R(x, y), G(x, y), B(x, y)) be transformed into the image \(I^{'}(R^{'}(x^{'},y^{'}),G^{'}(x^{'},y^{'}),B^{'}(x^{'},y^{'}))\) by (1) and (2), \((x_{p}^{'},y_{p}^{'}),(x_{q}^{'},y_{q}^{'})\) and \((x_{r}^{'},y_{r}^{'})\) in \(I^{'}\) are the corresponding points of \((x_{p},y_{p}),(x_{q},y_{q})\) and \((x_{r},y_{r})\) in I, respectively. Suppose that R(x, y), G(x, y), B(x, y), \(R^{'}(x^{'},y^{'}), G^{'}(x^{'},y^{'})\), \(B^{'}(x^{'},y^{'})\) have the k-order partial derivatives \((k=0,1,2,...)\). Then there is a relation

$$\begin{aligned} SCP_{k}^{'}(p,q,r)=|CA| \cdot SCP_{k}(p,q,r) \end{aligned}$$
(19)

where

$$\begin{aligned} \begin{aligned} SCP_{k}^{'}(p,q,r)= \left| \begin{array}{ccc} F^{k}_{R^{'}}(x_{p}^{'},y_{p}^{'})&{}F^{k}_{R^{'}}(x_{q}^{'},y_{q}^{'})&{}F^{k}_{R^{'}}(x_{r}^{'},y_{r}^{'})\\ F^{k}_{G^{'}}(x_{p}^{'},y_{p}^{'})&{}F^{k}_{G^{'}}(x_{q}^{'},y_{q}^{'})&{}F^{k}_{G^{'}}(x_{r}^{'},y_{r}^{'})\\ F^{k}_{B^{'}}(x_{p}^{'},y_{p}^{'})&{}F^{k}_{B^{'}}(x_{q}^{'},y_{q}^{'})&{}F^{k}_{B^{'}}(x_{r}^{'},y_{r}^{'})\\ \end{array} \right| \end{aligned} \end{aligned}$$
(20)

Further, the following relation can be obtained

$$\begin{aligned} scCore_{k}^{'}(N,M;D_{1},D_{2},...,D_{N})=|CA|^{M} \cdot scCore_{k}(N,M;D_{1},D_{2},...,D_{N}) \end{aligned}$$
(21)

where

$$\begin{aligned} scCore_{k}^{'}(N,M;D_{1},D_{2},...,D_{N})=\underbrace{SCP_{k}^{'}(1,2,3)SCP_{k}^{'}(G,K,L)...SCP_{k}^{'}(P,Q,N)}_M\ \end{aligned}$$
(22)

By using Maple2015, the proof of Theorem 1 is obvious. We can find that (10) is a special case of (21), when \(k=0\). So, when we replace \(cCore(N,M;D_{1},D_{2},...,D_{N})\) in (11) with \(scCore_{k}(N,M;D_{1},D_{2},...,D_{N})\), (12) is still tenable. Now, we can define SCDMIs.

Theorem 2

$$\begin{aligned} \begin{aligned}&SCDMIs_{k}(n,m,N,M;d_{1},...,d_{n};D_{1},...,D_{N})\\ {}&= \frac{In(sCore(n,m,d_{1},...,d_{n})\cdot scCore_{k}(N,M,D_{1},...,D_{N}))}{In(sCore(1,0))^{max(n+N)+m-\frac{3M}{2}}\cdot In(scCore_{k}(3,2;2,2,2))^{\frac{M}{2}}}\\ \end{aligned} \end{aligned}$$
(23)

Then there is a relation

$$\begin{aligned} \begin{aligned} SCDMIs_{k}^{'}(n,m,N,M;d_{1},...,d_{n};D_{1},...,D_{N})&\\=SCDMIs_{k}(n,m,N,M;d_{1},...,d_{n};D_{1},...,D_{N})\ \end{aligned} \end{aligned}$$
(24)

where

$$\begin{aligned} \begin{aligned}&SCDMIs_{k}^{'}(n,m,N,M;d_{1},...,d_{n};D_{1},...,D_{N})\\ {}&= \frac{In(sCore^{'}(n,m,d_{1},...,d_{n})\cdot scCore_{k}^{'}(N,M,D_{1},...,D_{N}))}{In(sCore^{'}(1,0))^{max(n+N)+m-\frac{3M}{2}}\cdot In(scCore_{k}^{'}(3,2;2,2,2))^{\frac{M}{2}}}\\ \end{aligned} \end{aligned}$$
(25)

(23) can be expressed as the polynomial of \(SCM^{k}_{pq\alpha \beta \gamma }\). (11) is a special case of (23) when \(k=0\). The proof of (24) is exactly the same as that of (12) proposed in [14].

3.5 The Instances of SCDMIs

We can use (23) to construct instances of SCDMIs by setting different k values. However, color images are discrete functions. The partial derivatives can’t be accurately calculated. As the order of the partial derivative increases, the calculation error is also increasing, which will greatly affect the stability of SCDMIs. So, we only set \(k=0,1\) in this paper.

When \(k=0\), \(SCDMIs_{0}\) are equivalent to SCAMIs. We construct \(SCDMIs_{0}\) of which degrees \(\leqslant 4\), shape orders \(\leqslant 4\) and color orders \(\leqslant 1\). Gong et al. [14] pointed out that in order to obtain \(SCDMIs_{0}\) of which degrees \(\leqslant 4\), shape orders \(\leqslant 4\) and color orders \(\leqslant 1\), \(scCore_{0}(3,1;1,1,1)\) must be C(1, 2, 3). This judgment is wrong. In fact, \(scCore_{0}(3,1;1,1,1)\) can be C(1, 2, 3), C(1, 2, 4), C(1, 3, 4) and C(2, 3, 4). Thus, lots of \(SCDMIs_{0}\) were missed in [14]. By correcting this shortcoming, we get 25 \(SCDMIs_{0}\) that satisfy the independency of functions by using the method proposed by [17].

At the same time, when \(k=1\), \(SCP_{1}(p,q,r)\) is defined by

$$\begin{aligned} SCP_{1}(p,q,r)= \left| \begin{array}{ccc} F^{1}_{R}(x_{p},y_{p})&{}F^{1}_{R}(x_{q},y_{q})&{}F^{1}_{R}(x_{r},y_{r})\\ F^{1}_{G}(x_{p},y_{p})&{}F^{1}_{G}(x_{q},y_{q})&{}F^{1}_{G}(x_{r},y_{r})\\ F^{1}_{B}(x_{p},y_{p})&{}F^{1}_{B}(x_{q},y_{q})&{}F^{1}_{B}(x_{r},y_{r})\\ \end{array} \right| \end{aligned}$$
(26)

where

$$\begin{aligned} F^{1}_{C}(x,y)=(x-\bar{x})\frac{\partial C}{\partial x}+(y-\bar{y})\frac{\partial C}{\partial y} ~~~C \in \{R,G,B\}. \end{aligned}$$
(27)

By replacing C(1, 2, 3), C(1, 2, 4), C(1, 3, 4) and C(2, 3, 4) with \(SCP_{1}(1,2,3)\), \(SCP_{1}(1,2,4)\), \(SCP_{1}(1,3,4)\) and \(SCP_{1}(2,3,4)\), 25 \(SCDMIs_{1}\) can be obtained. Therefore, we can construct the feature vector SCDMI50, which is defined by

$$\begin{aligned} SCDMI50=[SCDMIs_{0}^{1},...,SCDMIs_{0}^{25},SCDMIs_{1}^{1},...,SCDMIs_{1}^{25}] \end{aligned}$$
(28)
Table 1. The construction methods of SCDMI50
Full size table

The construction methods of 50 instances are shown in Table 1. In order to more clearly explain that SCDMIs can be expanded into the polynomial of \(SCM^{k}_{pq\alpha \beta \gamma }\), we give the shape-color moment polynomial of \(SCMIs_{0}^{3}\).

$$\begin{aligned} \begin{aligned} SCDMIs^{3}_{0}=&\{6SCM^{0}_{02001}SCM^{0}_{11010}SCM^{0}_{20100}-6SCM^{0}_{02001}SCM^{0}_{11100}SCM^{0}_{20010}\\ {}&-\,6SCM^{0}_{02010}SCM^{0}_{11001}SCM^{0}_{20100}+6SCM^{0}_{02010}SCM^{0}_{11100}SCM^{0}_{20001}\\ {}&+\,6SCM^{0}_{02100}SCM^{0}_{11001}SCM^{0}_{20010}-6SCM^{0}_{02100}SCM^{0}_{11010}SCM^{0}_{20001}\}\\ {}&/\,\{6SCM^{0}_{00002} SCM^{0}_{00020} SCM^{0}_{00200}-6 SCM^{0}_{00002} (SCM^{0}_{00110})^{2}\\ {}&-\,6 (SCM^{0}_{00011})^{2} SCM^{0}_{00200}+12 SCM^{0}_{00011} SCM^{0}_{00101} SCM^{0}_{00110}\\ {}&-\,6 SCM^{0}_{00020} (SCM^{0}_{00101})^{2}\}\\ \end{aligned} \end{aligned}$$
(29)

4 Experimental Results

In this section, some experiments are designed to evaluate the performance of SCDMI50. Firstly, we verify the stability and discriminability of SCDMI50 by using synthetic images. Then, some retrieval experiments based on real image databases are performed. Also, we chose some commonly used image descriptors for comparison.

It is worth noting that we have to choose a appropriate method to calculate the first order partial derivatives of R(x, y), G(x, y) and B(x, y). In [21], the 5 points difference formulas were used for approximating the first partial derivatives of discrete grayscale images and achieved good results. For color images, they can be defined by

$$\begin{aligned} \begin{aligned}&\frac{\partial C(x,y)}{\partial x}=C(x-2,y)-8C(x-1,y)+8C(x+1,y)-C(x+2,y)\\&\frac{\partial C(x,y)}{\partial y}=C(x,y-2)-8C(x,y-1)+8C(x,y+1)-C(x,y+2)\\ \end{aligned} \end{aligned}$$
(30)

where \(C \in \{R,G,B\}\). We choose this method because it guarantees the computational accuracy of the first partial differential to a certain extent and also maintains a relatively fast calculation speed.

4.1 The Stability and Discriminability of SCDMI50

We select 50 different kinds of butterfly images, which are shown in Fig. 2(a). Then, 5 shape affine transforms and 4 color affine transforms are applied to each image. One image can get 20 transformed versions which are shown in Fig. 2(b). Thus, we obtain the database containing 1000 images. \(10\%\) images selected randomly are used as the training data and the rest make up the testing data.

Fig. 2.
figure 2

Sample images from the butterfly image database (Color figure online)

Full size image

For comparison with SCDMI50, we chose some commonly used global features of gray images or color images.

  • Hu moments, which were composed of 7 invariants under the shape similarity transform, proposed in [1]. Hu moments were designed for gray images.

  • AMIs which were composed of 17 invariants under the shape affine transform proposed in [18]. AMIs were designed for gray images.

  • RGhistogram, which consisted of 60-dimensional features and was scale-invariant in color space, proposed in [19]. The pixel range of each color channel was divided into 20 intervals for statistics.

  • Transformed color distribution, which consisted of 60-dimensional features, was proposed in [19]. Transformed color distribution was scale-invariant and sift-invariant in color space. The pixel range of each color channel was divided into 20 intervals for statistics.

  • Color moments consisted of the first, second and third geometric moments of each channel of the color image, which were proposed in [15]. Color moments are sift-invariant in color space.

  • GPSOs were invariant under the shape affine transforms and the color diagonal-offset transform, which were proposed in [10]. GPSOs consisted of 21 moment invariants.

Fig. 3.
figure 3

The visualization of the distance matrix. As the distance increases, the color changes from white to black.

Full size image

Subsequently, image classification is performed on the butterfly database by using different kinds of features. We use the Nearest Neighbor classifier based on the Chi-Square distance to estimate the species of test images. Finally, we list the classification accuracies obtained by using different features in Table 2.

Table 2. The classification accuracies obtained by using different features
Full size table

On the one hand, we can find that the classification result obtained by using SCDMI50 is better than those obtained by using other features. And, color information is very important for the classification. On the other hand, in order to observe the property of SCDMI50 more clearly, we use Chi-Square distance to calculate the feature distance between any two images. So, we can get a \(1000 \times 1000\) distance matrix which is shown in Fig. 3. Obviously, the color of the area near the diagonal is lighter than those of other regions, indicating that SCDMI50 of similar images are similar in values, and vice versa. Also, these results demonstrate that the construction formula of SCDMIs designed in Sect. 3 is correct.

4.2 Image Retrieval on Real Image Database

In order to further test the performance of SCDMI50, the database COIL-100 proposed in [20] is chosen for our experiment. COIL-100 contains 7202 images of 100 categories, each of which has 72 images taken from different angles. We choose 20 classes, each class contains 8 images \((0^{\circ }, 10^{\circ }, 20^{\circ }, 30^{\circ }, 40^{\circ }, 50^{\circ }, 60^{\circ }, 70^{\circ })\). For each image, 6 color affine transforms are applied. Thus, 960 images of 20 categories are obtained, each of categories contains 48 images which are shown in Fig. 4.

Fig. 4.
figure 4

Sample images from COIL-100 (Color figure online)

Full size image

Then, retrieval experiment is performed on this database. Similar to Sect. 4.1, We choose Chi-Square distance to measure the similarity of two images. The Precision-Recall curves obtained by using different features are shown in Fig. 5(a).

Fig. 5.
figure 5

The Precision-Recall curves on COIL-100 (Color figure online)

Full size image

Obviously, the result obtained by using SCDMI50 is far superior to those obtained by using other features. This is because that traditional image features are constructed by using only color information or shape information. However, we use two kinds of information while constructing SCDMI50. In addition, traditional image descriptors are only stable for simple changes in color space, such as the diagonal-offset transforms. When the color space changes drastically, they are less robust.

Finally, we compare the retrieval results of SCDMI50, \(SCDMI_{0}25\) and \(SCDMI_{1}25\) which are shown in Fig. 5(b). \(SCDMI_{0}25\) and \(SCDMI_{1}25\) are defined by

$$\begin{aligned} SCDMI_{0}25=[SCDMIs_{0}^{1},SCDMIs_{0}^{2},...,SCDMIs_{0}^{25}] \end{aligned}$$
(31)
$$\begin{aligned} SCDMI_{1}25=[SCDMIs_{1}^{1},SCDMIs_{1}^{2},...,SCDMIs_{1}^{25}] \end{aligned}$$
(32)

Because of the calculation error of partial derivatives, the result obtained by using \(SCDMI_{1}25\) slightly worse than that obtained by using \(SCDMI_{0}25\), but far better than those obtained by using traditional features. Meanwhile, \(SCDMI_{1}25\) increase the number of invariants which have simple structures and good properties. We combine \(SCDMI_{1}25\) and \(SCDMI_{0}25\) to get SCDMI50 which achieve the best retrieval result in our experiment.

5 Conclusion

In this paper, we propose a kind of shape-color differential moment invariants (SCDMIs) for color images, which are invariant to the shape affine and color affine transforms, by using partial derivatives of each color channel. It is obvious that all SCAMIs proposed in [14] are the special cases of SCDMIs, when \(k=0\). Then, we correct the mistake in [14] and obtain 50 instances of SCDMIs, which have simple structures and good properties. Finally, several commonly used image descriptors and SCDMIs are used in color image classification and retrieval, respectively. By comparing the experimental results, we find that SCDMIs get better results.