Gradient-Aligned convolution neural network

Highlights

• We propose a general Convolution operation, called GAConv, which can replace conventional operations in CNN to help it achieve rotation invariance.

• With GAConv, Gradient-Aligned CNN (GACNN) can achieve rotation invariance without any data augmentation, feature-map augmentation, and filter enrichment.

• In GACNN, rotation invariance does not learn from the training set, but bases on the network model. Different from the vanilla CNN, GACNN will output invariant results for all rotated versions of an object, no matter whether the network is trained or not.

• We conduct classification experiments on designed dataset and realistic datasets. The results show that with the same computation cost, GACNN achieved better results than conventional CNN and some rotational invariant CNN.

Abstract

Although Convolution Neural Networks (CNN) have achieved great success in many applications of computer vision in recent years, rotation invariance is still a difficult problem for CNN. Especially for some images, the content can appear in the image at any angle of rotation, such as medical images, microscopic images, remote sensing images and astronomical images. In this paper, we propose a novel convolution operation, called Gradient-Aligned Convolution (GAConv), which can help CNN achieve rotation invariance by replacing vanilla convolutions in CNN. GAConv is implemented with a prior pixel-level gradient alignment operation before regular convolution. With GAConv, Gradient-Aligned CNN (GACNN) can achieve rotation invariance without any data augmentation, feature-map augmentation, and filter enrichment. In GACNN, rotation invariance does not learn from the training set, but bases on the network model. Different from the vanilla CNN, GACNN will output invariant results for all rotated versions of an object, no matter whether the network is trained or not. This means that we only need to train the network with one canonical version of the object and all other rotated versions of this object should be recognized with the same accuracy. Classification experiments have been conducted to evaluate GACNN compared with some rotation invariant approaches. GACNN achieved the best results on the ${360}^{\circ }$ rotated test set of MNIST-rotation, Plankton-sub-rotation, and Galaxy Zoo 2.

Introduction

Since the birth of computer vision, rotation invariance has long been an important issue in the field. When we capture images from nature, the object may appear on the image at different rotation angles due to the relative position between the object and the camera. Especially for medical images, microscopic images, remote sensing images, and astronomical images, the angle of rotation can be arbitrary without being affected by gravity constraints in natural images.

When dealing with such images of the same object in different orientations, as shown in Fig. 1, we always want to get the same response so that our descriptors won’t be affected by the orientation of input. We can say that these responses or descriptors are rotation invariants. The formal definition of rotation invariance can be represented by Equation (1),

$$invariance:f\left(I\right)=f\left(R\right(I\left)\right).$$

In the image coordinate system, we can define 2D rotation transformation $R$ on the input image $I$. The function $f(·)$, which has the same output for different rotated versions of image $I$, has rotation invariance. The GACNN proposed in this paper is a special case of $f(·)$.

In recent years, although CNN has achieved great success in many applications of computer vision, achieving rotation invariance through CNN has remained a challenge. CNN encodes local translation invariance somewhat implicitly by weight-shared convolution, pooling operation, and so on. However, there are some difficulties for encoding rotation. Interpolation is needed when rotating an image with an angle that is not multiple of ${90}^{\circ }$, and the results may not be in the pixel grid of the original image domain. These complications make encoding rotation invariance in CNN quite challenging.

For human vision, psychophysical experiments have proven that prior rotation alignment exists to recognize rotated objects. Shepard & Metzler [1] introduced the concept of mental rotation in 1971, which has become one of the best-known experiments in the field. The conclusion is that the time it takes for humans to determine whether two rotated objects are the same depends on the angle of rotation between them. As shown in Fig. 1, the times it takes to recognize ‘7’ in the rotated versions are positively related to the angle between the rotated image and the first canonical version. It means that in order to recognize the image, humans actually rotate the object mentally.

Based on this basic mechanism of human vision, we propose a special convolution, called Gradient-Aligned Convolution (GAConv). Before the regular convolution operation, the gradient at each pixel will be calculated by the proposed Extended Circle Sobel operator and aligned to the canonical direction. It means that all the neighbors will be rotated to make sure the gradient at the central pixel is aligned to a chosen and fixed direction, as illustrated in Fig. 2. The left two figures show the gradients of the first two images in Fig. 1. In the right two figures, the gradients of the first two images in Fig. 1 are aligned to the same fixed canonical direction. From another perspective, the direction information of gradient will be ignored and only the magnitude of gradient participates in the subsequent convolution.

With GAConv, Gradient-Aligned CNN (GACNN) can achieve global rotation invariance without any data augmentation, feature map augmentation, and filter enrichment. Only one canonical direction is encoded into the filters. In GACNN, rotation invariance does not learn from the training set, but bases on the network model. Different from the vanilla CNN, GACNN will output invariant results for all rotated versions of an object, no matter whether the network is trained or not. This means we only need to train the network with one canonical version of the object. All other rotated versions of this object should be recognized with the same accuracy. Further, convolution operations in any CNN architecture can be replaced by GAConv to make the network achieve invariance to rotations. Classification experiments have been conducted to evaluate GACNN compared with regular CNN and some other rotation invariant approaches.