Abstract
Symmetric positive definite (SPD) matrix has recently been used as an effective visual representation. When learning this representation in deep networks, eigen-decomposition of covariance matrix is usually needed for a key step called matrix normalisation. This could result in significant computational cost, especially when facing the increasing number of channels in recent advanced deep networks.
This work proposes a novel scheme called Relation Dropout (ReDro). It is inspired by the fact that eigen-decomposition of a block diagonal matrix can be efficiently obtained by decomposing each of its diagonal square matrices, which are of smaller sizes. Instead of using a full covariance matrix as in the literature, we generate a block diagonal one by randomly grouping the channels and only considering the covariance within the same group. We insert ReDro as an additional layer before the step of matrix normalisation and make its random grouping transparent to all subsequent layers. Additionally, we can view the ReDro scheme as a dropout-like regularisation, which drops the channel relationship across groups. As experimentally demonstrated, for the SPD methods typically involving the matrix normalisation step, ReDro can effectively help them reduce computational cost in learning large-sized SPD visual representation and also help to improve image recognition performance.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Let \(\mathbf {C}\) be a SPD matrix and its eigen-decomposition is \(\mathbf {C}=\mathbf {U}\mathbf {D}\mathbf {U}^{\top }\). The columns of \(\mathbf {U}\) are eigenvectors while the diagonal of the diagonal matrix \({\mathbf{D}}\) consists of eigenvalues. Matrix normalisation with a function f is defined as \(f(\mathbf {C})=\mathbf {U}f(\mathbf {D})\mathbf {U}^{\top }\), where \(f(\mathbf {D})\) means f is applied to each diagonal entry of \({\mathbf{D}}\). Matrix-logarithm and matrix-power based normalisations correspond to \(f(x)=\log (x)\) and \(f(x)=x^p\).
- 2.
In matrix analysis, a permutation matrix \({\mathbf{P}}\) is a square binary matrix. It has one and only one “1” entry in each row and each column, with all the remainder being “0”. It is easy to verify that \({\mathbf{P}}{\mathbf{P}}^{\top }={\mathbf{P}}^{\top }{\mathbf{P}}={\mathbf{I}}\), where \({\mathbf{I}}\) is an identity matrix.
- 3.
For the matrix \({\mathbf{G}}\), its (i, j)th entry \(g_{ij}\) is defined as \( \frac{f(\lambda _i)-f(\lambda _j)}{\lambda _i - \lambda _j}\) if \(\lambda _i\ne {\lambda _j}\) and \(f'(\lambda _i)\) otherwise, where \(\lambda _i\) is the ith diagonal element of \(\hat{\mathbf{D}}_b\).
- 4.
For a symmetric matrix, the complexity of eigen-decomposition could be improved up to the order of \({\mathcal {O}}(d^{2.38})\) by more sophisticated algorithms though [5].
References
Brooks, D., Schwander, O., Barbaresco, F., Schneider, J.Y., Cord, M.: Riemannian batch normalization for SPD neural networks. In: Advances in Neural Information Processing Systems, pp. 15489–15500 (2019)
Chakraborty, R., Bouza, J., Manton, J., Vemuri, B.C.: A deep neural network for manifold-valued data with applications to neuroimaging. In: Chung, A.C.S., Gee, J.C., Yushkevich, P.A., Bao, S. (eds.) IPMI 2019. LNCS, vol. 11492, pp. 112–124. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-20351-1_9
Cimpoi, M., Maji, S., Vedaldi, A.: Deep filter banks for texture recognition and segmentation. In: Proceedings of the International Conference on Computer Vision and Pattern Recognition, pp. 3828–3836. IEEE (2015)
Cui, Y., Zhou, F., Wang, J., Liu, X., Lin, Y., Belongie, S.: Kernel pooling for convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921–2930 (2017)
Demmel, J., Dumitriu, I., Holtz, O.: Fast linear algebra is stable. Numer. Math. 108(1), 59–91 (2007)
Engin, M., Wang, L., Zhou, L., Liu, X.: DeepKSPD: learning kernel-matrix-based SPD representation for fine-grained image recognition. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11206, pp. 629–645. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01216-8_38
Gao, Y., Beijbom, O., Zhang, N., Darrell, T.: Compact bilinear pooling. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 317–326. IEEE (2016)
Ghiasi, G., Lin, T.Y., Le, Q.V.: Dropblock: a regularization method for convolutional networks. In: Proceedings of the Advances in Neural Information Processing Systems, pp. 10727–10737 (2018)
Hou, S., Wang, Z.: Weighted channel dropout for regularization of deep convolutional neural network. Proc. AAAI Conf. Artif. Intell. 33, 8425–8432 (2019)
Huang, Z., Van Gool, L.: A Riemannian network for SPD matrix learning. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)
Ionescu, C., Vantzos, O., Sminchisescu, C.: Matrix backpropagation for deep networks with structured layers. In: Proceedings of the International Conference on Computer Vision, pp. 2965–2973. IEEE (2015)
Jayasumana, S., Hartley, R., Salzmann, M., Li, H., Harandi, M.: Kernel methods on Riemannian manifolds with Gaussian RBF kernels. IEEE Trans. Pattern Anal. Mach. Intell. 37(12), 2464–2477 (2015)
Kong, S., Fowlkes, C.: Low-rank bilinear pooling for fine-grained classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 365–374 (2017)
Koniusz, P., Wang, L., Cherian, A.: Tensor representations for action recognition. IEEE Trans. Pattern Anal. Mach. Intell. (2020)
Koniusz, P., Zhang, H.: Power normalizations in fine-grained image, few-shot image and graph classification. IEEE Trans. Pattern Anal. Mach. Intell. (2020)
Koniusz, P., Zhang, H., Porikli, F.: A deeper look at power normalizations. In: 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, 18–22 June 2018, pp. 5774–5783. IEEE Computer Society (2018). https://doi.org/10.1109/CVPR.2018.00605
Krause, J., Stark, M., Deng, J., Fei-Fei, L.: 3D object representations for fine-grained categorization. In: Proceedings of the International Conference on Computer Vision Workshops, pp. 554–561. IEEE (2013)
Li, P., Xie, J., Wang, Q., Gao, Z.: Towards faster training of global covariance pooling networks by iterative matrix square root normalization. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 947–955. IEEE (2018)
Li, P., Xie, J., Wang, Q., Zuo, W.: Is second-order information helpful for large-scale visual recognition? In: Proceedings of the International Conference on Computer Vision, pp. 2070–2078. IEEE (2017)
Lin, T.Y., Maji, S.: Improved Bilinear Pooling with CNNs. arXiv preprint arXiv:1707.06772 (2017)
Lin, T.Y., RoyChowdhury, A., Maji, S.: Bilinear CNN models for fine-grained visual recognition. In: Proceedings of the International Conference on Computer Vision, pp. 1449–1457. IEEE (2015)
Maji, S., Rahtu, E., Kannala, J., Blaschko, M., Vedaldi, A.: Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151 (2013)
Ng, A.Y., Jordan, M.I., Weiss, Y.: On spectral clustering: analysis and an algorithm. In: Proceedings of the Advances in Neural Information Processing Systems, pp. 849–856 (2002)
Nguyen, X.S., Brun, L., Lézoray, O., Bougleux, S.: A neural network based on SPD manifold learning for skeleton-based hand gesture recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12036–12045 (2019)
Porikli, F., Tuzel, O., Meer, P.: Covariance tracking using model update based on lie algebra. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 728–735. IEEE (2006)
Quattoni, A., Torralba, A.: Recognizing indoor scenes. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 413–420. IEEE (2009)
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)
Tompson, J., Goroshin, R., Jain, A., LeCun, Y., Bregler, C.: Efficient object localization using convolutional networks. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 648–656. IEEE (2015)
Tuzel, O., Porikli, F., Meer, P.: Region covariance: a fast descriptor for detection and classification. In: Leonardis, A., Bischof, H., Pinz, A. (eds.) ECCV 2006. LNCS, vol. 3952, pp. 589–600. Springer, Heidelberg (2006). https://doi.org/10.1007/11744047_45
Tuzel, O., Porikli, F., Meer, P.: Human detection via classification on riemannian manifolds. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 1–8. IEEE (2007)
Vedaldi, A., Lenc, K.: MatConvNet: convolutional neural networks for Matlab. In: Proceedings of the 23rd ACM International Conference on Multimedia, pp. 689–692 (2015)
Wang, Q., Xie, J., Zuo, W., Zhang, L., Li, P.: Deep CNNs meet global covariance pooling: better representation and generalization. IEEE Trans. Pattern Anal. Mach. Intell. (2020)
Wang, R., Guo, H., Davis, L.S., Dai, Q.: Covariance discriminative learning: A natural and efficient approach to image set classification. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 2496–2503. IEEE (2012)
Welinder, P., et al.: Caltech-UCSD birds 200. Technical report CNS-TR-2010-001, California Institute of Technology (2010)
Yu, K., Salzmann, M.: Second-order convolutional neural networks. arXiv preprint arXiv:1703.06817 (2017)
Zhang, H., Zhang, J., Koniusz, P.: Few-shot learning via saliency-guided hallucination of samples. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp. 2770–2779. IEEE (2019)
Zhang, H., Zhang, L., Qui, X., Li, H., Torr, P.H.S., Koniusz, P.: Few-shot action recognition with permutation-invariant attention. In: Proceedings of the European Conference on Computer Vision (ECCV) (2020)
Zheng, H., Fu, J., Zha, Z.J., Luo, J.: Learning deep bilinear transformation for fine-grained image representation. In: Proceedings of the Advances in Neural Information Processing Systems, pp. 4279–4288 (2019)
Zhu, X., Xu, C., Hui, L., Lu, C., Tao, D.: Approximated bilinear modules for temporal modeling. In: Proceedings of the International Conference on Computer Vision, pp. 3494–3503. IEEE (2019)
Acknowledgement
This work was supported by the CSIRO Data61 Scholarship; the University of Wollongong Australia IPTA scholarship; the Australian Research Council (grant number DP200101289); and the Multi-modal Australian ScienceS Imaging and Visualisation Environment (MASSIVE).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Rahman, S., Wang, L., Sun, C., Zhou, L. (2020). ReDro: Efficiently Learning Large-Sized SPD Visual Representation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, JM. (eds) Computer Vision – ECCV 2020. ECCV 2020. Lecture Notes in Computer Science(), vol 12360. Springer, Cham. https://doi.org/10.1007/978-3-030-58555-6_1
Download citation
DOI: https://doi.org/10.1007/978-3-030-58555-6_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58554-9
Online ISBN: 978-3-030-58555-6
eBook Packages: Computer ScienceComputer Science (R0)