Abstract
The methods of symmetric positive definite (SPD) matrix learning have attracted considerable attention in many pattern recognition tasks, as they are eligible to capture and learn appropriate statistical features while respecting the Riemannian geometry of SPD manifold where the data reside on. Accompanied with the advanced deep learning techniques, several Riemannian networks (RiemNets) for SPD matrix nonlinear processing have recently been studied. However, it is pertinent to ask, whether greater accuracy gains can be realized by simply increasing the depth of RiemNets. The answer appears to be negative, as deeper RiemNets may be difficult to train. To explore a possible solution to this issue, we propose a new architecture for SPD matrix learning. Specifically, to enrich the deep representations, we build a stacked Riemannian autoencoder (SRAE) on the tail of the backbone network, i.e., SPDNet [23]. With this design, the associated reconstruction error term can prompt the embedding functions of both SRAE and of each RAE to approach an identity mapping, which helps to prevent the degradation of statistical information. Then, we implant several residual-like blocks using shortcut connections to augment the representational capacity of SRAE, and to simplify the training of a deeper network. The experimental evidence demonstrates that our DreamNet can achieve improved accuracy with increased depth.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The source code will be released on: https://github.com/GitWR/DreamNet.
References
Absil, P.A., Mahony, R., Sepulchre, R.: Optimization algorithms on matrix manifolds. Princeton University Press (2009)
Arsigny, V., Fillard, P., Pennec, X., Ayache, N.: Geometric means in a novel vector space structure on symmetric positive-definite matrices. SIAM J. Matrix Anal. Appl. 29, 328–347 (2007)
Barachant, A., Bonnet, S., Congedo, M., Jutten, C.: Classification of covariance matrices using a riemannian-based kernel for BCI applications. Neurocomputing 112, 172–178 (2013)
Brooks, D., Schwander, O., Barbaresco, F., Schneider, J.Y., Cord, M.: Riemannian batch normalization for spd neural networks. arXiv preprint arXiv:1909.02414 (2019)
Chakraborty, R., Bouza, J., Manton, J., Vemuri, B.C.: ManifoldNet: a deep neural network for manifold-valued data with applications. IEEE Trans. Pattern Anal. Mach. Intell. 44, 799–810 (2022)
Chen, Z., Xu, T., Wu, X.J., Wang, R., Kittler, J.: Hybrid Riemannian graph-embedding metric learning for image set classification. IEEE Trans. Big Data 9, 75–92 (2021) https://doi.org/10.1109/TBDATA.2021.3113084
Chu, X., Zhou, T., Zhang, B., Li, J.: Fair DARTS: eliminating unfair advantages in differentiable architecture search. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12360, pp. 465–480. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58555-6_28
Dai, M., Zhang, Z., Srivastava, A.: Analyzing dynamical brain functional connectivity as trajectories on space of covariance matrices. IEEE Trans. Med. Imaging 39, 611–620 (2019)
Dhall, A., Goecke, R., Joshi, J., Sikka, K., Gedeon, T.: Emotion recognition in the wild challenge 2014: baseline, data and protocol. In: ICMI, pp. 461–466 (2014)
Du, Y., Wang, W., Wang, L.: Hierarchical recurrent neural network for skeleton based action recognition. In: CVPR, pp. 1110–1118 (2015)
Edelman, A., Arias, T.A., Smith, S.T.: The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. 20, 303–353 (1998)
Feichtenhofer, C., Pinz, A., Zisserman, A.: Convolutional two-stream network fusion for video action recognition. In: CVPR, pp. 1933–1941 (2016)
Gao, Z., Wu, Y., Harandi, M., Jia, Y.: A robust distance measure for similarity-based classification on the SPD manifold. IEEE Trans. Neural Netw. Learn. Syst. 31, 3230–3244 (2020)
Garcia-Hernando, G., Kim, T.K.: Transition forests: learning discriminative temporal transitions for action recognition and detection. In: CVPR, pp. 432–440 (2017)
Garcia-Hernando, G., Yuan, S., Baek, S., Kim, T.K.: First-person hand action benchmark with RGB-D videos and 3D hand pose annotations. In: CVPR, pp. 409–419 (2018)
Hamm, J., Lee, D.D.: Grassmann discriminant analysis: a unifying view on subspace-based learning. In: ICML, pp. 376–383 (2008)
Harandi, M., Salzmann, M.: Riemannian coding and dictionary learning: kernels to the rescue. In: CVPR, pp. 3926–3935 (2015)
Harandi, M., Salzmann, M., Hartley, R.: Joint dimensionality reduction and metric learning: a geometric take. In: ICML, pp. 1404–1413 (2017)
Harandi, M., Salzmann, M., Hartley, R.: Dimensionality reduction on SPD manifolds: the emergence of geometry-aware methods. IEEE Trans. Pattern Anal. Mach. Intell. 40, 48–62 (2018)
Harandi, M.T., Sanderson, C., Hartley, R., Lovell, B.C.: Sparse coding and dictionary learning for symmetric positive definite matrices: a Kernel approach. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) ECCV 2012. LNCS, vol. 7573, pp. 216–229. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33709-3_16
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)
Hu, J.F., Zheng, W.S., Lai, J., Zhang, J.: Jointly learning heterogeneous features for RGB-D activity recognition. In: CVPR, pp. 5344–5352 (2015)
Huang, Z., Van Gool, L.: A Riemannian network for SPD matrix learning. In: AAAI, pp. 2036–2042 (2017)
Huang, Z., Wang, R., Shan, S., Chen, X.: Hybrid Euclidean-and-Riemannian metric learning for image set classification. In: ACCV, pp. 562–577 (2014)
Huang, Z., Wang, R., Shan, S., Chen, X.: Projection metric learning on Grassmann manifold with application to video based face recognition. In: CVPR, pp. 140–149 (2015)
Huang, Z., Wang, R., Shan, S., Li, X., Chen, X.: Log-Euclidean metric learning on symmetric positive definite manifold with application to image set classification. In: ICML, pp. 720–729 (2015)
Huang, Z., Wu, J., Van Gool, L.: Building deep networks on Grassmann manifolds. In: AAAI, pp. 1137–1145 (2018)
Ionescu, C., Vantzos, O., Sminchisescu, C.: Training deep networks with structured layers by matrix backpropagation. arXiv preprint arXiv:1509.07838 (2015)
Kim, T.S., Reiter, A.: Interpretable 3D human action analysis with temporal convolutional networks. In: CVPRW, pp. 1623–1631 (2017)
Li, T., Liu, J., Zhang, W., Ni, Y., Wang, W., Li, Z.: UAV-Human: a large benchmark for human behavior understanding with unmanned aerial vehicles. In: CVPR, pp. 16266–16275 (2021)
Liu, H., Simonyan, K., Yang, Y.: Darts: Differentiable architecture search. In: ICLR (2019)
Lohit, S., Wang, Q., Turaga, P.: Temporal transformer networks: joint learning of invariant and discriminative time warping. In: CVPR, pp. 12426–12435 (2019)
Nguyen, X.S., Brun, L., Lézoray, O., Bougleux, S.: A neural network based on SPD manifold learning for skeleton-based hand gesture recognition. In: CVPR, pp. 12036–12045 (2019)
Pennec, X., Fillard, P., Ayache, N.: A Riemannian framework for tensor computing. Int. J. Comput. Vis. 66, 41–66 (2006). https://doi.org/10.1007/s11263-005-3222-z
Rahmani, H., Mian, A.: 3D action recognition from novel viewpoints. In: CVPR, pp. 1506–1515 (2016)
Sanin, A., Sanderson, C., Harandi, M.T., Lovell, B.C.: Spatio-temporal covariance descriptors for action and gesture recognition. In: WACV Workshop, pp. 103–110 (2013)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Sun, H., Zhen, X., Zheng, Y., Yang, G., Yin, Y., Li, S.: Learning deep match Kernels for image-set classification. In: CVPR, pp. 3307–3316 (2017)
Tekin, B., Bogo, F., Pollefeys, M.: H+O: unified egocentric recognition of 3D hand-object poses and interactions. In: CVPR, pp. 4511–4520 (2019)
Tosato, D., Farenzena, M., Spera, M., Murino, V., Cristani, M.: Multi-class classification on Riemannian manifolds for video surveillance. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010. LNCS, vol. 6312, pp. 378–391. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15552-9_28
Tuzel, O., Porikli, F., Meer, P.: Pedestrian detection via classification on Riemannian manifolds. IEEE Trans. Pattern Anal. Mach. Intell. 30, 1713–1727 (2008)
Vemulapalli, R., Arrate, F., Chellappa, R.: Human action recognition by representing 3D Skeletons as points in a lie group. In: CVPR, pp. 588–595 (2014)
Vemulapalli, R., Pillai, J.K., Chellappa, R.: Kernel learning for extrinsic classification of manifold features. In: CVPR, pp. 1782–1789 (2013)
Wang, R., Wu, X.J., Chen, Z., Xu, T., Kittler, J.: Learning a discriminative SPD manifold neural network for image set classification. Neural Netw. 151, 94–110 (2022)
Wang, R., Wu, X.J., Kittler, J.: Graph embedding multi-kernel metric learning for image set classification with Grassmann manifold-valued features. IEEE Trans. Multimedia 23, 228–242 (2021)
Wang, R., Wu, X.J., Kittler, J.: SymNet: a simple symmetric positive definite manifold deep learning method for image set classification. IEEE Trans. Neural Netw. Learn. Syst. 33, 2208–2222 (2022)
Wang, R., Wu, X.J., Xu, T., Hu, C., Kittler, J.: Deep metric learning on the SPD manifold for image set classification. In: IEEE Transactions on Circuits and Systems for Video Technology (2022)
Wang, R., Guo, H., Davis, L.S., Dai, Q.: Covariance discriminative learning: a natural and efficient approach to image set classification. In: CVPR, pp. 2496–2503 (2012)
Xu, T., Feng, Z.H., Wu, X.J., Kittler, J.: An accelerated correlation filter tracker. Pattern Recognit. 102, 107172 (2020)
Yan, S., Xiong, Y., Lin, D.: Spatial temporal graph convolutional networks for skeleton-based action recognition. In: AAAI (2018)
Zhang, T., et al.: Deep manifold-to-manifold transforming network for skeleton-based action recognition. IEEE Trans. Multimedia 22, 2926–2937 (2020)
Zhang, X., Wang, Y., Gou, M., Sznaier, M., Camps, O.: Efficient temporal sequence comparison and classification using gram matrix embeddings on a Riemannian manifold. In: CVPR, pp. 4498–4507 (2016)
Zhao, S., Xu, T., Wu, X.J., Zhu, X.F.: Adaptive feature fusion for visual object tracking. Pattern Recognit. 111, 107679 (2021)
Zhou, L., Wang, L., Zhang, J., Shi, Y., Gao, Y.: Revisiting metric learning for SPD matrix based visual representation. In: CVPR, pp. 3241–3249 (2017)
Zhu, X.F., Wu, X.J., Xu, T., Feng, Z.H., Kittler, J.: Complementary discriminative correlation filters based on collaborative representation for visual object tracking. IEEE Trans. Circuits Syst. Video Technol. 31, 557–568 (2020)
Acknowledgements
This work was supported by the National Natural Science Foundation of China (62020106012, U1836218, 61672265, 62106089, 62006097), the 111 Project of Ministry of Education of China (B12018), the Postgraduate Research & Practice Innovation Program of Jiangsu Province (KYCX21-2006), and the UK EPSRC EP/N007743/1, MURI/EPSRC/DSTL EP/R018456/1 grants.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, R., Wu, XJ., Chen, Z., Xu, T., Kittler, J. (2023). DreamNet: A Deep Riemannian Manifold Network for SPD Matrix Learning. In: Wang, L., Gall, J., Chin, TJ., Sato, I., Chellappa, R. (eds) Computer Vision – ACCV 2022. ACCV 2022. Lecture Notes in Computer Science, vol 13846. Springer, Cham. https://doi.org/10.1007/978-3-031-26351-4_39
Download citation
DOI: https://doi.org/10.1007/978-3-031-26351-4_39
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-26350-7
Online ISBN: 978-3-031-26351-4
eBook Packages: Computer ScienceComputer Science (R0)