Skip to main content
SpringerLink
Log in

TP: tensor product layer to compress the neural network in deep learning

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Tensor decomposition is widely used to reduce the amount of model parameters and release the pressure on resource-constrained devices. However, exiting studies on tensor decomposition need tensor rank selection, which introduce a large number of additional hyper-parameters and extensive combinatorial searches on tensor ranks. In this paper, we present a new tensor product (TP) based linear layer that can replace the original convolution layer, fully-connected (FC) layer, and vector FC layer in capsule networks without tensor rank selection. Specifically, tensor-matrix product and tensor-vector product are used to the original matrix multiplication. Tensor-matrix product and tensor-outer product are used to replace element product operation. Tensor-outer product is for convolution. These tensor product operations are made up of numerous factors and have fewer parameters. The experimental results show that, when compared to the tensor decomposition algorithm, our algorithm can compress the parameter amount several times while maintaining accuracy without fine-tuning and tensor rank selection.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT press

  2. Zhang X, Zou J, He K, Sun J (2016) Accelerating very deep convolutional networks for classification and detection. IEEE Trans Pattern Anal Mach Intell 38(10):1943–1955

    Article  Google Scholar 

  3. Kim Y D, Park E, Yoo S, Choi T, Yang L, Shin (2015) Compression of deep convolutional neural networks for fast and low power mobile applications. Comput Sci 71(2):576–584

    Google Scholar 

  4. Wang Y, Guo W G, Yue X (2021) Tensor decomposition to compress convolutional layers in deep learning. IISE Trans:1–60

  5. Novikov A, Podoprikhin D, Osokin A, Vetrov D P (2015) Tensorizing neural networks. In: Cortes C, Lawrence N, Lee D, Sugiyama M, Garnett R (eds) advances in neural information processing systems. https://proceedings.neurips.cc/paper/2015/file/6855456e2fe46a9d49d3d3af4f57443d-Paper.pdf, vol 28. Curran Associates, Inc.

  6. Bengua J A, Ho P N, Tuan H D, Do M N (2017) Matrix product state for higher-order tensor compression and classification. IEEE Trans Signal Process 65(15):4019–4030

    Article  MathSciNet  MATH  Google Scholar 

  7. Yu R, Zheng S, Liu Y (2017) Learning chaotic dynamics using tensor recurrent neural networks. In: Proceedings of the ICML, vol 17

  8. Tjandra A, Sakti S, Nakamura S (2017) Compressing recurrent neural network with tensor train. In: 2017 International Joint Conference on Neural Networks (IJCNN). IEEE, pp 4451–4458

  9. Wu B, Wang D, Zhao G, Deng L, Li G (2020) Hybrid tensor decomposition in neural network compression. Neural Netw 132:309–320

    Article  MATH  Google Scholar 

  10. Hillar C J, Lim L-H (2013) Most tensor problems are np-hard. J ACM (JACM) 60(6):1–39

    Article  MathSciNet  MATH  Google Scholar 

  11. Liu J, Musialski P, Wonka P, Ye J (2012) Tensor completion for estimating missing values in visual data. IEEE Trans Pattern Anal Mach Intell 35(1):208–220

    Article  Google Scholar 

  12. Lu C, Feng J, Chen Y, Liu W, Lin Z, Yan S (2019) Tensor robust principal component analysis with a new tensor nuclear norm. IEEE Trans Pattern Anal Mach Intell 42(4):925–938

    Article  Google Scholar 

  13. Zhang Z, Weng T-W, Daniel L (2016) Big-data tensor recovery for high-dimensional uncertainty quantification of process variations. IEEE Trans Compon Packaging Manuf Technol 7(5):687–697

    Article  Google Scholar 

  14. Guhaniyogi R, Qamar S, Dunson D B (2017) Bayesian tensor regression. J Mach Learn Res 18(1):2733–2763

    MathSciNet  MATH  Google Scholar 

  15. Hawkins C, Zhang Z (2021) Bayesian tensorized neural networks with automatic rank selection. Neurocomputing 453:172–180

    Article  Google Scholar 

  16. Lin S, Ji R, Li Y, Deng C, Li X (2019) Toward compact convnets via structure-sparsity regularized filter pruning. IEEE Trans Neural Netw Learn Syst 31(2):574–588

    Article  MathSciNet  Google Scholar 

  17. Luo J-H, Zhang H, Zhou H-Y, Xie C-W, Wu J, Lin W (2018) Thinet: pruning cnn filters for a thinner net. IEEE Trans Pattern Anal Mach Intell 41(10):2525–2538

    Article  Google Scholar 

  18. Lin Y, Tu Y, Dou Z (2020) An improved neural network pruning technology for automatic modulation classification in edge devices. IEEE Trans Veh Technol 69(5):5703–5706

    Article  Google Scholar 

  19. Lin M, Ji R, Wang Y, Zhang Y, Zhang B, Tian Y, Shao L (2020) Hrank: Filter pruning using high-rank feature map. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 1529–1538

  20. Conti F, Schiavone P D, Benini L (2018) Xnor neural engine: A hardware accelerator ip for 21.6-fj/op binary neural network inference. IEEE Trans Comput-Aided Des Integr Circ Syst 37(11):2940–2951

    Article  Google Scholar 

  21. Deng L, Jiao P, Pei J, Wu Z, Li G (2018) Gxnor-net: Training deep neural networks with ternary weights and activations without full-precision memory under a unified discretization framework. Neural Netw 100:49–58

    Article  MATH  Google Scholar 

  22. Liang T, Glossner J, Wang L, Shi S, Zhang X (2021) Pruning and quantization for deep neural network acceleration: a survey. Neurocomputing 461:370–403

    Article  Google Scholar 

  23. Tung F, Mori G (2018) Deep neural network compression by in-parallel pruning-quantization. IEEE Trans Pattern Anal Mach Intell 42(3):568–579

    Article  Google Scholar 

  24. Liu Y, Shu C, Wang J, Shen C (2020) Structured knowledge distillation for dense prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence

  25. Chen Y, Wang N, Zhang Z (2018) Darkrank: Accelerating deep metric learning via cross sample similarities transfer. In: Proceedings of the AAAI conference on artificial intelligence, vol 32

  26. Zhou G, Fan Y, Cui R, Bian W, Zhu X, Gai K (2018) Rocket launching: A universal and efficient framework for training well-performing light net. In: Thirty-second AAAI conference on artificial intelligence

  27. Yang Y, Qiu J, Song M, Tao D, Wang X (2020) Distilling knowledge from graph convolutional networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 7074–7083

  28. Wang L, Yoon K-J (2021) Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks. IEEE Transactions on Pattern Analysis and Machine Intelligence

  29. Wang D, Zhao G, Chen H, Liu Z, Deng L, Li G (2021) Nonlinear tensor train format for deep neural network compression. Neural Netw 144:320–333

    Article  Google Scholar 

  30. Ballester-Ripoll R, Lindstrom P, Pajarola R (2019) Tthresh: Tensor compression for multidimensional visual data. IEEE Trans Vis Comput Graph 26(9):2891–2903

    Article  Google Scholar 

  31. Kasiviswanathan S P, Narodytska N, Jin H (2018) Network approximation using tensor sketching.. In: IJCAI, pp 2319–2325

  32. Iandola F N, Han S, Moskewicz M W, Ashraf K, Dally W J, Keutzer K (2016) Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv:1602.07360

  33. Howard A G, Zhu M, Chen B, Kalenichenko D, Wang W, Weyand T, Andreetto M, Adam H (2017) Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861

  34. Sandler M, Howard A, Zhu M, Zhmoginov A, Chen L-C (2018) Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4510–4520

  35. Zhang X, Zhou X, Lin M, Sun J (2018) Shufflenet: An extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 6848–6856

  36. Ma N, Zhang X, Zheng H-T, Sun J (2018) Shufflenet v2: Practical guidelines for efficient cnn architecture design. In: Proceedings of the European conference on computer vision (ECCV), pp 116–131

  37. Carroll J D, Chang J-J (1970) Analysis of individual differences in multidimensional scaling via an n-way generalization of “eckart-young” decomposition. Psychometrika 35(3):283–319

    Article  MATH  Google Scholar 

  38. Kossaifi J, Lipton Z C, Kolbeinsson A, Khanna A, Furlanello T, Anandkumar A (2020) Tensor regression networks. J Mach Learn Res 21:1–21

    MathSciNet  MATH  Google Scholar 

  39. Imaizumi M, Maehara T, Hayashi K (2017) On tensor train rank minimization: Statistical efficiency and scalable algorithm. In: advances in neural information processing systems, pp 3930–3939

  40. Zhao Q, Zhang L, Cichocki A (2015) Bayesian cp factorization of incomplete tensors with automatic rank determination. IEEE Trans Pattern Anal Mach Intell 37(9):1751–1763

    Article  Google Scholar 

  41. Rai P, Wang Y, Guo S, Chen G, Dunson D, Carin L (2014) Scalable bayesian low-rank decomposition of incomplete multiway tensors. In: International conference on machine learning, pp 1800–1808

  42. Holtz S, Rohwedder T, Schneider R (2012) The alternating linear scheme for tensor optimization in the tensor train format. SIAM J Sci Comput 34(2):A683–A713

    Article  MathSciNet  MATH  Google Scholar 

  43. Ji Y, Wang Q, Li X, Liu J (2019) A survey on tensor techniques and applications in machine learning. IEEE Access 7:162950–162990

    Article  Google Scholar 

  44. Kolda T G, Bader B W (2009) Tensor decompositions and applications. SIAM Rev 51 (3):455–500

    Article  MathSciNet  MATH  Google Scholar 

  45. Kossaifi J, Panagakis Y, Anandkumar A, Pantic M (2019) Tensorly: tensor learning in python. J Mach Learn Res 20(1):925–930

    Google Scholar 

  46. Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado G S, Davis A, Dean J, Devin M et al (2016) Tensorflow: Large-scale machine learning on heterogeneous distributed systems. arXiv:1603.04467

  47. Duan H, Xiao X, Long J, Liu Y (2020) Tensor alternating least squares grey model and its application to short-term traffic flows. Appl Soft Comput 89:106145

    Article  Google Scholar 

  48. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  49. Sabour S, Frosst N, Hinton G E (2017) Dynamic routing between capsules. In: Advances in neural information processing systems, pp 3856–3866

  50. Cheng Z, Sun H, Takeuchi M, Katto J (2019) Deep residual learning for image compression.. In: CVPR Workshops, p 0

  51. Tan M, Le Q (2019) Efficientnet: Rethinking model scaling for convolutional neural networks. In: International Conference on Machine Learning. PMLR, pp 6105–6114

Download references

Acknowledgements

This paper is supported by Beijing Natural Science Foundation No.L182037, National Natural Science Foundation of China No.61871045 and DiDi Cooperation.

Author information

Authors and Affiliations

  1. Beijing University of Posts and Telecommunications, 10 Xitucheng Road, Beijing, China

    Wang Qiang & Yuwang Ji

Authors
  1. Wang Qiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuwang Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yuwang Ji.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qiang, W., Ji, Y. TP: tensor product layer to compress the neural network in deep learning. Appl Intell 52, 17133–17144 (2022). https://doi.org/10.1007/s10489-022-03260-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-03260-6

Keywords

Search

Navigation