Skip to main content
SpringerLink
Log in

Matrix-product neural network based on sequence block matrix product

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Convolution neural networks (CNNs) based on the discrete convolutional operation have achieved great success in image processing, voice and audio processing, natural language processing and other fields. However, it is still an open problem how to develop new models instead of CNNs. Using the idea of the sequence block matrix product, we propose a novel operation and its corresponding neural network, namely two-dimensional discrete matrix-product operation (TDDMPO) and matrix-product neural network (MPNN). We present the definition of the TDDMPO, a series of its properties and matrix-product theorem in detail, and then construct its corresponding MPNN. Experimental results on Fashion-MNIST, SVHN, FLOWER17 and FLOWER102 datasets show that MPNNs obtain 1.65–13.04% relative performance improvement in comparison with the corresponding CNNs, and the amount of calculation of matrix-product layers of MPNNs obtains 41× to 57× reduction in comparison with the corresponding convolutional layers of CNNs. Hence, it is a potential model that may open some new directions for deep neural networks, particularly alternatives to CNNs.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data Availability Statement

Fashion-MNIST dataset that supports the findings of this study is openly available in [GRAVITI] at [https://www.graviti.cn/open-datasets/FashionM-NIST], reference number [31]. SVHN dataset that supports the findings of this study is openly available in [GRAVITI] at [https://www.graviti.cn/open-datasets/SVHN], reference number [32]. 17 Category Flower (FLOWER17) dataset that supports the findings of this study is openly available in [GRAVITI] at [https://www.graviti.cn/open-datasets/Flower17], reference number [33]. 102 Category Flower (FLOWER102) dataset that supports the findings of this study is openly available in [GRAVITI] at [https://www.graviti.cn/open-datasets/Flower102], reference number [33].

References

  1. Hubel DH, Wiesel T (1962) Receptive fields, binocular interaction, and functional architecture in the cats visual cortex. J Physiol 160(1):106–154

    Article  Google Scholar 

  2. Wiesel T, Hubel DH (1959) Receptive fields of single neurons in the cats striate cortex. J Physiol 148(3):574–591

    Article  Google Scholar 

  3. Fukushima K (1979) Neural network model for a mechanism of pattern recognition unaffected by shift in position-Neocognitron. IEICE Techn Rep 62(10):658–665

    Google Scholar 

  4. Fukushima K (1980) Neocognitron: a self-organizing neural network for a mechanism of pattern recognition unaffected by shift in position. Biol Cybern 36(4):193–202

    Article  Google Scholar 

  5. Fukushima K (2013) Artificial vision by multi-layered neural networks: neocognitron and its advances. Neural Netw 37:103–119

    Article  Google Scholar 

  6. LeCun Y, Boser B, Denker JS et al (1989) Backpropagation applied to handwritten zip code recognition. Neural Comput 1(4):541–551

    Article  Google Scholar 

  7. LeCun Y, Bottou L, Bengio Y et al (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  8. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst, pp 1097–1105

  9. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. Comput Sci

  10. Russakovsky O, Deng J, Su H et al (2015) Imagenet large scale visual recognition challenge. Int J Comput Vis 115(3):211–252

    Article  MathSciNet  Google Scholar 

  11. Wengrowski E, Purri M, Dana K et al (2019) Deep CNNs as a method to classify rotating objects based on monostatic RCS. IET Radar Sonar Navig 13(7):1092–1100

    Article  Google Scholar 

  12. Wu X, Zhang Z, Zhang W et al (2021) A convolutional neural network based on grouping structure for scene classification. Remote Sens 13(13):2457–2477

    Article  Google Scholar 

  13. Hagag A, Omara I, Alfarra ANK, Mekawy F (2021) Handwritten chemical formulas classification model using deep transfer convolutional neural networks. In: International Conference on Electronic Engineering (ICEEM), pp 1–6

  14. Teli MN (2021) TeliNet, a simple and shallow convolution neural network (CNN) to classify CT scans of COVID-19 patients. arXiv:2107.04930

  15. Shawky OA, Hagag A, El-Dahshan E et al (2020) Remote sensing image scene classification using CNN-MLP with data augmentation. Optik Int J Light Electron Opt 165356

  16. He K, Gkioxari G, Dollr P et al (2017) Mask r-cnn. In: 2017 IEEE International Conference on Computer Vision (ICCV). IEEE, pp 2980–2988

  17. Liu B, Liu Q, Zhang T et al (2019) MSSTResNet-TLD: a robust tracking method based on tracking-learning-detection framework by using multi-scale spatio-temporal residual network feature model. Neurocomputing 175–194

  18. Liu Z, Waqas M, Yang J et al (2021) A multi-task CNN for maritime target detection. IEEE Signal Process Lett 28:434–438

    Article  Google Scholar 

  19. Fan M, Tian S, Liu K et al (2021) Infrared small target detection based on region proposal and CNN classifier. SIViP 1–10

  20. Hou F, Lei W, Li S et al (2021) Deep learning-based subsurface target detection from GPR scans. IEEE Sens J 21(6):8161–8171

    Article  Google Scholar 

  21. Mnih V, Kavukcuoglu K, Silver D et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533

    Article  Google Scholar 

  22. Silver D, Schrittwieser J, Simonyan K et al (2017) Mastering the game of Go without human knowledge. Nature 550(7676):354–359

    Article  Google Scholar 

  23. Zoughi T, Homayounpour MM (2019) A gender-aware deep neural network structure for speech recognition, Iranian Journal of Science and Technology-Transactions of. Electr Eng 43(3):635–644

    Google Scholar 

  24. Perdana BBSP, Irawan B, Setianingsih C (2019) Hate speech detection in indonesian language on instagram comment section using deep neural network classification method. In: 2019 IEEE Asia Pacific Conference on Wireless and Mobile (APWiMob). IEEE

  25. Krishnan PT, Balasubramanian P (2019) Detection of alphabets for machine translation of sign language using deep neural net. In: 2019 International Conference on Data Science and Communication (IconDSC)

  26. Hinton GE, Sabour S, Frosst N (2018) Matrix capsules with EM routing. In: International Conference on Representation Learning

  27. Gonzalez RC, Wintz P (1997) Digital image processing. Addison-Wesley, New York

    MATH  Google Scholar 

  28. Bhabatosh C (1977) Digital image processing and analysis. PHI Learning Pvt Ltd, New Delhi

    Google Scholar 

  29. Zhang XD (2017) Matrix analysis and applications. Cambridge University Press, Cambridge

    Book  Google Scholar 

  30. Bouvrie J (2006) Notes on convolutional neural networks. Center for Biological and Computational Learning, Massachusetts, pp 38–44

    Google Scholar 

  31. Xiao H, Rasul K, Vollgraf R (2017) Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms. arXiv:1708.07747

  32. Netzer Y, Wang T, Coates A et al (2011) Reading digits in natural images with unsupervised feature learning. Adv Neural Inf Process Syst 4–12

  33. Nilsback ME, Zisserman A (2008) Automated flower classification over a large number of classes. In: Sixth Indian Conference on Computer Vision, Graphics and Image Processing, ICVGIP 2008, Bhubaneswar, India, 16–19 December 2008. IEEE

Download references

Author information

Authors and Affiliations

  1. College of Electrical Engineering, Anhui Polytechnic University, Wuhu, 241000, China

    Chuanhui Shan

  2. TianFu Software Park, Chengdu, 610041, China

    Jun Ou

  3. College of Biological and Food Engineering, Anhui Polytechnic University, Wuhu, 241000, China

    Xiumei Chen

Authors
  1. Chuanhui Shan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jun Ou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiumei Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chuanhui Shan.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work is supported by Anhui Polytechnic University Introduced Talent Research Startup Fund (No. 2020YQQ039).

Appendix 1

Appendix 1

In “Appendix 1” section, we offer the detailed proofs of related conclusions in Sect. 2.

1. Proof of Formula (6)

Proof

According to Formula (2),

$$\begin{aligned}&(f_e(m,n)\otimes g_e(m,n))\otimes h_e(m,n)\\&= \left( \sum _{i=0}^{M-1}{\sum _{j=0}^{N-1}{f_e(i,j)\times g_e(m-i,n-j)}}\right) \otimes h_e(m,n)\\&= \sum _{{\bar{m}}=0}^{M-1}{\sum _{{\bar{n}}=0}^{N-1}{\left( \sum _{i=0}^{M-1}{\sum _{j=0}^{N-1}{f_e(i,j)\times g_e({\bar{m}}-i,{\bar{n}}-j)}}\right) }}\times h_e(m-{\bar{m}},n-{\bar{n}}) \\&= \sum _{i=0}^{M-1}{\sum _{j=0}^{N-1}{f_e(i,j)\times }}\left( \sum _{{\bar{m}}=0}^{M-1}{\sum _{{\bar{n}}=0}^{N-1}{g_e({\bar{m}}-i,{\bar{n}}-j)}}\times h_e(m-{\bar{m}},n-{\bar{n}})\right) \\&\overset{Let \ p={\bar{m}}-i,q={\bar{n}}-j}{=}\sum _{i=0}^{M-1}{\sum _{j=0}^{N-1}{f_e(i,j)}} \\&\times \left( \sum _{p=-i}^{M-1-i}{\sum _{q=-j}^{N-1-j}{g_e(p,q)\times h_e(m-p-i,n-q-j)}}\right) \\&= \sum _{i=0}^{M-1}{\sum _{j=0}^{N-1}{f_e(i,j)\times }}\left( \sum _{p=0}^{M-1}{\sum _{q=0}^{N-1}{g_e(p,q)\times h_e(m-i-p,n-j-q)}}\right) \\&= f_e(m,n)\otimes (g_e(m,n)\otimes h_e(m,n)) \end{aligned}$$

So Formula (6) holds. \(\square \)

2. Proof of Formula (7)

Proof

According to Formula (2),

$$\begin{aligned}&f_e(m,n)\otimes (g_e(m,n)+ h_e(m,n))\\&= \sum _{i=0}^{M-1}{\sum _{j=0}^{N-1}{f_e(i,j)\times }}(g_e(m-i,n-j)+ h_e(m-i,n-j))\\&= \sum _{i=0}^{M-1}{\sum _{j=0}^{N-1}{f_e(i,j)\times g_e(m-i,n-j)}} \\&+ \sum _{i=0}^{M-1}{\sum _{j=0}^{N-1}{f_e(i,j)\times h_e(m-i,n-j)}}\\&= f_e(m,n)\otimes g_e(m,n)+f_e(m,n)\otimes h_e(m,n) \end{aligned}$$

So Formula (7) holds. \(\square \)

3. Proof of Formula (8)

Proof

According to Formula (2),

$$\begin{aligned}&\nabla f_e(m,n) \otimes g_e(m,n) \\&\quad = (f_e(m,n)-f_e(m-1,n-1))\otimes g_e(m,n)\\&\quad = f_e(m,n)\otimes g_e(m,n)-f_e(m-1,n-1)\otimes g_e(m,n)\\&\quad = \sum _{i=0}^{M-1}{\sum _{j=0}^{N-1}{f_e(i,j)\times g_e(m-i,n-j)}}\\&\qquad - \sum _{i=0}^{M-1}{\sum _{j=0}^{N-1}{f_e(i-1,j-1)\times g_e(m-i,n-j)}}\\&\quad = \sum _{i=0}^{M-1}{\sum _{j=0}^{N-1}{f_e(i,j)\times g_e(m-i,n-j)}} \\&\qquad - \sum _{p=-1}^{M-2}{\sum _{q=-1}^{N-2}{f_e(p,q)\times g_e(m-p-1,n-q-1)}} \\&\quad = \sum _{i=0}^{M-1}{\sum _{j=0}^{N-1}{f_e(i,j)\times g_e(m-i,n-j)}} \\&\qquad - \sum _{p=0}^{M-1}{\sum _{q=0}^{N-1}{f_e(p,q)\times g_e(m-p-1,n-q-1)}} \\&\quad = \sum _{i=0}^{M-1}{\sum _{j=0}^{N-1}{f_e(i,j)\times g_e(m-i,n-j)}} \\&\qquad - \sum _{i=0}^{M-1}{\sum _{j=0}^{N-1}{f_e(i,j)\times g_e(m-i-1,n-j-1)}} \\ \end{aligned}$$
$$\begin{aligned}&= \sum _{i=0}^{M-1}{\sum _{j=0}^{N-1}{f_e(i,j) }} \\&\times (g_e(m-i,n-j)-g_e(m-1-i,n-l-j))\\&= f_e(m,n) \otimes \nabla g_e(m,n) \end{aligned}$$

So Formula (8) holds. \(\square \)

4. Proof of Formulas (9)–(11)

Proof

(1) According to Formula (2),

$$\begin{aligned}&f_e(m,n) \otimes \varDelta (m,n) \\&= \sum _{i=0}^{M-1}{\sum _{j=0}^{N-1}{f_e(i,j)\times \varDelta (m-i,n-j)}} \\&= f_e(m,n). \end{aligned}$$

(2) According to Formula (2),

$$\begin{aligned}&f_e(m+k,n+k) \otimes \varDelta (m,n) \\&= \sum _{i=0}^{M-1}{\sum _{j=0}^{N-1}{f_e(i+k,j+k)\times \varDelta (m-i,n-j)}} \\&= f_e(m+k,n+k). \end{aligned}$$

3) According to Formula (2),

$$\begin{aligned}&f_e(m,n) \otimes \varDelta (m+k,n+k) \\&= \sum _{i=0}^{M-1}{\sum _{j=0}^{N-1}{f_e(i,j)\times \varDelta (m+k-i,n+k-j)}} \\&= f_e(m+k,n+k). \end{aligned}$$

In summary, Formulas (9)-(11) hold. \(\square \)

5. Proof of Formula (13)

Proof

Substitute Formula (13) into Formula (12). We have

$$\begin{aligned}&IDFT(F(u,v)) = \sum _{u=0}^{M-1}{\sum _{v=0}^{N-1}{F(u,v)e^{j2\pi (um/M+vn/N)}}} \\ =&\sum _{u=0}^{M-1}{\sum _{v=0}^{N-1}{\frac{1}{MN}}} \sum _{m'=0}^{M-1}{\sum _{n'=0}^{N-1}{f(m',n')}} e^{-j2\pi (um'/M+vn'/N)}\\&e^{j2\pi (um/M+vn/N)} \\ =&\sum _{m'=0}^{M-1}{\sum _{n'=0}^{N-1}{f(m',n') }}\frac{1}{MN} \sum _{u=0}^{M-1}{\sum _{v=0}^{N-1}{e^{-j2\pi (um'/M+vn'/N)}}}\\&e^{j2\pi (um/M+vn/N)} \\ \end{aligned}$$
$$\begin{aligned} =&\sum _{m'=0}^{M-1}{\sum _{n'=0}^{N-1}{f(m',n')}}\frac{1}{MN}\\&\sum _{u=0}^{M-1}{\sum _{v=0}^{N-1}{e^{j2\pi [u(m-m')/M+v(n-n')/N]}}} \end{aligned}$$

Because

$$\begin{aligned} \begin{aligned}&\frac{1}{MN}\sum _{u=0}^{M-1}{\sum _{v=0}^{N-1}{e^{j2\pi [u(m-m')/M+v(n-n')/N]}}} \\&= {\left\{ \begin{array}{ll} 1,m = m'+z_1M, n = n'+z_2N \\ 0,m \ne m'+z_1M, n \ne n'+z_2N \end{array}\right. } , \end{aligned} \end{aligned}$$

where \(z_1\) and \(z_2\) are integer, the following formula is satisfied in the transformation interval, namely

$$\begin{aligned} \begin{aligned}&IDFT(F(u,v)) = f(m,n), \\&m =0,1,\ldots ,M-1,n =0,1,\ldots ,N-1. \end{aligned} \end{aligned}$$

It can be seen that the two-dimensional discrete Fourier transform defined by Formula (13) is unique. \(\square \)

6. Proof of Formula (14)

Proof

According to Formula (13),

$$\begin{aligned}&f(m+k_1 M, n+k_2 N) \\&= \sum _{u=0}^{M-1}{\sum _{v=0}^{N-1}{F(u,v)e^{j2\pi (u(m+k_1 M)/M+v(n+k_2 N)/N)}}} \\&= \sum _{u=0}^{M-1}{\sum _{v=0}^{N-1}{F(u,v)e^{j2\pi (um/M+vn/N)}e^{j2\pi (k_1u+k_2v)}}} \\&= \sum _{u=0}^{M-1}{\sum _{v=0}^{N-1}{F(u,v)e^{j2\pi (um/M+vn/N)}}}=f(m,n) \end{aligned}$$

According to Formula (12),

$$\begin{aligned}&F(u+k_1M, v+k_2N) \\&= \frac{1}{MN} \sum _{m=0}^{M-1}{\sum _{n=0}^{N-1}{f(m,n)}} e^{-j2\pi ((u+k_1M)m/M+(v+k_2N)n/N)} \\&= \frac{1}{MN} \sum _{m=0}^{M-1}{\sum _{n=0}^{N-1}{f(m,n)}} e^{-j2\pi (um/M+vn/N)}e^{-j2\pi (k_1m+k_2n/N)} \\&= \frac{1}{MN}\sum _{m=0}^{M-1}{\sum _{n=0}^{N-1}{f(m,n)e^{-j2\pi (um/M+vn/N)}}} = F(u,v) \end{aligned}$$

In summary, Formula (14) holds. \(\square \)

7. Proof of Formula (15)

Proof

According to Formula (12),

$$\begin{aligned}&DFT(af(m,n)+bg(m,n)) \\&= \frac{1}{MN}\sum _{m=0}^{M-1}{\sum _{n=0}^{N-1}{(af(m,n)+bg(m,n))}} e^{-j2\pi (um/M+vn/N)} \\&= \frac{1}{MN}\sum _{m=0}^{M-1}{\sum _{n=0}^{N-1}{af(m,n)e^{-j2\pi (um/M+vn/N)}}} \\&\quad + \frac{1}{MN}\sum _{m=0}^{M-1}{\sum _{n=0}^{N-1}{bg(m,n)e^{-j2\pi (um/M+vn/N)}}} \\&= a\frac{1}{MN}\sum _{m=0}^{M-1}{\sum _{n=0}^{N-1}{f(m,n)e^{-j2\pi (um/M+vn/N)}}} \\&\quad + b\frac{1}{MN}\sum _{m=0}^{M-1}{\sum _{n=0}^{N-1}{g(m,n)e^{-j2\pi (um/M+vn/N)}}} \\&= aF(u,v)+bG(u,v) \end{aligned}$$

So Formula (15) holds. \(\square \)

8. Proof of Formula (16)

Proof

According to Formula (13),

$$\begin{aligned} f(-m,-n)&= \sum _{u=0}^{M-1}{\sum _{v=0}^{N-1}{F(u,v)e^{-j2\pi (um/M+vn/N)}}}\\&= MN \cdot \frac{1}{MN} \sum _{u=0}^{M-1}{\sum _{v=0}^{N-1}{F(u,v)e^{-j2\pi (um/M+vn/N)}}} \\&= MN \cdot DFT(F(u,v)) \end{aligned}$$

So \(F(u,v) \Leftrightarrow \frac{1}{MN}f(-m,-n)\). According to Formula (30),

$$\begin{aligned} F(-u,-v)&= \frac{1}{MN}\sum _{m=0}^{M-1}{\sum _{n=0}^{N-1}{f(m,n)e^{j2\pi (um/M+vn/N)}}} \\&= \frac{1}{MN} IDFT(f(m,n)) \end{aligned}$$

So \(MN\cdot F(-u,-v)\Leftrightarrow f(m,n)\). In summary, Formula (16) holds. \(\square \)

9. Proof of Formula (17)

Proof

According to Formula (12),

$$\begin{aligned}&DFT(f(m\pm m_0,n\pm n_0))\\&= \frac{1}{MN}\sum _{m=0}^{M-1}{\sum _{n=0}^{N-1}{f(m\pm m_0,n\pm n_0)e^{-j2\pi (um/M+vn/N)}}}\\&\overset{Let \ m\pm m_0=p,m=p\mp m_0;n\pm n_0=q,n=q\mp n_0}{=} \\&\frac{1}{MN}\sum _{p=\pm m_0}^{M-1\pm m_0}{\sum _{q=\pm n_0}^{N-1\pm n_0}{f(p,q)e^{-j2\pi (u(p\mp m_0)/M+v(q\mp n_0)/N)}}} \\&= \frac{1}{MN}\sum _{p=\pm m_0}^{M-1\pm m_0}{\sum _{q=\pm n_0}^{N-1\pm n_0}{f(p,q)e^{-j2\pi (up/M+vq/N)}}} \\&e^{\pm j2\pi (um_0/M+vn_0/N)} \\&= e^{\pm j2\pi (um_0/M+vn_0/N)}\frac{1}{MN}\sum _{p=\pm m_0}^{M-1\pm m_0}{\sum _{q=\pm n_0}^{N-1\pm n_0}{f(p,q)}}\\&e^{-j2\pi (up/M+vq/N)} \\&\overset{According \ to \ periodicity}{=} e^{\pm j2\pi (um_0/M+vn_0/N)} F(u,v) \end{aligned}$$

So \(f(m\pm m_0,n\pm n_0)\Leftrightarrow e^{\pm j2\pi (um_0/M+vn_0/N)}F(u,v)\). According to Formula (13),

$$\begin{aligned}&IDFT(F(u\pm u_0,v\pm v_0))\\&= \sum _{u=0}^{M-1}{\sum _{v=0}^{N-1}{F(u\pm u_0,v\pm v_0)e^{j2\pi (um/M+vn/N)}}} \\&\overset{Let \ u\pm u_0=\lambda ,u=\lambda \mp u_0;v\pm v_0=\tau ,v=\tau \mp v_0}{=} \\&\sum _{\lambda =\pm u_0}^{M-1\pm u_0}{\sum _{\tau =\pm v_0}^{N-1\pm v_0}{F(\lambda ,\tau )e^{j2\pi ((\lambda \mp u_0)m/M+(\tau \mp v_0)n/N)}}} \\&= \sum _{\lambda =\pm u_0}^{M-1\pm u_0}{\sum _{\tau =\pm v_0}^{N-1\pm v_0}{F(\lambda ,\tau )e^{j2\pi (\lambda m/M+\tau n/N)}}} \\&e^{\mp j2\pi (u_0m/M+v_0n/N)} = e^{\mp j2\pi (u_0m/M+v_0n/N)} \\&\sum _{\lambda =\pm u_0}^{M-1\pm u_0}{\sum _{\tau =\pm v_0}^{N-1\pm v_0}{F(\lambda ,\tau )e^{j2\pi (\lambda m/M+\tau n/N)}}} \\&\overset{According \ to \ periodicity}{=} e^{\mp j2\pi (u_0m/M+v_0n/N)} f(m,n) \end{aligned}$$

So \(e^{\mp j2\pi (um_0/M+vn_0/N)}f(m,n)\Leftrightarrow F(u\pm u_0,v\pm v_0)\). In summary, Formula (17) holds. \(\square \)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shan, C., Ou, J. & Chen, X. Matrix-product neural network based on sequence block matrix product. J Supercomput 78, 8467–8492 (2022). https://doi.org/10.1007/s11227-021-04194-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-021-04194-5

Keywords

Search

Navigation