Skip to main content
SpringerLink
Log in

Multi-scale confusion and filling mechanism for pressure footprint recognition

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Footprint data have large intra-class variances and small inter-class variances; therefore, the key to the footprint recognition problem is to mine and learn the discriminative local details in the footprint images. In this paper, an algorithm based on a multi-scale confusion and filling mechanism is proposed to address the problem of footprint recognition from the perspective of fine-grained image recognition. Firstly, the pressure footprint image is divided evenly into several sub-regions, and the score of each sub-region is calculated by a joint confidence function. Secondly, using the filling mechanism of the Region Filling Module, the region with the lowest score in the split image is filled with a higher one for data enhancement. Then, the filled image is confused once using the Multi-Scale Region Confusion Module, and the regions with high confidence score are confused again to obtain an image with multi-scale information. Finally, the footprint features of the filled image and the confused image are extracted by the backbone network and optimized by the joint loss function to carry out the task of footprint recognition. Comprehensive experiments show that the proposed algorithm achieves 89.3%, 93.4% and 86.5% on three benchmark dataset including CUB-200-2011, Aircraft and Stanford Dogs. Meanwhile, it obtains 97.8% on the footprint dataset.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Data availibility

The footprint dataset generated and analysed during the current study is not publicly available because the study is ongoing and the project has not been completed as well as to protect the privacy of the subjects but is available from the corresponding author on reasonable request.

References

  1. Bao W, Qu J, Wang N et al (2020) Force-tactile footprint recognition based on spatial aggregation weighted convolutional neural network. J Southeast Univ (Natl Sci Edn) 50(5):959–964 (in Chinese)

    Google Scholar 

  2. Basheer S, Nagwanshi K, Bhatia S et al (2021) Fesd: an approach for biometric human footprint matching using fuzzy ensemble learning. IEEE Access 9:26,641-26,663. https://doi.org/10.1109/ACCESS.2021.3057931

    Article  Google Scholar 

  3. Chang D, Ding Y, Xie J et al (2020) The devil is in the channels: mutual-channel loss for fine-grained image classification. IEEE Trans Image Process 29:4683–4695. https://doi.org/10.1109/TIP.2020.2973812

    Article  MATH  Google Scholar 

  4. Chattopadhay A, Sarkar A, Howlader P, et al (2018) Grad-cam++: generalized gradient-based visual explanations for deep convolutional networks. In: 2018 IEEE winter conference on applications of computer vision (WACV), IEEE, pp 839–847. https://doi.org/10.1109/WACV.2018.00097

  5. Chen D, Chen Y, Ma J et al (2021) An ensemble deep neural network for footprint image retrieval based on transfer learning. J Sens 6631:029. https://doi.org/10.1155/2021/6631029

    Article  Google Scholar 

  6. Chen Y, Bai Y, Zhang W, et al (2019) Destruction and construction learning for fine-grained image recognition. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 5152–5161. https://doi.org/10.1109/CVPR.2019.00530

  7. DeVries T, Taylor G (2017) Improved regularization of convolutional neural networks with cutout. Preprint arXiv:1708.04552

  8. Ding Y, Zhou Y, Zhu Y, et al (2019) Selective sparse sampling for fine-grained image recognition. In: 2019 IEEE/CVF international conference on computer vision (ICCV), pp 6598–6607. https://doi.org/10.1109/ICCV.2019.00670

  9. Dubey A, Gupta O, Guo P, et al (2018) Pairwise confusion for fine-grained visual classification. In: Computer Vision—ECCV 2018—15th European Conference, Munich, Germany, September 8–14, 2018, Proceedings, Part XII, pp 71–88. https://doi.org/10.1007/978-3-030-01258-8_5

  10. Dubey A, Gupta O, Raskar R, et al (2018) Maximum-entropy fine grained classification. In: Advances in neural information processing systems 31, pp 637–647. http://papers.nips.cc/paper/7344-maximum-entropy-fine-grained-classification.pdf

  11. Fu J, Zheng H, Mei T (2017) Look closer to see better: recurrent attention convolutional neural network for fine-grained image recognition. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR), pp 4476–4484. https://doi.org/10.1109/CVPR.2017.476

  12. Gao Y, Beijbom O, Zhang N, et al (2016) Compact bilinear pooling. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 317–326. https://doi.org/10.1109/CVPR.2016.41

  13. Gergely S, Lóránt F (2021) Data augmentation by guided deep interpolation. Appl Soft Comput 111(107):680. https://doi.org/10.1016/j.asoc.2021.107680

    Article  Google Scholar 

  14. Goodfellow I, Pouget-Abadie J, Mirza M et al (2014) Generative adversarial networks. Adv Neural Inf Process Syst 3:2672–2680. https://doi.org/10.1145/3422622

    Article  Google Scholar 

  15. Gurney J, Kersting U, Rosenbaum D (2008) Between-day reliability of repeated plantar pressure distribution measurements in a normal population. Gait Posture 27(4):706–709. https://doi.org/10.1016/j.gaitpost.2007.07.002

    Article  Google Scholar 

  16. Han F, Zhu S, Ling Q et al (2022) Gene-cwgan: a data enhancement method for gene expression profile based on improved cwgan-gp. Neural Comput Appl. https://doi.org/10.1007/s00521-022-07417-9

    Article  Google Scholar 

  17. He K, Zhang X, Ren S, et al (2016) Deep residual learning for image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 770–778. https://doi.org/10.1109/CVPR.2016.90

  18. Huang G, Jafari A (2021) Enhanced balancing gan: minority-class image generation. Neural Comput Appl. https://doi.org/10.1007/s00521-021-06163-8

    Article  Google Scholar 

  19. Huang G, Liu Z, Van Der Maaten L, et al (2017) Densely connected convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4700–4708. https://doi.org/10.1109/CVPR.2017.243

  20. Huang S, Xu Z, Tao D, et al (2016) Part-stacked CNN for fine-grained visual categorization. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 1173–1182. https://doi.org/10.1109/CVPR.2016.132

  21. Jia D, Wei D, Richard S, et al (2009) Imagenet: a large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition, pp 248–255. https://doi.org/10.1109/CVPR.2009.5206848

  22. Jia W, Cai H, Gui J et al (2012) Newborn footprint recognition using orientation feature. Neural Comput Appl 21:1855–1863. https://doi.org/10.1007/s00521-011-0530-9

    Article  Google Scholar 

  23. Khokher R, Singh R, Kumar R (2015) Footprint recognition with principal component analysis and independent component analysis. In: Macromolecular symposia, pp 16–26. https://doi.org/10.1002/masy.201400045

  24. Khosla A, Jayadevaprakash N, Yao B, et al (2011) Novel dataset for fine-grained image categorization. In: First workshop on fine-grained visual categorization, IEEE conference on computer vision and pattern recognition. http://people.csail.mit.edu/khosla/papers/fgvc2011.pdf

  25. Krizhevsky A, Sutskever I, Hinton G (2012) Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst 25:1097–1105. https://doi.org/10.1145/3065386

    Article  Google Scholar 

  26. Kulkarni P, Kulkarni V (2015) Human footprint classification using image parameters. In: 2015 international conference on pervasive computing (ICPC), pp 1–5. https://doi.org/10.1109/PERVASIVE.2015.7087011

  27. Kushwaha R, Singal G, Nain N (2021) A texture feature based approach for person verification using footprint bio-metric. Artif Intell Rev 54:1581–1611. https://doi.org/10.1007/s10462-020-09887-6

    Article  Google Scholar 

  28. Lin D, Shen X, Lu C, et al (2015) Deep LAC: deep localization, alignment and classification for fine-grained recognition. In: 2015 IEEE conference on computer vision and pattern recognition (CVPR), pp 1666–1674. https://doi.org/10.1109/CVPR.2015.7298775

  29. Lin T, RoyChowdhury A, Maji S (2015) Bilinear CNN models for fine-grained visual recognition. In: 2015 IEEE international conference on computer vision (ICCV), pp 1449–1457. https://doi.org/10.1109/ICCV.2015.170

  30. Liu X, Xia T, Wang J, et al (2016) Fully convolutional attention localization networks: efficient attention localization for fine-grained recognition. CoRR arXiv:abs/1603.06765

  31. Loshchilov I, Hutter F (2017) Sgdr: stochastic gradient descent with warm restarts. In: 5th international conference on learning representations, ICLR 2017, Toulon, France, April 24–26, 2017, Conference Track Proceedings. https://openreview.net/forum?id=Skq89Scxx

  32. Maji S, Kannala J, Rahtu E, et al (2013) Fine-grained visual classification of aircraft. CoRR arXiv:abs/1306.5151

  33. Mariani G, Scheidegger F, Istrate R, et al (2018) Bagan: data augmentation with balancing gan. arXiv:1803.09655

  34. Moustakidis S, Theocharis J, Giakas G (2008) Subject recognition based on ground reaction force measurements of gait signals. IEEE Trans Syst Man Cybern Part B (Cybern) 38(6):1476–1485. https://doi.org/10.1109/TSMCB.2008.927722

    Article  Google Scholar 

  35. Nguyen D, Phan C, Koo S (2018) Predicting body movements for person identification under different walking conditions. For Sci Int 290:303–309. https://doi.org/10.1016/j.forsciint.2018.07.022

    Article  Google Scholar 

  36. Odena A, Olah C, Shlens J (2017) Conditional image synthesis with auxiliary classifier gans. In: Proceedings of the 34th international conference on machine learning, vol 70. PMLR, pp 2642–2651. http://proceedings.mlr.press/v70/odena17a.html

  37. Osisanwo F, Adetunmbi A, Álese B (2014) Barefoot morphology: a person unique feature for forensic identification. In: The 9th international conference for internet technology and secured transactions (ICITST-2014), pp 356–359. https://doi.org/10.1109/ICITST.2014.7038837

  38. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. In: 3rd international conference on learning representations, ICLR 2015, San Diego, CA, USA, May 7–9, 2015, Conference Track Proceedings. arXiv:1409.1556

  39. Summers C, Dinneen M (2019) Improved mixed-example data augmentation. In: 2019 IEEE winter conference on applications of computer vision (WACV), pp 1262–1270. https://doi.org/10.1109/WACV.2019.00139

  40. Sun M, Yuan Y, Zhou F, et al (2018) Multi-attention multi-class constraint for fine-grained image recognition. In: Computer Vision—ECCV 2018—15th European Conference, Munich, Germany, September 8–14, 2018, Proceedings, Part XVI, pp 834–850. https://doi.org/10.1007/978-3-030-01270-0_49

  41. Sun Q, Wang Q, Zhang J et al (2018) Hyperlayer bilinear pooling with application to fine-grained categorization and image retrieval. Neurocomputing 282:174–183. https://doi.org/10.1016/j.neucom.2017.12.020

    Article  Google Scholar 

  42. Szegedy C, Vanhoucke V, Ioffe S, et al (2016) Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2818–2826. https://doi.org/10.1109/CVPR.2016.308

  43. Wah C, Branson S, Welinder P, et al (2011) The Caltech–UCSD Birds-200-2011 Dataset. Tech. Rep. CNS-TR-2011-001, California Institute of Technology. http://www.vision.caltech.edu/visipedia/papers/CUB_200_2011.pdf

  44. Wang N, Chen X, Zhang Y et al (2020) FtH-Net method for predicting height based on footprint image. J South China Univ Technol (Natl Sci Edn) 48(6):106–113 (in Chinese)

    Google Scholar 

  45. Wang W, Cui Y, Li G et al (2020) A self-attention-based destruction and construction learning fine-grained image classification method for retail product recognition. Neural Comput Appl 32:14,613-14,622. https://doi.org/10.1007/s00521-020-05148-3

    Article  Google Scholar 

  46. Wang Z, Bovik A, Sheikh H et al (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612. https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  47. Wei X, Xie C, Wu J et al (2018) Mask-CNN: localizing parts and selecting descriptors for fine-grained bird species categorization. Pattern Recogn 76:704–714. https://doi.org/10.1016/j.patcog.2017.10.002

    Article  Google Scholar 

  48. Wei X, Zhang Y, Gong Y, et al (2018) Grassmann pooling as compact homogeneous bilinear pooling for fine-grained visual classification. In: Computer Vision—ECCV 2018—15th European Conference, Munich, Germany, September 8–14, 2018, Proceedings, Part III, pp 365–380. https://doi.org/10.1007/978-3-030-01219-9_22

  49. Wei X, Wu J, Cui Q (2019) Deep learning for fine-grained image analysis: a survey. CoRR arXiv:1907.03069

  50. Xia R, Ma Z, Yao Z et al (2013) Gait recognition based on spatio-temporal hog feature of plantar pressure distribution. J Pattern Recogn Artif Intell 26(6):529–536 (in Chinese)

    Google Scholar 

  51. Yang Z, Luo T, Wang D, et al (2018) Learning to navigate for fine-grained classification. In: Computer Vision—ECCV 2018—15th European Conference, Munich, Germany, September 8–14, 2018, Proceedings, Part XIV, pp 438–454. https://doi.org/10.1007/978-3-030-01264-9_26

  52. Yu C, Zhao X, Zheng Q, et al (2018) Hierarchical bilinear pooling for fine-grained visual recognition. In: Computer Vision—ECCV 2018—15th European Conference, Munich, Germany, September 8–14, 2018, Proceedings, Part XVI, pp 595–610. https://doi.org/10.1007/978-3-030-01270-0_35

  53. Yun S, Han D, Oh S, et al (2019) Cutmix: Regularization strategy to train strong classifiers with localizable features. In: Proceedings of the IEEE/CVF international conference on computer vision, pp 6023–6032. https://doi.org/10.1109/ICCV.2019.00612

  54. Zhang C, Pan S, Qi Y et al (2019) A footprint extraction and recognition algorithm based on plantar pressure. Traitement du Signal (5) 36(5):419–424. https://doi.org/10.18280/ts.360506

    Article  Google Scholar 

  55. Zhang H, Cissé M, Dauphin Y, et al (2018) mixup: beyond empirical risk minimization. In: 6th international conference on learning representations, ICLR 2018, Vancouver, BC, Canada, April 30–May 3, 2018, Conference Track Proceedings. https://openreview.net/forum?id=r1Ddp1-Rb

  56. Zhang N, Donahue J, Girshick R, et al (2014) Part-based R-CNNs for fine-grained category detection. In: Computer Vision—ECCV 2014—13th European Conference, Zurich, Switzerland, September 6–12, 2014, Proceedings, Part I, pp 834–849. https://doi.org/10.1007/978-3-319-10590-1_54

  57. Zhang X, Xiong H, Zhou W, et al (2016) Picking deep filter responses for fine-grained image recognition. In: 2016 IEEE conference on computer vision and pattern recognition (CVPR), pp 1134–1142. https://doi.org/10.1109/CVPR.2016.128

  58. Zhang Y, Wang Q, Wang N et al (2019) Footprint recognition algorithm based on multi-modal features. J Huazhong Univ of Sci Tech (Natl Sci Edn) 47(5):73–78 (in Chinese)

    Google Scholar 

  59. Zhao B, Wu X, Feng J et al (2017) Diversified visual attention networks for fine-grained object classification. IEEE Trans Multimed 19(6):1245–1256. https://doi.org/10.1109/TMM.2017.2648498

    Article  Google Scholar 

  60. Zhao F (2018) Biometric identification technology and development trend of physiological characteristics. J Phys Conf Ser. https://doi.org/10.1088/1742-6596/1060/1/012047

    Article  Google Scholar 

  61. Zhao J, Peng Y, He X (2020) Attribute hierarchy based multi-task learning for fine-grained image classification. Neurocomputing 395:150–159. https://doi.org/10.1016/j.neucom.2018.02.109

    Article  Google Scholar 

  62. Zheng H, Fu J, Mei T, et al (2017) Learning multi-attention convolutional neural network for fine-grained image recognition. In: 2017 IEEE international conference on computer vision (ICCV), pp 5219–5227. https://doi.org/10.1109/ICCV.2017.557

  63. Zheng H, Fu J, Zha Z, et al (2019) Looking for the Devil in the Details: Learning trilinear attention sampling network for fine-grained image recognition. In: 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR), pp 5007–5016. https://doi.org/10.1109/CVPR.2019.00515

  64. Zhong Z, Zheng L, Kang G, et al (2020) Random erasing data augmentation. In: Proceedings of the AAAI conference on artificial intelligence, pp 13,001–13,008. https://aaai.org/ojs/index.php/AAAI/article/view/7000

Download references

Acknowledgements

This work was supported by the National Key R &D Program of China (Grant No. 2018YFC0807302), National Natural Science Foundation of China (Grant Nos. 61772032 and 31971789) and the University Natural Science Research Project of Anhui Province (Grant No. KJ2019A0027).

Author information

Authors and Affiliations

  1. School of Electronics Information Engineering, Anhui University, Hefei, 230601, China

    Yan Zhang, Yongsheng Sun, Nian Wang, Zijian Gao, Jing Zhu & Jun Tang

Authors
  1. Yan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yongsheng Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nian Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zijian Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jing Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Jun Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nian Wang.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, Y., Sun, Y., Wang, N. et al. Multi-scale confusion and filling mechanism for pressure footprint recognition. Neural Comput & Applic 35, 375–392 (2023). https://doi.org/10.1007/s00521-022-07777-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-022-07777-2

Keywords

Search

Navigation