Skip to main content
SpringerLink
Log in

A deep learning based bank card detection and recognition method in complex scenes

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

With the widespread use of mobile Internet, mobile payment has become a part of daily life, and bank card recognition in natural scenes has become a hot topic. Although printed character recognition has achieved remarkable success in recent years, bank card recognition is not limited to traditional printed character recognition. There are two types of bank cards: unembossed bank cards, such as most debit cards which usually use printed characters, and embossed bank cards, such as most credit cards which mainly use raised characters. Recognition of raised characters is very challenging due to its own characteristics, and there is a lack of fast and good methods to handle it. To better recognize raised characters, we propose an effective method based on deep learning to detect and recognize bank cards in complex natural scenes. The method can accurately recognize the card number characters on embossed and unembossed bank cards. First, to break the limitation that YOLOv3 algorithm is usually used for object detection, we propose a novel approach that enables YOLOv3 to be used not only for bank card detection and classification, but also for character recognition. The CANNYLINES algorithm is used for rectification and the Scharr operator is introduced to locate the card number region. The proposed method can satisfy bank card detection, classification and character recognition in complex natural scenes, such as complex backgrounds, distorted card surfaces, uneven illumination, and characters with the same or similar color to the background. To further improve the recognition accuracy, a printed character recognition model based on ResNet-32 is proposed for the unembossed bank cards. According to the color and morphological characteristics of embossed bank cards, raised character recognition model combining traditional morphological methods and LeNet-5 convolutional neural network is proposed for the embossed bank cards. The experimental results on the collected bank card dataset and bank card number dataset show that our proposed method can effectively detect and identify different types of bank cards. The accuracy of the detection and classification of bank cards reaches 100%. The accuracy of the raised characters recognition on the embossed bank card is 99.31%, and the accuracy of the printed characters recognition on the unembossed bank card reaches 100%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  1. Cai S, Wen J, Xu H et al. (2016) Bank card and ID card number recognition in android financial APP[C]. International Conference on Smart Computing & Communication, 205–213

  2. Ge J, Fang Z, Tao Q (2020) Bank card number recognition based on deep learning[C]. IEEE 4th information technology, networking, Electronic and Automation Control Conference (ITNEC), 863–867

  3. Yun L, Ruidi M, Hui L, Xuehong C (2019) End-to-end Bank card number recognition algorithm based on DenseNet [C]. International Workshop on Electromagnetics: Applications and Student Innovation Competition (iWEM)

  4. Dong Q, Zhang R, Jing C, et al. (2019) A method of Bank card number recognition and end-to-end integration based on deep learning[C]. 2019 Chinese Automation Congress (CAC), 5060–5063

  5. Xin Y, Shi P, Han S (2019) An Automatic Location and Recognition Method for Bank Card Number[C]. Proceedings of the International Conference on Robotics, Intelligent Control and Artificial Intelligence, 728–732

  6. Chen B, Shi L, Ke X (2019) A robust moving object detection in multi-scenario big data for video surveillance [J]. IEEE Transactions on Circuits and Systems for Video Technology 29(4):982–995

    Article  Google Scholar 

  7. Zhong Z, Sun L, Huo Q (2019) Improved localization accuracy by LocNet for faster R-CNN based text detection in natural scene images[J]. Pattern Recogn 96:106986

    Article  Google Scholar 

  8. Bazazian D, Gómez R, Nicolaou A et al (2019) Fast: facilitated and accurate scene text proposals through fcn guided pruning[J]. Pattern Recogn Lett 119:112–120

    Article  Google Scholar 

  9. Saha S, Chakraborty N, Kundu S et al (2020) Multi-lingual scene text detection and language identification[J]. Pattern Recogn Lett 138:16–22

    Article  Google Scholar 

  10. He W, Zhang XY, Yin F et al (2018) Multi-oriented and multi-lingual scene text detection with direct regression[J]. IEEE Trans Image Process 27(11):5406–5419

    Article  MathSciNet  Google Scholar 

  11. Rong X, Yi C, Tian Y (2019) Unambiguous scene text segmentation with referring expression comprehension[J]. IEEE Trans Image Process 29:591–601

    Article  MathSciNet  Google Scholar 

  12. Khare V, Shivakumara P, Chan CS et al (2019) A novel character segmentation-reconstruction approach for license plate recognition[J]. Expert Syst Appl 131:219–239

    Article  Google Scholar 

  13. Zhang Z, Zhao H, Ling K et al (2019) Effective subword segmentation for text comprehension[J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing 27(11):1664–1674

    Article  Google Scholar 

  14. Bernard M, Thiolliere R, Saksida A et al (2020) WordSeg: standardizing unsupervised word form segmentation from text[J]. Behav Res Methods 52(1):264–278

    Article  Google Scholar 

  15. Bonechi S, Bianchini M, Scarselli F et al (2020) Weak Supervision for Generating Pixel–Level Annotations in Scene Text Segmentation[J]. Pattern Recogn Lett 138:1–7

    Article  Google Scholar 

  16. Wu B, Wei B, Liu J, et al. (2020) Faceted text segmentation via multitask learning[J]. IEEE Transactions on Neural Networks and Learning Systems

  17. Xie H, Fang S, Zha ZJ et al (2019) Convolutional attention networks for scene text recognition[J]. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 15(1):1–17

    Google Scholar 

  18. Chen X, Wang T, Zhu Y et al (2020) Adaptive embedding gate for attention-based scene text recognition[J]. Neurocomputing 381:261–271

    Article  Google Scholar 

  19. Yang L, Wang P, Li H et al (2020) A holistic representation guided attention network for scene text recognition[J]. Neurocomputing 414:67–75

    Article  Google Scholar 

  20. Luo C, Jin L, Sun Z (2019) Moran: a multi-object rectified attention network for scene text recognition[J]. Pattern Recogn 90:109–118

    Article  Google Scholar 

  21. Yousef M, Hussain KF, Mohammed US (2020) Accurate, data-efficient, unconstrained text recognition with convolutional neural networks[J]. Pattern Recogn 108:107482

    Article  Google Scholar 

  22. Huang Y, Sun Z, Jin L et al (2020) EPAN: effective parts attention network for scene text recognition[J]. Neurocomputing 376:202–213

    Article  Google Scholar 

  23. Girshick R (2015) Fast r-cnn[C]. Proceedings of the IEEE International Conference on Computer Vision: 1440–1448

  24. Ren S, He K, Girshick R et al (2017) Faster R-CNN: towards real-time object detection with region proposal networks[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence 6:1137–1149

    Article  Google Scholar 

  25. Redmon J, Divvala S, Girshick R et al (2016) You only look once: unified, real-time object detection[C]. Proc IEEE Conf Comput Vis Pattern Recognit:779–788

  26. Xu ZF, Jia RS, Sun HM et al (2020) Light-YOLOv3: fast method for detecting green mangoes in complex scenes using picking robots[J]. Appl Intell 50(12):4670–4687

    Article  Google Scholar 

  27. Sun C, Ai Y, Wang S et al (2020) Mask-guided SSD for small-object detection[J]. Appl Intell 20:1–12

    Google Scholar 

  28. Ke X, Zhou M, Niu Y, Guo W (2017) Data equilibrium based automatic image annotation by fusing deep model and semantic propagation [J]. Pattern Recogn 71:60–77

    Article  Google Scholar 

  29. Law H, Deng J (2018) CornerNet: detecting objects as paired Keypoints [C]. The European Conference on Computer Vision (ECCV), 734–750

  30. Duan K, Bai S, Xie L, et al. (2019) CenterNet: Keypoint Triplets for Object Detection[C]. Proceedings of the IEEE conference on computer vision and pattern recognition

  31. Lu X, Yao J, Li K, et al. (2015) Cannylines: a parameter-free line segment detector[C], 2015 IEEE International Conference on Image Processing (ICIP), 507–511

  32. Grompone von Gioi R, Jakubowicz J, Morel J-M, Gregory Randall LSD (2010) A fast line segment detector with a false detection control[J]. IEEE Trans Pattern Anal Mach Intell 32(4):722–732

    Article  Google Scholar 

  33. Akinlar C, Topal C (2011) EDLines: a real-time line segment detector with a false detection control[J]. Pattern Recogn Lett 32(13):1633–1642

    Article  Google Scholar 

  34. Kaiming H et al. (2016) Deep residual learning for image recognition[C]. Proceedings of the IEEE conference on computer vision and pattern recognition

  35. Ke X, Zou J, Niu Y (2019) End-to-end automatic image annotation based on deep CNN and multi-label data augmentation[J]. IEEE Transactions on Multimedia 21(8):2093–2106

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant 61,972,097, in part by the Natural Science Foundation of Fujian Province under Grant 2020 J01494, in part by the Industry-Academy Cooperation Project of Fujian Province under Grant 2018H6010, in part by the University Production Project of Fujian Province under Grant 2017H6008, in part by the Fujian Collaborative Innovation Center for Big Data Application in Governments, and in part by the Fujian Engineering Research Center of Big Data Analysis and Processing. The authors also gratefully acknowledge the helpful comments and suggestions of the reviewers.

Author information

Authors and Affiliations

  1. School of Computer Science and Communications Engineering, Jiangsu University, Jiangsu, China

    Hanyang Lin & Yongzhao Zhan

  2. Jiangsu Start Dima Data Processing Co., Ltd., Jiangsu, China

    Hanyang Lin

  3. College of Mathematics and Computer Science, Fuzhou University, Fujian, China

    Shiqin Liu, Xiao Ke & Yuzhong Chen

Authors
  1. Hanyang Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yongzhao Zhan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shiqin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xiao Ke
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yuzhong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hanyang Lin.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, H., Zhan, Y., Liu, S. et al. A deep learning based bank card detection and recognition method in complex scenes. Appl Intell 52, 15259–15277 (2022). https://doi.org/10.1007/s10489-021-03119-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-021-03119-2

Keywords

Search

Navigation