Skip to main content
Springer Nature Link
Log in

Development of a character CAPTCHA recognition system for the visually impaired community using deep learning

  • Original Paper
  • Published:
Machine Vision and Applications Aims and scope Submit manuscript

Abstract

This study proposed an assistive system to recognize the special character CAPTCHAs for the visually impaired community in the Chinese region. To improve the recognition precision, a convolutional neural network (CNN), which is named Captchanet for recognition, was proposed. Firstly, a ten-layer network architecture was designed and three improved training strategies were proposed for the deep learning model. Secondly, a customized Chinese character training set was designed using a novel and fast method, with the view of overcoming the limitation in labeled data collection and uneven data distribution. Finally, the experiments were conducted on the test set gathered from public websites to test the effectiveness of the proposed Captchanet. The statistical results demonstrated that the Captchanet has better classification performance and has obtained higher success rates of recognition than the well-known machine learning approaches and CNN-based approaches.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Source HWDB 1.0

Fig. 11
Fig. 12

Source: HWDB 1.0

Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Meutzner, H., Kolossa, D.: Reducing the cost of breaking audio CAPTCHAs by active and semi-supervised learning. In: CMLA, pp. 67–73 (2014)

  2. Obimbo, C., Halligan, A., Freitas, P.D.: Captchall: an improvement on the modern text-based captcha. In: ICCS, pp. 496–501 (2013)

  3. Lv, G.: Recognition of multi-fontstyle characters based on convolutional neural network. In: ISCID, pp. 223–225 (2011)

  4. Santosh, K.C.: Character recognition based on DTW-Radon. In: ICDAR, pp. 264–268 (2011)

  5. Dai, F., Gao, H., Liu, D.: Breaking CAPTCHAs with second template matching and BP neural network algorithms. In: IJIM, pp. 126–133 (2013)

  6. Fujioka, H., Zhu, W., Hidaka, A., Kano, H.: Reconstructing dynamic font-based Chinese characters using support vector machine. In: SMC, pp. 2408–2413 (2017)

  7. Zhou, M.K., Zhang, X.Y., Yin, F., Liu, C.L.: Discriminative quadratic feature learning for handwritten Chinese character recognition. Pattern Recogn. 49, 7–18 (2016)

    Article  Google Scholar 

  8. Wang, N., Zhu, X., Zhang, J.: License plate segmentation and recognition of Chinese vehicle based on BPNN. In: CIS, pp. 403–406 (2016)

  9. Kim, H.J., Kim, K.H., Kim, S.K., Lee, J.K.: On-line recognition of handwritten Chinese characters based on hidden markov models. Pattern Recogn. 30(9), 1489–1500 (1997)

    Article  Google Scholar 

  10. Lin, D., Lin, F., Lv, Y., Cai, F., Cao, D.: Chinese character CAPTCHA recognition and performance estimation via deep neural network. Neurocomputing 288, 11–19 (2018)

    Article  Google Scholar 

  11. Tang, Y., Wu, B., Peng, L., Liu, C.: Semi-supervised transfer learning for convolutional neural network based Chinese character recognition. In: ICDAR, pp. 441–447 (2018)

  12. Wang, Y., Huang, Y., Zheng, W., Zhou, Z., Liu, D., Lu, M.: Combining convolutional neural network and self-adaptive algorithm to defeat synthetic multi-digit text-based CAPTCHA. In: ICIT, pp. 980–985 (2017)

  13. Ren, X., Zhou, Y., He, J., Chen, K., Yang, X., Sun, J.: A convolutional neural network based Chinese text detection algorithm via text structure modeling. IEEE Trans. Multimed. 19, 506–518 (2017)

    Article  Google Scholar 

  14. Chen, J., Luo, X., Guo, Y., Zhang, Y., Gong, D.: A survey on breaking technique of text-based CAPTCHA. Secur. Commun. Netw. (2017). https://doi.org/10.1155/2017/6898617

    Article  Google Scholar 

  15. Mori, G., Malik, J.: Recognizing objects in adversarial clutter: breaking a visual CAPTCHA. In: CVPR, pp. 134–141 (2003)

  16. Santosh, K.C., Wendling, L.: Character recognition based on non-linear multi-projection profiles measure. Front. Comput. Sci. 9(5), 678–690 (2015)

    Article  Google Scholar 

  17. Starostenko, O., Cruzperez, C., Ucedaponga, F., Alarconaquino, V.: Breaking text-based CAPTCHAs with variable word and character orientation. Pattern Recogn. 48(4), 1101–1112 (2015)

    Article  Google Scholar 

  18. Guo, P., Deng, Y. W., Zhang, H.: A CAPTCHA image recognition algorithm based on edit distance. In: KEM, pp. 2203–2207 (2011)

  19. Wu, X., Dai, S., Guo, Y., Fujita, H.: A machine learning attack against variable-length Chinese character CAPTCHAs. Appl. Intell. 49(4), 1548–1565 (2019)

    Article  Google Scholar 

  20. Hussain, R., Gao, H., Shaikh, R.A.: Segmentation of connected characters in text-based CAPTCHAs for intelligent character recognition. Multimed. Tools Appl. 76(24), 25547–25561 (2017)

    Article  Google Scholar 

  21. Rui, C., Jing, Y., Ronggui, H., Shuguang, H.: A novel LSTM-RNN decoding algorithm in CAPTCHA recognition. In: IMCCC, pp. 766–771 (2013)

  22. Zhang, T., Zheng, H., Zhang, L.: Verification CAPTCHA based on deep learning. In: CCC, pp. 484–488 (2018)

  23. Chen, J., Luo, X., Liu, Y., Wang, J., Ma, Y.: Selective learning confusion class for text-based CAPTCHA recognition. IEEE Access 7, 22246–22259 (2019)

    Article  Google Scholar 

  24. Guha, R., Das, N., Kundu, M., Nasipuri, M., Santosh, K.C., Member, I.S.: DevNet: an efficient CNN architecture for handwritten Devanagari character recognition. In: IJPRAI (2019)

  25. Song, X., Gao, X., Ding, Y., Wang, Z.: A handwritten Chinese characters recognition method based on sample set expansion and CNN. In: ICSAI, pp. 843–849 (2017)

  26. Wang, Y., Li, X., Liu, C., Ding, X., Chen, Y.: An mqdf-cnn hybrid model for offline handwritten Chinese character recognition. In: ICFHR, pp. 246–249 (2014)

  27. Wang, Q., Lu, Y.: Similar handwritten chinese character recognition using hierarchical CNN model. In: ICDAR, pp. 603–608 (2017)

  28. Li, Z., Teng, N., Jin, M., Lu, H.: Building efficient cnn architecture for offline handwritten Chinese character recognition. In: IJDAR, pp. 233–240 (2018)

  29. Wang, J., Qin, J., Xiang, X., Tan, Y., Pan, N.: CAPTCHA recognition based on deep convolutional neural network. Math. Biosci. Eng. 16(5), 5851–5861 (2019)

    Article  MathSciNet  Google Scholar 

  30. Hu, J., Ma, W., Khan, A., Liu, L.: Recognizing character-matching CAPTCHA using convolutional neural networks with triple loss. In: KSEM, pp. 209–220 (2018)

  31. Liu, C., Yin, F., Wang, D., Wang, Q.: Casia online and offline Chinese handwriting databases. In: ICDAR, pp. 37–41 (2011)

  32. Zhang, L.L., Xie, Y., Luan, X., He, J.: Captcha automatic segmentation and recognition based on improved vertical projection. In: ICCSN, pp. 1167–1172 (2017)

  33. Otsu, N.: A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man. Cybern 9(1), 62–66 (2007)

    Article  MathSciNet  Google Scholar 

  34. Canny, J.: A computational approach to edge detection. IEEE Trans. Pattern Anal. 8(6), 679–698 (1986)

    Article  Google Scholar 

  35. Xiao, X., Jin, L., Yang, Y., Yang, W., Sun, J., Chang, T.: Building fast and compact convolutional neural networks for offline handwritten Chinese character recognition. Pattern Recogn. 72, 72–81 (2017)

    Article  Google Scholar 

  36. Yao, D., Zhu, W., Chen, Y., Zhang, L.: Chinese license plate character recognition based on convolution neural network. In: CAC, pp. 1547–1552 (2017)

  37. Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: ICML, pp. 448–456 (2015)

  38. Xu, B., Wang, N., Chen, T., Li, M.: Empirical evaluation of rectified activations in convolutional network. In: Computer Science (2015)

  39. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)

    MathSciNet  MATH  Google Scholar 

  40. Wang, S., Chen, L., Xu, L.: Deep knowledge training and heterogeneous CNN for handwritten Chinese text recognition. In: ICFHR, pp. 84–89 (2016)

  41. Zhang, W., Li, C., Peng, G., Chen, Y., Zhang, Z.: A deep convolutional neural network with new training methods for bearing fault diagnosis under noisy environment and different working load. Mech. Syst. Signal Process. 100, 439–453 (2018)

    Article  Google Scholar 

  42. Taigman, Y., Yang, M., Ranzato, M., Wolf, L.: DeepFace: closing the gap to human-level performance in face verification. In: CVPR, pp. 1701–1708 (2014)

  43. Mishkin, D., Sergievskiy, N., Matas, J.: Systematic evaluation of convolution neural network advances on the imagenet. Comput. Vis. Image Underst. 161, 11–19 (2017)

    Article  Google Scholar 

  44. Needleman, M.: The Unicode standard. Serials Rev. 26(2), 51–54 (2000)

    Article  Google Scholar 

  45. Masko, D., Hensman, P.: The impact of imbalanced training data for convolutional neural networks, pp. 12–20 (2015)

  46. Si, L., Wang, Z., Xu, R., Tan, C., Liu, X., Xu, J.: Image enhancement for surveillance video of coal mining face based on single-scale retinex algorithm combined with bilateral filtering. Symmetry 9(6), 93 (2017)

    Article  Google Scholar 

  47. Abu-Ain, W., Abdullah, S.N.H.S., Bataineh, B., Abu-Ain, T., Omar, K.: Skeletonization algorithm for binary images. In: Procedia Technology, pp. 704–709 (2013)

  48. Zhang, T.Y., Suen, C.Y.: A fast parallel algorithm for thinning digital patterns. Commun. ACM 27(3), 236–239 (1984)

    Article  Google Scholar 

  49. Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. J Mach. Learn. Res. 9, 249–256 (2010)

    Google Scholar 

  50. Boer, P.T.D., Kroese, D.P., Mannor, S., Rubinstein, R.Y.: A tutorial on the cross-entropy method. Ann. Oper. Res. 134(1), 19–67 (2005)

    Article  MathSciNet  Google Scholar 

  51. Powers, D.M.W.: Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. Mach. Learn. Technol. 2(1), 37–63 (2011)

    MathSciNet  Google Scholar 

  52. Tang, Z., Jiang, W., Zhang, Z.: DenseNet with up-sampling block for recognizing texts in images. Neural Comput. Appl. 32, 7553–7561 (2020)

    Article  Google Scholar 

  53. Zeiler, M. D., Fergus, R.: Visualizing and understanding convolutional networks. In: ECCV, pp. 818–833 (2013)

Download references

Acknowledgements

The support of National Natural Science Foundation of China (No. 51975568), National Natural Science Foundation of Jiangsu Province (No. BK20191341) and Priority Academic Program Development of Jiangsu Higher Education Institutions (PAPD) in carrying out this research are gratefully acknowledged.

Author information

Authors and Affiliations

  1. School of Mechatronics Engineering, China University of Mining and Technology, Xuzhou, 221116, China

    Xiaohui Zhang & Xinhua Liu

  2. Technology and Innovation Research Center of JiangYan EDZ, Taizhou, 225500, China

    Xinhua Liu

  3. College of Engineering, University of Texas, 500 W University Ave, El Paso, TX, 79968, USA

    Thompson Sarkodie-Gyan

  4. School of Engineering, Ocean University of China, Tsingtao, 266100, China

    Zhixiong Li

  5. Yonsei Frontier Lab, Yonsei University, 50 Yonsei-ro, Seodaemun-gu, Seoul, 03722, Republic of Korea

    Zhixiong Li

Authors
  1. Xiaohui Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xinhua Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Thompson Sarkodie-Gyan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhixiong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Xinhua Liu or Zhixiong Li.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, X., Liu, X., Sarkodie-Gyan, T. et al. Development of a character CAPTCHA recognition system for the visually impaired community using deep learning. Machine Vision and Applications 32, 29 (2021). https://doi.org/10.1007/s00138-020-01160-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s00138-020-01160-8

Keywords