Abstract
Document image classification is one of the most important components in business automation workflow. Therefore, a range of different supervised image classification methods have been proposed, which rely on a large amount of labeled data, which is rarely available in practice. Furthermore, retraining of these models is necessary upon the introduction of new classes. In this paper, we analyze the potential of zero-shot document image classification based on computing the agreement between the images and the textual embeddings of class names/descriptions. This enables the deployment of document image classification models without the availability of any training data at zero training cost, alongside providing seamless integration of new classes. Our results show that using zero-shot recognition achieves significantly better than chance performance on document image classification benchmarks (49.51% accuracy on Tobacco-3482 in contrast to 10% random classifier accuracy and 39.22% on RVL-CDIP dataset in contrast to 6.25% random classifier accuracy). We also show that the representation learned by a vision transformer using image-text pairs is competitive to CNNs by training a linear SVM on top of the pre-computed representations which achieves comparable performance to state-of-the-art convolutional networks (85.74% on Tobacco-3482 dataset in contrast to 71.67% from ImageNet pretrained ResNet-50). Even though the initial results look encouraging, there is still a large gap to cover for zero-shot recognition within the domain of document images in contrast to natural scene image classification, which achieves comparable performance to fully supervised baselines. Our preliminary findings pave the way for the deployment of zero-shot document image classification in production settings.
Keywords
This work was supported by the BMBF project DeFuseNN (Grant 01IW17002) and the NVIDIA AI Lab (NVAIL) program.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Afzal, M.Z., et al.: Deepdocclassifier: document classification with deep convolutional neural network. In: 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pp. 1111–1115 (2015). https://doi.org/10.1109/ICDAR.2015.7333933
Afzal, M.Z., Kölsch, A., Ahmed, S., Liwicki, M.: Cutting the error by half: investigation of very deep CNN and advanced training strategies for document image classification. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), vol. 1, pp. 883–888. IEEE (2017)
Asim, M.N., Khan, M.U.G., Malik, M.I., Razzaque, K., Dengel, A., Ahmed, S.: Two stream deep network for document image classification. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1410–1416. IEEE (2019)
Baevski, A., Zhou, H., Mohamed, A., Auli, M.: wav2vec 2.0: a framework for self-supervised learning of speech representations. arXiv preprint arXiv:2006.11477 (2020)
Chen, N., Blostein, D.: A survey of document image classification: problem statement, classifier architecture and performance evaluation. IJDAR 10(1), 1–16 (2007)
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 (2020)
Chen, Y., et al.: Bayesian optimization in AlphaGo. arXiv preprint arXiv:1812.06855 (2018)
Cosma, A., Ghidoveanu, M., Panaitescu-Liess, M., Popescu, M.: Self-supervised representation learning on document images. arXiv preprint arXiv:2004.10605 (2020)
Das, D., Lee, C.G.: Zero-shot image recognition using relational matching, adaptation and calibration. In: 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1–8. IEEE (2019)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
Ferrando, J., et al.: Improving accuracy and speeding up document image classification through parallel systems. In: Krzhizhanovskaya, V.V., et al. (eds.) ICCS 2020. LNCS, vol. 12138, pp. 387–400. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50417-5_29
Grill, J.B., et al.: Bootstrap your own latent: a new approach to self-supervised learning. arXiv preprint arXiv:2006.07733 (2020)
Harley, A.W., Ufkes, A., Derpanis, K.G.: Evaluation of deep convolutional nets for document image classification and retrieval. In: International Conference on Document Analysis and Recognition (ICDAR) (2015)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)
Kang, L., Kumar, J., Ye, P., Li, Y., Doermann, D.: Convolutional neural networks for document image classification. In: 2014 22nd International Conference on Pattern Recognition, pp. 3168–3172. IEEE (2014)
Kölsch, A., Afzal, M.Z., Ebbecke, M., Liwicki, M.: Real-time document image classification using deep CNN and extreme learning machines. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), vol. 1, pp. 1318–1323. IEEE (2017)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Commun. ACM 60(6), 84–90 (2017)
Kumar, J., Doermann, D.: Unsupervised classification of structurally similar document images. In: 2013 12th International Conference on Document Analysis and Recognition, pp. 1225–1229. IEEE (2013)
Kumar, J., Ye, P., Doermann, D.: Learning document structure for retrieval and classification. In: Proceedings of the 21st International Conference on Pattern Recognition (ICPR 2012), pp. 1558–1561. IEEE (2012)
Kumar, J., Ye, P., Doermann, D.: Structural similarity for document image classification and retrieval. Pattern Recogn. Lett. 43, 119–126 (2014)
Noce, L., Gallo, I., Zamberletti, A., Calefati, A.: Embedded textual content for document image classification with convolutional neural networks. In: Proceedings of the 2016 ACM Symposium on Document Engineering, pp. 165–173 (2016)
Radford, A., et al.: Learning transferable visual models from natural language supervision. Image 2, T2 (2021)
Reddy, K.V.U., Govindaraju, V.: Form classification. In: Yanikoglu, B.A., Berkner, K. (eds.) Document Recognition and Retrieval XV, vol. 6815, pp. 302–307. International Society for Optics and Photonics, SPIE (2008). https://doi.org/10.1117/12.766737
Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vision 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y
Sarkar, P.: Learning image anchor templates for document classification and data extraction. In: 2010 20th International Conference on Pattern Recognition, pp. 3428–3431. IEEE (2010)
Shin, C., Doermann, D., Rosenfeld, A.: Classification of document pages using structure-based features. Int. J. Doc. Anal. Recogn. 3(4), 232–247 (2001)
Shin, C.K., Doermann, D.S.: Document image retrieval based on layout structural similarity. In: IPCV, pp. 606–612 (2006)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.: Inception-v4, inception-resnet and the impact of residual connections on learning. arXiv preprint arXiv:1602.07261 (2016)
Vaswani, A., et al.: Attention is all you need. arXiv preprint arXiv:1706.03762 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Siddiqui, S.A., Dengel, A., Ahmed, S. (2021). Analyzing the Potential of Zero-Shot Recognition for Document Image Classification. In: Lladós, J., Lopresti, D., Uchida, S. (eds) Document Analysis and Recognition – ICDAR 2021. ICDAR 2021. Lecture Notes in Computer Science(), vol 12824. Springer, Cham. https://doi.org/10.1007/978-3-030-86337-1_20
Download citation
DOI: https://doi.org/10.1007/978-3-030-86337-1_20
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86336-4
Online ISBN: 978-3-030-86337-1
eBook Packages: Computer ScienceComputer Science (R0)