Skip to main content
SpringerLink
Log in

Complex Table Structure Recognition in the Wild Using Transformer and Identity Matrix-Based Augmentation

  • Conference paper
  • First Online:
Frontiers in Handwriting Recognition (ICFHR 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13639))

Included in the following conference series:

Abstract

Tables are a widely used and efficient data structure. Although people can intuitively understand table contents, it remains challenging for machines, especially the tables taken in the wild. Previous methods mainly focus on scanned or PDF tables, but ignore investigating camera-based tables. This paper treats table structure recognition (TSR) as an image-to-sequence recognition task and adopts an end-to-end trainable model for complex TSR in the wild. Specifically, the model consists of a CNN-based encoder and two Transformer-based decoding branches, which can simultaneously predict the logical and physical structures of a table. Currently available camera-based table datasets are scarce, but deep learning methods heavily rely on large-scale datasets. To alleviate data insufficiency and boost model’s performance, we propose a new and effective table data augmentation method, called TabSplitter. Due to the complex structure caused by cells spanning multiple rows or columns, directly cropping will lead to damage and change the properties of these cells. To solve this problem, we proposed a matrix representation, named Identity Matrix (IM), to describe the table structure. Based on IM, we crop the tables and correct the cells whose attributes have changed, thus enhancing data diversity. Furthermore, the proposed IM facilitates the pre-processing of data and post-processing of predictions. Experimental results on several datasets demonstrate the effectiveness of the model and the TabSplitter for TSR, especially for complex tables in the wild.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Chi, Z., Huang, H., Xu, H.D., Yu, H., Yin, W., Mao, X.L.: Complicated table structure recognition. arXiv preprint arXiv:1908.04729 (2019)

  2. Coüasnon, B., Lemaitre, A.: Recognition of tables and forms. In: Handbook of Document Image Processing and Recognition (2014)

    Google Scholar 

  3. Desai, H., Kayal, P., Singh, M.: TabLeX: a benchmark dataset for structure and content information extraction from scientific tables. In: Lladós, J., Lopresti, D., Uchida, S. (eds.) ICDAR 2021. LNCS, vol. 12822, pp. 554–569. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86331-9_36

    Chapter  Google Scholar 

  4. Graves, A., Fernández, S., Gomez, F., Schmidhuber, J.: Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: International Conference on Machine Learning, pp. 369–376 (2006)

    Google Scholar 

  5. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  6. He, Y., et al.: PingAN-VCGroup’s solution for ICDAR 2021 competition on scientific table image recognition to latex. arXiv preprint arXiv:2105.01846 (2021)

  7. Hirayama, Y.: A method for table structure analysis using DP matching. In: Proceedings of 3rd International Conference on Document Analysis and Recognition, vol. 2, pp. 583–586 (1995)

    Google Scholar 

  8. Itonori, K.: Table structure recognition based on textblock arrangement and ruled line position. In: International Conference on Document Analysis and Recognition, pp. 765–768 (1993)

    Google Scholar 

  9. Khan, S.A., Khalid, S.M.D., Shahzad, M.A., Shafait, F.: Table structure extraction with bi-directional gated recurrent unit networks. In: International Conference on Document Analysis and Recognition, pp. 1366–1371 (2019)

    Google Scholar 

  10. Kieninger, T., Dengel, A.: The T-recs table recognition and analysis system. In: Lee, S.-W., Nakano, Y. (eds.) DAS 1998. LNCS, vol. 1655, pp. 255–270. Springer, Heidelberg (1999). https://doi.org/10.1007/3-540-48172-9_21

    Chapter  Google Scholar 

  11. Li, M., Cui, L., Huang, S., Wei, F., Zhou, M., Li, Z.: TableBank: table benchmark for image-based table detection and recognition. In: Language Resources and Evaluation Conference, pp. 1918–1925 (2020)

    Google Scholar 

  12. Li, Y., Huang, Z., Yan, J., Zhou, Y., Ye, F., Liu, X.: GFTE: graph-based financial table extraction. In: International Conference on Pattern Recognition, pp. 644–658 (2021)

    Google Scholar 

  13. Liu, H., et al.: Show, read and reason: table structure recognition with flexible context aggregator. In: ACM International Conference on Multimedia, pp. 1084–1092 (2021)

    Google Scholar 

  14. Liu, Y., et al.: Exploring the capacity of an orderless box discretization network for multi-orientation scene text detection. Int. J. Comput. Vision 129(6), 1972–1992 (2021)

    Article  Google Scholar 

  15. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)

    Google Scholar 

  16. Long, R., et al.: Parsing table structures in the wild. In: IEEE International Conference on Computer Vision, pp. 944–952 (2021)

    Google Scholar 

  17. Lu, N., et al.: MASTER: multi-aspect non-local network for scene text recognition. Pattern Recognit. 117, 107980 (2021)

    Google Scholar 

  18. Nassar, A., Livathinos, N., Lysak, M., Staar, P.: TableFormer: table structure understanding with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4614–4623 (2022)

    Google Scholar 

  19. Paliwal, S.S., D, V., Rahul, R., Sharma, M., Vig, L.: TableNet: Deep learning model for end-to-end table detection and tabular data extraction from scanned document images. In: International Conference on Document Analysis and Recognition, pp. 128–133 (2019)

    Google Scholar 

  20. Prasad, D., Gadpal, A., Kapadni, K., Visave, M., Sultanpure, K.: CascadeTabNet: an approach for end to end table detection and structure recognition from image-based documents. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 572–573 (2020)

    Google Scholar 

  21. Qasim, S.R., Mahmood, H., Shafait, F.: Rethinking table recognition using graph neural networks. In: International Conference on Document Analysis and Recognition, pp. 142–147 (2019)

    Google Scholar 

  22. Qiao, L., et al.: LGPMA: complicated table structure recognition with local and global pyramid mask alignment. In: Lladós, J., Lopresti, D., Uchida, S. (eds.) ICDAR 2021. LNCS, vol. 12821, pp. 99–114. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86549-8_7

    Chapter  Google Scholar 

  23. Raja, S., Mondal, A., Jawahar, C.V.: Table structure recognition using top-down and bottom-up cues. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12373, pp. 70–86. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58604-1_5

    Chapter  Google Scholar 

  24. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, vol. 28 (2015)

    Google Scholar 

  25. Schreiber, S., Agne, S., Wolf, I., Dengel, A., Ahmed, S.: DeepDeSRT: deep learning for detection and structure recognition of tables in document images. In: International Conference on Document Analysis and Recognition, pp. 1162–1167 (2017)

    Google Scholar 

  26. Shi, B., Bai, X., Yao, C.: An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39(11), 2298–2304 (2017)

    Article  Google Scholar 

  27. TAL_OCR_TABLE: Tal table recognition technology challenge (2021). https://ai.100tal.com/dataset

  28. Tensmeyer, C., Morariu, V.I., Price, B., Cohen, S., Martinez, T.: Deep splitting and merging for table structure decomposition. In: International Conference on Document Analysis and Recognition, pp. 114–121 (2019)

    Google Scholar 

  29. Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  30. Wright, L., Demeure, N.: Ranger21: a synergistic deep learning optimizer. arXiv preprint arXiv:2106.13731 (2021)

  31. Xue, W., Yu, B., Wang, W., Tao, D., Li, Q.: TGRNet: a table graph reconstruction network for table structure recognition. In: IEEE International Conference on Computer Vision, pp. 1295–1304 (2021)

    Google Scholar 

  32. Zheng, X., Burdick, D., Popa, L., Zhong, X., Wang, N.X.R.: Global table extractor (GTE): a framework for joint table identification and cell structure recognition using visual context. In: IEEE Winter Conference on Applications of Computer Vision, pp. 697–706 (2021)

    Google Scholar 

  33. Zhong, X., ShafieiBavani, E., Jimeno Yepes, A.: Image-based table recognition: data, model, and evaluation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12366, pp. 564–580. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58589-1_34

    Chapter  Google Scholar 

  34. Zhou, X., Wang, D., Krähenbühl, P.: Objects as points. In: arXiv preprint arXiv:1904.07850 (2019)

Download references

Acknowledgments

This research is supported in part by NSFC (Grant No. 61936003), GD-NSF (No. 2017A030312006, No. 2021A1515011870), Zhuhai Industry Core and Key Technology Research Project (No. 2220004002350), and the Science and Technology Foundation of Guangzhou Huangpu Development District (Grant 2020GH17).

Author information

Authors and Affiliations

  1. South China University of Technology, Guangzhou, China

    Bangdong Chen, Dezhi Peng, Jiaxin Zhang, Yujin Ren & Lianwen Jin

  2. SCUT-Zhuhai Institute of Modern Industrial Innovation, Zhuhai, China

    Lianwen Jin

Authors
  1. Bangdong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dezhi Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiaxin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yujin Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lianwen Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Lianwen Jin .

Editor information

Editors and Affiliations

  1. Walmart Inc., Hoboken, NJ, USA

    Utkarsh Porwal

  2. Universitat Autònoma de Barcelona, Barcelona, Spain

    Alicia Fornés

  3. National University of Sciences and Technology (NUST), Islamabad, Pakistan

    Faisal Shafait

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Chen, B., Peng, D., Zhang, J., Ren, Y., Jin, L. (2022). Complex Table Structure Recognition in the Wild Using Transformer and Identity Matrix-Based Augmentation. In: Porwal, U., Fornés, A., Shafait, F. (eds) Frontiers in Handwriting Recognition. ICFHR 2022. Lecture Notes in Computer Science, vol 13639. Springer, Cham. https://doi.org/10.1007/978-3-031-21648-0_37

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-21648-0_37

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-21647-3

  • Online ISBN: 978-3-031-21648-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation