Skip to main content
SpringerLink
Log in

Dewarping Document Image in Complex Scene by Geometric Control Points

  • Conference paper
  • First Online:
Pattern Recognition (ACPR 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14408))

Included in the following conference series:

  • 231 Accesses

Abstract

In the process of document digitization, document images captured by mobile devices suffer from physical distortion, which is detrimental to subsequent document processing. Geometric information of the distorted document images provide global and local constraints that can assist in document dewarping. In this paper, we propose a novel document dewarping method which focus on utilizing the geometric control points such as document boundaries and textlines. Specifically, our method first extracts the boundary source control points and textline source control points and predicts their corresponding forward mapping as target control points. Eventually the sparse mapping between control points is converted into a dense backward mapping by Thin Plate Splines interpolation. Our method can obtain the backward mapping directly and explicitly by interpolation between control points, without solving the time-consuming optimization problem. Quantitative and qualitative evaluation show that our method can dewarp document images with various distortion types, and improve the inference speed by a factor of three over the existing geometric element based rectification methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 74.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Zhang, L., Zhang, Y., Tan, C.: An improved physically-based method for geometric restoration of distorted document images. IEEE Trans. Pattern Anal. Mach. Intell. 30(4), 728–34 (2008)

    Article  Google Scholar 

  2. Brown, M.S., Seales, W.B.: Document restoration using 3D shape: a general deskewing algorithm for arbitrarily warped documents. In: Proceedings Eighth IEEE International Conference on Computer Vision, ICCV 2001 2001 Jul 7, vol. 2, pp. 367–374. IEEE (2001)

    Google Scholar 

  3. Yamashita, A., Kawarago, A., Kaneko, T., Miura, K.T.: Shape reconstruction and image restoration for non-flat surfaces of documents with a stereo vision system. In: Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004. 2004 Aug 26, vol. 1, pp. 482–485. IEEE

    Google Scholar 

  4. Meng, G., Wang, Y., Qu, S., Xiang, S., Pan, C.: Active flattening of curved document images via two structured beams. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3890–3897 (2014)

    Google Scholar 

  5. Koo, H.I., Kim, J., Cho, N.I.: Composition of a dewarped and enhanced document image from two view images. IEEE Trans. Image Process. 18(7), 1551–62 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  6. Tsoi, Y.C., Brown, M.S.: Multi-view document rectification using boundary. In: 2007 IEEE Conference on Computer Vision and Pattern Recognition 2007 Jun 17, pp. 1–8. IEEE (2007)

    Google Scholar 

  7. You, S., Matsushita, Y., Sinha, S., Bou, Y., Ikeuchi, K.: Multiview rectification of folded documents. IEEE Trans. Pattern Anal. Mach. Intell. 40(2), 505–11 (2017)

    Article  Google Scholar 

  8. Zhang, L., Yip, A.M., Brown, M.S., Tan, C.L.: A unified framework for document restoration using inpainting and shape-from-shading. Pattern Recogn. 42(11), 2961–78 (2009)

    Article  MATH  Google Scholar 

  9. Cao, H., Ding, X., Liu, C.: Rectifying the bound document image captured by the camera: A model based approach. In: Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings. 2003 Aug 6, pp. 71–75. IEEE

    Google Scholar 

  10. Brown, M.S., Tsoi, Y.C.: Geometric and shading correction for images of printed materials using boundary. IEEE Trans. Image Process. 15(6), 1544–54 (2006)

    Article  Google Scholar 

  11. He, Y., Pan, P., Xie, S., Sun, J., Naoi, S.: A book dewarping system by boundary-based 3D surface reconstruction. In: 2013 12Th international Conference on Document Analysis and Recognition 2013 Aug 25, pp. 403–407. IEEE (2013)

    Google Scholar 

  12. Wada, T., Ukida, H., Matsuyama, T.: Shape from shading with interreflections under a proximal light source: distortion-free copying of an unfolded book. Int. J. Comput. Vision 24, 125–35 (1997)

    Article  Google Scholar 

  13. Meng, G., Su, Y., Wu, Y., Xiang, S., Pan, C.: Exploiting vector fields for geometric rectification of distorted document images. In: Proceedings of the European Conference on Computer Vision (ECCV) 2018, pp. 172–187 (2018)

    Google Scholar 

  14. Ezaki, H., Uchida, S., Asano, A., Sakoe, H.: Dewarping of document image by global optimization. In: Eighth International Conference on Document Analysis and Recognition (ICDAR’05) 2005 Aug 31, pp. 302–306. IEEE (2005)

    Google Scholar 

  15. Ma, K., Shu, Z., Bai, X., Wang, J., Samaras, D.: Docunet: document image unwarping via a stacked u-net. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2018, pp. 4700–4709 (2018)

    Google Scholar 

  16. Das, S., Ma, K., Shu, Z., Samaras, D., Shilkrot, R.: Dewarpnet: single-image document unwarping with stacked 3d and 2d regression networks. In: Proceedings of the IEEE/CVF International Conference on Computer Vision 2019, pp. 131–140 (2019)

    Google Scholar 

  17. Xie, G.-W., Yin, F., Zhang, X.-Y., Liu, C.-L.: Dewarping document image by displacement flow estimation with fully convolutional network. In: Bai, X., Karatzas, D., Lopresti, D. (eds.) DAS 2020. LNCS, vol. 12116, pp. 131–144. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-57058-3_10

    Chapter  Google Scholar 

  18. Xie, G.-W., Yin, F., Zhang, X.-Y., Liu, C.-L.: Document dewarping with control points. In: Lladós, J., Lopresti, D., Uchida, S. (eds.) ICDAR 2021. LNCS, vol. 12821, pp. 466–480. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86549-8_30

    Chapter  Google Scholar 

  19. Liu, X., Meng, G., Fan, B., Xiang, S., Pan, C.: Geometric rectification of document images using adversarial gated unwarping network. Pattern Recogn. 1(108), 107576 (2020)

    Article  Google Scholar 

  20. Li, X., Zhang, B., Liao, J., Sander, P.V.: Document rectification and illumination correction using a patch-based CNN. ACM Trans. Graph. (TOG) 38(6), 1–1 (2019)

    Google Scholar 

  21. Das, S., Singh, K.Y., Wu, J., Bas, E., Mahadevan, V., Bhotika, R., Samaras, D.: End-to-end piece-wise unwarping of document images. In: Proceedings of the IEEE/CVF International Conference on Computer Vision 2021, pp. 4268–4277 (2021)

    Google Scholar 

  22. Feng, H., Wang, Y., Zhou, W., Deng, J., Li, H.: Doctr: Document image transformer for geometric unwarping and illumination correction. arXiv preprint arXiv:2110.12942. 2021 Oct 25

  23. Zhang, J., Luo, C., Jin, L., Guo, F., Ding, K.: Marior: margin removal and iterative content rectification for document dewarping in the wild. arXiv preprint arXiv:2207.11515. 2022 Jul 23

  24. Ma, K., Das, S., Shu, Z., Samaras, D.: Learning from documents in the wild to improve document unwarping. In: ACM SIGGRAPH 2022 Conference Proceedings 2022 Jul 27, pp. 1–9

    Google Scholar 

  25. Xue, C., Tian, Z., Zhan, F., Lu, S., Bai, S.: Fourier document restoration for robust document dewarping and recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2022, pp. 4573–4582

    Google Scholar 

  26. Jiang, X., Long, R., Xue, N., Yang, Z., Yao, C., Xia, G.S.: Revisiting document image dewarping by grid regularization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2022, pp. 4543–4552 (2022)

    Google Scholar 

  27. Feng H, Zhou W, Deng J, Wang Y, Li H. Geometric representation learning for document image rectification. In: European Conference on Computer Vision 2022 Oct 22, pp. 475–492. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19836-6_27

  28. Wang, Z., Simoncelli, E.P., Bovik, A.C.: Multiscale structural similarity for image quality assessment. In: The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003 2003 Nov 9, vol. 2, pp. 1398–1402. IEEE (2003)

    Google Scholar 

  29. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–12 (2004)

    Article  Google Scholar 

  30. Liu, C., Yuen, J., Torralba, A., Sivic, J., Freeman, W.T.: Sift flow: dense correspondence across different scenes. In: Computer Vision-ECCV 2008: 10th European Conference on Computer Vision, Marseille, France, October 12–18, 2008, Proceedings, Part III 10 2008, pp. 28–42. Springer, Heidelberg

    Google Scholar 

Download references

Acknowledgments

This work has been supported by the National Key Research and Development Program Grant 2020AAA0109702.

Author information

Authors and Affiliations

  1. Beijing Jiaotong University, Beijing, China

    Run-Xi Li & Lin-Lin Huang

  2. National Laboratory of Pattern Recognition, Institute of Automation of Chinese, Academy of Sciences, Beijing, China

    Fei Yin

Authors
  1. Run-Xi Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fei Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lin-Lin Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Run-Xi Li .

Editor information

Editors and Affiliations

  1. Kyushu Institute of Technology, Kitakyushu, Fukuoka, Japan

    Huimin Lu

  2. The University of Sydney, Sydney, NSW, Australia

    Michael Blumenstein

  3. Yonsei University, Seoul, Korea (Republic of)

    Sung-Bae Cho

  4. Chinese Academy of Sciences, Bejing, China

    Cheng-Lin Liu

  5. Osaka University, Osaka, Ibaraki, Japan

    Yasushi Yagi

  6. Kyushu Institute of Technology, Kitakyushu, Japan

    Tohru Kamiya

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Li, RX., Yin, F., Huang, LL. (2023). Dewarping Document Image in Complex Scene by Geometric Control Points. In: Lu, H., Blumenstein, M., Cho, SB., Liu, CL., Yagi, Y., Kamiya, T. (eds) Pattern Recognition. ACPR 2023. Lecture Notes in Computer Science, vol 14408. Springer, Cham. https://doi.org/10.1007/978-3-031-47665-5_22

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-47665-5_22

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-47664-8

  • Online ISBN: 978-3-031-47665-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation