Skip to main content
Springer Nature Link
Log in

High-precision page information extraction from 3D scanned booklets using physics-informed neural network

  • Regular Paper
  • Published:
Journal of Visualization Aims and scope Submit manuscript

Abstract

X-ray-based computerized tomography scans are used to analyze page information in closed booklets noninvasively. An important task is to extract the page information. Previously, the Laplace equation was used to calculate the page number field and extract the page information as an iso-surface. However, this technique cannot extract the page information properly. To solve this problem and improve the accuracy of the extracted page information, we propose a page information extraction method using a physics-informed neural network. The proposed method employs a structural similarity measure—often used in image processing research—to numerically evaluate the appropriateness of the page extraction. New history booklet is used to verify the effectiveness of this method in addition to the conventional booklet data.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  • Andersson J, Åkesson J, Diehl M (2012) Casadi: a symbolic package for automatic differentiation and optimal control. In: Recent advances in algorithmic differentiation. Springer, pp 297–307

  • Baum D, Lindow N, Hege H-C, Lepper V, Siopi T, Kutz F, Mahlow K, Mahnke H-E (2017) Revealing hidden text in rolled and folded papyri. Appl Phys A 123(3):1–7

    Article  Google Scholar 

  • Baydin AG, Pearlmutter BA, Radul AA, Siskind JM (2018) Automatic differentiation in machine learning: a survey. J Mach Learn Res 18:1–43

    MathSciNet  MATH  Google Scholar 

  • Bergmann U (2007) Archimedes brought to light. Phys World 20(11):39

    Article  Google Scholar 

  • Bergmann U, Manning PL, Wogelius RA (2012) Chemical mapping of paleontological and archeological artifacts with synchrotron x-rays. Annu Rev Anal Chem 5:361–389

    Article  Google Scholar 

  • Cheng C, Meng H, Li Y-Z, Zhang G-T (2021) Deep learning based on pinn for solving 2 dof vortex induced vibration of cylinder. Ocean Eng 240:109932

    Article  Google Scholar 

  • Cybenko G (1989) Approximation by superpositions of a sigmoidal function. Math Control Signals Syst 2(4):303–314

    Article  MathSciNet  MATH  Google Scholar 

  • Diringer D (2013) The book before printing: ancient, medieval and oriental. Courier Corporation, North Chelmsford

    Google Scholar 

  • Hascoet L, Pascual V (2013) The tapenade automatic differentiation tool: principles, model, and specification. ACM Trans Math Softw (TOMS) 39(3):1–43

    Article  MathSciNet  MATH  Google Scholar 

  • Heinecke A, Ho J, Hwang W-L (2020) Refinement and universal approximation via sparsely connected relu convolution nets. IEEE Signal Process Lett 27:1175–1179

    Article  Google Scholar 

  • Higo Y, Oka F, Sato T, Matsushima Y, Kimoto S (2013) Investigation of localized deformation in partially saturated sand under triaxial compression using microfocus x-ray ct with digital image correlation. Soils Found 53(2):181–198

    Article  Google Scholar 

  • Hore A, Ziou D (2010) Image quality metrics: Psnr vs. ssim. In: 2010 20th international conference on pattern recognition. IEEE, pp 2366–2369

  • Horng A, Brun E, Mittone A, Gasilov S, Weber L, Geith T, Adam-Neumair S, Auweter SD, Bravin A, Reiser MF et al. (2014) Cartilage and soft tissue imaging using x-rays: propagation-based phase-contrast computed tomography of the human knee in comparison with clinical imaging techniques and histology. Investig Radiol 49(9):627–634

    Article  Google Scholar 

  • Hu K, Wang Q, Koyamada K, Ohtani H, Goto T, Miyazawa J (2020) Visualization of the plasma shape in a force free helical reactor, ffhr. J Adv Simul Sci Eng 7(1):151–167

    Google Scholar 

  • Hu K, Koyamada K, Ohtani H, Goto T, Miyazawa J (2021) Visualization of plasma shape in the lhd-type helical fusion reactor, ffhr, by a deep learning technique. J Vis 1–14

  • Huang G-B, Chen L, Siew CK et al. (2006) Universal approximation using incremental constructive feedforward networks with random hidden nodes. IEEE Trans Neural Netw 17(4):879–892

    Article  Google Scholar 

  • Hubbe MA, Bowden C (2009) Handmade paper: a review of its history, craft, and science. BioResources 4(4):1736–1792

    Article  Google Scholar 

  • Jadhav S, Dmitriev K, Marino J, Barish M, Kaufman A (2020) 3d virtual pancreatography. IEEE Trans Vis Comput Graph

  • Jais IKM, Ismail AR, Nisa SQ (2019) Adam optimization algorithm for wide and deep neural network. Knowl Eng Data Sci 2(1):41–46

    Article  Google Scholar 

  • Knoche M (2005) The Herzogin anna Amalia library after the fire. IFLA J 31(1):90–92

    Article  Google Scholar 

  • Kolar J, Štolfa A, Strlič M, Pompe M, Pihlar B, Budnar M, Simčič J, Reissland B (2006) Historical iron gall ink containing documents-properties affecting their condition. Analytica chimica acta 555(1):167–174

    Article  Google Scholar 

  • Lippmann RP (1989) Pattern classification using neural networks. IEEE Commun Mag 27(11):47–50

    Article  Google Scholar 

  • Lu L, Meng X, Mao Z, Karniadakis GE (2021) Deepxde: a deep learning library for solving differential equations. SIAM Rev 63(1):208–228

    Article  MathSciNet  MATH  Google Scholar 

  • Markidis S (2021) The old and the new: Can physics-informed deep-learning replace traditional linear solvers? Front Big Data 92

  • Meng L, Dong D, Chen X, Fang M, Wang R, Li J, Liu Z, Tian J (2020) 2d and 3d ct radiomic features performance comparison in characterization of gastric cancer: a multi-center study. IEEE J Biomed Health Inform 25(3):755–763

    Article  Google Scholar 

  • Mocella V, Brun E, Ferrero C, Delattre D (2015) Revealing letters in rolled herculaneum papyri by x-ray phase-contrast imaging. Nat Commun 6(1):1–6

    Article  Google Scholar 

  • Moreira DA, Hage C, Luque EF, Willrett D, Rubin DL (2015) 3d markup of radiological images in epad, a web-based image annotation tool. In: 2015 IEEE 28th international symposium on computer-based medical systems. IEEE, pp 97–102

  • Nguyen DT, Hua B-S, Yu L-F, Yeung S-K (2017) A robust 3d–2d interactive tool for scene segmentation and annotation. IEEE Trans Vis Comput Graph 24(12):3005–3018

    Article  Google Scholar 

  • Nielsen MA (2015) Neural networks and deep learning, vol 25. Determination press, San Francisco

    Google Scholar 

  • Ou J, Han Z, Koyamada K (2021) Three-dimensional book data page segmentation and extraction method using Laplace equation. J Adv Simul Sci Eng 8(2):223–236

    Google Scholar 

  • Park J, Sandberg IW (1991) Universal approximation using radial-basis-function networks. Neural Comput 3(2):246–257

    Article  Google Scholar 

  • Paszke A, Gross S, Chintala S, Chanan G, Yang E, DeVito Z, Lin Z, Desmaison A, Antiga L, Lerer A (2017) Automatic differentiation in pytorch

  • Perekrestenko D, Grohs P, Elbrächter D, Bölcskei H (2018) The universal approximation power of finite-width deep relu networks. arXiv preprint arXiv:1806.01528

  • Raissi M, Perdikaris P, Karniadakis GE (2017a) Physics informed deep learning (part I): data-driven solutions of nonlinear partial differential equations. arXiv preprint arXiv:1711.10561

  • Raissi M, Perdikaris P, Karniadakis GE (2017b) Physics informed deep learning (part II): data-driven solutions of nonlinear partial differential equations. arXiv preprint arXiv:1711.10566

  • Raissi M, Perdikaris P, Karniadakis GE (2019) Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J Comput phys 378:686–707

    Article  MathSciNet  MATH  Google Scholar 

  • Rubin DL, Mongkolwat P, Kleper V, Supekar K, Channin DS (2009) Annotation and image markup: accessing and interoperating with the semantic content in medical imaging. IEEE Intell Syst 24(1):57–65

    Article  Google Scholar 

  • Samko O, Lai Y-K, Marshall D, Rosin PL (2014) Virtual unrolling and information recovery from scanned scrolled historical documents. Pattern Recognit 47(1):248–259

    Article  Google Scholar 

  • Sara U, Akter M, Uddin MS (2019) Image quality assessment through fsim, ssim, mse and psnr: a comparative study. J Comput Commun 7(3):8–18

    Article  Google Scholar 

  • Schabenberger O, Gotway CA (2017) Statistical methods for spatial data analysis. CRC press, Cambridge

    Book  MATH  Google Scholar 

  • Seales B, Delattre D (2013) Virtual unrolling of carbonized Herculaneum scrolls: research status (2007–2012). Macchiaroli editore, Napoli

    Google Scholar 

  • Seales WB, Parker CS, Segal M, Tov E, Shor P, Porath Y (2016) From damage to discovery via virtual unwrapping: reading the scroll from en-gedi. Sci Adv 2(9):e1601247

    Article  Google Scholar 

  • Stijnman A (2004) Historical iron-gall ink recipes: art technological source research for inkcor. Papierrestaurierung 5(3):14–17

    Google Scholar 

  • Stromer D, Christlein V, Huang Y, Zippert P, Helmecke E, Hausotte T, Maier A (2018a) Dose reduction for historical books digitization by 3-d x-ray ct. In: U of Applied Sciences Upper Austria (ed) Proceedings of 8th conference on industrial computed tomography (iCT 2018), pp 1–2

  • Stromer D, Christlein V, Martindale C, Zippert P, Haltenberger E, Hausotte T, Maier A (2018b) Browsing through, sealed historical manuscripts by using 3-d computed tomography with low-brilliance x-ray sources. Sci Rep 8(1):1–10

    Article  Google Scholar 

  • Su X, Xu H, Zhang Y, Kang Y, Gao G et al. (2019) An end-to-end preprocessor based on adversiarial learning for mongolian historical document ocr. In: Pacific Rim international conference on artificial intelligence. Springer, pp 266–272

  • Wang H, Liu Y, Wang S (2022) Dense velocity reconstruction from particle image velocimetry/particle tracking velocimetry using a physics-informed neural network. Phys Fluids 34(1):017116

    Article  Google Scholar 

  • Yarotsky D (2018) Universal approximations of invariant maps by neural networks. arXiv preprint arXiv:1804.10306

  • Yu B, Doraiswamy H, Chen X, Miraldi E, Arrieta-Ortiz ML, Hafemeister C, Madar A, Bonneau R, Silva CT (2014) Genotet: an interactive web-based visual exploration framework to support validation of gene regulatory networks. IEEE Trans Vis Comput Graph 20(12):1903–1912

    Article  Google Scholar 

  • Zhou D-X (2020) Universality of deep convolutional neural networks. Appl Comput Harmon Anal 48(2):787–794

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

The authors wish to thank A, B, and C. This work was partly supported by a Grant from XYZ (# 12345-67890).

Author information

Authors and Affiliations

  1. Kyoto University, Kyoto, Japan

    Zhongjiang Han, Jiarui Ou & Koji Koyamada

Authors
  1. Zhongjiang Han
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Jiarui Ou
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Koji Koyamada
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Zhongjiang Han.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Han, Z., Ou, J. & Koyamada, K. High-precision page information extraction from 3D scanned booklets using physics-informed neural network. J Vis 26, 335–349 (2023). https://doi.org/10.1007/s12650-022-00877-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12650-022-00877-0

Keywords