Abstract
Illustrations are an essential transmission instrument. For an historian, the first step in studying their evolution in a corpus of similar manuscripts is to identify which ones correspond to each other. This image collation task is daunting for manuscripts separated by many lost copies, spreading over centuries, which might have been completely re-organized and greatly modified to adapt to novel knowledge or belief and include hundreds of illustrations. Our contributions in this paper are threefold. First, we introduce the task of illustration collation and a large annotated public dataset to evaluate solutions, including 6 manuscripts of 2 different texts with more than 2 000 illustrations and 1 200 annotated correspondences. Second, we analyze state of the art similarity measures for this task and show that they succeed in simple cases but struggle for large manuscripts when the illustrations have undergone very significant changes and are discriminated only by fine details. Finally, we show clear evidence that significant performance boosts can be expected by exploiting cycle-consistent correspondences. Our code and data are available on http://imagine.enpc.fr/~shenx/ImageCollation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Bourdaillet, J., Ganascia, J.G.: Practical block sequence alignment with moves. In: LATA (2007)
Chen, X., Fan, H., Girshick, R., He, K.: Improved baselines with momentum contrastive learning. arXiv (2020)
Deng, J., Dong, W., Socher, R., Li, L. J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR (2009)
Dutta, A., Zisserman, A.: The VIA annotation software for images, audio and video. In: ACM Multimedia (2019)
En, S., Petitjean, C., Nicolas, S., Heutte, L.: A scalable pattern spotting system for historical documents. Pattern Recognit. 54, 149–161 (2016)
Ezra, D.S.B., Brown-DeVost, B., Dershowitz, N., Pechorin, A., Kiessling, B.: The dead sea scrolls. In: ICFHR, Transcription alignment for highly fragmentary historical manuscripts (2020)
Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)
Gordo, A., Almazan, J., Revaud, J., Larlus, D.: End-to-end learning of deep visual representations for image retrieval. IJCV 124(2), 237–254 (2017)
Haentjens Dekker, R., Van Hulle, D., Middell, G., Neyt, V., Van Zundert, J.: Computer-supported collation of modern manuscripts: collatex and the Beckett Digital Manuscript Project. DSH 30(3), 452–470 (2015)
Hassner, T., Wolf, L., Dershowitz, N.: OCR-free transcript alignment. In: ICDAR (2013)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
Hobby, J.D.: Matching document images with ground truth. IJDAR 1(1), 52–61 (1998)
Kornfield, E.M., Manmatha, R. and Allan, J.: Text alignment with handwritten documents. In: DIAL (2004)
Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: ICCV (2017)
Lin, T.Y., et al.: Microsoft COCO: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48
Lowe, D.G.: Distinctive image features from scale-invariant keypoints. IJCV 60(2), 91–110 (2004)
Monnier, T., Aubry, M.: docExtractor: an off-the-shelf historical document element extraction. In: ICFHR (2020)
Radenović, F., Tolias, G., Chum, O.: Fine-tuning CNN image retrieval with no human annotation. TPAMI 41(7), 1655–1668 (2018)
Razavian, A.S., Sullivan, J., Carlsson, S., Maki, A.: Visual instance retrieval with deep convolutional networks. MTA 4(3), 251–258 (2016)
Revaud, J., Almazán, J., Rezende, R.S., Souza, C.R.D.: Learning with average precision: training image retrieval with a listwise loss. In: ICCV (2019)
Sadeh, G., Wolf, L., Hassner, T., Dershowitz, N., Ben-Ezra, D.S.: Viral transcript alignment. In: ICDAR (2015)
Schmidt, D., Colomb, R.: A data structure for representing multi-version texts online. Int. J. Hum.-Comput. Stud. 67(6), 497–514 (2009)
Shen, X., Efros, A.A., Aubry, M.: Discovering visual patterns in art collections with spatially-consistent feature learning. In: CVPR (2019)
Shen, X., et al.: Large-scale historical watermark recognition: dataset and a new consistency-based approach. In: ICPR (2020)
Sivic, J., Zisserman, A.: Video google: a text retrieval approach to object matching in videos. In: ICCV (2003)
Smith, S.E.: The eternal verities verified: Charlton Hinman and the roots of mechanical collation. Stud. Bibliogr. 53, 129–161 (2000)
Úbeda, I., Saavedra, J.M., Stéphane, N., Caroline, P., Heutte, L.: Pattern spotting in historical documents using convolutional models. In: HIP (2019)
Acknowledgements
This work was supported by ANR project EnHerit ANR-17-CE23-0008, project Rapid Tabasco, and gifts from Adobe. We thank Alexandre Guilbaud for fruitful discussions.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Kaoua, R., Shen, X., Durr, A., Lazaris, S., Picard, D., Aubry, M. (2021). Image Collation: Matching Illustrations in Manuscripts. In: Lladós, J., Lopresti, D., Uchida, S. (eds) Document Analysis and Recognition – ICDAR 2021. ICDAR 2021. Lecture Notes in Computer Science(), vol 12824. Springer, Cham. https://doi.org/10.1007/978-3-030-86337-1_24
Download citation
DOI: https://doi.org/10.1007/978-3-030-86337-1_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86336-4
Online ISBN: 978-3-030-86337-1
eBook Packages: Computer ScienceComputer Science (R0)
