MOSAIC+: Fragment retrieval and reconstruction enhancement for virtual restoration

Abstract

When a piece of art from the past is found, it is often broken into several fragments. This is a very common case with frescoes and pottery. Reconstruction from these fragments requires human expertize and it is almost always very hard, if not impossible, to be completely automated. Actually, the problem is an example of jigsaw puzzle solving, which is known to be NP-complete from a computational point of view. The possible high number of fragments and their possible fragility make the task formidable. This work describes software tools that help in two ways. First, reconstruction is assisted by a content-based database of the available pieces. Once acquired by suitable photo equipment and suitably annotated, the fragments can be left untouched and manipulated virtually to find out the best combination before proceeding with the actual reconstruction, if called for. Second, the unavoidable gaps and cracks in the reconstruction are filled by an inpainting module. The results have been assessed by running the system on both artificial datasets and actual case studies.

Introduction

It happens, either fortunately during archeological excavation campaigns or unfortunately after destructive events, e.g., earthquakes or bombing, to retrieve ruins originally covered by frescos, or fragmented pottery. In the fortunate situation, the reverse of the medal is that the scene depicted is not known and in many cases can be hardly imagined. In the unfortunate situation, since the breaking happened in recent time, it is very often the case that pictures may exist. However, if in this latter case the situation may not that better if fragmentation is dense. In both cases the automatic re-combination of fragments is very hard and requires manual ability, domain expertize, and a lot of patience and careful, slow work. The inherent difficulty of the task can be even further increased by the possibly extreme fragility of the materials and therefore by the greater required caution. On the one hand, the expert often faces a collection of fragments that crumble to dust if not handled with the utmost gentleness. On the other hand, unfortunately, reconstructing the original design can truly require a lot of repeated manipulation. In general, correctly putting the pieces side by side may require an infinite number of repeated rotations, tentative alignments, and more operations on single pieces as well as batched of already dovetailed ones. Even worse, the smaller the fragments (and then the more fragile), the more manipulation may be required. In this scenario, the most time consuming but also potentially dangerous action is to repeatedly explore the pool of available fragments to locate possible candidate pieces to join. Each time a piece is touched or moved it might break, or at least its edges might be further damaged. Even when one arrives at the end of the reconstruction process, it is the case that the obtained surface appears still cracked by missing pieces as well as by patchy edges. For both the expert and the final visitor of the artwork, the artistic experience may be significantly improved by attenuating the visual effect of such irregularities.

This paper presents the extension of a set of tools formerly proposed as Multi-Object Segmentation for Assisted Image reConstruction (MOSAIC) in [1]. MOSAIC+ also includes procedures to virtually attenuate the visual effect of craquelure. All tools have been designed and evaluated together with field experts, to support the work of archeologists and cultural heritage operators, when reconstructing fragmented (plain) artifacts. We want to preliminarily underline that, at the moment, we have not implement yet a procedure for the complete virtual reconstruction for 3D artifacts, e.g., pottery, but even in this case the system supports the expert in retrieving the most suitable fragments to join. No information about the original appearance of the whole artwork is assumed to be available. The system provides the operator with a complete workflow from photo-acquisition onwards. During the process, the fragments are photographed, and their captured images are suitably processed and stored in a repository. In the repository, images are suitably indexed according to features such as color distribution, shape and texture, so that they can be later retrieved through query-by-example. During query phase, any fragment image can be used as the key. If more results are returned to a quesry, as it is almost always the case, they are displayed to the user from the most to the least similar to the key. The operator can pick returned fragment images, rotate and translate them, and try to dovetail them to reconstruct the original picture, as when solving a puzzle. In most cases, holes will be present and the result will appear as highly fragmented, even in the virtual reconstruction. For this reason, once the reconstruction is completed, a technique to attenuate craquelure can be applied. The usefulness of this further tool is twofold. On the one end, it further supports both the operator and the final visitor by providing a better reconstruction of the original appearance of the artwork. On the other hand, it can be is also very useful as a preprocessing operation during population, before extracting shape and color information, when the fragments may present inner craquelure. We applied our techniques on a number of simulations, and on the real use case of the reconstruction of a fresco from fragments found in the St. Trophimena church in Salerno (Italy).