A graph-based optimization algorithm for fragmented image reassembly

doi:10.1016/j.gmod.2014.03.001

Graphical Models

Volume 76, Issue 5, September 2014, Pages 484-495

https://doi.org/10.1016/j.gmod.2014.03.001 Get rights and content

Abstract

We propose a graph-based optimization framework for automatic 2D image fragment reassembly. First, we compute the potential matching between each pair of the image fragments based on their geometry and color. After that, a novel multi-piece matching algorithm is proposed to reassemble the overall image fragments. Finally, the reassembly result is refined by applying the graph optimization algorithm. We perform experiments to evaluate our algorithm on multiple torn real-world images, and demonstrate the robustness of this new assembly framework outperforms the existing algorithms in both reassembly accuracy (in handling accumulated pairwise matching error) and robustness (in handling small image fragments).

Graphical abstract

Introduction

Fragmented image reassembly recomposes a group of picture fragments into the original complete image. It is a geometric processing problem that has important applications in many fields such as archaeology and forensics. For example, archaeologists need to spend a lot of efforts to recompose fractured ancient paintings to restore their original appearances; forensic specialists manually compose damaged documents or pictures to recover the original evidence that are broken by humans. Developing robust geometric algorithms for automatic image reassembly can reduce the expensive human labors and improve the restoration efficiency, and hence is highly desirable.

Existing automatic reassembly algorithms can be generally divided into two categories: color-based approaches and geometry-based approaches. Color-based methods mainly use the color information to predict the adjacency relationship of the fragments and guide the matching [1], [2], [3]. These algorithms are usually efficient, but they sometimes suffer from composition accuracy and may fail when the textures of different fragments are similar. Geometry-based methods reassemble the fragments by matching their boundary curves [4], [5], [6], [7]. They can more accurately align adjacent fragments along their breaking regions, but they are sometimes slow and can easily get trapped in local optima then fail to reach the correct result. Therefore, incorporating both the geometry and color information in the reassembly computation could make process more efficient, accurate, and reliable. This paper presents a novel 3-step composition algorithm, as illustrated in Fig. 1.

The first step is called the pairwise matching, which aims to identify adjacent pairs and compute their initial alignments. Geometry-based pairwise matching methods rely on analyzing the shape of the boundary curve contours; color-based pairwise matching methods match fragments using their color information. Our integrated algorithm (1) extracts the border of each fragment and represents it as a curve contour, (2) clusters such a curve contour into multiple segments based on both color and geometry, and (3) matches contour segments to suggest the potential alignment between adjacent fragments. This pairwise matching algorithm will result in a set of possible matching between identified pairs of fragments, some of which are correct while some are not.

The pairwise matching produces redundant matches, which we intentionally generate in order to tolerate erroneous adjacency identification due to noise and local minima. A global groupwise matching can better filter out these false matches. Extensive examination of this is a combinatorial optimization problem and is NP-hard. Many previous works use a best-first search strategy (i.e., to explore a graph by expanding the most promising node) with backtracking to solve this problem. In our work, the global composition is formulated as a novel graph-based searching problem, solving which will give us a more reliable global reassembly of multiple fragments.

Groupwise matching can compose the fragments globally. However, the result is dependent on the composition sequence, and the alignment errors will be accumulated which may even cause a failure in the reassembly (i.e., fragment intersection, see Section 6 for details). We derive a graph optimization algorithm to reduce the global matching errors between adjacent fragments.

The main contributions of this work are as follows:

1.
We integrate both geometry and color information in pairwise matching computation to obtain more reliable alignment between adjacent fragments.
2.
We propose a graph-based algorithm that performs groupwise matching to better handle the errors resulted from pairwise alignments and obtain correct adjacency information of all the fragments.
3.
We develop a variational graph optimization in the end to reduce the accumulated errors to refine the reassembly and achieve a global optimal result.

Section snippets

Local pairwise matching

The essence of fragment reassembly is to find the relative transformations between adjacent fragments. In the procedure of 2D images composition, fragments can be modeled and aligned using their boundaries, which are 2D curve contours. Therefore, pairwise matching often reduces to a partial curve matching problem, which is mainly solved in existing literatures via either geometry-based or color-based approaches.

Wolfson [8] approximates the curves as the polygonal shapes and solves their

Problem formulation and experimental settings

Given a set of fragments $F = {F_{i}}, i = 1, \dots, n, F_{i} \subset R^{2}$ , which are from a torn image I, we want to solve the transformation for each fragment $F_{i}$ , i.e., a $3 \times 3$ matrix $T_{i}$ , so that these transformations compose all the fragments back into the original image $I^{'} = ⋃ T_{i} (F_{i})$ .

In our experiments, each image is torn into multiple fragmented pieces. We scan all the fragments. For each fragment $F_{i}$ we obtain a digital image scan $I_{i}$ which has a colorful region appeared on a white background. We can segment such a region $F_{i}$

Pairwise matching between fragments

This section elaborates our algorithm of computing the pairwise matching between two image fragments. First, the boundary of each fragment $F_{i}$ is extracted from the scan image $I_{i}$ ; the boundary is a 2D curve loop. Then, the pairwise matching between two image fragments can reduce to a partial curve matching problem. Our pairwise matching algorithm utilizes both the color and geometry information. Unlike existing algorithms that directly use dynamic programming [5] or geometric hashing [8] for

Global image reassembly

The pairwise matching step provides us a set of possible matchings between each pair of the image fragments; each matching is associated with a matching score. This score is usually a good indicator to whether this alignment is correct. A straightforward strategy is to iteratively pick the matching following the scores from high to low. However, this may not always work as the highest score is not a guarantee to the correct match, especially when there are many small fragments: small fragment’s

Refinement of overall reassembly

In the previous section, we obtain a global reassembly of all the fragmented images. But small error may be inevitable in the transformations computed in the previous two steps, caused during e.g. polygon approximation, the inconsistency of the pixelization in image scan. In Step 2, we start with one fragment and iteratively glue other fragments. Transformation errors will be accumulated and sometimes this could even cause a failure in the reassembly (Fig. 7(a)). The error becomes especially

Experimental results

We perform multiple experiments to evaluate our reassembly algorithm. In our experiments, we print out randomly selected digital images on papers. Most images, after printed, are with the size of about 20 cm × 25 cm. Then we randomly tear an image into multiple pieces. The size of the fragments is ranging from 4 cm × 4 cm–7 cm × 7 cm. Then each image fragment is scanned (in our experiments, with a scanner of 150-dpi resolution). Our reassembly algorithm is then used to recompose the scanned digital image

Conclusion and future work

We present a novel computational pipeline for the automatic reassembly of fragmented images. It consists of three main steps: pairwise matching between two image fragments, graph-based global fragment reassembly, and refinement of the reassembly via graph optimization. In Step 1, we present a better curve-matching algorithm: each pair of image fragments are aligned via a pairwise matching integrating both geometry and color; In step 2, we present a reliable graph-based global search algorithm,

Acknowledgments

This research is partially supported by National Science Foundation – United States (IIS-1320959), IBM Faculty Award, and National Natural Science Foundation of China (No. 61170323).

References (25)

E. Justino et al.
Reconstructing shredded documents through feature matching
Forensic Sci. Int.
(2006)
G. Ucoluk et al.
Automatic reconstruction of broken 3-d surface objects
Comput. Graph.
(1999)
G. Papaioannou et al.
On the automatic assemblage of arbitrary broken solid artefacts
Image Vis. Comput.
(2003)
W. Yu et al.
Fragmented skull modeling using heat kernels
Graph. Models
(2012)
X. Li et al.
Symmetry and template guided completion of damaged skulls
Comput. Graph.
(2011)
F. Amigoni, S. Gazzani, S. Podico, A method for reassembling fragments in image reconstruction, in: Proceedings of...
M. Sagiroglu, A. Ercil, A texture based matching approach for automated assembly of puzzles, in: 18th International...
E. Tsamoura et al.
Automatic color based reassembly of fragmented images and paintings
IEEE Trans. Image Process.
(2010)
W. Kong, B. Kimia, On solving 2d and 3d puzzles using curve matching, in: Proceedings of the IEEE Computer Society...
H. da Gama Leitao et al.
A multiscale method for the reassembly of two-dimensional fragmented objects
IEEE Trans. PAMI
(2002)

L. Zhu et al.

Globally consistent reconstruction of ripped-up documents

IEEE Trans. Pattern Anal. Mach. Intell.

(2008)

H. Wolfson

On curve matching

IEEE Trans. Pattern Anal. Mach. Intell.

(1990)

Cited by (53)

Comprehensive survey of the solving puzzle problems
2023, Computer Science Review
Solving puzzle problems using computer-aided methods is becoming more common with applications in forensic science, restoration, banking system, and multimedia. However, only a few surveys have been published on this topic, the most recent being more than a decade old. The scope of 2D puzzle problems is extensive, and the number of computer-aided methods has increased in recent years. In this paper, we have presented a comprehensive survey to pave a roadmap for researchers dealing with puzzle problems. This study classifies 2D puzzle problems in a novel way, considering many examples such as dissection, combinatorial and double-sided puzzles and reclassifies computer-aided methods to cover the studies carried out in recent years. Various strategies (pre-grouping and global consistency approach) have been investigated to solve the puzzle problem effectively. The computer-aided methods have been examined deeply, including many recent methods related to squared jigsaw puzzles, torn photographs, banknotes, and fragmented documents, and they are compared to each other. In addition, new topics such as combining mosaic pieces and Islamic architectural motif puzzle problems have been proposed to the interest of researchers. In conclusion, our study shows many research opportunities that are not yet solved by any computer-aided method.
Solving archaeological puzzles
2021, Pattern Recognition
Citation Excerpt :
Handling 2D general-shaped pieces and gaps. Though not in archaeology, the most closely related works to ours are [31,49], which assume general-shaped pieces of natural images. However, in [31] there are no gaps between the pieces.
This paper focuses on the re-assembly of an archaeological artifact, given images of its fragments. This problem can be considered as a special challenging case of puzzle solving. The restricted case of re-assembly of a natural image from square pieces has been investigated extensively and was shown to be a difficult problem in its own right. Likewise, the case of matching “clean” 2D polygons/splines based solely on their geometric properties has been studied. But what if these ideal conditions do not hold? This is the problem addressed in the paper. Three unique characteristics of archaeological fragments make puzzle solving extremely difficult: (1) The fragments are of general shape; (2) They are abraded, especially at the boundaries (where the strongest cues for matching should exist); and (3) The domain of valid transformations between the pieces is continuous. The key contribution of this paper is a fully-automatic and general algorithm that addresses puzzle solving in this intriguing domain. We show that our approach manages to correctly reassemble dozens of broken artifacts and frescoes.
A Fragment Fracture Surface Segmentation Method Based on Learning of Local Geometric Features on Margins Used for Automatic Utensil Reassembly
2021, CAD Computer Aided Design
To achieve the automatic reassembly (piecing) of utensil fragments, a fracture surface extraction method based on the learning of local geometric features (core focus) and a utensil reassembly method (secondary focus) are presented in this paper. The steps of the methodological framework are as follows. First, based on obtained 3D models of utensil fragments, a triangle cell descriptor is proposed to describe the geometric features of spatial neighborhoods. Second, a set of feature mapping images (FMIs) is established as a training dataset. Third, after labeling of the ground-truth data, a convolutional neural network (CNN) is trained using the FMIs. Fourth, based on processing to eliminate mislabeled triangle cells, skeletons of the fracture surface margins can be generated. Fifth, a shortcut-based strategy is proposed to eliminate residual triangle cells to extract the fracture surfaces. Sixth, a control-point- and vector-based strategy is proposed to achieve the matching and prealignment of the fracture surfaces. Finally, a cyclic error iteration strategy is designed to assemble the fragments into a holonomic utensil. This learning-based framework is more effective at extracting the key geometric data (fracture surfaces) of utensil fragments than several classical methods. It may also enable a new strategy for 3D graph processing.
DAFNE: A dataset of fresco fragments for digital anastlylosis
2020, Pattern Recognition Letters
Citation Excerpt :
More precisely, matching curves and contour-shape based methods were used to recompose 2D and 3D puzzles [6], pottery [7], wall paintings [8] and archaeological fragments [9]. Considering fragmented image reconstruction, the available solutions include pure color-based approaches [10], the combination of color and geometric features [11,12] and, more recently, also deep learning [13]. All these methods, however, deal with a situation similar to a puzzle, where all (or quite all) the pieces are available, and there is an almost perfect match between fragment edges.
Restoring artworks seriously damaged or completely destroyed is a challenging task. In particular, the reconstruction of frescoes has to deal with problems such as very small fragments, irregular shapes and missing pieces. Several attempts have been done to develop new techniques for helping restorers in the matching process, starting from traditional image processing methods to the more recent deep learning approaches. However, as often happens in the Cultural Heritage field, the availability of labeled data to test new strategies is limited, and publicly available datasets contain only few samples. For this reason, in this paper we introduce DAFNE, a large dataset that includes hundreds of thousands of images of fresco fragments artificially generated to guarantee a high variability in terms of shapes and dimensions. Fragments have been obtained starting from 62 images of famous frescoes of various artists and historical periods, in order to consider different artistic styles, subjects and colors.
Reunion helper: an edge matcher for sibling fragment identification of the Dunhuang manuscript
2024, Heritage Science
Pictorial and Apictorial Polygonal Jigsaw Puzzles from Arbitrary Number of Crossing Cuts
2024, International Journal of Computer Vision

View all citing articles on Scopus

View full text

A graph-based optimization algorithm for fragmented image reassembly

Abstract

Graphical abstract

Introduction

Section snippets

Local pairwise matching

Problem formulation and experimental settings

Pairwise matching between fragments

Global image reassembly

Refinement of overall reassembly

Experimental results

Conclusion and future work

Acknowledgments

Forensic Sci. Int.

Comput. Graph.

Image Vis. Comput.

Graph. Models

Comput. Graph.

Automatic color based reassembly of fragmented images and paintings

IEEE Trans. Image Process.

A multiscale method for the reassembly of two-dimensional fragmented objects

IEEE Trans. PAMI

Globally consistent reconstruction of ripped-up documents

IEEE Trans. Pattern Anal. Mach. Intell.

On curve matching

IEEE Trans. Pattern Anal. Mach. Intell.