Keywords

1 Introduction

Texture Synthesis from Samples (TSFS) is proposed after texture mapping and procedural texture synthesis. It can not only overcome the shortcomings of traditional texture mapping, such as the noticeable seams between texture patches, but also avoid rectifying parameters repeatedly in procedural texture synthesis. It has the widespread application in the areas of the image editing, the incomplete image filling, the data compression, the network-data’s speedy transmission, the large-scale scene’s generation as well as the realistic and non-photorealistic rendering and so on, so it has become one of the research focus in the present computer graphics, computer vision and the image processing areas.

TSFS technology always seeks to improve the synthesis speed and synthesis effect. In 1999, Efros et al. proposed the classic point matching algorithm [1], which used Markov Random Field (MRF) model to carry on the texture synthesis. For most of the texture sample, MRF model could well describe the texture features, but its calculation amount was very large and the algorithm speed was very slow.

In 2001, Efros [2] and Liang [3] independently proposed a synthesis method based on patch, becoming the important milestone of TSFS. The algorithm sought best texture blocks from texture synthesis sample image and stitched them based on the best sew line to generate a new texture. This algorithm greatly increased the synthesis speed, but sometimes the texture block appeared repetitive effect. Wang Yiping [4] thoroughly analyzed the effect of the synthesis parameters such as the block’s shape, size and overlap region between adjacent blocks et al., proposed an adaptive optimized synthesis parameter method that improving the effect of patch-based texture synthesis. Chen Xin [5] gradually used the partial texture of synthesized texture to generate a larger texture block and raise the synthesis efficiency. Another kind of patch -based synthesis algorithm based on “Wang Tile” theory, was proposed by Cohen et al. [6]. It used a group of synthesized Wang Tiles to carry out a series of simple stitching to generate the periodic texture. This algorithm’s synthesis speed and synthesis effect were significantly improved. Zou Kun, et al. [7, 15] proposed the fast texture synthesis algorithm based on the Graph Cut and discrete optimization that had very quick synthesis speed. In 2005, Lefebvre et al. proposed the parallel controllable texture synthesis method [8], using GPU for parallel Calculation. In 2008 a multi-scale texture synthesis method [9] was proposed, which took the reference [8] ’s work as a foundation, and could realize the multi-scale texture synthesis. Lasram [10] presented a parallel patch-based texture synthesis technique that use two algorithms: one quickly find a good cut around a patch, and another deformation algorithm to further align features crossing the patch boundary. Caron [11] used Partition of unity parametrics (PUPs) for procedural texture synthesis, the problem of texture synthesis can be approached from the perspective of point placement and attribute assignment. Aguerrebere [12] proposed an acceleration method based on PCA, yielding results that are generally visually indistinguishable from the original results. Wei et al. [13, 14] introduced a fast texture synthesis using tree-structured vector quantization and a coherence concept that pixels that are close in the input image are more likely to be close in the output image. In order to solve the problem of structure mismatching, Yan [16] proposed a desired distance metric to measure structural similarity. This method could significantly alleviate structure discontinuities in synthesized textures. Kim [17] proposed a symmetry-guided texture synthesis algorithm to the long-standing problem of how to optimize, transfer, and control the spatial patterns in textures.

Based on Efros’s Image Quilting algorithm [2], this paper proposes an improved texture synthesis algorithm to improve the synthesis speed. It firstly synthesizes the row of texture using the patch texture, and then synthesizes the whole texture using the line texture. This new algorithm mainly makes improvements in the following three areas:

  • Replacing the L-overlap region of Image Quilting algorithm with r-overlap region. It can reduce the number of pixels which are involved in computing the SSD error of overlap region, and can eliminate the redundancy computation of the original algorithm.

  • Using parallel calculation to select the best matching texture patch to improve the synthesis speed.

  • Enlarging the searching range of the horizontal suture from the patch to the line of patches to guarantee and improve the texture synthesis effect.

Through the comparative experiment with Image Quilting Algorithm, it can be shown that the synthesis speed of the improved algorithm is increased substantially, and the synthesis results don’t decline. What is more, it can reduce the complexity of the algorithm.

2 Synthesis Algorithm Based on Line

2.1 Image Quilting Algorithm

Image quilting algorithm [2] is a patch-based texture synthesis algorithms, which was proposed by Efros in SIGGRAPH 2001. The main steps of the algorithm are as follows:

(1) Take an arbitrary texture block B1 in the input sample texture map, place it in the goal texture map, and then search a new block B2 in the input sample texture map, the search method as follows:

a. According to the order of the scan-line, and according to size of the synthesis texture block in the sample texture map, traverse the sample texture map, and then get the set of candidate texture blocks.

b. For each candidate texture block, according to the width of overlap to find out its overlap region, which has been synthesized block in the target map, the overlap region with synthesized block is recorded as \({\text{B}_{1}^{\text{ov}}}\), the overlap region with current found texture block is recorded as \({\text{B}_{2}^{\text{ov}}}\), calculate the error of overlap, and use the color’s Sum-of-Squared Differences (SSD), which belong to corresponding pixels in the overlap region, to measure the error, the calculation as follow:

$$ Error = \sum\limits_{i,j \in overlap} {[(R_{{1_{ij} }}^{ov} - R_{{2_{ij} }}^{ov} )^{2} + (G_{{1_{ij} }}^{ov} - G_{{2_{ij} }}^{ov} )^{2} + (B_{{1_{ij} }}^{ov} - B_{{2_{ij} }}^{ov} )^{2} ]} $$
(1)

In the above formula (1), \( R_{{1_{ij} }}^{ov} \), \( G_{{1_{ij} }}^{ov} \) and \( B_{{1_{ij} }}^{ov} \) are the RGB color values of the corresponding pixel in the B ov1 , while \( R_{{2_{ij} }}^{ov} \),\( G_{{2_{ij} }}^{ov} \) and \( B_{{2_{ij} }}^{ov} \) is the RGB color value of the corresponding pixel in the B ov2 .

The overlap region has three different kinds in the Image quilting algorithm, the vertical, the horizontal and the L-overlap region, which are shown by the blue rectangles in the Fig. 1, the left being a synthesized patch, the right patch with grey color being the new patch.

Fig. 1.
figure 1

The overlap region in the Image Quilting Algorithm. (a) vertical overlap. (b) horizontal overlap. (c) L-overlap.

Full size image

c. Look for the set of the texture blocks, whose overlapping error within the error tolerance, then randomly select a texture block as a new found optimal texture block B2 in this set.

(2) Place B2 in the goal texture map and splice B2 with the synthesized texture block, so as to find out its best synthetic suture with the synthesized texture block. There are three types of stitching line in the Image quilting algorithm, which are shown in Fig. 2. The best suture is gotten by using Dijkstra’s shortest path algorithm.

Fig. 2.
figure 2

Three types of stitching of the Image Quilting Algorithm (a) the best vertical stitching. (b) the best horizontal stitching. (c) the best L-overlap stitching.

Full size image

(3) Holding the found best suture as a new texture border, we can put B2 in the target texture map, and then splice them by the best suture.

(4) Repeat the above steps until the goal texture map is covered by the synthesis texture.

From above figures, we can find that the selection of the best matching texture patch is a time-consuming process; the calculation amount of SSD error is very large. The dark gray part in Fig. 1(c) is used several times to calculate the SSD error. Therefore, making efforts to reduce the calculation amount and redundancy calculation of SSD error is an effective way to improve the speed of the texture synthesis.

2.2 Improved Algorithm Principle

In the L-overlap region of the Image Quilting algorithm, there is a dark gray area shown in Fig. 1(c). Different SSD error calculations are involved three times which are in the top, the left and the current texture patch synthesis processing. This causes redundant calculation. Moreover, the processing of looking for the best sew line also has the redundant calculation to some extent. So we propose the synthesis algorithm based on line to reduce the redundant calculation. The line-based synthesis algorithm also has three types overlap region: the vertical, the horizontal and the r-overlap. The vertical and horizontal overlaps are the same with those in the Image Quilting algorithm. The r-overlap region is shown with blue rectangles in Fig. 3.

Fig. 3.
figure 3

The r-overlap region in the line-based synthesis algorithm.

Full size image

We can see from Fig. 3 that, the vertical and horizontal overlap no longer coincide in the r-overlap region, and the upper-left corner part of the pixels are no longer involved in SSD error calculation in current synthesis patch. It eliminates the redundancy calculation, as a result, to raise the synthesis speed to a certain extent.

The basic idea of the line-based texture synthesis is as follow:

  • Using the Image quilting algorithm to synthesize the first row texture of the target texture image.

  • Starting from the second row texture of the target texture image, except the first patch of each row texture synthesized by using the Image Quilting; the rest patches are synthesized based on r-overlap region showed in Fig. 3 to calculate the SSD error.

  • Finding the best horizontal sew line in the whole row texture, and then according to this sew line to stitch the two rows texture. There are two stitching types as shown in Fig. 4.

Fig. 4.
figure 4

Two types of stitching in the line-based synthesis (a) the best vertical stitching in the line-based synthesis. (b) the best horizontal texture stitching in the line-based synthesis.

Full size image

Repeating the previous two steps until the target texture image is totally covered by the synthesis texture.

2.3 The Upgrade Rate of Synthesis Speed Analysis

Next we will analyze the upgrade rate of synthesis speed of the line-based synthesis algorithm; this upgrade rate is relative to the Image Quilting algorithm.

Both algorithms have the vertical and horizontal overlap, the vertical overlap appears at the first row of the target texture image, and the horizontal overlap appears at the first column, while the other location are L-overlap or r-overlap. When analyze the synthetic speed upgrade rate, so we just consider the calculation amount of the L-overlap and r-overlap SSD error.

We use the number of the pixels that are used to calculate the SSD error in the L-overlap and r-overlap to approximately measure the upgrade rate of synthesis speed. We use the following formula to calculate.

$$ ratio = \frac{LS\_pixels - rS\_pixels}{LS\_pixels} $$
(2)

In the above formula, the LS_pixels is the number of pixels in the L-overlap, while the rS_pixels is the number of pixels in the r-overlap, and the ratio represents the synthesis speed upgrade rate.

Substituting the synthesis parameters into Eq. (1), such as the width of the synthesis patch, we can get the following formula:

$$ ratio = \frac{oW \times oW}{(bW + bH) \times oW - oW \times oW} $$
(3)

In the above formula, bW and bH are the width and height of the synthesis texture patch, while oW is the width of the overlap. Supposing the sample texture image, the synthesis texture patch and the target texture image are square, and the proportion between the oW and bW the is 1:n, then we can get:

$$ ratio = \frac{1}{{2{\text{n - 1}}}} $$
(4)

As can be seen from the above formula, the ratio n has a close relation with the proportion n. When the proportion n increases, the ratio n will be correspondingly increase. In the case of n = 3, the synthesis speed upgrade rate is 20 %, while in case of n = 6, the synthesis speed upgrade rate is 9.09 %.

2.4 The Search for the Best Horizontal Sew Line

Different from the Image Quilting algorithms, the line-based synthesis algorithms firstly synthesize a row texture, then stitch it with the previous row texture. The search range of the best horizontal sew line is enlarged from a synthesis patch to a row of patches. It will improve the quality of texture synthesis, and make the sew line have the global optimization in a line texture range.

3 Parallel Traversal of the Candidate Synthesis Texture Patch

According to the principles of the line-based synthesis algorithm, we need to traverse the texture sample image to build a set of candidate synthesis texture patches and find the best matching texture patch based on their SSD error. The SSD error calculation of each candidate synthesis patch is independent each other, so we can calculate at the same time, that is, at the same time we can carry on the error calculation for two, four or even multiple synthesis texture patches. This method can shorten the time to find the best matching texture patch. In this paper, the traversal procedure is separated into two parallel processes; the dual-core CPU is used for parallel calculation to improve the speed of the synthesis.

4 Experimental Result Contrast and Analysis

We use Visual C ++ 2005 to realize the Image Quilting algorithm and the line-based synthesis algorithm that proposed in this paper. We have carried on a large number of synthesis experiments, and have contrasted the result images to show the effectiveness of line-based synthesis algorithm and the parallel synthesis algorithm. The notebook computer used in these experiments is the IBM R400. Its configurations are as follows:

CPU: Intel(R) Core(TM) 2 Duo P8700

CPU frequency: 2.53 GHz; Memory: 2 GB

We downloaded many texture sample images from the Efros’s homepage [18], the Image Quilting results of them are shown in this homepage that can be used to compare easily with the results of our algorithm.

4.1 The Experiments of the Synthesis Speed

In order to verify the effectiveness that how much the line-based synthesis algorithm can raise the synthesis speed, we separately use the Image Quilting algorithm and line-based synthesis algorithm to texture synthesis for the ten texture sample images downloaded from Efros’s homepage. This experiment mainly inspect the influence of parameter n to the synthesis speed, while n is the proportion that the overlap occupy in the synthetic texture patch. We suppose the target sample image is square, so the number of the synthesis texture patch on each row and each column is the same, while the other synthesis parameters should be decided by the sample texture image. For example, the size of the synthesis texture patch should accord to the shape and size of the texture primitives in the sample texture image. To avoid the reaction time caused by other computer processes in the single synthesis, each texture sample carries on synthesis experiment for 50 times, and the final synthesis time is the average time of this 50 times. We select two values of n which are commonly used in the synthesis. These two common values are 3 and 6. Suppose the number of the synthesis texture patches in each row and each column, is bN, and the width of the target texture image is tw, so the size of the target texture image can be gotten from Eq. (5).

$$ tW = (bW - oW) \times (bN - 1) + bW $$
(5)

The experimental results are listed in Table 1. As can be seen from Table 1, when n = 3, the synthesis speed average upgrade rate of the ten samples is 23.38 %, this is very close to 20 % calculated by Eq. (3), but more than 20 %. This is because Eq. (3) just considers the overlap area error’s calculation. In fact, the line-based synthesis algorithm also reduce the calculation redundant in the process of searching the horizontal sew line. So in this experiment, the upgrade rate is higher than result calculated by Eq. (3). When n = 6, the average upgrade rate is 11.1 %, this is also higher than the rate that calculated by Eq. (3), but because of reducing of the overlap region, this upgrade rate is lower than that of n = 3.

Table 1. The comparison of the synthesis time between the Image Quilting algorithm (IQ) and the line-based synthesis algorithm
Full size table

4.2 Experiments of Synthesis Effect

In this experiment, we compare the effect of the two synthesis algorithms. The synthesis parameters of two methods are the same with the parameters of these of first group experiment in the Table 1. The target texture image and the synthesis texture patch are square, the number of the synthesis texture patches is 6 × 6. The experimental results are shown in Table 2.

Table 2. The comparation of the synthesis result of the Image Quilting algorithm and the line-based synthesis algorithm
Full size table

From the synthesis results shown in Table 2, when the synthesis parameters are the same, the line-based synthesis algorithm can raise the synthesis speed, but can not reduce the synthesis result effect. Moreover, because the search range of the best horizontal sews lines is enlarged, the synthesis results effect can be raised more.

4.3 The Results of the Parallel Calculation

In this experiment, we use the parallel calculation to search the best matching texture patch. The synthesis parameters are the same with the synthesis parameters of the first group in Table 1. The experimental results are shown in Table 3.

Table 3. The experimental results of the line-base synthesis by parallel calculation (n = 3 bN = 6)
Full size table

As we can see in the table above, for the line-based synthesis, the calculation speed raises a lot by joining the parallel calculation than using serial calculation, and the average time saving rate is about 45.55 %, which fully illustrates the feasibility and effectiveness of parallel calculation.

5 Conclusions

In this paper, we improve the Image quilting texture synthesis algorithm, change the L-overlap region to the r-overlap, and eliminate the redundancy calculation in the original algorithm. When looking for the best sew line, we change the patch-based suture to line-based. Then based on these two changes, we propose the line-based synthesis algorithm, which raises the speed of the texture synthesis and also reduces the complexity of the algorithm. At the same time, we adopt parallel calculation to select the best matching synthesis patch. A large number of experiments are used to compare the synthesis results and time between the patch-based synthesis and line-based synthesis. The experimental results show that line-based synthesis can reduce the algorithm complexity, and raise the synthesis speed. When joining the parallel calculation, the synthesis speed can raise nearly about 45 %, while the synthesis results not reduce. The further research will be expanded by going on raising the texture synthesis speed. GPU as the currently popular graphics processor can be considered to use for increasing the speed of texture synthesis, and at the same time, studying the general GPU texture synthesis algorithm has important meaning.