Elsevier

Computers & Graphics

Volume 32, Issue 6, December 2008, Pages 682-694
Computers & Graphics

Technical Section
Image-based three-dimensional model reconstruction for Chinese treasure—Jadeite Cabbage with Insects

https://doi.org/10.1016/j.cag.2008.08.001Get rights and content

Abstract

This paper presents a novel 3D reconstruction system for the famous Chinese treasure, Jadeite Cabbage with Insects, from uncalibrated image sequences. There are two major challenges for this 3D model reconstruction problem. The first is the difficult image registration problem due to the semi-diaphaneity and the highly specular property of jadeite materials. Secondly, the unknown camera information, including the intrinsic (calibration) and extrinsic (position and orientation) parameters, to be recovered from the uncalibrated image sequences makes the 3D reconstruction problem very challenging. The proposed 3D modeling process first recovers the camera information as well as sparse 3D structure by using a robust structure from motion algorithm. Then an approximate 3D object model is recovered from the silhouettes at the corresponding multiple views by using the visual hull technique. The final process refines the 3D model by further integrating the 3D information extracted from dense correspondences between image patches of different views. In the proposed 3D reconstruction system, we successfully combine the structure from motion and visual hull techniques to accomplish this challenging task of reconstructing an accurate 3D model for jadeite object from uncalibrated multi-view images. Finally, we assess the 3D reconstruction results for the Chinese jadeite treasure and simulated data by using the proposed 3D reconstruction system.

Introduction

In March 2007, it was reported that a priceless jadeite treasure, Jadeite Cabbage with Insects, of National Palace Museum in Taiwan was damaged for some unknown reason. It reveals the importance of careful preservation of historical treasure. The three-dimensional (3D) digital preservation provides the advantages of invariant preservation, remote display, ease of browsing, 3D model copy, etc. There have been quite some pioneering researches on preserving cultural and historical relics, e.g. famous pictures, stone carvings, and well-known architectures and landscape. There are many priceless Chinese treasures of jadeite material, but existing 3D scanning techniques usually fail to acquire 3D models of such jadeite objects because of their semi-transparent and reflective material properties. In addition, the safety consideration also makes the 3D laser scanning or other contacting methods unfavorable. Based on these considerations, we developed a semi-automatic system for 3D reconstruction of jadeite treasures from images of different views.

In this work, a sequence of 72 images taken from different views for the Jadeite Cabbage with Insects are provided by National Palace Museum with some sample views shown in Fig. 1. Unfortunately, the information about the camera settings, camera parameters, and the lighting conditions for the image acquisition are unavailable. This image sequence was acquired originally for the purpose of image-based rendering, and re-capturing is very difficult due to the protection rules of the national treasure. Thus, the major challenges for the 3D reconstruction from this image sequence come from two aspects: the first is the semi-diaphaneity and the highly specular property of jadeite materials under unknown lighting conditions; and the second is the unknown camera information from the acquired image sequences, including the intrinsic and extrinsic (position and orientation) camera parameters. The special material property makes the image registration between images of different views very difficult. To justify the first challenge, we applied the SIFT [1] feature extraction and matching on some of this image sequence and the result showed a very high percentage of mismatches in Fig. 2. Furthermore, the unknown camera information causes ambiguity in determining the camera calibration matrix, camera pose, and the 3D structure simultaneously under the uncertain correspondences inherited from the first challenge.

To overcome the above challenges, we proposed a semi-automatic 3D reconstruction system. This system integrates several state-of-art 3D reconstruction techniques in a novel way. First of all, from a set of manually selected corresponding points, we estimate the camera information by using the uncalibrated structure from motion algorithm. Secondly, we build an initial 3D model as a convex shape from the object silhouettes by using the visual hull technique [2], since the silhouettes usually can provide quite reliable but limited information for 3D reconstruction. From this rough 3D model, we subsequently apply a modified optical flow technique to establish dense point correspondences between images of neighboring views by considering the variant color illumination in a regularization framework. Finally, all the extracted information is integrated into the dense depth estimation to refine the final 3D model. The details of the proposed 3D reconstruction system are given in Section 3.

The main contribution of this paper is to develop a 3D reconstruction system by integrating the structure from motion (SfM) and visual hull techniques in combination with the robust image registration algorithm for object of jadeite material. Previous techniques fail to provide accurate point correspondences for the jadeite object due to its complex material reflection property. By using a generalized brightness variation model and considering the reflective highlight effect, we developed a robust optical flow computational technique to reliably compute the dense matching between the image patches. Based on the extracted camera information and registered image patches, the dense depth information of the jadeite object can be estimated, thus the 3D model is refined.

This paper is organized as follows. In the next section, we begin by giving a brief review of the background for the related information. Then the system overview is presented in Section 3. In Section 4 and Section 5, we describe the robust structure from motion algorithm and the modified optical flow computational algorithm for objects with reflective and semi-diaphaneity property, respectively. Experimental results on the Chinese treasure, Jadeite Cabbage with Insects, are presented in Section 6. Finally, we conclude this paper in the last section.

Section snippets

Previous works

In this section, we briefly review some representative 3D reconstruction systems as well as the related technologies employed in the proposed 3D reconstruction system.

System overview

Fig. 3 shows the flow diagram of the developed system. The two orange blocks depict the extracted information from the input sequence, including the manually selected point correspondences and the silhouette information of the object in image space. The green blocks represent the associated algorithm, and the blue ones are the outputs of each algorithm.

This system can be divided into two major stages: initial rough model reconstruction and model refinement. For the initial model reconstruction,

Metric structure from motion

The structure from motion problem is to simultaneously recover the camera and the structure information from the feature correspondences extracted from multi-view images. The camera information includes the camera intrinsic and extrinsic parameters.

Let us begin with a brief description of the standard procedure for projective reconstruction and metric upgrade. There are 11° of freedom for each 3×4 projection matrix with scaling ambiguity, and 3 degrees of freedom for each 3D point represented

Dense optical flow computation

Image registration is a crucial and challenging problem in this work due to the particular jadeite material, and its accuracy directly influences the final 3D reconstruction results. Optical flow computation is generally applied under the assumption of small motion. For large motion or displacement, we need an approximate motion field as the initial solution to the optical flow computational algorithm. In this work, we use the radial basis function (RBF) interpolation from the corresponding

3D reconstruction of the Jadeite Cabbage with Insect

In this section, we show the results of applying the proposed system for reconstructing the famous national treasure—Jadeite Cabbage with Insect. This image sequence was originally taken for making quicktime VR by National Palace Museum, Taipei, Taiwan. This jadeite treasure was positioned on a turntable, and the images are taken with a fixed camera at roughly every 10° along both the latitude and the longitude. Note that the camera calibration information is unknown in this 3D reconstruction

Conclusions

In this paper, we presented a novel 3D reconstruction system for 3D digital preservation for objects of jadeite material. We successfully applied this 3D reconstruction technology to one of the most famous Chinese treasures, Jadeite Cabbage with Insects. Due to the special property of jadeite material as well as the unknown camera and lighting variation information, the 3D reconstruction of the jadeite object from multi-viewed images is very challenging. To overcome these difficulties, we

Acknowledgments

The authors would like to thank National Palace Museum, Taipei, Taiwan, for providing the images of the national treasure, Jadeite Cabbage with Insects, for this research work. This research was partially supported by National Palace Museum, Taiwan, Industrial Technology Research Institute, Taiwan, and National Science Council (Project NSC 93-2213-E-007-003), Taiwan, ROC.

References (28)

  • B.K.P. Horn et al.

    Determining optical flow

    Artificial Intelligence

    (1981)
  • C.H. Teng et al.

    Accurate optical flow computation under non-uniform brightness variations

    Computer Vision and Image Understanding

    (2005)
  • D. Lowe

    Distinctive image features from scale-invariant keypoints

    International Journal of Computer Vision

    (2004)
  • A. Laurentini

    The visual hull concept for silhouette-based image understanding

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (1994)
  • R. Hartley et al.

    Multiple view geometry in computer vision

    (2000)
  • A. Bartoli et al.

    Constrained structure and motion from multiple uncalibrated views of a piecewise planar scene

    International Journal of Computer Vision

    (2003)
  • Pollefeys M. Self-calibration and metric 3D reconstruction from uncalibrated image sequences. PhD thesis, ESAT-PSI,...
  • P. Chen et al.

    Recovering the missing components in a large noisy low-rank matrix: application to SFM

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (2004)
  • C. Tomasi et al.

    Shape and motion from image streams under orthography: a factorization approach

    International Journal of Computer Vision

    (1992)
  • S. Christy et al.

    Euclidean shape and motion from multiple perspective views by affine iterations

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (1996)
  • Cheung GKM, Backer S, Kanade T. Visual hull alignment and refinement across time: a 3D reconstruction alignment...
  • Stich T, Tevs A, Magnor M. Global depth from epipolar volumes—a general framework for reconstructing non-Lambertian...
  • Ebrahimnezhad H, Ghassemian H. 3D shape reconstruction of moving object by tracking the sparse singular points. In:...
  • G. Vogiatzis et al.

    Using frontier points to recover shape, reflectance and illumination

    Proceedings of the IEEE Conference on Computer Vision

    (2005)
  • Cited by (0)

    View full text