Technical SectionImage-based three-dimensional model reconstruction for Chinese treasure—Jadeite Cabbage with Insects
Introduction
In March 2007, it was reported that a priceless jadeite treasure, Jadeite Cabbage with Insects, of National Palace Museum in Taiwan was damaged for some unknown reason. It reveals the importance of careful preservation of historical treasure. The three-dimensional (3D) digital preservation provides the advantages of invariant preservation, remote display, ease of browsing, 3D model copy, etc. There have been quite some pioneering researches on preserving cultural and historical relics, e.g. famous pictures, stone carvings, and well-known architectures and landscape. There are many priceless Chinese treasures of jadeite material, but existing 3D scanning techniques usually fail to acquire 3D models of such jadeite objects because of their semi-transparent and reflective material properties. In addition, the safety consideration also makes the 3D laser scanning or other contacting methods unfavorable. Based on these considerations, we developed a semi-automatic system for 3D reconstruction of jadeite treasures from images of different views.
In this work, a sequence of 72 images taken from different views for the Jadeite Cabbage with Insects are provided by National Palace Museum with some sample views shown in Fig. 1. Unfortunately, the information about the camera settings, camera parameters, and the lighting conditions for the image acquisition are unavailable. This image sequence was acquired originally for the purpose of image-based rendering, and re-capturing is very difficult due to the protection rules of the national treasure. Thus, the major challenges for the 3D reconstruction from this image sequence come from two aspects: the first is the semi-diaphaneity and the highly specular property of jadeite materials under unknown lighting conditions; and the second is the unknown camera information from the acquired image sequences, including the intrinsic and extrinsic (position and orientation) camera parameters. The special material property makes the image registration between images of different views very difficult. To justify the first challenge, we applied the SIFT [1] feature extraction and matching on some of this image sequence and the result showed a very high percentage of mismatches in Fig. 2. Furthermore, the unknown camera information causes ambiguity in determining the camera calibration matrix, camera pose, and the 3D structure simultaneously under the uncertain correspondences inherited from the first challenge.
To overcome the above challenges, we proposed a semi-automatic 3D reconstruction system. This system integrates several state-of-art 3D reconstruction techniques in a novel way. First of all, from a set of manually selected corresponding points, we estimate the camera information by using the uncalibrated structure from motion algorithm. Secondly, we build an initial 3D model as a convex shape from the object silhouettes by using the visual hull technique [2], since the silhouettes usually can provide quite reliable but limited information for 3D reconstruction. From this rough 3D model, we subsequently apply a modified optical flow technique to establish dense point correspondences between images of neighboring views by considering the variant color illumination in a regularization framework. Finally, all the extracted information is integrated into the dense depth estimation to refine the final 3D model. The details of the proposed 3D reconstruction system are given in Section 3.
The main contribution of this paper is to develop a 3D reconstruction system by integrating the structure from motion (SfM) and visual hull techniques in combination with the robust image registration algorithm for object of jadeite material. Previous techniques fail to provide accurate point correspondences for the jadeite object due to its complex material reflection property. By using a generalized brightness variation model and considering the reflective highlight effect, we developed a robust optical flow computational technique to reliably compute the dense matching between the image patches. Based on the extracted camera information and registered image patches, the dense depth information of the jadeite object can be estimated, thus the 3D model is refined.
This paper is organized as follows. In the next section, we begin by giving a brief review of the background for the related information. Then the system overview is presented in Section 3. In Section 4 and Section 5, we describe the robust structure from motion algorithm and the modified optical flow computational algorithm for objects with reflective and semi-diaphaneity property, respectively. Experimental results on the Chinese treasure, Jadeite Cabbage with Insects, are presented in Section 6. Finally, we conclude this paper in the last section.
Section snippets
Previous works
In this section, we briefly review some representative 3D reconstruction systems as well as the related technologies employed in the proposed 3D reconstruction system.
System overview
Fig. 3 shows the flow diagram of the developed system. The two orange blocks depict the extracted information from the input sequence, including the manually selected point correspondences and the silhouette information of the object in image space. The green blocks represent the associated algorithm, and the blue ones are the outputs of each algorithm.
This system can be divided into two major stages: initial rough model reconstruction and model refinement. For the initial model reconstruction,
Metric structure from motion
The structure from motion problem is to simultaneously recover the camera and the structure information from the feature correspondences extracted from multi-view images. The camera information includes the camera intrinsic and extrinsic parameters.
Let us begin with a brief description of the standard procedure for projective reconstruction and metric upgrade. There are 11° of freedom for each 3×4 projection matrix with scaling ambiguity, and 3 degrees of freedom for each 3D point represented
Dense optical flow computation
Image registration is a crucial and challenging problem in this work due to the particular jadeite material, and its accuracy directly influences the final 3D reconstruction results. Optical flow computation is generally applied under the assumption of small motion. For large motion or displacement, we need an approximate motion field as the initial solution to the optical flow computational algorithm. In this work, we use the radial basis function (RBF) interpolation from the corresponding
3D reconstruction of the Jadeite Cabbage with Insect
In this section, we show the results of applying the proposed system for reconstructing the famous national treasure—Jadeite Cabbage with Insect. This image sequence was originally taken for making quicktime VR by National Palace Museum, Taipei, Taiwan. This jadeite treasure was positioned on a turntable, and the images are taken with a fixed camera at roughly every 10° along both the latitude and the longitude. Note that the camera calibration information is unknown in this 3D reconstruction
Conclusions
In this paper, we presented a novel 3D reconstruction system for 3D digital preservation for objects of jadeite material. We successfully applied this 3D reconstruction technology to one of the most famous Chinese treasures, Jadeite Cabbage with Insects. Due to the special property of jadeite material as well as the unknown camera and lighting variation information, the 3D reconstruction of the jadeite object from multi-viewed images is very challenging. To overcome these difficulties, we
Acknowledgments
The authors would like to thank National Palace Museum, Taipei, Taiwan, for providing the images of the national treasure, Jadeite Cabbage with Insects, for this research work. This research was partially supported by National Palace Museum, Taiwan, Industrial Technology Research Institute, Taiwan, and National Science Council (Project NSC 93-2213-E-007-003), Taiwan, ROC.
References (28)
- et al.
Determining optical flow
Artificial Intelligence
(1981) - et al.
Accurate optical flow computation under non-uniform brightness variations
Computer Vision and Image Understanding
(2005) Distinctive image features from scale-invariant keypoints
International Journal of Computer Vision
(2004)The visual hull concept for silhouette-based image understanding
IEEE Transactions on Pattern Analysis and Machine Intelligence
(1994)- et al.
Multiple view geometry in computer vision
(2000) - et al.
Constrained structure and motion from multiple uncalibrated views of a piecewise planar scene
International Journal of Computer Vision
(2003) - Pollefeys M. Self-calibration and metric 3D reconstruction from uncalibrated image sequences. PhD thesis, ESAT-PSI,...
- et al.
Recovering the missing components in a large noisy low-rank matrix: application to SFM
IEEE Transactions on Pattern Analysis and Machine Intelligence
(2004) - et al.
Shape and motion from image streams under orthography: a factorization approach
International Journal of Computer Vision
(1992) - et al.
Euclidean shape and motion from multiple perspective views by affine iterations
IEEE Transactions on Pattern Analysis and Machine Intelligence
(1996)