Rendering complex scenes using spatial subdivision and textured LOD meshes

Abstract

We present a hybrid rendering scheme that explores the locality of visibility at the cost of extra storage and prefetching, and makes a tradeoff between image quality and rendering efficiency by using textured level-of-detail (LOD) meshes. The space is first subdivided into cells. For each cell, inside objects are rendered as normal while outside objects are rendered as textured LOD meshes using projective texture mapping. The textured LOD meshes are object based and derived from the original meshes based on the captured depth images viewed at the centers of the cell and its adjacent cells. With such a textured LOD mesh, problems commonly found in image-based rendering, such as the hole problem due to occlusion among objects and the gap problems due to resolution mismatch, can be avoided. The size of holes due to self-occlusion is constrained to be within a user-specified tolerance. Several scenes with millions of polygons have been tested and higher than 200 FPS has been achieved with a little loss of image quality.

Introduction

In order to achieve an immersive visual effect during the VR navigation, rendering with photo-realistic scene images in high frame rate has been an ultimate goal of real-time rendering. In the traditional geometry-based rendering, very complex scenes often consist of numerous polygons that cannot be rendered at an acceptable frame rate even using a state-of-the-art hardware. Many techniques have been proposed in last decades on reducing the polygon count while preserving the visual realism of the complex scenes, including visibility culling, level-of-detail (LOD) modeling, and image-based rendering (IBR). Although IBR is capable of rendering complex scenes with photo-realistic images in the time that is independent of the scene complexity, it has been suffered from the static lighting, the limited viewing degree of freedom, and some losses of image quality due to gaps and holes. As a consequence, hybrid rendering that combines geometry- and image-based technique has become a viable alternative.

As a representation for an object or a region of the scene, several image-based or hybrid representations have been proposed. Shade et al. [1] described a paradigm in which regions or objects could be represented by environment map, planar sprite, sprite with depth, layered depth image (LDI), and polygonal mesh, depending on their distances to the viewer. Although the scheme integrates several existing representations, each individual form has its own problems. For example, sprites in general have gap problem due to resolution mismatch, and have to be re-computed once the viewer is outside the safe-region. LDI can only be drawn using software rendering with splatting. Finally, transition between different representations may produce noticeable popping effects.

To reduce gap problems due to resolution mismatch and to improve the efficiency of pixel-based rendering, depth meshes are extracted from the sprite with depth based on depth variation. However, rubber artifacts between disjoint surfaces are often encountered, and re-projecting pixel coordinates back to 3D coordinates may result in precision problems. The depth mesh approach can be incorporated by space subdivision, in which, when navigating inside a cell, distant objects are rendered using depth meshes with textures, while near objects are rendered by selected LOD models. With such approaches, the polygon count of a complex scene can be still high and, most importantly, the transition between LOD and depth mesh with texture will generally results in visually noticeable popping effects.

Another more uniform representation is LOD modeling, which can be incorporated with texture mapping for recovering surface details. View-independent LOD modeling has no control over silhouette during navigation. View-dependent LOD modeling, however, has to deal with silhouette problems at run-time by maintaining a mesh of fine resolution along silhouettes. Silhouette clipping that incorporates LOD modeling and normal/texture map needs to extract fine silhouettes at run-time, which is in general time consuming.

A hybrid rendering scheme that aims to render complex scenes in a constant and high frame rate with only a little or an acceptable quality loss is presented in this paper. To this end, view space is partitioned into cells to explore the locality of visibility, and for a view cell, each object outside the cell is represented by a LOD mesh together with textures that are derived with respect to the view cell. All these are done in a preprocessing. In contrast with IBR or depth mesh approach, the object-based LOD mesh derivation avoids hole problems due to occlusion among objects. In the meantime, to reduce hole problems due to self-occluding, the LOD mesh is classifised into either single-view LOD mesh (termed as SVMesh) or multi-view LOD mesh (termed as MVMesh), depending on the object's self-occluding error (w.r.t. the viewcell). The SVMesh is chosen if the object's self-occluding error is smaller than a user-specified tolerance, otherwise MVMesh is chosen. Such a condition on SVMesh ensures that the potential holes possibly found in the images viewed from any point inside the cell will have size less than the user-specified tolerance. Hence all the information necessary to guide the derivation of SVMesh and the texture associated with the SVMesh come from the captured image and captured depth image of the cell's center. On the other hand, the MVMesh presents geometry and texture necessary to avoid holes on images viewed from some points in the cell. Therefore, the derivation of MVMesh and its texture associations are based on captured images and depth images from the cell's center as well as the centers of adjacent cells. In the proposed scheme, prefetching is also implemented to preload the data necessary for the following cells such that sudden drops in the frame rate at the cell transition can be avoided.

The proposed approach explores locality of visibility at the cost of extra storage and prefetching, and makes a tradeoff between image quality and rendering efficiency by using the SVMesh and MVMesh together with textures. Our experiments have shown that for a scene of 8 million polygons we have achieved higher than 200 frames/s with a little loss of image quality (average PSNR 37.34 dB). The polygons and textures require about 1260 MB hard disk storage and about 287 MB run-time memory on average. With such high frame rates, the overhead of prefetching is hardly noticeable.