Visual Localization Through Virtual Views

Abstract

This paper addresses the problem of camera localization, i.e. 6 DoF pose estimation, with respect to a given 3D reconstruction. Current methods often use a coarse-to-fine image registration framework, which integrates image retrieval and visual keypoint matching. However, the localization accuracy is restricted by the limited invariance of feature descriptors. For example, when the query image has been acquired at the illumination (day/night) not consistent with the model image time, or from a position not covered by the model images, retrieval and feature matching may fail, leading to false pose estimation. In this paper, we propose to increase the diversity of model images, namely new viewpoints and new visual appearances, by synthesizing novel images with neural rendering methods. Specifically, we build the 3D model on Neural Radiance Fields (NeRF), and use appearance embeddings to encode variation of illuminations. Then we propose an efficient strategy to interpolate appearance embeddings and place virtual cameras in the scene to generate virtual model images. In order to facilitate the model image management, the appearance embeddings are associated with image acquisition conditions, such as daytime, season, and weather. Query image pose is estimated through similar conditional virtual views using the conventional hierarchical localization framework. We demonstrate the approach by conducting single smartphone image localization in a large-scale 3D urban model, showing the improvement in the accuracy of pose estimation.

This work was completed when Zhenbo Song was an intern at ByteDance. This work was supported in part by the Jiangsu Funding Program for Excellent Postdoctoral Talent under Grant 2022ZB268.