Optical signal processing with illumination-encoded filters

Abstract

Recently, computer vision researchers have shown that orthogonal functions and computational techniques from the signal processing framework can be mapped directly into the scene using projector–camera systems. These scene-space signal processing algorithms are achieved with illumination-encoded functions as primitives and computations derived from surface reflection models. Some examples of this new optical approach include convolution filtering and aliasing-canceling filter banks.

In this paper we present computational techniques for realizing fundamental elements of the signal processing framework in the 3D scene domain. The motivation for optical computation directly in the scene is to avoid information loss when the rich 3D scene is reduced to an image. The computations are at subpixel resolution because they are performed within each camera sensor. Scene-space filtering applies 2D operators to locally planar 3D surfaces based on the optical coupling of the scene surface topology with projector–camera devices. Signal processing issues such as sampling geometry, dynamic range, mathematical operators, and resolution are addressed. We report a novel subpixel point correspondence technique for accurate camera sensor footprint localization in general scenes. It is a parallelizable optical clipping algorithm related to the polygon clipping and box filter anti-aliasing algorithms used in computer graphics. It replaces the regular coordinates of an image with a local surface parametrization in projector coordinates. The result is subpixel resolution filter responses. We provide experiments and results to evaluate the performance of our scene-space filtering techniques with both planar and non-planar objects.

Introduction

A common projector–camera technique is to encode the pixel positions of the projector with binary structured light to resolve point correspondences with the camera pixels that recover the code [1]. Techniques using high-frequency primitives have been used to probe optical properties such as defocus blur to compute depth [2], [3]. A high-frequency pattern is projected into the scene and the reflected signal is analyzed for low-pass filtering effects (blur). Other schemes use binary patterns to discern the global and local illumination components [4]. There are other variations of these techniques highlighting the effectiveness of projector–camera systems in computer vision problems.

Recently, researchers have investigated light patterns based on orthogonal bases from the signal processing domain. The use of analytical functions as structured light primitives allows for new approaches to computer vision problems. Computations are performed directly in the scene by modeling signal processing formulations in the optical domain with digital projectors and cameras. The work of Damera-Venkata and Chang [5], Jean [6], [7], and Ghosh et al. [8] clearly demonstrate that signal processing formulations can be implemented in scene-space by optically encoding analytical functions and leveraging the native reflection models in the scene. This is the core computational model and is easily extended with processor-based image processing. Mapping signal processing tools into scene-space also requires satisfying the requirements for signal representation and the analytical constraints of sampling theory in the optical domain.

The motivation for optical computation directly in the scene is to avoid information loss when the rich 3D scene is reduced to an image. Passive computer vision techniques suffer from the inherent ambiguities and overall challenge of inverse mapping from an image to the real world phenomena. For example, stereo vision systems must address the point correspondence problems between an image pair before the surface reconstruction.

Active vision techniques have been successful in removing some of these ambiguities. One solution is to replace a camera with a digital projector and use structured light techniques [1] to introduce a spatial code into the scene to disambiguate the pixel correspondences. Depth discontinuities are another challenge in stereo images and researchers have utilized a simpler form of active light, multiple flash images [9], to localize these features in an image. Clearly, introducing a known signal into an unknown scene, like in radar, is a valuable strategy in solving computer vision problems (i.e., photometric stereo [10]). And the extra images required can be produced at very high rates with current consumer-grade digital projectors [11], [12].

In this paper we present a technique for producing subpixel filter responses of scenes with locally planar topology. We add to our early results in [7], an orthographic projection calibration technique designed to produce higher accuracy light fields with only consumer-grade digital projector technology, and some unpublished results. The calibration technique is used to produce an orthographic projection system, and specifically, to bring the regular sampling geometry of signal processing to scene-space techniques. We report a novel subpixel projector–camera correspondence technique using an optical analog of polygon clipping and anti-aliasing from computer graphics. The goal is to replace the regular coordinates of an image with a local surface parametrization in projector coordinates. We compare processor-based image processing to our optical method and perform experiments using objects of varying topology.

In Section 2 we review some topics in the projector–camera literature for probing the environment and scene-space techniques, in particular. A technique for more general scenes is presented in Section 5. Experiments with various objects are documented in Section 6 and a discussion of the results is in Section 7.