In this paper, we present an adaptive background generation method for automatic selection of initial object regions, which realizes simultaneous object detection and depth estimation using multiple color-filter aperture (MCA) camera. Since the conventional background generation method does not fit the depth estimation using the MCA camera, we propose a novel color-based background generation method which can reduce interference in the object region for stable depth estimation. For efficient estimation of color shifting vectors in the extracted object region, a simplified elastic registration (ER) algorithm is used. The proposed simplified method is essential factor to realize realtime depth estimation and tracking, which is the primary condition for consumer applications. Finally, the object distance is determined by using the relationship between the pre-specified distance transformation function and the estimated shifting vectors of the corresponding object region. Although traditional depth estimation methods generally use dual cameras for stereo vision, the proposed method uses only a single camera for both object detection and depth estimation. Experimental results show that the proposed MCA camera-based object detection system can be used in a variety of consumer surveillance systems such as intelligent transport systems, 3D-based cameras and advanced safety vehicles.