In this paper, we present an adaptive background generation method for automatic selection of initial object region for simultaneous object detection and depth estimation using the MCA camera. We then estimate color shifting vectors (CSVs) of the extracted object's region using simplified elastic registration algorithm in the pyramid-based multi-resolution framework. Using the color shifting property of the MCA camera, the depth of the object's region is estimated from CSVs. The proposed method uses only one camera for object detection and depth estimation. Experimental results show that the MCA camera-based object tracking system can be used in various consumer surveillance systems.