In smart CCTV systems, the capability of automatic detection and recognition of the objects from the input sequence of images is a critical factor in evaluation of the system. However, CCTV images based on traditional RGB model commonly have limitation in object detection and tracking due to the lack of topological information of the objects in the images. This problem can be resolved by adding the depth information of the object created by using a 3D camera. In this paper, we present a novel approach for detecting and tracking moving objects with depth information of 3D CCTV images. The proposed object detection and tracking scheme based on RGB-D model can alleviate the drifting problem, especially when the object frequently interacts with similar background or other object in long sequence tracking periods. In this paper, stereo-based depth maps are utilized by using 3D camera in addition to the RGB color data. The RGB-based segmented region is set as a domain for extracting depth information, and depth-based segmentation is performed within the domain. In order to track the direction of the detected object, we adopt CAMShift, which is well-known object tracking algorithm. From the experiment, we can prove that the proposed RGB-D model-based object detecting and tracking approach is efficiently work in a complicate scene rather than using only traditional RGB based CCTV framework.