Background modeling is an important component of many computer vision systems. The numerous approaches to this problem differ in the statistical models used to describe the temporal behavior of single pixels. Without proper use of spatial coherence between pixel values, these models suffer greatly from memory consumption. In order to reduce spatial redundancy in the data, we propose a novel block- based background model which clusters pixel values within each small block of frames, and build weighted indexes for each pixel to track color values temporally. Compared with traditional models, the proposed model greatly reduces average number of bytes needed to model a pixel, and can be used in real-time video surveillance systems.