Effective background reconstruction is the key for real time traffic flow monitoring. High traffic density and complexity of background scene make reconstruction more difficult. Background estimation based on the median method is imprecise under a complex traffic flow condition. In this paper, a new background estimation method based on the similarity of background using parameters of gray mean and variance is proposed. Therefore, a two-dimensional clustering and merging mechanism is introduced. At last, accurate decision about the category of the background is made by analyzing the distribution characteristic of the frame numbers in one category. Our algorithm works on the difficult condition of traffic congestion with higher reliability. The proposed method can be used in background reconstruction of the crossroads based on video sequences.