Video segmentation is a key operation for MPEG-4 and MPEG-7. It is a computationally intensive task and cannot be afforded by general microprocessors. In this paper, we propose a hardware accelerator for video segmentation. Since most of the core operations of video segmentation algorithms can be mapped to morphological operations, the core of this accelerator is a programmable morphology PE array. With the programmable ability of each PE and the interconnection between them, this accelerator can achieve both high throughput and high flexibility. Simulation shows the proposed hardware accelerator can speed up the most important video segmentation algorithms 20 times to achieve realtime performance and has high programmability with a little hardware cost overhead.