In this paper, we describe a cost-efficient programmable video signal processor (PVSP) to implement various video encoding and decoding schemes. Hierarchical (two level) programmable control architecture, flexible memory address mapping strategies and a programmable VLC/VLD module are applied in order to achieve sufficient programmability. Thus, PVSP can support various video compression algorithms and standards, such as MPEG-1, MPEG-2 H.263, and MPEG-4. Meanwhile, to improve the throughput of this codec system, some paralleling approaches are exploited on different levels, which include pipeline, tree adder, and SIMD (single instruction stream, multiple data streams). PVSP is estimated to have approximately 320 k gates and it can accomplish MPEG-2 MP@ML encoding in real-time at a frequency of 133 MHz.