Real-time object tracking is demanded in versatile embedded computer vision applications. To this end, a low-cost high-speed VLSI system is proposed for object tracking, based on robust and computationally simple unified textural and dynamic compressive sensing features and simple elliptic matching with fast online template updating capability. The system introduces a memory-centric architectural paradigm, multiple-level pipelines and parallel processing circuits to achieve high frame rate while consuming as few hardware resources as possible. We implemented an FPGA prototype of the proposed VLSI tracking system. Under a 100 MHz clock frequency, the prototype achieves over 600 frame/s processing speed under $320\times240$ image resolution and obtains robust tracking results.