In this paper, we develop an object-based rate control algorithm for MPEG-4 arbitrarily shaped video object coding. An improved quadratic rate-distortion model for texture information is derived, which is more accurate for arbitrarily shaped video objects. We also introduce an accurate rate-distortion model for shape information. Combining the rate-distortion models for texture and shape information, an object level rate control algorithm is proposed. To achieve constant bit rate, the quantizer is adjusted based on buffer fullness. The tradeoff between picture quality and buffer fullness is considered. Comparing with the rate control algorithm for the MPEG-4 verification model, our proposed scheme can achieve a more stable buffer without sacrificing picture quality.