This paper presents a constant quality video rate control (CQVRC) scheme for MPEG-4 FGS (fine-grain scalability) video. The proposed scheme can mitigate quality variations among consecutive frames during the period when there is significant change in the video source or the transmission bandwidth. CQVRC utilizes both non-scalable base layer (BL) and scalable enhancement layer (EL) of MPEG-4 for smoother video quality. To take advantage of the relaxed delay requirement of streaming video, CQVRC for BL exploits a larger decoder buffer, future frame information, and temporal scene segmentation. CQVRC for EL is achieved by embedding one rate-distortion (R-D) pair in each bitplane to construct a piecewise linear R-D model. Experiment results show higher than 1 dB PSNR gain and smaller PSNR variation by comparing the proposed CQVRC with current MPEG-4 FGS VM approach, where BL is encoded with MPEG-4 VM rate control and uniform bit allocation is applied to EL.