A rate control (RC) algorithm is a primitive block of video encoders that fulfills the bandwidth and buffer constraints for given channel and application properties. State-of-the-art RC schemes perform inefficient in terms of buffer and quality smoothness when handling varying rate distortion characteristics of different picture types (I, P, B) and different MBs in one picture (e.g. bright, textured, static/moving MBs) while posing a high computational overhead. In this paper, we propose a novel RC scheme that covers GOP, picture/ slice, and basic unit levels. It treats different picture types (I, P, B) in a non-linear fashion with consideration of whether they are referenced or non-referenced pictures. Our novel RC scheme prioritizes Macroblocks depending upon their spatial and temporal characteristics (considering eye-catching regions) for refined Quantization Parameter allocation. Compared to RC-Mode-3 (i.e. the latest RC Mode in JM reference software), our RC achieves up to 77.8% and 72.4% reduced buffer-and quality fluctuations, respectively. Compared to RC-Mode-0, our RC provides 2.97dB (i.e. 7.2%) better PSNR for the mixed Susie sequence. Moreover, our proposed RC is 16.6× faster than the RC-Mode-0 when executing on Intel Core2Duo T5500 (1.66 GHz).