In this work, we propose a novel scheme to minimize drift in scalable wavelet based video coding, which gives a balanced performance between compression efficiency and reconstructed quality with less drift. Our drift control mechanism maintains two frame buffers in the encoder and decoder; one that is based on the base layer and the one that is based on the base plus enhancement layers. Drift control is achieved by switching between these two buffers for motion estimation and compensation and base layer information is used for predicting enhancement layers. Our coder is designed for variable bit rate wireless channels and offers high compression efficiency and sustained video quality.