Video transmission over mobile worldwide interoperability for microwave access (WiMAX) can be serverly degraded due to the effect of fading and handoff. In this paper, we propose a channel adaptive error resilience scheme for video transmission over mobile WiMAX. When the channel condition begins to trigger handoff, the current frame is stored in the long-term memory for the forward error correction, and the following frames are encoded by using double motion vectors (MVs) in the sense of multi-hypothesis motion compensation. Even if a whole frame is lost, we can reconstruct the following frames using the stored frame in the long-term memory. However, the error propagation still remains in this forward error resilience method. To refresh the erroneous frames to the decoder, the encoder utilizes the channel adaptive refreshing (CAR). In the CAR, the channel rate is first predicted using channel parameter, a carrier to interference and noise ratio (CINR), and the encoder adaptively determines the number of blocks to be encoded in the intra mode based on the feedback information. Performance evaluations are presented to demonstrate the effectiveness of the proposed method.