Motion-compensated temporal filtering (MCTF) is the core technology of the next generation scalable video coding schemes. There are two key steps in MCTF process, prediction step and update step. In this paper, an efficient scheme along with the hardware architecture of update step for VLSI implementation is proposed. The proposed update step scheme can reduce 55% off-chip memory access and turn the irregular access into regular access. 25% on-chip memory access is also reduced in deriving inverse motion vectors. 80% hardware area is saved by reusing the hardware resources of prediction step.