Multiple reference frame motion estimation (MRF ME) increases the video coding efficiency at the expense of increased computational complexity and energy consumption. Therefore, in this paper, a low complexity H.264 MRF ME algorithm and a low energy adaptive hardware for its real-time implementation are proposed. The proposed MRF ME algorithm reduces the computational complexity of MRF ME by using a dynamically determined number of reference frames for each Macroblock (MB) and early termination. The proposed H.264 MRF ME hardware is implemented in Verilog HDL. The proposed H.264 MRF ME hardware has 29-72% less energy consumption than an H.264 MRF ME hardware using 5 reference frames for all MBs with a negligible PSNR loss. Therefore, it can be used in consumer electronics products that require real-time video processing or compression with low power consumption.