Inverse dynamics modeling is a critical problem for the computed-torque control of robotic manipulator. This paper presents a novel recurrent network based on the modified Simple Recurrent Unit (SRU) with hierarchical memory (SRU-HM), which is achieved by the nested SRU structure. In this way, it enables the capability to retain the long-term information in the distant past, compared with the conventional stacked structure. The hidden state of SRU is able to provide more complete information relevant to current prediction. Experimental results demonstrate that the proposed method can improve the accuracy of dynamics model greatly, and outperforms the state-of-the-art methods.