Although a recurrent neural network (RNN) has achieved tremendous advances in video summarization, there are still some problems remaining to be addressed. In this article, we focus on two intractable problems when applying an RNN to video summarization: first the extremely large feature-to-hidden matrices. Since video features are usually in a high-dimensional space, it leads to extremely large feature-to-hidden mapping matrices in the RNN model, which increases the training difficulty. Second, the deficiency in long-range temporal dependence exploration. Most videos contain thousands of frames at least, which is such a long sequence that traditional RNNs cannot deal well with. Facing the abovementioned two problems, we develop a tensor-train hierarchical recurrent neural network (TTH-RNN) for the video summarization task. It contains a tensor-train embedding layer to avert the large feature-to-hidden matrices, together with a hierarchical structure of an RNN to explore the long-range temporal dependence among video frames. Practically, the experimental results on four benchmark datasets, including SumMe, TVsum, MED, and VTW, have demonstrated the excellent performance of a TTH-RNN in video summarization.