In this paper we propose Group k-Sparse Temporal Convolutional Neural Networks for unsupervised pretraining using video data. Our work is the first to consider the recurrent extension of structured sparsity, thus enhancing representational power and explainability. We show that our architecture is able to outperform several state-of-the-art baselines on Rotated MNIST, Scanned CIFAR-10, COIL-100 and NEC Animal pretraining benchmarks for video classification using limited labeled data.