Hardness is one of the most essential tactile clues for robots to recognize objects. However, methods for robots to recognize hardness are limited. In this paper, based on the Capsule Network (CapsNet), we propose a novel tactile capsule network (TactCapsNet) for object hardness recognition. Specifically, we collect a tactile dataset on the silicone samples with three different shapes, and the silicone samples of each shape have thirteen hardness levels ranging from 0A (Shore A scale) to 60A at 5A intervals. Furthermore, we construct the tactile image as the input of the CapsNet to make full use of the spatio-temporal information of the tactile hardness dataset. The experimental results prove that the proposed approach achieves higher accuracy and quadratic weighted kappa (QWK) than support vector machine (SVM), long short-term memory (LSTM), convolutional neural network (CNN), and CapsNet.