We propose to learn spatiotemporal features for video-based facial expression recognition with multi-layer independent subspace analysis (ISA) algorithm. On the first layer, a set of ISA filters are learned from small 3D patches of the video data, and then more abstract and powerful features on the second layer are learned from the feature responses of the first layer. Two public facial expression databases, extended Cohn-Kanade and MMI are used to evaluate our method. Experimental results show that the features learned by multi-layer architecture achieve better recognition performance than that of single-layer model. Furthermore, our method outperforms popular hand-crafted features, and the overall accuracy of our method is comparable to some related feature learning based methods.