Continuous action recognition plays an important role for human behavior analysis. Most existing approaches require fully labelled action videos, which is labour and time consuming to get. In this paper, we propose a continuous action recognition approach for weakly labelled videos data, where only the orders of action labels are needed without its temporal location's. We build a deep network combing convolutional neural network (CNN) and latent-dynamic dynamic conditional random field (LDCRF) to learn action features and recognize actions in a unified procedure. A visual similarity extended connectionist temporal classification (CTC) layer is put on the top of the network to evaluate all possible of temporal locations of weakly labelled videos data. The whole network can be trained end-to-end under weakly supervision. Experimental results on dataset HumanEva show our approach is promising and practical.