Anomaly detection and localization in the surveillance scenes is still a challenge in the filed of computer vision. Previous methods take this task as a one-class deviation problem, where the deviations between test samples and normal patterns are computed. In this paper, an Intra-frame Classification Network (ICN) is proposed to take the advantages of deep learning to transform this task to a multi-class classification problem. For video sequences, the feature maps are extracted by a set of Spatial Temporal Convolutional Layers (STCLs). The feature maps are then split into sub-regions by an Adaptive Region Pooling Layer (ARPL). The feature vectors of each sub-region are labelled as one class. The classification result for a sub-region is used to evaluate the abnormality of it. The proposed method is examined on UCSD Ped l dataset and UCSD Ped2 dataset. The results are further compared with previous state-of-the-art approaches to confirm the effectiveness and the efficiency of our method.