Computed Tomography (CT) scanning is widely used for chest diseases detection including pneumonia due to its diagnostic efficacy and efficiency. Recent studies have shown that the distribution of infection regions caused by COVID19 in CT images is different from other pneumonia, and the COVID-19 cases are more likely to suffer severe and large-area infections. However, most deep learning methods only focus on intra-slice features and ignore cross-slice features. In this paper, we propose a novel two-stage method to fully exploit deep cross-slice features from volumetric CT data, including a dual-task supervised CNN and a context-aware Bi-LSTM. To further demonstrate the effectiveness of our model, we conduct extensive experiments on a chest CT imaging dataset with a total of 801 patients (250 healthy people, 238 COVID-19 patients, 191 H1N1 patients, and 122 CAP patients). The experimental results indicate the superiority of our proposed model on the multi-class pneumonia classification task.