Many deep learning methods have been proposed for the diagnosis of COVID-19 since the global pandemic. However, few studies have focused on the disease course classification of COVID-19, which is crucial for radiologists to determine treatment plans. This paper proposes a Multi-Modal Fusion Cascade (MMFC) framework for this task, which can make the most of multi-modal information, including CT image and bio-information (laboratory examination, clinical characterization, etc.). The proposed framework consists of two parts: Bio-Visual Feature Learning Module (BFL) and Joint Decision Module (JD). Firstly, BFL learns the discriminative visual features from the mediastinal window with the assistance of bio-information. According to the official Treatment Protocol of China, the bio-information is chosen and helps the BFL better extract the images’ bio-visual features and then obtained a disease course classification result based on CT images. Secondly, JD uses bio-information again and fuses the confidence of BFL’s result to make the joint decision. Experimental results show that our framework significantly improves accuracy and sensitivity compared to the baseline.