Semantic communication has sparked great interest, due to the rising demands of emerging applications on high communication capacity and low latency. The majority of existing semantic communication methods are task-oriented, which transmit task-related semantic information via synchronous trained deep learning-based (DL-based) encoders and decoders. However, these methods have limitations in handling multi-task communications. Moreover, the synchronous training paradigm also leads to significant communication overhead in the establishing phase. In this article, we propose an asynchronous multi-task semantic communication method. In the proposed method, the DL-based encoder is trained independently using a contrastive learning method to extract task-independent semantic knowledge. Then, the receiver trains different DL-based decoders to perform various communication tasks based on the pre-trained encoder. Our method enables the accomplishment of multiple communication tasks in a single transmission. Moreover, the asynchronous training paradigm can reduce the communication overhead during the training phase of our system. The experimental results demonstrate that the proposed method achieves state-of-the-art performance in image classification and reconstruction tasks while requiring less than 10% of the training communication time compared to existing semantic communication systems.