Recently, Self-Supervised Learning (SSL) has achieved great success in various famous applications e.g., BERT and ChatGPT. However, when applying SSL to safety-critical downstream tasks, such as self-driving cars, potential adversarial attacks can completely change the final decisions and thus lead to serious security issues. To overcome this issue, existing methods combine adversarial training with pre-training to improve the adversarial robustness of SSL. However, combining these two computationally complex processes may largely amplify the computation cost. Moreover, whether performing adversarial training in pre-training or fine-tuning, current methods may degrade the accuracy due to the famous catastrophic forgetting problem. The computation cost of current adversarial training methods based on full parameter updating is still high even in the fine-tuning stage.To address the above challenges, we propose an effective robust fine-tuning framework for SSL based on Low-Rank Adaptation (LoRA), named LoFT. First, LoFT performs adversarial training in the fine-tuning stage rather than in the pre-training stage. Second, LoFT innovatively and elaborately integrates LoRA into adversarial training to avoid the catastrophic forgetting problem. Third, LoFT exploits a low-rank matrix in LoRA which enables efficient fine-tuning by updating only a small set of parameters, which contains only 1%-5% of the parameters of the pre-trained model. The whole pre-training and fine-tuning stages take only 9.44 hours, which reduces training time by 3× over the current SOTA method. Furthermore, compared with existing SOTA robust pre-training methods for SSL, LoFT improves accuracy by 5.97% (77.41%⇒83.38%) and robustness by 13% (45.04%⇒58.44%) on the CIFAR-10 dataset.