In visual anomaly detection, anomalies are often rare and unpredictable. For this reason, we aim to build a detection framework that can detect unseen anomalies with only anomaly-free examples. In this paper, we introduce a Self-supervised Anomaly Detection approach with Synthetic Anomalies (SADSA), a two-stage framework for detecting and localizing anomalies in images with self-supervised learning. We create anomalies through the designed augmentation strategy. In the first stage, the model learns to distinguish normal data from synthetic anomalies. In the second stage, we extend the self-supervised task to downstream anomaly representation extraction and aggregate features from different semantic levels to improve the detection and localization performance. Without extra training samples and pre-trained models, SADSA achieves 96.4% detection AUROC and 96.1% localization AUROC on the MVTec AD benchmark. This is competitive against existing unsupervised methods. The results demonstrate the potential of SADSA for industrial applications.