Crossing different working conditions is a common scenario in rotating machinery fault diagnosis, which can be solved by cross-domain transfer learning. However, the existing diagnosis methods do not consider possibly new and unknown faults, i.e., open-set fault diagnosis scenarios, which would cause diagnosis performance degradation. To address this issue, in this article, the self-supervised-enabled open-set cross-domain (SEOC) approach is proposed for fault diagnosis of rotary machines under various working conditions. Specifically, open-set risk minimization and self-supervised contrastive learning are proposed to improve distinguishability and stability. A pseudolabel consistency self-training is designed to decrease the domain shift. A novel open-set identification strategy with the designed squeeze confidence rule is developed for unknown- and known-class fault detection. Experiments on three-phase motor and bearing datasets illustrate the superior and efficient performance of the proposed SEOC method. The proposed SEOC framework improves the overall classification accuracies by at least 9%, and the average accuracy of unknown fault identification is more than 97.68% in motor and bearing fault diagnosis.