Atrial fibrillation (AF) is the most common cardiac arrhythmia and is associated with 4-5 fold increase in stroke risk. An electrocardiogram (ECG) is normally used for its identification. However, ECG is not readily available. Additionally, the rhythm strip at echocardiogram is often misleading. Here, we propose Echo-Rhythm Net, a deep learning-based method to automate AF detection based solely on echocardiogram imagery (echo) without the need for an ECG. The proposed framework consists of three main components: an encoder that is trained using a self-supervised method, a temporal self-similarity matrix layer, and a final supervised detector trained with labels of cardiac rhythm assigned by sonographers. Our Echo-Rhythm Net, trained with 3947 cines of which only 583 are labeled, achieves an accuracy of 79% on the detection of AF in an independent test dataset of 260 cines. This result is superior to that of a trained echocardiographer, who when given the same test data without ECG information, scored an AF detection accuracy of 63%.