Atrial Fibrillation (AFib) is one of the most common cardiovascular diseases that can cause serious consequences if it is not treated on time. Although AFib is defined as an irregular rhythm with the absence of P waves in the electrocardiogram signal, still the algorithms for AFib detection are not perfect. When analyzing the state-of-the-art literature, we found several problems in identification consistent methodology that will allow comparison of different methods. The inconsistency problems are found in defining the set of benchmark tests and in interpretation of the key performance indicators. In this paper, we present three different methodologies and argue which key performance indicators should reveal the best performance evaluation. Choosing the F1 score allows conveying the balance between the positive predictivity value (precision) and sensitivity (recall). Also, we found that the way the true and false positives, and true and false negatives are determined varies from one approach to another and argue on which performance evaluation approach is the most relevant.