In the past few decades, machine learning tools have been successfully used in several medical diagnostic problems. While they often significantly outperform expert physicians (in terms of diagnostic accuracy, sensitivity and specificity), they are mostly not being used in practice. One reason for this is that it is difficult to obtain an unbiased estimation of the diagnosis's reliability. We discuss how the reliability of diagnoses is assessed in medical decision-making and propose a general framework for reliability estimation in machine learning, based on transductive inference. We compare our approach with the usual machine-learning probabilistic approach, as well as with classical step-wise diagnostic process, where the reliability of a diagnosis is presented as its post-test probability. The proposed transductive approach is evaluated in a practical problem of the clinical diagnosis of coronary artery disease. Significant improvements over existing techniques are achieved.