Machine-learning (ML) techniques have grown to be among the leading research topics within the health care systems and particularly for clinical decision support systems (CDSS), which are commonly used in helping physicians to make more accurate diagnosis. However, applying these techniques for CDSS is most likely would face a lack of criteria for adequate use. Therefore, a range of recent studies have focused on evaluating different machine learning classifiers with the aim of identifying the most appropriate classifier to be used for particular decision making problem-domain. The majority of these studies have used a single dataset within a certain medical-related classification domain. Nevertheless, evaluating machine-learning classifiers with one sample of data appears to be unsatisfying, perhaps it is not reflecting the classifiers capabilities or their behavioral patterns under different circumstances. In this study, five well-known supervised machine-learning classifiers were examined using five different real-world datasets with a range of attributes. The main aim was to illustrate not only the impact of the datasets volume and attributes on the evaluation, but also and more importantly, present the classifiers capabilities and shortcomings under certain conditions, which potentially provide a guidance or instructions to help health analysts and researchers to determine the most suitable classifier to address a particular medical-related decision making problem.