We evaluate different cross-validation (CV) protocols for an automated classification of colonic polyps. For this purpose we select six previously developed methods which achieved promising results already in the past. We then evaluate the methods using the cross-validation protocols leave-one-image-out (LOO-CV), leave-one-parent-image-out (LOPIO-CV), leave-one-lesion-out (LOLO-CV), and leave-one-patient-out (LOPO-CV). We show that, in general, the more restrictive cross-validation protocols lead to high results drops. While in case of LOO-CV the accuracies are rather high across all methods evaluated, the picture changes the more strictness a cross-validation mode imposes on the set of training images.