A new type of digital filter for removing impulsive noise in color images is proposed using interactive evolutionary computing. This filter is realized as a rule-based system containing switching median filters. This filter detects impulsive noise in color images with rules and applies switching median filters only at the noisy pixel. Interactive evolutionary computing (IEC) is adopted to optimize the filter parameters, considering the subjective assessment by human vision. In order to detect impulsive noise precisely, complicated rules with multiple parameters are required. Here, the relationship between color components and the degree of peculiarity of the pixel value are utilized in the rules. Usually, optimization of such a complicated rule-based system is difficult, but IEC enables such optimization easily. Moreover, human taste and subjective sense are highly considered in the filter performance. Computer simulations are shown for noisy images to verify its high performance.