Colonoscopy is the primary method for detecting and removing polyps - precursors to colon cancer, but during colonoscopy, a significant number of polyps are missed - the pooled miss-rate for all polyps is 22% (95% CI, 19%-26%). This paper presents an automatic polyp detection system for colonoscopy, aiming to alert colonoscopists to possible polyps during the procedures. Given an input image, our method first collects a crude set of edge pixels, then refines this edge map by effectively removing many non-polyp boundary edges through a classification scheme, and finally localizes polyps based on the retained edges with a novel voting scheme. This paper makes three original contributions: (1) a fast and discriminative patch descriptor for precisely characterizing image appearance, (2) a new 2-stage classification pipeline for accurately excluding undesired edges, and (3) a novel voting scheme for robustly localizing polyps from fragmented edge maps. Evaluations demonstrate that our method outperforms the state-of-the-art.