Colorectal cancer (CRC) is a leading cause of cancer. The incidence and mortality rates of CRC are expected to steadily increase in the future. Colonoscopy is the most popular and effect method for curing and screening CRC. However, 25% polyps were reported to be missed during colonoscopy examinations. In this study, we proposed a method to classify polyps from background based on bag-of-visual-words (BoW) from colonoscopy images. This method generates a histogram of visual word occurrences to represent an image. The histograms of a dataset were used to train an image category classifier. Validation was performed on 35 subjects' data with an average specificity of 97.01%, an average sensitivity of 99.43%, and an average accuracy of 97.8%.