The proper planning of repairs and rehabilitation of the asphalt pavement is one of the important tasks for safe driving. The most common form of distress on asphalt pavements are potholes, which can compromise safety, and result in vehicle damage. Timely repairing potholes is crucial in ensuring the safety, quality of driving, and reducing the cost of vehicle maintenance. Many of the existing methods for pothole detection often use sophisticated equipment and algorithms, which require substantial amount of data for filtering and training. Consequently, as a result of intensive computational processing, this can lead to long execution time and increased power consumption. In this paper, we propose an efficient unsupervised vision-based method for pothole detection without the process of training and filtering. Our method first extracts asphalt pavements by analysing RGB color space and performing image segmentation. When the asphalt pavement is detected, the search continues in detected region only. The method is tested on online image data set captured from different cameras and angles, with different irregular shapes and number of potholes. The results indicate that the method is suitable as a pre-processing step for other supervised methods.