This paper presents a novel architecture for a classification system based on the visual saliency of images. The work is motivated by the difficulty of reviewing large numbers of images as a human operator in the context of Autonomous Underwater Vehicle (AUV) surveys. We formulate a feature space in which an algorithm operates over color and texture to determine saliency and illustrate how this can be used to find interesting or unusual images within a large data set. The saliency classification based on these general image features allows for overlays highlighting interesting benthos or geologic structures on large scale 3D seafloor reconstructions, quickly providing spatial context to human observers. These results are validated using a set of human trials in which images are classified into salient and non-salient categories by a number of test subjects. The trials show good agreement both between subjects and between the human labels and the automated classification system. The results of the automated technique are also compared directly to a more traditional SVM classification system showing favorable results for our system for generalizing to new environments.