Traditional image classifiers suffer from the so called "Background clutter problem", where they have to, simultaneously, localize and recognize an object of interest. In real-world video scenes, many objects appear at the same time and several objects of interest could be present. We decompose the problem into two stages. The first involves extracting a moving object from the video scene based on its coherent motion. This step makes no assumption about the nature of the object being tracked and extracted. We, then apply a classifier to the extracted object to determine if it falls into one of the classes of interest.