We describe a novel appearance based scheme for extraction and representation of video objects. The tracking algorithm used for video object extraction is based upon a new eigen-space update scheme. We propose a scheme for organisation of video objects in an appearance based hierarchy. The appearance based hierarchy is constructed using a new SVD based eigen-space merging algorithm. The hierarchy enables approximate query resolution. Experiments performed on a large number of video sequences have yielded promising results.