The basic problem we address in this paper is the following. Given a set of sparse point correspondences, how does one obtain a dense motion field and a motion-based segmentation of the image sequence? We propose a novel solution to this problem by formulating it as pursuit in segmentation space. This segmentation is defined by level set evolution equations, allowing changes in segmentation topology.