This paper presents a technique to efficiently index and retrieve video clips in terms of motion-trajectory-based similarity. We describe the motion trajectory in three representations: the horizontal and vertical movements of the trajectory, and the motion trail that indicates shape of the trajectory. Each representation is approximated by a polynomial function. We index the polynomial coefficients and combine different spatio-temporal characteristics to provide flexible retrieval processes. A unified framework is also proposed to deal with various query types: query-by-example, query-by-sketch, and query-by-specification.