NIST has been conducting a series of evaluations in the automatic analysis of information in video since 2001. These began within the NIST Text Retrieval Evaluation (TREC) as a pilot track in searching for information in large collections of video. The evaluation series was spun off into its own evaluation/workshop series called TRECVID. TRECVID continues to examine the challenge of extracting features for search technologies. In 2004, NIST also began an evaluation series dedicated to assessing video object detection and tracking technologies using training and test sets that were significantly larger than those used in the past - facilitating novel machine learning approaches and supporting statistically-informative evaluation results. Eventually this effort was merged with other video processing evaluations being implemented in Europe under the Classification of Events, Activities, and Relationships (CLEAR) Consortium. NIST’s goal is to evolve these evaluations of video processing technologies towards a focus on the detection of visually observable events and 3D modeling and to help the Computer Vision community make strides in the areas of accuracy, robustness, and efficiency.