This paper presents an edge-based semantic classification of sports video sequences. The paper presents an algorithm for edge detection, and illustrates the usage of edges for semantic analysis of video content. We first propose an algorithm for detecting edges within video frames directly on the MPEG format without a decompression process. The algorithm is based on a spatial-domain synthetic edge model, which is defined using interrelationship of two DCT edge features: horizontal and vertical. We then use a multi-step approach to classify video sequences into meaningful semantic segments such as "goal", "foul", and "crowd" in basketball games using the "edgeness" criteria. We then show how an audio feature ("whistles") can be used as a filter to enhance edge-based semantic classification.