A framework for scrutinizing baseball videos is proposed. By applying the well-defined baseball rules, this work exactly identifies what happens in a game rather than roughly finding some interesting parts. After extracting the information changes on the superimposed caption, a rule-based decision is applied to detect meaningful events. Only three types of information, including number of outs, number of scores, and base-occupation situation, have to be considered in the detection process. The experimental results show the effectiveness of this framework and demonstrate some research opportunities about generating semantic-level summary or indexing for sports videos.