Ball locations over frames facilitate tennis video analysis to a great extent. But so far no algorithm is able to obtain satisfactory result in locating the ball in broadcast tennis video (BTV). This paper presents a trajectory-based algorithm to detect and track the ball in BTV. Unlike the object-based algorithm, it does not decide whether an object is the ball. Instead it decides whether a candidate trajectory is a ball trajectory. This algorithm is able to obtain ball locations for most frames in a BTV, making use of four cues, namely, (1) an antimodel method to produce ball candidates from each frame, (2) a trajectory-based scheme to generate, identify and extend the ball trajectories from a set of candidates, (3) a method to infer the ball locations according to players' locations and the points of hitting, (4) a method to estimate missing ball locations from known ball locations. The experimental results show that our algorithm obtains the ball locations for above 96% frames in a sufficient accuracy for summarization.