Ball tracking is a key technology in processing and analyzing a ball game. Because of the complexity of visual scenes, a large number of objects are often selected as candidates for the ball, leading to incorrect identification, and conversely, the true position of the ball may sometimes be missed because of occlusion and blur, which can both be frequent and severe. Several tennis ball tracking algorithms have been reported in literature. In this paper, we propose a two-layered data association method to improve the robustness of tennis ball tracking. At the local layer, a shift token transfer method is proposed, based on shift window processing, to generate a set of short trajectories or “ trajectorylets .” At the global layer, a unique ball trajectory is obtained by applying a dynamic programming based splice method to a directed acyclic graph consisting of trajectorylets. We evaluated our approach on tennis matches from the Australian Open and the U.S. Open, and the results obtained show that our approach outperforms current state-of-the-art approach.