In a sport video, the complex and dynamic background and the complexity of player actions bring considerable difficulty to analyze the player's action. In this paper we propose a method to represent the player's action on the panoramic background in a whole. It is helpful for watching and analyzing the player's action. We first build the panoramic background by the inter-frame relationship. Then object segmentation is implemented by subtraction between the panoramic background and each warped frame. Finally, the player's action in each frame is represented on the panoramic background. And the trajectory of the player action is presented on the background. The experimental results show that the proposed method is effective for the sports video like diving, jumps, which the player performs his or her action in a large arena, and the camera motion mainly includes horizontal and vertical direction.