Computing the relative homography between the sports field template and the corresponding field in a video frame is an important task in camera calibration. In this paper, a fast and effective framework is proposed for addressing this task. The proposed framework has three processing modules. First, a semantic segmentation network is presented to obtain the segmented video frames. Second, a regression network is developed and combined with the direct linear transformation (DLT) algorithm to compute the homography. Third, the enhanced correlation coefficient (ECC) technique is leveraged to refine the estimated homography. The proposed framework is evaluated on 2014 World Cup dataset. The experimental results are compared to the state-of-the-art approaches. The experimental results demonstrate that the accuracy in the proposed framework is superior and the computation speed is competitive.