Knee arthroplasty (KA) has led to substantial clinical results through the advancement of surgical procedures. Hand frame extraction is one of the important and crucial tasks to analyze video images of Orthopaedic surgery (OS). Although a number of vision-based techniques have been used to interpret hand movements in the field of computer vision, proper hand frame extraction in OS video is still a challenging task in the surgical work atmosphere. Therefore, this paper proposes a hand frame extraction method in KA surgical video images. For each video frame, the proposed method automatically recognizes three classes, hand, no-hand, or non-surgical area, by using Resnet-50. The method was validated by using five videos of UKA (Unicompartmental knee arthroplasty), a type of KA. The experimental results demonstrated that the model achieved a recognition accuracy of 96%. Moreover, the model has shown significantly good performance in extracting hand class frames in the test dataset (Precision = 0.95, Recall = 0.98, and F1-score = 0.97).