A path along which the human knee joint moves can be estimated from real-time moving images or a sequence of static images. In case of many algorithms solving this problem, it is essential to locate the characteristic points (i.e., key-points) on each image and find the correspondence between them in the image sequence. In this paper we present an algorithm, which detects such key-points facilitating effective femur tracking in a sequence of X-ray images. We use a set of X-ray images manually labeled with the key-point positions, to train a Convolutional Neural Network (CNN) for the purposes of solving a regression task corresponding to finding key-point positions in previously unknown images. CNN hyperparameters such as number of convolutions and layers, learning rate, regularization parameters, and activation functions were optimized using a tree of Parzen estimators guiding the process of training multiple models. Results for models with the best mean-square estimation error computed for a validation set and lowest structural complexity are presented. Key-point positions predicted by the CNN are on par with human predictions, even though the actual key-point position is ambiguous in some cases. The feasibility of detected key-points for femur tracking has been verified by several case studies.