To address the tracking challenges such as weak feature expression ability of small targets and susceptibility to interference by similar objects in complex backgrounds, we use the principle of feature enhancement in the field of small target detection to redesign the backbone network of the siamese network tracker and propose a small target tracking algorithm based on a bidirectional feature pyramid fusion framework. The algorithm first constructs a deep feature pyramid with semantic and contextual information in the second half of the backbone network using successive deconvolution; then, considering that tracking also has a strong dependence on shallow information, we construct a shallow feature pyramid with location and spatial structure information in the first half of the backbone network as a complement to the deep feature pyramid, which is used to repair small target internal structure details and enhance its localization ability. Finally, the features processed from the deep and shallow pyramids are merged to construct a bidirectional pyramid fusion framework, while a self-attention mechanism is introduced to treat each type of information after fusion in a targeted manner, and the asymmetric convolution is used to lighten the fusion framework. The algorithm in this paper achieves a more advanced performance compared with existing algorithms for experiments on four publicly available datasets, GOT-10k, LaSOT, UAV123, and DTB70.