In the process of downsampling, the detection of small infrared targets encounters problems such as imaging area, missing texture features, and disappearance of targets. This study proposes a local contrast attention guide network (LCAGNet), which implements and uses an SPD-CSP Resblock (i.e., CSPResblock based on subpixel DownSample), an MALC (i.e., multiple attention local contrast) module, and MCAF (i.e., multiscale cooperative attention fusion) module, as well as e-maxmin (i.e., natural logarithm normalization) loss function for reconstructing images. First, by segmenting the gradient flow, the SPD-CSP Resblock propagates it in different paths. Second, under the guidance of the MALC module, the accuracy and convergence speed of the network are improved by constructing local nonlinear feature layers. Third, the weighted low-level detail feature channel is integrated with the deep semantic feature by MCAF to enhance the semantic information of the target. By standardizing the NL (i.e., natural logarithm) probability of the network output, the loss of target shape and position could be calculated accurately. Also, in the upsampling stage, the subpixel conv method reconstructs features with high resolution and further boosts detection accuracy. A series of detailed ablation experiments were conducted on each module in the network architecture to verify the effectiveness of the module and compare LCAGNet with other advanced data- and model-driven methods. The experimental results showed that LCAGNet achieved better results than other advanced benchmarking methods on two datasets of infrared small-dim target detection and tracking in both single-frame infrared small target benchmark (SIRST) and ground-sky background (KBT).