Developing fault diagnosis algorithms presents a significant challenge for rotary bearing signals with composite defects due to their inherent characteristics of nonlinearity, time variability, instability, and uncertainty. Hence, this article proposes a novel diagnostic architecture, IRTCog, based on variable visual-angle infrared thermography (V-IRT) images and an asymmetric convolutional neural network (ACNN), so as to overcome the insufficient samples, excessive parameters, and overfitting. V-IRT images that adequately characterize composite defects are considered for model training. Besides, hybrid activation functions and asymmetric convolution processes are designed to improve the accuracy and efficiency of the diagnostic model without increasing the parameter count. Finally, transfer learning is introduced to reduce model dependence on sample size and training time. The experimental results demonstrate that the proposed method reduces the training time by 72.9% and the diagnosis accuracy is close to 99%, indicating its superiority compared with other mainstream models.