The identification and prioritization of immune-regulatory cancer driver mutations present a promising study for precision immunotherapy of cancer but remain considerable challenges. Here we introduced a novel method ImmRegInformer to systematically explore the regulatory relationship between cancer driver mutations and immune response by leveraging the powerful Transformer model and the lasso-regularised ordinal regression. In particular, our method integrated the mutation co-occurrence information with the self-attention weight to discern the underlying relationships between different driver mutations when regulating the immune cytolytic activity (CYT). Using ImmRegInformer, we identified 250 immune-regulating driver mutations in 8223 pan-cancer samples. They were verified in terms of the mutation frequency and interactions with the cytolytic signature genes. Further, we found the complementary roles of self-attention weight and mutation co-occurrence in prioritizing the driver mutations exhibited dominant associations with CYT. In conclusion, this study underscored the importance of employing deep learning methods like Transformer to unlock hidden insights into the biological complexities of cancer immunity, offering a new avenue in dissecting the immune regulatory mechanism and potential clinical applications. ImmRegInformer is freely available at https://github.com/Liwen-Liberty/ImmregInformer.