With the advancement of intelligent and networked technology, the Industrial Internet of Things (IIoT) faces an escalating threat from cyberattacks, especially by Advanced Persistent Threat (APT) attacks. These novel and complex attacks, characterized by their dynamic nature and life-long duration, pose significant challenges to existing security protection methods. The challenges are twofold, i.e., sparse reward problem in the long-lasting attack, and partial observation of attack actions. To this end, we propose a Security-as-a-Service based reinforcement learning method, namely Attention Augmented Dueling Deep Q-learning Network (AD2QN), to make real-time defense strategies for the hot standby IIoT. First, we build the attack-defend confrontation model as black boxes interact with the IIoT environment to play a long-lasting partially observable zero-sum stochastic game on the server. Then, to dynamically generate optimal defense strategies as the service, AD2QN is proposed employing information completion and prediction to more informed action selection. Furthermore, AD2QN utilizes an iteratively updated reward network to deal with the sparse reward problem. Extensive simulation results shown that the defense strategies generated by our method have a higher defense success rate and a stable defense performance with the average success rate of 0.7384, while the average success rate of baseline methods was 0.7375, in the best case.