This paper solves the containment problem of multi-agent systems on undirected graph with multiple active leaders using off-policy reinforcement learning (RL). The leaders are active in the sense that there exists bounded control input in the dynamics which is unknown to all followers and the followers are heterogeneous with different dynamics. Not only the steady states of agent i but also the transient trajectories are taken into account to impose optimality to the proposed containment control. Inhomogeneous algebraic Riccati equations (ARE) are derived to solve the optimal containment control protocol. To avoid the requirement of agents' dynamics to obtain containment control, an off-policy RL algorithm is developed to solve the inhomogeneous AREs online in real time and without requiring any knowledge of the agents' dynamics. Finally, a simulation example is presented to illustrate the effectiveness of the proposed algorithm.