In-network caching is frequently used to reduce unnecessary network traffic and lower server workload by allowing content to be accessed from the caching nodes in both wired and wireless networks. The drastic increasing mobile data traffic demands have conceived cooperative content sharing between nodes for addressing the storage limitation of a single node. In this paper, we explore the problem of minimizing transmission cost among cooperative nodes by jointly optimizing caching and routing in a hybrid network with vital support of service differentiation. We show that the optimal routing policy is a route-to-least cost-cache (RLC) policy for fixed caching policy. We formulate the cooperative caching problem as a Markov decision process (MDP) with the goal of maximizing the long-term average caching reward, which is NP-hard even when assuming users demands are perfectly known. Our study then proposes C-MAAC, a collaborative multi-agent deep reinforcement learning (MADRL) algorithm employing actor-critic learning model. C-MAAC has a key characteristic of centralized training and decentralized execution, with which challenge from the unstable training process caused by simultaneous decision of all agents can be addressed. After completing the offline training process, each node independently makes caching decision online during the execution process. Simulation results demonstrate the effectiveness of our proposed algorithms under dynamic environment where user request traffic change rapidly.