As the requirements of wireless communication networks change, the space-air-ground integrated network (SAGIN) architecture will be the primary goal of future communication networks. However, due to the pressure of user devices (UDs) to process compute-intensive tasks and the limited power of aircraft, this remains a challenge of reducing energy consumption in SAGIN system. Multi-access edge computing (MEC) provides a promising solution by offloading tasks to edge nodes with more computing resources. In this paper, we study a multi-node cooperative task offloading problem, where user devices generate computing tasks that can be processed locally, offloaded to multiple unmanned aerial vehicles (UAVs) or a satellite. The aim is to minimize the total energy consumption of the system while maintaining the task latency constraints by allocating the optimal mmWave band bandwidth, and jointly optimizing UAV trajectories, offloading decisions making, and computing resource allocation. Since the proposed problem optimization is NP-hard, the proposed Deep Reinforcement learning based Cooperative Task Offloading (DRCTO) algorithm, which leverage the proximal policy optimization method, can accelerate the learning process of searching for the optimal solution. Numerical results indicate that the proposed DRCTO algorithm performs better than other comparable algorithms and significantly reduces the total system energy consumption.