Autonomous navigation of multiple unmanned aerial vehicles (UAVs) serves as the foundation for their widespread applications in various fields. However, multi-UAV cooperative navigation is a challenging problem because it is difficult for UAVs to plan efficient paths while avoiding collisions with both neighboring UAVs and other obstacles. In this work, an efficient reinforcement learning (RL)-based cooperative navigation algorithm (RL-CN) is proposed to make optimal decisions in dense and dynamic environments. In RL-CN, the multi-UAV cooperative navigation is formulated as a Markov decision process and an enhanced RL method is proposed for continuous control of multiple UAVs. To solve the sparse reward problem, a group of reward functions are designed in reward shaping. Next, a staged-tuning with two subactor networks strategy is developed to accelerate the training process and improve the navigation performance. Then, an enhanced prioritized experience replay strategy is designed by considering both Temporal Difference (TD) error and connectivity trend. Thus, high-value transitions are more frequently selected to further enhance the training process. Finally, we conduct comprehensive simulation experiments and provide comparative results to substantiate the effectiveness and robustness of RL-CN algorithm.