The problem of resource allocation in a high mobility network is always meaningful while challenging. Due to the mobility characteristic, the main difficulty lies in solving different optimization problems in a limited time, so traditional time-consuming optimization solvers are no longer applicable. This paper considers NGMA-enabled heterogeneous networks (HetNets) and uses end-to-end multi-agent reinforcement learning (MARL) to optimize the resource allocation problem. Even though MARL can use many cooperative agents to divide action space, it may lead to a significant increase in agent number, which is harmful to credit assignment. To tackle this problem, we employ a unique design that can fix the number of agents in any case. Agent credit assignment is then considered to guide agents to work cooperatively and ensure each efficient action gets a suitable reward. Also, a novel learning process named Learn to Improve is utilized to make our method more general. Numerical results and comparison experiments show the effectiveness and robustness of our methods.