Nonorthogonal multiple access (NOMA) technology shows the potential for improving spectral efficiency and enables massive connectivity in future wireless networks. Unlike orthogonal schemes that require separate resources for each user, NOMA allows multiple users to share the same frequency and time resource. However, joint cell association, subchannel assignment, and power allocation problem in uplink multi-cell NOMA systems is NP-hard to solve, posing a significant challenge. In this paper, we formulate this joint problem to maximize energy efficiency and propose a multi-agent deep reinforcement learning-based approach as a solution. In this approach, we adopt the multi-agent twin delayed deep deterministic algorithm (MATD3) for the power allocation and deep Q network for the cell association and subchannel assignment. Simulation results demonstrate that the proposed approach improves the energy efficiency performance of the uplink multi-cell NOMA system and outperforms other methods.