Energy efficiency (EE) is essential for uplink communication in massive machine-type communication (mMTC) since Internet of Things (IoT) devices are always energy-limited. This article refers to a relay-assisted uplink mMTC system that uses the non-orthogonal multiple access (NOMA) and orthogonal multiple access (OMA) hybrid resource allocation strategy that aims to enhance EE and access rate. In this way, some two-hop uplink relays based on amplifying and forwarding (AF) are deployed to connect the base station (BS) and IoT device groups. Each relayed group contains several adjacent IoT devices with poor channel conditions and uses the NOMA scheme. Firstly, a clustering algorithm named A-CFSFDP is proposed to efficiently and quickly divide IoT devices into different NOMA groups. Secondly, this article presents a hypergraph-based joint EE optimization model of the channel assignment, power allocation, and relay selection. The 3D channel matching among the NOMA IoT devices group, relay IoT devices, and ordinary OMA IoT devices is formulated as a mixed integer programming (MIP) problem. The acquirement of feasible EE weights in the problem is treated as an oligopolistic competition of power and modeled and solved as a Cournot game. Then, the largest independent set and the hypergraph-based method (HGM) algorithm solves the MIP problem. The numerical results show: 1) the designed A-CFSFDP clustering algorithm can more accurately distinguish NOMA-group halos and NOMA-group cores compared to the CFSFDP one; 2) the total EE achieved by the proposed hypergraph-based 3D channel matching is higher than the iterative Hungarian method (IHM) and minimum zero-surface preferred allocation (MZPA) method, respectively; and 3) the computational complexity of the HGM algorithm is also lower than that of the IHM and MZPA method by determining the largest independent set.