Resource allocation is an important strategy to optimize system performance in multiuser cooperative orthogonal frequency-division multiple-access (OFDMA) systems. In this paper, we jointly consider three different transmission modes in cooperative OFDMA systems, i.e., direct transmission (DT) mode, half-duplex (HD) relay cooperative transmission (HDRCT) mode, and full-duplex (FD) relay transmission (FDRT) mode. The joint optimization problem of transmission mode selection, subcarrier assignment, relay selection, subcarrier pairing, and power allocation (PA) is investigated. In this paper, we transform the binary assignment problem into a maximum-weighted bipartite matching problem, which can be solved by the classical Hungarian method. Based on the dual method, we solve the joint PA and binary assignment problem iteratively. Specifically, since the direct link is considered interference in the FD relay transmission mode, the PA problem in FD relay transmission mode is nontrivial. Thus, we provide a novel hierarchical dual method to solve the PA problem in FD relay transmission mode. In addition, in HDRCT mode, the joint transmission of both the source and relay is taken into account, and we provide a simple and insightful PA scheme. Simulation results show that the proposed algorithms can significantly enhance the overall system throughput, compared with previous works.