In this era of Internet of Everything (IoE) and rapid development of mobile wireless communication technology, time delay and spectrum efficiency are difficult to achieve in the development process. Fog computing (FC) is a kind of edge computing, which is more suitable for the Internet of Things (IoT) with high dense connectivity. Non-orthogonal multiple access (NOMA), as a promising multiple access technology combined with FC, is considered here. In this paper, we study how to adopt deep reinforcement learning (DRL) algorithms to optimize energy efficiency (EE) in the multi-server and multi-user scenario of downlink fog radio access network (F-RAN) based on NOMA. In order to solve this nonconvex problem, it is divided into two subproblems: subchannel allocation and power allocation. The former adopts deep Q network (DQN) and the latter adopts deep deterministic policy gradient (DDPG) to obtain the best allocation strategy. DRL has high applicability in dealing with high-dimensional data, so it can be well applied in dynamic communication environment. The simulation results show that the combination scheme of “DQN-DDPG” has achieved remarkable advantages in terms of the faster convergence speed and the better final result of the system EE than other schemes.