As an emerging paradigm for the fifth generation mobile communication system, cloud radio access networks (C-RANs) with millimeter-wave (mmWave) wireless fronthauls have been proved with a strong capability in reducing both capital expenses and operating expenses, as well as providing high energy efficiency (EE). In this paper, we investigate the energy-efficient downlink resource allocation in an OFDMA-based C-RAN with TDMA-based mmWave fronthauls. We first formulate the energy efficiency maximization problem as a mixed-integer nonlinear fractional programming (MINLFP) problem with the constraint of mmWave fronthaul, and then transform the MINLFP problem into a subtractive optimization problem by means of the Dinkelbach method. Then, we work out the transformed problem using the Lagrangian dual decomposition method. Finally, we evaluate the EE performance of our proposed algorithm by comparing to several benchmark schemes, and it shows that our proposed algorithm can achieve great EE performance gain over the benchmark schemes.