In this article, we propose a secrecy-aware energy-efficient scheme for a two-tier heterogeneous network (HetNet), consisting of a sub-6 GHz macrocell and multiple millimeter wave (mmWave) picocells. Each picocell is assumed to have several users and an eavesdropper (Eve) which intercepts the signal of the picocell users. In the proposed scheme, firstly, to maximize the secrecy energy-efficiency (SEE) of picocell users, a joint optimization problem of power control, channel allocation, and beamforming is formulated by considering the minimum secrecy rate and signal-to-interference-plus-noise ratio (SINR) constraints. Due to the non-convex nature of the aforementioned optimization problem in a highly dynamic HetNet environment, we transform it into a reinforcement learning (RL) problem using the Markov decision process (MDP). Then, a multi-agent reinforcement learning (MARL) technique is used to obtain the maximum long-term reward. Moreover, we propose a multi-agent cooperative deep reinforcement learning (DRL) scheme known as SecBoost to solve the MDP with large number of action and state spaces. It uses the dueling and double-Q architecture of dueling double deep Q-network (D3QN) to optimize power control, channel allocation, and beamforming vectors to maximize the SEE of picocells. Also, prioritized experience replay is used to increase the sampling efficiency of SecBoost. The SEE performance of SecBoost is compared with MARL, multi-agent deep Q-network (MA-DQN), state-of-the-art joint beamforming based secrecy energy efficiency maximization (JBF-SEEM) scheme, and one-time pad based encrypted data transmission (O-EDT). Simulation results demonstrated that the proposed SecBoost scheme achieves 14.7%, 8.33%, 30%, and 69% better average SEE in comparison to MARL, MA-DQN, JBF-SEEM, and O-EDT schemes, respectively, which reveals its effectiveness in improving SEE of picocells.