The application of the cross entropy (CE) method to tone injection (TI) can make the peak-to-average power ratio (PAPR) of the orthogonal frequency division multiplexing (OFDM) signals reduced efficiently with great complexity savings. However, as a population-based search method, the CE method requires large samples to find a high-quality solution, which means higher complexity and more time consuming concerning convergence. To reduce the complexity while achieving promising PAPR performance, an improved CE based TI scheme is proposed in this paper. First, a structured extended constellation for rectangular quadrature amplitude modulation is designed to provide more freedom degrees. With proper parameter selection, the proposed extended constellation can offer more design flexibilities and achieve a better tradeoff between freedom degree and power increase. Then, by analyzing the impact of the subcarrier candidate population on the conventional CE based TI scheme, we derive an explicit expression for the determination of the size of the subcarrier candidate population. And, based on the prior information carried by the clipping signal, proper candidate subcarriers are selected. Finally, an improved CE based TI scheme is proposed, which can efficiently enhance the solution quality and greatly accelerate the convergence speed. Simulation results demonstrate that the improved CE based TI scheme achieves great improvements on the performance of PAPR, bit error rate, power increase and complexity compared with the conventional CE based TI scheme.