A commonly used assumption in the existing unmanned aerial vehicle (UAV) channel models is that the UAV flies along linear trajectories. However, this assumption does not always hold in realistic UAV communication scenarios. In this paper, we relax the linear trajectory restriction and present a temporally non-stationary channel model for UAV-to-ground communication scenario. The temporal non-stationarity of the channel stemming from time-varying heading directions of the UAV is properly modeled. This is achieved by combining a geometry-based stochastic model (GBSM) and an aeronautic random mobility model (RMM), which is introduced to characterize the movement pattern of UAVs. Based on UAV and ground station (GS) trajectories, time-varying model parameters and statistics including space-time-frequency correlation function (STF-CF), Doppler power spectrum density (PSD), delay PSD, and stationary interval are derived. The influences of the RMM-related parameters on the statistics are presented and analyzed. Some of those statistics are verified by measurements, showing the practicability of the model. The work presented in this paper is helpful in designing UAV communication system considering realistic UAV trajectories.