Cellular-connected unmanned aerial vehicles (UAVs) is considered as integral components for sixth generation (6G) cellular networks. Cell-free radio access network (CF-RAN) with network-assisted fullduplex (NAFD) possesses global collaborative capabilities and enable uplink and downlink transmission simultaneously, which can be considered as a potential technology for supporting air-ground communication. In this paper, we investigate the performance of cellular-connected UAV in CF-RAN with NAFD. We propose a modified beamforming training scheme to mitigate cross-link interference (CLI) and a location-aware access point (AP) clustering strategy to reduce fronthaul overhead. We derive closed-form expressions for uplink and downlink achievable rates of GUEs and UAVs, respectively. Based on these expressions, we propose an efficient global spectral efficiency optimization scheme by solving a multi-objective optimization problem (MOOP) aiming to maximize the uplink sum rates and downlink sum rates simultaneously with deep Q-network (DQN). Numerical results verify the accuracy of the derived closed-form expressions. The effectiveness of the modified beamforming training scheme and location-aware AP clustering strategy are proved. In addition, the impact of system parameters and the advantages of NAFD system are analyzed. We also illustrate the convergence and benefits of DQN-based optimization scheme on different type of users.