The growing popularity of 4 G/5 G mobile devices has led to an increase in demand for wireless security. Radio frequency fingerprint (RFF) technique is an emerging approach for device authentication using intrinsic and unique hardware impairments. In this paper, we propose an RFF-based method to identify rogue/unknown long term evolution (LTE) terminals. This is achieved by combining wavelet transform (WT) coefficient graphs and differential spectrum. The proposed method involves extracting 48 levels of wavelet coefficients from the transient power-off of the physical random access channel (PRACH) signal and representing them in a WT graph. The steady-state part of the PRACH signal after a frequency domain differential processing between the adjacent spectrum is extracted. To detect unknown attack devices, an identification scheme based on an autoencoder (AE) is designed. Two different AE network structures are designed based on the proposed features, and a hybrid identification structure is proposed. An experimental evaluation system is set up with seven mobile phones from three categories and one universal software radio peripheral (USRP) software-defined radio (SDR) platform. Training and testing datasets are collected under different conditions such as location, working times, and dates. Experimental results show that rogue devices can be identified with an accuracy up to 98.84% for different categories and 90.27% for different individuals.