To meet industrial demand, plenty of research works have been dedicated to monitoring the health status of planetary gearboxes. For the same purpose, a new path is explored based on rotating encoder signal (RES) by considering time-varying speed condition. In proposed method, generalized demodulation is firstly applied on second-order derivative of RES to eliminate the frequency variability caused by time-varying speed, and a combined comb filter is performed in this process to improve the signal-to-noise ratio of demodulated signal. The toothwise meshing information is then integrated as individual samples for the detection of local anomalies induced by gear defect. After reducing the dimensionality of these toothwise samples by reordered kernel principal component analysis, a normalized robust statistical distance is constructed to indicate the anomalies contained in RES. Simulation and experimental cases are provided to reveal the efficacy of the proposed method.