Recently, neural-network-based approaches for spectral analysis have been proposed and are proven to be successful. However, since neural networks require huge computational as well as storage resources, the application of neural networks may face latency, memory footprint, storage size and power consumption issues, especially on low-power edge devices. In this paper, we propose a first hybrid approach in spectral analysis to optimize and accelerate the neural network execution while consistently preserving the final performance. First, feature selection is performed to reduce the data dimension and guarantee an efficient input throughput. Then, after the neural network training, we further prune the network to obtain a more compact network architecture. Finally, the network will be quantized to reach a higher compression ratio with low-cost operations for edge devices. We conduct extensive experiments on various target hardware platforms to demonstrate the effectiveness of our method, and results show that our approach can achieve up to 52x mode size compression and 600x speedup with even better performance in most cases. As a representative example, we successfully deploy a DenseNet with only a 0.1 MB model size and 0.9 ms inference time on a Raspberry Pi, which enables real-time on-site spectral analysis for industrial and commercial applications.