Speech-to-text conversion has been extensively deployed for a variety of applications. To implement speech-to-text conversion on energy-constrained edge devices, a hybrid algorithm is adopted in this work. A bidirectional recurrent neural network (BRNN), composed of the light gated recurrent units (LiGRUs), is included to achieve a high speech-to-text accuracy with fewer network parameters. A network compression scheme, including scaling factor pruning (SFP), multi-bit clustering (MBC), and linear quantization (LQ), is proposed to minimize the complexity of the BRNN. The network size and the computational complexity are reduced by $29.8\times$ and $73.2\times$ , respectively, with only a 1% accuracy drop. An array of processing elements (PEs) are designed to perform BRNN inference. To minimize the system latency under the silicon area constraint, an optimal design with four PEs that can achieve 100% utilization for BRNN inference is selected. A memory-efficient decoding scheme without backtracking is utilized to reduce the memory usage by 21% compared to the direct-mapped implementation. The proposed speech-to-text accelerator achieves a phone error rate (PER) of 15.2% on the TIMIT with a network size of 1.51 MB. Fabricated in 28-nm CMOS, the chip consumes energy of 12.7 mJ/frame when operated at 1.25 MHz from a 0.6-V supply. Compared to the state-of-the-art designs, this work achieves a 6.5– $177\times$ lower normalized energy and a 37.5– $50\times$ lower attainable latency, with a 3.3%–8.5% lower PER.