This article presents a silent speech recognition approach based on high-density (HD) surface electromyogram (sEMG) using hybrid neural networks that support anomaly detection. In the hybrid networks, both a convolutional long short-term memory module and an autoencoder module were designed to extract discriminative spatio-temporal features and potentially identify any anomaly patterns, respectively. To verify the effectiveness of the proposed method, experimental data were recorded using HD-sEMG arrays with 64 channels from 11 subjects subvocalizing 33 Chinese words and articulating 9 anomaly patterns. The proposed method significantly outperformed other comparison methods (p < 0.05) and achieved the highest anomaly detection rate of 90.61% while maintaining a high level of target word-pattern classification accuracy of 82.30%. These findings demonstrate the effectiveness of the proposed method for improving the robustness of the SSR approach based on HD-sEMG recordings against anomaly muscular activities. This article also provides a novel solution for building practical and robust sEMG-based SSR systems with broad applications, such as instant messaging and human-computer interaction.