Wearable intelligent phonocardiogram (PCG) sensors provide a noninvasive method for long-term monitoring of cardiac status, which is crucial for the early detection of cardiovascular diseases (CVDs). As one of the key technologies for intelligent PCG sensors, PCG classification techniques based on computer audition (CA) have been widely leveraged in recent years, such as convolutional neural networks (CNNs), generative adversarial nets, and long short-term memory (LSTM). However, the limitation of these methods is that the models have a sizeable computational complexity, which is not suitable for wearable devices. To this end, we propose an end-to-end neural network for PCG classification with low-computational complexity [52.67k parameters and 1.59M floating point operations per second (FLOPs)]. We utilize two public datasets to test the model, and experimental results demonstrate that the proposed model achieves an accuracy of 93.1% in the 2016 PhysioNet/CinC Challenge 2016 dataset with considerable complexity reduction compared with the state-of-the-art works. Moreover, we design an energy-efficient wearable PCG sensor and deploy the proposed algorithms on it. The experimental results show that our proposed model consumes only 245.1 mW for PCG classification with an accuracy of 89.8% on test datasets. This means that the proposed model obtains excellent performance compared with previous work while consuming lower power, which is significant in practical application scenarios.