Many traffic accidents are caused by distracted drivers. Drivers often use mobile phones and navigation when driving, which puts them at risk. Early warning for distracted drivers can reduce traffic accidents. To improve the recognition accuracy of distracted driving behaviors and reduce the scale of deep learning models, we propose a driver behavior recognition model called ENet-CBAM based on EfficientNet and Convolutional Block Attention Module(CBAM). In this paper, our proposed ENet-CBAM has only 4.0 M parameters and achieves 98.86% accuracy on our mixed dataset. We conduct experiments on different Convolutional Neural Networks (CNNs) and conclude that our architecture outperforms early methods. Ablation experiments on the attention module are conducted to show that the attention module really works in different model. We analyze the classification accuray by embedding the attention module at different layers of our proposed CNN. The results show that the attention module works at the early layers of a CNN. Finally, we conclude that Our proposed CNN is able to detect distracted driving behaviors with good accuracy and few parameters.