In recent years, action recognition has been widely explored and attains significant performance improvement. In this paper, we propose a real-time action recognition specified convolutional 3D (AR-C3D) neural network for human-computer interaction. The CNN structure is optimized to decrease the complexity. Furthermore, Winograd algorithm is adopted to accelerate computation. It achieves 89.9% accuracy in the application which refers to the robot classifies the video captured by itself and would either imitate human's action or give verbal feedback. The Artix-7 FPGA implementation result outperforms previous work in terms of resource utilization and no external storage is consumed. One video can be processed in 6.6ms, and the power consumption is only 2.7W.