Unmanned Aerial Vehicles (UAV) have been recently applied in several various civilian applications. Based on this, there is a growing need for intuitive UAV control interfaces. In this work, we report on the Body-Machine Interface (BMI), helping a human operator to control a quadrotor through the gesture commands. We perform the human motion capture through wearable sensors and Kalman filter to reduce the noise. For the gesture command recognition, we designed the Recurrent Neural Network recognizing gestures within 65 ms. For the quadrotor orientation estimation, we designed the Extended Kalman Filter (EKF). We assess the proposed BMI via the simulations and experiments: the standard deviation of the trajectories varies for up to 10 cm.