This paper proposes a new deep neural network-based approach of human activity recognition for virtual multi-sports. The proposed approach acquires initial motion data of sports balls, not full motion data of human activities, with a single high-speed camera. Then, a deep neural network model is trained to recognize corresponding human activities, such as baseball batting, soccer kicking, golf swing, and so on, by learning those initial motion data. The proposed approach is very efficient and effective for practical applications of virtual multi-sports. In practice, the proposed approach was successfully applied to a unified virtual multi-sport platform. The effectiveness and efficiency of proposed approach was verified with real experimental results.