Many tasks have demand on precise predictions of agents or moving objects. Previous prediction methods usually only focus on the kinematic model of moving objects or the environment. However, the target tasks of agents may influence the prediction of agents in great sense, especially in tasks of confrontation. Therefore traditional methods cannot work well in such scenes. In this paper, we propose a heterogeneous graph neural network method to deal with the multi-agent trajectory prediction problem. Our method can aggregate and pass messages representing environment and also agents' tasks due to the elaborate design of graph neural network structure. We validate our method on the Robocup Small Size League simulation platform which focuses on multi-agent coordination and confrontation in the form of soccer games. After making our own ZJUNlictSSL dataset, we predict the position of all robots on the pitch of certain time gaps based on the limited information we get from vision. The results prove that our method is of high prediction accuracy and low prediction error compared with conventional kinematic motion methods. Code is available here.