To navigate a complex urban environment, it is essential for autonomous vehicles to make educated assumptions and accurate predictions of the movement of other traffic agents. Beyond single object tracking, this task involves understanding behavior of other participants and predicting their trajectories. In this paper, we present a data-driven approach to learn the behavior of traffic agents at an intersection by observing several episodes of real-life scenarios captured through a static camera. We develop a feed-forward artificial neural network called the influence-network, which can simultaneously reason over the influence that agents and the environment have on each other. We compare it to an extension of popularly used Dynamic Bayesian Network. Based on data captured at a busy city intersection, we show that our model can predict trajectories of different classes of traffic agents with improved accuracy, and capture higher-level agent behavior.