The driving behavior classification of road agents is important for driver assistance systems and self-driving cars. The driving safety can be improved by identifying the normal or dangerous behavior of a road agent. In this paper, we propose an end-to-end system for road agent classification. Multi-object tracking is first carried out using the images acquired from an onboard camera. Future trajectories of the targets with unique IDs are predicted and used for behavior classification. It is then followed by an overtaking evaluation based on the conservative or aggressive driving behavior of individual road agents. In the system, we design an efficient parallelization mechanism for the interdependence and sharing between modules. The experiment performed with real scene data has demonstrated the feasibility of the proposed method. Source code is available at https://github.com/leisurecodog/E2E-Behavior-Classification.