Augmentation of virtual agents in real crowd videos is an important task for different applications from design simulations of social environments to modeling abnormalities in crowd behavior. We propose a framework for this task, namely for augmenting virtual agents in real crowd videos. Our framework utilizes homography-based video stabilization, Dalal-Triggs detector [1] for pedestrian detection and state-based tracking algorithms to automatically locate the pedestrians in video frames and project them into our 3D simulated environment, where the navigable area of the simulated environment is available as a manually designed and located navigation mesh. We represent the real pedestrians in the video as simple three-dimensional (3D) models in our simulation environment. 3D models representing real, projected agents and the augmented virtual agents are simulated using local path planning coupled with a collision detection and avoidance algorithm, called Reciprocal Velocity Obstacles (RVO) [2]. The virtual agents augmented into the video move plausibly without colliding with static and dynamic obstacles, including other virtual agents and real pedestrians. We provide an extensive graphical user interface for controlling the virtual agents in the scene, including collision avoidance parameters, adjusting the camera in the scene and some standard video player options.