The goal of this work is to propose a hardware-in-the-loop, human-in-the-loop agent-based simulation which incorporates the human crowd characteristics and behaviors captured by computer vision techniques, for an effective crowd control using unmanned vehicles (UVs). Three major functions needed in our autonomous surveillance system include: 1) detection, 2) modeling, and 3) tracking. The proposed simulation communicates with the crowd detection module in the UVs' onboard computer in real-time, developing plans for a number of simulated crowd-individuals based on the parameters extracted from real crowds. Next, the social-force-based crowd modeling is used in the simulation to interpolate waypoints for moving the simulated individuals to their planned destinations. Finally, these waypoints are sent to the tracking module for a more realistic prediction of crowd's future location for the UVs' path planning purposes. Preliminary results reveal significant improvements in performance measures for this human-inthe- loop simulation, which demonstrate the effectiveness of the proposed methodology.