An agent-based hardware-in-the-loop simulation framework is proposed to model the UAV/UGV surveillance and crowd control system. To this end, a planning and control system architecture is discussed first, which includes various modules such as sensory data collection, crowd detection, tracking, motion planning, control command generation, and control strategy evaluation. The modules that are highly related with agent-based modeling (focus of this paper) are then discussed, which includes the UAV/UGV motion planning considering multi-objectives, crowd motion modeling via social force model, and enhancement of simulation environment via GIS 3D coordinates conversion. In the experiment, Repast Simphony is used as the agent-based modeling tool, which transmits sensory data and control commands with QGroundControl as hardware interface that further conducts radio communications with ArduCopter as a real UAV. Preliminary results show that finer grid scale and larger vehicle detection range generate a better crowd coverage percentage. Finally, conclusions and future works are discussed.