There has been an increased trend in the use of Unmanned Aerial Vehicles (UAVs) providing essential assistance and enhanced situational awareness in various application domains. Considering the sensing and intelligent processing capabilities of UAV platforms, operational managers are able to make smart decisions by having a more complete picture in real-time about the state of the managed event. In this paper, a preliminary design of a multipurpose UAV based situational awareness platform is proposed by exploiting advanced on-board processing capabilities and efficient computer vision, image processing and machine learning techniques. The proposed design consists of aerial and terrestrial surveillance assets able to provide rapid and reliable real-time event information of areas of interests in emergency operations. Experimental results are provided using RGB and thermal video datasets and applying novel object detection and tracking algorithms. The results show the utility and the potentials of the proposed design, while future directions for extension and optimization of the multipurpose UAV based situational awareness system are presented.