The proliferation of Unmanned Aerial Vehicles (UAVs) such as drones has caused serious security and privacy concerns in the recent past which calls for their efficient localization and tracking. Drone localization plays a pivotal role in defense operations and it is of prime importance for drone identification systems to reduce false alarm rates in cases of close resemblance to birds. Small target localization and tracking is highly challenging under unfavorable topographical conditions such as long-range target detection, uneven illumination, and weak background contrast. To overcome such limitations in localization of drones and birds, SETNET is proposed which is a sparse ensemble network of YOLOv5 models compressed using static pruning and quantization for achieving robust inference speed, hence making it deployable in real-time surveillance systems. A hyperparameter evolution-based genetic algorithm is adopted to obtain optimal model parameters and reduce generalization error. An ensemble of five different versions of YOLOv5 is constructed based on non-maximum suppression. An overall improvement in target localization and a five-fold improvement in inference speed is achieved with the sparse ensemble network. This is combined with a Contrastive Language Image Pre-training (CLIP)-based zero shot drone tracking algorithm that assigns a unique ID to drones spotted in video instances and helps track them using feature similarity.