Visual target tracking involves object localization in image sequences. This is achieved by optimizing image feature similarity based objective functions in object state space. Meta-heuristic algorithms have shown promising results in solving hard optimization problems where gradients are not available. This motivated us to use Firefly algorithms in visual object tracking. The object state is represented by its bounding box parameters and the target is modeled by its color distribution. This work has two significant contributions. First, we propose a hybrid firefly algorithm where genetic operations are performed using Real-coded Genetic Algorithm(RGA). Here, the crossover operation is modified by incorporating parent velocity information. Second, the firefly brightness is computed from both foreground and background information (as opposed to only foreground). This helps in handling scale implosion and explosion problems. The proposed approach is benchmarked on challenging sequences from VOT2014 dataset and is compared against other baseline trackers and metaheuristic algorithms.