Multi-object tracking is a difficult vision task that is necessary for most real world vision applications. This task becomes almost indomitable if the objects collectively act to avoid being tracked. In this paper we present a solution strategy by utilising of two novel trackers, one based on a recurrent deep neural network and the other a one shot associative memory. We solve the problem at its highest difficulty level by incorporating a phenomenon seen in nature called the confusion effect used by swarming animals to evade predators. This behaviour has evolved to actively disrupt the predator's ability to accurately track targets, which makes it an extremely challenging testbed for computer vision. We use our findings to propose a strategy that takes advantage of both the robust tracking accuracy of recurrent neural networks and the rapid training of the one shot associative memory to predict the swarm's next moves.