Visual analytics systems can greatly help in the analysis of urban data allowing domain experts from academia and city governments to better understand cities, and thus enable better operations, informed planning and policies. Effectively designing these systems is challenging and requires bringing together methods from different domains. In this paper, we discuss the challenges involved in designing a visual analytics system to interactively explore large spatio-temporal data sets and give an overview of our research that combines visualization and data management to tackle these challenges.