In this paper we describe several indoor scenarios for ambient intelligence settings with a data collection setup that uses two omni-directional cameras, and challenges posed by realistic illumination and occlusion settings. Single-person and multi-person activities are recorded and annotated. We describe baseline and feature-based tracking systems and comparatively analyse tracker performance in this setting. Our results show that omni-directional cameras have great potential for indoor settings.