Sensibility Testbed is a framework for developing sensor-based applications that can run on user-provided smart-phones, and is easy to program. Over the past four years, we have been organizing hackathons at SAS in order to perform semi-controlled experiments with this platform. Any smartphone user can install Sensibility Testbed and develop a simple sensor application in less than a day. One of the problems with developing and testing such a framework is that there are many possible hardware platforms and system configurations. Hackathons provide an effective venue for observing the development of applications on a range of devices by users with no previous knowledge of the framework. In this paper, we describe our experiences with hosting hackathons in a variety of venues, including the challenges of working in unfamiliar environments and with researchers who had no prior knowledge of the testbed. The feedback from participants has been very useful in identifying usability issues, hardware issues, and the types of sensor applications that users want to create.