The Internet of Things (IoT) continues to expand; as daily new smart-devices are connected to Internet and adding to a deluge of data created by our society. Compounding the challenges is that this data is very heterogeneous, including device or human activity trace data, structured data, sensor information, and media data. The computing needs for the IoT continue to rise in terms of scalable compute power, storage, and complex data processing pipelines to accommodate these diverse sources of data. For this reason, it becomes essential to develop a flexible framework that is able to efficiently manage the IoT data in a real-time and scalable approach. In this paper, we propose a novel framework to handle IoT data. Our framework is dynamically extensible, lightweight, resources efficient, and has the ability to handle stream data as well as batch data. We leverage autonomous agents along with the publish-subscribe pattern to achieve a run-time extensible, event-driven, and high-performance computational architecture. Additionally, we have incorporated localized and centralized databases into the framework to support structured and unstructured data for compute processing and analytical tasks. We have implemented the proposed framework and evaluated its performance using a visual object-detection case study on both a local cluster and within cloud-computing infrastructure. Our analysis shows that this framework utilizes the CPU, memory, and network resources efficiently. Additionally, the framework can scale horizontally as adding more processing nodes reduces the time and increases the goodput.