An Architecture and Methods for Big Data Analysis

Abstract

Data production has recently witnessed explosive growth, reaching an insurmountable amount (larger than 4 ZB in 2013). This includes data sources such as sensors used to gather climate information, reports on household parameters, posts to social media sites containing digital pictures and videos, purchase transaction records, and cell phone GPS signals, to name a few. Not yet having more than an intuitive and ad hoc definition, big data is challenging the IT infrastructure of companies and organizations, forcing them to look for viable solutions leading to data processing such that enterprises can deploy a better business strategy. In essence, big data implies collecting, extracting, transforming, transporting, loading (ETL), classifying, analyzing, interpreting, and visualizing, among many other operations, on large amounts of structured, semi-structured, and unstructured data, in the order of a few petabytes per day, executed and terminated in critical time. This paper will introduce the architecture and the corresponding functions of a platform and tools implementing part of these challenging operations, while others are being obtained via composing elementary operations. The architecture is built around a distributed network of virtual servers called “agents,” which can migrate around a network of hardware servers whenever available resources are provided or created. A control center makes decisions on moving the agents based on the availability of resources when needed. An example from the telecommunications industry will illustrate how the platform is applied to this domain of big data.