A Scalable Heterogeneous Solution for Massive Data Collection and Database Loading

Abstract

Massive collection of data at high rates is critical for many industries. Typically, a massive stream of records is gathered from the business information network at a very high rate. Because of the complexity of the collection process, the classical database solution falls short. The high volume and rate of records involved requires a heterogeneous pipeline comprised of two major parts: a system that carries out massive collection and then uploads the information to a database, and a subsequent data analysis and management system consisting of an Extract Transform and Load component. We developed a massive collection and loading system, based on a highly scalable heterogeneous architecture solution. The solution has been applied successfully for Telco revenue assurance, and can be applied to other industrial areas. The solution was successful in scaling up a Telco client system to handle streams of records ten times larger than was previously possible.