Data Quality (Poor Quality Data: The Fly in the Data Analytics Ointment)

Introduction and Summary

In recent years a powerful combination of database technologies, data mining techniques (see Data Mining) and analytics software have created vast new opportunities for data analysts and statisticians. For example, corporations have duly stored the results of their customer transactions in corporate databases for over a generation. There are, quite literally millions of records. Massively parallel engines can examine these data in heretofore unimagined ways. The potentials to understand customer profitability, develop better understandings of customers’ past needs and predict future ones, and to use those insights to develop new product niches are enormous.

Yet all is not well in the world of data analytics. Unlocking the mysteries data have to offer is difficult at best. And putting the discoveries to work can be even harder. One major reason is poor quality data. Bad data camouflage the hidden nuggets in data or, worse, send an analysis in the wrong direction...