Entity Matching from Unstructured and Dissimilar Data Collections: Semantic and Content Distribution Approach

This paper describes a solution to the problem of extracting data features from a collection of dissimilar, unstructured data sets, gathered from multiple data sources in the web or databases. In this work we present a method of feature extraction and normalization, aiming at closing the gap between a workable data set of uniform content, and a large collection of unstructured and un-normalized collection of unworkable data set. The feature extraction we modeled creates focused, structured data sets as output, and with Big-Data and Analytics perspective. The solution we present automates data ingestion from public data sources and it applies Machine Learning methodology to build data relationships across unstructured data sets. Our research is aiming at extracting key features by using semi-supervised process, semantic relations, and statistical analysis of the distribution of content. The mapping across dissimilar datasets is solved through matching problem of these metrics, constructing a scoring value that maps different entities. We proposed a three-layer matching process of homogenous covariates from different sources semantic and measures are nonstandard using pattern recognition. This work presents a novel way to tackle the entity resolution problem. The result shows that the method works well on real industrial data and provides immediate ROI value for the data management system.