Efficient sample generation for scalable meta learning | IEEE Conference Publication | IEEE Xplore

Efficient sample generation for scalable meta learning


Abstract:

Meta learning techniques such as cross-validation and ensemble learning are crucial for applying machine learning to real-world use cases. These techniques first generate...Show More

Abstract:

Meta learning techniques such as cross-validation and ensemble learning are crucial for applying machine learning to real-world use cases. These techniques first generate samples from input data, and then train and evaluate machine learning models on these samples. For meta learning on large datasets, the efficient generation of samples becomes problematic, especially when the data is stored distributed in a block-partitioned representation, and processed on a shared-nothing cluster. We present a novel, parallel algorithm for efficient sample generation from large, block-partitioned datasets in a shared-nothing architecture. This algorithm executes in a single pass over the data, and minimizes inter-machine communication. The algorithm supports a wide variety of sample generation techniques through an embedded user-defined sampling function. We illustrate how to implement distributed sample generation for popular meta learning techniques such as hold-out tests, k-fold cross-validation, and bagging, using our algorithm and present an experimental evaluation on datasets with billions of datapoints.
Date of Conference: 13-17 April 2015
Date Added to IEEE Xplore: 01 June 2015
Electronic ISBN:978-1-4799-7964-6

ISSN Information:

Conference Location: Seoul, Korea (South)

References

References is not available for this document.