A Research Study on Running Machine Learning Algorithms on Big Data with Spark

Abstract

The design and implementation of proactive fault diagnosis systems concerning the bearings during their manufacturing process requires the selection of robust representation learning techniques, which belong to the broader scope of the machine learning techniques. Particular systems, such as those that are based on machine learning libraries like Scikit-learn, favor the actual processing of the data, while essentially disregarding relevant computational parameters, such as the speed of the data processing, or the consideration of scalability as an important design and implementation feature. This paper describes an integrated machine learning-based data analytics system, which processes the large amounts of data that are generated by the bearings manufacturing processes using a multinode cluster infrastructure. The data analytics system uses an optimally configured and deployed Spark environment. The proposed data analytics system is thoroughly assessed using a large dataset that stores real manufacturing data, which is generated by the respective bearings manufacturing processes. The performance assessment demonstrates that the described approach ensures the timely and scalable processing of the data. This achievement is relevant, as it exceeds the processing capabilities of significant existing data analytics systems.