Implementation of Neural Network Regression Model for Faster Redshift Analysis on Cloud-Based Spark Platform

Abstract

Since observational astronomy has turned into data-driven astronomy recently, analyzing this huge data effectively to extract useful information is becoming an important and essential task day by day. In this paper, we developed a neural network model to analyze redshift data of million of extragalactic objects. In order to do that, two different approaches for faster training of neural networks have been proposed. The first approach deals with the training model using Lipschitz-based adaptive learning rate in a single node/machine whereas the second approach discusses processing astronomy data in a multinode clustered environment. This approach can scale up to accommodate multiple nodes when necessary to handle bulk data using Apache spark and Elephas. Additionally, this paper also addresses the scalability and storage issue by implementing the model on the cloud. We used the distributed processing capability of the spark that reads data directly from HDFS (Hadoop Distributed File System) of multiple machines and our experimental results show that using these approaches we can reduce training time and CPU time tremendously which is a crucial requirement while dealing with the extensive dataset. Although we have tested our experiment on a subset of huge data it can be scaled to process data of any size as well without much hurdle.