A private cloud-based Datalab for scalable DSML pipelines

In an era where businesses continuously try to conquer new markets and at-tract new customers, Big Data usage is becoming a strategic enabler to sup-port such ambitions. The upward trend of enterprises offering omnichannel experiences to customers and prospects gives rise to vast amounts of non-traditional data sources, which need to be leveraged to turn them into action-able insights. Enterprises are becoming aware of these challenges and are trying to integrate their existing corporate data with non-traditional acquired Big Data to unlock meaningful use cases. The IT landscape has coincided with these trends, offering modern technology stacks in infrastructure, Big Data platforms, and artificial intelligence tools and frameworks. However, the path toward data analytics in the enterprise concerning business requirements, time to market, and industrialization constraints remain a real challenge. Implementing a data analytics use case efficiently requires heterogeneous steps ranging from data preparation, model training, validation, serving, and managing the product lifecycle to guarantee sustainability and ultimately reach the expected business outcomes. Therefore, the enterprises’ data ecosystem should support these steps by providing a suitable platform enabling data scientists’ collaboration with tools to unleash data analytics at scale. This paper addresses these needs and suggests a scalable cloud-ready architecture that fits into data science pipelines with all their steps, specificities, and requirements. The propositions made through this contribution are driven by business ambitions and confronted with a benchmark that presents the available solutions in the IT marketplace with careful consideration of their strengths and weaknesses. Finally, the suggested architecture is exposed and discussed with perspectives and future research areas.