expanAI: A Smart End-to-End Platform for the Development of AI Applications

Abstract

Building Modern Artificial Intelligence (AI) applications is a complicated process involving data preparation, model selection, and intensive training over large-scale data. It usually requires expertise in various domains, namely resource management, distribute storage, parallel computing, machine learning and deep learning. Acquiring all these skills for many small and medium companies to build an efficient AI application can be extremely hard. ExpanAI is proposed as a smart end-to-end platform for building efficient AI applications. ExpanAI provides a set of microservices to abstract away low-level implementation, like infrastructure and resource management, from the end users. Frequently used middleware, such as Spark, Kafka, Cassandra, etc., are first-class residences in the ExpanAI and are always available to users. Furthermore, ExpanAI introduces a smart interpreter to provide easy-to-use interface to execute data-intensive jobs. This interpreter automatically optimizes the execution plans according to the profile of the data and available resources. Lastly, a workflow optimization recommender is also proposed to conduct self-analysis over all jobs and automatically generates reports to suggest ways to improve performance or to avoid failures.