Distributed Artificial Intelligent Model Training and Evaluation

Abstract

Machine Learning (ML) and in particular Neural Networks (NN) are currently being used for different image/video processing, speech recognition and other tasks. The goal of supervised NN is to classify raw input data according to the patterns learned from an input training set. Training and validation of NN is very computationally intensive. In this paper we present an NN infrastructure to accelerate model training, specifically tuning of hyper-parameters, and model inference or prediction using distributed systems techniques. By accelerating model training, we give ability to researchers to obtain a large set of potential models to use and compare in a shorter amount of time. Automating this process not only reduces development time but will provide an easy means for comparing results for different classifiers and/or different hyper-parameters. With a single set of training data, our application will run different classifiers on different servers each running models with tweaked hyper-parameters. To give more control over the automation process the degree by which these hyper-parameters will be tweaked can be set by the user prior to running. The prediction step in most ML algorithms can also be very slow, especially on video prediction where current systems calculate their inference predictions on an entire input video, and then evaluate accuracy based on human annotations of objects of interest within the video. To reduce this bottleneck, we also accelerate and distribute this important part of the ML algorithm development. This process involves sending to each server the data; the model weights; and human annotations within the video segmentation. Our efficient distribution of input frames among each node greatly reduced the amount taken for testing and to generate accuracy metrics. To make our implementation robust to common distributed system failures (servers going down, lost of communication among some nodes, and others) we use heartbeat/gossip style protocol for failure detection and recovery. We tested our infrastructure for fast testing and inference of ML on video with data generated by a group of marine biologists researching the behavior of different marine species on the deep sea. Results show that by using our infrastructure times improved by a factor of 15.