Strategies & Principles for Distributed Machine Learning
Abstract
The rise of Big Data has led to new demands for Machine Learning (ML) systems to learn complex models with millions to billions of parameters that promise adequate capacity to digest massive datasets
and offer powerful predictive analytics (such as high-dimensional latent features, intermediate representations, and decision functions) thereupon.
In order to run ML algorithms at such scales, on a distributed cluster with 10s to 1000s of machines, it is often the case that significant engineering efforts are required ---
and one might fairly ask if such engineering truly falls within the domain of ML research or not. Taking the view that Big ML systems can indeed benefit greatly from ML-rooted statistical and algorithmic insights ---
and that ML researchers should therefore not shy away from such systems design ---
we discuss a series of principles and strategies distilled from our resent effort on industrial-scale ML solutions that involve a continuum from application, to engineering,
and to theoretical research and development of Big ML system and architecture, on how to make them efficient, general, and with convergence and scaling guarantees.
These principles concern four key questions which traditionally receive little attention in ML research
: How to distribute an ML program over a cluster? How to bridge ML computation with inter-machine communication?
How to perform such communication?
What should be communicated between machines? By exposing underlying statistical and algorithmic characteristics unique to ML programs but not typical in traditional computer programs,
and by dissecting successful cases of how we harness these principles to design both high-performance distributed ML software and general-purpose ML framework,
we present opportunities for ML researchers and practitioners to further shape and grow the area that lies between ML and systems.
