Ray Data Streaming for Large-Scale ML Training and Inference

Some of the most demanding ML use cases involve pipelines that span both CPU and GPU devices in distributed environments. Most frequently, this situation occurs in batch inference, which involves a CPU-intensive preprocessing stage (e.g., video decoding or image resizing) before utilizing a GPU-intensive model to make predictions. It also occurs in distributed training, where similar CPU-heavy transformations are required to prepare or augment the dataset prior to GPU training. In this talk, we examine how Ray data streaming works and how to use it for your own machine learning pipelines to address these common workloads utilizing all your compute resources–CPUs and GPUs–at scale.

Takeaways

• Ray Data streaming is the new execution strategy for Ray Data in Ray 2.6

• Ray Data streaming scales data preprocessing for training and batch inference to heterogeneous CPU/GPU clusters