Faster and Cheaper Offline Batch Inference with Ray

The popularity of machine learning (ML) in the real world has exploded recently, with offline batch inference as one of the primary workloads. Yet existing production ML systems fall short for this workload in scale and simplicity. To address this, the Ray community has built Ray Data, an open-source library for building large-scale data processing for ML applications. In this talk we'll discuss:

• How to use Ray Data for efficient inference of Terabytes of data and a pretrained model

• Why traditional data processing tools are difficult, expensive, and inefficient, particularly for modern deep learning applications.

• How to easily leverage modern ML models, multiple times faster and cheaper than other common solutions (like Spark or SageMaker) using - Ray Data

• How and why offline inference can be useful with LLMs and when building LLM applications

• Demonstrate an e2e offline batch inference use case with Ray Data

Takeaways:

• Ray Data is the best solution for offline batch inference/processing, particularly when working with unstructured data and with deep learning models

• Discuss why having a performant batch inference solution is important for LLM workloads, and show how Ray Data can help

• Share user success stories of using Ray Data for batch inference