Building Production AI Applications with Ray Serve

Productionizing modern machine learning workloads is challenging. Not only do you need to train and optimize your models, but also find a way to serve them efficiently without too much operational cost. Ray Serve solves these complex requirements to enable you to go to production safely and at low cost: you can flexibly scale and coordinate multiple models, deploy and upgrade safely, and maximize your hardware utilization with minimal management overhead.

This talk will demonstrate Ray Serve’s production-ready capabilities, including a demo of serving an ML-powered application using Ray Serve on the Anyscale platform. Some highlights include improvements around scalability, high availability, fault tolerance, and observability.

Takeaways:

• Learn about patterns of production ML serving and how Ray Serve is tailored to solve them.

• Hear how users in the community are using Ray Serve in production to lower their ML inference costs.

• Watch a real time demo of how to serve an ML application using Ray Serve on the Anyscale platform. This will highlight recent improvements around observability, autoscaling, and cost savings.