Registration opens at 8:30 AM

Breakfast and Networking

Co-founder and CEO

Executive Chairman and President, Anyscale

VP of Engineering, Head of Data and AI

Co-founder

Head of Product

VP of Enterprise Products

Head of Cloud Infrastructure and Platform Engineering

Keynote, Day 1

Lunch and Networking Break

Rayground Open/Expo 

We live in an era that's set to change the world in ways we haven't witnessed yet. While GenAI excitement and momentum is high, Enterprises are grappling with challenges around data pre-processing complexities, never ending AI model selection debates, dilemma over leveraging APIs that carry confidential queries outside of their security perimeter and many more. This session will shed light on practical concerns we have encountered when dealing with Partners and Customers and our approach to successful GenAI implementations at scale

Director of AI Strategic Engagements

GenAI for Enterprises—Practical Considerations From Pilot to Scale

Attend this session to learn about AWS purpose built ML accelerators including AWS Trainium and AWS Inferentia. These accelerators deliver a high performance, low-cost solution for Generative AI applications in the cloud. You will also learn about the new native support for these accelerators available in Ray.

Sr Manager

Running ML Workloads with AWS Purpose Build ML Accelerators and Ray

TPUs (Tensor Processing Units) are Google's specialized chips for machine learning workloads which are available for access through GCP (Google Cloud Platform). In this discussion, we'll be providing an overview of how to develop applications using Ray on Cloud TPUs. Attendees should expect to gain an understanding of how to develop performant, large scale ML applications and increase their ML productivity using Ray on Cloud TPUs.

Software Engineer

Group Product Manager

Developing Ray Applications on Google Cloud TPUs

Attend this session to learn how VMware can provide dramatic performance improvements in certain steps for Ray deployments on-premise, and how that aligns to general data privacy, control, and choice of AI services in the enterprise.

Vice President, VMware AI Labs

Standing Up Ray Clusters in the Blink of an AI to Drive Private AI

At Instacart, we are building our next generation ML Training Platform on Ray. We build standard training runtime for broad ML use cases across Instacart, and at the same time provide capabilities for more advanced users to fine tune their distributed deep learning model built by Ray AIR. We'll demonstrate the end to end system built on top of KubeRay for workflow orchestration, as well as the standard runtime deeply integrated with Ray Data, Ray Train and Ray Tune to utilize distributed computation capabilities across all training lifecycles. We'll also demonstrate a more custom use case, Instacart personalization ranker, which is also built on Ray AIR but with more complex data preprocessing steps. We'll tell the story of how we fine tune p13n ranker model to achieve higher training throughput and other optimizations brought by streaming executor etc.

Senior Machine Learning Engineer

Engineering Leader

Build Instacart Training Platform on Ray

In this talk we will share how we use Ray to address various challenges Samsara faced building its machine learning platform from scratch, specifically, creating a unify but flexible development experience for a team that needs to build AI in both cloud and edge. We will deep dive on how Ray and its ecosystem (Spark on Ray, Dagster etc.) enables ML research and development on a non-traditional setup, which improves data exploration and model training experience. We will also share our experience on managing our Ray infrastructure and some of the best practices we learn along the way.

Staff Machine Learning Engineer

Sr Machine Learning Engineer

Senior Applied Scientist

Building Samsara's Machine Learning Platform with Ray

The needs of AI developers at LinkedIn go far beyond simple serving of models. In this talk, we will share our journey to serving complex real-world Machine Learning (ML) workflows. We will talk about how we leverage Ray serving now, and in the future. We will also discuss the solutions we are developing to address the growing complexity of AI-based applications and to make the work of AI engineers more productive.

Sr. Staff Engineer

Inference Graphs at Linkedin Using Ray-Serve

At DoorDash, machine learning is a key component, used to enhance the experience of merchants, dashers, and customers. As our machine learning use cases keep growing, our forecasting and training pipelines are faced with several challenges like scalability, growing costs, reduced user development velocity and lack of proper debugging/observability. Driven by these challenges, we started learning about Ray and implemented a POC to verify the feasibility of incorporating Ray into our existing ML Platform architecture. In this talk, we'll share our journey towards adopting Ray, working with the open source KubeRay, how we built our POC and benchmarking setup and also share what the future of ML Platform at DD looks like with Ray being a core component.

Staff Engineer

From Good to Great: Elevating DoorDash Model Training with Ray

How can companies best build useful and differentiated applications on top of language models? Many of the products and companies built do this by providing the relevant context to LLMs and asking it to reason appropriately. In this talk, Harrison will discuss the different types of context you should be aware of, the different levels of cognitive architectures that are emerging, and how LangChain and LangSmith are built to help with this journey. In this session, Harrison will walk through common problems in evaluating context-aware reasoning applications. He will then talk about how LangSmith - a platform for managing these types of applications - helps address some of these issues, as well as other practical techniques you can implement to tackle this.

Co-Founder and CEO

Building Context-Aware Reasoning Applications with LangChain and LangSmith

Deep reinforcement learning (RL) research has made tremendous progress over the past years. Today it is widely used in various industries as an automatic decision making tool. Many Ray users have presented remarkable successes in using RLlib at recent Ray summits, for example Riot Games, Microsoft, AWS, TwoSigma, and McKinsey/QuantumBlack - just to name a few. Despite wide adoption, there remains one challenge: Tuning an RL algorithm to a specific problem environment demands engineering expertise and compute. The RLlib team at Anyscale has implemented a brand new, model-based RL algorithm, DreamerV3, published in Jan 2023 by Google DeepMind. DreamerV3 is very easy to tune as it requires less than a handful of hyperparameters and is extremely versatile with respect to the different environments it can solve, covering discrete- and continuous action domains, dense- and sparse reward functions, as well as image and vector observations. Due to its model-based nature, it is also very sample efficient, learning reliably on a minimum number of actual environment interactions. Furthermore, since our implementation utilizes RLlib's new multi-node/multi-GPU &quot;Learner API&quot;, it achieves linear walltime scalability compared to the originally published DeepMind code. We will explain the basics of model-based RL and the DreamerV3 architecture. Then we'll show how anyone can rapidly implement such complex algorithms on top of the new &quot;RLlib light&quot; APIs. Finally, we will demo the linearly scalable learning capabilities of DreamerV3, running on dozens of GPUs and mastering very challenging example environments. Takeaways: • Help explain the inner workings and learning capabilities of the SOTA DreamerV3 algo and its implementation on top of RLlib%E2%80%99s new API stack, offering multi-GPU training (OSS RLlib) and multi-node/multi-GPU training (on Anyscale).

Team Lead Manager Reinforcement Learning

DeepMind's DreamerV3 on RLlib with Multiple GPUs: Out-of-the-Box, Fast Model-Based RL

Foundation Models perform well at natural language tasks such summarization, text generation and question answering on a broad variety of topics but struggle to provide accurate answers and without hallucinations about content that they haven't seen in their training data. Retrieval Augmented Generation (RAG) is a popular approach to mitigate this problem. When building next generation Information retrieval (IR) applications using a RAG architecture, enterprises encounter challenges of scale and operational complexity. In this talk, we will address these challenges and walkthrough a set open source technologies on AWS for building enterprise vector-based IR applications.

Sr. AI/ML Specialist Partner SA

Enterprise Vector-based Information Retrieval & Prompt Engineering at Scale using Open Source on AWS

Join us for an exclusive glimpse into the future of Anyscale! In this second of two interactive roadmap sessions, we'll arm you with the knowledge you need to unlock ever more AI value for your organization. In this session, our Product team will unveil exciting updates and demos, share insights into upcoming features, and share our Anyscale roadmap. You'll have the opportunity to ask questions, provide feedback, and be a part of Anyscale's evolution. Don't miss this opportunity to get a sneak peek at what's ahead and how you can be a part of it!

Product Manager

Product Management Team

The Interactive Roadmap Experience Part 1 of 2: Focus on Anyscale

Break

Chroma is an open-source vector database for building production AI apps. AI is powerful, but we need to make it useful by being able to deterministically set its knowledge. In this session, Jeff will walk through how teams are approaching &quot;programming AI with data&quot; today and how they will do this in the future.

CEO and Co-Founder

Embeddings and Vectors are the Key to Production AI

At Netflix, Machine Learning algorithms are at the heart of various use cases such as recommendations, content understanding, content demand modeling, trailer and artwork generation and various other content creation use cases. Scaling these use cases to entertain our members can significantly leverage deep learning techniques. The Machine Learning Platform team at Netflix is tasked with constructing the necessary infrastructure and tools to optimize the effectiveness of all machine learning practitioners across the company. We are constantly striving to ensure that our machine learning models are trained and deployed in a reliable, scalable and robust way. Deep learning models have grown in complexity, requiring significantly more computational resources to train. In this Talk, we explore the benefits of using Ray for building a heterogeneous training cluster, and discuss the steps involved in setting up such a cluster. We demonstrate how to run distributed training jobs on the cluster with a mix of CPU instances and GPU instances, and show how Ray's automatic resource allocation and management can facilitate the scheduling of different types of workers .Additionally, we discuss the challenges and considerations that come with building and managing persistent clusters using Ray, and provide best practices for effective cluster configuration and management.

Machine Learning Engineer

Heterogeneous Training Cluster with Ray at Netflix

Serverless computing is a development model that lets developers build and run applications without having to manage servers. Two popular open source frameworks for running serverless workloads are Ray Serve and Knative Serving. Each framework takes a slightly different approach to serverless: Ray focuses primarily on serving machine learning models, whereas Knative focuses on building automatic HTTP services more generally. But despite these differences, there are many opportunities for both communities to learn from one another, which this talk will highlight. Drawing on experience participating in both communities and building open technologies using both frameworks, this talk compares and contrasts the different approaches that Ray and Knative take to serverless. We also uncover best practices and lessons learned for serverless development as well as potential pitfalls and difficulties that serverless users should be aware of. Furthermore, we highlight key pillars for the next generation of serverless applications, including possible areas of collaboration between the Ray and Knative communities.

Senior Software Engineer

Distinguished Engineer

Ray, Knative, and Running Serverless Workloads in the Cloud

Large Language Models (LLM’s) are starting to revolutionize how users can search for, interact with, and generate new content. Some recent stacks and toolkits around Retrieval Augmented Generation (RAG) have emerged where users are building applications such as chatbots using LLMs on their own private data. This opens the door to a vast array of applications. However while setting up a naive RAG stack is easy, there is a long-tail of data challenges that the user must tackle in order to make their application production-ready. In this talk, we give practical tips on how to manage data for building a scalable/robust/reliable LLM software system, and how LlamaIndex + Ray provide the tools to do so. Bio: Jerry is the co-founder/CEO of LlamaIndex, an open-source tool that provides a central data management/query interface for your LLM application. Before this, he has spent his career at the intersection of ML, research, and startups. He led the ML monitoring team at Robust Intelligence, did self-driving AI research at Uber ATG, and worked on recommendation systems at Quora. He graduated from Princeton in 2017 with a degree in CS.

Co-Founder/CEO

Practical Data Considerations for Building Production-Ready LLM Applications

This case study revisits the key milestones that allowed Amazon to successfully transition core exabyte-scale data catalog management jobs from Spark to Ray. We'll review the challenges encountered, concessions made, and future vision to more deeply incorporate Ray into critical batch and streaming business intelligence pipelines at Amazon. Topics covered include techniques for developing large-scale serverless Ray job management infrastructure, build and deploy management, risk mitigation strategies leveraged to ease the migration to Ray, and operational excellence methods employed to ensure a smooth rollout to production.

Principal Engineer

From Spark to Ray: An Exabyte-Scale Production Migration Case Study

In this session we will cover best practices in preparing your org for the challenges and opportunities offered with GenAI, and what you need to do in the GenAI space to ensure the GenAI capabilities are on a par with Enterprise security, privacy, guardrails. We will provide an overview of Google Cloud's innovation with GenAI in the Enterprise: This session will discuss how GenAI can be effectively utilized to meet the demanding requirements of Enterprise environments, resulting in a more relevant and scalable solutions focused on AI/ML workloads. GenAI is more than a buzzword and our enterprises are seeing the true value of our generative AI offerings in action.

Director, Cloud Partner Engineering, Head of AI CoE

Making your Enterprise GenAI Ready and GenAI Enterprise Ready

Infrastructure challenges like high compute costs, GPU availability, scalability, and the burden of managing cloud resources slow down LLM and generative AI development. Anyscale provides the solutions to tackle these problems so our customers can focus on building and deploying high-performing custom models and applications. Our infrastructure powers our fast, cost-efficient, and scalable Anyscale Endpoints product. In this talk, you will hear about how we: • Leverage all available GPU across different clouds to satisfy your compute needs • Build intelligent features such as autoscaling and fully utilizing preemptible instances to cut cost • Speed up instance start time to accelerate development cycle • Manage compute, networking, storage and other cloud resources Takeaways • There is growing interest in self-hosting open source LLMs due to its flexibility, data privacy and cost-effectiveness, but it comes with challenges. • Anyscale platform provides the solutions to the infrastructure challenges that come with self-hosting LLM, such as high compute costs, GPU availability, scalability, and the burden of managing cloud resources.

Engineering Manager

Managed Cloud Infrastructure for LLMs

In this talk, we will discuss how Pinterest integrated Ray to streamline ML innovations. With Ray, ML engineers at Pinterest can scale the end-to-end ML lifecycle from data processing to model training all within a single Job, minimizing the need for a workflow framework and greatly enhancing the ML developer velocity. We’ll visit how we integrated Ray with Pinterest’s in-house Kubernetes infrastructure, and how it supports business-critical use cases. Furthermore, we’ll share insights into the challenges faced, lessons learned, and the benefits Ray brings to our business.

Last Mile Data Processing for ML Training using Ray

LinkedIn has seen a surge in the use of machine learning over the past few years, driven by more advanced and sophisticated models such as LLM (Large Language Model), LPM (Large Personalized Model), and GNN (Graph Neural Network). These technologies are enabling LinkedIn members to identify new opportunities, receive personalized recommendations, and connect with other professionals in a more effective and efficient way. As a result, it has become necessary to design a training platform that can support these innovations, providing a flexible, user-friendly, and scalable training experience for our AI engineers. In this talk, we will present LinkedIn's cloud-native deep learning training platform on Kubernetes; and how the training platform team leverages Ray(Tune) to enable hyper-parameter tuning in our training pipeline. We treat hyper-parameter tuning as a first class citizen in our training platform as we envision an AutoML end goal that will democratize AI for all engineers.

Machine Learning Infra. Eng.

Hyperparameter Tuning with Ray[Tune] for Next-Gen Training Platform at LinkedIn

With KubeRay, a single command can get your ray-cluster up and running on Kubernetes. In this talk we start by demo-ing to the users how to deploy KubeRay and create their first cluster in seconds. And then we go beyond the basic use case of using Kubectl and Helm to create ray-clusters. We present the extended API to manage ray-clusters from an end-user perspective. We also present the clients libraries allowing your application running inside a ray-cluster to scale its own ray-cluster on demand, or even spin up new ray-clusters and manage them programmatically. This will include the newly added Python-Client KubeRay library (https://github.com/ray-project/kuberay/tree/master/clients/python-client) which developers will appreciate the ease with which they can setup clusters (in K8s) without dealing with the K8s platform or Kubectl and Yaml files. At the end of the talk, you will be familiar with how ray clustering works on Kubernetes and all the different mechanisms the KubeRay project offers you to create and manage ray-clusters.

Principal Software Engineer

The Different Shades of using KubeRay with Kubernetes

Foundation models, trained on broad data and adapted to a wide range of tasks, represent a paradigm shift in AI. But open questions remain: How do we make training much more efficient and accurate through *technical advances*? How do we improve *social responsibility* through documentation and thoughtful benchmarking? What are novel *applications* that can benefit from the power of foundation models? In this talk, I will discuss recent projects that advance each of the three pillars above, and discuss the promise of efforts to connect them.

Associate Professor of Computer Science and Statistics

Advances in Foundation Models — Technology, Society, and Applications

There are a lot of different moving pieces when it comes to developing and serving LLM applications. This talk will provide a comprehensive guide for developing retrieval augmented generation (RAG) based LLM applications — with a focus on scale (embed, index, serve, etc.), evaluation (component-wise and overall) and production workflows. We’ll also explore more advanced topics such as hybrid routing to close the gap between OSS and closed LLMs. Takeaways: • Evaluating RAG-based LLM applications are crucial for identifying and productionizing the best configuration. • Developing your LLM application with scalable workloads involves minimal changes to existing code. • Mixture of Experts (MoE) routing allows you to close the gap between OSS and closed LLMs.

Co-founder and CTO

AI Education Lead

Developing and Serving RAG-Based LLM Applications in Production

We demonstrate the integration of Ray with CodeFlare and Red Hat OpenShift Data Science Pipelines (RHODS Pipelines) for automatically scaling the execution of end-to-end workflows to train and validate foundation models on an OpenShift Container Platform (OCP). Workflow pipelines in foundation model development typically involves running various pre-processing steps to deduplicate data sources, filter out biased and low-quality data, and remove hate and profanity contents. The preprocessed and cleaned-up data are then tokenized and used to further train or fine-tune existing generative pre-trained models. Auto scaling is critical in the execution of this workflow because these steps are usually very compute intensive, with some of the steps iterated several times. RHODS Pipelines is a tool for specifying workflow pipelines as DAGs. It uses Tekton as the workflow engine to deploy pods to execute the workflow DAG in a Kubernetes cluster. However, RHODS Pipelines +Tekton lacks a way for the user to automatically scale up with parallel pods to run a task in the DAG. CodeFlare is a tool to create the necessary configurations for deploying a Ray cluster in an OCP and submitting a task programmed with Ray to execute in parallel. We explore the integration of Ray with CodeFlare and RHODS Pipelines, such that the entire end-to-end workflow DAG, or any subset of them, can be easily specified, independently managed, automatically scaled up by individual developers. We will show foundation model use cases that leverage and benefit from a simple interface to provide specific parameters and specify the DAG in RHODS Pipelines, allowing the tool to generate all the necessary configurations and artifacts for effectively running foundation model workflows in parallel with Ray on OpenShift.

Research Staff Member

Software Engineering Manager

Automatically scaling the execution of Foundation Models workflows: Ray with CodeFlare and Red Hat OpenShift Data Science Pipelines

Data loading and preprocessing can easily become the performance bottleneck in ML training pipelines. Data preprocessing requirements are also becoming more complicated as the types of data being processed are becoming more diverse. With Ray Data, data loading can be fast, flexible and scalable. In this talk, we’ll dive into the performance of different open-source data loader solutions. We’ll show how Ray Data can match PyTorch DataLoader and tf.data in performance on a single node, while also providing advanced features necessary for scale, such as in-memory streaming, automatic recovery from out-of-memory failures, and support for heterogeneous clusters. Takeaway: • Ray Data provides a combination of speed, scale, and flexibility unmatched by other open-source data loaders.

Fast, Flexible, and Scalable Data Loading for ML Training with Ray Data

It can often be useful to leverage short-lived Ray clusters as a part of a broader ML workflow (ex: to perform distributed training as part of a pipeline with multiple data and modeling steps). Using these single-purpose, ephemeral Ray clusters can unlock opportunities for improved reproducibility, efficiency, and observability. KubeRay provides a natural way to manage these ephemeral Ray clusters on Kubernetes. Sematic (https://sematic.dev) shares learnings from leveraging KubeRay &amp; Ray in this way.

Founding Engineer

On-Demand Ray Clusters in ML Workflows via KubeRay & Sematic

Productionizing modern machine learning workloads is challenging. Not only do you need to train and optimize your models, but also find a way to serve them efficiently without too much operational cost. Ray Serve solves these complex requirements to enable you to go to production safely and at low cost: you can flexibly scale and coordinate multiple models, deploy and upgrade safely, and maximize your hardware utilization with minimal management overhead. This talk will demonstrate Ray Serve’s production-ready capabilities, including a demo of serving an ML-powered application using Ray Serve on the Anyscale platform. Some highlights include improvements around scalability, high availability, fault tolerance, and observability. Takeaways: • Learn about patterns of production ML serving and how Ray Serve is tailored to solve them. • Hear how users in the community are using Ray Serve in production to lower their ML inference costs. • Watch a real time demo of how to serve an ML application using Ray Serve on the Anyscale platform. This will highlight recent improvements around observability, autoscaling, and cost savings.

Building Production AI Applications with Ray Serve

LLMs have seen an impressive wave of advances, with models now excelling in a variety of tasks, such as mathematical reasoning and program synthesis. However, their potential to effectively use tools via API calls remains unfulfilled. This is a challenging task even for today's state-of-the-art LLMs, largely due to their inability to generate accurate input arguments, and their tendency to hallucinate the wrong usage of an API call. We release Gorilla LLM that surpasses the performance of all close-sourced and open-sourced LLMs on writing API calls. When combined with a document retriever, Gorilla demonstrates a strong capability to adapt to test-time document changes, enabling flexible API updates and version changes. Gorilla also substantially mitigates the issue of hallucination, commonly encountered in LLMs. Gorilla is an open-source project having served hundreds of thousand user requests and an energetic community supporting it. Check out the project at https://gorilla.cs.berkeley.edu/.

CS PhD Student

Gorilla: Large Language Models Connected with Massive APIs

Freenome is developing next-generation blood tests for early cancer detection, powered by multiomics and machine learning. We are continuously exploring ways to make model development more efficient and scalable because the faster we can evaluate new models, the faster we can test their potential utility for use in new cancer diagnosis products. This presentation will describe our journey using Ray at Freenome to improve the scalability and ease-of-use of our machine learning platform. The presentation is general enough to be applicable to other industries and is appropriate for any group interested in improving machine learning model development. It is especially appropriate for teams that are looking for opportunities to scale and simplify machine learning operations (MLOps).

Senior Machine Learning Research Engineer

A Ray-based ML Platform for Early Cancer Detection

Millions of customers visited the Verizon Website and stores monthly to shop, from plans to products and accessories. Numerous task-specific recommender systems have been built at Verizon to enhance customer experience. Motivated by the latest development in deep learning, multi-task learning architectures can better understand customer behavior therefore greatly improve model performance, and also save time to learn new tasks. However, it's challenging to build such a system to: 1) digest heterogeneous data from purchases, online clicks, store visits to customer services and many more; and 2) accomplish various tasks where some of them might contradict with each other; and 3) efficiently train and tune large DL models at scale . At Verizon, we greatly benefited from the Ray ecosystem and built a novel multi-task learning recommender system, namely Pathways Recommender system (PaRS), which directly handles multiple abstract forms of data and is able to generalize across multiple recommendation tasks at Verizon. Last but not least, we explored built-in bias mitigation in PaRS to promote diverse, relevant and fair recommendations.

Lead Distinguished Scientist

Associate Director of Responsible AI

Large Scale Pathways Recommender Systems (PaRS) at Verizon

Airbnb lists more than 6M properties across more than 100,000 cities and towns spanning 220 countries. LLMs are a strategic opportunity for Airbnb across multiple use cases. Airbnb's ML Platform team wanted to augment their existing ML platform to enable internal teams to effectively and efficiently use open source LLMs. This is difficult because LLMs present scalability challenges that push the boundaries of existing tooling. Working with Ray, the Airbnb team is seeing improvements in developer productivity, and significantly better performance. Beyond that, they're now able to better utilize A100 GPUs on AWS for their training workloads, providing long term efficiency and cost savings for these strategic LLM applications. This session will describe before and after states and share real-world benchmarks demonstrating cloud efficiency, comparing Sagemaker to Ray, and also describe some of the applications that this work enables.

Optimizing LLM Training with Airbnb's Next-Gen ML Platform

With the growing complexity of deep learning models and the emergence of Large Language Models (LLMs) and generative AI, scaling training efficiently and cost-effectively has become an urgent need. Enter Ray Train, a cutting-edge library designed specifically for seamless, production-ready distributed deep learning. In this talk, we will take a deep dive into the architecture of Ray Train, emphasizing its advanced resource scheduling and the simplicity of its APIs designed for effortless ecosystem integrations. We will cover a detailed breakdown of Ray Train's design, from its robust architecture to its exclusive features for LLM training, including Distributed Checkpointing and the seamless Ray Data Integration. Takeaways: • Ray Train offers production-ready open-source solutions for large-scale distributed training. • Ray Train seamlessly integrates into the deep learning ecosystem (such as PyTorch, Lightning, HuggingFace) with easy-to-use APIs. • Ray Train accelerates your LLM development with built-in fault tolerance and resource management capabilities.

Ray Train: A Production-Ready Library for Distributed Deep Learning

Ray.io is a widely adopted open-source framework for distributed computing that offers scalable and efficient execution of machine learning tasks. IN this session we will cover the details of how to adapt OSS to meet more stringent Enterprise-ready requirements such as security and networking. We will examine a set of ten best-practices to adapt OSS for the Enterprise and demonstrate how to further enhance Ray.io's suitability for Enterprise-level computing. Our discussion will encompass topics ranging from bolstering security features to improving scalability and reliability through integration with GKE. Our goal is to illustrate how Ray.io can be optimized to meet the demanding requirements of Enterprise environments, resulting in a more robust and scalable solutions esp focused on AI/ML workloads. We invite you to join us as we explore the exciting possibilities of Ray.io in the realm of Enterprise computing.

Integrating Ray.io into the Google Cloud Ecosystem for the Enterprise

OneDigital, the data and digital division of one of Australia's leading conglomerates, Wesfarmers, faces the challenge of building machine learning (ML) products utilising vast amounts of data. The growing demand for better scalability and a more efficient end-to-end productionization process prompted the need to uplift our existing platform with modern architecture and improved performance capabilities. In this talk we will explore the journey of building a new ML platform using Ray, Anyscale, and other complementary products. We will delve into the key requirements for building this comprehensive platform that provides a rich user experience and reduces model deployment time and complexity, and cover the architecture, strategies and techniques used. Finally, we will showcase some retail use cases which use advanced ML models fine-tuned on Ray, to better identify customer segments, behaviors and preferences.

Principal Machine Learning Engineer

OneDigital ML Platform: Unleashing the Power of Machine Learning at Scale

Join us for an exclusive glimpse into the future of Ray! In this first of two interactive roadmap sessions, we'll arm you with the knowledge you need to unlock ever more AI value for your organization. In this session, our Product team will unveil exciting updates and demos, share insights into upcoming features, and discuss our open source roadmap. You'll have the opportunity to ask questions, provide feedback, and be a part of Ray's evolution. Don't miss this opportunity to get a sneak peek at what's ahead and how you can be a part of it!

The Interactive Roadmap Experience Part 2 of 2: Focus on Ray and Open Source Projects

In this presentation, we discuss the implementation of a computer vision model using the Ray open-source framework and compare its training performance against another popular distributed training framework, Kubeflow. Our goal was to evaluate the cost effectiveness, training speed, gpu utilization, and throughput of training the machine learning model using both frameworks. Ray is an open-source framework that is designed to make it easy to scale and optimize distributed Python applications. We chose to evaluate Ray for this project because it has built-in support for distributed training of machine learning models and can utilize an inexpensive data storage framework, Amazon S3, for data storage, which is can be more cost-effective than the use of Amazon elastic file storage (EFS). To compare the performance of Ray and Kubeflow, we trained our computer vision model on a large production dataset using both frameworks. We measured the training time, cost, and throughput for each framework and found that Ray was more cost-effective and faster than Kubeflow, even while using S3 for data storage in comparison with EFS. One of the key reasons for the performance improvement of Ray is its use of distributed training. By distributing the workload across multiple nodes, Ray was able to complete the training process faster and at a lower cost than Kubeflow. We utilized in-expensive computational optimizations to process data quickly from S3. We used Ray Dataset to parallelize our data IO, preprocessing and ingesting to GPU trainers at scale. From our experiments, trainer GPUs can achieve desired saturation with image data stored on commodity S3 storage, where data workload happens on heterogeneous, commodity CPU workers for best cost efficiency. In addition, Ray Dataset also has built-in support for full streaming execution that lowers memory and compute requirement for training data at larger scale. With optimizations of vectorized preprocessing UDF, batching and prefetching, we have shown Ray AIR can effectively handle and scale computer vision workloads with large amount of training data without sacrificing performance. In conclusion, our project demonstrates certain benefits of using Ray for computer vision applications. By leveraging its distributed training capabilities along with its use of cost-effective S3 data storage, we were able to achieve improved performance in terms of cost-effectiveness and training speed compared to Kubeflow's use of a more expensive EFS storage. Our findings are that this can have important implications for businesses and organizations that rely on computer vision models to drive their operations.

Machine Learning DevOps Engineer

Scaling Computer Vision Models with Ray: A Cost-Effective and Efficient Distributed Training Framework

Snorkel helps the world's largest organizations solve their toughest ML challenges. To continue our growth and achieve our product initiatives around foundation and large language models, we needed to fully redesign our interactive ML systems so that our products stay performant under increasing data and model scale. However, building low-latency ML products for enterprises is challenging. Some enterprises are on-premises only and have limited compute resources. Others have very large datasets that need to be processed interactively. Still, others want to use the latest and greatest large language models. How do you go from sprawling requirements to an architecture that is performant for everyone? In this talk, we share how we went from the customer requirements all the way to using Ray to design and build the interactive ML system that now powers our flagship enterprise product, Snorkel Flow. We'll dive into: * Distributed data/task parallelism to run ML workloads of any scale for resource-constrained customers. * Scalable in-memory processing for blazing fast ML workloads for resource-abundant customers. * How we combined the above two approaches into a single architecture deployable on any customer. * Lessons learned from using Ray to build performant ML systems.

Member of Technical Staff

How Snorkel Builds Interactive Enterprise ML Products Using Ray

Llama and Llama 2 have taken the Generative AI world by storm with publicly available state of the art performance under a commercially permissive license. This has been a game changer for startups, enterprises and a community looking to harness the power of Generative AI but need a platform to build on. Llama is that platform. Join Joe Spisak on a journey through the evolution of Llama — Meta’s latest open source large language model (LLM). Understand its growth and the range of available models including the most recently released Code Llama. Joe will also dive deeper into some of the major platforms built on Ray that enabled Llama 2 and Code Llama such that you yourself can leverage state of the art performance on infrastructure along with the most popular open models available.

Product Director, Head of Generative AI Open Source

Llama, Scaling Up LLMs in an Open Ecosystem

Ray, including Ray/RLLib, has been actually speeding up many reinforce-learning-based services in NetEase Games. It has greatly promoted the user experience of game players and the general profit of various game products as well. We deployed our first reinforcement learning based recommendation system application using ray RLLib. Reinforcement learning is a promising direction since the RL paradigm is inherently suitable for tackling multi-step decision-making problems, optimizing long-term user satisfaction directly, and exploring the combination spaces efficiently. As a way of giving back to the community, we open-sourced the RL4RS (Reinforcement Learning for Recommender Systems) dataset - a new resource fully collected from industrial applications to train and evaluate RL algorithms with special concerns on the reality gaps, contains two real-world datasets, data understanding tools, tuned simulation environments, related advanced RL baselines, batch RL baselines, and counterfactual policy evaluation algorithms. The RL4RS suit can be found at this repo (https://github.com/fuxiAIlab/RL4RS).

Senior AI Researcher

Ray/RLLib-Empowered Reinforcement Learning Based Recommender Systems in NetEase Game

Managing Ray clusters and workloads on Kubernetes can be challenging. KubeRay provides an open-source toolkit to manage the lifecycle of Ray clusters and simplify the deployment process of Ray Applications on Kubernetes. In this talk, we will talk about: 1. How KubeRay integrates with the Kubernetes ecosystem? (e.g. observability, ingress, prototyping, scheduling) 2. What capabilities exist in KubeRay? (e.g. autoscaling, fault-tolerance, high-available serving deployment, batch job) 3. Performance/Stability/Scalability benchmarks

KubeRay: A Ray cluster management solution on Kubernetes

Extended Networking Break

Ray has quickly become the distributed framework of choice for training, tuning and serving Generative AI and Large Language Models (LLMs). The foundation of any LLM/Generative AI effort is the ability to have access to high quality data. Cloudera provides an open data lakehouse that is secure by design and flexible to fit scenarios ranging from stream processing, data warehousing, data engineering, and AI and advanced ML. In this session, we will show you how Cloudera Machine Learning makes it easy to spin up Ray clusters on Kubernetes with a couple lines of code. This enables data scientists to leverage the power of Ray on top of enterprise data. With this data loaded up into a Ray dataset, we will finish showing how the data can be transformed into features and used to train and deploy an ML model.

Principal Product Manager

Making it easy to provision Ray clusters to support enterprise AI/ML efforts

With the rising demand for Generative AI applications in businesses, existing operational ecosystems are being tested for their adaptability to this new technology. This talk aims to address the arising question: &quot;Can existing MLOps frameworks accommodate LLMOps while maintaining risk and compliance standards across enterprises?&quot; We will delve into the intricacies of both MLOps and LLMOps, discussing which MLOps components are feasible for reuse, and which require major modifications or total replacement to support generative AI and LLMs efficiently. We invite IT professionals tasked with supporting LLMOps systems, Engineers and Data Scientists tasked with building LLM-based applications, and Risk Officers seeking to establish governance frameworks to join us as we chart the roadmap for the transformation of your existing MLOps framework to make it Gen-AI-ready.

Senior Director, Product Management

Bridging MLOps and LLMOps: Integrating Governed Generative AI into your Enterprise

Community Bash (Networking)

This presentation explores io.net's revolutionary approach to achieving high-throughput inference by deploying thousands of decentralized, globally-distributed GPU clusters powered by Ray Core and Ray Serve. Demonstrations will include a step-by-step guide on rapidly scaling &quot;KubeRay&quot; and Ray Core clusters to meet any computational demands. Key Insights: Decentralization for Lower Latency: Understand how io.net has used decentralized architectures to bring machine learning models closer to end-users, thereby reducing inference time. Global Distribution for Resiliency: Learn about the importance of globally-distributed GPU clusters in achieving fault tolerance and maximizing uptime. High Throughput with Ray Serve: Discover how Ray Serve can be tuned for high-throughput inference pipelines, allowing for more requests per second without sacrificing quality. Rapid Deployment via KubeRay: Witness a live demo that showcases the speed and ease with which KubeRay enables deployment of Ray clusters, scaling to any size almost instantaneously. The world's first decentralized Kubernetes clusters. Adaptability and Flexibility: Gain insights into the adaptability of this architecture, suitable for various industry applications.

Co-Founder & CTO

Co-Founder & COO

Decentralized Ray - Scaling Ray for Global Inference

Samsara's machine learning infrastructure team shares how they used Ray serve to scale their cloud IOT inference platform developing computer vision and sensor fusion detection models. The team will share pragmatic best practices learned creating production serve clusters. Providing demos such as: templates to provision a production cluster setup turn-key, common utilities we've developed, and dashboards for observability and maintenance. In the IOT space, configuration and customization is key, we'll touch on how Ray serve helped Samsara ML engineering dev-velocity by simplifying the composition of complex pipelines to target our specific customer's use-cases. (( Key Talking Points )) (Section) How is Samsara's Machine learning platform special and why did we choose ray serve? 1. For teams already familiar with ray, what's the transition process like? 2. Applications are generally event detection pipelines with configuration and conditionals. 3. Data comes in many forms - computer vision and sensor fusion. 4. Enabling real-time inferences for our application: ex. relevancy. (Section) Learnings from Deployment 1. Guidelines for cluster configuration - ex. autoscaling, logging. 2. Metrics to monitor - dashboards, what to alert on. 3. Composing pipelines as code. 4. How we maintain our clusters. (Section) Utility Sharing - Give back to the community 1. Terraform for EKS cluster setup 2. Template/example ray cluster cluster configuration 3. Example complex serve inference pipeline 4. Utils like model LRU cache to share endpoints

Ray Serve for IOT at Samsara

Working with data has always been the hardest part of developing applications. As applications evolve and requirements change, so do the technologies used to support them. The adoption of different data stores, the creation of data transfer processes, and the increasing complexity of query access patterns can introduce a lot of friction into the SDLC. In this session we will learn how the MongoDB Atlas Developer Data Platform along with the benefits of the Document Model can liberate your data and enable innovation, while integrating with Anyscale to leverage the most advanced models, at scale.

Executive Solutions Architect

Real-Time Vectorization for Transactional Workloads with MongoDB Atlas + Anyscale

AWS Glue has been pioneering in the space of automating ETL processes by providing a fully managed serverless data integration service. This service is a simple and cost-effective way for customers to categorize their data, clean it, enrich it, and move it swiftly and reliably between various data stores. AWS Glue is made up of a Data Catalog (i.e a metadata store), sophisticated ETL engines with automated code generation and visual interfaces for every persona to do ETL tasks. AWS Glue customers use Apache Spark and Python engines for data integration. Our Python customers have asked for scaling their Python workloads over large datasets. To enable these use-cases, AWS Glue added support for Ray.io (http://ray.io/) and launched AWS Glue for Ray. AWS Glue for Ray provides data engineers a distributed Pythonic data analytics platform for performing distributed data integration at scale with Ray core APIs. Using Ray's powerful abstraction of tasks/actors, we were able to horizontally scale python workloads. The simple distributed collection APIs provided by Ray dataset helped our python customers to perform ETL operations efficiently on very large datasets. Distinctly, we launch Ray clusters on ARM based platforms and using IPv6 addressing based workers. Data engineers are comfortable with Pandas and given that popularity, we integrated Modin at scale with Ray. We will cover our experiences with Ray datasets and distributed Pandas at scale. We will also talk about the innovations we did integrating with Ray's robust cluster manager and demand based autoscaler, to offer an instant-on, interactive, easy to use serverless Ray platform for our customers.

Sr Software Engineer

Software Development Manager

Building an Instant-On Serverless Platform for Large-Scale Data Processing Using Ray

KServe is a Opensource production-ready model inference framework on Kubernetes utilizing many knative's features such as routing for canary traffic and payload logging. However, the one model per container paradigm limits the concurrency and throughput when sending multiple inference requests. With RayServe integration, a model can be deployed as individual Python workers allowing for parallel inference. This enables concurrent inference requests to be processed simultaneously, improving overall efficiency. In this talk, we will share how you can configure, run, and scale machine learning models in Kubernetes using KServe and Ray.

Parallel inferencing with KServe Ray integration

In this talk, we will dive deep into operationalizing a generative AI project lifecycle when using Kubernetes for compute. We will demonstrate how machine learning and platform engineers can use JupyterHub, Argo Workflows and Kuberay to enable experimentation for data scientists in order to adapt a large language model from Hugging Face and provide a workflow for fine-tuning and deploying trained model into production.

Sr. Specialist Solutions Architect

Unleashing the Power of GenAI: Scaling with Ray and Orchestrating with Kubernetes

Time series—data ordered chronologically—constitutes the underlying fabric of systems, enterprises, and institutions. Its impact spans from measuring ocean tides to tracking the daily closing value of the Dow Jones. This type of data representation is indispensable in sectors such as finance, healthcare, meteorology and social sciences. However, the current theoretical and practical understanding of time series hasn't yet achieved a level of consensus among practitioners that mirrors the widespread acclaim for generative models in other fundamental domains of the human condition, like language and perception. Our field is still divided and highly specialized. Efforts in forecasting science have fallen short of fulfilling the promises of genuinely universal pre-trained models. In this talk, we will and introduce TimeGPT, the first pre-trained foundation model for time series forecasting that can produce accurate predictions across a diverse array of domains and applications without additional training. A general pre-trained model constitutes a groundbreaking innovation that opens the path to a new paradigm for the forecasting practice that is more accessible and accurate, less time-consuming, and drastically reduces computational complexity. During the talk, we will guide practitioners to leverage TimeGPT and the distributed capabilities from Ray to perform forecasting and anomaly detection at scale.

CEO & Co-Founder

TimeGPT: A Foundation Large Time Series Model

VP of Engineering

Senior Vice President of Engineering

Senior Engineering Manager

Principal Software Engineer/Tech Lead Manager

Co-founder and General Partner

Engineering Manager, ML Data Platform

Sr. Director, AI/ML Platform and Data

Director Neuron Software (Inferentia and Trainium)

Keynote, Day 2

Consumer-based internet companies are leading the charge of the current &quot;AI trend&quot; – bringing its power to the transportation, delivery, and social media apps we use everyday. The AI infrastructure teams behind these cutting-edge applications increasingly rely on Ray. In this panel discussion, leaders of AI infrastructure teams at Uber, DoorDash, and Pinterest will share their insights on this topic. Learn from their experiences as we talk about the state of AI in their organizations, the challenges they face as they evolve, and the values Ray has brought about. Moderated by Zhe Zhang, Head of Open Source Engineering, Anyscale

Head of Open Source Engineering

Building Next Generation AI Infrastructure on Ray

Join us for a hands-on demo of the CodeFlare-SDK, an open-source project that simplifies cloud-native data pre-processing, model training and validation with an intuitive Python interface to Ray, PyTorch/TorchX, and Kubernetes. With the CodeFlare-SDK, you can easily manage your cloud resources, submit jobs, and monitor job status, without worrying about the complexities of DevOps and cloud infrastructure. In this demo, Mustafa Eyceoz and Atin Sood will guide you through the CodeFlare-SDK workflow, from resource allocation to ML job submission and monitoring. You will see firsthand how easy it is to train large scale models (foundation models) in the cloud using the CodeFlare-SDK. Don't miss this opportunity to learn how CodeFlare can make cloud-native model training more accessible and manageable for developers.

Program Manager - IBM Research

How to simplify execution of cloud-native model training and validation with CodeFlare: A Hands-On Demo

Are you looking for ways to improve the efficiency and success of your multi-agent reinforcement learning (MARL) experiments? Ray can automate MARL tuning by dynamically tracing and optimizing experiments across parameter spaces, while utilizing your computational resources via its tools such as AIR, Tune, and RLLib. Combined with Weights &amp; Biases, MARL engineers get a comprehensive view of their experiments, centralizing all details and assets into one ML system of record. This enables them to save time on experiment iterations and quickly determine which tunings produce optimal performance outcomes, providing faster results with fewer hiccups. To demonstrate this, we will perform experiments on two scenarios: Autonomous Vehicle Driving and Drone Flying.

Tend and Tune Your MARL Experiments with Ray and Weights & Biases

Deep learning systems are consistently producing state-of-the-art results for tasks using unstructured data such as text, images, video, and audio, but unlocking this value in production requires taking best-in-class pretrained models (GPT, BERT, ViT, etc.) and tuning them to your domain-specific datasets and tasks. Ludwig is a low-code deep learning framework (developed and open sourced by Uber into the Linux Foundation) that integrates and scales natively with Ray to declaratively fine-tune state-of-the-art foundation models, customized to your domain-specific business data – including tabular metadata, or any other feature types. In this talk, we present 3 ways to fine-tune powerful foundation models like LLMs on your data in less than 10 lines of YAML using Ludwig on Ray: Modify the weights of a pretrained model to adapt them to a downstream task. Keep the pretrained model weights fixed and train a stack of dense layers that sit downstream of the pretrained embeddings. Use the pretrained model embeddings as inputs to a tree-based model (gradient-boosted machine). We explore the tradeoffs between these approaches by comparing model quality against the training time / cost, and show you how Ludwig leverages the advanced capabilities of Ray AIR to provide out-of-the-box scale and optimized performance on any hardware. Key takeaways from this talk include: - Use the Ludwig framework to flexibly train a model for any task in just a few lines of YAML, while retaining the flexibility to customize any and all parts of the model architecture. - Automatically scale your Ludwig training job out across a cluster of machines with zero code or config changes using native integration with Ray. - Apply best practices to speed up fine-tuning when modifying the pretrained weights with just a few lines of YAML, including: automatic mixed precision, batch-size auto-tuning, ghost batch normalization, low-rank adaptation, and distributed training. - Understand how Ludwig leverages Ray Train to provide out-of-the-box support for both data parallel (DDP) and model parallel (FSDP) training through a single parameter in a YAML configuration. - Enable Ludwig's encoder embedding cache capability, available through a single YAML parameter, to speed up fine-tuning with fixed pretrained weights by over 50x, and producing an ordinary model at the end of training ready to serve in production. - Understand how Ray Datasets parallelizes and scales the encoder embedding caching, making it possible to fine-tune state-of-the-art models on commodity CPU hardware. - Explore using the same cached pretrained embeddings to fine-tune a neural network or a gradient boosted tree, switching between the two through a single parameter in the Ludwig config, and leveraging Ray AIR's support for distributed PyTorch and distributed LightGBM. Project repository: https://github.com/ludwig-ai/ludwig

Chief Technology Officer

50x Faster Fine-Tuning in 10 Lines of YAML with Ludwig and Ray

AI copilots – LLM-powered assistants embedded into applications – are rapidly becoming a core feature of almost every application. In this talk, I'll discuss how to build a production AI copilots for your application, including best practices and common pitfalls. I'll cover connecting to private data and APIs, orchestration and planning, authentication and authorization, and copilot UX patterns.

CEO 

How to Build an AI Copilot for Your Application

Ray is a powerful distributed computing framework. However, as data sets grow and computation requirements become more complex, managing memory usage across multiple computing nodes becomes increasingly challenging. Issues that slow down performance include the data copying between the computing nodes, data spilling out of memory into storage, and the data skew among computing nodes. We'll introduce Gismo, a multi-node shared memory object store based on Compute Express Link (CXL) technology to address this challenge. With the help of Gismo, Ray does not need to serialize and transfer the extra copies of the objects across network. Data spill and skew issues will be minimized because each host has direct memory access to the whole object store. This talk will demonstrate how Gismo is integrated with Ray and showcase how it improves overall performance and reduces memory overhead for Ray users.

Co-Founder & CEO

Gismo for Ray: A Multi-Node Shared Memory Object Store That Accelerates Ray Workloads

SkyPilot is an open-source framework for running LLMs, AI, and batch jobs on any cloud. In this talk, we describe how SkyPilot offers AI practitioners high GPU availability, maximum cost savings, and managed execution — all with a simple cloud-agnostic interface. SkyPilot is open source (Apache 2.0) and under active development at UC Berkeley. SkyPilot's new approach is to view all clouds and regions (that a user has access to) as a coherent collection called the Sky. SkyPilot transparently sends a job to the cheapest and most available location in the Sky and manages its execution. It cuts cloud costs by supporting spot instances with automatic recovery (potentially across zones, regions, clouds) and automatically cleaning up idle cloud instances. We'll also share our experiences deploying SkyPilot to users in the past year. SkyPilot has powered the development of popular LLM projects including the Vicuna chatbots and the vLLM inference system. It is being used by 10s of organizations for diverse use cases, including AI training on GPUs/TPUs, AI serving, CPU batch jobs, and interactive development.

Postdoctoral Researcher

SkyPilot: Run AI on Any Cloud

In recent developments, large open language models have achieved remarkable advancements, unlocking new possibilities for commercially-scalable enterprise applications. Among these models, Meta's Llama-2 series has set a new benchmark for open-source capabilities. While comprehensive language models like GPT-4 and Claude-2 offer versatile utility, they often exceed the needs of specialized applications. This presentation will explore our insights gained from fine-tuning open-source models for task-specific applications, demonstrating how tailored solutions can outperform even GPT-4 in specialized scenarios. We'll also discuss how leveraging Anyscale + Ray's suite of libraries has enabled efficient fine-tuning processes, particularly in an era where GPU availability presents a critical bottleneck for many organizations. Takeaways: • Where to apply fine-tuning and when would it shine? • How to set up an LLM fine-tuning problem? • How does Ray and its libraries help with building a fine-tuning infrastructure? • What does it take to do parameter efficient fine-tuning? • How does Anyscale platform help with LLM-based fine tuning?

Team Lead

Lessons From Fine-Tuning Llama-2

Serving numerous models is essential today due to diverse business needs and various customized use-cases. However, this raises the challenge of how to efficiently deploy and manage these models while considering both ease of use and cost-effectiveness. This talk aims to provide a comprehensive insight into various patterns of serving many models using Ray Serve. We will delve into how 3 features in Ray Serve - model composition, multi-application, model multiplexing - enable seamless deployment of numerous models while optimizing resource utilization. Takeaways: • Discuss common industry patterns for serving many models. • Learn how to simplify management and enhance performance of many-model serving through Ray Serve's model composition, multi-application, and model multiplexing features. • Deep dive into case studies of Ray Serve users running many-model applications in production.

Sponsors

Event Organizer

Platinum Sponsors

Gold Sponsors

Silver Sponsors

Become a Ray Summit Sponsor

Why Become a Sponsor?

Share Your Point of View

Plug into ML/AI Trends

Discover New Talent

Join the Conversation