Build Augmented Reality Maps With Ray

Niantic is a leading augmented reality company. The AR mapping organization at Niantic is responsible for building and serving the best 3D maps to users all around the world through Niantic Lightship visual positioning system (VPS).

VPS is like GPS for the virtual version of the outside world, only far more precise, down to centimeter. With VPS developers can build persistent AR content that is anchored at real-world locations. This means people can have more meaningful and compelling immersive AR experiences in the real world because the visuals are blended perfectly.

We leverage Ray to solve 3 problems: 1. Research to production 2. Scan processing 3. Map building.

1: VPS is new and we need to constantly ship research outcomes into production. Often researchers, CV engineers and server engineers use different development environments, codebases and toolchains. Such gaps make debugging, testing and deployment difficult and extremely painful. With Ray we are building a serverless framework which different groups can run without code changes which fills the gaps.

2: Millions of people playing Niantic titles like Pokemon Go and Ingress upload a large volume of scans of places of interest (POIs) from all around the world to Niantic servers, every minute. There are a number of steps in scan processing, including blurring potentially recognizable objets like faces and license plates, running AI models for quality control, generating scan previews and etc. Different steps run on different hardware platform, e.g. CPUs and GPUs. Low overhead scheduling, efficient data passing between machines (because the scans files could be large) are critical for good performance and low cost, which Ray is good at.

3. Creating 3D maps from scans is a complex process. There are number of steps involved - splitting scans, location mapping, connecting relevant maps, 3D dense reconstruction, context + semantics, map coordinate system accuracy improvement, merging connected maps etc. The overall pipeline is a large distributed system and some individual steps are also distributed. Easily connecting different components and distributing workloads, efficiently passing data around and scheduling different steps onto different hardware platforms are very important. With Ray we are able to make the pipeline simple, efficient and performant. What's more, with Ray we can also make overall system highly composable. Each step can be made a building block that can be run both locally and remotely, and users can easily assemble different building blocks to achieve their goals.