Perplexity Details GPU Embedding Stack Serving pplx-embed

According to the post, embedding serving is framed as two workloads. Batch embedding occurs when building or re-indexing the vector database, where throughput minimizes cost. Online embedding occurs at query time, where a short query must be embedded fast. Scoring sits in between: after vector search, large document batches are ranked, balancing both. The company said it did not build a separate embedding engine. Because embedding models are small Transformers, batch embedding resembles compute-bound prefill and online embedding, often a few tokens, resembles memory-bound decode. The team therefore reused the prefill and decode kernels from its LLM stack.

Three services handle a request. Ivy is a Rust HTTP gateway that performs CPU-side work such as JSON parsing, tokenization, input templating and batch splitting. It translates requests into a custom gRPC protocol, splits large-batch requests into chunks and load-balances them across replicas. Tulip is the inference server interface: a gRPC server built with Rust, tokio and tonic, handling scheduling and batching before dispatching to the engine. ROSE, or Runtime-Optimized Serving Engine, implements model inference in Python, providing kernels, layers and model definitions, managing CUDA graphs and exposing a step() function to Tulip.

The scheduler is deliberately simple, picking sequences first-come, first-served while requests accumulate. Perplexity said that for small embedding models at the sequence lengths it serves, the linear cost of dense layers dominates the quadratic cost of attention, so latency is roughly proportional to token count rather than sequence count. Once a batch saturates the GPU, around 512 tokens on a sub-billion-parameter model, adding more sequences does not improve efficiency.

To reduce CPU-side kernel launching on small batches, Perplexity builds whole-model CUDA graphs for all embedding models, capturing every launch into a single driver call. Some attention implementations block full-model graphs by depending on dynamic host-side inputs; Perplexity upstreamed changes to FlashInfer to enable capture. Graphs must be captured per configuration, so token counts are padded to buckets that are multiples of 64 or 256, yielding thousands of graphs and multiple minutes of capture per model. The fix is lazy capture: each configuration gets an eager warmup run, then triggers capture and replay on its second hit. This costs p99 latency at startup but spreads minutes of eager work across hours.

The second piece is the LazyTensor, which tracks a page-locked host buffer plus a cudaMemcpyAsync and a CUDA event. Instead of step() blocking on the device, it returns a LazyTensor, letting a Rust async task wait on batch N while the CPU enqueues N+1.