Google Cloud ships llm-d 1.0, a Kubernetes-native inference stack for TPUs and GPUs

Google reported a published benchmark on a 17,000-node multi-region Google Kubernetes Engine deployment, MarkTechPost said. The stack runs on Kubernetes with NVIDIA, AMD or Intel GPUs or Google TPUs, over vLLM or SGLang. Vendor-reported gains come from cache-aware routing, tiered KV cache offload and prefill/decode disaggregation.

In the best case, median time to first token was cut by more than 85% with 2.4x throughput on an Ironwood TPU coding workload, according to the report. In the worst case, RL time-slicing lifted accelerator duty cycle only to above 70%, leaving roughly 30% idle time.

llm-d was founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave and NVIDIA. Red Hat, IBM and Google are the lead maintainers. The project counts more than 500 contributors from over 25 organizations; it is a CNCF Sandbox project accepted on March 12, 2026, and is licensed under Apache 2.0. Prefix-aware routing, KV cache offloading, prefill/decode disaggregation, wide expert parallelism and batch inference already shipped. Version 1.0 is about running them reliably in production, which IBM calls day 2 operations.

New in 1.0 are multi-tenant quality of service with priority and fairness-aware flow control to stop bursty agents from starving latency-sensitive requests; v1 APIs and a hardened request path validated on multiple clouds and accelerators; zero-downtime upgrades with router high availability and hot standby plus DisaggregatedSet, which rolls prefill and decode pools together; agent-aware serving with session-aware routing, turn-priority scheduling and native Responses API state; asynchronous inference through llm-d-async-processor, which pulls jobs from Google Cloud Pub/Sub or Redis into slack capacity; and expanded TPU support via TorchTPU, which runs PyTorch models natively inside upstream vLLM and SGLang.

The llm-d router builds on the Kubernetes Gateway API Inference Extension. It tracks which replicas hold each cached prefix, then weighs load and predicted latency before picking a replica. Agents call one endpoint, and each step lands where its context already lives. When accelerator memory fills, KV cache moves to host memory, then SSD or Managed Lustre, using vLLM’s OffloadingConnector, Mooncake or SGLang’s HiCache. Prefill and decode run as separate pools because prefill is compute-heavy and decode is memory-bound.