IBM-backed llm-d 1.0 adds controls for self-hosted AI inference

The release was announced ahead of the PyTorch Conference North America in San Jose. It focuses on managing competing workloads, maintaining service during upgrades and improving hardware efficiency as AI agents generate growing volumes of model requests.

Built for Kubernetes, llm-d coordinates model serving across computing infrastructure. IBM, its Red Hat subsidiary and Google LLC are lead maintainers. They co-founded the project last year with CoreWeave Inc. and Nvidia Corp. Contributors include Advanced Micro Devices Inc., Cisco Systems Inc., Hugging Face Inc., Intel Corp. and Mistral AI SAS.

The project is intended to give organizations more control over inference costs and infrastructure choices and reduce dependence on a single model provider, cloud or hardware platform.

Version 1.0 adds support for quality of service across multiple tenants, zero-downtime operations and serving patterns tailored to AI agents. Flow controls and prioritization are designed to prevent bursts of agent activity from overwhelming services that need fast responses, such as coding assistants used by people.

The release “adds the maturity to run it like any other critical service: a stable [application programming interface], multiple tenants with competing priorities sharing the same infrastructure and upgrades that don’t interrupt operations,” said Carlos Costa, distinguished engineer at IBM Research and a project co-founder and lead maintainer.

Workloads can behave differently even when they reach models through the same endpoint. An agent may issue many requests, pause to use an external tool and return for more model calls. The framework can keep requests directed to appropriate infrastructure across multiple turns, increasing opportunities to reuse previously processed context. That matters because agents repeatedly consult the same code, documents, instructions and conversation history while completing assignments.

A routing system directs requests toward replicas that already hold relevant cached context. Another capability moves cached information among graphics processing unit memory, CPU memory and storage, preserving reusable information when GPU capacity becomes constrained. IBM said these techniques were previously available, but version 1.0 integrates them with operational controls, along with methods that separate prompt processing from response generation, optimize work batches and distribute large mixture-of-experts models across accelerators. IBM cited a recent agentic coding test in which the framework’s router captured 97% of available cache reuse.

Ruchir Puri, IBM Research’s chief scientist, vice president of IBM’s Technology & Technical Community and IBM Fellow, said in e-mailed comments that open large language models and llm-d can greatly lower inference costs in computationally demanding problems such as designing hardware for quantum error correction, helping achieve an 11.8X efficiency advantage. “Llm-d 1.0 extends these savings by delivering the operational readiness needed for enterprises to extract more out of their computational capacity,” he said.

In a separate demonstration, IBM and Red Hat said they ran 3,000 software-engineering agents concurrently on self-hosted infrastructure at a cost five to 10 times lower than serving the same open model through a hosted service. IBM said the economics become more significant as agents perform multiple rounds of reasoning, tool use and revision for a single assignment, because reprocessing identical material at each step consumes additional computing resources.

Llm-d works with inference engines including vLLM and SGLang, supplying distributed routing and operational capabilities around model servers. It is currently hosted as a sandbox project by the Cloud Native Computing Foundation.

“It delivers high-performance, distributed inference with the reliability and scale enterprises need within an operational model they already know,” said Stephen Watt, vice president in Red Hat’s chief technology officer’s office.

Editor's Summary

IBM and its collaborators released llm-d 1.0, adding multi-tenant service controls, zero-downtime upgrades and agent-specific routing to support self-hosted AI inference. The project is backed by IBM, Red Hat and Google and works with engines such as vLLM and SGLang under the CNCF sandbox. IBM and Red Hat also reported running 3,000 software-engineering agents at a lower cost than a hosted service.