llm-d 1.0 ships multi-tenant QoS and zero-downtime upgrades
llm-d 1.0 shipped October 20 with multi-tenant QoS and zero-downtime upgrades for self-hosted inference. Disaggregated setups need LeaderWorkerSet v0.11.1+.
llm-d is a Kubernetes-native framework for distributed LLM inference. It does not replace model servers such as vLLM or SGLang. It sits in front of them and handles routing, scheduling and rolling upgrades. The project shipped llm-d 1.0.0 on October 20, and GitHub timestamps the release at 02:30 UTC that day. The release is less about new techniques than about turning existing ones (prefix-aware routing, KV-cache offload, prefill/decode disaggregation) into a service that runs long-term: multi-tenant flow control, router hot standby, zero-downtime rolling upgrades and session affinity for agents.
Key points
- Release and license: v1.0.0 is marked Latest on GitHub and is licensed under Apache 2.0. The project reports more than 500 contributors from over 25 organizations.
- Operations: multi-tenant flow control still defaults to first-come-first-served, so operators must turn it on. A hot-standby router takes over scheduling when the active instance fails or is replaced, and DisaggregatedSet rolls prefill and decode replicas as one unit.
- Upgrade gate: the prefill/decode disaggregation guide requires LeaderWorkerSet v0.11.1 or newer with DisaggregatedSet enabled. Builds of the llm-d-rocm image are deprecated.
Background
Red Hat, IBM and Google are the lead maintainers, and IBM’s post names CoreWeave and NVIDIA as co-founders. In March 2026, IBM Research, Red Hat and Google Cloud announced at KubeCon Europe that llm-d would move to the Cloud Native Computing Foundation as a sandbox project, so that status is not new in 1.0.
Nor is 1.0 introducing new techniques. IBM says the ideas predate this release and that what is new is the integration and operational maturity around them. SiliconANGLE also says the techniques were already available. This article therefore calls 1.0 an integration release, not a set of new features.
English search suggestions include “llm-d vs vllm”, so the relationship is a common question. llm-d treats vLLM and SGLang as model servers and adds routing and operational features around them. The same day, DeepSeek published DeepInfer, an inference framework. Its README says it covers offline inference only, with no KV-cache system or prefill/decode load balancing for a full online service (our same-day report has the details).
The facts
The release notes group the changes into a few areas. Multi-tenant QoS uses priority- and fairness-aware flow control, with a queue per workload or tenant. Under overload, a probabilistic admission controller sheds load gracefully instead of failing hard.
Zero-downtime operations rest on two changes: the Endpoint Picker’s hot standby, and DisaggregatedSet’s rollout screener. The screener is an alpha plugin that must be enabled in the EPP configuration. During a rollout it picks one revision per request, weighted by Ready pods, and a revision gets no traffic until each required role has a Ready pod. In a reference run cited by the safe-rollout guide (DeepSeek-V2-Lite on H200), 61% of requests paired across revisions without the screener.
For agents, a session-state-producer plugin keeps follow-up turns on one backend until it saturates. The Responses API also accepts previous_response_id, so the server keeps context across turns.
The release’s component table shows the version changes that matter for operators:
| Component | 1.0.0 | Previous |
|---|---|---|
| Endpoint Picker (llm-d-router-endpoint-picker) | v1.0.0 | v0.11.0 |
| Disaggregation sidecar (llm-d-router-disagg-sidecar) | v1.0.0 | v0.11.0 |
| LeaderWorkerSet, with DisaggregatedSet | v1.0.0 | v0.11.0 |
| llm-d-rocm image | Deprecated | v0.10.0 |
Source: the component table in the GitHub release notes, published October 20, 2026.
The notes also align component versions across the project and aim to do the same for future major and minor releases. AMD users should plan to move from llm-d-rocm to upstream vLLM images. For this release, the vLLM v0.32.0-rc.2 ROCm image was copied into llm-d’s own registry so it would not be garbage-collected.
What others say
The maintainers tell the same story. Carlos Costa of IBM Research, a co-founder and lead maintainer of llm-d, says 1.0 adds “the maturity to run it like any other critical service”: a stable API, multiple tenants with competing priorities, and upgrades that don’t interrupt operations.
The one independent English report we could read in full is SiliconANGLE’s, by Paul Gillin. It attributes its figures to IBM, and we found no sign they were tested independently. It also quotes IBM Research’s chief scientist, Ruchir Puri, in e-mailed comments on an “11.8X efficiency advantage”. That is an efficiency measure, not the 11.5x cost gap in IBM’s October 13 post, so we keep the two apart.
Google Cloud’s post, published October 20 UTC (the page is dated October 21), reports, as vendor figures, that prefix-aware routing plus host-memory offload cut median time to first token by more than 85% on an agentic coding workload on TPU 7x (Ironwood), and raised total throughput 2.4x over a plain Kubernetes Service. It names neither the model nor the test length. For a different TPU result on a different workload, see our Inferact report.
Community discussion is thin. The two Hacker News submissions about llm-d 1.0 each have one point and no comments, and we found no substantive independent critique.
Reading the numbers
Every figure here comes from a vendor or maintainer, and they are not directly comparable. The table lists what each one was measured under.
| Figure | Source and date | Conditions stated | Not stated |
|---|---|---|---|
| 97% cache reuse | IBM, October 20 | One agentic coding test | Hardware, model, duration |
| 97% (figure caption) | llm-d benchmark, July 22 | GLM-5.2-FP8 on H200; DEP8 1P1D on 16 GPUs; sessions that fit a 142K-token context; concurrency 64; 91.4% prefill hit rate against a 93.9% ceiling | Whether it is the same test as IBM’s |
| 5–10x lower cost per token | IBM, September 8 | GLM-5.2 (a ~753B-parameter MoE, ~39B active) self-hosted on 544 H100s at current cloud rental rates, against equivalent commercial API pricing; output reached 6.60 million tokens per minute at 3,000 concurrent coding agents, near the serving limit | Rental prices, utilization, the calculation |
| 11.5x cost gap | IBM, October 13 | Quantum error-correction hardware design; 100 hypotheses per search; $299 all-proprietary versus $26 all-open; speedups of 52x for open (HLS estimate) and 204x for proprietary; one search run each | Model and hardware names; inference cost only |
The October 20 IBM post says IBM and Red Hat ran 3,000 software-engineering agents concurrently at five to ten times less than hosted pricing for the same open model. It gives no test conditions and does not say whether it is the September run. The project blog puts 98% cache hits, $0.11 per million tokens and the 11.5x in one sentence with the 3,000-agent figure, but all three come from the chip-design workload.
The July 22 benchmark also prices GPU rental. At an assumed $2.71 per H200 GPU-hour, two 64-GPU layouts come to $0.05 to $0.08 per million input tokens and $1.92 to $3.55 per million output tokens. Its own caveat matters most: “These figures cover GPU rental only, not the full cost of operating an API service.”
Our take
For platform teams that run inference on their own GPUs and serve several teams, or many agents, from one endpoint, 1.0 matters. Tenant isolation, a standby router and rolling upgrades of split prefill and decode pools were things teams had to build themselves. Teams that only call hosted APIs will see no direct change.
Three things to check before upgrading. The LeaderWorkerSet version: the release notes warn that “almost all guides will break without v0.11.1+.” ROCm: if you still run llm-d-rocm, plan the move to upstream vLLM. Flow control: it does not change behavior by default, so the multi-tenant benefit needs explicit configuration.
The cost figures and “zero-downtime” both rest on the project’s or IBM’s own word, as the table shows, and the rollout screener behind safe rollouts is still alpha. We found no independent upgrade test, and the release notes open by saying “not all of that was accomplished” against the release goal.
The documents also disagree on one point. The README’s v0.5 changelog, dated February 2026, lists “active-active HA”. The 1.0 blog puts active-active router configurations on its next-steps list and describes the 1.0 router as hot standby. We could not tell which one 1.0 delivers, so teams that depend on router high availability should confirm it on GitHub first.
How to try it
The shortest route is the official Quickstart, which covers the Standalone mode of v1.0. You need kubectl, helm and a Kubernetes cluster; the default model server uses NVIDIA GPUs. Clone the repository and switch to the release-1.0 branch:
git clone https://github.com/llm-d/llm-d.git && cd llm-d && git checkout release-1.0
export REPO_ROOT=$(realpath $(git rev-parse --show-toplevel))
source ${REPO_ROOT}/guides/env.sh
export GUIDE_NAME="quickstart" NAMESPACE=llm-d-quickstart
Then install the Gateway API Inference Extension CRDs, create a Hugging Face token secret, deploy the router with helm (it includes a built-in Envoy proxy), and apply the vLLM model server with kubectl apply -k. The default is two replicas of Qwen/Qwen3-32B; the guide also offers a CPU alternative with Llama-3.2-3B-Instruct. This is not a one-minute setup. The guide says model pods “may take several minutes” to start while the weights download.
What to watch
The project blog lists active-active router configurations, failover and migration for inference requests, broader vLLM wide expert parallelism on managed Kubernetes, multimodal and audio serving, and deeper GitOps integration with tools such as ArgoCD. The blog presents these as work in progress, not shipped features.