NVIDIA Unveils Switchyard, an Open-Source Rust Proxy for Multi-Provider LLM Routing

Switchyard lets clients keep their native API. The proxy decodes incoming requests into provider-neutral Rust types, runs a routing algorithm to pick a backend, re-encodes the request in that backend's own wire format, calls it, and translates the response, including streaming events, back into the shape the client expects. The server accepts three inbound formats: OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages. Any of the three can address any route, and each configured LLM client selects one upstream format of its own, so the agent's API and the backend's API no longer have to match.

NVIDIA documents three ways to run Switchyard. The launcher path targets coding agents and is installed with uv tool install --python 3.12 "nemo-switchyard[cli]", then used through commands such as switchyard launch claude or switchyard launch codex. The server path installs a standalone proxy with cargo install --locked switchyard-server, validates a config with --dry-run, and serves on a host and port chosen by the user. The library path, switchyard-libsy, embeds the routing algorithms in a Rust application without owning an HTTP stack and never calls a model itself; the algorithm decides which target to use and hands every model call back to the caller.

Four routing algorithms are available. Passthrough sends every request to one target. Random splits traffic across targets using optional relative weights and an optional reproducible seed, supporting A/B and cost experiments. Llm_classifier calls a classifier target for a capability verdict, then routes to a weak or strong target; it requires base_threshold, and can be tuned with min_confidence, capability_elevated_floor and session_affinity. Setting mode = "escalation" runs every turn on the weak tier first and lets a judge decide whether to rerun it on the strong tier. Stage_router scores tool-result and agent-progress signals from recent turns to pick a capable or efficient target, avoiding an extra classifier call on most turns. Strong, weak, capable, and efficient are roles inside a route, not fixed properties of a model; the same upstream model can serve different roles in different routes.

Switchyard exposes observability through GET /metrics, which returns Prometheus text from the server's process-wide OpenTelemetry provider. The metric families cover requests, errors, model-call latency, full-turn latency, prompt, completion, cached, cache-creation and reasoning tokens, plus upstream HTTP attempts by outcome and code. A dedicated metric, switchyard_routing_overhead_ms, reports the algorithm's run time minus the model call that served the request. An optional --routing-log-file appends a JSON record per completed response, and GET /v1/routing/session-stats returns per-session call and token totals.

Configuration uses a TOML deployment with three layers: llm_clients define base URL, wire format, credential environment variable and retry policy; targets bind one upstream model ID to a client; and routes expose one client-visible model ID with its algorithm. Secrets never sit in the config file, since api_key_env only names an environment variable. max_retries defaults to 2 and covers transport failures, timeouts, HTTP 408/429 and 5xx responses. Documentation is available at docs.nvidia.com/nemo/switchyard.