Respan launches open-source Rust AI gateway built around plugins

Respan said the gateway is decoupled so that providers such as OpenAI, Anthropic and Gemini; protocols including chat, realtime voice and MCP; modalities such as images and videos; and reliability or security features including smart routing, fallbacks and PII redaction are all plugins. Instead of packaging many providers and features into one gateway, Respan offers a small core and public interfaces that teams can use to assemble the AI infrastructure their applications need. Users can add their own features as plugins without changing the core, the company said.

The launch follows Respan's experience building on LiteLLM since 2024. Respan said its gateway now serves thousands of organizations, each with different API key settings, routes and limits. To support those requirements, the company kept wrapping LiteLLM and adding features, and at one stage its wrapper code exceeded 100,000 lines. In January, a routine LiteLLM upgrade caused intermittent connection errors and the company rolled back, Respan said. It has not determined the cause, and since then it has not upgraded LiteLLM directly, instead manually backporting upstream fixes.

Respan said those experiences led it to conclude that continuing to patch LiteLLM was not sustainable and that it had to redesign the gateway. Inspired by Nginx, the company decided an AI gateway should have a very small core responsible only for dispatching events. It also wanted to make adding features easier without breaking existing ones. Adding a provider is intended to be simple because provider-specific quirks are plugins; Respan said Mistral's provider file is 22 lines, and each quirk, such as Groq placing token usage in its own field, gets its own test.

The gateway's architecture hands each HTTP or WebSocket connection to the first client format that recognizes it. That format exposes the request as a two-way stream of typed input and output events. The core runs plugins in registration order, allowing them to choose a provider, filter or wrap either stream, make additional calls, return a response or refuse the request. A provider format then opens the upstream connection.

If the client and provider use the same format and no plugin changes the payload, the original bytes and protocol-specific fields can pass through unchanged, according to Respan. When formats differ, they translate through shared types for messages, tool calls, results, streams and usage. The stream model can represent both one-shot HTTP requests and long-lived bidirectional sessions.

Respan said the gateway handles tools, images and reasoning protocols automatically. An application based on the OpenAI Realtime API can also use Gemini Live through the gateway, though not every feature translates, and a simple chat application can use MCP tools configured in the gateway without implementing MCP itself. The gateway parses requests into a shared structure; when the client and provider use the same API, it preserves provider-specific fields in their native format even if the shared structure lacks a field for them, so the gateway can write them back instead of dropping them.

The project is open source on GitHub under the MIT or Apache-2.0 license. Respan said users do not need its hosted platform and can run the gateway locally, build it from source, add provider keys and swap the base URL in a supported client while keeping the API the client already uses. The software is still pre-alpha. Respan makes money from a hosted platform that runs the gateway with native observability and evals, along with unified billing. The company said it wants developers to contribute plugins for routing, tools, filtering, observability and security, and that a PLUGINS.md file in the repository explains how to write one.