Open-Source Model Gateway Offers No-Markup Routing and Custom Model Training

Written in Rust and designed for concurrency, the gateway handles configuration differences across models and providers, including streaming formats, tool calls, model parameters, rate limits and error behavior. According to the post, it connects to every major inference provider and supports more than 1,000 models, refreshed daily by a Codex agent that opens a pull request.

The project differs from similar services by being open source, taking no markup and allowing users to mix local models with a marketplace. Users can also opt in to let their traffic be used to train a custom model. 'Simple routing doesn't warrant a 10% token markup,' the team wrote.

To choose the best model for each request, the gateway uses standardized OTel traces to mine representative real tasks, then simulates rollouts for various models with text world models. An LLM judge evaluates the results, and a nearest-neighbor classifier fitted on prompt embeddings decides the optimal model. The team said this usually maps a better cost-quality Pareto curve than calling a single model, but acknowledged it is not perfect.

The simulations are also used to suggest cache-hit optimizations, recommend new models and train models. The software is open source, so it can be deployed on a user's own infrastructure or run on the hosted version, which the team says carries zero markup. Additional details on availability design are published on the project website.