LLM Gateway

What it does

LLM Gateway is an open-source API gateway that routes requests to 250+ models from 40+ providers, including OpenAI, Anthropic and Google, through one OpenAI-compatible API. It adds cost tracking, caching, failover, spend caps and guardrails, and can run in the cloud or self-hosted. It suits developers and teams building multi-model applications.

LLM Gateway sits between your app and model providers. You keep your existing OpenAI SDK code and change the base URL, then send requests to 250+ models from 40+ providers. The gateway handles routing, caching and failover. When a provider errors, times out or rate-limits you, it retries up to twice on another provider for that model. Requests pinned to one provider stay there. The dashboard shows requests, tokens, total spend and average cost per 1K tokens across 7 or 30 days, with breakdowns by provider and model. Spend caps can be set per key or member, and requests stop once a cap is reached. Guardrails, SSO, audit logs and a 99.9% SLA sit on Enterprise. The core is open source under AGPLv3 and the enterprise code is source-available. Self-hosted enterprise features need a license key. Free models are limited to 20 requests per minute. Credit purchases carry a 5% platform fee, and non-US cards may add a 1.5% card fee.