AI Gateway

What it is#

The platform already governs every model call it makes. Until now only callers that spoke its own request shape could reach an agent. A gateway key opens the OpenAI shape without opening anything else: it is minted by one user, reaches only that user's agents and the models that user may call, pays from that user's budgets, and is traced and audited under that user's name. This is the inbound half of the gateway; the outbound half — routing the platform's own traffic through LiteLLM, Portkey or Helicone — is the setting beside it on the same tab and is unchanged.

Keys#

Mint a key under Integrations → LLM Gateway → API access. The plaintext gw_… is shown once; the row keeps a hash and the first characters. Revoking is immediate and permanent, and every change to a key is audited by trigger.

Calling it#

Base URL https://<your host>/api/v1/. GET /models lists what the key may name; POST /chat/completions runs one turn, streamed as server-sent events when stream is true.

curl https://<your host>/api/v1/chat/completions \
  -H "Authorization: Bearer gw_..." \
  -H "Content-Type: application/json" \
  -d '{"model": "agent:Support triage", "stream": true,
       "messages": [{"role": "user", "content": "A customer reports a failed payment"}]}'

from openai import OpenAI
client = OpenAI(base_url="https://<your host>/api/v1/", api_key="gw_...")
reply = client.chat.completions.create(
    model="openrouter/openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "What is a lakehouse?"}],
)
print(reply.choices[0].message.content)

• Taken from the request: model, messages (system and developer messages become the instruction; tool messages are dropped, because an agent runs its own tools), stream, temperature, max_tokens, and stream_options.include_usage. Client-supplied tools are ignored: an agent's tools are the ones its owner configured, and a bare model has none.
• Added to the reply: an agentswarms object with the trace id, the model fallen back from, citations, tool calls and any guardrail note; headers X-Trace-Id, X-Gateway-Model (the model that answered), X-Gateway-Fallback and X-Gateway-Cache.
• No memory between calls. The caller holds the conversation and replays it, the way the OpenAI API works; a key is not a person.

The semantic layer#

A key with the metrics scope answers governed questions without an agent in between. GET /metrics lists the semantic models the key may query, described for a client: names, labels, each metric's aggregation and format, each dimension's type, synonyms and sampled values, the declared parameters, the hierarchies, and the grains and period comparisons a time dimension accepts — never the SQL behind them. A shared model with a restricted grant is listed with its access_note and without the masked fields. POST /metrics/query runs one query in the same structured shape the runner and the metric_query agent tool use.

curl https://<your host>/api/v1/metrics -H "Authorization: Bearer gw_..."

curl https://<your host>/api/v1/metrics/query \
  -H "Authorization: Bearer gw_..." \
  -H "Content-Type: application/json" \
  -d '{"model": "revenue", "metrics": ["net_revenue", "orders"],
       "dimensions": ["region", "order_date"], "grains": {"order_date": "month"},
       "filters": [{"field": "order_date", "op": "last_n_days", "value": 180}],
       "order_by": [{"field": "order_date", "dir": "desc"}], "limit": 500}'

• The body: model (a name or id from GET /metrics), metrics, dimensions, filters (field, op, value — the comparison ops and the relative-date windows such as last_n_days, this_month, ytd), grains (dimension → day, week, month, quarter, year or a fiscal grain), order_by, limit, compare (prior_period, mom, yoy) and params for declared parameters. A request that fails validation answers 400 naming the field; one the compiler refuses answers 400 in the compiler's words.
• The answer: { object: "metrics.result", model, columns, rows, row_count, truncated, sql }, with access_note for a restricted share, rollup when a pre-aggregate answered and resolution_notes when a synonym was resolved. Rows stop at the request's limit, the instance cap (AI_GATEWAY_METRICS_MAX_ROWS, 10,000 by default) or the semantic layer's own ceiling of 10,000 rows, whichever is smallest; truncated says whether more matched.
• Governance is the semantic layer's own. The query runs as the key's owner through the same chokepoint as a dashboard tile or an agent's metric_query call — the owner's models plus the ones IAM shares with them, a grantee's row filters and field masks rewritten into the query, the data read as the model owner — and audits metric.query with via: gateway, the key, the compiled SQL and a digest of the result. The key's allow-list narrows that access and never widens it. A metrics query makes no model call, so budgets are not touched. See Semantic layer.

Fallback#

When the requested model fails with a provider error — throttling, exhausted credits, a timeout, a 5xx — the gateway tries the key's chain, then the instance-wide chain, each entry once, and answers with the first that works. For an agent only the model changes; the prompt, tools and knowledge stay the agent's. A caller's own mistake is never retried, and a policy refusal of the model the caller asked for answers 403; a forbidden model that only appears in a chain is skipped. A fallback happens only before any token has reached the caller. Every switch is audited as gateway.fallback.

Governance#

• Identity. The turn runs on the chat route's internal channel as the key's owner; data tools read what the owner may read.
• Model rules. IAM allow-lists apply to the model that answers, requested or fallen back to. See Access control.
• Budgets. The owner's personal and group budgets apply, and the key's own monthly ceiling on top; every trace made through the key carries the key as its cost scope, so the ceiling is measured. Over budget answers 429 insufficient_quota.
• Audit. gateway.chat for every turn — including whether the cache answered it — gateway.cache.clear when an owner empties the cache, gateway.fallback for every switch, gateway.access.denied for a revoked, expired, out-of-scope or throttled key with the caller's address; agent turns also audit agent.chat. A metric query audits metric.query with via: gateway, the key, the compiled SQL and a digest of the result.
• Traces. Each turn is an execution trace under the owner with the agent's name, so Observability shows gateway traffic beside everything else.

Semantic cache#

The same question asked twice costs twice. A key can switch on a cache that matches on the meaning of a question rather than its bytes, so “what was revenue last quarter” and “how much revenue did we make last quarter” are one question, answered once. It is off unless a key turns it on — on the key when it is minted, or from the Cache column afterwards. Nobody inherits one by upgrading, because a cache that answers a question with a nearly identical question's answer is a correctness risk that has to be chosen deliberately.

An entry belongs to one owner, one target (the agent id, or provider/model for a bare model call) and one system instruction. No answer crosses a user, an agent, or a differently instructed run of the same agent. There is no global cache and no way to ask for one.

• Never cached: a conversation with a history. “And for Europe?” means nothing without what came before it, and matching on the last message alone would answer it with whatever the last person who asked that got.
• Never cached: a turn above the temperature ceiling. A high temperature asks for variety; serving it from cache answers a different question than the one asked.
• Never cached: an answer that used a tool. It read something live, and freezing it for a day would serve yesterday's number tomorrow.
• What you get back. Every reply carries X-Gateway-Cache — hit, miss, skip (the cache is on but this call was not eligible) or off. A hit sets X-Gateway-Model to the model that wrote the answer, puts cached in the agentswarms object with the similarity and the question it matched, and reports zero usage, because a hit spends nothing at the provider.
• It still costs one embedding of the question, on the owner's connected embedding provider — the same one the knowledge bases use, roughly three orders of magnitude less than a completion. With no embedding provider connected, every call simply misses.

The cache is consulted after every check that can refuse a call — inactive key, wrong scope, forbidden model, exhausted budget — and before the first provider call. A refused key stays refused whether or not the answer happens to be sitting in a table. A hit audits gateway.chat with cache: hit, the similarity and zero tokens; flipping the switch on a key audits through the key table's own trigger; and emptying the cache from the same card audits gateway.cache.clear.

Limits#

How this compares#

LiteLLM and Portkey are gateways in front of model providers: keys, routing, fallbacks and spend for raw models. This endpoint is a gateway in front of your agents as well as your models, and it inherits the platform's governance instead of carrying its own copy. If you already run one of those gateways, keep it as the outbound route and put this endpoint in front of the agents.

Troubleshooting#