Add a provider

A provider is a name, a dialect and a URL. That reaches OpenAI, a self-hosted vLLM, a vendor nobody has heard of and an internal proxy alike. For the vendors Gate knows, a recipe writes the entry out to copy.

The whole thing

Add this configuration:

providers:
  - name: vllm
    dialect: openai
    base_url: http://vllm.inference.svc.cluster.local:8000
    api_key: ${secret:VLLM_API_KEY}
    models: ["llama-*", "qwen-*"]

Set these fields:

• name identifies the provider in routes, rules, budgets, logs, and metrics. Required.
• dialect names the protocol the endpoint speaks, openai (the default) or anthropic. A server that serves /v1/chat/completions is openai. One that serves /v1/messages is anthropic. There is no third value, and no way to declare one from config. A dialect ships with a translator, a probe, and a coverage fixture.
• base_url is the root the request path is appended to. Any absolute URL, including http for your own servers.
• api_key (or credential) is sent upstream in the header the dialect expects, in place of the caller’s own credential. A named credential can instead sign with SigV4 (aws) or send a client_credentials token as the bearer (oauth). It is required whenever auth is set even if the server ignores it. Without one, the gateway would forward the caller’s credential upstream. If you omit auth, omit this field to forward the caller’s credential unchanged.
• models gates what routes here, exact names or trailing-* prefixes. The gateway also uses this list for GET /v1/models.

Send a request and watch it arrive:

$ curl -s localhost:8484/v1/chat/completions \
    -H "Authorization: Bearer $VIRTUAL_KEY" \
    -d '{"model":"llama-3.1-8b","messages":[{"role":"user","content":"hi"}]}'

Check GET /admin/v1/backends to inspect the resolved URL and credential.

Start from a recipe

Gate ships one complete entry per vendor it knows: OpenAI, Anthropic, Bedrock, Gemini, Groq, Mistral and the rest. A recipe is the entry above with the vendor’s endpoint, credential reference and quirks filled in. Copy it from the provider recipes page, or in the console choose New provider and pick the vendor under Start from. Either way the whole entry lands in your document and every field is yours to edit.

A recipe is a starting point and nothing more. Nothing in a saved configuration refers back to it, and the gateway reads no recipe at runtime, so a vendor moving an endpoint is an edit to your document like any other. The recipes reference page, the console and GET /admin/v1/recipes/providers all read one embedded file.

A vendor with two surfaces

Anthropic serves its Messages API and an OpenAI-compatible endpoint. Bedrock serves both, split by model family. Each surface is one entry in its own dialect, and the recipes carry both:

providers:
  - name: anthropic
    dialect: anthropic
    base_url: https://api.anthropic.com
    api_key: ${secret:ANTHROPIC_API_KEY}
    models: ["claude-*"]
  - name: anthropic-openai
    dialect: openai
    base_url: https://api.anthropic.com
    api_key: ${secret:ANTHROPIC_API_KEY}
    models: ["claude-*"]

Two entries may claim one model when their dialects differ, and a request goes to the one that speaks its dialect, passed through byte for byte. Declare only the first and an OpenAI-dialect request is translated by the gateway instead, under on_fidelity_loss. Fidelity covers which to prefer and what each costs.

A provider behind an authorization server

An endpoint that takes a short-lived token rather than a static key names an oauth credential. The gateway runs the client_credentials grant, caches the token, refreshes it before expiry, and sends it as Authorization: Bearer whatever the provider’s dialect. Azure OpenAI under Microsoft Entra ID is the common case. It takes a token for the https://ai.azure.com/.default scope from the tenant’s token endpoint:

credentials:
  - name: entra
    oauth:
      token_url: https://login.microsoftonline.com/YOUR-TENANT-ID/oauth2/v2.0/token
      client_id: YOUR-APP-CLIENT-ID
      client_secret: ${secret:AZURE_CLIENT_SECRET}
      scopes: ["https://ai.azure.com/.default"]
providers:
  - name: azure-openai
    dialect: openai
    base_url: https://YOUR-RESOURCE-NAME.openai.azure.com/openai
    credential: entra
    models: ["gpt-*"]

The app registration needs the Cognitive Services OpenAI User role on the resource. A token the authorization server will not issue refuses the request with 502 upstream_auth_failed before anything is sent. The server’s own error goes to the gateway log and not to the caller, and kirto_gate_auth_failures_total{reason="upstream_oauth_failed"} counts it.

Claude on Vertex AI

A provider with a vertex block reaches Claude on Google Cloud’s Vertex AI. Vertex AI serves the Anthropic Messages API with two differences: the model is in the URL instead of the body, and the body carries anthropic_version. The gateway makes both edits on the way out. Claude Code on /v1/messages reaches it passed through, and an OpenAI SDK on /v1/chat/completions reaches it translated. The entry takes dialect: anthropic and a gcp credential:

credentials:
  - name: vertex
    gcp: {}
providers:
  - name: vertex
    dialect: anthropic
    credential: vertex
    vertex:
      project: my-project
      region: global
    models: ["claude-*"]

region picks the host. global is aiplatform.googleapis.com, us and eu are the multi-region hosts, and a region such as us-east5 is its regional host. Leave base_url empty. A base_url that is set must be an https host under googleapis.com, such as a Private Service Connect endpoint.

gcp: {} uses Application Default Credentials on the node: GOOGLE_APPLICATION_CREDENTIALS, the gcloud well-known file, or the metadata server, which is how GKE Workload Identity reaches the pod. A service account key instead goes in credentials_json: ${secret:VERTEX_SA_KEY}. Only the service_account key type is accepted, because the other types can direct the loader at URLs and local files. Either identity needs permission to call Vertex AI predictions on the project. A gcp credential is refused on a provider without a vertex block. Under ADC it is the node’s own Google identity, and the check keeps it on Google’s hosts.

What reaches Vertex AI, and what does not:

• The Messages API and /v1/messages/count_tokens are served. Any other path, such as /v1/models or message batches, is refused with 422 route_not_served.
• Model IDs are Vertex AI’s. Older ones carry a dated version after an @, as in claude-sonnet-4-5@20250929. A model name with any character besides letters, digits, ., _, - and that one @ is refused with 400, because it is placed in the URL.
• An error Google raises itself, such as a missing permission, a quota or an unknown project, arrives in Google’s error shape. A passed through request gets it verbatim. A translated one keeps the status and message.
• An llm guardrail detector cannot name a Vertex AI provider yet.

Three things that bite

• A server that rejects unknown fields. Gate adds stream_options.include_usage to streamed OpenAI-dialect requests so usage can be metered, and removes the chunk it produces. A server that returns 400 on a field it does not know needs stream_usage: false, and its streamed requests then settle against the reservation estimate. A server that rejects a named field on every request, as Groq does with logit_bias, takes remove_request_fields listing it.
• A path prefix the server does not serve. The incoming path is appended to base_url as it is. A server that serves /chat/completions without the /v1 takes strip_path_prefix: /v1, as the Gemini and Perplexity recipes do.
• region. It is the SigV4 signing region, read only when the credential is aws, and refused on any other provider. The endpoint’s own region is spelled in base_url.

From the console or the API

In the console, New provider offers a recipe to start from or a blank entry: a name, a dialect and a URL. Either way the object lands in the source you chose and is validated as the whole deployment on write.

Through the API it is one object write, naming the source only because the provider does not exist yet:

$ curl -s -X PUT 'localhost:8485/admin/v1/config/providers/vllm?source=team-a' \
    -H "Authorization: Bearer $ADMIN" -H 'If-None-Match: *' \
    -H 'Content-Type: application/yaml' \
    --data-binary $'name: vllm\ndialect: openai\nbase_url: http://vllm:8000\napi_key: ${secret:VLLM_API_KEY}\nmodels: ["llama-*"]\n'

See manage the configuration through the API for the precondition and what a refused write looks like.

Related

• Provider recipes, every vendor entry Gate ships.
• Passthrough, what “byte-for-byte” means when the dialects match.
• Fidelity, how the entry is chosen and what a translation costs.
• Configuration reference, every field on providers[].