Skip to content
Edit page
Copy page

Configure App LLM Proxy

An App LLM Proxy exposes a managed endpoint that your Gen AI applications or agents use to reach an LLM Provider. Once deployed, the proxy handles authentication and enforces any guardrails you configure. While the provider can be called directly, the proxy lets you create specialized endpoints for different applications, agents, or environments with independent keys, guardrails, and access controls on top of the same provider.

This guide walks you through creating an App LLM Proxy, optionally attaching guardrails, and deploying it to a gateway.

Prerequisites

Before You Begin

Create a New App LLM Proxy

  1. Navigate to LLM > App LLM Proxies in the left navigation menu.

  2. Click Create App LLM Proxy.

  3. Fill in the required proxy details:

    1. Name* (Required): Enter a unique name for the proxy (e.g., support-chat-api, sales-agent-proxy). The Proxy ID is auto-generated from the name (lowercase, hyphen-separated).

    2. LLM Provider* (Required): Under Provider Configuration, select the LLM provider this proxy routes to. This becomes the proxy's primary provider — use + Add Additional LLM Provider to attach more, which is covered in Route to Several Providers.

    3. Version* (Required): The version is pre-filled (e.g., v1.0). You can edit this if needed.

    4. Description (Optional): Add a brief description to identify the Gen AI application or agent use case this proxy is created for.

    5. Context (Optional): The context path is derived from the name as you type, for example /test/multi-provider-demo. Edit it if you need a different base path for the proxy's endpoints (normalized with a "/" prefix).

  4. Optionally expand Advanced Configurations to set the Inbound Interface — the request format the proxy accepts from client applications. Leave it alone unless you need a format other than the selected provider's own; see Choose the inbound interface.

  5. Click Create Proxy to save the proxy.

Add Guardrails (Optional)

You can attach policies and guardrails to your proxy that apply to all requests:

  1. In the Guardrails tab, click + Add Guardrail.

  2. A sidebar will open showing available guardrails and policies.

  3. Click on a guardrail to select it and configure its settings.

  4. Click Submit to attach it to the proxy.

Advanced Settings

Each guardrail includes advanced configuration options that allow you to fine-tune its behavior. After selecting a guardrail, you can configure these settings before attaching it to the proxy.

Info

Learn more about available guardrails in the Guardrails Overview. For the full list of policies and their specifications, visit the Policy Hub.

Deploy Proxy to Gateway

After creating your proxy, you must deploy it to a gateway before it can be used.

Required Step

Your proxy will not be functional until it is deployed to at least one gateway.

  1. Click the Deploy to Gateway button in the top-right corner of the proxy details page.

  2. The Deploy to Gateway page opens, showing all available gateways. Use the search bar to find a specific gateway.

  3. Each gateway card displays:

    • Gateway name and status (Active or Not Active)
    • Current Deployment identifier (if previously deployed)

  4. Click Deploy next to the gateway you want to deploy to.

  5. Once deployed, expand the gateway card to view deployment details:

    Field Description
    Deployment Status Active or Inactive
    Deployment ID Unique identifier for the deployment
    Deployed Time since deployment
    Stop Button to stop an active deployment

  6. The API Deployment History panel on the right shows the deployment timeline:

    • Deployment identifier and timestamp
    • Status tags (Latest, Deployed)

To stop a deployment:

Click the Stop button next to an active deployment to undeploy the proxy from that gateway.

Click Back to App LLM Proxy to return to the proxy details page.

Deployment Status

Monitor deployment progress on this page. Changes take effect within seconds of successful deployment.

Get Started

Once the proxy is deployed, the proxy details page shows the Get Started panel on the right.

Invoke URL

Select a gateway from the Gateways dropdown to see the base URL for calling this proxy through that gateway.

The URL follows the format:

https://{gateway-host}/{proxy-name}

To call a specific resource, append the resource path:

https://{gateway-host}/{proxy-name}/chat/completions

App LLM Proxy Keys

Generate an API key to authenticate requests to the deployed gateway.

  1. Click Generate API Key in the Get Started panel.
  2. Copy and save your API key immediately.

Important

API keys are only displayed once. Store it in a secure location immediately — you will not be able to retrieve it again.

Deployed Gateways

The Deployed Gateways section lists all gateways this proxy is deployed to, along with the deployment status.

Next Steps