Skip to main content
logoTetrate Agent Operations DirectorVersion: Latest

Redirecting LLM Traffic

Learn how to configure the Director gateway to intercept and manage LLM API requests, enabling effective governance without requiring significant application changes.

Overview​

The Tetrate Agent Operations Director uses a Gateway component to intercept and control LLM API traffic. This approach allows organizations to implement governance and cost controls for resources and consumers without extensive modifications to existing applications.

The Gateway sits between your applications and LLM providers, providing:

  • Request interception and routing
  • Usage metering and quota enforcement
  • Credential management and security
  • Logging and telemetry collection

Configuration Steps​

Step 1: Prepare Gateway Deployment​

Before configuring traffic redirection:

  1. Verify the Gateway is deployed and accessible
  2. Ensure network connectivity between applications, gateway, and LLM providers
  3. Confirm that appropriate credentials are configured for target providers

Step 2: Configure Resource Definitions​

Associate the gateway with specific LLM resources:

  1. Navigate to the Resources section in the Director console
  2. Select or add the target LLM resources (e.g., OpenAI, AWS Bedrock)
  3. Verify API endpoints and authentication settings
  4. Associate resources with the appropriate providers

Step 3: Set Up Budget Controls​

Before redirecting traffic, establish governance policies:

  1. Create or select appropriate budget templates
  2. Configure budgets for specific consumer/resource pairs
  3. Set initial mode to Monitored

Step 4: Update Application Configuration​

Modify applications to route requests through the Gateway:

  1. API Endpoint Change: Update application configuration to point to the gateway endpoint

    • Original: https://api.openai.com/v1/chat/completions
    • New: https://ai-gateway.example.com/openai/v1/chat/completions

  2. Authentication: Continue using existing API keys (gateway will handle provider authentication)

Step 5: Verify Gateway Traffic​

After configuration, confirm traffic is flowing correctly:

  1. Make test requests from the application
  2. Check gateway logs for successful request proxying
  3. Verify metrics appear in the Director dashboard
  4. Confirm that usage is being tracked against the appropriate budget