East-West Traffic Management

Use the Tetrate Hosted Agent to expose services between Kubernetes clusters

Overview

The Tetrate Hosted Agent configures east-west gateways for cross-cluster service-to-service communication using an opt-in model (disabled by default). It supports two distinct exposure modes:

1. Namespace-Level Exposure: Expose ALL services in a namespace
2. Service-Level Exposure: Expose only specifically labeled services

Key Concepts

• Dual-Mode Support: Choose between namespace-wide or selective service exposure
• Opt-In Model: East-west disabled by default - requires explicit enablement
• Zero Agent Overhead: TSB automatically routes traffic - no per-service agent action
• Fully Customizable: Supports custom gateway configurations via namespace annotations
• Consistent Configuration: All gateway settings (name, namespace, cloud annotations) configured at namespace level only

Quick Start: Mode 1 - Namespace-Level Exposure (All Services)

Use this mode when you want to expose all services in a namespace across clusters.

1. Create Namespace with Namespace-Level Opt-In

apiVersion: v1
kind: Namespace
metadata:
  name: payments
  annotations:
    traffic.tetrate.io/global: "true"  # Exposes ALL services in namespace
  labels:
    tetrate.io/rev: default  # Required for workspace creation
    istio-injection: enabled

note

The tetrate.io/rev=default label is required for the Tetrate Hosted Agent to create a workspace for this namespace. This is controlled by the agent.kubernetes.triggerLabels configuration. Alternative trigger labels include istio-injection=enabled or istio.io/rev=default.

The agent then:

• Deploys east-west-gateway to tetrate-system namespace
• Creates WorkspaceSettings with empty exposedServices array
• TSB automatically exposes ALL services in the namespace

2. Deploy Services (No Labels Required)

apiVersion: v1
kind: Service
metadata:
  name: payment-api
  namespace: payments
  # No label needed - all services in namespace are exposed
spec:
  selector:
    app: payment
  ports:
  - port: 8080
    targetPort: 8080
    name: http

3. Access from Another Cluster

# From any pod in another cluster (same workspace)
curl http://payment-api.payments.svc.cluster.local:8080/health

Quick Start: Mode 2 - Service-Level Exposure (Labeled Services Only)

Use this mode when you want to expose only specific services across clusters.

How Service-Level Mode Works

Service-Level exposure works without requiring a namespace annotation. The agent automatically:

1. Detects services with the traffic.tetrate.io/global: "true" label
2. Deploys the east-west gateway using default settings (or namespace-level gateway configuration if provided)
3. Creates WorkspaceSettings with a serviceLabels selector that matches only labeled services

You can optionally add namespace annotations (like traffic.tetrate.io/eastwest-gateway-name) to customize the gateway deployment, but they are not required for basic service-level exposure.

1. Deploy Services with Global Label

apiVersion: v1
kind: Service
metadata:
  name: payment-api
  namespace: payments
  labels:
    traffic.tetrate.io/global: "true"  # Add this label to expose this service
spec:
  selector:
    app: payment
  ports:
  - port: 8080
    targetPort: 8080
    name: http
---
apiVersion: v1
kind: Service
metadata:
  name: internal-cache
  namespace: payments
  # No label - this service stays internal to the cluster
spec:
  selector:
    app: redis
  ports:
  - port: 6379
    targetPort: 6379
    name: tcp-redis

The agent then:

• Detects service with traffic.tetrate.io/global: "true" label
• Deploys east-west-gateway to tetrate-system namespace (using namespace-level config or defaults)
• Creates WorkspaceSettings with serviceLabels selector
• TSB exposes only labeled services (payment-api), not internal-cache

2. Access from Another Cluster

# From any pod in another cluster (same workspace)
curl http://payment-api.payments.svc.cluster.local:8080/health  # Works - service is labeled
curl http://internal-cache.payments.svc.cluster.local:6379      # Not exposed - no label

Choosing the Right Exposure Mode

Scenario	Mode	Annotation/Label
All services need cross-cluster access	Namespace-Level	Namespace annotation: traffic.tetrate.io/global: "true"
Only specific services need exposure	Service-Level	Service label: traffic.tetrate.io/global: "true"
Mixed: some exposed, some internal	Service-Level	Label only the services that need exposure
Temporary exposure for testing	Service-Level	Add/remove label without changing namespace
Entire microservices platform	Namespace-Level	Simpler - no per-service labeling needed

Key Differences:

• Namespace-Level: exposedServices field omitted (not set) in WorkspaceSettings → TSB exposes all services
• Service-Level: serviceLabels selector in WorkspaceSettings → TSB exposes only matching services
• Configuration Source: Both modes use namespace annotations for gateway configuration (name, namespace, cloud annotations)
• Precedence Rule: If namespace has annotation AND services have labels → namespace annotation takes precedence (namespace-level mode wins)

Mode Transition Safety

Switching from namespace-level to service-level is blocked to prevent production outages. This transition would unexpectedly stop exposing services that don't have the traffic.tetrate.io/global: "true" label. The agent preserves the existing namespace-level configuration and returns an error.

Safe transition path: Delete the namespace and recreate it without the annotation, ensuring services have the required label.

Switching from service-level to namespace-level is allowed (safe direction - expands exposure).

Advanced Configuration

Custom East-West Gateway (Works with Both Modes)

To use a custom gateway configuration:

apiVersion: v1
kind: Namespace
metadata:
  name: production
  annotations:
    # Custom gateway configuration
    traffic.tetrate.io/eastwest-gateway-namespace: "custom-gateways"
    traffic.tetrate.io/eastwest-gateway-name: "prod-east-west"
  labels:
    tetrate.io/rev: default  # Required for workspace creation
    istio-injection: enabled

Cloud Provider Integration

Configure AWS NLB for east-west gateway:

apiVersion: v1
kind: Namespace
metadata:
  name: payments
  annotations:
    # AWS Network Load Balancer - use dedicated cloud annotations annotation
    traffic.tetrate.io/eastwest-gateway-cloud-annotations: |
      service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
      service.beta.kubernetes.io/aws-load-balancer-internal: "true"
      service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"

    # Custom gateway name
    traffic.tetrate.io/eastwest-gateway-name: "payments-east-west"
  labels:
    tetrate.io/rev: default  # Required for workspace creation
    istio-injection: enabled

Configure GCP Internal Load Balancer:

apiVersion: v1
kind: Namespace
metadata:
  name: analytics
  annotations:
    # GCP Internal Load Balancer - use dedicated cloud annotations annotation
    traffic.tetrate.io/eastwest-gateway-cloud-annotations: |
      cloud.google.com/load-balancer-type: "Internal"
      networking.gke.io/load-balancer-type: "Internal"

    # Custom configuration
    traffic.tetrate.io/eastwest-gateway-namespace: "networking"
  labels:
    tetrate.io/rev: default  # Required for workspace creation
    istio-injection: enabled

Supported East-West Annotations

All annotations are configured at the namespace level:

Annotation	Default	Description	Mode
traffic.tetrate.io/global	false (disabled)	Enable namespace-level exposure (all services)	Namespace-Level
traffic.tetrate.io/eastwest-gateway-deploy	true	Auto-deploy gateway.install resource	Both
traffic.tetrate.io/eastwest-gateway-namespace	tetrate-system	Target namespace for gateway	Both
traffic.tetrate.io/eastwest-gateway-name	east-west-gateway	Custom gateway name (derives workload selector if not specified)	Both
traffic.tetrate.io/eastwest-gateway-workload-selector	app=<gateway-name>	Custom workload selector (derives gateway name from app label if not specified)	Both
traffic.tetrate.io/eastwest-gateway-cloud-annotations	none	Cloud provider annotations (YAML format)	Both

Service Labels (Service-Level Mode):

Label	Description	Mode
traffic.tetrate.io/global: "true"	Expose this specific service cross-cluster	Service-Level

Cloud Provider Annotations Format:

The traffic.tetrate.io/eastwest-gateway-cloud-annotations annotation accepts cloud provider annotations in YAML format:

traffic.tetrate.io/eastwest-gateway-cloud-annotations: |
  service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
  service.beta.kubernetes.io/aws-load-balancer-internal: "true"
  cloud.google.com/load-balancer-type: "Internal"

Parsing Behavior:

• First attempts YAML parsing with yaml.Unmarshal
• Falls back to line-by-line parsing if YAML fails
• Handles values with colons (URLs, ARNs) using strings.Index
• Removes surrounding quotes (single and double) from values
• Logs parse errors as warnings when some annotations succeed
• Returns empty map when all parsing fails

Supported Cloud Providers:

• service.beta.kubernetes.io/* - AWS LoadBalancer annotations
• service.kubernetes.io/* - Standard Kubernetes service annotations
• cloud.google.com/* - GCP-specific annotations
• networking.gke.io/* - GKE networking annotations
• service.beta.kubernetes.io/azure-* - Azure LoadBalancer annotations

Annotation Derivation Logic

The eastwest-gateway-name and eastwest-gateway-workload-selector annotations work independently with smart defaults:

• Both provided: Use both independently (full control over gateway name and workload selector)
• Only gateway-name: Workload selector auto-derived as app=<gateway-name>
• Only workload-selector: Gateway name auto-derived from app label in selector
• Neither provided: Defaults to name=east-west-gateway with app=east-west-gateway

Examples:

# Example 1: Only gateway name (selector derived)
annotations:
  traffic.tetrate.io/eastwest-gateway-name: "prod-gateway"
  # Result: gateway-name=prod-gateway, workload-selector=app=prod-gateway

# Example 2: Only workload selector (name derived from app label)
annotations:
  traffic.tetrate.io/eastwest-gateway-workload-selector: "app=my-gw"
  # Result: gateway-name=my-gw, workload-selector=app=my-gw

# Example 3: Both provided (full control)
annotations:
  traffic.tetrate.io/eastwest-gateway-name: "custom-name"
  traffic.tetrate.io/eastwest-gateway-workload-selector: "app=custom-gw"
  # Result: Uses both exactly as specified

Disabling East-West for a Namespace

East-west is disabled by default. Simply omit the traffic.tetrate.io/global: "true" annotation:

apiVersion: v1
kind: Namespace
metadata:
  name: isolated-services
  # No traffic.tetrate.io/global annotation = east-west disabled (default)
  labels:
    tetrate.io/rev: default  # Required for workspace creation
    istio-injection: enabled

Deployment Workflow

1. Namespace Creation with Opt-In → Add traffic.tetrate.io/global: "true" annotation to namespace
2. East-West Gateway Deployed → Deployed to tetrate-system (default) when annotation is present
3. WorkspaceSettings Created → Contains defaultEastWestGatewaySettings with service label selector
4. User Labels Service → Add traffic.tetrate.io/global: "true" label to service
5. TSB Auto-Exposure → Service automatically exposed via east-west gateway
6. Cross-Cluster Access → Service accessible from other clusters in same workspace

Multi-Cluster Scenarios

Scenario 1: Multiple Clusters, Same Service

# Cluster 1 - us-west
apiVersion: v1
kind: Namespace
metadata:
  name: api
  annotations:
    traffic.tetrate.io/global: "true"  # Enable east-west
  labels:
    tetrate.io/rev: default  # Required for workspace creation
    istio-injection: enabled

---
apiVersion: v1
kind: Service
metadata:
  name: user-api
  namespace: api
  labels:
    traffic.tetrate.io/global: "true"
    region: us-west
spec:
  selector:
    app: user-api
  ports:
  - port: 8080
    targetPort: 8080
    name: http

# Cluster 2 - us-east
apiVersion: v1
kind: Namespace
metadata:
  name: api
  annotations:
    traffic.tetrate.io/global: "true"  # Enable east-west
  labels:
    tetrate.io/rev: default  # Required for workspace creation
    istio-injection: enabled

---
apiVersion: v1
kind: Service
metadata:
  name: user-api
  namespace: api
  labels:
    traffic.tetrate.io/global: "true"
    region: us-east
spec:
  selector:
    app: user-api
  ports:
  - port: 8080
    targetPort: 8080
    name: http

Both services are accessible as user-api.api.svc.cluster.local:8080 from any cluster. TSB handles multi-cluster routing automatically.

Scenario 2: Service Dependencies Across Clusters

# Cluster 1: Frontend service calls backend
apiVersion: v1
kind: Service
metadata:
  name: frontend
  namespace: web
  labels:
    traffic.tetrate.io/global: "true"  # Exposed to other clusters
spec:
  selector:
    app: frontend
  ports:
  - port: 8080
    targetPort: 8080
    name: http
---
# Cluster 2: Backend service
apiVersion: v1
kind: Service
metadata:
  name: backend-api
  namespace: services
  labels:
    traffic.tetrate.io/global: "true"  # Exposed to other clusters
spec:
  selector:
    app: backend-api
  ports:
  - port: 8080
    targetPort: 8080
    name: http

Frontend in Cluster 1 can call:

curl http://backend-api.services.svc.cluster.local:8080/api/data

Best Practices

1. Label Services Selectively: Only add traffic.tetrate.io/global: "true" to services that need cross-cluster access
2. Use Custom Gateways for Isolation: Create separate east-west gateways for different security zones
3. Apply Cloud Annotations: Use cloud provider annotations for optimized LoadBalancer configuration
4. Monitor Gateway Health: Check /debug/workspaces endpoint for WorkspaceSettings status
5. Test Connectivity: Verify cross-cluster communication before production deployment

Debugging East-West Traffic

Check WorkspaceSettings

# Port-forward to agent
kubectl port-forward -n istio-system deployment/tetrate-hosted-agent 8080:8080

# View workspace settings
curl http://localhost:8080/debug/workspaces | jq '.workspaces[] | select(.name=="ws-payments")'

Verify Gateway Deployment

# Check if east-west gateway exists
kubectl get gateway.install.tetrate.io -n tetrate-system east-west-gateway

# Check gateway pods
kubectl get pods -n tetrate-system -l app=east-west-gateway

Test Service Exposure

# From a pod in another cluster
kubectl exec -it test-pod -- curl http://payment-api.payments.svc.cluster.local:8080/health

Common Issues

Issue: Service not accessible across clusters

• Check: Service has traffic.tetrate.io/global: "true" label (not annotation) - this is critical!
• Check: Namespace has east-west enabled (must have traffic.tetrate.io/global: "true" annotation - disabled by default)
• Check: WorkspaceSettings contains defaultEastWestGatewaySettings with correct service label selector
• Verify: Use kubectl get service <name> -n <namespace> -o jsonpath='{.metadata.labels}' to check service labels

Issue: Gateway not deploying

• Check: traffic.tetrate.io/eastwest-gateway-deploy not set to "false" on namespace
• Check: Target namespace exists (default: tetrate-system)
• Check: Agent logs for deployment errors: kubectl logs -n istio-system deployment/tetrate-hosted-agent
• Verify: Check if gateway.install resource exists: kubectl get gateway.install -n <gateway-namespace>

Issue: Custom workload selector not working

• Check: Annotation derivation logic is working correctly (name/selector interdependence)
• Check: Workload selector format: app=name,tier=value (comma-separated, no spaces around =)
• Check: Pods exist with matching labels in target namespace: kubectl get pods -n <namespace> --show-labels
• Verify: If only gateway-name provided, selector should auto-derive as app=<gateway-name>
• Verify: If only workload-selector provided, name should auto-derive from app label value

Detailed East-West Debugging

Verify Service Label (Critical!)

The most common issue is using an annotation instead of a label:

# WRONG - using annotation (will NOT work!)
kubectl annotate service payment-api -n payments traffic.tetrate.io/global=true

# CORRECT - using label
kubectl label service payment-api -n payments traffic.tetrate.io/global=true

# Check current labels on a service
kubectl get service payment-api -n payments -o jsonpath='{.metadata.labels}' | jq .

Quick Status Check

# Check if east-west is enabled for namespace
kubectl get namespace payments -o jsonpath='{.metadata.annotations.traffic\.tetrate\.io/global}'

# List all services with global label
kubectl get services -n payments --show-labels | grep "traffic.tetrate.io/global=true"

Verify WorkspaceSettings Configuration

# Check WorkspaceSettings exists and has east-west config
kubectl get workspacesettings.tsb.tetrate.io ws-payments-settings -o yaml | grep -A 20 "defaultEastWestGatewaySettings"

Check Gateway Deployment Status

# Verify gateway.install resource exists
kubectl get gateway.install -n tetrate-system

# Check gateway pods are running
kubectl get pods -n tetrate-system -l app=east-west-gateway

# Check LoadBalancer service
kubectl get service -n tetrate-system -l app=east-west-gateway

Test Cross-Cluster Connectivity

# From a pod in another cluster (same workspace)
kubectl exec -it <pod-name> -n <namespace> -- curl http://<service>.<namespace>.svc.cluster.local:<port>/

# Check service endpoints
kubectl get endpoints <service> -n <namespace>