Troubleshooting your Configuration

Use the troubleshooting steps to understand the configuration generated by the Tetrate Hosted Agent, and diagnose problems that might occur

Checking Gateway Status

The system provides real-time status feedback through the gateway.tetrate.io/gateway-status annotation:

# Check service annotations
kubectl get service my-app -o jsonpath='{.metadata.annotations.gateway\.tetrate\.io/gateway-status}' | jq .

Example status output:

{
  "configured": true,
  "gatewayName": "api-example-com-80",
  "lastUpdated": "2024-01-15T10:30:00Z",
  "conflicts": [],
  "errors": []
}

Common Issues and Solutions

1. Path Conflicts

Problem: Multiple services trying to use the same path

{
  "configured": false,
  "errors": [
    {
      "type": "PathConflict",
      "message": "Path '/api' conflicts with service 'other-service' in namespace 'other-ns'",
      "conflictingServices": ["other-ns/other-service"]
    }
  ]
}

Solution: Use more specific paths or coordinate with the conflicting service owner.

2. Missing TLS Secret

Problem: HTTPS configured but TLS secret not found

{
  "configured": false,
  "errors": [
    {
      "type": "ValidationError",
      "message": "TLS secret 'my-tls' not found in namespace 'default'"
    }
  ]
}

Solution: Create the TLS secret before exposing the service:

kubectl create secret tls my-tls \
  --cert=path/to/cert.pem \
  --key=path/to/key.pem \
  -n default

3. Invalid Annotation Values

Problem: Incorrect annotation format or values

{
  "configured": false,
  "errors": [
    {
      "type": "ValidationError",
      "message": "Invalid protocol 'HTPPS', supported: HTTP, HTTPS",
      "field": "gateway.tetrate.io/protocol"
    }
  ]
}

Solution: Check the annotation value and correct the typo.

Debug Endpoints Reference

The agent provides comprehensive debug endpoints for troubleshooting:

Endpoint	Description	Key Information
/health	Agent health status	Overall agent health
/ready	Agent readiness	Ready to serve requests
/metrics	Prometheus metrics	Performance and usage metrics
/debug/gateways	Gateway infrastructure	Workspace/gateway group status + annotation stats
/debug/annotation-gateways	Annotation gateway details	Exposed services, routes, TSB gateways, conflicts
/debug/tsb-api-objects	Raw TSB API payloads	Exact TSB gateway configurations sent to TSB API
/debug/workspaces	Workspace management	TSB workspace creation and memberships
/debug/namespaces	Namespace management	Watched namespaces and trigger labels
/debug/services	Service management	Service events and processing

# View all gateways status
curl http://tetrate-hosted-agent:8080/debug/gateways

# View annotation-based gateway details
curl http://tetrate-hosted-agent:8080/debug/annotation-gateways

# Check specific namespace services
curl http://tetrate-hosted-agent:8080/debug/services?namespace=default

Advanced Debugging Commands

Use these commands with jq to analyze specific issues:

# List exposed services
curl http://agent:8080/debug/annotation-gateways | jq '.exposedServices'

# Check for service conflicts
curl http://agent:8080/debug/annotation-gateways | jq '.serviceStatuses | to_entries[] | select(.value.conflicts | length > 0)'

# Find failed configurations
curl http://agent:8080/debug/annotation-gateways | jq '.serviceStatuses | to_entries[] | select(.value.configured == false)'

# View exact TSB API payloads
curl http://agent:8080/debug/tsb-api-objects | jq '.gateways[].tsb_api_payload'

# Check TLS configurations
curl http://agent:8080/debug/tsb-api-objects | jq '.gateways[] | {name, tls_configs: [.tsb_api_payload.http[]? | select(.tls) | .tls]}'

# Check workspace settings for east-west
curl http://agent:8080/debug/workspaces | jq '.workspaces[] | select(.name | startswith("ws-"))'

4. TLS Configuration Issues

Problem: HTTPS service not working or certificate errors

Diagnosis:

• Verify TLS secret exists: kubectl get secret <tls-secret> -n <namespace>
• Check secret has tls.crt and tls.key fields
• For cross-namespace secrets, verify ReferenceGrant was created

# Check if ReferenceGrant exists
kubectl get referencegrant -n <secret-namespace>

# Verify TLS secret format
kubectl get secret <tls-secret> -n <namespace> -o jsonpath='{.data}' | jq 'keys'

Solution:

• Use gateway.tetrate.io/tls-secret-namespace for cross-namespace secrets (requires shared gateway)
• Ensure secret name follows Kubernetes naming (lowercase alphanumeric + hyphens)
• Verify the secret contains valid tls.crt and tls.key data

TSB Connectivity Troubleshooting

5. Agent Cannot Connect to TSB Management Plane

Problem: Agent fails to communicate with the TSB API

Symptoms:

• Agent logs show connection errors or timeouts
• Gateways are not being created in TSB
• /health endpoint returns unhealthy status

Diagnosis:

# Check agent logs for TSB connection errors
kubectl logs -n istio-system deployment/tetrate-hosted-agent | grep -i "error\|failed\|timeout"

# Verify ControlPlane object exists
kubectl get controlplane -n istio-system

# Check if cluster-service-account secret exists and has required fields
kubectl get secret cluster-service-account -n istio-system -o jsonpath='{.data}' | jq 'keys'

# Verify TSB management plane address is reachable
kubectl exec -n istio-system deployment/tetrate-hosted-agent -- wget -qO- --timeout=5 https://<tsb-address>/v2/version || echo "Connection failed"

Common Causes and Solutions:

Cause	Solution
Missing cluster-service-account secret	Ensure the secret exists with valid credentials from TSB onboarding
Invalid or expired credentials	Re-generate credentials from TSB and update the secret
Network connectivity issues	Verify firewall rules allow egress to TSB management plane
TLS certificate errors	Check agent.tsb.insecureSkipVerify setting (testing only) or ensure proper CA certificates
Incorrect organization name	Verify TSB_ORGANIZATION matches your TSB organization

# Check the organization configured in the secret
kubectl get secret cluster-service-account -n istio-system -o jsonpath='{.data.organization}' | base64 -d

# Verify agent configuration
kubectl get deployment tetrate-hosted-agent -n istio-system -o jsonpath='{.spec.template.spec.containers[0].env}' | jq '.[] | select(.name | startswith("TSB"))'

6. Authentication/Authorization Failures

Problem: Agent authenticates but cannot create/update resources in TSB

Symptoms:

• Agent logs show 403 Forbidden errors
• Gateways created locally but not appearing in TSB
• Workspace creation fails

Diagnosis:

# Check for permission errors in logs
kubectl logs -n istio-system deployment/tetrate-hosted-agent | grep -i "403\|forbidden\|unauthorized\|permission"

# Verify the tenant exists in TSB
# (requires tctl or TSB UI access)
tctl get tenants --org <organization>

# Check agent's configured tenant
kubectl get deployment tetrate-hosted-agent -n istio-system -o jsonpath='{.spec.template.spec.containers[0].env}' | jq '.[] | select(.name=="TSB_TENANT")'

Solutions:

• Verify the service account has appropriate RBAC permissions in TSB
• Ensure the configured tenant (TSB_TENANT) exists and the service account has access
• Check that the cluster is properly registered with TSB

7. Validating cluster-service-account Secret

Problem: Uncertain if the service account secret is properly configured

Validation Steps:

# Check secret exists
kubectl get secret cluster-service-account -n istio-system

# Verify required fields exist
kubectl get secret cluster-service-account -n istio-system -o json | jq '.data | keys'
# Expected: ["ca.crt", "cluster-name", "organization", "service-account.jwk", "token"]

# Check organization value
kubectl get secret cluster-service-account -n istio-system -o jsonpath='{.data.organization}' | base64 -d && echo

# Check cluster name
kubectl get secret cluster-service-account -n istio-system -o jsonpath='{.data.cluster-name}' | base64 -d && echo

# Verify token is not expired (if using JWT)
kubectl get secret cluster-service-account -n istio-system -o jsonpath='{.data.token}' | base64 -d | cut -d'.' -f2 | base64 -d 2>/dev/null | jq '.exp | todate' 2>/dev/null || echo "Token format check failed"

If secret is missing or invalid:

1. Re-onboard the cluster following TSB documentation
2. Regenerate service account credentials from TSB
3. Apply the new secret to the cluster

# Example: Apply new service account secret
kubectl apply -f cluster-service-account.yaml -n istio-system

# Restart agent to pick up new credentials
kubectl rollout restart deployment/tetrate-hosted-agent -n istio-system