Scraping Gateway Proxy Metrics

Tetrate Service Bridge can configure the Control Plane otel-collector to scrape Envoy metrics directly from selected data plane proxies. This is useful when you need deeper operational insight into gateway behavior, such as upstream cluster state, circuit breakers, connection pools, listener activity, or HTTP/2 behavior that is exposed by Envoy's Prometheus endpoint.

The feature is opt-in. Scraping every proxy in every namespace can create a large number of time series and add unnecessary load to the collector. In most production environments, gateway proxies are the best first target: they carry important north-south and east-west traffic, but they are usually much fewer than application sidecars. This gives operators more detail where it matters without overwhelming the collector.

How it works

Configure spec.components.collector.envoyMetricScrapingSelectors in the TSB ControlPlane resource. For each selector, the Control Plane operator adds a Prometheus scrape job to the otel-collector configuration.

The generated scrape jobs:

• discover pods in the selected namespace,
• scrape Envoy's Prometheus endpoint at /stats/prometheus,
• rewrite the target address to port 15090, and
• expose the scraped metrics through the Control Plane otel-collector service on port 9090.

When you configure a selector with only namespace, TSB scrapes gateway pods in that namespace. The generated scrape job keeps pods with the istio pod label, for example istio=ingressgateway. Application sidecars in the same namespace are not scraped by the namespace selector unless they also have that label.

Use a workloadSelector when you need to target a specific proxy workload by label.

Start with gateways

Prefer namespace-level gateway scraping first. It gives useful Envoy-level visibility into gateway traffic while keeping the number of scraped targets small. Add workload selectors only for specific workloads that need deeper debugging or long-term monitoring.

Scrape gateway proxies in a namespace

The following example enables Envoy metric scraping for gateway proxies in the echo namespace:

apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
  name: controlplane
  namespace: istio-system
spec:
  components:
    collector:
      envoyMetricScrapingSelectors:
      - namespace: echo

You can also patch an existing Control Plane:

kubectl patch controlplane controlplane -n istio-system --type=merge -p \
'{"spec":{"components":{"collector":{"envoyMetricScrapingSelectors":[{"namespace":"echo"}]}}}}'

The operator renders a scrape job similar to this in the otel-collector-conf ConfigMap:

- job_name: 'envoy-stats-echo-0'
  metrics_path: '/stats/prometheus'
  kubernetes_sd_configs:
    - role: pod
      namespaces:
        names:
          - echo
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_istio]
    action: keep
    regex: .+
  - source_labels: [__address__]
    action: replace
    regex: '([^:]+)(?::\d+)?'
    replacement: $1:15090
    target_label: __address__
  - source_labels: []
    target_label: tsb_cluster_name
    replacement: "cluster-0"
  - source_labels: [__meta_kubernetes_namespace]
    target_label: namespace
  - source_labels: [__meta_kubernetes_pod_name]
    target_label: pod

In the demo environment, the echo namespace has application pods and a gateway pod:

echo-v1-ff4747944-drs7m             app=echo,version=v1
echo-v2-86d44d54f4-6lg77            app=echo,version=v2
tsb-gateway-echo-6bb8557577-c5fhq   app=tsb-gateway-echo,istio=ingressgateway

With namespace: echo, only the gateway pod is scraped because it has the istio label:

pod="tsb-gateway-echo-6bb8557577-c5fhq"

Scrape a specific workload

Use workloadSelector to scrape proxies for pods that match specific labels. The following example targets the tsb-gateway-echo workload in the echo namespace:

apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
  name: controlplane
  namespace: istio-system
spec:
  components:
    collector:
      envoyMetricScrapingSelectors:
      - workloadSelector:
          namespace: echo
          labels:
            app: tsb-gateway-echo

You can patch an existing Control Plane with:

kubectl patch controlplane controlplane -n istio-system --type=merge -p \
'{"spec":{"components":{"collector":{"envoyMetricScrapingSelectors":[{"workloadSelector":{"namespace":"echo","labels":{"app":"tsb-gateway-echo"}}}]}}}}'

The generated scrape job keeps only pods matching the configured labels:

- job_name: 'envoy-stats-echo-0'
  metrics_path: '/stats/prometheus'
  kubernetes_sd_configs:
    - role: pod
      namespaces:
        names:
          - echo
  relabel_configs:
  - source_labels: [__meta_kubernetes_pod_label_app]
    action: keep
    regex: tsb-gateway-echo
  - source_labels: [__address__]
    action: replace
    regex: '([^:]+)(?::\d+)?'
    replacement: $1:15090
    target_label: __address__
  - source_labels: []
    target_label: tsb_cluster_name
    replacement: "cluster-0"
  - source_labels: [__meta_kubernetes_namespace]
    target_label: namespace
  - source_labels: [__meta_kubernetes_pod_name]
    target_label: pod

To target application sidecars instead, select the application labels. For example, the demo echo application pods can be selected with:

envoyMetricScrapingSelectors:
- workloadSelector:
    namespace: echo
    labels:
      app: echo

Use this carefully. Scraping application sidecars can add many more targets and time series than scraping gateways.

Verify the configuration

Check the selector configured on the Control Plane:

kubectl get controlplane controlplane -n istio-system \
  -o jsonpath='{.spec.components.collector.envoyMetricScrapingSelectors}'

Check the rendered collector scrape job:

kubectl get cm otel-collector-conf -n istio-system \
  -o jsonpath='{.data.otel-collector-config}' \
  | grep -A 35 "envoy-stats"

Expose the Control Plane collector locally:

kubectl port-forward -n istio-system svc/otel-collector 29090:9090

In another terminal, verify which pods are being scraped:

curl -s localhost:29090/metrics \
  | grep 'job="envoy-stats-echo-0"' \
  | grep -o 'pod="[^"]*"' \
  | sort -u

For the demo namespace-level gateway example, the output should include only the gateway pod:

pod="tsb-gateway-echo-6bb8557577-c5fhq"

You can also inspect the metric names coming from the scraped Envoy proxy:

curl -s localhost:29090/metrics \
  | grep 'job="envoy-stats-echo-0"' \
  | awk -F'[ {]' '{print $1}' \
  | sort -u \
  | head -20

Example metric names include:

envoy_cluster_assignment_stale_total
envoy_cluster_assignment_timeout_received_total
envoy_cluster_bind_errors_total
envoy_cluster_circuit_breakers_default_cx_open
envoy_cluster_http2_goaway_sent_total