Health Checks

Tetrate Global Load Balancer monitors the health and readiness of TSB Gateways that have been annotated with the name of a Gateway set gslb.tetrate.io/name: edge-internal or similar.

The Tetrate GSLB controller does not send its own synthetic HTTP or TCP probes at the Edge Gateway's public IP address. Instead, it determines health from signals already available inside the cluster, combines them into a single healthy/unhealthy verdict per Gateway, and then shares that verdict with every other cluster and region using the DNS TXT heartbeat records described above.

What is Monitored

For each Kubernetes Service that carries a gslb.tetrate.io/name annotation, the controller checks:

• Endpoint readiness: whether the Service has at least one Ready endpoint, as reported by Kubernetes EndpointSlices. If Kubernetes considers every backing Pod not-ready (for example, because Pods are still starting, failing readiness probes, or have all been terminated), the Gateway is considered unhealthy.
• External address: whether the Service has been assigned a LoadBalancer IP address (or hostname, which the controller resolves in the background on providers such as AWS that publish a hostname rather than an IP). A Gateway with no external address cannot be healthy.
• Istio proxy configuration sync (optional): when the Helm value enableIstioProxyStatusCheck is set to true, and the Service matches an Istio ingress gateway (selector istio: ingressgateway), the controller additionally queries the Gateway's istio-proxy sidecar for its xDS sync status with istiod. All of Listeners, Routes, Clusters and Endpoints (LDS/RDS/CDS/EDS) must report as synced before the Gateway is considered healthy. This catches cases where the Gateway Pod is Ready but has not yet received (or has lost) its Istio configuration, and so cannot correctly route traffic. This check is disabled by default; enable it if you want GSLB failover to also account for Istio configuration propagation delays.

A Gateway must pass all of the checks that are enabled for it to be marked healthy. Any single failed check (an EndpointSlice going not-ready, the LoadBalancer address disappearing, or an Istio sync going stale) is enough to mark it unhealthy immediately — there is no "wait and see" grace period for the unhealthy transition.

Recovery back to healthy, when the Istio proxy status check is enabled, requires three consecutive successful sync checks (checked once per reconcile) before the Gateway is marked healthy again. This avoids flapping a Gateway's DNS status healthy and unhealthy while its proxy configuration is still converging.

Propagating Health Between Clusters and Regions

Each cluster's Tetrate GSLB controller only has direct visibility into the Gateways running in its own cluster. To build a complete picture across all clusters and regions, every controller periodically writes a timestamped TXT heartbeat record (health-001-<cluster-name>.<local/global-dns-zone>) recording that its local Gateway(s) are healthy, and reads the equivalent heartbeat records written by every other cluster:

• Local heartbeat: written to the Local Private DNS Zone, and read by other clusters in the same region, to build the Regional Gateway A record.
• Global heartbeat: written to the Global Private DNS Zone, and read by other regions, to build the Failover Gateway A record.

A heartbeat is considered stale — and the corresponding cluster or region is treated as unhealthy — if its timestamp has not been refreshed within the configured healthy duration:

• localTXTRecordHealthyDurationSeconds (default: 30 seconds) for local (regional) heartbeats
• globalTXTRecordHealthyDurationSeconds (default: 30 seconds) for global heartbeats

Heartbeats are refreshed roughly every reconcileRequeueSeconds (default: 10 seconds; see Metrics for counters that track these updates), giving a healthy margin under the default 30-second staleness window.

If a Gateway or an entire cluster becomes unhealthy, its address is removed from the Regional Gateway and Failover Gateway DNS records. However, if removing unhealthy addresses would leave a DNS record with no healthy targets at all, the Tetrate GSLB solution deliberately leaves the existing (stale) record in place rather than publishing an empty answer, since DNS providers handle empty record sets inconsistently. In this situation, clients continue to be directed towards the last-known addresses until at least one Gateway recovers.

Self-Healing and Rate-Limit Protection

DNS providers can rate-limit or throttle frequent writes, which could otherwise cause a cluster's own heartbeat to appear stale (and so falsely trigger a failover) simply because the controller was unable to write its update in time. To protect against this, if the controller detects that its own heartbeat is stale, it writes a self-heal heartbeat timestamped up to selfHealInterval (default: 3 minutes) into the future. This suppresses further, unnecessary failover flapping caused purely by write throttling, while still allowing a genuine outage to be detected once the self-healed timestamp naturally expires.

You can monitor this behavior using the tetrate_gslb_controller_local_healthcheck_is_recovering_from_rate_limiting and tetrate_gslb_controller_global_healthcheck_is_recovering_from_rate_limiting metrics, and observe overall health and failover status using the tetrate_gslb_controller_cluster_healthy, tetrate_gslb_controller_region_healthy and tetrate_gslb_controller_failover_in_progress metrics — see Metrics for details.