Timeouts, Retries and Circuit Breakers

This guide shows how to make calls to a service more resilient with three settings that TSB applies in the Envoy proxies of the clients:

• Timeouts limit how long a client waits for a response.
• Retries send a failed request again, when it is safe to do so.
• Circuit breakers temporarily remove a failing instance of the service from load balancing, so that clients stop sending it requests.

All three are configured in the resilience section of a TrafficSetting. The examples use a payments service, payments.payments.svc.cluster.local, on port 8080.

Before you begin

• The service and its clients are onboarded to the mesh with sidecars. Upstream traffic settings are not supported in ambient mode.
• You have a Traffic Group for the clients, in BRIDGED mode. Traffic Groups in DIRECT mode do not receive these settings.
• You know the normal latency and error behavior of the service, for example from its metrics. The values in this guide are examples, not recommendations.

Where the settings apply

A TrafficSetting configures the clients of a service, not the service itself. The settings in spec.outbound.upstreamTrafficSettings apply to the proxies in the namespaces of the Traffic Group, for the calls they make to the listed hosts:

• To apply settings to calls between services (east-west), attach the TrafficSetting to the Traffic Group of the calling namespaces.
• To apply settings to calls from an Ingress Gateway to the service (north-south), attach the TrafficSetting to the Traffic Group that contains the gateway's namespace.

If the clients belong to another team's Workspace, that team owns the setting for their calls. To define defaults for a whole Workspace, use defaultTrafficSetting in the WorkspaceSetting. Traffic Group settings take precedence.

One TrafficSetting per Traffic Group, one entry per host

A Traffic Group can have only one TrafficSetting. Put the timeouts, retries and circuit breaker for a host in the same upstreamTrafficSettings entry. If two entries list the same host, the later entry replaces the earlier one; the settings are not merged.

Summary

Setting	Default	What you need to decide
Request timeout	None: requests can wait indefinitely	The longest time a client should wait, including all retries
Connect timeout	10s	How long to wait for a new connection to an instance
Retries	On: 2 retries for connection failures, resets and 503 responses	Which failures are safe to retry, how many times, and how long each attempt may take
Circuit breaker	Off, unless you set a sensitivity	How many consecutive failures an instance may return before it is ejected

Complete example

The following TrafficSetting combines all three settings for calls to payments. The sections below explain each part.

apiVersion: traffic.tsb.tetrate.io/v2
kind: TrafficSetting
metadata:
  organization: tetrate
  tenant: tetrate
  workspace: payments-ws
  group: payments-traffic
  name: payments-traffic-settings
spec:
  outbound:
    upstreamTrafficSettings:
      - hosts:
          - 'payments.payments.svc.cluster.local'
        settings:
          resilience:
            connectionPool:
              tcp:
                connectTimeout: 2s
              http:
                requestTimeout: 4s
                retries:
                  attempts: 2
                  perTryTimeout: 1s
                  retryOn: 'gateway-error,connect-failure,refused-stream'
                  retryBackOff:
                    baseInterval: 25ms
                    maxInterval: 250ms
            circuitBreakerSensitivity: MEDIUM

Apply it with tctl apply -f, or with kubectl apply -f if you use GitOps.

Timeouts

connectionPool:
  tcp:
    connectTimeout: 2s
  http:
    requestTimeout: 4s

• requestTimeout is the total time a client waits for a response. It includes every retry, so it is the hard limit for the whole request. It is not set by default.
• connectTimeout is the time allowed to open a TCP connection to an instance of the service. The default is 10s.

A request that exceeds requestTimeout fails with a 504 response. Set the timeout above the normal response time of the service, including its slowest legitimate requests, and below the timeout of the caller, so that the caller receives a clear error instead of giving up first.

Retries

connectionPool:
  http:
    retries:
      attempts: 2
      perTryTimeout: 1s
      retryOn: 'gateway-error,connect-failure,refused-stream'
      retryBackOff:
        baseInterval: 25ms
        maxInterval: 250ms

• attempts is the number of retries after the first attempt. In this example, a request is tried up to three times.
• perTryTimeout limits each attempt. The retries must fit within requestTimeout: here, three attempts of up to 1s, plus the back-off, fit within 4s. If requestTimeout is reached first, no further retries are made.
• retryOn lists the failures to retry. gateway-error covers 502, 503 and 504 responses. See the Envoy retry conditions for the full list.
• retryBackOff sets the delay between retries. The delay grows exponentially from baseInterval, up to maxInterval.

Retries are on by default

If you do not configure retries, TSB still retries failed requests twice, for connection failures, resets and 503 responses. If you configure retries without retryOn, these default conditions apply. To turn retries off, set attempts: 0.

Only retry requests that are safe to repeat. A 5xx response or a timeout does not prove that the service did nothing: it may have processed the request before it failed. For requests that change data, such as payments or orders, either make the operation idempotent in the service, or limit retryOn to failures where the request did not reach the service, such as connect-failure and refused-stream.

Circuit breakers

circuitBreakerSensitivity: MEDIUM

When an instance of the service returns too many consecutive errors, the client's proxy ejects it from load balancing for a period of time, and sends requests to the other instances. The sensitivity sets how many consecutive errors are tolerated:

Sensitivity	Consecutive errors before ejection
LOW	20
MEDIUM	10
HIGH	5
CUSTOM	Set in outlierDetection

Which errors count

The LOW, MEDIUM and HIGH levels count gateway errors (502, 503 and 504 responses) and connection failures. An instance that returns other errors, such as 500, is not ejected. To include all 5xx responses, use CUSTOM.

With CUSTOM, you set the outlier detection values yourself:

circuitBreakerSensitivity: CUSTOM
outlierDetection:
  consecutive5xx: 5
  enforcingConsecutive5xx: 100
  interval: 10s
  baseEjectionTime: 30s
  maxEjectionPercent: 50

• consecutive5xx and enforcingConsecutive5xx eject an instance after 5 consecutive 5xx responses. enforcingConsecutive5xx defaults to 0, which disables this check, so set it to 100.
• baseEjectionTime is how long an instance stays ejected. It grows each time the same instance is ejected again.
• maxEjectionPercent limits the share of instances that can be ejected at the same time. It defaults to 100, which means that if all instances fail, all of them can be ejected and the service stops receiving traffic. A value such as 50 keeps some instances in service.

See OutlierDetection for all the fields.

Circuit breaking is most useful when the service has several instances. With a single instance, ejecting it leaves no instance to send requests to.

Verify the configuration

TSB applies these settings to the client proxies through EnvoyFilter resources. Check that they exist in the namespace of the clients:

kubectl get envoyfilter -n <client-namespace>

Inspect the configuration of a client proxy, for example a pod in the checkout namespace that calls payments:

# Timeouts and retries are part of the route configuration
istioctl proxy-config route <client-pod> -n checkout -o json | grep -E '"timeout"|"numRetries"|"perTryTimeout"'

# Connect timeout and outlier detection are part of the cluster configuration
istioctl proxy-config cluster <client-pod> -n checkout --fqdn payments.payments.svc.cluster.local -o json | grep -E '"connectTimeout"|outlierDetection' -A3

To watch the settings take effect, read the proxy's statistics for the payments cluster. upstream_rq_retry counts retries, and outlier_detection.ejections_active shows the number of ejected instances:

kubectl exec <client-pod> -n checkout -c istio-proxy -- pilot-agent request GET stats | grep payments.payments.svc.cluster.local | grep -E 'upstream_rq_retry|upstream_rq_timeout|ejections_active'

Related settings

• Rate limiting protects a service from too many requests.
• Client Side Load Balancing includes session affinity (sticky sessions) by header, cookie or source IP.
• Canary Releases shifts traffic between versions of a service.