Skip to main content
logoTetrate Service BridgeVersion: next

Connectivity and Failover between Clusters

Use this section when an application runs on more than one cluster. Single-cluster applications can skip it.

Tetrate makes it easy for the App Owner to deploy an application across multiple clusters. East-West gateways can be deployed in selected Workspaces, and expose some or all services to other clusters. Tetrate's central Service Registry is used to then create virtual endpoints (ServiceEntries) for these exposed services in other clusters, so that clients can discover and consume these services exactly as if they were running locally.

This capability can be used to:

  • Distribute components of an application across different clusters
  • Access centralized, shared services such as a Database microservice from remote clusters
  • Provide Failover between instances of services across different clusters
  • Simplify testing by ensuring that the same application and addressing scheme works in one cluster, or distributed across multiple

The Platform Owner ("Platform") prepares the platform for failover use cases, and the Application Owner ("Apps") can take advantage of these capabilities with minimal modification to their workflow.

The Platform Owner will prepare the platform as follows:

The Application Owner can then:

RunBook​

  1. Platform: Before you begin​

    Before you begin, you will need to know:

    • The Workspace where the shared services are located
    • Which services should be exposed; by default, all services within the Workspace are exposed

    You also need to check the following:

    • Shared root of trust: All the clusters must share the same root of trust, so that mTLS works across clusters. Populate the cacerts secret with the correct certificates before you deploy the Control Planes. For more information, refer to Multi-cluster traffic failover with EastWest Gateways.
    • Cluster regions: Place clusters that back each other up in different regions. Services in the same region are considered identical, and requests can be routed to any cluster in that region, even when the local instance is healthy.
    • Rate limits: From TSB 1.12, rate limits are also enforced on traffic that passes through port 15443, which is used for multi-cluster traffic.

  2. Platform: Deploy an East-West Gateway​

    Deploy an East-West Gateway in the namespace containing services to be shared:

    cat <<EOF > eastwest-gateway.yaml
    apiVersion: install.tetrate.io/v1alpha1
    kind: Gateway
    metadata:
      name: eastwest-gateway
      namespace: bookinfo
    spec:
      type: EASTWEST
    EOF

    kubectl apply -f eastwest-gateway.yaml

    List the East-West Gateways running in the local cluster:

    kubectl get pods -A -l app=eastwest-gateway

  3. Platform: Update the Workspace to expose the required services​

    Update the WorkspaceSettings associated with the Workspace:

    cat <<EOF > bookinfo-settings.yaml
    apiVersion: api.tsb.tetrate.io/v2
    kind: WorkspaceSetting
    metadata:
      name: bookinfo-ws-settings
      workspace: bookinfo-ws
      tenant: tetrate
      organization: tetrate
    spec:
      defaultSecuritySetting:
        authenticationSettings:
          trafficMode: REQUIRED
        authorization:
          mode: WORKSPACE
      defaultEastWestGatewaySettings:
        - workloadSelector:
            namespace: bookinfo
            labels:
              app: eastwest-gateway
          exposedServices:
            - serviceLabels:
                app: details
            - serviceLabels:
                app: reviews
    EOF

    tctl apply -f bookinfo-settings.yaml

    Note how the defaultEastWestGatewaySettings uses workloadSelector to identify the EastWest Gateway, and has an optional exposedServices stanza.

    The Application Owner can discover and consume these shared services from the remote cluster. They are exposed as ServiceEntries in the xcp-multicluster namespace, and use the same FQDN as in the first cluster:

    kubectl get serviceentry -n xcp-multicluster

    Note that if the Application Owner deploys the same services in the remote cluster, then the Tetrate control plane will then remove the ServiceEntries so that clients in that cluster prefer service instances in the same cluster.

  4. Apps: Access the Exposed Services​

    Once an East-West gateway is deployed in the origin cluster, and services are selected for exposure using WorkspaceSettings:defaultEastWestGatewaySettings, you can then deploy services in the origin cluster. Matching services are then exposed within remote clusters.

    You can view the list of exposed services in the remote cluster as follows:

    kubectl get serviceentry -n xcp-multicluster

    The FQDNs of the services are addressable in the remote cluster, meaning that any client services in the remote cluster can access the origin service without any modification. The Tetrate platform maintains the ServiceEntries in sync with the presence and state of the origin services.

  5. Platform: Deploy East-West Gateways for Internal Failover​

    In this use case, we'll prepare the platform for failover of internal services from one cluster to another. For example, an Application Owner could deploy a multi-component application (such as bookinfo) into two clusters in different regions. Each application will use the local version of the dependent services unless the local instance fails, in which case, the Tetrate platform will automatically fail traffic over to the other cluster. If the clusters are in the same region, you can apply a custom istio-locality label to the East-West Gateway to get the same behavior. No application modifications are required on behalf of the Application Owner.

    Prerequisites​

    1. The namespaces for the application must be created on each workload cluster.
    2. The Tetrate workspace must include the namespaces, and must span each workload clusters.

    For example, if the application is to be deployed in the namespace bookinfo, this namespace must be present on each cluster and the Workspace configuration should reference all instances, for example, using the following namespaceSelector:

    spec:
      namespaceSelector:
        names:
          - "*/bookinfo"

    Deploy an East-West gateway​

    Deploy additional East-West gateways in the other clusters, exactly as before:

    cat <<EOF > eastwest-gateway.yaml
    apiVersion: install.tetrate.io/v1alpha1
    kind: Gateway
    metadata:
      name: eastwest-gateway
      namespace: bookinfo
    spec:
      type: EASTWEST
    EOF

    kubectl apply -f eastwest-gateway.yaml

  6. Apps: Test Failover across Clusters​

    With the above configuration, you can deploy your application into several clusters. If a service instance in one cluster were to fail for any reason, the Tetrate platform will detect this and transparently switch traffic over to a remote, working instance of the service.

    The Tetrate platform creates shadow WorkloadEntries for each remote instance of a service in the cluster and namespace where the local instance exists. For example, if you deploy the bookinfo app into two clusters which are equipped with the HA configuration, you can then check each cluster for the presence of the shadow WorkloadEntries.

    In the case below, we exposed the details and reviews services only, and the Cloud platform exposed the eastwest-gateway on three IP addresses:

    kubectl get workloadentries -n bookinfo
    # NAME                                           AGE   ADDRESS
    # k-details-fc556d47e94d1cb435e513fa016c2243     17m   18.135.167.198
    # k-details-fc556d47e94d1cb435e513fa016c2243-2   17m   18.168.99.230
    # k-details-fc556d47e94d1cb435e513fa016c2243-3   17m   35.179.51.164
    # k-reviews-3ab8d1334c8f22513cd591f84c978f88     17m   18.135.167.198
    # k-reviews-3ab8d1334c8f22513cd591f84c978f88-2   17m   18.168.99.230
    # k-reviews-3ab8d1334c8f22513cd591f84c978f88-3   17m   35.179.51.164

    kubectl get svc -n bookinfo eastwest-gateway
    # NAME               TYPE           CLUSTER-IP      EXTERNAL-IP                                                                     PORT(S)           AGE
    # eastwest-gateway   LoadBalancer   10.100.17.100   k8s-bookinfo-eastwest-00a17af379-bdda7f4eb8c5da5c.elb.eu-west-2.amazonaws.com   15443:30082/TCP   82m

    For more information, check out the TSB Getting Started exercise Failover Across Clusters.