Cross-Cluster Communications

TSE makes it easy to securely connect clusters, availability zones and regions

Once you scale your deployment to more than one cluster, this creates the challenge of securely connecting these clusters so that a client in one cluster can access a service in a second cluster. TSE handles all of the complexity, providing service discovery and secure mTLS-based connectivity between clusters, without the need to expose services to external (non-mesh) clients:

Using an East-West Gateway to expose services to remote clusters

• You don't need to use an Ingress Gateway. One approach would be to publish the target service for external access, via an Ingress Gateway. Clients in and outside of the mesh could access the service via its public entry point. This approach is not optimal for services that should only be accessed from internal (mesh-based) clients because it exposes the service to internal and external users.

• TSE provides an internal 'East-West' gateway. The 'East-West' gateway exposes sets of services in a cluster so that other TSE-managed clusters can access them. These services are not exposed to non-mesh clients. Furthermore, these services can be discovered and accessed using their FQDN (service name) alone, exactly as if they were running in the local cluster, so clients can access them by name without knowing any details about where the service is deployed.

• The 'East-West' gateway provides high-availability. As we will see in the next exercise, the 'East-West' gateway provides high-availability for internal services, failing-over to a remote service instance if the local instance(s) become unavailable.

Prerequisites

• You have installed TSE and onboarded an 'application' cluster
• You have installed the Bookinfo App and verified it is working
• You have disabled the Deny-All setting in the TSE Administration Settings

Pay particular attention to the cluster!

When you follow this exercise, pay particular attention to which cluster you apply kubectl commands to. Some steps are specific to one cluster, and some are applied to both!

You may find it useful to create shell aliases to easily switch between clusters:

alias k1c="kubectl config use-context arn:aws:eks:us-east-1:901234567890:cluster/tse-cluster-1"
alias k2c="kubectl config use-context arn:aws:eks:us-east-2:901234567890:cluster/tse-cluster-2"

Recall that tctl commands apply to the TSE management console, so you don't need to consider the kubectl context / cluster.

Access a Service from a Remote Cluster, as if it were Local

The bookinfo app is installed in a TSE-managed cluster-1. We will onboard a second cluster-2, and use the sleep application to access the bookinfo services. It will appear that the services are local to cluster-2, but TSE will securely route traffic to cluster-1.

1. Prepare the first cluster

   Deploy a test sleep client in the bookinfo namespace in the first application cluster. Verify that it can access the productpage service.

2. Onboard a second cluster and deploy the sleep app

   Onboard a second client cluster, deploy the sleep app and verify that it cannot access the productpage service in the first cluster.

3. Deploy and configure an East-West Gateway in the application cluster

   Deploy an East-West (EW) gateway in the first cluster, and configure the Bookinfo workspace.

4. Access productpage.bookinfo remotely from cluster-2

   Verify that cluster-2 clients can reach the productpage.bookinfo service on cluster-1.

Prepare the First Cluster

Begin with the following deployment (you can adapt it to the names you have used):

• A single workload cluster cluster-1, onboarded to TSE
• The Bookinfo app deployed within the bookinfo namespace on cluster-1
• A TSE Workspace bookinfo-ws with the namespace selector */bookinfo

We are going to:

1. Deploy the sleep service into the bookinfo namespace on cluster-1
2. Verify that the sleep service can access the productpage.bookinfo service

Deploy the sleep service in cluster-1's bookinfo namespace:

kubectl apply -n bookinfo -f https://raw.githubusercontent.com/istio/istio/master/samples/sleep/sleep.yaml

Verify that the sleep service can access the productpage.bookinfo service:

kubectl exec deploy/sleep -n bookinfo -- curl -s productpage.bookinfo:9080/productpage | grep -o "<title>.*</title>"

Onboard a second cluster and deploy the sleep app

We are going to:

1. Onboard a second cluster cluster-2 to TSE, for the purposes of running a remote client. Locate this cluster in a different AWS region.
2. Deploy the sleep service into the bookinfo namespace on cluster-2
3. Verify that the sleep client service cannot access the productpage.bookinfo service in cluster-1

You may wish to review the onboarding process before proceeding.

Choose a different region

It's not needed for this exercise, but the next Failover across Clusters exercise works much better when the two clusters are in different regions. For that reason, it's strongly recommended to create your second cluster in a different AWS region.

Prepare TSE to accept the new cluster:

cat <<EOF > cluster-2.yaml
apiVersion: api.tsb.tetrate.io/v2
kind: Cluster
metadata:
    name: cluster-2
    organization: tse
EOF

tctl apply -f cluster-2.yaml 
tctl x cluster-install-template cluster-2 > cluster-2-values.yaml

Perform the helm install for tse-cp on cluster-2, using cluster-2-values.yaml to configure the install. Allow several minutes for the TSE management plane to fully onboard and synchronize with the second cluster.

In the TSE UI, check the Clusters pane. You should see the new cluster, with no services.

Deploy the sleep app on cluster-2

Run the following commands on client cluster-2:

kubectl create namespace bookinfo
kubectl label namespace bookinfo istio-injection=enabled

kubectl apply -n bookinfo -f https://raw.githubusercontent.com/istio/istio/master/samples/sleep/sleep.yaml

Test cross-cluster communications from client cluster-2 to application cluster-1

Run the following commands on client cluster-2:

kubectl exec deploy/sleep -n bookinfo -- curl -s productpage.bookinfo:9080/productpage

The service call fails (as expected), with the error Could not resolve host: productpage.bookinfo.

Deploy and configure an East-West Gateway in the application cluster

We are going to:

1. Deploy an East-West Gateway in the first application cluster
2. Expose the Bookinfo services on the East-West gateway
3. Wait for TSE to create ServiceEntries for the Bookinfo services

Deploy an East-West Gateway in the first cluster

Apply the following configuration in cluster-1:

cat <<EOF > eastwest-gateway.yaml
apiVersion: install.tetrate.io/v1alpha1
kind: IngressGateway
metadata:
  name: eastwest-gateway
  namespace: bookinfo
spec:
  eastWestOnly: true
EOF

kubectl apply -f eastwest-gateway.yaml

On cluster-1, verify that the East-West Gateway service is running:

kubectl get svc -n bookinfo eastwest-gateway

You should see a service with an external IP, similar to the following:

NAME               TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)           AGE
eastwest-gateway   LoadBalancer   10.100.132.235   a30e28ec7646043b6a22fb31826d2322-543394208.eu-west-2.elb.amazonaws.com   15443:31568/TCP   2m

Expose the Bookinfo services on the East-West gateway

We will reconfigure the Bookinfo workspace by adding a setting that exposes the Workspace's services on the East-West Gateway:

cat <<EOF > bookinfo-ws-eastwest.yaml
apiVersion: api.tsb.tetrate.io/v2
kind: WorkspaceSetting
metadata:
  organization: tse
  tenant: tse
  workspace: bookinfo-ws
  name: bookinfo-ws-setting
spec:
  defaultEastWestGatewaySettings:
    - workloadSelector:
        namespace: bookinfo
        labels:
          app: eastwest-gateway
EOF

tctl apply -f bookinfo-ws-eastwest.yaml

Wait for TSE to create ServiceEntries for the Bookinfo services

TSE will identify services that exposed by an East-West Gateway, and will then create ServiceEntries for these in the other clusters.

On cluster-2, watch for the creation of the ServiceEntries in the xcp-multicluster namespace used by TSE:

while true ; do echo -n ... ; kubectl get se -n xcp-multicluster ; sleep 5 ; done

ServiceEntries are typically created promptly, but may be delayed by several minutes if TSE batches updates.

Access productpage.bookinfo remotely from cluster-2

Recall that previously, the sleep client in cluster-2 could not access the productpage.bookinfo service in cluster-1.

In cluster-2, try again to invoke the productpage.bookinfo service:

kubectl exec deploy/sleep -n bookinfo -- curl -s productpage.bookinfo:9080/productpage

The service call now succeeds.

'503 Service Unavailable', 'No healthy Upstreams' or similar error?

What if it does not work? Allow 5 minutes or so for all resources to be deployed. In particular, the EastWest gateway requires a load balancer, and this may take several minutes to be fully provisioned:

Provisioning the Load Balancer for the East-West Gateway

What have we achieved?

TSE Topology: cross-cluster communications

1. We have onboarded a second cluster into TSE
2. We have deployed an East-West Gateway in cluster-1, to expose cluster-1 services without using an external Ingress Gateway
3. We can now access the productpage.bookinfo service from remote clusters, through the East-West Gateway

Fine-tuning Configuration (optional)

There are several ways that you can fine-tune the configuration generated by TSE.

Limit what services are exposed in a workspace

The defaultEastWestGatewaySettings object is associated with a Workspace, and exposes all services within that Workspace on the Eest-West gateway.

You can use service selectors to fine-tune which services are exposed in the EastWest gateway:

apiVersion: api.tsb.tetrate.io/v2
kind: WorkspaceSetting
metadata:
  organization: tse
  tenant: tse
  workspace: bookinfo-ws
  name: bookinfo-ws-setting
spec:
  defaultEastWestGatewaySettings:
    - workloadSelector:
        namespace: eastwest
        labels:
          app: eastwest-gateway
      exposedServices:
        - serviceLabels:
            failover: enable
        - serviceLabels:
            app: details

Limit what services are exposed to other clusters

You can use "Network Reachability" to declare which clusters can reach each other, and hence which ServiceEntry resources are created. This is not a formal access-control method; rather, it's an optimization to limit the number of ServiceEntry resources that are created so that clusters are not polluted with unneeded entries. A user could manually create additional ServiceEntries to access other clusters.

To configure Network Reachability, you need to:

1. Label the networks in each cluster
2. Define reachability rules using these labels

Label the networks and configure reachability

Using the TSE UI, make the following changes:

1. Edit cluster-1 and assign the name cluster-1-network to the cluster's network

   TSE Cluster Configuration: label the cluster network

2. Repeat for cluster-2, using the name cluster-2-network

3. Go to Settings and Network Reachability. Add a reachability setting to allow cluster-2-network to reach cluster-1-network:

   TSE Network Reachability: cluster-2 can reach cluster-1

TSE will now just create the ServiceEntry resources that allows cluster-2 to reach cluster-1.

Cleaning Up

You can remove the East-West Gateway and related services as follows:

On cluster-1, remove the East-West gateway and the sleep app:

kubectl delete ingressgateway -n bookinfo eastwest-gateway
kubectl delete -n bookinfo -f https://raw.githubusercontent.com/istio/istio/master/samples/sleep/sleep.yaml

Allow several minutes for the AWS Load Balancer and the cluster-2 service-entries to be deprovisioned.

On cluster-2, remove the sleep app and bookinfo namespace:

kubectl delete -n bookinfo -f https://raw.githubusercontent.com/istio/istio/master/samples/sleep/sleep.yaml
kubectl delete namespace bookinfo