Multi cluster traffic shifting with Tier-1 Gateway
This document describes how to use a Tier-1 Gateway for multi cluster traffic shifting. You will create one cluster for a Tier-1 Gateway deployment and two clusters for running bookinfo applications.
Each application cluster will have an Ingress Gateway configured to route traffic to the bookinfo application. Finally, you will configure the Tier-1 Gateway to shift the traffic from an application running on one cluster to another application on running on another cluster.
Before you get started, make sure you:
✓ Familiarize yourself with TSB concepts
✓ Familiarize yourself with TSB management plane and cluster onboarding. Following scenarios will assume that you have already installed a TSB management plane and you have tctl
configured to the correct management plane.
The following scenario has been tested on GKE Kubernetes clusters. However the steps described here should be generic enough to be used in other Kubernetes providers.
This scenario uses self-signed certificates for Istio CA and cert-manager to create certificates for XCP communications. The instructions here are for demo purposes only. For a production cluster setup, it is highly recommended to use a production-ready CA.
Tier-1 Gateway
There are two kinds of Gateways that receive incoming traffic in TSB: Tier-1 Gateways and Ingress Gateways (also called Tier-2 gateways). A Tier-1 Gateway distributes traffic across one or more ingress gateways in other clusters over Istio mTLS. An Ingress Gateway distributes traffic to one or more workloads (business application services) running in the cluster where the gateways is deployed.
There are several caveats you should be aware of with regards to Tier-1 deployments:
First, clusters that have a Tier-1 gateway deployed may not have any other gateways or workloads. You must use dedicated clusters for Tier-1.
Second Istio that runs on Tier-1 and Application clusters must share the same root CA. Refer to Istio docs on Plug in CA Certificates on how to set root and intermediate CA on Istio for multiple clusters. The TSB Control Plane Operator will deploy Istio and Istio's CA will read certificates from secrets-mount files from steps described in Plug in CA Certificates.
And finally, the application must be deployed in the same namespace on both clusters. This is because you will use one Ingress Gateway configuration for both application clusters.
Preparing Clusters
The following image shows the deployment architecture that you will use in this document. The management plane should already be deployed.
You will create a single Tier-1 Gateway cluster and two application clusters. Each application cluster has one Ingress Gateway and application workloads.
In your cloud provider, create the above three clusters: one for the Tier-1 gateway, and two for the applications.
Then plug in the certificates and keys into each cluster as described in Plug in CA Certificates documentation.
Onboarding Tier-1 Gateway and Application Clusters
Create a file called traffic-shifting-clusters.yaml with the following content. This will create the cluster resources for our use: The Tier-1 cluster is named t1
, and the application clusters are named c1
and c2
. You will need to use these names when referring to them in TSB configuration objects later.
# Application cluster 1.
apiVersion: api.tsb.tetrate.io/v2
kind: Cluster
metadata:
name: c1
organization: tetrate
spec:
displayName: 'Cluster 1'
network: tier2
tier1Cluster: false
tokenTtl: '8760h'
---
# Application cluster 2.
apiVersion: api.tsb.tetrate.io/v2
kind: Cluster
metadata:
name: c2
organization: tetrate
spec:
displayName: 'Cluster 2'
network: tier2
tier1Cluster: false
tokenTtl: '8760h'
---
# Tier-1 cluster
apiVersion: api.tsb.tetrate.io/v2
kind: Cluster
metadata:
name: t1
organization: tetrate
spec:
displayName: 'Tier-1 Cluster'
network: tier1
tier1Cluster: true
tokenTtl: '8760h'
Apply this using tctl
:
tctl apply -f traffic-shifting-clusters.yaml
Network Reachability
A cluster has a network
field representing a network boundary like a VPC on AWS/GCP/Azure. All clusters within the same network are assumed to be reachable to each other for multi-cluster routing. If your clusters are on different networks, you must configure them properly so that they are reachable from each other.
Please take note that in the cluster resources you have created, the Tier-1 cluster and application clusters have been assigned different networks: For the Tier-1 cluster the network is tier1
, and for the two application clusters the network is tier2
.
You will use these network names to tell TSB that tier1
and tier2
are reachable. Create a file named organization-settings.yaml with the following contents.
apiVersion: api.tsb.tetrate.io/v2
kind: OrganizationSetting
metadata:
name: tetrate-settings
organization: tetrate
spec:
networkSettings:
networkReachability:
# clusters that belong to tier1 networks can reach
# clusters that belong to tier2 networks.
tier1: tier2
Apply this using tctl
:
tctl apply -f organization-settings.yaml
Installing Control Plane components in Clusters.
At this point the clusters have been registered to TSB, but there are a few more things that need to be deployed onto these clusters. For each of the clusters, you need to perform the following operations:
- Deploy control plane operators
- Install control plane secrets
- Apply control plane custom resource to install TSB control plane components
When all clusters are onboarded properly, you should see the following information in the TSB UI. Note that clusters are reporting back the Istio and TSB agent versions.
Deploy Applications and Ingress Gateway to Application Cluster
For both application clusters, do the following
- Deploy the bookinfo application
- Deploy an Ingress Gateway
To deploy the Ingress gateway, create a file called bookinfo-ingress-deploy.yaml
with the following contents
apiVersion: install.tetrate.io/v1alpha1
kind: IngressGateway
metadata:
name: tsb-gateway-bookinfo
namespace: bookinfo
spec:
kubeSpec:
service:
type: LoadBalancer
Apply this using kubectl
:
kubectl apply -f bookinfo-ingress-deploy.yaml
Make sure that you are pointing kubectl
to the correct clusters when you apply the YAML file.
Note that we are using kubectl
for the previous step when deploying application and ingress gateway. In TSB, deployment and configuration are separate concepts and handled differently. You deploy with kubectl
directly to clusters and you configure with tctl
through the TSB management plane.
In this example, you use a LoadBalancer as a gateway service type. Depending on your Kubernetes environment (e.g. bare metal), you might need to use NodePort.
Typically LoadBalancer types are available from cloud providers. On GKE, this will spin up a Network Load Balancer that will give you a single IP address that will forward all traffic to your service. When using Kubernetes on your own infrastructure and not installing a load balancer service like MetalLB or PureLB, you will need to use NodePort. NodePort, opens a specific port on all the Nodes (the VMs), and any traffic that is sent to this port is forwarded to the service.
Tenant and Workspaces
In this example you are going to associate the Tier-1 gateway to a workspace and the two ingress gateways a another workspace. You should make sure that workspaces and a tenant that the workspaces belong to are configured properly.
Create a Tenant
If you have already configured a tenant in TSB, you can skip this section.
Create a file called traffic-shifting-tenant.yaml
with the following contents.
apiVersion: api.tsb.tetrate.io/v2
kind: Tenant
metadata:
organization: tetrate
name: tetrate
Apply this using tctl
.
tctl apply -f traffic-shifting-tenant.yaml
Create Workspaces
Create workspaces to associate the gateways. Create a file named traffic-shifting-workspaces.yaml
.
# workspace for bookinfo
apiversion: api.tsb.tetrate.io/v2
kind: Workspace
metadata:
tenant: tetrate
organization: tetrate
name: bookinfo-workspace
spec:
description: for bookinfo
displayName: bookinfo
namespaceSelector:
names:
- 'c1/bookinfo'
- 'c2/bookinfo'
---
# workspace for tier-1
apiversion: api.tsb.tetrate.io/v2
kind: Workspace
metadata:
tenant: tetrate
organization: tetrate
name: tier1-workspace
spec:
description: for tier1
displayName: tier1
namespaceSelector:
names:
- 't1/tier1'
Apply this using tctl
:
tctl apply -f traffic-shifting-workspaces.yaml
If you want to use existing workspaces, you can update the workspace to include clusters and namespaces that you just created by updating the workspace namespace selector.
Configure Ingress Gateway
Next, you will configure the Ingress Gateway to receive traffic for bookinfo applications in both application clusters.
Before configuring the Ingress gateways, create a TLS certificate using this script. Make sure to create the secrets in the bookinfo
namespace in both application clusters.
Create a file named traffic-shifting-bookinfo-ingress-config.yaml
.
apiVersion: gateway.tsb.tetrate.io/v2
kind: Group
metadata:
tenant: tetrate
organization: tetrate
workspace: bookinfo-workspace
name: bookinfo-gateway-group
spec:
displayName: bookinfo-gateway-group
description: for bookinfo-gateway
namespaceSelector:
names:
- 'c1/bookinfo'
- 'c2/bookinfo'
configMode: BRIDGED
---
apiVersion: gateway.tsb.tetrate.io/v2
kind: IngressGateway
metadata:
name: bookinfo-gateway
group: bookinfo-gateway-group
workspace: bookinfo-workspace
tenant: tetrate
organization: tetrate
spec:
workloadSelector:
namespace: bookinfo
labels:
app: tsb-gateway-bookinfo
http:
- name: bookinfo-gateway
port: 443
hostname: bookinfo.tetrate.com
tls:
mode: SIMPLE
# make sure to use correct secret name that you created previously
secretName: bookinfo-certs
routing:
rules:
- route:
host: 'bookinfo/productpage.bookinfo.svc.cluster.local'
port: 9080
Apply this using tctl
:
tctl apply -f traffic-shifting-bookinfo-ingress-config.yaml
The Ingress gateway configuration will automatically be pushed to both application clusters, as the configuration above specifies the clusters in the namespaceSelector
section of the Group
object.
Deploy and configure Tier-1 Gateway
Create a file named traffic-shifting-tier1-deploy.yaml
with the following contents.
apiVersion: install.tetrate.io/v1alpha1
kind: Tier1Gateway
metadata:
name: tier1-gateway
namespace: tier1
spec:
kubeSpec:
service:
type: LoadBalancer
Deploy this using kubectl
:
kubectl apply -f traffic-shifting-tier1-deploy.yaml
Create a file named traffic-shifting-tier1-config.yaml
with the following contents.
You will use the same bookinfo TLS certificate that you have created earlier for the Ingress gateway. In the following yaml you route all incoming traffic to the first application cluster, that you named c1 when you onboarded the clusters in the previous step.
apiVersion: gateway.tsb.tetrate.io/v2
kind: Group
metadata:
tenant: tetrate
organization: tetrate
workspace: tier1-workspace
name: tier1-gateway-group
spec:
displayName: tier1-gateway-group
description: for tier1-gateway-group
namespaceSelector:
names:
- 't1/tier1'
configMode: BRIDGED
---
apiVersion: gateway.tsb.tetrate.io/v2
kind: Tier1Gateway
metadata:
name: tier1-gateway
group: tier1-gateway-group
workspace: tier1-workspace
tenant: tetrate
organization: tetrate
spec:
workloadSelector:
namespace: tier1
labels:
app: tier1-gateway
externalServers:
- hostname: bookinfo.tetrate.com
name: bookinfo
port: 443
tls:
mode: SIMPLE
# make sure to use correct secret name that you created previously
secretName: bookinfo-certs
clusters:
- name: c1
weight: 100
Configure the Tier-1 gateway using tctl
:
tctl apply -f traffic-shifting-tier1-config.yaml
At this point, you should be able to send requests to the Tier-1 Gateway. Get the Tier-1 public IP address using the Tier-1 cluster kubeconfig.
export GATEWAY_IP=$(kubectl -n tier1 get service tier1-gateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
curl -H "X-B3-Sampled: 1" "https://bookinfo.tetrate.com/productpage" --resolve "bookinfo.tetrate.com:443:${GATEWAY_IP}" -v --cacert bookinfo-ca.crt
Traffic Shifting
Now that you have a Tier-1 gateway installed and configured, you can configure traffic shifting using it. Traffic shifting is the act of gradually migrating traffic from one version to another version of applications or services.
In the previous configuration, all traffic from Tier-1 Gateway was routed to the Ingress Gateway in the bookinfo application that runs in the cluster c1
. Suppose that you have a newer version of the bookinfo application and it runs in the other cluster c2
.
In a scenario like this, it is usually desirable to configure it such that only a small percentage of the traffic is routed to the new cluster c2
, so that you can test and observe if the application in the new cluster is working as expected. When you have verified that the there are no issues, the traffic percentage that is routed to c2
can be increased incrementally until all traffic is routed to c2
. At which point c1
can safely be taken offline.
To shift application traffic from cluster c1
to cluster c2
, create a file called traffic-shifting-tier1-config2.yaml
(or you may edit the previous configuration file) and apply with tctl
.
apiVersion: gateway.tsb.tetrate.io/v2
kind: Group
metadata:
tenant: tetrate
organization: tetrate
workspace: tier1-workspace
name: tier1-gateway-group
spec:
displayName: tier1-gateway-group
description: for tier1-gateway-group
namespaceSelector:
names:
- 't1/tier1'
configMode: BRIDGED
---
apiVersion: gateway.tsb.tetrate.io/v2
kind: Tier1Gateway
metadata:
name: tier1-gateway
group: tier1-gateway-group
workspace: tier1-workspace
tenant: tetrate
organization: tetrate
spec:
workloadSelector:
namespace: tier1
labels:
app: tier1-gateway
externalServers:
- hostname: bookinfo.tetrate.com
name: bookinfo
port: 443
tls:
mode: SIMPLE
# make sure to use correct secret name that you created previously
secretName: bookinfo-certs
clusters:
- name: c1
weight: 90
- name: c2
weight: 10
Below is a diff between the configuration the original YAML and the new one. Note that the Group
definition has been included again for completeness, but it can be omitted.
@@ -36,4 +36,6 @@
secretName: bookinfo-certs
clusters:
- name: c1
- weight: 100
+ weight: 90
+ - name: c2
+ weight: 10
Using this configuration, the Tier-1 gateway will route 10% of traffic to the Ingress Gateway in cluster c2
, and 90% to the cluster c1
. You can then increase the traffic that is routed to c2
incrementally until it reaches 100%.