TSB Helm Upgrade
This document explains how you can leverage Helm Charts to upgrade the different elements of TSB. The document assumes that Helm is already installed in the system.
This document only applies to TSB instances created with Helm, not to upgrade from TCTL based installations.
Before you start, make sure that you have:
✓ Checked the new version's requirements
✓ Read the upgrade notes upgrade notes
Please ensure you thoroughly understand the upgrade notes before proceeding with the upgrade.
Prerequisites
Configure the Helm repository
- Add the repository:
helm repo add tetrate-tsb-helm 'https://charts.dl.tetrate.io/public/helm/charts/'
helm repo update - List the available versions:
helm search repo tetrate-tsb-helm -l
Backup the PostgreSQL database
Create a backup of your PostgreSQL database.
The exact procedure for connecting to the database may differ depending on your environment, please refer to the documentation for your environment.
Upgrade process
Verify the available TSB versions to upgrade and always aim to upgrade to the latest patch version:
helm search repo tetrate-tsb-helm --versions
And export the version as a variable:
export TSB_VERSION="<version-to-upgrade>"
Management Plane
Upgrade the management plane chart:
helm upgrade mp tetrate-tsb-helm/managementplane --set image.registry=${HUB} --set image.tag=${TSB_VERSION} --namespace tsb -f values-mp.yaml
Control Plane
Upgrade the control plane chart:
helm upgrade cp tetrate-tsb-helm/controlplane --set image.registry=${HUB} --set image.tag=${TSB_VERSION} --namespace istio-system -f values-cp.yaml --set-file secrets.clusterServiceAccount.JWK=/tmp/<cluster>.jwk
Data Plane
Upgrade the control plane chart:
helm upgrade dp tetrate-tsb-helm/dataplane --set image.registry=${HUB} --set image.tag=${TSB_VERSION} --namespace istio-gateway -f values-dp.yaml
Check the upgrade is successful
Once the upgrade is completed, review the following information from the Grafana dashboards to verify the upgrade status.
Management Plane dashboards
-
TSB Health
provides the status for the different management plane components. All of them must be green before and after the upgrade. -
TSB Operational Status
provides more details from the TSB API. Checking previous values and compare with the values after the upgrade will give an insight of the performance. It also shows the error rate and latency, not only for the TSB API but also for the Postgres database. -
Global Configuration Distribution
provides information about config propagation from the management plane to the control planes. -
MPC Operational Status
shows the latency or errors that config generation system have. It's important to compare theGet All Config Objects duration
as it will show the performance after the upgrade. Received configs shouldn't be empty and it will also show the updates from the TSB API to XCP.
Control Plane dashboards
-
Gitops Operational Status
if Gitops is enabled in the control planes, provides information about accepted/rejected configurations and performance. -
Control Plane Token Status
shows if tokens are being rotated and if they are valid for each control plane. -
XCP Edge status
shows the status of XCP Edge in each control plane. Latency and performance for the Edges is shown and also if communication with Central is successful. -
Istio Operational Status
shows important information for each control plane like the error rate fromistiod
, the proxy convergence time and the time to root CA expiration.
Rollback
In case something goes wrong and you want to rollback TSB to the previous version, you will need to rollback the Management Plane, the Control Planes and the Data Planes charts.
Rollback the Control Plane
You can use helm rollback
to rollback the current revision. To see the current revisions, you can run:
helm history cp -n istio-system
And then you can rollback to the previous revision:
helm rollback cp <REVISION> -n istio-system
Rollback the Management Plane
Scale Down Pods in Management Plane
Scale down all of the pods that talk to Postgres in the Management Plane so that the it is inactive.
kubectl scale deployment tsb iam -n tsb --replicas=0
Restore PostgreSQL
Restore your PostgreSQL database from your backup. The exact procedure for connecting to the database may differ depending on your environment, please refer to the documentation for your environment.
Restore Management Plane
helm rollback mp <REVISION> -n tsb