Upgrade a Cluster

Before You Begin

Ensure the following tools are available and properly configured:

• kubectl
• helm
• tctl (optional - recommended for health checks)

Upgrading a Cluster

Set these once for the commands below:

export CP=<cp-context>          # your kube context for the cluster to be upgraded
export NS=istio-system          # control-plane namespace
export TARGET_VERSION=1.14.2    # the TSB version you're upgrading to

How to upgrade

The following instructions appear lengthy. Most of the instructions relate to checking the health of the cluster and backing the configuration up before performing the upgrade. These processes can be performed at any time to verify the health of a cluster.

The upgrade is performed with a single helm upgrade command, which typically takes a minute or two to apply and converge.

1. Verify the current state of the cluster

   # Check that all pods in $NS are running without error
   kubectl --context $CP -n $NS get pods
   
   # Check the health of the installation
   helm --kube-context $CP -n $NS history controlplane | tail -2
   
   # Note the current version of the control plane operator
   kubectl --context $CP -n $NS get deploy tsb-operator-control-plane \
     -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}' 

2. Verify the health of the cluster

   Do not proceed if any of the following checks do not pass; contact Tetrate Technical Support if necessary to resolve any errors.

   Configuration prior to health checks

   Point kubectl at the control plane cluster before starting. The tctl debug commands use the current kubeconfig context.

   If the control plane is installed in non-default namespaces, pass -n (control plane), -d (data plane), or -m (management plane) to each tctl debug command. Defaults are istio-system, istio-gateway, and tsb.

   1. List the Tetrate components

   tctl debug list-components

   You should not see any errors when you list the components.

   2. Check component health

   Check the health of the following control plane components:

   tctl debug health cp/tsb-operator
   tctl debug health cp/onboarding
   tctl debug health cp/wasmfetcher

   Each returns All Services Operational when healthy.

   3. Check Pod Status

   kubectl get pods -n istio-system

   Verify that every pod is Running or Completed. Investigate any pod that is not ready, or whose restart count is increasing.

   For edge, xcp-operator-edge, and otel-collector, you can inspect recent logs:

   kubectl logs -n istio-system deploy/edge -c edge --since=24h | grep -E $'\t(error|fatal)\t'
   kubectl logs -n istio-system deploy/xcp-operator-edge -c xcp-operator --since=24h | grep -E $'\t(error|fatal)\t'
   kubectl logs -n istio-system deploy/otel-collector -c otel-collector --since=24h | grep -E $'\t(error|fatal)\t'

   For oap-deployment:

   kubectl logs -n istio-system deploy/oap-deployment -c oap --since=24h | grep -E '\b(ERROR|FATAL)\b'

   No output means no errors in the "--since" window.

   Error Messages are common and expected when installing or upgrading

   Note that Error Messages are common when installing or upgrading while the system waits for all services to start and synchronize. If you installed or upgraded the control plane recently, reduce the --since window to only check logs after the install or upgrade.

   4. Check configuration propagation

   List the managed clusters with tctl get clusters. Check the status of the cluster you plan to upgrade:

   tctl status cluster <cluster-name>

   A healthy cluster reports a status of READY. Do not proceed if the cluster is not healthy.

   5. Capture full Tetrate system state

   If desired, you can generate a full configuration and status dump; this information will prove useful for Tetrate Technical Support should you need assistance.

   tctl collect --disable-archive

   Check the bundle for credentials before sharing it (grep -rI "Authorization: Bearer" tctl-debug-*/)

3. Back up your cluster

   Capture everything needed to repair or re-onboard the control plane:

   mkdir -p cp-backup && cd cp-backup
   
   # Control Plane configuration
   kubectl --context $CP -n $NS get controlplane -o yaml > 01-controlplane-cr.yaml
   helm   --kube-context $CP -n $NS get values controlplane -a > 02-controlplane-values.yaml
   
   # Cluster secrets — REQUIRED to re-onboard (store these encrypted, never in git)
   kubectl --context $CP -n $NS get secret -o yaml > 03-secrets.yaml
   
   # The annotated Services that drive the agent (your source of truth for gateways)
   kubectl --context $CP get svc -A -o yaml > 04-services.yaml
   
   # Snapshot of agent-generated gateways (for an after-upgrade comparison)
   kubectl --context $CP -n tetrate-system get gateways.install.tetrate.io,gateway.networking.istio.io -o yaml > 05-gateways.yaml
   cd -

   If you have installed the Tetrate Hosted Agent, you should also back that configuration up:

   # Hosted Agent configuration (if present)
   helm   --kube-context $CP -n $NS get values tetrate-hosted-agent -a > 06-hosted-agent-values.yaml 2>/dev/null \
     || kubectl --context $CP -n $NS get deploy tetrate-hosted-agent -o yaml > 06-hosted-agent-deploy.yaml

   Keep cp-backup/ safe. The secrets.yaml file contains credentials — store it encrypted.

4. Upgrade the Control Plane

   helm repo add tetrate-tsb-helm 'https://charts.dl.tetrate.io/public/helm/charts/' && helm repo update
   
   # Capture the values that are ALREADY running your control plane (your onboarding config)
   
   helm --kube-context $CP -n $NS get values controlplane > cp-values.yaml
   
   # Upgrade using exactly those values — only the version changes
   
   helm upgrade controlplane tetrate-tsb-helm/controlplane \
     --kube-context $CP \
     --namespace $NS \
     --version $TARGET_VERSION --timeout 10m \
     --values cp-values.yaml

   tip

   helm get values pulls the live configuration (MP address, cluster identity, registry, overlays) from the running release, so you don't need the original onboarding files. Only the version changes; the operator then rolls the mesh components (istiod, xcp-edge, oap) to $TARGET_VERSION automatically.

   Watch it converge:

   kubectl --context $CP -n $NS get deploy tsb-operator-control-plane \
     -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
   watch "kubectl --context $CP -n $NS get pods"

   The upgrade is complete when istiod, edge, and xcp-operator-edge are all Running/Ready on the new version.

5. Check the Hosted Agent

   No change is needed to the agent for this upgrade — just confirm it's healthy:

   kubectl --context $CP -n $NS logs deploy/tetrate-hosted-agent --tail=50

   Expect INFO lines only; a couple of benign and transient WARNs are normal.

6. Verify the Upgrade

   Verify that your gateways survived the upgrade

   Gateways should be updated in place, not recreated. Compare their uid and creationTimestamp values to the backup; these values should not change:

   kubectl --context $CP -n tetrate-system get gateway.networking.istio.io \
     -o custom-columns='NAME:.metadata.name,UID:.metadata.uid,CREATED:.metadata.creationTimestamp'

   Verify that the agent still creates gateways

   Apply a test Service and confirm a gateway appears:

   kubectl --context $CP apply -f - <<'EOF'
   apiVersion: v1
   kind: Service
   metadata:
     name: echo-check
     namespace: <your-app-namespace>     # must have: istio-injection: enabled
     annotations:
       gateway.tetrate.io/host: "echo.example.com"
       gateway.tetrate.io/protocol: "HTTP"
       gateway.tetrate.io/port: "80"
       gateway.tetrate.io/expose: "local"
       gateway.tetrate.io/auto-deploy: "true"
   spec:
     selector: { app: echo-check }
     ports: [ { name: http, port: 8080, targetPort: 8080 } ]
   EOF
   
   # The agent log should show "Gateway created successfully", and a new Istio Gateway appears:
   kubectl --context $CP -n $NS logs deploy/tetrate-hosted-agent --tail=20
   kubectl --context $CP -n tetrate-system get gateway.networking.istio.io

   Clean up the test when done, using kubectl --context $CP -n <your-app-namespace> delete svc echo-check.

Reload Existing Sidecar-enabled Deployments

Existing gateways are upgraded when the control plane is upgraded, but the sidecars deployed on services are not upgraded. The Tetrate Control Plane can manage sidecars of a previous generation, and the latest sidecar version is deployed whenever a new pod is rolled out, so upgrading sidecars is often not an immediate priority.

At any point, you can rolling-restart a deployment to upgrade the sidecar:

kubectl rollout restart -n NS deploy DEPLOYMENT

Re-install the Previous Control Plane

In the event of a failed upgrade, you can re-install the previous control-plane version from your backup:

helm upgrade controlplane tetrate-tsb-helm/controlplane --kube-context $CP \
  --version <previous-version> --namespace $NS \
  --values cp-backup/02-controlplane-values.yaml

Do not skip versions backward across CRD changes. If in doubt, contact Tetrate support and provide your cp-backup/ folder.