CP: Revisioned to Revisioned

Before you continue, make sure you are familiar with Istio Isolation Boundaries feature.

Control plane Upgrade Strategy

For revision-to-revision upgrades, when the user upgrades the control plane cluster to the latest TSB release using Helm or tctl, TSB automatically upgrades the Istio installation to the latest version if the IsolationBoundary configuration does not specify istio.tsbVersion.

We recommend the following IsolationBoundary configuration to make the upgrade seamless as mentioned in Revisioned Upgrades

spec:
  ...
  components:
    xcp:
      isolationBoundaries:
      - name: global
        revisions:
        - name: default
          istio:
            tsbVersion: 1.11.0 // Step 2 - Update old release version from `1.11.0` to `1.12.0` after validating `canary` release.
        - name: canary         // Step 1 - TSB automatically upgrades istio installation under canary revision to new release version i.e `1.12.0`.

Control plane Canary Upgrades

For canary upgrades of control plane clusters, you can follow the steps below to facilitate a controlled upgrade of Dataplane components.

Step 1: Upgrade control plane cluster to the new TSB release version

Assume you have TSB version 1.11.0 running on the control plane clusters and you want to upgrade to 1.12.0. This step assumes you already have IsolationBoundary enabled and installed based on Isolation Boundaries Installation.

Upgrade Using Helm

Obtain Helm values of control plane used before for installation

helm get values controlplane -n istio-system -o yaml > <CLUSTER_NAME>-cp.yaml

Upgrade the control plane cluster to the latest TSB release version.

helm upgrade controlplane -n istio-system -f <CLUSTER_NAME>-cp.yaml tetrate-tsb-helm/controlplane --version <UPGRADE_VERSION>

This would by default update istio control plane (revision canary) to the new TSB release version i.e 1.12.0. This state will be reconciled by the xcp-operator-edge operator in istio-system namespace. You can check the status of istio operators and istiod deployments to verify.

kubectl get iop -n istio-system

# Output
NAMESPACE      NAME              REVISION   STATUS    AGE
istio-system   xcp-iop-canary    canary     HEALTHY   8m36s
istio-system   xcp-iop-default              HEALTHY   8m36s

kubectl get deployment -n istio-system | grep istiod

# Output
istiod                       1/1     1            1           9m55s
istiod-canary                1/1     1            1           9m56s

Step 2: Upgrade default revision to the new TSB release version

Once you verify the new release version and corresponding istio proxy changes by verifying the workloads under canary revision, you can safely upgrade the default revision to the new TSB release version.

Update the istio.tsbVersion under default revision to the latest ( i.e 1.12.0) in the control plane yaml after verifying the new release changes

spec:
  ...
  components:
    xcp:
      isolationBoundaries:
      - name: global
        revisions:
        - name: default
          istio:
            tsbVersion: 1.12.0
        - name: canary

Apply the changes using helm/tctl

helm upgrade controlplane -n istio-system -f <CLUSTER_NAME>-cp.yaml tetrate-tsb-helm/controlplane --version <UPGRADE_VERSION>

Control plane In-Place Upgrades

For an in-place upgrade, omit istio.tsbVersion under the default revision. TSB will automatically upgrade the Istio control plane to the latest version after you upgrade the control plane to the latest TSB release.

spec:
  ...
  components:
    xcp:
      isolationBoundaries:
      - name: global
        revisions:
        - name: default

Once you upgrade, Istio control plane components will be upgraded automatically to the latest release version. This state will be reconciled by the xcp-operator-edge operator in istio-system namespace.

Workload Upgrade - In-Place

Gateway upgrade

By default, gateways will be upgraded automatically to use latest tsbVersion. See Gateway upgrades for more details on gateway upgrade behavior.

Application upgrade

Since the revision name does not change, no updates are required in the workload namespaces (workload-ns in this example). However you still need to restart the application workloads. A rolling update is preferred to avoid traffic disruptions.

kubectl rollout restart deployment -n workload-ns

VM workload upgrade

To upgrade VM workload, download latest Istio sidecar from your onboarding plane using revisioned link then reinstall Istio sidecar on the VM.

Then restart onboarding-agent running in the VM.

Workload Upgrade - Revisioned

Gateway upgrade

To upgrade the Gateways, update the spec.revision in the Ingress/Egress/Tier1Gateway resource. This will reconcile the existing gateway pods to connect to the new revisioned Istio control plane. TSB by default configures the Gateway install resources with a RollingUpdate strategy that ensures zero downtime.

You can also update spec.revision by patching gateway CR.

kubectl patch ingressgateway.install <name> -n <namespace> --type=json --patch '[{"op": "replace","path": "/spec/revision","value": "1-6-1"}]'; \

Application upgrade

To upgrade sidecars, replace istio.io/rev=stable workload namespace label and apply the new revision.

kubectl label namespace workload-ns istio.io/rev=1-6-1 --overwrite=true

Then restart the application workloads. A rolling update is preferred to avoid traffic disruptions.

kubectl rollout restart deployment -n workload-ns

VM workload upgrade

To upgrade VM workload, download latest Istio sidecar from your onboarding plane using revisioned link then reinstall Istio sidecar on the VM.

Update revision value in onboarding-agent configuration then restart onboarding-agent.

Post-upgrade cleanup

A revision that is no longer in use can either be removed or marked "disabled" in the ControlPlane CR as mentioned below. Marking it disabled helps in enabling the revision back at any point in time.

Disabling the revision

spec:
  ...
  components:
    xcp:
      isolationBoundaries:
      - name: global
        revisions:
        - name: default
          istio:
            tsbVersion: 1.8.0
          disable: true
        - name: canary
          istio:
            tsbVersion: 1.9.0

Removing the revision

spec:
  ...
  components:
    xcp:
      isolationBoundaries:
      - name: global
        revisions:
        - name: default
          istio:
            tsbVersion: 1.9.0

Stale Istio control plane components

After disabling/removing the revision under isolation boundaries, a few stale components might remain. For instance, IstioOperator resource, istio-operator (revisioned) deployment or istiod (revisioned) deployment. This happens due to a race condition in removing the IstioOperator resource and istio-operator deployment. In that case, such Istio components can be removed like regular kubernetes objects using

kubectl delete iop xcp-iop-stable -n istio-system
kubectl delete deployment istio-operator-stable -n istio-system
kubectl delete configmap istio-sidecar-injector-stable -n istio-system
kubectl delete deployment istiod-stable -n istio-system

Rollback from revisioned to revisioned

This workflow is similar to upgrading from revisioned to revisioned control plane. You need to update your workloads to use old revision.