Non-revisioned to Revisioned

Before you continue, make sure you are familiar with Istio Isolation Boundaries feature.

Before you upgrade

Upgrading from a non-revisioned to revisioned control plane setup involves enabling the Istio isolation boundary feature. Once enabled, a revision can be configured within the isolation boundary to which the control plane must upgrade to. Follow the steps mentioned in Isolation Boundaries Installation to deploy the control plane with isolation boundary feature enabled.

Once the Istio isolation boundary feature is enabled, the TSB data plane operator components will be automatically scaled down. This includes -

1. Scaling down the tsb-operator-data-plane deployment and the istio-operator deployment in istio-gateway namespace.
2. Removing the TSB webhooks in the istio-gateway namespace.
3. Marking the istio-gateway namespace for deletion. This does not automatically deletes the namespace, since there might be stale IOP objects in the namespace. These stale IOPs will be removed once the upgrade is complete.

Control plane upgrade strategy

In case of non-revisioned to revisioned upgrades, TSB rolls out revisioned Istio control plane in place of non-revisioned Istio control plane. This means that Canary upgrade of the Istio control plane is not supported. This allows non-revisioned workloads to connect to a default revisioned istio control plane. Therefore while upgrading from non-revisioned to a revisioned setup, the workload namespaces do not require any label changes.

Control plane

Configure an isolation boundary in your ControlPlane CR. If you use Helm, you can add isolation boundary configuration in your Helm values file.

spec:
  hub: <registry-location>
  telemetryStore:
    elastic:
      host: <tsb-address>
      port: <tsb-port>
      version: <elastic-version>
      selfSigned: <is-elastic-use-self-signed-certificate>
  managementPlane:
    host: <tsb-address>
    port: <tsb-port>
    clusterName: <cluster-name-in-tsb>
    selfSigned: <is-mp-use-self-signed-certificate>
  components:
    xcp:
      isolationBoundaries:
      - name: global
        revisions:
        - name: default
      centralAuthMode: 'JWT'

"global" isolation boundary

Existing workspaces that are already deployed in the cluster will NOT be bound to a specific isolation boundary, therefore a workspace with no isolation boundary specified is considered to be a part of the "global" boundary.

Although you can deploy multiple revisioned control planes after enabling isolation boundaries support, with any boundary "name" whatsoever. But if you are configuring a single isolation boundary, then the boundary name has to be configured as "global", so that existing Workspaces can be considered as part of the "global" isolation boundary. TSB Webhook validation will ensure this in each control-plane cluster.

Now with this change, TSB makes sure multi cluster connectivity and traffic distribution will not be impacted during upgrade, as the workloads in non-revisioned clusters and clusters upgraded with "global" boundary would be able to discover each other, hence the existing connectivity and service discovery will work as is without any change.

"default" revision

It is recommended to use the "default" revision name while upgrading from a non-revisioned setup, so the workload namespaces wouldn't require any label change i.e. the existing label istio-injection=enabled can remain as is.

Configuring an isolation boundary in the ControlPlane CR will setup a revisioned control plane in the istio-system namespace, although the name of the istio control plane components will be unchanged.

Gateway upgrade

All the existing Gateways are automatically upgraded to have the default revision configured in the spec.revision, once ISTIO_ISOLATION_BOUNDARIES is enabled, and the ControlPlane CR is configured as shown above. As part of this, the ownership of data plane reconciliation is also handed over to the XCP Edge Operator, which reconciles the new/upgraded data plane Gateways. TSB by default configures the Gateway install resources with a RollingUpdate strategy that ensures minimal downtime during the upgrade.

Application upgrade

Since the configured revision is set to "default" in the ControlPlane CR, there is no need to update the workload namespace labels, and the existing label istio-injection=enabled can remain as is. Although, setting the workload namespace labels to istio.io/rev=default will also have the same effect.

Restart application workloads so they connect to the new revisioned Istio control plane. A rolling update is preferred to avoid traffic disruptions.

kubectl rollout restart deployment -n workload-ns

VM workload upgrade

To upgrade VM workload, download latest Istio sidecar from your onboarding plane using revisioned link then reinstall Istio sidecar on the VM.

Update onboarding-agent configuration with revision value then restart onboarding-agent. Istio sidecar will connect to revisioned Istio control plane.

Post-upgrade cleanup

Once all sidecars have moved to the revisioned proxy and all application gateways have revisioned gateways running, and a healthy upgrade is ensured, we can proceed to cleanup the old non-revisioned resources from the cluster which are now stale.

1. Cleanup the stale IOP resources that are present in the istio-gateway namespace. This will automatically trigger the deletion of the istio-gateway namespace.

kubectl get iop -n istio-gateway -o name | xargs -I % kubectl patch % -n istio-gateway -p '{"metadata":{"finalizers":null}}' --type=merge;

2. Delete IstioOperator resource named tsb-istiocontrolplane from the namespace istio-system using kubectl.

kubectl delete iop tsb-istiocontrolplane -n istio-system; \
kubectl patch iop tsb-istiocontrolplane -n istio-system -p '{"metadata":{"finalizers":null}}' --type=merge;