Non-revisioned to Revisioned

Starting from TSB 1.12.0, IsolationBoundary is a required feature that cannot be disabled. This guide provides a walkthrough for enabling Istio isolation boundary feature on your existing pre-1.12.0 TSB control plane and migrate your workloads from non-revisioned to revisioned before upgrading to the latest TSB version.

Before you continue, make sure you are familiar with Istio Isolation Boundaries feature.

Important Notes Before Upgrade

1. Only migration to the global boundary with default revision is supported for rollback. Rollback with multiple revisions is not supported as it is unclear which revision to roll back to.
2. Before proceeding with the upgrade:
   1. Backup your Helm values files.
   2. Verify all workloads are healthy and connected to Istiod.
   3. Ensure you have necessary image pull secrets configured.

Enabling Isolation Boundary

Upgrading from a non-revisioned to revisioned control plane setup involves enabling the Istio isolation boundary feature. Once enabled, a revision can be configured within the isolation boundary to which the control plane must upgrade to.

To enable Istio isolation boundary feature, you need to set the ISTIO_ISOLATION_BOUNDARIES environment variable to true in the TSB control plane operator. Following is an example of how to set the environment variable in the TSB control plane Helm values file.

operator:
  deployment:
    env:
      - name: ISTIO_ISOLATION_BOUNDARIES 
        value: "true"

By default TSB installs global boundary and default revision in the controlplane clusters, this allows non-revisioned workloads to connect to a default revisioned istio control plane.

If you manage the TSB control plane via GitOps by maintaining the ControlPlane CR YAML in a Git repository, we recommend including the following IsolationBoundary configuration in your control plane YAML:

spec:
  ...
  components:
    xcp:
      isolationBoundaries:
      - name: global
        revisions:
          name: default

Control Plane Effects

Once isolation boundaries are enabled:

1. A revisioned control plane will be set up in the istio-system namespace. The names of the Istio control plane components remain unchanged.
2. Multi-cluster connectivity and traffic distribution are maintained:
   1. Workloads in non-revisioned clusters can discover workloads in clusters upgraded with the global boundary
   2. Existing service discovery continues to work without changes

Important Notes

1. Using revision name default allows existing workload namespaces to keep their istio-injection=enabled label
2. The global boundary name is required when configuring a single isolation boundary
3. Consider adding a canary revision during initial setup for future revisioned-to-revisioned upgrades

Data Plane Operator Effects

When isolation boundaries are enabled, several changes occur in the data plane operator:

1. The TSB data plane operator components in istio-gateway namespace are automatically scaled down:
   1. tsb-operator-data-plane deployment is scaled down
   2. TSB webhooks in the istio-gateway namespace are removed
2. Gateway management changes:
   1. Ownership of gateway reconciliation transfers from TSB operator to XCP Edge operator
   2. Existing gateways are automatically upgraded to use the default revision
   3. Gateway deployments use RollingUpdate strategy to ensure minimal downtime
3. The istio-gateway namespace:
   1. Is marked for deletion after operator components are scaled down
   2. May require manual cleanup of finalizers if deletion is blocked
   3. Contains stale IOP resources that should be cleaned up during post-upgrade steps

Gateway Transition

During the transition, gateway management is handled gracefully with no traffic disruption. Gateways continue serving traffic while being upgraded to use the new revisioned control plane.

Workload Upgrade

Gateway upgrade

All the existing Gateways are automatically upgraded to have the default revision configured in the spec.revision, once ISTIO_ISOLATION_BOUNDARIES is enabled, and the ControlPlane CR is configured as shown above.

As part of this, the ownership of data plane reconciliation is also handed over to the XCP Edge Operator, which reconciles the new/upgraded data plane Gateways. TSB by default configures the Gateway install resources with a RollingUpdate strategy that ensures minimal downtime during the upgrade.

Application upgrade

Since the configured revision is set to default in the ControlPlane CR, there is no need to update the workload namespace labels, and the existing label istio-injection=enabled can remain as is. Although, setting the workload namespace labels to istio.io/rev=default will also have the same effect.

Restart application workloads so they connect to the new revisioned Istio control plane. A rolling update is preferred to avoid traffic disruptions.

kubectl rollout restart deployment -n workload-ns

VM workload upgrade

To upgrade VM workload, download latest Istio sidecar from your onboarding plane using revisioned link then reinstall Istio sidecar on the VM.

Update onboarding-agent configuration with revision value then restart onboarding-agent. Istio sidecar will connect to revisioned Istio control plane.

Post Upgrade Cleanup

Once all sidecars have moved to the revisioned proxy and all application gateways have revisioned gateways running, and a healthy upgrade is ensured, we can proceed to cleanup the old non-revisioned resources from the cluster which are now stale.

1. Cleanup the stale IOP resources that are present in the istio-gateway namespace. This will automatically trigger the deletion of the istio-gateway namespace.

   kubectl get iop -n istio-gateway -o name | xargs -I % kubectl patch % -n istio-gateway -p '{"metadata":{"finalizers":null}}' --type=merge;

2. Delete IstioOperator resource named tsb-istiocontrolplane from the namespace istio-system using kubectl.

   kubectl delete iop tsb-istiocontrolplane -n istio-system; \
   kubectl patch iop tsb-istiocontrolplane -n istio-system -p '{"metadata":{"finalizers":null}}' --type=merge;

3. Delete stale webhook created by IstioOperator

   kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io istiod-default-validator

Rollback to Non-revisioned Setup

If you need to roll back from isolation boundary to non-revisioned setup, follow these steps:

1. First, reinstall the data plane operator in the istio-gateway namespace:

   kubectl create ns istio-gateway
   helm install tsb-dp tetrate-tsb-helm/dataplane \
     --version <your-tsb-version> \
     --namespace istio-gateway \
     --create-namespace \
     --values <your-dp-values.yaml> \
     --set image.registry=<your-registry>

2. Roll back the control plane using Helm:

   Check Helm revision history

   helm history tsb-cp -n istio-system

   Roll back to the previous revision (before IB was enabled)

   helm rollback tsb-cp <previous-revision-number> -n istio-system

3. After rollback completes:

   • The control plane components will restart
   • Workloads will automatically reconnect to the rolled-back Istiod
   • Gateway ownership will return to TSB operator

4. Clean up any remaining XCP-managed IOPs:

   kubectl delete iop xcpgw-ingress-<uuid> -n istio-system
   kubectl patch iop xcpgw-ingress-<uuid> -n istio-system -p '{"metadata":{"finalizers":null}}' --type=merge

Important Rollback Considerations

• Only rollback to global boundary with default revision is supported
• Ensure you have backups of your Helm values before attempting rollback
• Monitor application traffic during rollback to ensure no disruption
• Verify all workloads successfully reconnect to the rolled-back control plane

Istio overlays

If you are using overlays to customize the Istio behavior, you would need to update the overlays by moving overlays from the spec.components.istio to spec.components.xcp.isolationBoundaries section under the correct revision. If you have multiple revisions, you would need to add the overlays for each revision.

For example, if you have an overlay to customize the Istio logging level, you would need to update the overlay from following

spec:
  components:
    istio:
      kubeSpec:
        overlays:
        - apiVersion: install.istio.io/v1alpha1
          kind: IstioOperator
          name: tsb-istiocontrolplane
          patches:
          - path: spec.values.global.proxy.logLevel
            value: warn
          - path: spec.values.global.logging.level
            value: default:warn

to following

spec:
  components:
    xcp:
      centralAuthMode: JWT
      centralProvidedCaCert: true
      configProtection: {}
      isolationBoundaries:
      - name: global
        revisions:
        - name: default
          istio:
            kubeSpec:
              overlays:
              - apiVersion: install.istio.io/v1alpha1
                kind: IstioOperator
                name: xcp-iop-default
                patches:
                - path: spec.values.global.proxy.logLevel
                  value: warn
                - path: spec.values.global.logging.level
                  value: default:warn
        - name: canary
          istio:
            kubeSpec:
              overlays:
              - apiVersion: install.istio.io/v1alpha1
                kind: IstioOperator
                name: xcp-iop-canary
                patches:
                - path: spec.values.global.proxy.logLevel
                  value: info
                - path: spec.values.global.logging.level
                  value: default:info