Upgrading TSB

Managing TSB Upgrades - a quick overview

• Upgrade your Management Plane minor release at least once per year, to stay within the (N-2) support policy
• Upgrade your Control Plane versions to stay within the N-2 compatibility range of your Management Plane version

Staying up-to-date with the latest releases in your selected minor version:

• ensures compatibility and avoids complications such as the OpenCensus/OTEL change and the need to upgrade to later calendar versions at each stage
• provides access to the latest features and security updates.

This document explains the potential complications that you should be aware of before upgrading, and the recommended practices.

TSB Management Planes, Control Plane and Data Plane compatibility

Control Planes: TSB follows an 'N-2' policy for compatibility and stability between Management Plane and Control Plane. This assures that the latest 1.N.x release of the TSB Management Plane (MP) is fully compatible with the latest Control Plane (CP) versions released up to two versions prior 1.(N-2).x.

Data Planes: TSB follows an 'M-1' policy for compatibility between Control Plane and Data Plane. When a Control Plane is upgraded, the Data Plane (Gateways and Sidecar proxies) are not always automatically updated. Gateway upgrades may be held back, and sidecars require explicit pod reloads. The TSB Control Plane version 1.M.x supports Data Plane components from one previous minor version 1.(M-1).x.

Good Upgrade Practice

It is good practice and recommended to:

• Maintain your chosen Management Plane release at the latest point release for your chosen minor release
• Ensure that Control Plane clusters are maintained at the latest point release, and do not fall more than 2 minor versions (1.(N-2)) behind the management plane
• Restart application workloads as necessary

What Upgrade Paths are supported?

TSB Management Plane and Control Plane instances can be upgraded in-place from releases up to 2 minor versions behind (N-2). For example, a 1.10.x installation can be upgraded to 1.12.x, but should not be directly upgraded to 1.13.x. If you wish to upgrade by more than 2 minor versions, you should first upgrade to an intermediate release.

There are two important caveats to bear in mind:

• Always upgrade the Management Plane first. The Control Plane version must not exceed the Management Plane version. For example, a control plane 1.13.1 is not supported with a Management Plane 1.13.0.
• Always upgrade to a later calendar release. Tetrate make point releases for each supported version, which include current CVE fixes and necessary dependency updates. For example, a recent 1.11.4 release may contain fixes not present in an earlier 1.12.0 release. To enforce good security hygiene, you cannot upgrade to an earlier release, even if it has a later minor version number.

Good Upgrade Practice

It is always safe to upgrade to the latest point release in your chosen minor release. Doing so will avoid complications from the 'later calendar release' requirement.

Planning your Upgrades

Always upgrade the Management Plane first. When planning to upgrade to a version, e.g. 1.13.1, upgrade the Management Plane first, and then upgrade your control plane clusters to this version.

Always upgrade to the latest point release in your chosen minor version. For example, if you are running TSB version 1.12.x, always upgrade to the latest point release in that version to ensure that you meet the 'later calendar release' requirement.

Upgrade Notes by Version

OpenCensus and OTEL support in TSB 1.14.0, 1.13.1, 1.12.7, 1.11.5

Prior to these releases, TSB relied on OpenCensus to transmit metrics from Control Plane clusters to Management Plane clusters. OpenCensus has been sunsetted, and from these releases onwards, Control Plane clusters will use OTEL to transmit metrics to the Management Plane. To ease migration, the TSB Management Plane will support both OpenCensus and OTEL to meet the N-2 support commitment:

Release	Management Plane	Control Plane
1.16.x	OTEL	OTEL
1.15.x	OTEL and OpenCensus	OTEL
1.14.x	OTEL and OpenCensus	OTEL
1.13.1+	OTEL and OpenCensus	OTEL
1.13.0	OpenCensus	OpenCensus
1.12.7+	OTEL and OpenCensus	OTEL
1.12.0-6	OpenCensus	OpenCensus
1.11.5+	OTEL and OpenCensus	OTEL
1.11.0-4	OpenCensus	OpenCensus

As a general rule, an older Control Plane version (using OpenCensus) will function correctly with a newer Management Plane instance, provided the (N-2) compatibility is met.

OpenCensus support in the Management Plane may be disabled in two situations:

• New Management Plane installations will only enable OTEL support. Matching control plane installations will use OTEL, so no compatibility issues should arise
• An administrator can explicitly disable the legacy OpenCensus support in the MP, using the components.collectors.enableOpencensusBackwardsCompatibility: false flag in the Management Plane custom resource. They may wish to do this once all CP instances have been verified to use OTEL

TSB 1.14.3 upgrade and rollback notes

Upgrading to 1.14.3: align IsolationBoundary revision versions

If your ControlPlane CR pins an Istio version in its IsolationBoundary revisions (spec.components.xcp.isolationBoundaries[].revisions[].istio.tsbVersion), update any revision still set to 1.14.0, 1.14.1, or 1.14.2 to 1.14.3 as part of the upgrade. This is due to an incompatible API change introduced in upstream Istio in 1.14.3. In-place upgrade within patch versions is safe.

spec:
  components:
    xcp:
      isolationBoundaries:
      - name: global
        revisions:
        - name: default
          istio:
            tsbVersion: 1.14.3   # update from 1.14.0 / 1.14.1 / 1.14.2

Revisions that do not set tsbVersion follow the current release automatically and need no change.

Rolling back the control plane from 1.14.3

A control plane rollback moves to an older set of APIs, and a few resources written by 1.14.3 carry newer values that older releases do not recognize yet. These are known issues with simple workarounds - apply the ones relevant to your rollback target.

Gateway API CRDs block xcp-operator-edge

1.14.3 advances the stored version of the experimental Gateway API CRDs (backendtlspolicies.gateway.networking.k8s.io, tlsroutes.gateway.networking.k8s.io). After the rollback, xcp-operator-edge can fail to reconcile these CRDs because the newer stored version is no longer listed in the older CRD spec (... must remain in spec.versions until a storage migration ...). Tell the edge operator to skip re-creating these CRDs by setting DISABLE_K8S_GATEWAY_API_CRD_CREATION=true on the xcp component, then restart it:

spec:
  components:
    xcp:
      kubeSpec:
        deployment:
          env:
          - name: DISABLE_K8S_GATEWAY_API_CRD_CREATION
            value: "true"

kubectl rollout restart deploy/xcp-operator-edge -n istio-system

GatewayDeployment status phase not recognized (target older than 1.12.14)

When rolling back from 1.14.3 to a release older than 1.12.14 (that is, 1.12.13 or earlier), the older control-plane operator may not recognize a newer GatewayDeployment status phase written by 1.14.3 (RECONCILIATION_DIRTY / RECONCILIATION_PAUSED) and stops reconciling with an error such as unknown value "RECONCILIATION_DIRTY" for enum ... GatewayPhase. Releases 1.12.14 and later already understand these values, so this step only applies to older targets.

Reset the affected GatewayDeployments to a phase the older operator understands, then restart the control-plane operator:

kubectl get gatewaydeployments.install.xcp.tetrate.io -A -o json \
  | jq -r '
      .items[]
      | select(.status.phase == "RECONCILIATION_DIRTY" or .status.phase == "RECONCILIATION_PAUSED")
      | [ .metadata.namespace, .metadata.name,
          (if (.status.workload.deployment.isReady and .status.workload.service.isReady)
           then "READY" else "PENDING" end) ]
      | @tsv' \
  | while IFS=$'\t' read -r ns name phase; do
      kubectl patch gatewaydeployment -n "$ns" "$name" \
        --subresource=status --type=merge -p "{\"status\":{\"phase\":\"$phase\"}}"
    done

kubectl rollout restart deploy/tsb-operator-control-plane -n istio-system