Skip to main content
Version: 1.2.x

1.0 Upgrade notes

TSB 1.0 introduces a new cross cluster control plane agent called XCP, deprecating tsbd. Because of this, there are some considerations that need to be taken into account on upgrade.

In order to upgrade, follow the instructions in sequence:

  1. Tier1 routing based on network
  2. Management Plane Upgrade
  3. Control Plane Upgrade
Multi Phase Upgrade

If you are planning for a multi-phase upgrade, read the note about OAP

Tier1 routing based on network

Up to TSB 0.9.x, you could define routing rules in Tier1Gateway objects that would forward traffic to all the clusters exposing a given host that have a given network tag. For instance:

  externalServers:
- name: bookinfo
hostname: bookinfo.1.2.3.4.nip.io
port: 443
tls:
mode: SIMPLE
secretName: tier1-cert
clusters:
- network: tier2

This is no longer supported with the introduction of XCP. You will need to switch to label based routing. The first thing you need to do is make sure your clusters have labels that you can use for routing.

apiVersion: api.tsb.tetrate.io/v2
kind: Cluster
metadata:
name: my-tier2-cluster
tenant: tetrate
labels:
network: tier2
spec:
network: tier2
tier1Cluster: false
tokenTtl: 31536000s

In the Cluster above, a label called network has been added with the same value as the network tag of the cluster. Once your Cluster objects have been updated with the right labels, you can change the Tier1Gateway configuration to route based on this label instead of the network attribute. The example shown above becomes:

  externalServers:
- name: bookinfo
hostname: bookinfo.1.2.3.4.nip.io
port: 443
tls:
mode: SIMPLE
secretName: tier1-cert
clusters:
- label:
network: tier2
service disruption

Depending on the cloud provider or infrastructure you use, this action can cause a brief service disruption (~5s).

Management Plane Upgrade

Another requirement of XCP is the use of TLS certificates for communication between the central component of XCP in the management plane, and their edge counterparts in the control planes.

One easy way to achieve this is to deploy cert-manager in your cluster and leverage tctl to create the needed cert-manager object that will generate such certificates.

tctl install manifest management-plane-secrets \
--elastic-password ${ELASTIC_PASSWORD} \
--elastic-username ${ELASTIC_USERNAME} \
--elastic-ca-certificate "${ELASTIC_CA}" \
--postgres-password ${POSTGRES_PASSWORD} \
--postgres-username ${POSTGRES_USERNAME} \
--tsb-admin-password admin \
--tsb-server-certificate "$(cat ~/src/tetrate/test/certs/dummy.cert.pem)" \
--tsb-server-key "$(cat ~/src/tetrate/test/certs/dummy.key.pem)" -y \
--xcp-certs | kubectl apply -f -

The --xcp-certs flag creates some Issuer and Certificate objects that populate the needed secrets in the management plane namespace.

If you are not using cert-manager, you can create the certificates using your existing PKI and store the TLS certificate in the Kubernetes secret called xcp-central-cert in the management namespace. Check the set up instructions for further information.

Once the central XCP is up and running, you will need to create XCP’s GlobalSetting object to specify network reachability, similar to what you did in TSB UI or API up to 0.9.x.

apiVersion: xcp.tetrate.io/v2
kind: GlobalSetting
metadata:
namespace: tsb
name: xcp-global-settings
spec:
networkSettings:
networkReachability:
tier1: tier2

The example above configures XCP so that clusters with network: tier1 will set up multi cluster routing with clusters having network: tier2. You will need to set this up mimicking your current setup in TSB API.

Control Plane Upgrade

After upgrading the management plane to TSB 1.0 (not before!), you can start upgrading existing control planes. Before starting the upgrade, you need to create the certificate that the edge XCP will use to authenticate to the central XCP.

For that, if you use the cert-manager approach, you can also leverage tctl to get the needed Kubernetes secret YAML. Having your kubectl context pointing to the management plane cluster:

tctl install cluster-certs --cluster <cluster-name> \
> /tmp/edge-cert.yaml

Where <cluster-name> is the TSB cluster name of the control plane you are about to upgrade. You can apply the resulting YAML in the control plane cluster. If you are not using cert-manager, check the set up instructions for the control plane for the requirements of the certificate that needs to be present in the control plane namespace.

At this point it is safe to upgrade the TSB control plane:

tctl install manifest cluster-operators \
--registry <registry-location>

The command above will output the needed Kubernetes manifests which you can dump to a file for review or modification (adding tolerations, etc.). Once the YAML is OK you can proceed and apply it to the cluster. The new pod will upgrade the control plane components after start up. You will see tsbd deployments disappearing, and the new XCP components showing up. Wait for all components to be ready.

Once XCP edge is up, you will need to remove the old tsbd configurations, if any. You can use the command below to that end:

while read p; do $p; done < <(kubectl get gw,vs,dr,se --all-namespaces -o=jsonpath='{range .items[?(@.metadata.annotations.tsb\.tetrate\.io/contentHash!="")]}kubectl delete -n {.metadata.namespace} {.kind}/{.metadata.name}{"\n"}{end}')
service disruption

Depending on the cloud provider or infrastructure you use, this could cause a brief service disruption (~1min).

NOTE about OAP

If you are not planning for a synchronized upgrade of all the control planes at the same time, you will need to run a CronJob in the TSB management plane namespace (by default tsb) that will create some additional indices that are deprecated in 1.0, but still needed for the 0.9.x OAP in the control planes to function. Once all the control planes have been upgraded to 1.0 you can remove the oap-init CronJob from the management plane namespace.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
labels:
app: oap-init
platform.tsb.tetrate.io/application: oap-init
platform.tsb.tetrate.io/component: oap
platform.tsb.tetrate.io/plane: management
name: oap-init
namespace: tsb
spec:
concurrencyPolicy: Forbid
failedJobsHistoryLimit: 1
jobTemplate:
spec:
template:
metadata:
creationTimestamp: null
labels:
platform.tsb.tetrate.io/application: oap-init
platform.tsb.tetrate.io/component: oap
platform.tsb.tetrate.io/plane: management
spec:
containers:
- env:
- name: JAVA_OPTS
value: -Xmx4096M -Dmode=init
# This denotes the target Elasticsearch nodes. Should match
# `spec.telemetryStore.elastic.host`:`spec.telemetryStore.elastic.port`
- name: SW_STORAGE_ES_CLUSTER_NODES
value: <host>:<port>
# This denotes the protocol to use when connecting to Elasticsearch nodes. Should match
# `spec.telemetryStore.elastic.protocol`
- name: SW_STORAGE_ES_HTTP_PROTOCOL
value: <http|https>
# Credentials will be the same as used by OAP
- name: SW_ES_USER
valueFrom:
secretKeyRef:
key: username
name: elastic-credentials
optional: true
- name: SW_ES_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: elastic-credentials
optional: true
- name: SW_SW_STORAGE_ES_SSL_JKS_PATH
value: /etc/skywalking/truststore.jks
- name: SW_SW_STORAGE_ES_SSL_JKS_PASS
value: tetrate
# Replace with your Docker registry URL. Image and tag should be kept as is.
image: <docker-registry>/spm-central:11b382507838a7051bad260c9188c7fed2ab83f9-es7
imagePullPolicy: IfNotPresent
name: oap-init
volumeMounts:
- mountPath: /skywalking/ext-config/log4j2.xml
name: config
subPath: log4j2.xml
- mountPath: /etc/skywalking
name: keystore-volume
initContainers:
- args:
- -c
- csplit -z -f crt- $CA_BUNDLE '/-----BEGIN CERTIFICATE-----/' '{*}' && for
file in crt-*; do keytool -import -noprompt -keystore $TRUSTSTORE_JKS -file
$file -storepass $PASSWORD -alias service-$file; done
command:
- /bin/bash
env:
- name: CA_BUNDLE
value: /var/run/secrets/elasticsearch.io/ca.crt
- name: TRUSTSTORE_JKS
value: /var/run/secrets/java.io/keystores/truststore.jks
- name: PASSWORD
value: tetrate
# Replace with your Docker registry URL. Image and tag should be kept as is.
image: <docker-registry>/sso71-openshift:1.1-16
imagePullPolicy: IfNotPresent
name: pem-to-truststore
volumeMounts:
- mountPath: /var/run/secrets/elasticsearch.io
name: es-certs
- mountPath: /var/run/secrets/java.io/keystores
name: keystore-volume
restartPolicy: OnFailure
serviceAccount: tsb-oap
serviceAccountName: tsb-oap
volumes:
- configMap:
defaultMode: 420
items:
- key: log4j2.xml
path: log4j2.xml
name: oap-config
name: config
- name: es-certs
secret:
defaultMode: 420
optional: true
secretName: es-certs
- emptyDir: {}
name: keystore-volume
schedule: 0 0 * * *
successfulJobsHistoryLimit: 3
suspend: false
note

This is a generic CronJob with the necessary images and configuration for OAP to create the necessary indices. If you need to add tolerations, affinity, or other Kubernetes requirements specific to your environment or distribution, you will need to add them to the CronJob depicted above.