Install a new Tetrate Management Plane (Embedded Postgres Database)

How to install a new Management Plane instance, when using the embedded Postgres Database.

Embedded Postgres Database

These instructions refer to installations that use a embedded Postgres database.

If you are using your own, external postgres implementation, refer to the reinstall when using external postgres instructions instead.

You should review the Helm Installation Procedure and any specific notes before proceeding. Please work with Tetrate Technical Support to go through the following procedure.

Backup the Existing Database

If possible, follow the admin instructions to back up the active PostgreSQL database.

Copy the backup to the desired storage location, for example, your local machine.

Prevent Updates to the Database

If the Management Plane that uses the database is active and receiving changes, you can make it inactive to prevent changes to the database. You should do this if you plan to deactivate this management plane, so that configuration changes are rejected rather than accepted and lost.

On the current Management cluster

kubectl scale deploy -n tsb tsb iam --replicas 0

Install the new, standby Management Plane instance

1. Option 1: Deploy a Replica Management Plane instance using Helm

   Use the helm-based deployment when the current Management Plane was originally deployed using helm (the recommended method).

   In the Kubernetes cluster of the original Management Plane:

   Take a snapshot of the operational Kubernetes secrets. These secrets were auto-generated on first use:

   kubectl get secrets -n tsb -o yaml iam-signing-key > source_mp_operational_secrets.yaml

   In the Kubernetes cluster intended for the new Management Plane:

   Create a k8s namespace for the replica MP:

   kubectl create ns tsb

   Apply the operational secrets from the the original Management Plane instance:

   kubectl apply -n tsb -f source_mp_operational_secrets.yaml

   Install the replica Management Plane using the same Helm values that were used for the original Management Plane:

   helm install mp tetrate-tsb-helm/managementplane \
     --version <tsb-version> \
     --namespace tsb  \
     --values source_mp_values.yaml \
     --timeout 10m \
     --set image.registry=<registry-location>

   Ensure that the front Envoy certificate and key, and the root CA and key are provided, for example through the Helm values.

2. Option 2: Generic deployment model

   Use the 'generic' deployment method when the Management Plane was originally deployed using the tctl CLI.

   In the Kubernetes cluster of the original Management Plane:

   Take a snapshot of the configurational and operational Kubernetes secrets. These secrets were auto-generated on first use:

   kubectl get secrets -n tsb -o yaml admin-credentials azure-credentials custom-host-ca elastic-credentials es-certs iam-oidc-client-secret ldap-credentials postgres-credentials tsb-certs xcp-central-cert iam-signing-key > source_mp_all_secrets.yaml

   In the Kubernetes cluster intended for the new Management Plane:

   Create a k8s namespace for the replica MP:

   kubectl create ns tsb

   Apply the secrets from the the original Management Plane instance:

   kubectl apply -n tsb -f source_mp_all_secrets.yaml

   Install the replica Management Plane using helm:

   helm install mp tetrate-tsb-helm/managementplane \
     --version <tsb-version> \
     --namespace tsb  \
     --values dr_mp_values.yaml \
     --timeout 10m \
     --set image.registry=<registry-location>

   ... where dr_mp_values.yaml:

   • Should include the spec field
   • Should NOT include the secrets field (as secrets were installed in the previous step)

3. Disable the new Management Plane (optional)

   You may wish to disable the new Management Plane so that it does not accidentally receive configuration updates that are intended for the current instance:

   kubectl scale deploy -n tsb tsb iam --replicas 0

   Limited Health Checks

   Note that if you do this, you will not be able to health-check the services within the management plane, as access is gated using the iam service. You can only health-check the envoy service that fronts the Management Plane services.

   You will need to reactivate these Management Plane services before starting a failover operation.

Import the configuration to the new Management Plane instance

1. Create PVC and Pod for Restoration

   At any point, you can restore the database backup to the new, standby Management Plane.

   On the new, standby Management Plane cluster, create a PersistentVolumeClaim (PVC) and pod to perform the restoration:

   cat <<EOF > pvc.yaml
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: restore
     namespace: tsb
   spec:
     accessModes:
     - ReadWriteOnce
     resources:
       requests:
         storage: 10Gi
     storageClassName: standard-rwo
     volumeMode: Filesystem
   EOF
   
   kubectl apply -f pvc.yaml
   
   cat <<EOF > restore-pod.yaml
   apiVersion: v1
   kind: Pod
   metadata:
     name: restore
     namespace: tsb
   spec:
     containers:
     - name: alpine
       image: alpine:latest
       command:
         - sleep
         - infinity
       volumeMounts:
       - name: restore-data
         mountPath: /var/lib/restore
         readOnly: false
     volumes:
     - name: restore-data
       persistentVolumeClaim:
         claimName: restore
   EOF
   
   kubectl apply -f restore-pod.yaml

2. Copy Backup File to PVC and Set Permissions

   Copy the backup file to the PVC:

   kubectl cp tsb-postgres-backup-26_09_2024_02_00_01.gz tsb/restore:/var/lib/restore/tsb-postgres-backup-26_09_2024_02_00_01.gz
   kubectl exec -n tsb -it restore -- chown root:root /var/lib/restore/tsb-postgres-backup-26_09_2024_02_00_01.gz
   kubectl exec -n tsb -it restore -- ls -l /var/lib/restore

3. Run Job to Restore PostgreSQL

   Create and apply the restoration job:

   cat <<EOF > restore-backup.yaml
   apiVersion: batch/v1
   kind: Job
   metadata:
     name: restore-backup
     namespace: tsb
   spec:
     backoffLimit: 1
     completions: 1
     parallelism: 1
     template:
       spec:
         restartPolicy: Never
         containers:
         - name: restore
           image: gcr.io/alclass1-ctkl-1/postgres:14.8-alpine3.18
           command:
             - sh
           args:
             - -c
             - |
               set -ex
               echo "Checking for backup file in /var/lib/restore"
               if [ -f /var/lib/restore/tsb-postgres-backup-26_09_2024_02_00_01.gz ]; then
                 echo "Backup file found, decompressing"
                 gzip -d /var/lib/restore/tsb-postgres-backup-26_09_2024_02_00_01.gz
               else
                 echo "Backup file not found!"
                 exit 1
               fi
               echo "Restoring PostgreSQL from the backup"
               psql "host=tsb-postgres dbname=postgres user=tsb password=Tetrate123 sslmode=verify-ca sslcert=/var/lib/postgresql/data/tls/tls.crt sslkey=/var/lib/postgresql/data/tls/tls.key sslrootcert=/var/lib/postgresql/data/tls/ca.crt" -f "/var/lib/restore/tsb-postgres-backup-26_09_2024_02_00_01"
           volumeMounts:
           - mountPath: /var/lib/postgresql/data/tls
             name: tsb-postgres-certs
             readOnly: true
           - mountPath: /var/lib/restore
             name: restore-volume
             readOnly: false
         volumes:
         - name: tsb-postgres-certs
           secret:
             defaultMode: 0600
             secretName: tsb-postgres-certs
         - name: restore-volume
           persistentVolumeClaim:
             claimName: restore  # Reference the same PVC as in the restore pod
             readOnly: false
   EOF
   
   kubectl apply -f restore-backup.yaml

4. Validate the Job

   Check the status of the job and logs:

   kubectl get jobs -n tsb restore-backup

   NAME             STATUS     COMPLETIONS   DURATION   AGE
   restore-backup   Complete   1/1           6s         3h58m

   kubectl get pod -n tsb restore-backup-pgvrq

   NAME                   READY   STATUS      RESTARTS   AGE
   restore-backup-pgvrq   0/1     Completed   0          3h59m

   kubectl logs -n tsb -l job-name=restore-backup

   CREATE INDEX
   CREATE INDEX
   CREATE INDEX
   CREATE INDEX
   ALTER TABLE
   ALTER TABLE
   ALTER TABLE
   ALTER TABLE
   ALTER TABLE
   GRANT

Optionally, Test the new Management Plane instance

Scale the TSB and IAM deployments back to normal:

kubectl scale deploy -n tsb tsb iam --replicas 1

This will allow you to monitor the health of the new, standby Management Plane. Obtain the front envoy public ip address, for example:

kubectl get svc -n tsb envoy

Log into the UI with Envoy IP Address:

• Verify that your Tetrate configuration has been preserved in the Postgres database; look for cluster configurations (clusters will not have synced at this point) and the organizational structure (organization, tenants, workspaces) that you expect to see
• Check the Elastic historical data if available

This confirms that the rebuild was successful.

Perform the Failover from old to new

When needed, follow the failover process to update DNS records and provoke the controlplane deployments to connect to the new Management Plane instance.