Using GitOps with Tetrate Service Express

GitOps support in TSE allows:

• Administrator teams to create TSE configuration resources directly in the Management Plane cluster, using the Management Plane UI and the tctl CLI
• Application teams to create TSE configuration resources directly in the application clusters using the Kubernetes API, exactly as they push changes to applications

In order to do this, all TSE configuration objects exist as Kubernetes Custom Resource Definitions (CRDs) so that they can be easily applied to the cluster. Once the resources are applied to the cluster, they are automatically reconciled and forwarded to the Management Plane.

Pushing Configuration to TSE

TSE Kubernetes Custom Resources

Kubernetes Custom Resources for TSE configuration are used like any other Kubernetes resource. They differ from TSE configuration as follows:

• The apiVersion and kind properties are the same for all resources except for the following:
  • Use the API group tsb.tetrate.io/v2, rather than api.tsb.tetrate.io/v2
• The metadata section does not have the TSB/TSE properties such as organization, tenant, etc. Instead, the hierarchy information is provided with the following annotations, where appropriate:
  • tsb.tetrate.io/organization
  • tsb.tetrate.io/tenant
  • tsb.tetrate.io/workspace
  • tsb.tetrate.io/trafficGroup
  • tsb.tetrate.io/securityGroup
  • tsb.tetrate.io/gatewayGroup
  • tsb.tetrate.io/istioInternalGroup
  • tsb.tetrate.io/application
• The contents of the spec are unchanged between the TSE native configuration and the Kubernetes-based config.

For example, a Workspace definition is defined as follows:

TSE Native Configuration	Kubernetes-based Configuration
apiversion: api.tsb.tetrate.io/v2 kind: Workspace metadata: name: bookinfo organization: tse tenant: tse spec: displayName: Bookinfo description: Test Bookinfo application namespaceSelector: names: - "*/bookinfo"	apiVersion: tsb.tetrate.io/v2 kind: Workspace metadata: name: bookinfo annotations: tsb.tetrate.io/organization: tse tsb.tetrate.io/tenant: tse spec: displayName: Bookinfo description: Test Bookinfo application namespaceSelector: names: - "*/bookinfo"

Using Istio direct mode resources

When using Istio direct mode resources with GitOps, an additional label needs to be added to the direct mode resource:

labels:
    istio.io/rev: "tsb"

For example, a Gateway object in a Gateway Group would look like:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: bookinfo-gateway
  namespace: bookinfo
  labels:
    istio.io/rev: tsb
  annotations:
    tsb.tetrate.io/organization: tetrate
    tsb.tetrate.io/tenant: tetrate
    tsb.tetrate.io/workspace: bookinfo
    tsb.tetrate.io/gatewayGroup: bookinfo
spec:
  selector:
    app: tse-gateway-bookinfo
  servers:
    - hosts:
        - "bookinfo.tetrate.io"
      port:
        number: 80
        name: http
        protocol: HTTP

This prevents the local Istio instance in the cluster from immediately processing that resource. Instead, the TSE relay pushes the resource to the Management Plane. There is a validation webhook that checks that all resources that need this label have it, and rejects them otherwise.

Applying TSE Custom Resources

TSE Custom Resources can be applied normally using kubectl. For example, to apply the workspace in the example above, you can run:

kubectl apply -f workspace.yaml

kubectl get workspaces -A
# NAMESPACE   NAME       PRIVILEGED   TENANT    AGE
# bookinfo    bookinfo                tse       4m20s

If you want to verify that the object has properly been created in the Management Plane, you can also use tctl to view the object there:

tctl get ws bookinfo
# NAME        DISPLAY NAME    DESCRIPTION
# bookinfo    Bookinfo        Test Bookinfo application

Triggering a GitOps operation

Your GitOps CD system needs to push TSE configuration to a TSE-managed cluster, or needs to run in the cluster and pull the configuration as needed:

The following documents provide examples of how to configure ArgoCD and Flux.

Monitoring GitOps health

The GitOps integration provides metrics and detailed logs that can be used to monitor the health of the different components involved in the GitOps process:

• The GitOps metrics provide insights about the latency experienced when sending configurations to the Management Plane, error rates, etc.
• The tsb-operator-control-plane provides the gitops logger that can be enabled at debug level to get detailed log messages from the different components that are part of the GitOps configuration propagation.

Fine-tuning GitOps Operation

The GitOps component can be configured through the ControlPlane custom resouce for each cluster.

kubectl edit -n istio-system controlplane/controlplane

Settings are defined in the gitops component:

spec:
  components:
    ...
    gitops:
      enabled: true
      reconcileInterval: 600s

You may wish to disable gitops (enabled: false) if you want to prevent K8s users from pushing TSE configuration using the K8s API.

Every time resources are applied by the CD system to the application cluster, the TSE GitOps component will push them to the Management Plane. Additionally, there is a periodic reconciliation process that ensures the application cluster remains the source of truth, and periodically pushes the information in it. The reconcileInterval attribute can be used to customize the interval at which the background reconciliation process runs. Further details and additional configuration options can be found in the TSE GitOps component reference.