Using GitOps with Tetrate Service Express
GitOps support in TSE allows:
- Administrator teams to create TSE configuration resources directly in the Management Plane cluster, using the Management Plane UI and the tctl CLI
- Application teams to create TSE configuration resources directly in the application clusters using the Kubernetes API, exactly as they push changes to applications
In order to do this, all TSE configuration objects exist as Kubernetes Custom Resource Definitions (CRDs) so that they can be easily applied to the cluster. Once the resources are applied to the cluster, they are automatically reconciled and forwarded to the Management Plane.
Pushing Configuration to TSE
TSE Kubernetes Custom Resources
Kubernetes Custom Resources for TSE configuration are used like any other Kubernetes resource. They differ from TSE configuration as follows:
kindproperties are the same for all resources except for the following:
- Use the API group
tsb.tetrate.io/v2, rather than
- Use the API group
- The metadata section does not have the TSB/TSE properties such as
tenant, etc. Instead, the hierarchy information is provided with the following annotations, where appropriate:
- The contents of the
specare unchanged between the TSE native configuration and the Kubernetes-based config.
For example, a Workspace definition is defined as follows:
|TSE Native Configuration||Kubernetes-based Configuration|
Using Istio direct mode resources
When using Istio direct mode resources with GitOps, an additional label needs to be added to the direct mode resource:
For example, a Gateway object in a Gateway Group would look like:
This prevents the local Istio instance in the cluster from immediately processing that resource. Instead, the TSE relay pushes the resource to the Management Plane. There is a validation webhook that checks that all resources that need this label have it, and rejects them otherwise.
Applying TSE Custom Resources
TSE Custom Resources can be applied normally using
kubectl. For example, to apply the workspace in the example above, you can run:
kubectl apply -f workspace.yaml
kubectl get workspaces -A
# NAMESPACE NAME PRIVILEGED TENANT AGE
# bookinfo bookinfo tse 4m20s
If you want to verify that the object has properly been created in the Management Plane, you can also use
tctl to view the object there:
tctl get ws bookinfo
# NAME DISPLAY NAME DESCRIPTION
# bookinfo Bookinfo Test Bookinfo application
Triggering a GitOps operation
Your GitOps CD system needs to push TSE configuration to a TSE-managed cluster, or needs to run in the cluster and pull the configuration as needed:
Monitoring GitOps health
The GitOps integration provides metrics and detailed logs that can be used to monitor the health of the different components involved in the GitOps process:
- The GitOps metrics provide insights about the latency experienced when sending configurations to the Management Plane, error rates, etc.
gitopslogger that can be enabled at debug level to get detailed log messages from the different components that are part of the GitOps configuration propagation.
Fine-tuning GitOps Operation
The GitOps component can be configured through the ControlPlane custom resouce for each cluster.
kubectl edit -n istio-system controlplane/controlplane
Settings are defined in the gitops component:
You may wish to disable gitops (
enabled: false) if you want to prevent K8s users from pushing TSE configuration using the K8s API.
Every time resources are applied by the CD system to the application cluster, the TSE GitOps component will push them to the Management Plane. Additionally, there is a periodic reconciliation process that ensures the application cluster remains the source of truth, and periodically pushes the information in it. The reconcileInterval attribute can be used to customize the interval at which the background reconciliation process runs. Further details and additional configuration options can be found in the TSE GitOps component reference.