Skip to main content
Version: 1.3.x

Using overlays in TSB

This document describes how to use overlays to perform advanced configuration of resources that are deployed by the Tetrate Service Bridge (TSB) operators, using examples.

Background​

TSB makes extensive use of the Operator pattern to deploy and configure the necessary parts in Kubernetes.

Normally customization and fine tuning of the parameters are done through the operator, which is responsible for creating the necessary resources and controlling their lifecycles.

For example, when you create an IngressGateway CR, a TSB operator picks up this information and deploys and/or updates the relevant resources, such as Kubernetes Service objects, by creating a manifest and applying them. The manifest will use certain parameters that you have provided, along with other default values that are computed by TSB.

However, TSB does not necessarily expose all of the knobs to fine tune the Service objects. If TSB were to provide all hooks to configure the Service object, TSB would have to effectively replicate the entire Kubernetes API, which is realistically not feasible nor desirable.

This is where we use overlays, which allows you to override and apply custom configurations to resources that are being deployed. For more details on how overlays work, please read the documentation for overlays in the reference.

Notes About the Examples​

In these examples that follow, the necessary configurations are applied using kubectl edit by directly editing the deployed manifest. If you own the original manifests you may opt to use kubectl apply as well, but you will have to provide the entire resource definition, not just the parts that you want to edit.

The sample manifests only show the minimum information required to be specified, along with information to specify the context (location) of where to make these changes.

note

Depending on your specific Kubernetes environment, you may need to modify the content of the samples for them to function properly.

Once you have studied these examples, you will most likely be working with far more complex overlays. One caveat that you should be aware of as you write complex overlays is that you can only have one overlay per object. For example, the following specification is syntactically valid, but only the last patch against quux.corge.grault will be applied: .

kubeSpec:
  overlays:
  - apiVersion: v1
    kind: ....
    name: my-object
    patches:
    - path: foo.bar.baz
      value: 1
  - apiVersion: v1
    kind: ....
    name: my-object
    patches:
    - path: quux.corge.grault
      value: hello

This is because the manifest contains multiple entries under overlays that point to the same object (my-object), and in such cases only the last entry is actually applied. To apply patches to both foo.bar.baz and quux.corge.grault, you must consolidate all the patch specifications under a single object, as follows:

kubeSpec:
  overlays:
  - apiVersion: v1
    kind: ....
    name: my-object
    patches:
    - path: foo.bar.baz
      value: 1
    - path: quux.corge.grault
      value: hello

Example Usage for Overlays​

Configure CNI with elevated privileges​

Certain environments such as SELinux or OpenShift require special privileges to write files in the host system. To enable this, the install-cni.securityContext.privileged property must be set to true by editing the ControlPlane CR.

Edit the ControlPlane CR for the TSB control plane using kubectl edit, and use the following snippet as a sample on how to edit the manifest.

kubectl edit controlplane -n istio-system
spec:
  components:
    istio:
      kubespec:
        overlays:
        - apiVersion: install.istio.io/v1alpha1
          kind: IstioOperator
          name: tsb-istiocontrolplane
          patches:
          - path: spec.components.cni.k8s
            value:
              overlays:
              - apiVersion: extensions/v1beta1
                kind: DaemonSet
                name: istio-cni-node
                patches:
                - path: spec.template.spec.containers.[name:install-cni].securityContext
                  value:
                    privileged: true

Preserve endpoint IP address​

Kubernetes provides a way to preserve the IP address of the client connecting to an application, which can be used to route traffic to node-local or cluster-wide endpoints.

For this example we assume that you have deployed the following ingress gateway with service type of LoadBalancer:

apiVersion: install.tetrate.io/v1alpha1
kind: IngressGateway
metadata:
  name: tsb-gateway-bookinfo
  namespace: bookinfo
spec:
  kubeSpec:
    service:
      type: LoadBalancer

Edit the IngressGateway CR for the application using kubectl edit, and use the following snippet as a sample on how to edit the manifest.

kubectl edit tsb-gateway-bookinfo -n bookinfo
spec:
  connectionDrainDuration: 10s
  kubeSpec:
    overlays:
    - apiVersion: v1
      kind: Service
      name: tsb-gateway-bookinfo
      patches:
      - path: spec.externalTrafficPolicy
        value: Local

Add a host alias to istiod​

In some scenarios, istiod may need to communicate with services that have no DNS records. A typical example would be when there is a need to fetch a custom Istio CA from Vault or other secret managers. The hostAlias patch will directly map hostnames to ip addresses in a way similar to statically adding an entry to a VM host file.

Edit the ControlPlane CR for the TSB control plane using kubectl edit, and use the following snippet as a sample on how to edit the manifest. Replace <hostname-FQDN> and <ip address> with the appropriate values.

kubectl edit controlplane -n istio-system
 spec:
  components:
    istio:
      kubeSpec:
        overlays:
        - apiVersion: install.istio.io/v1alpha1
          kind: IstioOperator
          name: tsb-istiocontrolplane
          patches:
          - path: spec.components.pilot.k8s.overlays
            value:
            - apiVersion: apps/v1
              kind: Deployment
              name: istiod
              patches:
              - path: spec.template.spec.hostAliases
                value:
                - hostnames:
                  - <hostname-FQDN>
                  ip: <ip address>

Configure sidecar resource limits​

The Sidecar API resource does not allow you to specify resource usage limit or definitions for the sidecar, but this is possible by adding an overlay in the control plane CR. In this example, we will overwrite resource limits under the resources field.

Edit the ControlPlane CR for the TSB control plane using kubectl edit, and use the following snippet as a sample on how to edit the manifest. Update the actual resource limit values as necessary.

kubectl edit controlplane -n istio-system
spec:
  components:
    istio:
      khttps://kubernetes.io/docs/tutorials/services/source-ip/ubeSpec:
        overlays:
        - apiVersion: install.istio.io/v1alpha1
          kind: IstioOperator
          name: tsb-istiocontrolplane
          patches:
          - path: spec.values.global.proxy
            value:
              resources:
                limits:
                  cpu: 2000m
                  memory: 1024Mi
                requests:
                  cpu: 100m
                  memory: 128Mi

Forwarding client information​

Some applications require knowing the certificate information of the connecting client. TSB uses the x-forwarded-client-cert header to pass this information along to the backend servers. In order to enable this feature you need to configure the Envoy proxies for ControlPlane and IngressGateway(s) as such.

For the ControlPlane, edit the ControlPlane CR for the TSB control plane using kubectl edit, and use the following snippet as a sample on how to edit the manifest

kubectl edit controlplane -n istio-system
spec:
  components:
    istio:
      kubeSpec:
        overlays:
        - apiVersion: install.istio.io/v1alpha1
          kind: IstioOperator
          name: tsb-istiocontrolplane
          patches:
          - path: spec.meshConfig.defaultConfig.gatewayTopology
            value:
              forwardClientCertDetails: APPEND_FORWARD

For the IngressGateway, edit the IngressGateway CR for the application using kubectl edit, and use the following snippet as a sample on how to edit the manifest. Replace the <ingress-name> and <namespace> values as appropriate.

kubectl edit <ingress-name> -n <namespace>
spec:
  kubeSpec:
    overlays:
    - apiVersion: apps/v1
      kind: Deployment
      name: <ingress-name>
      patches:
      - path: spec.template.metadata.annotations.proxy\.istio\.io/config
          gatewayTopology:
            forwardClientCertDetails: APPEND_FORWARD

Controlling user session inactivity time for TSB UI​

By default the user session for TSB UI expires after 15 minutes of inactivity. You may override this value by setting the SESSION_AGE_IN_MINUTES environment variable through the overlay mechanism.

Suppose you would like to allow the user to be logged into the Web UI for 60 minutes. Edit the ControlPlane CR for the TSB control plane using kubectl edit, and use the following snippet as a sample on how to edit the manifest.

kubectl edit managementplane -n tsb
spec:
  webUI:
    kubeSpec:
      overlays:
      - apiVersion: apps/v1
        kind: Deployment
        name: web
        patches:
        - path: spec.template.spec.containers.[name:web].env[-1]
          value:
            name: SESSION_AGE_IN_MINUTES
            value: "60"

It is highly recommended that SESSION_AGE_MINUTES be set to the minimum for security best practices with regards to accessing the UI.

References​

TSB reference Doc