Skip to main content
Version: 1.3.x

Kubernetes

When installing on Kubernetes, these configuration settings can be used to override the default Kubernetes configuration. Kubernetes configuration can be set on each component in the install API using the kubeSpec field.

The API allows for customization of every field in the rendered Kubernetes manifests. The more common configuration fields, such as resources and service type, are supported directly; and can be configured like so:

apiVersion: install.tetrate.io/v1alpha1
kind: ManagementPlane
metadata:
name: managementplane
spec:
hub: docker.io/tetrate
components:
apiServer:
kubeSpec:
service:
type: LoadBalancer
deployment:
resources:
limits:
memory: 500Mi
requests:
memory: 750Mi

All components have a deployment and service object. Some, such as apiServer, also have a job object associated with them. This can be configured in a similar manner:

apiVersion: install.tetrate.io/v1alpha1
kind: ManagementPlane
metadata:
name: managementplane
spec:
hub: docker.io/tetrate
components:
apiServer:
kubeSpec:
job:
podAnnotations:
annotation-key: annotation-value

Not all fields in a Kubernetes manifest can be configured directly. This is to avoid re-implementing the entire Kubernetes API within the install API. Instead, the kubeSpec object provides an overlays mechanism. This field is applied after the operator renders the initial manifests and enables support for customization of any field in a rendered manifest.

Overlays can be applied by selecting the Kubernetes object you wish to overlay and then describe a list of patches you wish to apply. For example, to add a hostPort on port 8443 to the frontEnvoy component, do the following:

apiVersion: install.tetrate.io/v1alpha1
kind: ManagementPlane
metadata:
name: managementplane
spec:
hub: docker.io/tetrate
components:
frontEnvoy:
kubeSpec:
overlays:
- apiVersion: apps/v1
kind: Deployment
name: envoy
patches:
- path: spec.template.spec.containers.[name:envoy].ports.[containerPort:8443].hostPort
value: 8443

The path refers to the location of the field in the Kubernetes object you with to patch. The format is a.[key1:value1].b.[:value2]. Where [key1:value1] is a selector for a key-value pair to identify a list element and [:value] is a value selector to identify a list element in a leaf list. All path intermediate nodes must exist.

Overlays are inspired by and bear a loose resemblence to [kustomize](https://kustomize.io/). We use the library from the Istio Operator. For more examples of how to construct paths take a look at the tests in the upstream.

Affinity

The scheduling constraints for the pod. https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity

FieldDescriptionValidation Rule

nodeAffinity

tetrateio.api.install.kubernetes.NodeAffinity
Group of node affinity scheduling rules. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#nodeaffinity-v1-core

podAffinity

tetrateio.api.install.kubernetes.PodAffinity
Group of inter-pod affinity scheduling rules. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#podaffinity-v1-core

podAntiAffinity

tetrateio.api.install.kubernetes.PodAntiAffinity
Group of inter-pod anti-affinity scheduling rules. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#podantiaffinity-v1-core

CNI

Configure Istio's CNI plugin For further details see: https://istio.io/docs/setup/additional-setup/cni/

FieldDescriptionValidation Rule

binaryDirectory

string
Directory on the host to install the CNI binary. Must be the same as the environment’s --cni-bin-dir setting (kubelet parameter).

configurationDirectory

string
Directory on the host to install the CNI config. Must be the same as the environment’s --cni-conf-dir setting (kubelet parameter).

chained

bool
Whether to deploy the configuration file as a plugin chain or as a standalone file in the configuration directory. Some Kubernetes flavors (e.g. OpenShift) do not support the chain approach.

configurationFileName

string
Leave unset to auto-find the first file in the cni-conf-dir (as kubelet does). Primarily used for testing install-cni plugin configuration. If set, install-cni will inject the plugin configuration into this file in the cni-conf-dir.

clusterRole

string
The ClusterRole Istio CNI will bind to in the ControlPlane namespace. This is useful if you use Pod Security Policies and want to allow istio-cni to run as privileged Pods.

ClientIPConfig

FieldDescriptionValidation Rule

timeoutSeconds

int32

ConfigMapKeySelector

FieldDescriptionValidation Rule

localObjectReference

tetrateio.api.install.kubernetes.LocalObjectReference

key

string

optional

bool

ContainerPort

ContainerPort represents a network port in a single container.

FieldDescriptionValidation Rule

name

string
If specified, this must be an IANA_SVC_NAME and unique within the pod. Each named port in a pod must have a unique name. Name for the port that can be referred to by services.

hostPort

int32
Number of port to expose on the host. If specified, this must be a valid port number, 0 < x < 65536.

containerPort

int32
Number of port to expose on the pod's IP address. This must be a valid port number, 0 < x < 65536.

protocol

string
Protocol for port. Must be UDP, TCP, or SCTP. Defaults to "TCP".

hostIP

string
What host IP to bind the external port to.

CrossVersionObjectReference

FieldDescriptionValidation Rule

kind

string

name

string

apiVersion

string

Deployment

The Kubernetes resource configuration for a Deployment

FieldDescriptionValidation Rule

pod_annotations

map<string, string>
Pod annotations are an unstructured key value map stored with the pod. https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/

env

List of tetrateio.api.install.kubernetes.EnvVar
Environment variables for all containers in the deployment. https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/

affinity

tetrateio.api.install.kubernetes.Affinity
The scheduling constraints for the pod. https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity

replicaCount

uint32
Number of desired pods. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#deploymentspec-v1-apps

resources

tetrateio.api.install.kubernetes.Resources
Compute Resources required by the primary container in the deployment PodSpec. https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container

strategy

tetrateio.api.install.kubernetes.DeploymentStrategy
The deployment strategy to use to replace existing pods with new ones. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#deploymentstrategy-v1-apps

tolerations

List of k8s.io.api.core.v1.Toleration
Tolerations are applied to pods, and allow (but do not require) the pods to schedule onto nodes with matching taints. Taints and tolerations work together to ensure that pods are not scheduled onto inappropriate nodes. One or more taints are applied to a node; this marks that the node should not accept any pods that do not tolerate the taints. https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/

hpaSpec

tetrateio.api.install.kubernetes.HorizontalPodAutoscalerSpec
Horizontal Pod Autoscaler automatically scales the number of pods in a deployment based on a specified metric. Kubernetes periodically adjusts the number of replicas in a deployment to match the observed metric to the target specified. The version of Horizontal Pod Autoscaler currently used is [v2beta1](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#horizontalpodautoscaler-v2beta1-autoscaling). https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

DeploymentStrategy

The deployment strategy to use to replace existing pods with new ones. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#deploymentstrategy-v1-apps

FieldDescriptionValidation Rule

type

string

rollingUpdate

tetrateio.api.install.kubernetes.RollingUpdateDeployment

EnvVar

FieldDescriptionValidation Rule

name

string

value

string

valueFrom

tetrateio.api.install.kubernetes.EnvVarSource

EnvVarSource

FieldDescriptionValidation Rule

fieldRef

tetrateio.api.install.kubernetes.ObjectFieldSelector

resourceFieldRef

tetrateio.api.install.kubernetes.ResourceFieldSelector

configMapKeyRef

tetrateio.api.install.kubernetes.ConfigMapKeySelector

secretKeyRef

tetrateio.api.install.kubernetes.SecretKeySelector

ExternalMetricSource

FieldDescriptionValidation Rule

metricName

string

metricSelector

k8s.io.apimachinery.pkg.apis.meta.v1.LabelSelector

targetValue

tetrateio.api.install.kubernetes.TypeIntOrString

targetAverageValue

tetrateio.api.install.kubernetes.TypeIntOrString

HorizontalPodAutoscalerSpec

Horizontal Pod Autoscaler automatically scales the number of pods in a deployment based on a specified metric. Kubernetes periodically adjusts the number of replicas in a deployment to match the observed metric to the target specified. This mirrors the Kubernetes spec except from the top level scaleTargetRef field, which we set for you. The version of Horizontal Pod Autoscaler currently used is [v2beta1](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#horizontalpodautoscaler-v2beta1-autoscaling). https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/

FieldDescriptionValidation Rule

minReplicas

int32
Must be set in order to create the HPA resource in Kubernetes

maxReplicas

int32
Must be set in order to create the HPA resource in Kubernetes

metrics

List of tetrateio.api.install.kubernetes.MetricSpec

Job

The Kubernetes resource configuration for a CronJob or Job

FieldDescriptionValidation Rule

pod_annotations

map<string, string>
Pod annotations are an unstructured key value map stored with the pod. https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/

env

List of tetrateio.api.install.kubernetes.EnvVar
Environment variables for all containers in the job. https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/

affinity

tetrateio.api.install.kubernetes.Affinity
The scheduling constraints for the pod. https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity

tolerations

List of k8s.io.api.core.v1.Toleration
Tolerations are applied to pods, and allow (but do not require) the pods to schedule onto nodes with matching taints. Taints and tolerations work together to ensure that pods are not scheduled onto inappropriate nodes. One or more taints are applied to a node; this marks that the node should not accept any pods that do not tolerate the taints. https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/

KubernetesComponentSpec

KubernetesComponentSpec is a common set of Kubernetes resource configuration for components.

FieldDescriptionValidation Rule

deployment

tetrateio.api.install.kubernetes.Deployment
Settings related to the component deployment

service

tetrateio.api.install.kubernetes.Service
Settings related to the component service

overlays

List of tetrateio.api.install.kubernetes.TypeK8SObjectOverlay
Post-render overlays to mutate Kubernetes manifests https://istio.io/latest/docs/reference/config/istio.operator.v1alpha1/#K8sObjectOverlay

KubernetesIstioComponentSpec

KubernetesIstioComponentSpec is the common set of Kubernetes resource configuration for Istio. It differs from the standard component specs in that it supports CNI configuration.

FieldDescriptionValidation Rule

deployment

tetrateio.api.install.kubernetes.Deployment
Settings related to the component deployment

service

tetrateio.api.install.kubernetes.Service
Settings related to the component service

CNI

tetrateio.api.install.kubernetes.CNI
Configure Istio's CNI plugin For further details see: https://istio.io/docs/setup/additional-setup/cni/

overlays

List of tetrateio.api.install.kubernetes.TypeK8SObjectOverlay
Post-render overlays to mutate Kubernetes manifests https://istio.io/latest/docs/reference/config/istio.operator.v1alpha1/#K8sObjectOverlay

KubernetesJobComponentSpec

KubernetesJobComponentSpec is a common set of Kubernetes resource configuration for components with a job associated with them.

FieldDescriptionValidation Rule

deployment

tetrateio.api.install.kubernetes.Deployment
Settings related to the component deployment

service

tetrateio.api.install.kubernetes.Service
Settings related to the component service

job

tetrateio.api.install.kubernetes.Job
Settings related to the component job or cronjob

overlays

List of tetrateio.api.install.kubernetes.TypeK8SObjectOverlay
Post-render overlays to mutate Kubernetes manifests https://istio.io/latest/docs/reference/config/istio.operator.v1alpha1/#K8sObjectOverlay

LocalObjectReference

FieldDescriptionValidation Rule

name

string

MetricSpec

FieldDescriptionValidation Rule

type

string

object

tetrateio.api.install.kubernetes.ObjectMetricSource

pods

tetrateio.api.install.kubernetes.PodsMetricSource

resource

tetrateio.api.install.kubernetes.ResourceMetricSource

external

tetrateio.api.install.kubernetes.ExternalMetricSource

NodeAffinity

Group of node affinity scheduling rules. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#nodeaffinity-v1-core

FieldDescriptionValidation Rule

requiredDuringSchedulingIgnoredDuringExecution

tetrateio.api.install.kubernetes.NodeSelector

preferredDuringSchedulingIgnoredDuringExecution

List of tetrateio.api.install.kubernetes.PreferredSchedulingTerm
The scheduler will prefer to schedule pods to nodes that satisfy the affinity expressions specified by this field, but it may choose a node that violates one or more of the expressions. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#preferredschedulingterm-v1-core

NodeSelector

FieldDescriptionValidation Rule

nodeSelectorTerms

List of tetrateio.api.install.kubernetes.NodeSelectorTerm

NodeSelectorRequirement

FieldDescriptionValidation Rule

key

string

operator

string

values

List of string

NodeSelectorTerm

FieldDescriptionValidation Rule

matchExpressions

List of tetrateio.api.install.kubernetes.NodeSelectorRequirement

matchFields

List of tetrateio.api.install.kubernetes.NodeSelectorRequirement

ObjectFieldSelector

FieldDescriptionValidation Rule

apiVersion

string

fieldPath

string

ObjectMetricSource

FieldDescriptionValidation Rule

target

tetrateio.api.install.kubernetes.CrossVersionObjectReference

metricName

string

targetValue

tetrateio.api.install.kubernetes.TypeIntOrString

selector

k8s.io.apimachinery.pkg.apis.meta.v1.LabelSelector

averageValue

tetrateio.api.install.kubernetes.TypeIntOrString

PodAffinity

Group of inter-pod affinity scheduling rules. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#podaffinity-v1-core

FieldDescriptionValidation Rule

requiredDuringSchedulingIgnoredDuringExecution

List of tetrateio.api.install.kubernetes.PodAffinityTerm

preferredDuringSchedulingIgnoredDuringExecution

List of tetrateio.api.install.kubernetes.WeightedPodAffinityTerm

PodAffinityTerm

FieldDescriptionValidation Rule

labelSelector

k8s.io.apimachinery.pkg.apis.meta.v1.LabelSelector

namespaces

List of string

topologyKey

string

PodAntiAffinity

Group of inter-pod anti-affinity scheduling rules. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#podantiaffinity-v1-core

FieldDescriptionValidation Rule

requiredDuringSchedulingIgnoredDuringExecution

List of tetrateio.api.install.kubernetes.PodAffinityTerm

preferredDuringSchedulingIgnoredDuringExecution

List of tetrateio.api.install.kubernetes.WeightedPodAffinityTerm

PodsMetricSource

FieldDescriptionValidation Rule

metricName

string

targetAverageValue

tetrateio.api.install.kubernetes.TypeIntOrString

selector

k8s.io.apimachinery.pkg.apis.meta.v1.LabelSelector

PreferredSchedulingTerm

FieldDescriptionValidation Rule

weight

int32

preference

tetrateio.api.install.kubernetes.NodeSelectorTerm

ResourceFieldSelector

FieldDescriptionValidation Rule

containerName

string

resource

string

divisor

tetrateio.api.install.kubernetes.TypeIntOrString

ResourceMetricSource

FieldDescriptionValidation Rule

name

string

targetAverageUtilization

tetrateio.api.install.kubernetes.TypeIntOrString

targetAverageValue

tetrateio.api.install.kubernetes.TypeIntOrString

Resources

Mirrors k8s.io.api.core.v1.ResourceRequirements for unmarshalling.

FieldDescriptionValidation Rule

limits

map<string, string>

requests

map<string, string>

RollingUpdateDeployment

Mirrors k8s.io.api.apps.v1.RollingUpdateDeployment for unmarshalling.

FieldDescriptionValidation Rule

maxUnavailable

tetrateio.api.install.kubernetes.TypeIntOrString

maxSurge

tetrateio.api.install.kubernetes.TypeIntOrString

SecretKeySelector

FieldDescriptionValidation Rule

localObjectReference

tetrateio.api.install.kubernetes.LocalObjectReference

key

string

optional

bool

Service

The Kubernetes resource configuration for a Service

FieldDescriptionValidation Rule

annotations

map<string, string>
Pod annotations are an unstructured key value map stored with the service. https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/

ports

List of tetrateio.api.install.kubernetes.ServicePort
List of ports exposed by the component's service. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#serviceport-v1-core

type

string
Determines how the Service is exposed. Valid options are ExternalName, ClusterIP, NodePort, and LoadBalancer. https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services-service-types

ServicePort

FieldDescriptionValidation Rule

name

string

protocol

string

port

int32

targetPort

tetrateio.api.install.kubernetes.TypeIntOrString

nodePort

int32

SessionAffinityConfig

FieldDescriptionValidation Rule

clientIP

tetrateio.api.install.kubernetes.ClientIPConfig

Toleration

FieldDescriptionValidation Rule

key

string

operator

string

value

string

effect

string

tolerationSeconds

int64

TypeIntOrString

GOTYPE: *IntOrString

NameNumberDescription

TypeK8SObjectOverlay

GOTYPE: *K8SObjectOverlayForPB

NameNumberDescription

WeightedPodAffinityTerm

FieldDescriptionValidation Rule

weight

int32

podAffinityTerm

tetrateio.api.install.kubernetes.PodAffinityTerm

k8s.io.api.core.v1.Toleration

The pod this Toleration is attached to tolerates any taint that matches the triple <key,value,effect> using the matching operator <operator>.

FieldDescriptionValidation Rule

key

string
Key is the taint key that the toleration applies to. Empty means match all taint keys. If the key is empty, operator must be Exists; this combination means to match all values and all keys. +optional

operator

string
Operator represents a key's relationship to the value. Valid operators are Exists and Equal. Defaults to Equal. Exists is equivalent to wildcard for value, so that a pod can tolerate all taints of a particular category. +optional

value

string
Value is the taint value the toleration matches to. If the operator is Exists, the value should be empty, otherwise just a regular string. +optional

effect

string
Effect indicates the taint effect to match. Empty means match all taint effects. When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute. +optional

tolerationSeconds

int64
TolerationSeconds represents the period of time the toleration (which must be of effect NoExecute, otherwise this field is ignored) tolerates the taint. By default, it is not set, which means tolerate the taint forever (do not evict). Zero and negative values will be treated as 0 (evict immediately) by the system. +optional

k8s.io.apimachinery.pkg.apis.meta.v1.LabelSelector

A label selector is a label query over a set of resources. The result of matchLabels and matchExpressions are ANDed. An empty label selector matches all objects. A null label selector matches no objects.

FieldDescriptionValidation Rule

matchLabels

map<string, string>
matchLabels is a map of {key,value} pairs. A single {key,value} in the matchLabels map is equivalent to an element of matchExpressions, whose key field is "key", the operator is "In", and the values array contains only "value". The requirements are ANDed.

matchExpressions

List of k8s.io.apimachinery.pkg.apis.meta.v1.LabelSelectorRequirement
matchExpressions is a list of label selector requirements. The requirements are ANDed. +optional

k8s.io.apimachinery.pkg.apis.meta.v1.LabelSelectorRequirement

A label selector requirement is a selector that contains values, a key, and an operator that relates the key and values.

FieldDescriptionValidation Rule

key

string
key is the label key that the selector applies to. +patchMergeKey=key +patchStrategy=merge

operator

string
operator represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists and DoesNotExist.

values

List of string
values is an array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. This array is replaced during a strategic merge patch. +optional