Kubernetes
When installing on Kubernetes, these configuration settings can be used to
override the default Kubernetes configuration. Kubernetes configuration can
be set on each component in the install API using the kubeSpec field.
The API allows for customization of every field in the rendered Kubernetes manifests. The more common configuration fields, such as resources and service type, are supported directly; and can be configured like so:
apiVersion: install.tetrate.io/v1alpha1
kind: ManagementPlane
metadata:
name: managementplane
spec:
hub: docker.io/tetrate
components:
apiServer:
kubeSpec:
service:
type: LoadBalancer
deployment:
resources:
limits:
memory: 750Mi
requests:
memory: 500Mi
All components have a deployment and service object. Some, such as
apiServer, also have a job object associated with them. This can be
configured in a similar manner:
apiVersion: install.tetrate.io/v1alpha1
kind: ManagementPlane
metadata:
name: managementplane
spec:
hub: docker.io/tetrate
components:
apiServer:
kubeSpec:
job:
podAnnotations:
annotation-key: annotation-value
Not all fields in a Kubernetes manifest can be configured directly. This is
to avoid re-implementing the entire Kubernetes API within the install API.
Instead, the kubeSpec object provides an overlays mechanism. This field is
applied after the operator renders the initial manifests and enables support
for customization of any field in a rendered manifest.
Overlays can be applied by selecting the Kubernetes object you wish to
overlay and then describe a list of patches you wish to apply. For example,
to add a hostPort on port 8443 to the frontEnvoy component, do the
following:
apiVersion: install.tetrate.io/v1alpha1
kind: ManagementPlane
metadata:
name: managementplane
spec:
hub: docker.io/tetrate
components:
frontEnvoy:
kubeSpec:
overlays:
- apiVersion: apps/v1
kind: Deployment
name: envoy
patches:
- path:
spec.template.spec.containers.[name:envoy].ports.[containerPort:8443].hostPort
value: 8443
The path refers to the location of the field in the Kubernetes object you
with to patch. The format is a.[key1:value1].b.[:value2]. Where
[key1:value1] is a selector for a key-value pair to identify a list element
and [:value] is a value selector to identify a list element in a leaf list.
All path intermediate nodes must exist.
Overlays are inspired by and bear a loose resemblance to
[kustomize](https://kustomize.io/). We use the library from the Istio
Operator. For more examples of how to construct paths take a look at the
tests in the
upstream.
Affinity
The scheduling constraints for the pod. https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
| Field | Description | Validation Rule |
|---|---|---|
nodeAffinity | tetrateio.api.install.kubernetes.NodeAffinity | – |
podAffinity | tetrateio.api.install.kubernetes.PodAffinity | – |
podAntiAffinity | tetrateio.api.install.kubernetes.PodAntiAffinity | – |
CNI
Configure Istio's CNI plugin For further details see: https://istio.io/docs/setup/additional-setup/cni/
| Field | Description | Validation Rule |
|---|---|---|
binaryDirectory | string | – |
configurationDirectory | string | – |
chained | bool | – |
configurationFileName | string | – |
clusterRole | string | – |
revision | string | – |
Capabilities
See k8s.io.api.core.v1.Capabilities.
| Field | Description | Validation Rule |
|---|---|---|
add | List of string | – |
drop | List of string | – |
ClientIPConfig
| Field | Description | Validation Rule |
|---|---|---|
timeoutSeconds | – |
ConfigMapKeySelector
| Field | Description | Validation Rule |
|---|---|---|
localObjectReference | – | |
key | – | |
optional | – |
ContainerPort
ContainerPort represents a network port in a single container.
| Field | Description | Validation Rule |
|---|---|---|
name | string | – |
hostPort | int32 | – |
containerPort | int32 | – |
protocol | string | – |
hostIP | string | – |
CrossVersionObjectReference
| Field | Description | Validation Rule |
|---|---|---|
kind | – | |
name | – | |
apiVersion | – |
Deployment
The Kubernetes resource configuration for all Deployments
| Field | Description | Validation Rule |
|---|---|---|
podAnnotations | map<string, string> | – |
env | List of tetrateio.api.install.kubernetes.EnvVar | – |
affinity | tetrateio.api.install.kubernetes.Affinity | – |
replicaCount | uint32 | – |
resources | tetrateio.api.install.kubernetes.Resources | – |
strategy | tetrateio.api.install.kubernetes.DeploymentStrategy | – |
tolerations | List of k8s.io.api.core.v1.Toleration | – |
hpaSpec | tetrateio.api.install.kubernetes.HorizontalPodAutoscalerSpec | – |
podSecurityContext | tetrateio.api.install.kubernetes.PodSecurityContext oneof __podSecurityContext | – |
containerSecurityContext | tetrateio.api.install.kubernetes.SecurityContext oneof __containerSecurityContext | – |
DeploymentStrategy
The deployment strategy to use to replace existing pods with new ones. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#deploymentstrategy-v1-apps
| Field | Description | Validation Rule |
|---|---|---|
type | – | |
rollingUpdate | – |
EnvVar
| Field | Description | Validation Rule |
|---|---|---|
name | – | |
value | – | |
valueFrom | – |
EnvVarSource
| Field | Description | Validation Rule |
|---|---|---|
fieldRef | – | |
resourceFieldRef | – | |
configMapKeyRef | – | |
secretKeyRef | – |
ExternalMetricSource
| Field | Description | Validation Rule |
|---|---|---|
metricName | – | |
metricSelector | – | |
targetValue | – | |
targetAverageValue | – |
GlobalDeployment
The Kubernetes resource configuration for a Deployment
| Field | Description | Validation Rule |
|---|---|---|
podAnnotations | map<string, string> | – |
env | List of tetrateio.api.install.kubernetes.EnvVar | – |
affinity | tetrateio.api.install.kubernetes.Affinity | – |
strategy | tetrateio.api.install.kubernetes.DeploymentStrategy | – |
tolerations | List of k8s.io.api.core.v1.Toleration | – |
podSecurityContext | tetrateio.api.install.kubernetes.PodSecurityContext oneof __podSecurityContext | – |
containerSecurityContext | tetrateio.api.install.kubernetes.SecurityContext oneof __containerSecurityContext | – |
GlobalJob
The Kubernetes resource configuration for all CronJob or Job
| Field | Description | Validation Rule |
|---|---|---|
podAnnotations | map<string, string> | – |
affinity | tetrateio.api.install.kubernetes.Affinity | – |
tolerations | List of k8s.io.api.core.v1.Toleration | – |
podSecurityContext | tetrateio.api.install.kubernetes.PodSecurityContext oneof __podSecurityContext | – |
containerSecurityContext | tetrateio.api.install.kubernetes.SecurityContext oneof __containerSecurityContext | – |
GlobalService
The Kubernetes resource configuration for all the Service
| Field | Description | Validation Rule |
|---|---|---|
annotations | map<string, string> | – |
HorizontalPodAutoscalerSpec
Horizontal Pod Autoscaler automatically scales the number of pods in a
deployment based on a specified metric. Kubernetes periodically adjusts the
number of replicas in a deployment to match the observed metric to the target
specified. This mirrors the Kubernetes spec except from the top level
scaleTargetRef field, which we set for you. The version of Horizontal Pod
Autoscaler currently used is
[v2beta1](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#horizontalpodautoscaler-v2beta1-autoscaling).
https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
| Field | Description | Validation Rule |
|---|---|---|
minReplicas | int32 | – |
maxReplicas | int32 | – |
metrics | – |
Job
The Kubernetes resource configuration for a CronJob or Job
| Field | Description | Validation Rule |
|---|---|---|
podAnnotations | map<string, string> | – |
env | List of tetrateio.api.install.kubernetes.EnvVar | – |
affinity | tetrateio.api.install.kubernetes.Affinity | – |
tolerations | List of k8s.io.api.core.v1.Toleration | – |
podSecurityContext | tetrateio.api.install.kubernetes.PodSecurityContext oneof __podSecurityContext | – |
containerSecurityContext | tetrateio.api.install.kubernetes.SecurityContext oneof __containerSecurityContext | – |
KubernetesComponentSpec
KubernetesComponentSpec is a common set of Kubernetes resource configuration for components.
| Field | Description | Validation Rule |
|---|---|---|
deployment | tetrateio.api.install.kubernetes.Deployment | – |
service | tetrateio.api.install.kubernetes.Service | – |
serviceAccount | tetrateio.api.install.kubernetes.ServiceAccount | – |
overlays | List of istio.operator.v1alpha1.K8sObjectOverlay | – |
KubernetesIstioComponentSpec
KubernetesIstioComponentSpec is the common set of Kubernetes resource configuration for Istio. It differs from the standard component specs in that it supports configuring the operator and istiod deployments separately and CNI configuration.
| Field | Description | Validation Rule |
|---|---|---|
deployment | tetrateio.api.install.kubernetes.Deployment | – |
service | tetrateio.api.install.kubernetes.Service | – |
serviceAccount | tetrateio.api.install.kubernetes.ServiceAccount | – |
operatorDeployment | tetrateio.api.install.kubernetes.Deployment | – |
istiodDeployment | tetrateio.api.install.kubernetes.Deployment | – |
CNI | tetrateio.api.install.kubernetes.CNI | – |
overlays | List of istio.operator.v1alpha1.K8sObjectOverlay | – |
KubernetesJobComponentSpec
KubernetesJobComponentSpec is a common set of Kubernetes resource configuration for components with a job associated with them.
| Field | Description | Validation Rule |
|---|---|---|
deployment | tetrateio.api.install.kubernetes.Deployment | – |
service | tetrateio.api.install.kubernetes.Service | – |
job | tetrateio.api.install.kubernetes.Job | – |
serviceAccount | tetrateio.api.install.kubernetes.ServiceAccount | – |
overlays | List of istio.operator.v1alpha1.K8sObjectOverlay | – |
KubernetesSpec
KubernetesSpec is a common set of Kubernetes resource configuration for the install CRs, that will be common to all of its components.
| Field | Description | Validation Rule |
|---|---|---|
deployment | tetrateio.api.install.kubernetes.GlobalDeployment | – |
service | tetrateio.api.install.kubernetes.GlobalService | – |
account | tetrateio.api.install.kubernetes.ServiceAccount | – |
job | tetrateio.api.install.kubernetes.GlobalJob | – |
LocalObjectReference
LocalObjectReference contains enough information to let you locate the referenced object inside the same namespace.
| Field | Description | Validation Rule |
|---|---|---|
name | string | – |
MetricSpec
| Field | Description | Validation Rule |
|---|---|---|
type | – | |
object | – | |
pods | – | |
resource | – | |
external | – |
MetricTarget
MetricTarget provides compatibility with k8s autoscaling/v2 API
| Field | Description | Validation Rule |
|---|---|---|
type | – | |
averageUtilization | – | |
averageValue | – | |
value | – |
NodeAffinity
Group of node affinity scheduling rules. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#nodeaffinity-v1-core
| Field | Description | Validation Rule |
|---|---|---|
requiredDuringSchedulingIgnoredDuringExecution | – | |
preferredDuringSchedulingIgnoredDuringExecution | List of tetrateio.api.install.kubernetes.PreferredSchedulingTerm | – |
NodeSelector
| Field | Description | Validation Rule |
|---|---|---|
nodeSelectorTerms | – |
NodeSelectorRequirement
| Field | Description | Validation Rule |
|---|---|---|
key | – | |
operator | – | |
values | List of string | – |
NodeSelectorTerm
| Field | Description | Validation Rule |
|---|---|---|
matchExpressions | List of tetrateio.api.install.kubernetes.NodeSelectorRequirement | – |
matchFields | List of tetrateio.api.install.kubernetes.NodeSelectorRequirement | – |
ObjectFieldSelector
| Field | Description | Validation Rule |
|---|---|---|
apiVersion | – | |
fieldPath | – |
ObjectMetricSource
| Field | Description | Validation Rule |
|---|---|---|
target | tetrateio.api.install.kubernetes.CrossVersionObjectReference | – |
metricName | – | |
targetValue | – | |
selector | – | |
averageValue | – |
PodAffinity
Group of inter-pod affinity scheduling rules. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#podaffinity-v1-core
| Field | Description | Validation Rule |
|---|---|---|
requiredDuringSchedulingIgnoredDuringExecution | – | |
preferredDuringSchedulingIgnoredDuringExecution | List of tetrateio.api.install.kubernetes.WeightedPodAffinityTerm | – |
PodAffinityTerm
| Field | Description | Validation Rule |
|---|---|---|
labelSelector | – | |
namespaces | List of string | – |
topologyKey | – |
PodAntiAffinity
Group of inter-pod anti-affinity scheduling rules. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#podantiaffinity-v1-core
| Field | Description | Validation Rule |
|---|---|---|
requiredDuringSchedulingIgnoredDuringExecution | – | |
preferredDuringSchedulingIgnoredDuringExecution | List of tetrateio.api.install.kubernetes.WeightedPodAffinityTerm | – |
PodSecurityContext
See k8s.io.api.core.v1.PodSecurityContext.
| Field | Description | Validation Rule |
|---|---|---|
seLinuxOptions | tetrateio.api.install.kubernetes.SELinuxOptions oneof __seLinuxOptions | – |
runAsUser | – | |
runAsNonRoot | – | |
supplementalGroups | List of uint32 | – |
fsGroup | – | |
runAsGroup | – | |
sysctls | – | |
windowsOptions | tetrateio.api.install.kubernetes.WindowsSecurityContextOptions oneof __windowsOptions | – |
fsGroupChangePolicy | – | |
seccompProfile | tetrateio.api.install.kubernetes.SeccompProfile oneof __seccompProfile | – |
PodsMetricSource
| Field | Description | Validation Rule |
|---|---|---|
metricName | – | |
targetAverageValue | – | |
selector | – |
PreferredSchedulingTerm
| Field | Description | Validation Rule |
|---|---|---|
weight | – | |
preference | – |
ResourceFieldSelector
| Field | Description | Validation Rule |
|---|---|---|
containerName | – | |
resource | – | |
divisor | – |
ResourceMetricSource
| Field | Description | Validation Rule |
|---|---|---|
name | – | |
targetAverageUtilization | – | |
targetAverageValue | – | |
target | – |
Resources
Mirrors k8s.io.api.core.v1.ResourceRequirements for unmarshalling.
| Field | Description | Validation Rule |
|---|---|---|
limits | – | |
requests | – |
RollingUpdateDeployment
Mirrors k8s.io.api.apps.v1.RollingUpdateDeployment for unmarshalling.
| Field | Description | Validation Rule |
|---|---|---|
maxUnavailable | – | |
maxSurge | – |
SELinuxOptions
See k8s.io.api.core.v1.SELinuxOptions.
| Field | Description | Validation Rule |
|---|---|---|
user | – | |
role | – | |
type | – | |
level | – |
SeccompProfile
See k8s.io.api.core.v1.SeccompProfile.
| Field | Description | Validation Rule |
|---|---|---|
type | – | |
localhostProfile | – |
SecretKeySelector
| Field | Description | Validation Rule |
|---|---|---|
localObjectReference | – | |
key | – | |
optional | – |
SecurityContext
See k8s.io.api.core.v1.SecurityContext.
| Field | Description | Validation Rule |
|---|---|---|
capabilities | tetrateio.api.install.kubernetes.Capabilities oneof __capabilities | – |
privileged | – | |
seLinuxOptions | tetrateio.api.install.kubernetes.SELinuxOptions oneof __seLinuxOptions | – |
windowsOptions | tetrateio.api.install.kubernetes.WindowsSecurityContextOptions oneof __windowsOptions | – |
runAsUser | – | |
runAsGroup | – | |
runAsNonRoot | – | |
readOnlyRootFilesystem | – | |
allowPrivilegeEscalation | – | |
procMount | – | |
seccompProfile | tetrateio.api.install.kubernetes.SeccompProfile oneof __seccompProfile | – |
Service
The Kubernetes resource configuration for a Service
| Field | Description | Validation Rule |
|---|---|---|
annotations | map<string, string> | – |
ports | List of tetrateio.api.install.kubernetes.ServicePort | – |
type | string | – |
labels | map<string, string> | – |
ServiceAccount
Settings related to the component service account
| Field | Description | Validation Rule |
|---|---|---|
imagePullSecrets | List of tetrateio.api.install.kubernetes.LocalObjectReference | – |
ServicePort
| Field | Description | Validation Rule |
|---|---|---|
name | – | |
protocol | – | |
port | – | |
targetPort | – | |
nodePort | – |
SessionAffinityConfig
| Field | Description | Validation Rule |
|---|---|---|
clientIP | – |
Sysctl
See k8s.io.api.core.v1.Sysctl.
| Field | Description | Validation Rule |
|---|---|---|
name | – | |
value | – |
Toleration
| Field | Description | Validation Rule |
|---|---|---|
key | – | |
operator | – | |
value | – | |
effect | – | |
tolerationSeconds | – |
WeightedPodAffinityTerm
| Field | Description | Validation Rule |
|---|---|---|
weight | – | |
podAffinityTerm | – |
WindowsSecurityContextOptions
See k8s.io.api.core.v1.WindowsSecurityContextOptions.
| Field | Description | Validation Rule |
|---|---|---|
gmsaCredentialSpecName | – | |
gmsaCredentialSpec | – | |
runAsUserName | – |