Kubernetes
When installing on Kubernetes, these configuration settings can be used to
override the default Kubernetes configuration. Kubernetes configuration can
be set on each component in the install API using the kubeSpec
field.
The API allows for customization of every field in the rendered Kubernetes manifests. The more common configuration fields, such as resources and service type, are supported directly; and can be configured like so:
apiVersion: install.tetrate.io/v1alpha1
kind: ManagementPlane
metadata:
name: managementplane
spec:
hub: docker.io/tetrate
components:
apiServer:
kubeSpec:
service:
type: LoadBalancer
deployment:
resources:
limits:
memory: 750Mi
requests:
memory: 500Mi
All components have a deployment
and service
object. Some, such as
apiServer
, also have a job
object associated with them. This can be
configured in a similar manner:
apiVersion: install.tetrate.io/v1alpha1
kind: ManagementPlane
metadata:
name: managementplane
spec:
hub: docker.io/tetrate
components:
apiServer:
kubeSpec:
job:
podAnnotations:
annotation-key: annotation-value
Not all fields in a Kubernetes manifest can be configured directly. This is
to avoid re-implementing the entire Kubernetes API within the install API.
Instead, the kubeSpec
object provides an overlays mechanism. This field is
applied after the operator renders the initial manifests and enables support
for customization of any field in a rendered manifest.
Overlays can be applied by selecting the Kubernetes object you wish to
overlay and then describe a list of patches you wish to apply. For example,
to add a hostPort
on port 8443 to the frontEnvoy
component, do the
following:
apiVersion: install.tetrate.io/v1alpha1
kind: ManagementPlane
metadata:
name: managementplane
spec:
hub: docker.io/tetrate
components:
frontEnvoy:
kubeSpec:
overlays:
- apiVersion: apps/v1
kind: Deployment
name: envoy
patches:
- path:
spec.template.spec.containers.[name:envoy].ports.[containerPort:8443].hostPort
value: 8443
The path refers to the location of the field in the Kubernetes object you
with to patch. The format is a.[key1:value1].b.[:value2]
. Where
[key1:value1]
is a selector for a key-value pair to identify a list element
and [:value]
is a value selector to identify a list element in a leaf list.
All path intermediate nodes must exist.
Overlays are inspired by and bear a loose resemblance to
[kustomize](https://kustomize.io/)
. We use the library from the Istio
Operator. For more examples of how to construct paths take a look at the
tests in the
upstream.
Affinity #
The scheduling constraints for the pod. https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
Field | Description | Validation Rule |
---|---|---|
nodeAffinity | tetrateio.api.install.kubernetes.NodeAffinity | – |
podAffinity | tetrateio.api.install.kubernetes.PodAffinity | – |
podAntiAffinity | tetrateio.api.install.kubernetes.PodAntiAffinity | – |
CNI #
Configure Istio's CNI plugin For further details see: https://istio.io/docs/setup/additional-setup/cni/
Field | Description | Validation Rule |
---|---|---|
binaryDirectory | string | – |
configurationDirectory | string | – |
chained | bool | – |
configurationFileName | string | – |
clusterRole | string | – |
revision | string | – |
Capabilities #
See k8s.io.api.core.v1.Capabilities.
Field | Description | Validation Rule |
---|---|---|
add | List of string | – |
drop | List of string | – |
ClientIPConfig #
Field | Description | Validation Rule |
---|---|---|
timeoutSeconds | – |
ConfigMapKeySelector #
Field | Description | Validation Rule |
---|---|---|
localObjectReference | – | |
key | – | |
optional | – |
ContainerPort #
ContainerPort represents a network port in a single container.
Field | Description | Validation Rule |
---|---|---|
name | string | – |
hostPort | int32 | – |
containerPort | int32 | – |
protocol | string | – |
hostIP | string | – |
CrossVersionObjectReference #
Field | Description | Validation Rule |
---|---|---|
kind | – | |
name | – | |
apiVersion | – |
Deployment #
The Kubernetes resource configuration for all Deployments
Field | Description | Validation Rule |
---|---|---|
podAnnotations | map<string, string> | – |
env | List of tetrateio.api.install.kubernetes.EnvVar | – |
affinity | tetrateio.api.install.kubernetes.Affinity | – |
replicaCount | uint32 | – |
resources | tetrateio.api.install.kubernetes.Resources | – |
strategy | tetrateio.api.install.kubernetes.DeploymentStrategy | – |
tolerations | List of k8s.io.api.core.v1.Toleration | – |
hpaSpec | tetrateio.api.install.kubernetes.HorizontalPodAutoscalerSpec | – |
podSecurityContext | tetrateio.api.install.kubernetes.PodSecurityContext oneof __podSecurityContext | – |
containerSecurityContext | tetrateio.api.install.kubernetes.SecurityContext oneof __containerSecurityContext | – |
DeploymentStrategy #
The deployment strategy to use to replace existing pods with new ones. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#deploymentstrategy-v1-apps
Field | Description | Validation Rule |
---|---|---|
type | – | |
rollingUpdate | – |
EnvVar #
Field | Description | Validation Rule |
---|---|---|
name | – | |
value | – | |
valueFrom | – |
EnvVarSource #
Field | Description | Validation Rule |
---|---|---|
fieldRef | – | |
resourceFieldRef | – | |
configMapKeyRef | – | |
secretKeyRef | – |
ExternalMetricSource #
Field | Description | Validation Rule |
---|---|---|
metricName | – | |
metricSelector | – | |
targetValue | – | |
targetAverageValue | – |
GlobalDeployment #
The Kubernetes resource configuration for a Deployment
Field | Description | Validation Rule |
---|---|---|
podAnnotations | map<string, string> | – |
env | List of tetrateio.api.install.kubernetes.EnvVar | – |
affinity | tetrateio.api.install.kubernetes.Affinity | – |
strategy | tetrateio.api.install.kubernetes.DeploymentStrategy | – |
tolerations | List of k8s.io.api.core.v1.Toleration | – |
podSecurityContext | tetrateio.api.install.kubernetes.PodSecurityContext oneof __podSecurityContext | – |
containerSecurityContext | tetrateio.api.install.kubernetes.SecurityContext oneof __containerSecurityContext | – |
GlobalJob #
The Kubernetes resource configuration for all CronJob or Job
Field | Description | Validation Rule |
---|---|---|
podAnnotations | map<string, string> | – |
affinity | tetrateio.api.install.kubernetes.Affinity | – |
tolerations | List of k8s.io.api.core.v1.Toleration | – |
podSecurityContext | tetrateio.api.install.kubernetes.PodSecurityContext oneof __podSecurityContext | – |
containerSecurityContext | tetrateio.api.install.kubernetes.SecurityContext oneof __containerSecurityContext | – |
GlobalService #
The Kubernetes resource configuration for all the Service
Field | Description | Validation Rule |
---|---|---|
annotations | map<string, string> | – |
HorizontalPodAutoscalerSpec #
Horizontal Pod Autoscaler automatically scales the number of pods in a
deployment based on a specified metric. Kubernetes periodically adjusts the
number of replicas in a deployment to match the observed metric to the target
specified. This mirrors the Kubernetes spec except from the top level
scaleTargetRef
field, which we set for you. The version of Horizontal Pod
Autoscaler currently used is
[v2beta1](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#horizontalpodautoscaler-v2beta1-autoscaling)
.
https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
Field | Description | Validation Rule |
---|---|---|
minReplicas | int32 | – |
maxReplicas | int32 | – |
metrics | – |
Job #
The Kubernetes resource configuration for a CronJob or Job
Field | Description | Validation Rule |
---|---|---|
podAnnotations | map<string, string> | – |
env | List of tetrateio.api.install.kubernetes.EnvVar | – |
affinity | tetrateio.api.install.kubernetes.Affinity | – |
tolerations | List of k8s.io.api.core.v1.Toleration | – |
podSecurityContext | tetrateio.api.install.kubernetes.PodSecurityContext oneof __podSecurityContext | – |
containerSecurityContext | tetrateio.api.install.kubernetes.SecurityContext oneof __containerSecurityContext | – |
KubernetesComponentSpec #
KubernetesComponentSpec is a common set of Kubernetes resource configuration for components.
Field | Description | Validation Rule |
---|---|---|
deployment | tetrateio.api.install.kubernetes.Deployment | – |
service | tetrateio.api.install.kubernetes.Service | – |
serviceAccount | tetrateio.api.install.kubernetes.ServiceAccount | – |
overlays | List of istio.operator.v1alpha1.K8sObjectOverlay | – |
KubernetesIstioComponentSpec #
KubernetesIstioComponentSpec is the common set of Kubernetes resource configuration for Istio. It differs from the standard component specs in that it supports CNI configuration.
Field | Description | Validation Rule |
---|---|---|
deployment | tetrateio.api.install.kubernetes.Deployment | – |
service | tetrateio.api.install.kubernetes.Service | – |
serviceAccount | tetrateio.api.install.kubernetes.ServiceAccount | – |
CNI | tetrateio.api.install.kubernetes.CNI | – |
overlays | List of istio.operator.v1alpha1.K8sObjectOverlay | – |
KubernetesJobComponentSpec #
KubernetesJobComponentSpec is a common set of Kubernetes resource configuration for components with a job associated with them.
Field | Description | Validation Rule |
---|---|---|
deployment | tetrateio.api.install.kubernetes.Deployment | – |
service | tetrateio.api.install.kubernetes.Service | – |
job | tetrateio.api.install.kubernetes.Job | – |
serviceAccount | tetrateio.api.install.kubernetes.ServiceAccount | – |
overlays | List of istio.operator.v1alpha1.K8sObjectOverlay | – |
KubernetesSpec #
KubernetesSpec is a common set of Kubernetes resource configuration for the install CRs, that will be common to all of its components.
Field | Description | Validation Rule |
---|---|---|
deployment | tetrateio.api.install.kubernetes.GlobalDeployment | – |
service | tetrateio.api.install.kubernetes.GlobalService | – |
account | tetrateio.api.install.kubernetes.ServiceAccount | – |
job | tetrateio.api.install.kubernetes.GlobalJob | – |
LocalObjectReference #
LocalObjectReference contains enough information to let you locate the referenced object inside the same namespace.
Field | Description | Validation Rule |
---|---|---|
name | string | – |
MetricSpec #
Field | Description | Validation Rule |
---|---|---|
type | – | |
object | – | |
pods | – | |
resource | – | |
external | – |
MetricTarget #
MetricTarget provides compatibility with k8s autoscaling/v2 API
Field | Description | Validation Rule |
---|---|---|
type | – | |
averageUtilization | – | |
averageValue | – | |
value | – |
NodeAffinity #
Group of node affinity scheduling rules. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#nodeaffinity-v1-core
Field | Description | Validation Rule |
---|---|---|
requiredDuringSchedulingIgnoredDuringExecution | – | |
preferredDuringSchedulingIgnoredDuringExecution | List of tetrateio.api.install.kubernetes.PreferredSchedulingTerm | – |
NodeSelector #
Field | Description | Validation Rule |
---|---|---|
nodeSelectorTerms | – |
NodeSelectorRequirement #
Field | Description | Validation Rule |
---|---|---|
key | – | |
operator | – | |
values | List of string | – |
NodeSelectorTerm #
Field | Description | Validation Rule |
---|---|---|
matchExpressions | List of tetrateio.api.install.kubernetes.NodeSelectorRequirement | – |
matchFields | List of tetrateio.api.install.kubernetes.NodeSelectorRequirement | – |
ObjectFieldSelector #
Field | Description | Validation Rule |
---|---|---|
apiVersion | – | |
fieldPath | – |
ObjectMetricSource #
Field | Description | Validation Rule |
---|---|---|
target | tetrateio.api.install.kubernetes.CrossVersionObjectReference | – |
metricName | – | |
targetValue | – | |
selector | – | |
averageValue | – |
PodAffinity #
Group of inter-pod affinity scheduling rules. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#podaffinity-v1-core
Field | Description | Validation Rule |
---|---|---|
requiredDuringSchedulingIgnoredDuringExecution | – | |
preferredDuringSchedulingIgnoredDuringExecution | List of tetrateio.api.install.kubernetes.WeightedPodAffinityTerm | – |
PodAffinityTerm #
Field | Description | Validation Rule |
---|---|---|
labelSelector | – | |
namespaces | List of string | – |
topologyKey | – |
PodAntiAffinity #
Group of inter-pod anti-affinity scheduling rules. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#podantiaffinity-v1-core
Field | Description | Validation Rule |
---|---|---|
requiredDuringSchedulingIgnoredDuringExecution | – | |
preferredDuringSchedulingIgnoredDuringExecution | List of tetrateio.api.install.kubernetes.WeightedPodAffinityTerm | – |
PodSecurityContext #
See k8s.io.api.core.v1.PodSecurityContext.
Field | Description | Validation Rule |
---|---|---|
seLinuxOptions | tetrateio.api.install.kubernetes.SELinuxOptions oneof __seLinuxOptions | – |
runAsUser | – | |
runAsNonRoot | – | |
supplementalGroups | List of uint32 | – |
fsGroup | – | |
runAsGroup | – | |
sysctls | – | |
windowsOptions | tetrateio.api.install.kubernetes.WindowsSecurityContextOptions oneof __windowsOptions | – |
fsGroupChangePolicy | – | |
seccompProfile | tetrateio.api.install.kubernetes.SeccompProfile oneof __seccompProfile | – |
PodsMetricSource #
Field | Description | Validation Rule |
---|---|---|
metricName | – | |
targetAverageValue | – | |
selector | – |
PreferredSchedulingTerm #
Field | Description | Validation Rule |
---|---|---|
weight | – | |
preference | – |
ResourceFieldSelector #
Field | Description | Validation Rule |
---|---|---|
containerName | – | |
resource | – | |
divisor | – |
ResourceMetricSource #
Field | Description | Validation Rule |
---|---|---|
name | – | |
targetAverageUtilization | – | |
targetAverageValue | – | |
target | – |
Resources #
Mirrors k8s.io.api.core.v1.ResourceRequirements for unmarshalling.
Field | Description | Validation Rule |
---|---|---|
limits | – | |
requests | – |
RollingUpdateDeployment #
Mirrors k8s.io.api.apps.v1.RollingUpdateDeployment for unmarshalling.
Field | Description | Validation Rule |
---|---|---|
maxUnavailable | – | |
maxSurge | – |
SELinuxOptions #
See k8s.io.api.core.v1.SELinuxOptions.
Field | Description | Validation Rule |
---|---|---|
user | – | |
role | – | |
type | – | |
level | – |
SeccompProfile #
See k8s.io.api.core.v1.SeccompProfile.
Field | Description | Validation Rule |
---|---|---|
type | – | |
localhostProfile | – |
SecretKeySelector #
Field | Description | Validation Rule |
---|---|---|
localObjectReference | – | |
key | – | |
optional | – |
SecurityContext #
See k8s.io.api.core.v1.SecurityContext.
Field | Description | Validation Rule |
---|---|---|
capabilities | tetrateio.api.install.kubernetes.Capabilities oneof __capabilities | – |
privileged | – | |
seLinuxOptions | tetrateio.api.install.kubernetes.SELinuxOptions oneof __seLinuxOptions | – |
windowsOptions | tetrateio.api.install.kubernetes.WindowsSecurityContextOptions oneof __windowsOptions | – |
runAsUser | – | |
runAsGroup | – | |
runAsNonRoot | – | |
readOnlyRootFilesystem | – | |
allowPrivilegeEscalation | – | |
procMount | – | |
seccompProfile | tetrateio.api.install.kubernetes.SeccompProfile oneof __seccompProfile | – |
Service #
The Kubernetes resource configuration for a Service
Field | Description | Validation Rule |
---|---|---|
annotations | map<string, string> | – |
ports | List of tetrateio.api.install.kubernetes.ServicePort | – |
type | string | – |
labels | map<string, string> | – |
ServiceAccount #
Settings related to the component service account
Field | Description | Validation Rule |
---|---|---|
imagePullSecrets | List of tetrateio.api.install.kubernetes.LocalObjectReference | – |
ServicePort #
Field | Description | Validation Rule |
---|---|---|
name | – | |
protocol | – | |
port | – | |
targetPort | – | |
nodePort | – |
SessionAffinityConfig #
Field | Description | Validation Rule |
---|---|---|
clientIP | – |
Sysctl #
See k8s.io.api.core.v1.Sysctl.
Field | Description | Validation Rule |
---|---|---|
name | – | |
value | – |
Toleration #
Field | Description | Validation Rule |
---|---|---|
key | – | |
operator | – | |
value | – | |
effect | – | |
tolerationSeconds | – |
WeightedPodAffinityTerm #
Field | Description | Validation Rule |
---|---|---|
weight | – | |
podAffinityTerm | – |
WindowsSecurityContextOptions #
See k8s.io.api.core.v1.WindowsSecurityContextOptions.
Field | Description | Validation Rule |
---|---|---|
gmsaCredentialSpecName | – | |
gmsaCredentialSpec | – | |
runAsUserName | – |