Kubernetes
Tetrate Service Express (TSE) utilizes many of the same components as the Tetrate Service Bridge(TSB) product but has the several distinctions. Go to Comparing TSE and TSB for more details.
When installing on Kubernetes, these configuration settings can be used to
override the default Kubernetes configuration. Kubernetes configuration can
be set on each component in the install API using the kubeSpec
field.
The API allows for customization of every field in the rendered Kubernetes manifests. The more common configuration fields, such as resources and service type, are supported directly; and can be configured like so:
apiVersion: install.tetrate.io/v1alpha1
kind: ManagementPlane
metadata:
name: managementplane
spec:
hub: docker.io/tetrate
components:
apiServer:
kubeSpec:
service:
type: LoadBalancer
deployment:
resources:
limits:
memory: 750Mi
requests:
memory: 500Mi
All components have a deployment
and service
object. Some, such as
apiServer
, also have a job
object associated with them. This can be
configured in a similar manner:
apiVersion: install.tetrate.io/v1alpha1
kind: ManagementPlane
metadata:
name: managementplane
spec:
hub: docker.io/tetrate
components:
apiServer:
kubeSpec:
job:
podAnnotations:
annotation-key: annotation-value
Not all fields in a Kubernetes manifest can be configured directly. This is
to avoid re-implementing the entire Kubernetes API within the install API.
Instead, the kubeSpec
object provides an overlays mechanism. This field is
applied after the operator renders the initial manifests and enables support
for customization of any field in a rendered manifest.
Overlays can be applied by selecting the Kubernetes object you wish to
overlay and then describe a list of patches you wish to apply. For example,
to add a hostPort
on port 8443 to the frontEnvoy
component, do the
following:
apiVersion: install.tetrate.io/v1alpha1
kind: ManagementPlane
metadata:
name: managementplane
spec:
hub: docker.io/tetrate
components:
frontEnvoy:
kubeSpec:
overlays:
- apiVersion: apps/v1
kind: Deployment
name: envoy
patches:
- path:
spec.template.spec.containers.[name:envoy].ports.[containerPort:8443].hostPort
value: 8443
The path refers to the location of the field in the Kubernetes object you
with to patch. The format is a.[key1:value1].b.[:value2]
. Where
[key1:value1]
is a selector for a key-value pair to identify a list element
and [:value]
is a value selector to identify a list element in a leaf list.
All path intermediate nodes must exist.
Overlays are inspired by and bear a loose resemblance to
[kustomize](https://kustomize.io/)
. We use the library from the Istio
Operator. For more examples of how to construct paths take a look at the
tests in the
upstream.
Affinity
The scheduling constraints for the pod. https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
Field | Description | Validation Rule |
---|---|---|
nodeAffinity | tetrateio.api.install.kubernetes.NodeAffinity | – |
podAffinity | tetrateio.api.install.kubernetes.PodAffinity | – |
podAntiAffinity | tetrateio.api.install.kubernetes.PodAntiAffinity | – |
CNI
Configure Istio's CNI plugin For further details see: https://istio.io/docs/setup/additional-setup/cni/
Field | Description | Validation Rule |
---|---|---|
binaryDirectory | string | – |
configurationDirectory | string | – |
chained | bool | – |
configurationFileName | string | – |
clusterRole | string | – |
revision | string | – |
Capabilities
See k8s.io.api.core.v1.Capabilities.
Field | Description | Validation Rule |
---|---|---|
add | List of string | – |
drop | List of string | – |
ClientIPConfig
Field | Description | Validation Rule |
---|---|---|
timeoutSeconds | – |
ConfigMapKeySelector
Field | Description | Validation Rule |
---|---|---|
localObjectReference | – | |
key | – | |
optional | – |
ContainerPort
ContainerPort represents a network port in a single container.
Field | Description | Validation Rule |
---|---|---|
name | string | – |
hostPort | int32 | – |
containerPort | int32 | – |
protocol | string | – |
hostIP | string | – |
CrossVersionObjectReference
Field | Description | Validation Rule |
---|---|---|
kind | – | |
name | – | |
apiVersion | – |
Deployment
The Kubernetes resource configuration for all Deployments
Field | Description | Validation Rule |
---|---|---|
podAnnotations | map<string, string> | – |
env | List of tetrateio.api.install.kubernetes.EnvVar | – |
affinity | tetrateio.api.install.kubernetes.Affinity | – |
replicaCount | uint32 | – |
resources | tetrateio.api.install.kubernetes.Resources | – |
strategy | tetrateio.api.install.kubernetes.DeploymentStrategy | – |
tolerations | List of k8s.io.api.core.v1.Toleration | – |
hpaSpec | tetrateio.api.install.kubernetes.HorizontalPodAutoscalerSpec | – |
podSecurityContext | tetrateio.api.install.kubernetes.PodSecurityContext oneof __podSecurityContext | – |
containerSecurityContext | tetrateio.api.install.kubernetes.SecurityContext oneof __containerSecurityContext | – |
DeploymentStrategy
The deployment strategy to use to replace existing pods with new ones. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#deploymentstrategy-v1-apps
Field | Description | Validation Rule |
---|---|---|
type | – | |
rollingUpdate | – |
EnvVar
Field | Description | Validation Rule |
---|---|---|
name | – | |
value | – | |
valueFrom | – |
EnvVarSource
Field | Description | Validation Rule |
---|---|---|
fieldRef | – | |
resourceFieldRef | – | |
configMapKeyRef | – | |
secretKeyRef | – |
ExternalMetricSource
Field | Description | Validation Rule |
---|---|---|
metricName | – | |
metricSelector | – | |
targetValue | – | |
targetAverageValue | – |
GlobalDeployment
The Kubernetes resource configuration for a Deployment
Field | Description | Validation Rule |
---|---|---|
podAnnotations | map<string, string> | – |
env | List of tetrateio.api.install.kubernetes.EnvVar | – |
affinity | tetrateio.api.install.kubernetes.Affinity | – |
strategy | tetrateio.api.install.kubernetes.DeploymentStrategy | – |
tolerations | List of k8s.io.api.core.v1.Toleration | – |
podSecurityContext | tetrateio.api.install.kubernetes.PodSecurityContext oneof __podSecurityContext | – |
containerSecurityContext | tetrateio.api.install.kubernetes.SecurityContext oneof __containerSecurityContext | – |
GlobalJob
The Kubernetes resource configuration for all CronJob or Job
Field | Description | Validation Rule |
---|---|---|
podAnnotations | map<string, string> | – |
affinity | tetrateio.api.install.kubernetes.Affinity | – |
tolerations | List of k8s.io.api.core.v1.Toleration | – |
podSecurityContext | tetrateio.api.install.kubernetes.PodSecurityContext oneof __podSecurityContext | – |
containerSecurityContext | tetrateio.api.install.kubernetes.SecurityContext oneof __containerSecurityContext | – |
GlobalService
The Kubernetes resource configuration for all the Service
Field | Description | Validation Rule |
---|---|---|
annotations | map<string, string> | – |
HorizontalPodAutoscalerSpec
Horizontal Pod Autoscaler automatically scales the number of pods in a
deployment based on a specified metric. Kubernetes periodically adjusts the
number of replicas in a deployment to match the observed metric to the target
specified. This mirrors the Kubernetes spec except from the top level
scaleTargetRef
field, which we set for you. The version of Horizontal Pod
Autoscaler currently used is
[v2beta1](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#horizontalpodautoscaler-v2beta1-autoscaling)
.
https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
Field | Description | Validation Rule |
---|---|---|
minReplicas | int32 | – |
maxReplicas | int32 | – |
metrics | – |
Job
The Kubernetes resource configuration for a CronJob or Job
Field | Description | Validation Rule |
---|---|---|
podAnnotations | map<string, string> | – |
env | List of tetrateio.api.install.kubernetes.EnvVar | – |
affinity | tetrateio.api.install.kubernetes.Affinity | – |
tolerations | List of k8s.io.api.core.v1.Toleration | – |
podSecurityContext | tetrateio.api.install.kubernetes.PodSecurityContext oneof __podSecurityContext | – |
containerSecurityContext | tetrateio.api.install.kubernetes.SecurityContext oneof __containerSecurityContext | – |
KubernetesComponentSpec
KubernetesComponentSpec is a common set of Kubernetes resource configuration for components.
Field | Description | Validation Rule |
---|---|---|
deployment | tetrateio.api.install.kubernetes.Deployment | – |
service | tetrateio.api.install.kubernetes.Service | – |
serviceAccount | tetrateio.api.install.kubernetes.ServiceAccount | – |
overlays | List of istio.operator.v1alpha1.K8sObjectOverlay | – |
KubernetesIstioComponentSpec
KubernetesIstioComponentSpec is the common set of Kubernetes resource configuration for Istio. It differs from the standard component specs in that it supports CNI configuration.
Field | Description | Validation Rule |
---|---|---|
deployment | tetrateio.api.install.kubernetes.Deployment | – |
service | tetrateio.api.install.kubernetes.Service | – |
serviceAccount | tetrateio.api.install.kubernetes.ServiceAccount | – |
CNI | tetrateio.api.install.kubernetes.CNI | – |
overlays | List of istio.operator.v1alpha1.K8sObjectOverlay | – |
KubernetesJobComponentSpec
KubernetesJobComponentSpec is a common set of Kubernetes resource configuration for components with a job associated with them.
Field | Description | Validation Rule |
---|---|---|
deployment | tetrateio.api.install.kubernetes.Deployment | – |
service | tetrateio.api.install.kubernetes.Service | – |
job | tetrateio.api.install.kubernetes.Job | – |
serviceAccount | tetrateio.api.install.kubernetes.ServiceAccount | – |
overlays | List of istio.operator.v1alpha1.K8sObjectOverlay | – |
KubernetesSpec
KubernetesSpec is a common set of Kubernetes resource configuration for the install CRs, that will be common to all of its components.
Field | Description | Validation Rule |
---|---|---|
deployment | tetrateio.api.install.kubernetes.GlobalDeployment | – |
service | tetrateio.api.install.kubernetes.GlobalService | – |
account | tetrateio.api.install.kubernetes.ServiceAccount | – |
job | tetrateio.api.install.kubernetes.GlobalJob | – |
LocalObjectReference
LocalObjectReference contains enough information to let you locate the referenced object inside the same namespace.
Field | Description | Validation Rule |
---|---|---|
name | string | – |
MetricSpec
Field | Description | Validation Rule |
---|---|---|
type | – | |
object | – | |
pods | – | |
resource | – | |
external | – |
MetricTarget
MetricTarget provides compatibility with k8s autoscaling/v2 API
Field | Description | Validation Rule |
---|---|---|
type | – | |
averageUtilization | – | |
averageValue | – | |
value | – |
NodeAffinity
Group of node affinity scheduling rules. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#nodeaffinity-v1-core
Field | Description | Validation Rule |
---|---|---|
requiredDuringSchedulingIgnoredDuringExecution | – | |
preferredDuringSchedulingIgnoredDuringExecution | List of tetrateio.api.install.kubernetes.PreferredSchedulingTerm | – |
NodeSelector
Field | Description | Validation Rule |
---|---|---|
nodeSelectorTerms | – |
NodeSelectorRequirement
Field | Description | Validation Rule |
---|---|---|
key | – | |
operator | – | |
values | List of string | – |
NodeSelectorTerm
Field | Description | Validation Rule |
---|---|---|
matchExpressions | List of tetrateio.api.install.kubernetes.NodeSelectorRequirement | – |
matchFields | List of tetrateio.api.install.kubernetes.NodeSelectorRequirement | – |
ObjectFieldSelector
Field | Description | Validation Rule |
---|---|---|
apiVersion | – | |
fieldPath | – |
ObjectMetricSource
Field | Description | Validation Rule |
---|---|---|
target | tetrateio.api.install.kubernetes.CrossVersionObjectReference | – |
metricName | – | |
targetValue | – | |
selector | – | |
averageValue | – |
PodAffinity
Group of inter-pod affinity scheduling rules. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#podaffinity-v1-core
Field | Description | Validation Rule |
---|---|---|
requiredDuringSchedulingIgnoredDuringExecution | – | |
preferredDuringSchedulingIgnoredDuringExecution | List of tetrateio.api.install.kubernetes.WeightedPodAffinityTerm | – |
PodAffinityTerm
Field | Description | Validation Rule |
---|---|---|
labelSelector | – | |
namespaces | List of string | – |
topologyKey | – |
PodAntiAffinity
Group of inter-pod anti-affinity scheduling rules. https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#podantiaffinity-v1-core
Field | Description | Validation Rule |
---|---|---|
requiredDuringSchedulingIgnoredDuringExecution | – | |
preferredDuringSchedulingIgnoredDuringExecution | List of tetrateio.api.install.kubernetes.WeightedPodAffinityTerm | – |
PodSecurityContext
See k8s.io.api.core.v1.PodSecurityContext.
Field | Description | Validation Rule |
---|---|---|
seLinuxOptions | tetrateio.api.install.kubernetes.SELinuxOptions oneof __seLinuxOptions | – |
runAsUser | – | |
runAsNonRoot | – | |
supplementalGroups | List of uint32 | – |
fsGroup | – | |
runAsGroup | – | |
sysctls | – | |
windowsOptions | tetrateio.api.install.kubernetes.WindowsSecurityContextOptions oneof __windowsOptions | – |
fsGroupChangePolicy | – | |
seccompProfile | tetrateio.api.install.kubernetes.SeccompProfile oneof __seccompProfile | – |
PodsMetricSource
Field | Description | Validation Rule |
---|---|---|
metricName | – | |
targetAverageValue | – | |
selector | – |
PreferredSchedulingTerm
Field | Description | Validation Rule |
---|---|---|
weight | – | |
preference | – |
ResourceFieldSelector
Field | Description | Validation Rule |
---|---|---|
containerName | – | |
resource | – | |
divisor | – |
ResourceMetricSource
Field | Description | Validation Rule |
---|---|---|
name | – | |
targetAverageUtilization | – | |
targetAverageValue | – | |
target | – |
Resources
Mirrors k8s.io.api.core.v1.ResourceRequirements for unmarshalling.
Field | Description | Validation Rule |
---|---|---|
limits | – | |
requests | – |
RollingUpdateDeployment
Mirrors k8s.io.api.apps.v1.RollingUpdateDeployment for unmarshalling.
Field | Description | Validation Rule |
---|---|---|
maxUnavailable | – | |
maxSurge | – |
SELinuxOptions
See k8s.io.api.core.v1.SELinuxOptions.
Field | Description | Validation Rule |
---|---|---|
user | – | |
role | – | |
type | – | |
level | – |
SeccompProfile
See k8s.io.api.core.v1.SeccompProfile.
Field | Description | Validation Rule |
---|---|---|
type | – | |
localhostProfile | – |
SecretKeySelector
Field | Description | Validation Rule |
---|---|---|
localObjectReference | – | |
key | – | |
optional | – |
SecurityContext
See k8s.io.api.core.v1.SecurityContext.
Field | Description | Validation Rule |
---|---|---|
capabilities | tetrateio.api.install.kubernetes.Capabilities oneof __capabilities | – |
privileged | – | |
seLinuxOptions | tetrateio.api.install.kubernetes.SELinuxOptions oneof __seLinuxOptions | – |
windowsOptions | tetrateio.api.install.kubernetes.WindowsSecurityContextOptions oneof __windowsOptions | – |
runAsUser | – | |
runAsGroup | – | |
runAsNonRoot | – | |
readOnlyRootFilesystem | – | |
allowPrivilegeEscalation | – | |
procMount | – | |
seccompProfile | tetrateio.api.install.kubernetes.SeccompProfile oneof __seccompProfile | – |
Service
The Kubernetes resource configuration for a Service
Field | Description | Validation Rule |
---|---|---|
annotations | map<string, string> | – |
ports | List of tetrateio.api.install.kubernetes.ServicePort | – |
type | string | – |
labels | map<string, string> | – |
ServiceAccount
Settings related to the component service account
Field | Description | Validation Rule |
---|---|---|
imagePullSecrets | List of tetrateio.api.install.kubernetes.LocalObjectReference | – |
ServicePort
Field | Description | Validation Rule |
---|---|---|
name | – | |
protocol | – | |
port | – | |
targetPort | – | |
nodePort | – |
SessionAffinityConfig
Field | Description | Validation Rule |
---|---|---|
clientIP | – |
Sysctl
See k8s.io.api.core.v1.Sysctl.
Field | Description | Validation Rule |
---|---|---|
name | – | |
value | – |
Toleration
Field | Description | Validation Rule |
---|---|---|
key | – | |
operator | – | |
value | – | |
effect | – | |
tolerationSeconds | – |
WeightedPodAffinityTerm
Field | Description | Validation Rule |
---|---|---|
weight | – | |
podAffinityTerm | – |
WindowsSecurityContextOptions
See k8s.io.api.core.v1.WindowsSecurityContextOptions.
Field | Description | Validation Rule |
---|---|---|
gmsaCredentialSpecName | – | |
gmsaCredentialSpec | – | |
runAsUserName | – |