Control Plane
ControlPlane resource exposes a set of configurations necessary to automatically install the Service Bridge control plane on a cluster. The installation API is an override API so any unset fields that aren't required will use sensible defaults.
Prior to creating the ControlPlane resource, a cluster needs to be created in the management plane. Control plane install scripts would create the following secrets in the Kubernetes namespace the control plane is deployed into. Make sure they exist:
- oap-token
- otel-token
If your Elasticsearch backend requires authentication, ensure you create the following secret:
- elastic-credentials
A minimal resource must have the container registry hub, telemetryStore, and managementPlane fields set.
apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
name: controlplane
namespace: istio-system
spec:
hub: docker.io/tetrate
telemetryStore:
elastic:
host: elastic
port: 5678
managementPlane:
host: tsb.tetrate.io
port: 8443
clusterName: cluster
To configure infrastructure specific settings such as resource limits in Kubernetes, set the relevant field in a component. Remember that the installation API is an override API so if these fields are unset the operator will use sensible defaults. Only a subset of Kubernetes configuration is available and only for individual components.
apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
name: controlplane
namespace: istio-system
spec:
hub: docker.io/tetrate
imagePullSecrets:
- name: my-registry-creds
telemetryStore:
elastic:
host: elastic
port: 5678
managementPlane:
host: tsb.tetrate.io
port: 8443
clusterName: cluster
components:
collector:
kubeSpec:
resources:
limits:
memory: 750Mi
requests:
memory: 500Mi
ControlPlaneComponentSet
The set of components that make up the control plane. Use this to override application settings or Kubernetes settings for each individual component.
Field | Description | Validation Rule |
---|---|---|
collector | tetrateio.api.install.controlplane.v1alpha1.OpenTelemetryCollector | – |
oap | – | |
xcp | – | |
istio | – | |
rateLimitServer | – | |
hpaAdapter | – | |
onboarding | tetrateio.api.install.controlplane.v1alpha1.Onboarding | – |
satellite | tetrateio.api.install.controlplane.v1alpha1.Satellite | – |
ngac | – | |
gitops | tetrateio.api.install.common.GitOps | – |
internalCertProvider | tetrateio.api.install.common.InternalCertProvider | – |
defaultKubeSpec | tetrateio.api.install.kubernetes.KubernetesSpec | – |
ControlPlaneSpec
ControlPlaneSpec defines the desired installed state of control plane components. Specifying a minimal ControlPlaneSpec with hub, clusterName, and managementPlane set will create an installation with sensible defaults.
Field | Description | Validation Rule |
---|---|---|
hub | string | string = { |
imagePullSecrets | List of tetrateio.api.install.kubernetes.LocalObjectReference List of references to secrets in the same namespace to use for pulling any images in pods that reference this ServiceAccount. ImagePullSecrets are distinct from Secrets because Secrets can be mounted in the pod, but ImagePullSecrets are only accessed by the kubelet. More info: https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#service_account-v1-core | – |
components | tetrateio.api.install.controlplane.v1alpha1.ControlPlaneComponentSet | – |
managementPlane | tetrateio.api.install.controlplane.v1alpha1.ManagementPlaneSettings | message = { |
meshExpansion | tetrateio.api.install.controlplane.v1alpha1.MeshExpansionSettings | – |
telemetryStore | tetrateio.api.install.controlplane.v1alpha1.ControlPlaneSpec.TelemetryStore | message = { |
meshObservability | tetrateio.api.install.controlplane.v1alpha1.ControlPlaneSpec.MeshObservability | – |
tier1Cluster | bool | – |
MeshObservability
Configure how the mesh should be observed, which observability functionalities should be
enabled to observe your registered services in the mesh, and the store properties
that TSB will use to persist application observability data like metrics, traces,
logs.
If omitted, the operator will assume
a demo installation and for your convenience install a demo grade mesh observability
setting.
Select one of the MeshObservability
settings to see complete examples.
Field | Description | Validation Rule |
---|---|---|
demoSettings | tetrateio.api.install.common.MeshObservabilitySettings oneof _mesh_observability | – |
settings | tetrateio.api.install.common.MeshObservabilitySettings oneof _mesh_observability | – |
TelemetryStore
Configure the store that TSB will use to persist application telemetry data.
Select one of the TelemetryStore
settings to see complete examples.
Field | Description | Validation Rule |
---|---|---|
elastic | tetrateio.api.install.controlplane.v1alpha1.ElasticSearchSettings oneof _telemetry_store | – |
HpaAdapter
Kubernetes settings for the OAP (SkyWalking) HPA adapter component.
Field | Description | Validation Rule |
---|---|---|
kubeSpec | tetrateio.api.install.kubernetes.KubernetesComponentSpec | – |
IsolationBoundary
IsolationBoundary is an isolated Istio environment which can spread across multiple revisioned control plane clusters.
Example:
isolationBoundaries:
- name: prod
revisions:
- name: stable
istio:
tsbVersion: 1.6.0
- name: staging
revisions:
- name: v1_6_3
istio:
tsbVersion: 1.6.3
- name: v1_6_1
istio:
tsbVersion: 1.6.1
disable: true
The tsbVersion
field can be left empty, which would then default to the
current TSB released version.
isolationBoundaries:
- name: global
istio:
- revisions: stable
For instance, if isolation boundaries are being added in TSB 1.6.1
, the default
would looks something like this:
isolationBoundaries:
- name: global
revisions:
- name: stable
istio:
tsbVersion: 1.6.1
Field | Description | Validation Rule |
---|---|---|
name | string | string = { |
revisions | List of tetrateio.api.install.controlplane.v1alpha1.IstioRevision | repeated = { |
Istio
Mesh and Kubernetes settings for Istio.
Field | Description | Validation Rule |
---|---|---|
tsbVersion | string If not provided explicitly, this defaults to the current tsb version. | – |
kubeSpec | tetrateio.api.install.kubernetes.KubernetesIstioComponentSpec | – |
traceSamplingRate | double | – |
defaultWorkloadCertTTL | google.protobuf.Duration | – |
maxWorkloadCertTTL | google.protobuf.Duration | – |
trustDomain | string | – |
baseOverlays | List of istio.operator.v1alpha1.K8sObjectOverlay | – |
pilotOverlays | List of istio.operator.v1alpha1.K8sObjectOverlay | – |
cniOverlays | List of istio.operator.v1alpha1.K8sObjectOverlay | – |
IstioRevision
Istio control plane settings for a specific revision.
Field | Description | Validation Rule |
---|---|---|
name | string Notice that the value constraints here are stricter than the ones in Istio. Apparently, Istio validation rules allow values that lead to internal failures at runtime, e.g. values with capital letters or values longer than 56 characters. Stricter validation rules here are meant to prevent those hidden pitfalls. | string = { |
istio | tetrateio.api.install.controlplane.v1alpha1.Istio | message = { |
disable | bool | – |
NGAC
Kubernetes settings for the NGAC component.
Field | Description | Validation Rule |
---|---|---|
enabled | bool | – |
kubeSpec | tetrateio.api.install.kubernetes.KubernetesComponentSpec | – |
logLevels | map<string, string> | – |
Oap
Kubernetes settings for the OAP (SkyWalking) component.
Field | Description | Validation Rule |
---|---|---|
streamingLogEnabled | bool | – |
onDemandEnvoyMetricsEnabled | bool | – |
storageIndexMergingEnabled | bool | – |
kubeSpec | tetrateio.api.install.kubernetes.KubernetesComponentSpec | – |
Onboarding
Settings for the Workload Onboarding
component.
Field | Description | Validation Rule |
---|---|---|
operator | tetrateio.api.install.controlplane.v1alpha1.OnboardingOperator | – |
repository | tetrateio.api.install.controlplane.v1alpha1.OnboardingRepository | – |
plane | tetrateio.api.install.controlplane.v1alpha1.OnboardingPlane | – |
OnboardingOperator
Kubernetes settings for the Workload Onboarding Operator
component.
Field | Description | Validation Rule |
---|---|---|
kubeSpec | tetrateio.api.install.kubernetes.KubernetesComponentSpec | – |
OnboardingPlane
Configure Workload Onboarding Plane
component.
Field | Description | Validation Rule |
---|---|---|
instance | tetrateio.api.onboarding.config.install.v1alpha1.OnboardingPlaneInstance | – |
OnboardingRepository
Kubernetes settings for the Workload Onboarding Repository
component.
Field | Description | Validation Rule |
---|---|---|
kubeSpec | tetrateio.api.install.kubernetes.KubernetesComponentSpec | – |
OpenTelemetryCollector
Kubernetes settings for the OpenTelemetryCollector component.
Field | Description | Validation Rule |
---|---|---|
kubeSpec | tetrateio.api.install.kubernetes.KubernetesComponentSpec | – |
RateLimitServer
Configuration settings for the RateLimit Server
Field | Description | Validation Rule |
---|---|---|
backend | tetrateio.api.install.controlplane.v1alpha1.RateLimitServer.Backend | message = { |
domain | string | – |
kubeSpec | tetrateio.api.install.kubernetes.KubernetesComponentSpec | – |
Backend
External Backend Database types. This points to the backend used by the ratelimit server as a key/value store.
Field | Description | Validation Rule |
---|---|---|
redis | tetrateio.api.install.controlplane.v1alpha1.RateLimitServer.Backend.RedisSettings oneof _backend_specifier | – |
RedisSettings
Configuration for the External Redis Backend Database
Field | Description | Validation Rule |
---|---|---|
uri | string | string = { |
Satellite
Kubernetes settings for the Satellite (SkyWalking-Satellite) component.
Field | Description | Validation Rule |
---|---|---|
enabled | bool | – |
kubeSpec | tetrateio.api.install.kubernetes.KubernetesComponentSpec | – |
XCP
Kubernetes settings for the XCP component.
Field | Description | Validation Rule |
---|---|---|
centralAuthMode | tetrateio.api.install.controlplane.v1alpha1.XCP.CentralAuthMode | – |
configProtection | tetrateio.api.install.common.ConfigProtection | – |
kubeSpec | tetrateio.api.install.kubernetes.KubernetesComponentSpec | – |
isolationBoundaries | List of tetrateio.api.install.controlplane.v1alpha1.IsolationBoundary | – |
enableHttpMeshInternalIdentityPropagation | bool | – |
ElasticSearchSettings
Configure an Elasticsearch connection.
apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
name: controlplane
namespace: istio-system
spec:
telemetryStore:
elastic:
host: elastic
port: 5678
protocol: https
selfSigned: true
version: 7
Field | Description | Validation Rule |
---|---|---|
host | string | string = { |
port | int32 | int32 = { |
protocol | tetrateio.api.install.controlplane.v1alpha1.ElasticSearchSettings.Protocol | – |
selfSigned | bool | – |
version | int32 | int32 = { |
ManagementPlaneSettings
Configure the management plane connection.
apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
name: controlplane
namespace: istio-system
spec:
managementPlane:
host: tsb.tetrate.io
port: 8443
selfSigned: true
clusterName: control-plane-cluster
Field | Description | Validation Rule |
---|---|---|
host | string | – |
port | int32 | int32 = { |
selfSigned | bool | – |
clusterName | string | string = { |
MeshExpansionSettings
Configure mesh expansion to connect workloads external to Kubernetes to the mesh.
To enable mesh expansion set it to an empty object:
apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
name: controlplane
namespace: istio-system
spec:
meshExpansion: \{\}
If external workloads are unable to communicate with the default mesh expansion gateway via external IPs or hostnames, then you must specify the gateway that enables them to do so. This custom gateway must be configured to forward this communication to the VM gateway service:
apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
name: controlplane
namespace: istio-system
spec:
meshExpansion:
customGateway:
host: customgateway.tetrate.io
port: 15443
To automate onboarding of workloads from auto-scaling groups of VMs, you need
to enable the Workload Onboarding Plane
.
Workload Onboarding Agent
, a component that you install next to the workload,
will connect to the Workload Onboarding Plane
to authenticate itself, ask
permission to join the mesh, register the workload into the mesh and retrieve
boot configuration required to start Istio Sidecar
.
All communication between the Workload Onboarding Agent
and the
Workload Onboarding Plane
must occur over TLS.
Therefore, to enable Workload Onboarding Plane
you must provide a TLS
certificate for the endpoint that exposes Workload Onboarding API
to
Workload Onboarding Agents
.
Make sure that TLS certificate is signed by the certificate authority known
to Workload Onboarding Agents
.
apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
name: controlplane
namespace: istio-system
spec:
meshExpansion:
onboarding:
endpoint:
hosts:
- onboarding.example.org
secretName: onboarding-tls-cert
tokenIssuer:
jwt:
expiration: 1h
localRepository: \{\}
To onboard workloads from custom on-premise environments, you can leverage support for OIDC ID Tokens.
If workloads in your custom environment can authenticate themselves by means of an
OIDC ID Token,
you can define a list of JWT issuers permitted by the Workload Onboarding Plane
.
For example,
apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
name: controlplane
namespace: istio-system
spec:
meshExpansion:
onboarding:
endpoint:
hosts:
- onboarding.example.org
secretName: onboarding-tls-cert
localRepository: \{\}
workloads:
authentication:
jwt:
issuers:
- issuer: "https://mycompany.corp"
jwksUri: "https://mycompany.corp/jwks.json"
shortName: "mycorp"
tokenFields:
attributes:
jsonPath: .custom_attributes
To ensure there will be no traffic loss when an onboarded workload gets shutdown, you can configure the time period to delay the shutdown for after deregistering the workload from the mesh, which will give enough time to reconfigure all affected mesh nodes to not load balance requests to the deregistered workload before it becomes unavailable.
For example,
apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
name: controlplane
namespace: istio-system
spec:
meshExpansion:
onboarding:
endpoint:
hosts:
- onboarding.example.org
secretName: onboarding-tls-cert
localRepository: \{\}
workloads:
deregistration:
propagationDelay: 15s
Field | Description | Validation Rule |
---|---|---|
customGateway | tetrateio.api.install.controlplane.v1alpha1.MeshExpansionSettings.Gateway | – |
onboarding | tetrateio.api.install.controlplane.v1alpha1.MeshExpansionSettings.OnboardingPlane | – |
Gateway
A custom mesh expansion gateway. This is required when the workload can't access the default gateway directly via the external IP or hostname.
Field | Description | Validation Rule |
---|---|---|
host | string | string = { |
port | int32 | int32 = { |
OnboardingPlane
Configuration of the Workload Onboarding Plane
.
Field | Description | Validation Rule |
---|---|---|
uid | string Is used in the workload authentication flow to prevent replay attacks
that abuse compromised workload credentials intended for a different
installation of the Defaults to an auto-generated UUID. | string = { |
endpoint | tetrateio.api.install.controlplane.v1alpha1.MeshExpansionSettings.OnboardingPlane.Endpoint | message = { |
tokenIssuer | tetrateio.api.install.controlplane.v1alpha1.MeshExpansionSettings.OnboardingPlane.TokenIssuer | message = { |
localRepository | tetrateio.api.install.controlplane.v1alpha1.MeshExpansionSettings.OnboardingPlane.LocalRepository Local repository is disabled by default. To enable it, set this
field to an empty value, i.e. | – |
workloads | tetrateio.api.onboarding.config.install.v1alpha1.WorkloadConfiguration | – |
Endpoint
Configuration of the endpoint exposing Workload Onboarding API
to
Workload Onboarding Agents
.
Field | Description | Validation Rule |
---|---|---|
hosts | List of string | repeated = { |
secretName | string | string = { |
LocalRepository
Configuration of the local repository with DEB
and RPM
packages
of the Workload Onboarding Agent
and Istio Sidecar
.
TokenIssuer
Configuration of the built-in Workload Onboarding Token Issuer
.
Field | Description | Validation Rule |
---|---|---|
jwt | tetrateio.api.install.controlplane.v1alpha1.MeshExpansionSettings.OnboardingPlane.TokenIssuer.JwtTokenIssuer oneof _token_issuer | – |
JwtTokenIssuer
Configuration of the built-in JWT Token Issuer.
Field | Description | Validation Rule |
---|---|---|
expiration | google.protobuf.Duration | duration = { |
CentralAuthMode
Authentication mode for connections from XCP Edges to XCP Central
Field | Number | Description |
---|---|---|
UNKNOWN | 0 | Default when unset, do not use |
MUTUAL_TLS | 1 | GRPC stream is encrypted with mutual TLS |
JWT | 2 | XCP Edges present a JWT bearer token in the GRPC headers |
Protocol
The list of supported protocols to communicate with Elasticsearch.
Field | Number | Description |
---|---|---|
https | 0 | |
http | 1 |