Skip to main content
logoTetrate Service BridgeVersion: 1.10.x

Isolation Boundary Installation

Before you begin

Enabling or Upgrading to Isolation boundaries is a complex process. Please make sure to review this entire document to correctly plan the install or upgrade.

Important Note - istio-gateway namespace

During the upgrade process, the tsb-operator will delete the existing istio-gateway namespace. In some situations, this delete will hang indefinitely due to stuck finalizers that prevent the delete from completing. You may need to edit the namespace and delete any finalizers in order to allow the deletion to complete.

Do also refer to the Troubleshooting section at the end of this document.

Installation - Single IsolationBoundary

If you are mainly looking for seamless control plane upgrade using revisions as the primary use-case, you can go with single IsolationBoundary configuration.

For a fresh installation, you can follow the standard steps to onboard a control plane cluster using tctl or helm along with the following changes:

  1. Enable isolation boundary in TSB control plane operator by setting ISTIO_ISOLATION_BOUNDARIES to true
  2. Add isolation boundaries definition in ControlPlane CR or control plane Helm values.

In the following example, you will use Helm to onboard a cluster with a single IsolationBoundary enabled.

Install with Helm

Follow the instructions in control plane installation with Helm using following Helm values to enable Istio Isolation Boundaries.

operator:
deployment:
env:
- name: ISTIO_ISOLATION_BOUNDARIES
value: "true"

spec:
managementPlane:
clusterName: <cluster-name-in-tsb>
host: <tsb-address>
port: <tsb-port>
telemetryStore:
elastic:
host: <tsb-address>
port: <tsb-port>
version: <elastic-version>
selfSigned: <is-elastic-use-self-signed-certificate>
components:
xcp:
isolationBoundaries:
- name: global
revisions:
- name: default
- name: canary

secrets:
clusterServiceAccount:
clusterFQN: organizations/jupiter/clusters/<cluster-name-in-tsb>
JWK: '$JWK'

Install with tctl

If you prefer to use tctl installation, you can use following command to generate cluster operator with Istio Isolation Boundaries enabled

tctl install manifest cluster-operators \
--registry <registry-location> \
--set "operator.deployment.env[0].name=ISTIO_ISOLATION_BOUNDARIES" \
--set "operator.deployment.env[0].value='true'" > clusteroperators.yaml

Then update xcp component in your ControlPlane CR or Helm values with isolationBoundaries:

spec:
...
components:
xcp:
isolationBoundaries:
- name: global
revisions:
- name: default
- name: canary

With these configurations, two revisioned control planes are deployed - with corresponding TSB released Istio images. When you leave tsbVersion under revisions as empty, TSB considers it as the current release version. Having multiple revisions in a single IsolationBoundary helps to upgrade workloads in a particular IsolationBoundary from one tsbVersion to another. See Revisioned to Revisioned Upgrades for more details.

After the installation steps are done, look at deployments, configmaps and webhooks in the istio-system namespace. All resources which are part of the revisioned Istio control plane will be having revisions.name as a suffix in the name. These resources are going to be present for every revision that is configured under isolationBoundaries.

kubectl get deployment -n istio-system | grep istio
# Output
istio-operator 1/1 1 1 2d1h
istio-operator-canary 1/1 1 1 45h
istiod 1/1 1 1 2d1h
istiod-canary 1/1 1 1 45h
kubectl get configmap -n istio-system | grep istio
# Output
istio 2 2d1h
istio-canary 2 45h
istio-sidecar-injector 2 2d1h
istio-sidecar-injector-canary 2 45h

Installation - Multiple IsolationBoundary

If you prefer to introduce environment segmentation or network isolation in a single cluster, you can go with multiple IsolationBoundary configuration.

Install with Helm

Follow the instructions in control plane installation with Helm using following Helm values to enable Istio Isolation Boundaries.

operator:
deployment:
env:
- name: ISTIO_ISOLATION_BOUNDARIES
value: "true"

spec:
managementPlane:
clusterName: <cluster-name-in-tsb>
host: <tsb-address>
port: <tsb-port>
telemetryStore:
elastic:
host: <tsb-address>
port: <tsb-port>
version: <elastic-version>
selfSigned: <is-elastic-use-self-signed-certificate>
components:
xcp:
isolationBoundaries:
- name: dev
revisions:
- name: dev-stable
- name: qa
revisions:
- name: qa-stable

secrets:
clusterServiceAccount:
clusterFQN: organizations/jupiter/clusters/<cluster-name-in-tsb>
JWK: '$JWK'

Install with tctl

If you prefer to use tctl installation, you can use following command to generate cluster operator with Istio Isolation Boundaries enabled

tctl install manifest cluster-operators \
--registry <registry-location> \
--set "operator.deployment.env[0].name=ISTIO_ISOLATION_BOUNDARIES" \
--set "operator.deployment.env[0].value='true'" > clusteroperators.yaml

Then update xcp component in your ControlPlane CR or Helm values with isolationBoundaries:

spec:
...
components:
xcp:
isolationBoundaries:
- name: dev
revisions:
- name: dev-stable
- name: qa
revisions:
- name: qa-stable

After the installation steps are done, look at deployments, configmaps and webhooks in the istio-system namespace. All resources which are part of the revisioned Istio control plane will be having revisions.name as a suffix in the name. These resources are going to be present for every revision that is configured under isolationBoundaries.

kubectl get deployment -n istio-system | grep stable
# Output
istio-operator-dev-stable 1/1 1 1 2d1h
istio-operator-qa-stable 1/1 1 1 45h
istiod-dev-stable 1/1 1 1 2d1h
istiod-qa-stable 1/1 1 1 45h
kubectl get configmap -n istio-system | grep stable
# Output
istio-dev-stable 2 2d1h
istio-qa-stable 2 45h
istio-sidecar-injector-dev-stable 2 2d1h
istio-sidecar-injector-qa-stable 2 45h

Using isolation boundary and revisions

Application deployment

Revisioned namespace labels
  • Default Revision & Sidecar Injection: TSB injects global boundary and default revision as a default settings for backward compatibility, even if it is not configured by the user. You can continue use the istio-injection: enabled label to automatically inject the Istio sidecar proxy into your application pods. This approach means you don't need to make any changes to your existing application namespaces.
  • Transitioning with Existing Labels: If a namespace is already labeled with istio-injection: enabled, adding istio.io/rev: default to it works seamlessly. To use a revision other than default, modify the labels by adding istio.io/rev: <revision-name> and removing istio-injection: enabled.
  • Isolation Boundary & Service Registry: If a namespace doesn’t have the istio-injection label and isolation boundary enabled, that namespace won’t be part of Istio's service registry. As a result, Istio will not be aware of, or manage traffic for, services within that namespace. However, this is not applicable for gateway namespaces, if gateways are installed in a namespace with no revision label, they will be considered to be a part of global boundary.
  • Integrating into Istio's Registry: To make any such namespaces to be part of the Istio's service registry, you must explicitly add the istio.io/rev: <revision-name> label to it. Alternatively, if you specified the revision name as default, you can use the istio-injection: enabled label.
  • Selective Traffic Management with Labels: If you want to keep services in a namespace outside of Istio but need to send traffic to services managed by Istio when isolation boundary is enabled, you can add the following labels to the namespace:
      labels:
    istio.io/rev: default # this namespace is part of service registry and will be visible to mesh
    istio-injection: disabled # this namespace is not injected with Istio sidecar proxy

Workloads can now be deployed in appropriate namespaces with a revision label. The revision label istio.io/rev determines the revisioned control plane for the proxies to connect to, for service discovery and xDS updates. Make sure to configure the workload Namespaces as follows:

apiVersion: v1
kind: Namespace
metadata:
labels:
istio.io/rev: dev-stable
name: dev-bookinfo
---
apiVersion: v1
kind: Namespace
metadata:
labels:
istio.io/rev: qa-stable
name: qa-bookinfo

Application Pods in these namespaces will be injected with the istio-proxy configurations that enable them to connect to their respective revisioned Istio control planes.

VM workload onboarding

By default, workload onboarding will use non-revisioned Istio control plane. To use revisioned control plane, You need to use revisioned link to download Istio sidecar from TSB workload onboarding repository.

Mesh Expansion per IsolationBoundary

By default, if workload onboarding is enabled using meshExpansion configuration as described here, it will use the first enabled revision from the ones specified in isolationBoundaries configuration in the Controlplane CR.

With multiple Isolation Boundary support added from TSB 1.8.0 onwards in Workload Onboarding, Users can configure meshExpansion configurations specific to each IsolationBoundary config in the Controlplane CR.

spec:
...
components:
xcp:
isolationBoundaries:
- name: dev
meshExpansion:
onboarding:
endpoint:
hosts:
- onboarding-endpoint.example
secretName: onboarding-endpoint-tls-cert
localRepository: {}
tokenIssuer:
jwt:
expiration: 3600s
revisions:
- name: dev-stable
istio:
tsbVersion: 1.8.1
- name: qa
revisions:
- name: qa-stable
istio:
tsbVersion: 1.8.0

You also need to update agent configuration in /etc/onboarding-agent/agent.config.yaml in your VM to add revision value.

apiVersion: config.agent.onboarding.tetrate.io/v1alpha1
kind: AgentConfiguration
sidecar:
istio:
revision: dev-stable

Then restart onboarding agent.

systemctl restart onboarding-agent

If you use cloud-init to provision your VM, then you need to add above AgentConfiguration in your cloud-init file. Since file /etc/onboarding-agent/agent.config.yaml might be created in advance, for Debian based OS, you need to pass -o Dpkg::Options::="--force-confold" when installing onboarding agent for non-interactive installation.

sudo apt-get install -y -o Dpkg::Options::="--force-confold" ./onboarding-agent.deb

Workspace configurations

Every workspace can be configured to be part of an isolation boundary by specifying the isolation boundary name as shown below:

apiVersion: api.tsb.tetrate.io/v2
kind: Workspace
metadata:
organization: tetrate
tenant: tetrate
name: dev-ws
spec:
isolationBoundary: dev
namespaceSelector:
names:
- "*/dev-bookinfo"
apiVersion: api.tsb.tetrate.io/v2
kind: Workspace
metadata:
organization: tetrate
tenant: tetrate
name: qa-ws
spec:
isolationBoundary: qa
namespaceSelector:
names:
- "*/qa-bookinfo"
Brownfield Setup

In a brownfield setup, existing Workspaces would not be configured with any particular isolation boundary. In that case, if Istio Isolation Boundaries are enabled and configured, the Workspace will look to be part of an isolation boundary - named "global". If this "global" isolation boundary is not configured in the ControlPlane CR, the Workspace will not be a part of any isolation boundary. Therefore, it is recommended to create a fallback isolation boundary named "global", for Workspaces that do not specify any isolation boundary in their spec.

spec:
...
components:
xcp:
isolationBoundaries:
- name: global
revisions:
- name: default

Gateway deployment

For each gateway (Ingress/Egress/Tier1) resource, if it is part of global boundary and default revision, there are no changes required from the user end. Otherwise, you must set a revision that belongs to the desired isolation boundary in the GW install resource.

For example in your Ingress gateway deployment:

apiVersion: install.tetrate.io/v1alpha1
kind: IngressGateway
metadata:
name: tsb-gateway-dev-bookinfo
namespace: dev-bookinfo
spec:
revision: dev-stable # Revision value

Once applied, this will result in revisioned gateway deployment.

Istio isolation

Having multiple isolation boundaries set up in a single/multiple clusters allow users to run multiple mesh environments that are segregated in terms of service discovery. This means that a Service in one isolation boundary will only be discoverable to clients in the same isolation boundary, and therefore will allow traffic to Services in the same isolation boundary. Services that are separated via isolation boundaries are not be able to discover each other, leading to no inter-boundary traffic.

As a simple example consider the following isolation boundary configuration

...
spec:
...
components:
xcp:
isolationBoundaries:
- name: dev
revisions:
- name: dev-stable
revisions:
- name: dev-testing
- name: qa
revisions:
- name: qa-stable

This could correspond to three separate namespaces dev-bookinfo, dev-bookinfo-testing and qa-bookinfo with the revision labels attached appropriately.

apiVersion: v1
kind: Namespace
metadata:
labels:
istio.io/rev: dev-testing
name: dev-bookinfo-testing
---
apiVersion: v1
kind: Namespace
metadata:
labels:
istio.io/rev: dev-stable
name: dev-bookinfo
---
apiVersion: v1
kind: Namespace
metadata:
labels:
istio.io/rev: qa-stable
name: qa-bookinfo

Note that the namespaces dev-bookinfo and dev-bookinfo-testing are part of the same isolation boundary dev, whereas the namespace qa-bookinfo is part of the isolation boundary qa.

This configuration will have the following affects in the cluster:

  1. Services in namespace dev-bookinfo will be discovered by proxies running in namespaces dev-bookinfo and dev-bookinfo-testing. This is because both namespaces dev-bookinfo and dev-bookinfo-testing are a part of the same isolation boundary.

  2. Services in namespace dev-bookinfo will NOT be discovered by proxies running in namespace qa-bookinfo. This is because namespaces dev-bookinfo and qa-bookinfo are part of different isolation boundaries.

And similarly for other namespaces.

Strict isolation

If a Service from one boundary tries to communicate to a Service in a different boundary, the "client" side proxy treats the outbound traffic to be destined outside the mesh. By default, this traffic is allowed, but can be restricted by using outboundTrafficPolicy in the IstioOperator resource mesh config. By default, the value for this outboundTrafficPolicy is set as

outboundTrafficPolicy:
mode: ALLOW_ANY

In order to restrict cross-isolation-boundary traffic you can set the .outboundTrafficPolicy.mode to REGISTRY_ONLY.

  • First list down the IstioOperator resources in the istio-system namespace.
kubectl get iop -n istio-system | grep xcp-iop
# Output
xcp-iop-dev-stable stable HEALTHY 25h
xcp-iop-dev-testing stable HEALTHY 25h
xcp-iop-qa-stable stable HEALTHY 25h
  • Choose the IstioOperator resource name for which the outboundTrafficPolicy config needs to be set. We will patch the xcp-iop-dev-stable IstioOperator resource with the policy configuration. Edit the Helm values for the TSB control plane above, and following overlay for istio component:
spec:
...
components:
istio:
kubeSpec:
overlays:
- apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
name: xcp-iop-dev-stable
patches:
- path: spec.meshConfig.outboundTrafficPolicy
value:
mode: REGISTRY_ONLY

This can be done for other IstioOperators in a similar fashion, by appending the overlays.

Troubleshooting

  1. Look for ingressdeployment, egressdeployment, tier1deployment resources in the istio-system namespace corresponding to TSB IngressGateway, EgressGateway, Tier1Gateway resources respectively.
kubectl get ingressdeployment -n istio-system
# Output
NAME AGE
tsb-gateway-dev-bookinfo 3h10

If missing, the TSB control plane operator did not reconcile TSB gateway resource to corresponding XCP resource. First, re-verify the revision match between the TSB control plane operator and Gateway resource. Next, the operator logs should give some hints.

  1. Check TSB Config UI page, the UI should indicate red with an error message that the config was not applied/rejected.

  2. Look for corresponding IstioOperator resource in the istio-system namespace. Example:

kubectl get iop -n istio-system | grep dev-stable
# Output
xcp-iop-dev-stable dev-stable HEALTHY 25h
xcpgw-tsb-dev-gateway-bookinfo dev-stable HEALTHY 3h14m

If missing, the xcp-operator-edge logs should give some hints.

  1. If the above two points are OK and the gateway deployment/services are still not getting deployed, or the deployment configurations not as configured in the IstioOperator resource, the Istio operator deployment logs should give some hints.

  2. To troubleshoot service discovery between revisioned namespace workloads, look at the control plane IstioOperator resource in the istio-system namespace. For multiple revisions there will be multiple resources.

kubectl get iop -n istio-system | grep xcp-iop
# Output
xcp-iop-dev-stable stable HEALTHY 25h
xcp-iop-dev-testing stable HEALTHY 25h
xcp-iop-qa-stable stable HEALTHY 25h

Service discovery in namespace workloads is determined by the .spec.meshConfig.discoverySelectors field in the control plane IstioOperator resources. Considering dev-stable and dev-testing revisions share an isolation boundary, the discoverySelectors in both the IstioOperator resources (xcp-iop-dev-stable and xcp-iop-dev-staging) must look like

  discoverySelectors:
- matchLabels:
istio.io/rev: dev-stable
- matchLabels:
istio.io/rev: dev-testing

but since the revision qa-stable is part of a separate isolation boundary the discoverySelectors in xcp-iop-qa-stable would look simply like

  discoverySelectors:
- matchLabels:
istio.io/rev: qa-stable
  1. To further debug service discovery by istio proxies and traffic between services, the following istioctl commands come in handy
istioctl pc endpoints -n <namespace> deploy/<deployment-name>
# Output
ENDPOINT STATUS OUTLIER CHECK CLUSTER
10.20.0.36:8090 HEALTHY OK outbound|80||echo.echo.svc.cluster.local
10.255.20.128:15443 HEALTHY OK outbound|80||echo.tetrate.io
...
127.0.0.1:15000 HEALTHY OK prometheus_stats
127.0.0.1:15020 HEALTHY OK agent
unix://./etc/istio/proxy/XDS HEALTHY OK xds-grpc
unix://./var/run/secrets/workload-spiffe-uds/socket HEALTHY OK sds-grpc

and

istioctl pc routes -n <namespace> deploy/<deployment-name>
# Output
NAME DOMAINS MATCH VIRTUAL SERVICE
80 echo, echo.echo + 1 more... /*
80 echo.tetrate.io, 240.240.0.1 /*
echo-gateway.echo.svc.cluster.local:15021 * /*
...
* /stats/prometheus*
* /healthz/ready*
inbound|7070|| * /*
InboundPassthroughClusterIpv4 * /*
inbound|8090|| * /*