Skip to main content
logoTetrate Service BridgeVersion: 1.11.x

Isolation Boundary Installation

Before you begin​

Enabling or Upgrading to Isolation boundaries is a complex process. Please make sure to review this entire document to correctly plan the install or upgrade.

Important Note - istio-gateway namespace

During the upgrade process, the tsb-operator will delete the existing istio-gateway namespace. In some situations, this delete will hang indefinitely due to stuck finalizers that prevent the delete from completing. You may need to edit the namespace and delete any finalizers in order to allow the deletion to complete.

Do also refer to the Troubleshooting section at the end of this document.

Installation - Single IsolationBoundary​

If you are mainly looking for seamless control plane upgrade using revisions as the primary use-case, you can go with single IsolationBoundary configuration.

For a fresh installation, you can follow the standard steps to onboard a control plane cluster using tctl or helm along with the following changes:

  1. Enable isolation boundary in TSB control plane operator by setting ISTIO_ISOLATION_BOUNDARIES to true
  2. Add isolation boundaries definition in ControlPlane CR or control plane Helm values.

In the following example, you will use Helm to onboard a cluster with a single IsolationBoundary enabled.

Install with Helm​

Follow the instructions in control plane installation with Helm using following Helm values to enable Istio Isolation Boundaries.

operator:
  deployment:
    env:
      - name: ISTIO_ISOLATION_BOUNDARIES
        value: "true"

spec:
  managementPlane:
    clusterName: <cluster-name-in-tsb>
    host: <tsb-address>
    port: <tsb-port>
  telemetryStore:
    elastic:
      host: <tsb-address>
      port: <tsb-port>
      version: <elastic-version>
      selfSigned: <is-elastic-use-self-signed-certificate>
  components:
    xcp:
      isolationBoundaries:
      - name: global
        revisions:
        - name: default
        - name: canary

secrets:
  clusterServiceAccount:
    clusterFQN: organizations/jupiter/clusters/<cluster-name-in-tsb>
    JWK: '$JWK'

Install with tctl​

If you prefer to use tctl installation, you can use following command to generate cluster operator with Istio Isolation Boundaries enabled

tctl install manifest cluster-operators \
  --registry <registry-location> \
  --set "operator.deployment.env[0].name=ISTIO_ISOLATION_BOUNDARIES" \
  --set "operator.deployment.env[0].value='true'" > clusteroperators.yaml

Then update xcp component in your ControlPlane CR or Helm values with isolationBoundaries:

spec:
  ...
  components:
    xcp:
      isolationBoundaries:
      - name: global
        revisions:
        - name: default
        - name: canary

With these configurations, two revisioned control planes are deployed - with corresponding TSB released Istio images. When you leave tsbVersion under revisions as empty, TSB considers it as the current release version. Having multiple revisions in a single IsolationBoundary helps to upgrade workloads in a particular IsolationBoundary from one tsbVersion to another. See Revisioned to Revisioned Upgrades for more details.

After the installation steps are done, look at deployments, configmaps and webhooks in the istio-system namespace. All resources which are part of the revisioned Istio control plane will be having revisions.name as a suffix in the name. These resources are going to be present for every revision that is configured under isolationBoundaries.

kubectl get deployment -n istio-system | grep istio
# Output
istio-operator             1/1     1            1           2d1h
istio-operator-canary      1/1     1            1           45h
istiod                     1/1     1            1           2d1h
istiod-canary              1/1     1            1           45h
kubectl get configmap -n istio-system | grep istio
# Output
istio                           2      2d1h
istio-canary                    2      45h
istio-sidecar-injector          2      2d1h
istio-sidecar-injector-canary   2      45h

Installation - Multiple IsolationBoundary​

If you prefer to introduce environment segmentation or network isolation in a single cluster, you can go with multiple IsolationBoundary configuration.

Install with Helm​

Follow the instructions in control plane installation with Helm using following Helm values to enable Istio Isolation Boundaries.

operator:
  deployment:
    env:
      - name: ISTIO_ISOLATION_BOUNDARIES
        value: "true"

spec:
  managementPlane:
    clusterName: <cluster-name-in-tsb>
    host: <tsb-address>
    port: <tsb-port>
  telemetryStore:
    elastic:
      host: <tsb-address>
      port: <tsb-port>
      version: <elastic-version>
      selfSigned: <is-elastic-use-self-signed-certificate>
  components:
    xcp:
      isolationBoundaries:
      - name: dev
        revisions:
        - name: dev-stable
      - name: qa
        revisions:
        - name: qa-stable

secrets:
  clusterServiceAccount:
    clusterFQN: organizations/jupiter/clusters/<cluster-name-in-tsb>
    JWK: '$JWK'

Install with tctl​

If you prefer to use tctl installation, you can use following command to generate cluster operator with Istio Isolation Boundaries enabled

tctl install manifest cluster-operators \
  --registry <registry-location> \
  --set "operator.deployment.env[0].name=ISTIO_ISOLATION_BOUNDARIES" \
  --set "operator.deployment.env[0].value='true'" > clusteroperators.yaml

Then update xcp component in your ControlPlane CR or Helm values with isolationBoundaries:

spec:
  ...
  components:
    xcp:
      isolationBoundaries:
      - name: dev
        revisions:
        - name: dev-stable
      - name: qa
        revisions:
        - name: qa-stable

After the installation steps are done, look at deployments, configmaps and webhooks in the istio-system namespace. All resources which are part of the revisioned Istio control plane will be having revisions.name as a suffix in the name. These resources are going to be present for every revision that is configured under isolationBoundaries.

kubectl get deployment -n istio-system | grep stable
# Output
istio-operator-dev-stable             1/1     1            1           2d1h
istio-operator-qa-stable              1/1     1            1           45h
istiod-dev-stable                     1/1     1            1           2d1h
istiod-qa-stable                      1/1     1            1           45h
kubectl get configmap -n istio-system | grep stable
# Output
istio-dev-stable                      2      2d1h
istio-qa-stable                       2      45h
istio-sidecar-injector-dev-stable     2      2d1h
istio-sidecar-injector-qa-stable      2      45h

Using isolation boundary and revisions​

Application deployment​

Revisioned namespace labels
  • Default Revision & Sidecar Injection: TSB injects global boundary and default revision as a default settings for backward compatibility, even if it is not configured by the user. You can continue use the istio-injection: enabled label to automatically inject the Istio sidecar proxy into your application pods. This approach means you don't need to make any changes to your existing application namespaces.
  • Transitioning with Existing Labels: If a namespace is already labeled with istio-injection: enabled, adding istio.io/rev: default to it works seamlessly. To use a revision other than default, modify the labels by adding istio.io/rev: <revision-name> and removing istio-injection: enabled.
  • Isolation Boundary & Service Registry: If a namespace doesn’t have the istio-injection label and isolation boundary enabled, that namespace won’t be part of Istio's service registry. As a result, Istio will not be aware of, or manage traffic for, services within that namespace. However, this is not applicable for gateway namespaces, if gateways are installed in a namespace with no revision label, they will be considered to be a part of global boundary.
  • Integrating into Istio's Registry: To make any such namespaces to be part of the Istio's service registry, you must explicitly add the istio.io/rev: <revision-name> label to it. Alternatively, if you specified the revision name as default, you can use the istio-injection: enabled label.
  • Selective Traffic Management with Labels: If you want to keep services in a namespace outside of Istio but need to send traffic to services managed by Istio when isolation boundary is enabled, you can add the following labels to the namespace: 
      labels:
        istio.io/rev: default          # this namespace is part of service registry and will be visible to mesh
        istio-injection: disabled      # this namespace is not injected with Istio sidecar proxy

Workloads can now be deployed in appropriate namespaces with a revision label. The revision label istio.io/rev determines the revisioned control plane for the proxies to connect to, for service discovery and xDS updates. Make sure to configure the workload Namespaces as follows:

apiVersion: v1
kind: Namespace
metadata:
  labels:
    istio.io/rev: dev-stable
  name: dev-bookinfo
---
apiVersion: v1
kind: Namespace
metadata:
  labels:
    istio.io/rev: qa-stable
  name: qa-bookinfo

Application Pods in these namespaces will be injected with the istio-proxy configurations that enable them to connect to their respective revisioned Istio control planes.

VM workload onboarding​

By default, workload onboarding will use non-revisioned Istio control plane. To use revisioned control plane, You need to use revisioned link to download Istio sidecar from TSB workload onboarding repository.

Mesh Expansion per IsolationBoundary

By default, if workload onboarding is enabled using meshExpansion configuration as described here, it will use the first enabled revision from the ones specified in isolationBoundaries configuration in the Controlplane CR.

With multiple Isolation Boundary support added from TSB 1.8.0 onwards in Workload Onboarding, Users can configure meshExpansion configurations specific to each IsolationBoundary config in the Controlplane CR.

spec:
  ...
  components:
    xcp:
      isolationBoundaries:
      - name: dev
        meshExpansion:
          onboarding:
            endpoint:
              hosts:
              - onboarding-endpoint.example
              secretName: onboarding-endpoint-tls-cert
            localRepository: {}
            tokenIssuer:
              jwt:
                expiration: 3600s
        revisions:
        - name: dev-stable
          istio:
            tsbVersion: 1.8.1
      - name: qa
        revisions:
        - name: qa-stable
          istio:
            tsbVersion: 1.8.0

You also need to update agent configuration in /etc/onboarding-agent/agent.config.yaml in your VM to add revision value.

apiVersion: config.agent.onboarding.tetrate.io/v1alpha1
kind: AgentConfiguration
sidecar:
  istio:
    revision: dev-stable

Then restart onboarding agent.

systemctl restart onboarding-agent

If you use cloud-init to provision your VM, then you need to add above AgentConfiguration in your cloud-init file. Since file /etc/onboarding-agent/agent.config.yaml might be created in advance, for Debian based OS, you need to pass -o Dpkg::Options::="--force-confold" when installing onboarding agent for non-interactive installation.

sudo apt-get install -y -o Dpkg::Options::="--force-confold" ./onboarding-agent.deb

Workspace configurations​

Every workspace can be configured to be part of an isolation boundary by specifying the isolation boundary name as shown below:

apiVersion: api.tsb.tetrate.io/v2
kind: Workspace
metadata:
  organization: tetrate
  tenant: tetrate
  name: dev-ws
spec:
  isolationBoundary: dev
  namespaceSelector:
    names:
      - "*/dev-bookinfo"
apiVersion: api.tsb.tetrate.io/v2
kind: Workspace
metadata:
  organization: tetrate
  tenant: tetrate
  name: qa-ws
spec:
  isolationBoundary: qa
  namespaceSelector:
    names:
      - "*/qa-bookinfo"
Brownfield Setup

In a brownfield setup, existing Workspaces would not be configured with any particular isolation boundary. In that case, if Istio Isolation Boundaries are enabled and configured, the Workspace will look to be part of an isolation boundary - named "global". If this "global" isolation boundary is not configured in the ControlPlane CR, the Workspace will not be a part of any isolation boundary. Therefore, it is recommended to create a fallback isolation boundary named "global", for Workspaces that do not specify any isolation boundary in their spec.

spec:
  ...
  components:
    xcp:
      isolationBoundaries:
      - name: global
        revisions:
          - name: default

Gateway deployment​

For each gateway (Ingress/Egress/Tier1) resource, if it is part of global boundary and default revision, there are no changes required from the user end. Otherwise, you must set a revision that belongs to the desired isolation boundary in the GW install resource.

For example in your Ingress gateway deployment:

apiVersion: install.tetrate.io/v1alpha1
kind: IngressGateway
metadata:
  name: tsb-gateway-dev-bookinfo
  namespace: dev-bookinfo
spec:
  revision: dev-stable         # Revision value

Once applied, this will result in revisioned gateway deployment.

Istio isolation​

Having multiple isolation boundaries set up in a single/multiple clusters allow users to run multiple mesh environments that are segregated in terms of service discovery. This means that a Service in one isolation boundary will only be discoverable to clients in the same isolation boundary, and therefore will allow traffic to Services in the same isolation boundary. Services that are separated via isolation boundaries are not be able to discover each other, leading to no inter-boundary traffic.

As a simple example consider the following isolation boundary configuration

...
spec:
  ...
  components:
    xcp:
      isolationBoundaries:
      - name: dev
        revisions:
        - name: dev-stable
        revisions:
        - name: dev-testing
      - name: qa
        revisions:
        - name: qa-stable

This could correspond to three separate namespaces dev-bookinfo, dev-bookinfo-testing and qa-bookinfo with the revision labels attached appropriately.

apiVersion: v1
kind: Namespace
metadata:
  labels:
    istio.io/rev: dev-testing
  name: dev-bookinfo-testing
---
apiVersion: v1
kind: Namespace
metadata:
  labels:
    istio.io/rev: dev-stable
  name: dev-bookinfo
---
apiVersion: v1
kind: Namespace
metadata:
  labels:
    istio.io/rev: qa-stable
  name: qa-bookinfo

Note that the namespaces dev-bookinfo and dev-bookinfo-testing are part of the same isolation boundary dev, whereas the namespace qa-bookinfo is part of the isolation boundary qa.

This configuration will have the following affects in the cluster:

  1. Services in namespace dev-bookinfo will be discovered by proxies running in namespaces dev-bookinfo and dev-bookinfo-testing. This is because both namespaces dev-bookinfo and dev-bookinfo-testing are a part of the same isolation boundary.

  2. Services in namespace dev-bookinfo will NOT be discovered by proxies running in namespace qa-bookinfo. This is because namespaces dev-bookinfo and qa-bookinfo are part of different isolation boundaries.

And similarly for other namespaces.

Strict isolation​

If a Service from one boundary tries to communicate to a Service in a different boundary, the "client" side proxy treats the outbound traffic to be destined outside the mesh. By default, this traffic is allowed, but can be restricted by using outboundTrafficPolicy in the IstioOperator resource mesh config. By default, the value for this outboundTrafficPolicy is set as

outboundTrafficPolicy:
  mode: ALLOW_ANY

In order to restrict cross-isolation-boundary traffic you can set the .outboundTrafficPolicy.mode to REGISTRY_ONLY.

  • First list down the IstioOperator resources in the istio-system namespace.
kubectl get iop -n istio-system | grep xcp-iop
# Output
xcp-iop-dev-stable               stable   HEALTHY     25h
xcp-iop-dev-testing              stable   HEALTHY     25h
xcp-iop-qa-stable                stable   HEALTHY     25h
  • Choose the IstioOperator resource name for which the outboundTrafficPolicy config needs to be set. We will patch the xcp-iop-dev-stable IstioOperator resource with the policy configuration. Edit the Helm values for the TSB control plane above, and following overlay for istio component:
spec:
  ...
  components:
    istio:
      kubeSpec:
        overlays:
        - apiVersion: install.istio.io/v1alpha1
          kind: IstioOperator
          name: xcp-iop-dev-stable
          patches:
          - path: spec.meshConfig.outboundTrafficPolicy
            value:
              mode: REGISTRY_ONLY

This can be done for other IstioOperators in a similar fashion, by appending the overlays.

Troubleshooting​

  1. Look for ingressdeployment, egressdeployment, tier1deployment resources in the istio-system namespace corresponding to TSB IngressGateway, EgressGateway, Tier1Gateway resources respectively.
kubectl get ingressdeployment -n istio-system
# Output
NAME                       AGE
tsb-gateway-dev-bookinfo   3h10

If missing, the TSB control plane operator did not reconcile TSB gateway resource to corresponding XCP resource. First, re-verify the revision match between the TSB control plane operator and Gateway resource. Next, the operator logs should give some hints.

  1. Check TSB Config UI page, the UI should indicate red with an error message that the config was not applied/rejected.

  2. Look for corresponding IstioOperator resource in the istio-system namespace. Example:

kubectl get iop -n istio-system | grep dev-stable
# Output
xcp-iop-dev-stable               dev-stable   HEALTHY     25h
xcpgw-tsb-dev-gateway-bookinfo   dev-stable   HEALTHY     3h14m

If missing, the xcp-operator-edge logs should give some hints.

  1. If the above two points are OK and the gateway deployment/services are still not getting deployed, or the deployment configurations not as configured in the IstioOperator resource, the Istio operator deployment logs should give some hints.

  2. To troubleshoot service discovery between revisioned namespace workloads, look at the control plane IstioOperator resource in the istio-system namespace. For multiple revisions there will be multiple resources.

kubectl get iop -n istio-system | grep xcp-iop
# Output
xcp-iop-dev-stable               stable   HEALTHY     25h
xcp-iop-dev-testing              stable   HEALTHY     25h
xcp-iop-qa-stable                stable   HEALTHY     25h

Service discovery in namespace workloads is determined by the .spec.meshConfig.discoverySelectors field in the control plane IstioOperator resources. Considering dev-stable and dev-testing revisions share an isolation boundary, the discoverySelectors in both the IstioOperator resources (xcp-iop-dev-stable and xcp-iop-dev-staging) must look like

  discoverySelectors:
  - matchLabels:
      istio.io/rev: dev-stable
  - matchLabels:
      istio.io/rev: dev-testing

but since the revision qa-stable is part of a separate isolation boundary the discoverySelectors in xcp-iop-qa-stable would look simply like

  discoverySelectors:
    - matchLabels:
        istio.io/rev: qa-stable
  1. To further debug service discovery by istio proxies and traffic between services, the following istioctl commands come in handy
istioctl pc endpoints -n <namespace> deploy/<deployment-name>
# Output
ENDPOINT                                                STATUS      OUTLIER CHECK     CLUSTER
10.20.0.36:8090                                         HEALTHY     OK                outbound|80||echo.echo.svc.cluster.local
10.255.20.128:15443                                     HEALTHY     OK                outbound|80||echo.tetrate.io
...
127.0.0.1:15000                                         HEALTHY     OK                prometheus_stats
127.0.0.1:15020                                         HEALTHY     OK                agent
unix://./etc/istio/proxy/XDS                            HEALTHY     OK                xds-grpc
unix://./var/run/secrets/workload-spiffe-uds/socket     HEALTHY     OK                sds-grpc

and

istioctl pc routes -n <namespace> deploy/<deployment-name>
# Output
NAME                                                        DOMAINS                                                         MATCH                  VIRTUAL SERVICE
80                                                          echo, echo.echo + 1 more...                                     /*
80                                                          echo.tetrate.io, 240.240.0.1                                    /*
echo-gateway.echo.svc.cluster.local:15021                   *                                                               /*
...
                                                            *                                                               /stats/prometheus*
                                                            *                                                               /healthz/ready*
inbound|7070||                                              *                                                               /*
InboundPassthroughClusterIpv4                               *                                                               /*
inbound|8090||                                              *                                                               /*