Skip to main content
logoTetrate Service BridgeVersion: 1.9.x

Multicluster Access Control using Identity Propagation

When traffic is forwarded by a gateway, by default it takes on the identity of that gateway. This default behavior allows an administrator to easily configure access controls for all external traffic. For example, an administrator may wish to classify all gateway-sourced traffic as internet-sourced and potentially not trusted, and create access-control rules accordingly.

In a multi-cluster environment, you may wish to perform more fine-grained access control. TSB can preserve the identity of a request through gateway hops. This makes it possible to perform cross-cluster authentication, so that you can:

  • propagate the consumer identity to the remote service it has called
  • configure detailed access control between consumers and services in different TSB-managed clusters
  • apply detailed access control rules to failover targets, so that they are not exposed to too-broad a set of consumers

The examples in this documentation assume you are familiar with TSB concepts, Ingress Gateways, Edge Gateways and EastWest gateways.

GitOps

Following examples use TSB GitOps feature that allow you to apply TSB configurations using kubectl. See Enabling GitOps to enable GitOps in your TSB environment and How GitOps works to understand you can leverage GitOps workflows with TSB.

What you need to know​

Service identities are not propagated through gateway hops by default because when a request goes through the gateway, due to TLS termination at gateways, the request takes the identity of the gateway instead of original source. TSB achieves identity propagation using an internal WASM extension on each gateway hop. This extension checks the validity of the client identity, and then append the clients identity to the requests XFCC header and then forwards the request.

You enable this behavior enabling two flags in ControlPlane CR or Helm values.

  1. enableHttpMeshInternalIdentityPropagation in the xcp component to enable identity propagation.
  2. mountInternalWasmExtensions is enabled by default in the istio component to automatically mount required WASM images on the gateways and workload sidecars.
spec:
  ...
  components:
    xcp:
      centralAuthMode: JWT
      enableHttpMeshInternalIdentityPropagation: true   # (1)
      ...
    istio:
      mountInternalWasmExtensions: true                 # (2)
      ...

Verify that ENABLE_HTTP_MESH_INTERNAL_IDENTITY_PROPAGATION is then enabled in XCP edge:

kubectl get deployment edge -n istio-system -o yaml | grep ENABLE_HTTP_MESH_INTERNAL_IDENTITY_PROPAGATION -A 1

For assistance, check the Troubleshooting instructions at the end of this page.

How to enable IdentityPropagation in a multi cluster setup?​

During upgrades in a multi-cluster setup, IdentityPropagation can be enabled gradually without disrupting live traffic or existing authorization policies. Starting from TSB version 1.9.0, a new property called IdentityMatch has been introduced. This property allows users to enable IdentityPropagation and policy validations incrementally based on originating client's source identity.

When you enable IdentityPropagation, by default IdentityMatch property would be configured as PERMISSIVE for all the existing authZ policies in the cluster, which would allow both peer-identity based validation as well as source-identity based validation if the XFCC header is present in the request.

This ensures that all existing policies will continue to function without any changes required from the user.

Once the upgrade is complete and IdentityPropagation is enabled across all clusters, users can configure IdentityMatch as SOURCE_IDENTITY wherever they want the source-identity of the originating client to be validated.

Use case 1: Propagating Service Identities through Application Edge Gateway and Application Ingress Gateways​

Configure two clusters cluster-1 and cluster-2, sharing the same root of trust. If necessary, follow this guide](https://istio.io/latest/docs/tasks/security/cert-management/plugin-ca-cert/) and the [repo to setup istio root and intermediate certs.

In edge-gw cluster:

  • Create a dedicated cluster for deploying an edge gateway.
  • Configure networkReachability setting to make sure reachability from cluster-1 to edge-gw cluster and from edge-gw cluster to cluster-2 are established.

In cluster-1:

  • create a tenant tenant-1 and its namespace tenant-1-ns, workspace tenant-1-ws and groups.
  • deploy the sleep pod, or a similar text client in the tenant-1-ns.

In cluster-2:

  • create a tenant tenant-2 and its namespace tenant-2-ns, workspace tenant-2-ws and groups.
  • deploy the bookinfo app into tenant-2-ns and an ingress gateway.

Verify that a request from the cluster-1 sleep pod can reach a service in the bookinfo app in cluster-2 via the Edge gateway. For example, you might use a command similar to the one below to invoke curl from the sleep pod against the Edge gateway.

export GATEWAY_IP=34.68.3.192   # use your Edge gateway IP
export POD=`kubectl get pod -n tenant-1 -l app=sleep -o jsonpath='{.items[0].metadata.name}')`
kubectl exec $POD -c sleep -- curl -sS -H "X-B3-Sampled: 1" "http://bookinfo.tetrate.io/api/v1/products" -v --resolve "bookinfo.tetrate.io:80:$GATEWAY_IP"
Output
* Added bookinfo.tetrate.io:80:34.68.3.192 to DNS cache
* Hostname bookinfo.tetrate.io was found in DNS cache
*   Trying 34.68.3.192:80...
* Connected to bookinfo.tetrate.io (34.68.3.192) port 80 (#0)
> GET /api/v1/products HTTP/1.1
> Host: bookinfo.tetrate.io
> User-Agent: curl/7.86.0-DEV
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< content-length: 395
< server: envoy
< date: Fri, 25 Nov 2022 04:05:03 GMT
< x-envoy-upstream-service-time: 30
<
{ [395 bytes data]
[{"id": 0, "title": "The Comedy of Errors", "descriptionHtml": "<a href=\"https://en.wikipedia.org/wiki/The_Comedy_of_Errors\">Wikipedia Summary</a>: The Comedy of Errors is one of <b>William Shakespeare's</b> early plays. It is his shortest and one of his most farcical comedies, with a major part of the humour coming from slapstick and mistaken identity, in addition to puns and word play."}]
* Connection #0 to host bookinfo.tetrate.io left intact

Denying Access at a Workspace level​

Apply a deny rule to deny communications from tenant-1-ws in cluster-1 to tenant-2-ws in cluster-2:

apiVersion: tsb.tetrate.io/v2
kind: WorkspaceSetting
metadata:
  name: tenant-2-wss
  annotations:
    tsb.tetrate.io/organization: tetrate
    tsb.tetrate.io/tenant: tenant-2
    tsb.tetrate.io/workspace: tenant-2-ws
spec:
  defaultSecuritySetting: 
    authenticationSettings:
      trafficMode: REQUIRED
    authorization: 
      mode: RULES
      rules:
        deny:
        - from:
            fqn: organizations/tetrate/tenants/tenant-1/workspaces/tenant-1-ws
          to:
            fqn: organizations/tetrate/tenants/tenant-2/workspaces/tenant-2-ws
      identityMatch: SOURCE_IDENTITY
kubectl apply -f tenant-2-wss.yaml

Test again a request from the cluster-1 sleep pod to the bookinfo app via the Edge gateway. This time, the Edge gateway should deny the request and return an RBAC: access denied message.

Allowing Tenant access and Denying Service access​

You can also allow access at a Tenant level, and then deny access at a service level using ServiceSecuritySetting. This allows for more fine-grained security rules.

First, apply a rule to allow access at a tenant level:

apiVersion: tsb.tetrate.io/v2
kind: TenantSetting
metadata:
  name: tenant-setting
  annotations: 
    tsb.tetrate.io/organization: tetrate
    tsb.tetrate.io/tenant: tenant-2
spec:
  displayName: default-setting
  defaultSecuritySetting:
    authenticationSettings:
      trafficMode: REQUIRED
    authorization:
      mode: RULES
      rules:
        allow:
        - from:
            fqn: organizations/tetrate/tenants/tenant-1
          to:
            fqn: organizations/tetrate/tenants/tenant-2
      identityMatch: SOURCE_IDENTITY
kubectl apply -f tenant-setting.yaml

Verify that the sleep pod in tenant-1 can access the bookinfo products service in tenant-2, as above.

The security Group is defined as follows:

apiVersion: security.tsb.tetrate.io/v2
kind: Group
metadata:
  name: tenant-2-sg
  annotations:
    tsb.tetrate.io/organization: tetrate
    tsb.tetrate.io/tenant: tenant-2
    tsb.tetrate.io/workspace: tenant-2-ws
spec:
  displayName: tenant-2-security-group
  namespaceSelector:
    names:
      - "cluster-2/tenant-2-ns"
  configMode: BRIDGED

We can then define the deny rule using a ServiceSecuritySetting as follows:

apiVersion: security.tsb.tetrate.io/v2
kind: ServiceSecuritySetting
metadata:
  name: productpage-service-ss
  annotations:
    tsb.tetrate.io/organization: tetrate
    tsb.tetrate.io/tenant: tenant-2
    tsb.tetrate.io/workspace: tenant-2-ws
    tsb.tetrate.io/securityGroup: tenant-2-sg
spec:
  service: tenant-2-ns/productpage.tenant-2-ns.svc.cluster.local
  settings:
    authorization: 
      mode: RULES
      rules:
        deny:
        - from:
            fqn: organizations/tetrate/tenants/tenant-1
          to:
            fqn: organizations/tetrate/tenants/tenant-2/workspaces/tenant-2-ws/securitygroups/tenant-2-sg
      identityMatch: SOURCE_IDENTITY

Use case 2 - Propagating Service Identities in EastWest gateway failover​

Review the documentation for EastWest failover before you proceed. In this use case, we'll propagate the source identity when a service fails-over to a remote instance.

  • In cluster-1, create the namespaces client-ns belonging to tenant Client, and bookinfo-ns belonging to tenant Bookinfo. Deploy the bookinfo and bookinfo-gateway services into the bookinfo-ns.
  • In cluster-2, create the namespace bookinfo-nsbelonging to tenant Bookinfo. Deploy the bookinfo and bookinfo-gateway services into the bookinfo-ns. - In TSB, configure bookinfo-ns/bookinfo-gateway for EW failover with defaultEastWestGatewaySettings:
apiVersion: tsb.tetrate.io/v2
kind: WorkspaceSetting
metadata:
  name: bookinfo-ws-settings
  annotations:
    tsb.tetrate.io/organization: tetrate
    tsb.tetrate.io/tenant: Bookinfo
    tsb.tetrate.io/workspace: Bookinfo-ws
spec:
  defaultEastWestGatewaySettings:
    - workloadSelector:
        namespace: bookinfo-ns
        labels:
          app: bookinfo-gateway

Apply the following allow rule to permit communications from clients in the Client tenant to services in the Bookinfo tenant:

apiVersion: tsb.tetrate.io/v2
kind: TenantSetting
metadata:
  name: default-setting
  annotations: 
    tsb.tetrate.io/organization: tetrate
    tsb.tetrate.io/tenant: Bookinfo
spec:
  displayName: default-setting
  defaultSecuritySetting:
    authenticationSettings:
      trafficMode: REQUIRED
    authorization:
      mode: RULES
      rules:
        allow:
        - from:
            fqn: organizations/tetrate/tenants/Client
          to:
            fqn: organizations/tetrate/tenants/Bookinfo
        - from:
            fqn: organizations/tetrate/tenants/Bookinfo
          to:
            fqn: organizations/tetrate/tenants/Bookinfo
      identityMatch: SOURCE_IDENTITY

Verify that a client in Client can access the bookinfo services:

kubectl exec deployment/sleep -n client-ns -c sleep -- curl -s -H "X-B3-Sampled: 1" http://productpage.bookinfo-ns:9080/api/v1/products -v
kubectl exec deployment/sleep -n client-ns -c sleep -- curl -s -H "X-B3-Sampled: 1" http://details.bookinfo-ns:9080/details/1 -v
kubectl exec deployment/sleep -n client-ns -c sleep -- curl -s -H "X-B3-Sampled: 1" http://reviews.bookinfo-ns:9080/reviews/1 -v
kubectl exec deployment/sleep -n client-ns -c sleep -- curl -s -H "X-B3-Sampled: 1" http://ratings.bookinfo-ns:9080/ratings/1 -v

Now apply a deny rule to prevent direct access to the details, reviews and ratings services from the Clients tenant. For example, to deny access to details:

apiVersion: security.tsb.tetrate.io/v2
kind: ServiceSecuritySetting
metadata:
  name: details-service-ss
  annotations:
    tsb.tetrate.io/organization: tetrate
    tsb.tetrate.io/tenant: Bookinfo
    tsb.tetrate.io/workspace: bookinfo-ws
    tsb.tetrate.io/securityGroup: bookinfo-sg
spec:
  service: bookinfo-ns/details.bookinfo-ns.svc.cluster.local
  settings:
    authorization: 
      mode: RULES
      rules:
        deny:
        - from:
            fqn: organizations/tetrate/tenants/Client
          to:
            fqn: organizations/tetrate/tenants/Bookinfo/workspaces/bookinfo-ws/securitygroups/bookinfo-sg
      identityMatch: SOURCE_IDENTITY

Repeat the client tests, verifying that a client can access the front-end products service, but none of the back-end details, reviews and ratings services.

Finally, scale down the details, reviews and ratings services to zero. This will provoke a failover.

  • Without service identity propagation, the client requests to the details, reviews and ratings services would now succeed because they would take on the identity of the gateway in cluster-2 (the Bookinfo tenant).
  • With service identity propagation, the client requests to the details, reviews and ratings services continue to be denied as the identity of the originator is forwarded through gateway hops.

Troubleshooting​

  1. Identity propagation is enabled from TSB ControlPlane CR by adding enableHttpMeshInternalIdentityPropagation to the xcp component:
spec:
  ...
  components:
    xcp:
      centralAuthMode: JWT
      configProtection: {}
      enableHttpMeshInternalIdentityPropagation: true
      kubeSpec:
        ..
        ..

Verify that ENABLE_HTTP_MESH_INTERNAL_IDENTITY_PROPAGATION is enabled in XCP edge:

kubectl get deployment edge -n istio-system -o yaml | grep ENABLE_HTTP_MESH_INTERNAL_IDENTITY_PROPAGATION -A 1
  1. From TSB version 1.7.0 onwards, Internal WASM extensions can be directly mounted in the Sidecar, Ingress and Egress Gateway pods, instead of being downloaded from the image registries. Make sure mountInternalWasmExtensions is enabled by edgexcp
Controlplane CR
  spec:
    components:
      istio:
        mountInternalWasmExtensions: true
kubectl get edgexcp -n istio-system -o yaml | grep mount

  xcp.tetrate.io/mount-internal-wasm-plugins: "true"
  mountPath: "/wasm-plugins/"
  mountPath: "/wasm-plugins/"
  1. Verify that the Istio wasmplugin extension has been successfully installed by the XCP operator in the ControlPlane cluster.
kubectl get wasmplugins.extensions.istio.io -A
Output
NAMESPACE      NAME                       AGE
istio-system   xfcc-extractor-internal    5m
istio-system   xfcc-hasher-internal       5m
istio-system   xfcc-validator-internal    5m
  1. Verify that the WASM extensions are configured with the url as file:// for referencing .wasm module files present locally within the proxy container.
kubectl get wasmplugins.extensions.istio.io xfcc-extractor-internal -n istio-system -o yaml
Output
apiVersion: extensions.istio.io/v1alpha1
kind: WasmPlugin
metadata:
  annotations:
    xcp.tetrate.io/contentHash: a2bc43f364e0d928e879b32419c8a7bb
    xcp.tetrate.io/created-by-installer: "true"
  creationTimestamp: "2023-12-04T08:18:33Z"
  generation: 1
  labels:
    install.xcp.tetrate.io/owner-kind: EdgeXcp
    install.xcp.tetrate.io/owner-name: edge-xcp
    install.xcp.tetrate.io/owner-version: v1alpha1
  name: xfcc-extractor-internal
  namespace: istio-system
  ownerReferences:
  - apiVersion: install.xcp.tetrate.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: EdgeXcp
    name: edge-xcp
    uid: 3a6b33b2-71e9-424d-a4ae-16bd762e89a1
  resourceVersion: "1028015"
  uid: b03264fb-3e14-4550-8128-29a30adc48b4
spec:
  phase: AUTHZ
  pluginConfig:
    mode: extractor
  pluginName: xfcc-extractor-internal
  priority: 10
  selector:
    matchLabels:
      xcp.tetrate.io/mount-internal-wasm-plugins: "true"
  url: file:///wasm-plugins/xcp-guard.wasm
  1. Verify inject-wasm-extension init container logs which fetches and mount the internal WASM plugins to the gateways and sidecars.
k logs -f deploy/t1-gateway -n tier1 -c inject-wasm-extensions
'/wasmfetcher/plugins/./xcp-guard.wasm' -> '/wasm-plugins/./xcp-guard.wasm'
'/wasmfetcher/plugins/./coraza-proxy-wasm.wasm' -> '/wasm-plugins/./coraza-proxy-wasm.wasm'
'/wasmfetcher/plugins/.' -> '/wasm-plugins/.'
  1. Verify XCP internal WASM plugins are locally mounted and Envoy is using local filename.
k exec -it deployment/t1-gateway -n tier1 -c istio-proxy  -- pilot-agent request GET config_dump > config.json
{
  "version_info": "2023-11-27T10:38:22Z/31",
  "ecds_filter": {
    "@type": "type.googleapis.com/envoy.config.core.v3.TypedExtensionConfig",
    "name": "istio-system.xfcc-hasher-internal",
    "typed_config": {
    "@type": "type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm",
    "config": {
      "name": "istio-system.xfcc-hasher-internal",
      "root_id": "xfcc-hasher-internal",
      "vm_config": {
      "runtime": "envoy.wasm.runtime.v8",
      "code": {
        "local": {
        "filename": "/wasm-plugins/xcp-guard.wasm"
        }
      },
      "environment_variables": {
        "key_values": {
        "ISTIO_META_WASM_PLUGIN_RESOURCE_VERSION": "59082"
        }
      }
      },
      "configuration": {
      "@type": "type.googleapis.com/google.protobuf.StringValue",
      "value": "{\"mode\":\"hasher\"}"
      }
    }
    }
  },
  "last_updated": "2023-11-27T10:38:22.413Z"
  },