Shared Gateway

In a service mesh deployment, the recommended architecture is to deploy separate Ingress and Egress gateways per namespace or Workspace. Gateways are lightweight, and this approach simplifies configuration and minimizes the impact of noisy neighbours and failures.

In some cases, particularly when there are large numbers of namespaces, a Shared Gateway approach is preferred.

Shared Ingress Gateway, publishing services from multiple application teams

Shared Gateway Responsibilities

Platform teams and Application teams both have a role in configuring this gateway to publish applications. The Platform and Application responsibilities are separated as follows:

Platform Team

Platform teams are typically responsible for managing the infrastructure and common services used by multiple application teams. Their responsibilities regarding gateway include:

• Infrastructure Provisioning: Platform teams set up and configure the underlying kubernetes resources ( like service and deployment objects) that make up the shared gateway.

• Scaling & Monitoring: They manage the gateway's scalability, performance, and availability, ensuring it can handle traffic for all applications in the mesh.

• Security Controls: Platform teams handle the setup of security policies like TLS termination, mTLS between services, and ensuring proper certificates are in place for the gateway.

• Traffic Routing Logic: Platform teams define high-level routing rules and configurations, such as global load balancing policies and route traffic for different tenants or environments (like dev, staging, production).

• Global Failover & Resilience: The platform team configures strategies for global failover and resiliency, including cross-zone or cross-region redundancy where applicable.

Application Team

Each Application team, on the other hand, focuses on their specific services and how they interact with the gateway. Their responsibilities include:

• Gateway Routing Configurations: Application teams are responsible for defining how their services should be exposed and routed via the gateway. This often involves configuring routing based on path, host headers, or weights for canary deployments.

• Service-Specific Traffic Management: Teams configure traffic splitting, fault injection, retries, or circuit-breaking rules for their specific services.

• Application-Level Security: Application teams configure service-specific mTLS policies, access policies (using AuthorizationPolicy resources), or implement JWT-based authentication if required for their services.

• Observability: While platform teams maintain the infrastructure-level observability stack (e.g., Prometheus and Grafana), application teams leverage service-level metrics, traces, and logs for their specific applications. They are responsible for investigating issues related to their own service traffic.

In essence, platform teams are responsible for building and maintaining the shared infrastructure, while application teams configure how their specific services leverage that infrastructure. Both teams need to collaborate closely, especially for troubleshooting, performance tuning, and policy enforcement

Two Approaches to Create a Shared Gateway in TSB

When you configure a shared gateway, there's a fundamental decision you need to make: who manages the shared gateway infra resources ( k8s services and deployments), and where does the per-application gateway configuration live?

TSB supports the following 2 approaches, one using TSB BRIDGED mode APIs (preferred) and the other using TSB DIRECT mode Istio resources.

Approach 1: Utilizing TSB Bridged Mode APIs

The Platform team deploys a shared Gateway instance on the Kubernetes cluster, and creates dedicated TSB Tenant, Workspace, and GatewayGroup resources. Each Application team then deploys Gateway resources for their applications into their own namespaces. Their Gateway resources reference the platform team's TSB resources, and TSB safely generates the appropriate underlying configuration to configure the Gateway instance.

Shared Ingress Gateway, TSB Config View

This approach is preferred because it is simpler and safer; TSB validates all configuration and ensures that the Gateway is correctly configured in this multi-tenant use case. It requires a dedicated Platform Team function in TSB to manage the related TSB resources, and the Platform Team will need to explicitly grant each application team access to the observability metrics so that they can monitor and troubleshoot their applications.

Approach 2: Utilizing TSB Direct Mode APIs

This approach is a fallback if you want manual control over the Gateway instance's configuration, and you find that the segmentation, validation and safety protections that the TSB Bridged Mode approach are too restrictive for your use case.

The Platform team deploys a shared Gateway instance on the Kubernetes cluster. Each Application Team creates their own high-level TSB configuration (Tenant, Workspace, and GatewayGroup), and then use Istio APIs (Gateway, VirtualService) to directly configure the shared Gateway.

This approach is not encouraged because it provides limited segmentation, validation and safety protections for each Application Team's configuration. For example, there is no protection to prevent one Application Team from mis-configuring their Gateway settings and claiming traffic they should not be able to, or interrupting service for other Application teams.

Approach 1: Utilizing TSB Bridged Mode APIs

In this setup, the Platform team is responsible for creating and managing the shared gateway installation resources needed to provision infrastructure components such as internal and external load balancers, NodePort services, and ClusterIP services. Meanwhile, the Application team manages their individual gateway configurations along with other deployment manifests in their source repositories, applying these resources to their application namespaces using GitOps. The key difference is that the Application team will utilize the same Tenant, Workspace, and GatewayGroup created by the Platform team to configure the gateway, but these configurations will be applied to their application namespaces rather than the shared-gateway namespace owned by the Platform team.

Application Deployment

Consider you have 2 application teams named marketing and payment shares a common k8s cluster. We will be deploying httpbin app on both the namespaces for demo purpose. Follow the instructions in this document to deploy httpbin service.

TSB Setup - Platform Team

Create a dedicated Tenant for platform team to configure shared gateway.

apiVersion: tsb.tetrate.io/v2
kind: Tenant
metadata:
  name: platform
  annotations:
    tsb.tetrate.io/organization: tetrate
spec:
  displayName: Platform

Create a Workspace to onboard shared-ingress namespace into TSB.

apiVersion: tsb.tetrate.io/v2
kind: Workspace
metadata:
  name: shared-ingress-ws
  annotations: 
    tsb.tetrate.io/organization: tetrate
    tsb.tetrate.io/tenant: platform
spec:
  namespaceSelector:
    names:
      - "*/shared-ingress"
  displayName: shared-ingress-ws

Create a GatewayGroup for the shared-ingress

shared-ingress-gg.yaml

apiVersion: gateway.tsb.tetrate.io/v2
kind: Group
metadata:
  name: shared-ingress-gg
  namespace: shared-ingress
  annotations:
    tsb.tetrate.io/organization: tetrate
    tsb.tetrate.io/tenant: platform
    tsb.tetrate.io/workspace: shared-ingress-ws
spec:
  displayName: Shared Ingress Gateway Group
  namespaceSelector:
    names:
      - "*/shared-ingress"
  configMode: BRIDGED

Deploy Shared Gateway Instance - Platform Team

This resource is responsible for creating Kubernetes resources like Services of type LoadBalancer,NodePort , k8s Deployments, HPA etc for Gateway workloads.

Apply this on targetted kubernetes clusters using kubectl as this is not a TSB API resource.

shared-gw-install.yaml

apiVersion: install.tetrate.io/v1alpha1
kind: Gateway
metadata:
  name: shared-ingress-gateway
  namespace: shared-ingress
spec:
  type: INGRESS
  kubeSpec:
    service:
      type: LoadBalancer

Configure Ingress Gateway - Application Team

Application teams belonging to different tenants, such as marketing and payment, will create their own ingress gateway configurations and publish them within their respective application namespaces.

TSB Setup - Marketing

Onboard application namespaces into TSB by creating a Tenant & Workspace.

---
apiVersion: tsb.tetrate.io/v2
kind: Tenant
metadata:
  name: marketing
  annotations:
    tsb.tetrate.io/organization: tetrate
spec:
  displayName: Marketing
---
apiVersion: tsb.tetrate.io/v2
kind: Workspace
metadata:
  name: marketing-ws
  annotations:
    tsb.tetrate.io/organization: tetrate
    tsb.tetrate.io/tenant: marketing
spec:
  namespaceSelector:
    names:
      - "*/marketing"
  displayName: marketing-ws

Configure the Ingress Gateway within the same Tenant, Workspace, and GatewayGroup established by the Platform team, but apply the configuration to the marketing namespace using GitOps.

configure shared gateway workload selector

Make sure workload selector is configured as shared-ingress-gateway in the gateway configuration to refer to the shared gateway instance created by platform team.

marketing-ingress-gw.yaml

apiVersion: gateway.tsb.tetrate.io/v2
kind: Gateway
metadata:
  annotations:
    tsb.tetrate.io/organization: tetrate
    tsb.tetrate.io/tenant: platform
    tsb.tetrate.io/workspace: shared-ingress-ws
    tsb.tetrate.io/gatewayGroup: shared-ingress-gg
  name: marketing-ingress-gateway
spec:
  workloadSelector:
    namespace: shared-ingress
    labels:
      app: shared-ingress-gateway
  http:
  - name: marketing-httpbin
    hostname: marketing.httpbin.com
    port: 80
    routing:
      rules:
      - route:
          serviceDestination:
            host: marketing/httpbin.marketing.svc.cluster.local

Apply using kubectl in a gitOps enabled app cluster

kubectl apply -f marketing-ingress-gw.yaml -n marketing

TSB Setup - payment

Onboard application namespaces into TSB by creating a Tenant & Workspace.

---
apiVersion: tsb.tetrate.io/v2
kind: Tenant
metadata:
  name: payment
  annotations:
    tsb.tetrate.io/organization: tetrate
spec:
  displayName: Payment
---
apiVersion: tsb.tetrate.io/v2
kind: Workspace
metadata:
  name: payment-ws
  annotations:
    tsb.tetrate.io/organization: tetrate
    tsb.tetrate.io/tenant: payment
spec:
  namespaceSelector:
    names:
      - "*/payment"
  displayName: payment-ws

Configure the Ingress Gateway within the same Tenant, Workspace, and GatewayGroup established by the Platform team, but apply the configuration to the payment namespace using GitOps.

Configure shared gateway workload selector

Make sure workload selector is configured as shared-ingress-gateway in the gateway configuration to refer to the shared gateway instance created by platform team.

payment-ingress-gw.yaml

apiVersion: gateway.tsb.tetrate.io/v2
kind: Gateway
metadata:
  annotations: 
    tsb.tetrate.io/organization: tetrate
    tsb.tetrate.io/tenant: platform
    tsb.tetrate.io/workspace: shared-ingress-ws
    tsb.tetrate.io/gatewayGroup: shared-ingress-gg
  name: payment-ingress-gateway
spec:
  workloadSelector:
    namespace: shared-ingress
    labels:
      app: shared-ingress-gateway 
  http:
  - name: payment-httpbin
    hostname: payment.httpbin.com
    port: 80
    routing:
      rules:
      - route:
          serviceDestination:
            host: payment/httpbin.payment.svc.cluster.local

Verify Correct Operation

Export GATEWAY_IP

 export GATEWAY_IP=$(kubectl -n shared-ingress get service shared-ingress-gateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

Verify requests

 curl http://payment.httpbin.com/get --resolve "payment.httpbin.com:80:$GATEWAY_IP" -v

 curl http://marketing.httpbin.com/get --resolve "marketing.httpbin.com:80:$GATEWAY_IP" -v

Limitations and Restrictions

The application ingress gateways are set up using the Tenant, Workspace, and GatewayGroup created by the Platform team. As a consequence, observability features like topology, metrics, and tracing require a read-only access grant to the shared-gateway workspace. This is required for the application team to fully leverage TSB's observability and troubleshooting capabilities.

Configuration distribution is a crucial factor when using GitOps. It's worth noting that a similar approach can be implemented without onboarding application teams into TSB. If the goal is solely for the teams to publish gateway configurations into their application namespaces via GitOps, this can be done without requiring TSB onboarding. However, the absence of TSB may limit other platform-level features.

Approach 2: Utilizing TSB Direct Mode APIs

In this setup, the Application team is responsible for managing the gateway configurations needed to expose their services. The Platform team handles the provisioning of infrastructure components such as internal and external load balancers, NodePort services, and ClusterIP services, as well as the deployments for gateway workloads. These gateway workloads will be shared among various application teams belonging to different namespaces.

The key difference is that the Application team will use Istio Direct Mode resources like Gateway, VirtualService, and DestinationRule to expose their services. These resources will be implemented within the TSB constructs of Tenant, Workspace, and GatewayGroup that are specifically created for each Application Team.

Application Deployment

Consider you have 2 application teams named marketing and payment shares a common k8s cluster. We will be deploying httpbin app on both the namespaces for demo purpose. Follow the instructions in this document to deploy httpbin service.

TSB Setup - Platform Team

Platform team deploys the shared gateway install resource to create gateway deployment resources in a shared namespace. This doesn't require platform teams to create TSB resources like Tenant/Workspace/Group as they are going to manage only the Gateway install resouce which is a k8s resource.

Apply the below configuration in shared-ingress namespace owned by platform teams.

shared-gw-install.yaml

apiVersion: install.tetrate.io/v1alpha1
kind: Gateway
metadata:
  name: shared-ingress-gateway
  namespace: shared-ingress
spec:
  type: INGRESS
  kubeSpec:
    service:
      type: LoadBalancer

Apply the resource yaml using kubectl

kubectl apply -f shared-gw-install.yaml -n shared-ingress

Set PILOT_SCOPE_GATEWAY_TO_NAMESPACE=false

TSB sets this to true by default so that a gateway workload can only select gateway resources in the same namespace. Gateways with same selectors in different namespaces will not be supported.

In a shared-gateway ( shared gateway workload ) use case, gateway workload needs to select resources in other app namespaces as well, hence we need to set this to false.

Configure it under istio in controlplane CR if you are using data-plane operator. If IsolationBoundary is enabled, make sure you configure this for all the revision specific IstioOperator created by XCP to apply it for corresponding istiod.

{
 "xcp": 
  {
    .
    .
    "isolationBoundaries":
    [
      {
        "name": "global",
        "revisions": [
            {
              "istio": {
                "tsbVersion": "1.10.0",
                "kubeSpec": {
                  "overlays": [
                    {
                      "apiVersion": "install.istio.io/v1alpha1",
                      "kind": "IstioOperator",
                      "name": "xcp-iop-default",
                      "patches": [
                        {
                          "path": "spec.components.pilot.k8s.env[-1]",
                          "value": {
                            "name": "PILOT_SCOPE_GATEWAY_TO_NAMESPACE",
                            "value": "false"
                          }
                        }
                      ]
                    }
                  ]
                }
              },
              "name": "default"
            },
            {
              .
              .
            }
        ]
      }
    ]
  }
}

TSB Setup for Application Teams

You will be creating TSB configurations for marketing team to expose their services in the application ingress gateway that each individual teams owns. You can repeat the same for payment just by replacing the occurrences of marketing to payment.

TSB Configuration

---
apiVersion: tsb.tetrate.io/v2
kind: Tenant
metadata:
  name: marketing
  annotations:
    tsb.tetrate.io/organization: tetrate
spec:
  displayName: Marketing
---
apiVersion: tsb.tetrate.io/v2
kind: Workspace
metadata:
  name: marketing-ws
  annotations: 
    tsb.tetrate.io/organization: tetrate
    tsb.tetrate.io/tenant: marketing
spec:
  namespaceSelector:
    names:
      - "*/marketing"
  displayName: marketing-ws

Gateway Configuration - DIRECT Mode

To enable application teams to configure the shared gateway from their own app namespace is only supported in TSB DIRECT mode. What this means is, application teams will have to create TSB DIRECT Mode Istio resources to deploy Gateway configurations.

Create GatewayGroup for marketing team and configure configMode: DIRECT to indicate DIRECT mode resources would be applying for the namespaces referenced in the GatewayGroup

marketing-gg.yaml

apiVersion: gateway.tsb.tetrate.io/v2
kind: Group
metadata:
  name: httpbin-marketing-gg
  annotations:
    tsb.tetrate.io/organization: tetrate
    tsb.tetrate.io/tenant: marketing
    tsb.tetrate.io/workspace: marketing-ws
spec:
  displayName: HTTPBIN Marketing Gateway Group
  namespaceSelector:
    names:
      - "*/marketing"
  configMode: DIRECT

Create Direct mode Istio resources i.e Gateway and VirtualService configuration by annotating TSB constructs like Org/Tenant/Workspace/Groups

Configure shared gateway workload selector

Make sure workload selector is configured as shared-ingress-gateway in the gateway configuration to refer to the shared gateway instance created by platform team.

httpbin-gateway.yaml

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: httpbin-marketing-gateway
  namespace: marketing
  annotations: 
    tsb.tetrate.io/organization: tetrate
    tsb.tetrate.io/tenant: marketing
    tsb.tetrate.io/workspace: marketing-ws
    tsb.tetrate.io/gatewayGroup: httpbin-marketing-gg
  labels:
    istio.io/rev: tsb
spec:
  selector:
    app: shared-ingress-gateway 
  servers:
  - port: 
      number: 80
      name: http
      protocol: HTTP
    hosts:
      - "marketing.httpbin.com"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin-marketing-vs
  namespace: marketing
  annotations:
    tsb.tetrate.io/organization: tetrate
    tsb.tetrate.io/tenant: marketing
    tsb.tetrate.io/workspace: marketing-ws
    tsb.tetrate.io/gatewayGroup: httpbin-marketing-gg
  labels:
    istio.io/rev: tsb
spec:
  hosts:
  - "marketing.httpbin.com"
  gateways:
  - httpbin-marketing-gateway
  http:
  - match:
    - uri:
        prefix: /get
    - method:
        exact: "GET"
    route:
      - destination:
          host: httpbin.marketing.svc.cluster.local

Where to store the TLS certificates

For TLS enabled application, shared-gateway will need a certificate applied in the same namespace as the shared-gateway

Repeat the process for other teams and applications that should use the same shared gateway.

Verify Correct Operation

Export $GATEWAY_IP

 export GATEWAY_IP=$(kubectl -n shared-ingress get service shared-ingress-gateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

Verify requests

curl http://payment.httpbin.com/get --resolve "payment.httpbin.com:80:$GATEWAY_IP" -v

curl http://marketing.httpbin.com/get --resolve "marketing.httpbin.com:80:$GATEWAY_IP" -v

Limitations and Restrictions

This approach bypasses the segmentation, validation and safety protections provided by TSB. Instead, this approach:

• Relies on a custom patch to the Istio operator to relax the Istio security posture
• Requires users to provide Istio configuration directly

It must be used with care, in a trusted environment, only if the alternative approach is not suitable.