Unified Gateway

This document introduces the concept of a Unified Gateway within the Tetrate Service Bridge (TSB) ecosystem, explaining its significance and providing detailed usage scenarios.

Introduction

Unified Gateway is a critical feature introduced in TSB 1.7.0 that merges the capabilities of Tier1Gateway and IngressGateway under a common resource called Gateway. This unification simplifies the gateway management process and offers a more cohesive experience.

From TSB 1.7.0 onwards, Tier1Gateway and IngressGateway resources will be deprecated, and we strongly recommend using the Gateway resource for all your gateway needs. The former Tier1 Gateway will now be collectively known as Edge Gateway.

The Unified Gateways tab is seamlessly integrated into the TSB UI, allowing easy configuration of any gateway, whether it functions as a Tier 1 or Tier 2 gateway.

Why the Unified Gateway?

Early in our journey, we recognized the distinctive needs of our customers for both cluster-specific (Tier 2) and cross-cloud vendor (Tier1) gateways. As a result, we developed separate gateway solutions to cater to these varying demands. However, as our Gateway API evolved and customer requirements grew more complex, the need to continually enhance the capabilities of our Tier1 gateway became evident.

This evolution brought challenges—ongoing engineering efforts, customer education on when to choose Tier1 or Tier2, and maintaining parallel codebases. We've embarked on a groundbreaking endeavor: Unified Gateway to streamline these complexities and provide a more cohesive experience.

The Unified Gateway Advantage

Unified Gateway isn't just a fusion of Tier 1 and Tier 2 gateways; it's a paradigm shift in gateway management. Here's what you need to know about this game-changing solution:

Comprehensive Capabilities

Unified Gateway combines the robust capabilities of Tier 1 and Tier 2 gateways from TSB version 1.6.X, ensuring you get the best of both worlds. Whether dealing with retries, failover, or any other advanced functionality, Unified Gateway has you covered, regardless of whether it's configured as a Tier 1 or Tier 2 gateway.

Seamless Transition

For our existing customers, we understand the importance of continuity. Fear not, as your Tier 1 and Tier 2 gateways will continue to function as usual with the capabilities provided by version 1.6.X. But we're not stopping there. We're introducing a seamless process to transition your existing gateways to the Unified Gateway model, enhancing Tier 1 functionality such as retries and more.

New API for Unified Gateway

Embracing innovation doesn't mean neglecting the past. While introducing the new Unified Gateway API for fresh opportunities, we're committed to supporting the previous APIs for the subsequent three TSB releases. This ensures you can switch at your own pace and without disruption.

Empowering Direct Mode

Unified Gateway isn't just about gateways—it's about empowerment. Both new and existing customers can harness the full capabilities of the Gateway API via the Direct mode, offering unparalleled control and customization over their mesh infrastructure.

Aligned with Open API Strategy

We believe in the power of open standards. Unified Gateway aligns seamlessly with our Open API strategy, enabling you to configure Unified Gateways using a standardized Open API specification. This approach promotes consistency and simplifies integration with your existing toolchain.

New API for Gateway Install

Starting from TSB version 1.9.0, Unified Gateway can be installed using a single Gateway API install resource for the following different use-cases.

apiVersion: install.tetrate.io/v1alpha1
kind: Gateway
metadata:
  name: gateway
spec:
  type: UNIFIED/INGRESS/EGRESS/EASTWEST
  kubeSpec:
    service:
      type: LoadBalancer

Functionality	Deployment Type	Service Type	Customizable to	Default Ports Exposed	Multicluster Service Entries	Istio Label	Additional Notes
Unified	UNIFIED (default)	LoadBalancer	NodePort	80(HTTP), 443(HTTPS), 15443(ISTIO_MTLS)	Yes (Transit, Ingress, Egress hosts)	istio: ingressgateway	-
Ingress	INGRESS	LoadBalancer	NodePort	80(HTTP), 443(HTTPS), 15443(ISTIO_MTLS)	Yes (Transit, Ingress hosts)	istio: ingressgateway	Validations: No EastWest or Egress Gateway
Egress	EGRESS	ClusterIP	NodePort/LoadBalancer	80(HTTP), 443(HTTPS), 15443(ISTIO_MTLS)	No	istio: egressgateway	Validations: No EastWest or Ingress Gateway
EastWest	EASTWEST	LoadBalancer	NodePort	15443(ISTIO_MTLS)	Yes (Canonical services)	istio: ingressgateway	Validations: No Ingress or Egress Gateway

Use Cases

Let's dive into the unified gateway usage scenarios.

Preparing Clusters

The following image shows the deployment architecture we use in this document. We created 3clusters in GKE, deployed TSB in one of them, loaded the other three clusters into TSB, and deployed the bookinfo application in the cluster with the infrastructure shown below.

The following table describes the roles and applications of these clusters:

Cluster	gke-jimmy-us-central1-1	gke-jimmy-us-west1-1	gke-jimmy-us-west1-2	gke-jimmy-us-west2-3
Region	us-central1	us-west1	us-west1	us-west2
TSB Role	Management Plane	Control Plane	Control Plane	Control Plane
Application	-	bookinfo-frontend	bookinfo-backend	httpbin
Services	-	productpage	productpage, ratings, reviews, details	httpbin
Network	tier1	cp-cluster-1	cp-cluster-2	cp-cluster-3

This section describes the unified gateway usage scenarios.

Scenario 1: Cluster-Based Routing with HTTP Path and Header Match

In this scenario, we will use the Gateway resource to expose bookinfo.tetrate.io and httpbin.tetrate.io. We will leverage cluster-based routing capabilities based on the path prefix on Gateway to route bookinfo frontend services to cp-cluster-1 and other backend services to cp-cluster-2. Using Gateway, users can expose multiple such hosts with clusterDestination, provided the host:port combination is unique.

Deployment Topology and Traffic Routing

We have set up the following deployment topology:

• Tier 1 Cluster: This cluster acts as the entry point for external traffic and routes it to the respective backend clusters.

• Backend Clusters: There are three backend clusters, each hosting different services:

  1. cp-cluster-1 hosts the frontend service of the "Bookinfo" application.
  2. cp-cluster-2 hosts the backend services of the "Bookinfo" application.
  3. cp-cluster-3 hosts an HTTP service called httpbin.

Configuration

1. Tier 1 Cluster Gateway (edge-gateway):

In the tier1 cluster, we deploy a Gateway named edge-gateway. This Gateway receives incoming traffic and routes it to the appropriate backend clusters based on the host and path prefix.

Gateway Install resource

apiVersion: install.tetrate.io/v1alpha1
kind: Gateway
metadata:
  name: gateway
spec:
  type: UNIFIED
  kubeSpec:
    service:
      type: LoadBalancer

Here's a snippet of the configuration for routing requests to the "Bookinfo" frontend and backend services:

apiVersion: gateway.tsb.tetrate.io/v2
kind: Gateway
 metadata:
   name: edge-gateway
   namespace: tier1
   annotations:
     tsb.tetrate.io/organization: tetrate
     tsb.tetrate.io/tenant: tier1
     tsb.tetrate.io/workspace: tier1
     tsb.tetrate.io/gatewayGroup: edge-gateway-group
 spec:
   workloadSelector:
     namespace: tier1
     labels:
       app: edge-gateway
   http:
   - name: bookinfo
     hostname: bookinfo.tetrate.io
     port: 80
     routing:
       rules:
         - match:
             - uri:
                 prefix: "/productpage"
               headers:
                 X-CLUSTER-SELECTOR:
                   exact: gke-jimmy-us-west1-1
           route:
             clusterDestination:
               clusters:
                 - name: gke-jimmy-us-west1-1
                   weight: 100
         - match:
             - uri:
                 prefix: "/productpage"
               headers:
                 X-CLUSTER-SELECTOR:
                   exact: gke-jimmy-us-west1-2
           route:
             clusterDestination:
               clusters:
                 - name: gke-jimmy-us-west1-2
                   weight: 100
         - match:
             - uri:
                 prefix: "/productpage"
           route:
             clusterDestination:
               clusters:
                 - name: gke-jimmy-us-west1-1
                   weight: 100
         - match:
             - uri:
                 prefix: "/api/v1/products"
           route:
             clusterDestination:
               clusters:
                 - name: gke-jimmy-us-west1-2
                   weight: 100
   - name: httpbin
     hostname: httpbin.tetrate.io
     port: 80
     routing:
       rules:
         - route:
             clusterDestination:
               clusters:
                 - name: gke-jimmy-us-west2-3
                   weight: 100

These rules ensure that requests to bookinfo.tetrate.io with different path prefixes are routed to the appropriate backend clusters. Similarly, requests to httpbin.tetrate.io are directed to the cp-cluster-3.

2. Ingress Gateways in Backend Clusters

In each backend cluster (cp-cluster-1, cp-cluster-2, and cp-cluster-3), we deploy Ingress Gateways to receive traffic from the tier1 cluster and route it to the respective services.

Gateway Install resource

apiVersion: install.tetrate.io/v1alpha1
kind: Gateway
metadata:
  name: gateway
spec:
  type: UNIFIED
  kubeSpec:
    service:
      type: LoadBalancer

Here's an example of the Ingress Gateway configuration in cp-cluster-1:

apiVersion: gateway.tsb.tetrate.io/v2
kind: Gateway
metadata:
  name: bookinfo-ingress-gateway
spec:
  # ... (metadata and selectors)
  http:
    - hostname: bookinfo.tetrate.io
      name: bookinfo-tetrate
      port: 80
      routing:
        rules:
          - route:
              serviceDestination:
                host: bookinfo-frontend/productpage.bookinfo-frontend.svc.cluster.local

This configuration ensures that traffic received by the Ingress Gateway in cp-cluster-1 for bookinfo.tetrate.io is routed to the frontend service.

Verification

We can use tools like curl to request the exposed services to verify the setup. For example, to test /productpage:

export GATEWAY_IP=$(kubectl -n tier1 get service edge-gateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

curl -Ss -H "X-B3-Sampled: 1" "http://bookinfo.tetrate.io/productpage" --resolve "bookinfo.tetrate.io:80:$GATEWAY_IP" -v

Similarly, you can test other routes and services per the defined routing rules.

Scenario 2: Host-Based Routing with Gateway Header Rewriting

This scenario demonstrates the unified gateway's authority rewriting or header rewriting function. We deploy an edge gateways in the tier1 cluster to route traffic between different clusters and use IngressGateways for each of the three control plane clusters to receive traffic.

Deployment Topology and Traffic Routing

We have set up the following deployment topology:

• Tier 1 Cluster: This cluster serves as the entry point for external traffic and routes it to the respective backend clusters.

• Backend Clusters: There are three backend clusters, each hosting different services:

  1. cp-cluster-1 hosts the frontend service of the "Bookinfo" application.
  2. cp-cluster-2 hosts the backend services of the "Bookinfo" application.

Configuration

1. Tier 1 Cluster Gateway (tier1-gateway)

In the Tier 1 cluster, we deploy a Gateway named tier1-gateway. This Gateway receives incoming traffic and routes it to the appropriate backend clusters based on the host and path prefix. Additionally, it performs host header rewriting for specific routes.

Gateway Install resource

apiVersion: install.tetrate.io/v1alpha1
kind: Gateway
metadata:
  name: gateway
spec:
  type: UNIFIED
  kubeSpec:
    service:
      type: LoadBalancer

Here's a snippet of the configuration for routing requests to the "Bookinfo" frontend and backend services with header rewriting:

apiVersion: gateway.tsb.tetrate.io/v2
kind: Gateway
metadata:
  name: tier1-gateway
  namespace: tier1
spec:
  # ... (metadata and selectors)
  http:
  - name: bookinfo
    hostname: bookinfo.tetrate.io
    port: 80
    routing:
      rules:
        - match:
            - uri:
                prefix: "/productpage"
          modify:
            rewrite:
              authority: 'internal-bookinfo-frontend.tetrate.io'
          route:
            clusterDestination:
              clusters:
                - name: gke-jimmy-us-west1-1
                  weight: 100
        - match:
            - uri:
                prefix: "/api/v1/products"
          modify:
            rewrite:
              authority: 'internal-bookinfo-backend.tetrate.io'
          route:
            clusterDestination:
              clusters:
                - name: gke-jimmy-us-west1-2
                  weight: 100

These rules ensure that requests to bookinfo.tetrate.io with different path prefixes are routed to the appropriate backend clusters. Additionally, the host header is rewritten for these routes.

2. Ingress Gateways in Backend Clusters

In each backend cluster (cp-cluster-1 and cp-cluster-2), we deploy Ingress Gateways to receive traffic from the tier1 cluster and route it to the respective services. These Ingress Gateways listen to the rewritten host headers.

Gateway Install resource

apiVersion: install.tetrate.io/v1alpha1
kind: Gateway
metadata:
  name: gateway
spec:
  type: UNIFIED
  kubeSpec:
    service:
      type: LoadBalancer

Here's an example of the Ingress Gateway configuration in cp-cluster-1:

apiVersion: gateway.tsb.tetrate.io/v2
kind: Gateway
metadata:
  name: bookinfo-ingress-gateway
  annotations:
    tsb.tetrate.io/organization: tetrate
    tsb.tetrate.io/tenant: payment
    tsb.tetrate.io/workspace: bookinfo-frontend-ws
spec:
  displayName: Bookinfo Ingress
  workloadSelector:
    namespace: bookinfo-frontend
    labels:
      app: bookinfo-gateway
  http:
    - hostname: internal-bookinfo-frontend.tetrate.io
      name: bookinfo-tetrate
      port: 80
      routing:
        rules:
          - route:
              serviceDestination:
                host: bookinfo-frontend/productpage.bookinfo-frontend.svc.cluster.local

This configuration ensures that traffic received by the Ingress Gateway in cp-cluster-1 with the rewritten host header is routed to the frontend service.

Verification

We can use tools like curl to request the exposed services to verify the setup. For example, to test /productpage:

export GATEWAY_IP=$(kubectl -n tier1 get service tier1-gateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

curl -Ss -H "X-B3-Sampled: 1" "http://bookinfo.tetrate.io/productpage" --resolve "bookinfo.tetrate.io:80:$GATEWAY_IP" -v

Similarly, you can test other routes and services per the defined routing rules.

Restrict Exposed Hostnames for a Limited Set of Apps/Namespaces

When services are exposed via an IngressGateway, by default, all workloads deployed within the same cluster across all namespaces gain access to the exposed services. This is because the ServiceEntry, created for the exposed services, automatically exports them to all namespaces within the cluster.

However this behaviour can be restricted using hostReachability in WorkspaceSettings, where you can configure only those hostnames that needs to be exported for the application namespaces.

Here internal-bookinfo-backend.tetrate.io hostname is restricted only for app-1 workspace namespace selectors

  apiVersion: tsb.tetrate.io/v2
  kind: WorkspaceSetting
  metadata:
    name: app-1-wss
    annotations:
      tsb.tetrate.io/organization: tetrate
      tsb.tetrate.io/tenant: app-1
      tsb.tetrate.io/workspace: app-1-ws
  spec:
    hostsReachability:
      hostnames:
      - exact: "internal-bookinfo-backend.tetrate.io"

Similarly internal-bookinfo-frontend.tetrate.io is restricted to app-2 workspace namespace selectors

  apiVersion: tsb.tetrate.io/v2
  kind: WorkspaceSetting
  metadata:
    name: app-2-wss
    annotations:
      tsb.tetrate.io/organization: tetrate
      tsb.tetrate.io/tenant: app-2
      tsb.tetrate.io/workspace: app-2-ws
  spec:
    hostsReachability:
      hostnames:
      - exact: "internal-bookinfo-frontend.tetrate.io"

Please verify the exportTo attribute in the translated ServiceEntry objects created under xcp-multicluster namespace once you configure the above settings.

kubectl get se -n xcp-multicluster

NAME                                                           HOSTS                                           LOCATION        RESOLUTION   AGE
default-gateway-internal-bookinfo-backend-tetrate-io           ["internal-bookinfo-backend.tetrate.io"]        MESH_INTERNAL   STATIC       3h33m
default-gateway-internal-bookinfo-frontend-tetrate-io          ["internal-bookinfo-frontend.tetrate.io"]       MESH_INTERNAL   STATIC       3h33m

Scenario 3: Create UnifiedGateway using TSB Application & API with OpenAPI Spec

This scenario demonstrates how to create Unified Gateways for Tier 1 and Tier 2 use cases using OpenAPI Specification with Tetrate Service Bridge (TSB). This approach allows you to define traffic routing using OpenAPI Specs for your application.

Deployment Topology and Traffic Routing

We aim to configure a Unified Gateway using OpenAPI Specifications for traffic routing. The following diagram illustrates the deployment topology and routing setup:

Configuration Steps

1. Tier 1 Cluster Configuration

   In the tier1 cluster, we configure the Application and API resources using OpenAPI Specs. These configurations use the x-tsb-clusters annotation for cluster-based routing to expose bookinfo.tetrate.io.

   x-tsb-clusters:
     clusters:
       - name: gke-jimmy-us-west1-2
         weight: 100

   This configuration routes traffic to multiple Tier 2 clusters specified in the x-tsb-clusters annotation.

2. Tier 2 Cluster Configuration

   In the Tier 2 cluster (cp-cluster-2 in this scenario), we configure the Application and API resources with service-based routing to expose bookinfo.tetrate.io. This configuration uses the x-tsb-service annotation to route to the productpage.bookinfo-backend service.

   x-tsb-service: productpage.bookinfo-backend

Verification

To verify the routing setup, you can use tools like curl to make requests to the exposed services. For example, to test the /api/v1/products/* route:

# Export the Load Balancer IP of the tier1-gateway
export GATEWAY_IP=$(kubectl -n tier1 get service tier1-gateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

# Send a request to the API
curl -Ss -H "X-B3-Sampled: 1" "http://bookinfo.tetrate.io/api/v1/products/1/reviews" --resolve "bookinfo.tetrate.io:80:$GATEWAY_IP" -v

Scenario 4: Implementing HTTP to HTTPS Redirection

This scenario demonstrates how to configure a Gateway resource to implement HTTP to HTTPS redirection. This is useful for exposing services securely via HTTPS while supporting legacy applications that use plain HTTP.

Deployment Topology

We aim to expose bookinfo.tetrate.io on port 80 with plain text (HTTP), and we'll configure HTTP to HTTPS redirection on port 443 (HTTPS) to ensure secure communication. The following diagram illustrates the deployment topology and the routing setup:

Configuration Steps

1. Tier 1 Cluster Configuration

   In the tier1 cluster, we create a Gateway resource named tier1-gateway. This gateway is responsible for HTTP to HTTPS redirection. We specify two HTTP listeners:

   • bookinfo-plaintext: This listener listens on port 80 and handles requests for bookinfo.tetrate.io. It performs an HTTP to HTTPS redirection to port 443 (HTTPS) using a 301 redirect code.

   • bookinfo: This listener listens on port 443 (HTTPS) for secure communication. It uses TLS with a secret named bookinfo-certs.

    apiVersion: install.tetrate.io/v1alpha1
    kind: Gateway
    metadata:
      name: gateway
    spec:
      type: UNIFIED
      kubeSpec:
        service:
          type: LoadBalancer

   apiVersion: gateway.tsb.tetrate.io/v2
   kind: Gateway
   metadata:
     name: tier1-gateway
     namespace: tier1
     annotations:
       tsb.tetrate.io/organization: tetrate
       tsb.tetrate.io/tenant: tier1
       tsb.tetrate.io/workspace: tier1
       tsb.tetrate.io/gatewayGroup: tier1-gateway-group
   spec:
     workloadSelector:
       namespace: tier1
       labels:
         app: tier1-gateway
     http:
     - name: bookinfo-plaintext
       port: 80
       hostname: bookinfo.tetrate.io
       routing:
         rules:
           - redirect:
               authority: bookinfo.tetrate.io
               port: 443
               redirectCode: 301
               scheme: https
     - name: bookinfo
       hostname: bookinfo.tetrate.io
       port: 443
       tls:
         mode: SIMPLE
         secretName: bookinfo-certs
       routing:
         rules:
           - match:
               - uri:
                   prefix: "/productpage"
             route:
               clusterDestination:
                 clusters:
                   - name: gke-jimmy-us-west1-2
                     weight: 100

2. Tier 2 Cluster Configuration

   In the Tier 2 cluster (cp-cluster-2 in this scenario), we configure an IngressGateway with eastWestOnly: true. This setting exposes the mTLS 15443 multi-cluster port only. We also define a Gateway resource named bookinfo-ingress-gateway to route requests.

    apiVersion: install.tetrate.io/v1alpha1
    kind: Gateway
    metadata:
      name: gateway
    spec:
      type: UNIFIED
      kubeSpec:
        service:
          type: LoadBalancer

   apiVersion: install.tetrate.io/v1alpha1
   kind: IngressGateway
   metadata:
     name: bookinfo-gateway
   spec:
     eastWestOnly: true
     kubeSpec:
       service:
         type: LoadBalancer
   apiVersion: gateway.tsb.tetrate.io/v2
   kind: Gateway
   metadata:
     name: bookinfo-ingress-gateway
     annotations:
       tsb.tetrate.io/organization: tetrate
       tsb.tetrate.io/tenant: payment
       tsb.tetrate.io/workspace: bookinfo-backend-ws
       tsb.tetrate.io/gatewayGroup: bookinfo-gg
   spec:
     displayName: Bookinfo Ingress
     workloadSelector:
       namespace: bookinfo-backend
       labels:
         app: bookinfo-gateway
     http:
       - hostname: bookinfo.tetrate.io
         name: bookinfo-tetrate
         routing:
           rules:
             - route:
                 serviceDestination:
                   host: bookinfo-backend/productpage.bookinfo-backend.svc.cluster.local

Verification

To verify the HTTP to HTTPS redirection, follow these steps:

1. To view the redirection in your browser, you need to update your /etc/hosts file to make bookinfo.tetrate.io resolve to your Edge Gateway IP:

   export GATEWAY_IP=$(kubectl -n tier1 get service tier1-gateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
   echo "$GATEWAY_IP bookinfo.tetrate.io" | sudo tee -a /etc/hosts

2. Access http://bookinfo.tetrate.io/productpage in your browser. You should be automatically redirected to https://bookinfo.tetrate.io/productpage, ensuring secure communication.

Scenario 5: Configuring External Service

In this scenario, we use a Unified Gateway to configure a dedicated egress gateway for forwarding external service traffic. We create an httpbin namespace, deploy IngressGateway in two clusters (cp-cluster-1 and cp-cluster-2), and set up ServiceEntry to define external services.

Deployment Topology

This deployment involves creating a new httpbin namespace and IngressGateway in both cp-cluster-1 and cp-cluster-2. To enable these clusters to access the external service httpbin.org, we add a ServiceEntry in both clusters and configure the gateway to override the request's authority.

Configuration Steps

1. Create ServiceEntry and IstioInternalGroup

   Define an external service using ServiceEntry and associate it with an IstioInternalGroup. This configuration enables the clusters to access httpbin.org. We create these resources in both clusters.

   apiVersion: v1
   kind: List
   items:
     - apiVersion: tsb.tetrate.io/v2
       kind: Workspace
       metadata:
         name: httpbin-ws
         annotations:
           tsb.tetrate.io/organization: tetrate
           tsb.tetrate.io/tenant: payment
       spec:
         namespaceSelector:
           names:
             - "gke-jimmy-us-west1-1/httpbin"
             - "gke-jimmy-us-west1-2/httpbin"
         displayName: httpbin-ws
     - apiVersion: istiointernal.tsb.tetrate.io/v2
       kind: Group
       metadata:
         name: httpbin-internal-gp
         annotations:
           tsb.tetrate.io/organization: tetrate
           tsb.tetrate.io/tenant: payment
           tsb.tetrate.io/workspace: httpbin-ws
       spec:
         namespaceSelector:
           names:
             - "gke-jimmy-us-west1-1/httpbin"
             - "gke-jimmy-us-west1-2/httpbin"
     - apiVersion: networking.istio.io/v1beta1
       kind: ServiceEntry
       metadata:
         name: httpbin-external-svc
         annotations:
           tsb.tetrate.io/organization: tetrate
           tsb.tetrate.io/tenant: payment
           tsb.tetrate.io/workspace: httpbin-ws
           tsb.tetrate.io/istioInternalGroup: httpbin-internal-gp
         labels:
           istio.io/rev: tsb
       spec:
         hosts:
           - httpbin.org
         exportTo:
           - "."
         location: MESH_EXTERNAL
         ports:
           - number: 443
             name: https
             protocol: HTTPS
         resolution: DNS

2. Apply Gateway Configuration

   Configure the Gateway resource to rewrite requests from httpbin.tetrate.io to httpbin.org. We set up this configuration in both clusters, cp-cluster-1 and cp-cluster-2.

    apiVersion: install.tetrate.io/v1alpha1
    kind: Gateway
    metadata:
      name: gateway
    spec:
      type: UNIFIED
      kubeSpec:
        service:
          type: LoadBalancer

   - apiVersion: gateway.tsb.tetrate.io/v2
     kind: Gateway
     metadata:
       name: httpbin-ingress-gateway
       annotations:
         tsb.tetrate.io/organization: tetrate
         tsb.tetrate.io/tenant: payment
         tsb.tetrate.io/workspace: httpbin-ws
         tsb.tetrate.io/gatewayGroup: httpbin-gg
     spec:
       displayName: Httpbin Ingress
       workloadSelector:
         namespace: httpbin
         labels:
           app: httpbin-gateway
       http:
         - hostname: httpbin.tetrate.io
           name: httpbin-tetrate
           port: 80
           routing:
             rules:
               - modify:
                   rewrite:
                     authority: httpbin.org
                 route:
                   serviceDestination:
                     host: httpbin/httpbin.org
                     tls:
                       mode: SIMPLE
                       files:
                         caCertificates: "/etc/ssl/certs/ca-certificates.crt"

Verification

To verify this configuration, you can follow these steps:

1. To obtain the IP address of the IngressGateway in both cp-cluster-1 and cp-cluster-2, run the following command in each cluster:

   export GATEWAY_IP=$(kubectl -n httpbin get service httpbin-gateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

2. Use the two gateways to proxy access to httpbin.org by executing the following command:

   curl -v -H "X-B3-Sampled: 1" 'http://httpbin.tetrate.io/get' --resolve "httpbin.tetrate.io:80:$GATEWAY_IP" -v

This will demonstrate that requests to httpbin.tetrate.io are correctly forwarded to httpbin.org through the egress gateway.

Scenario 6: Configure EgressGateway with TLS Origination for External Services

In this scenario, we use a Unified Gateway to configure a dedicated egress gateway for forwarding external service traffic with TLS origination.

Prerequisites

1. Self-signed certs for bookinfo.tetrate.com generated as per certificate-for-gateway.

2. bookinfo.tetrate.com accessible as an external endpoint. For this setup we used k8s cluster with Istio installed, bookinfo app, IngressGW exposing bookinfo.tetrate.com over port 443, TLS mode: SIMPLE with secret name bookinfo-tls-server-secret. You can create the secret using this command:

  kubectl -n bookinfo create secret tls bookinfo-tls-server-secret \
      --key bookinfo.key \
      --cert bookinfo.crt

3. TSB 1.9.0 environment is up and running, and GitOps has been enabled for the target cluster
   .

Deployment Topology

This deployment involves creating a new egressgw-ns namespace and EgressGateway in cp-cluster-1. To enable the sleep container to access the external service bookinfo.tetrate.com with TLS origination, we add a ServiceEntry in cp-cluster-1 and configure the gateway to override the request's authority.

Apply TSB Configuration

bookinfo-conf.yaml

apiVersion: v1
kind: List
items: 
  - apiVersion: tsb.tetrate.io/v2
    kind: Tenant
    metadata:
      name: payment
      annotations:
        tsb.tetrate.io/organization: tetrate
    spec:
      displayName: egressgw
  - apiVersion: tsb.tetrate.io/v2
    kind: Workspace
    metadata:
      name: egressgw-ws
      annotations:
        tsb.tetrate.io/organization: tetrate
        tsb.tetrate.io/tenant: payment
    spec:
      displayName: egressgw-ws
      namespaceSelector:
        names:
          - "*/egressgw-ns"
  - apiVersion: security.tsb.tetrate.io/v2
    kind: Group
    metadata:
      name: egressgw-gg
      annotations:
        tsb.tetrate.io/organization: tetrate
        tsb.tetrate.io/tenant: payment
        tsb.tetrate.io/workspace: egressgw-ws
    spec:
      displayName: egressgw-gg
      namespaceSelector:
        names:
          - "*/egressgw-ns"
  - apiVersion: istiointernal.tsb.tetrate.io/v2
    kind: Group
    metadata:
      name: egressgw-ig
      namespace: egressgw-ns
      annotations:
        tsb.tetrate.io/organization: tetrate
        tsb.tetrate.io/tenant: payment
        tsb.tetrate.io/workspace: egressgw-ws
    spec:
      namespaceSelector:
        names:
          - "*/egressgw-ns"

Apply EgressGateway Configuration

Create Egress Gateway Namespace

  kubectl create ns egressgw-ns

Apply EgressGateway deployment configuration

egress-deployment.yaml

  apiVersion: install.tetrate.io/v1alpha1
  kind: EgressGateway
  metadata:
    name: egressgw
    namespace: egressgw-ns
  spec:
    kubeSpec:
      deployment:
        labels:
          app: egressgw
      service:
        type: NodePort #specifiies the type of resrouce that will be created in the cloud provider (AWS)
        annotations:
          traffic.istio.io/nodeSelector: '{"beta.kubernetes.io/os": "linux"}'     

Create k8s secret for TLS client side that will be used by Egress GW.

  kubectl -n egressgw-ns create secret generic bookinfo-tls-client-secret --from-file=ca.crt=bookinfo.crt

Apply EgressGateway configuration

egress-gw-config.yaml

  apiVersion: gateway.tsb.tetrate.io/v2
  kind: Gateway
  metadata:
    name: egressgw
    namespace: egressgw-ns
    annotations:
      tsb.tetrate.io/organization: tetrate
      tsb.tetrate.io/tenant: payment
      tsb.tetrate.io/workspace: egressgw-ws
      tsb.tetrate.io/gatewayGroup: egressgw-gg
  spec:
    workloadSelector:
      namespace: egressgw-ns
      labels:
        app: egressgw
    http:
      - name: bookinfo
        port: 443
        hostname: "bookinfo.tetrate.com"
        routing:
          rules:
            - modify:
                rewrite:
                  authority: bookinfo.tetrate.com
              route:
                serviceDestination:
                  host: sleep/bookinfo.tetrate.com
                  port: 443
                  tls:
                    mode: SIMPLE
                    secretName: bookinfo-tls-client-secret

Deploy sleep service in the same cluster to generate traffic from its container to external bookinfo.tetrate.com via Egress Gateway with TLS origination.

  kubectl create ns sleep
  kubectl label namespace sleep istio-injection=enabled
  kubectl apply -n sleep -f https://docs.tetrate.io/assets/files/sleep-0e507364b65e01029474dddf45bc15df.yaml

Verification

Generate traffic from your TSB managed service mesh to external endpoint bookinfo.tetrate.com via Egress Gateway with TLS origination.

Run the below command from the sleep container.

  kubect exec -it -n sleep sleep-755c856df8-z5h5n sh
  kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.

  ~ $ while true; do curl http://bookinfo.tetrate.com -vvv -I ; sleep 1; done
  * Host bookinfo.tetrate.com:80 was resolved.
  * IPv6: (none)
  * IPv4: 240.240.102.57
  *   Trying 240.240.102.57:80...
  * Connected to bookinfo.tetrate.com (240.240.102.57) port 80
  > HEAD / HTTP/1.1
  > Host: bookinfo.tetrate.com
  > User-Agent: curl/8.5.0
  > Accept: */*
  >
  < HTTP/1.1 200 OK
  HTTP/1.1 200 OK
  < content-type: text/html; charset=utf-8
  content-type: text/html; charset=utf-8
  < content-length: 1683
  content-length: 1683
  < server: envoy
  server: envoy
  < date: Fri, 02 Feb 2024 20:33:49 GMT
  date: Fri, 02 Feb 2024 20:33:49 GMT
  < x-envoy-upstream-service-time: 18
  x-envoy-upstream-service-time: 18

As you can see, curl returns 200 - means we can successfully access bookinfo.

verify traffic

You need to run curl to http (port 80) as the traffic within your service mesh is encrypted using ISTIO_MUTUAL mode. When traffic hits Egress Gateway it will be converted to TLS encryption and port 443 using credentials from k8s bookinfo-tls-client-secret secret generated earlier.

Also, the request is sent to an automatically allocated address, 240.240.0.1. These addresses will be picked from the 240.240.0.0/16 reserved IP address range to avoid conflicting with real services. See Istio doc for more information.

Verify Egress Gateway logs.

  kubectl get pods -n egressgw
  NAME                               READY   STATUS    RESTARTS   AGE
  egressgw-7c56dc8b54-b4qhk   1/1     Running   0          2d4h

  k logs -n egressgw-7c56dc8b54-b4qhk
  [2024-02-02T20:33:49.190Z] "HEAD / HTTP/1.1" 200 - via_upstream - "-" 0 0 11 11 "172.21.25.49" "curl/8.5.0" "e48f4963-6d74-4db8-8c58-b5e7d2118d0d" "bookinfo.tetrate.com" "34.250.72.75:443" outbound|443||bookinfo.tetrate.com 172.21.4.217:34466 172.21.4.217:15443 172.21.25.49:34276 bookinfo.tetrate.com egressgw-bookinfo-external-0
  [2024-02-02T20:33:50.208Z] "HEAD / HTTP/1.1" 200 - via_upstream - "-" 0 0 5 5 "172.21.25.49" "curl/8.5.0" "e4df8b47-01fa-408f-aa2e-8b69e5246839" "bookinfo.tetrate.com" "34.250.72.75:443" outbound|443||bookinfo.tetrate.com 172.21.4.217:34466 172.21.4.217:15443 172.21.25.49:34276 bookinfo.tetrate.com egressgw-bookinfo-external-0
  [2024-02-02T20:33:51.222Z] "HEAD / HTTP/1.1" 200 - via_upstream - "-" 0 0 7 7 "172.21.25.49" "curl/8.5.0" "d0c29a43-db58-4b46-9c08-1d9ec9395ab6" "bookinfo.tetrate.com" "34.250.72.75:443" outbound|443||bookinfo.tetrate.com 172.21.4.217:46150 172.21.4.217:15443 172.21.25.49:52614 bookinfo.tetrate.com egressgw-bookinfo-external-0

As you can see from the logs, traffic generated from sleep container first hits the Egress Gateway and then goes to external bookinfo.tetrate.com endpoint resolving the IP address configured in bookinfo Service Entry which is our Ingress Gateway exposing bookinfo application on 34.250.72.75.

Troubleshooting

While using PKI generated certificates please check the ROOT_CA for both client and server workloads and ensure they are the same. Also, you may need to include the chain in your ca.crt before you create k8s secret. If the certificate is wrong you won’t be able to access your endpoint and might see this log in Egress GW pod

  [2024-02-01T19:00:35.364Z] "HEAD / HTTP/1.1" 503 UF,URX upstream_reset_before_response_started{remote_connection_failure,TLS_error:_268435581:SSL_routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED} - "TLS_error:_268435581:SSL_routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED" 0 0 113 - "10.76.33.96" "curl/8.6.0" "724ad8de-73d5-42c8-8c94-d5eb81075b2f" "bookinfo.tetrate.com" "10.53.84.5:443" outbound|443||bookinfo.tetrate.com - 10.76.34.113:15443 10.76.33.96:41084 bookinfo.tetrate.com egressgw-bookinfo-external-0
  [2024-02-01T19:00:35.509Z] "HEAD / HTTP/1.1" 503 UH no_healthy_upstream - "-" 0 0 0 - "10.76.33.96" "curl/8.6.0" "724ad8de-73d5-42c8-8c94-d5eb81075b2f" "bookinfo.tetrate.com" "-" outbound|443||bookinfo.tetrate.com - 10.76.34.113:15443 10.76.33.96:41084 bookinfo.tetrate.com egressgw-bookinfo-external-0