Tetrate Enterprise Gateway for Envoy (TEG)Version: v0.0.0-latest

Using TEG in Conjunction with an Istio Service Mesh

Istio has mature and flexible support for ingress gateways (indeed based on the same Envoy proxy as TEG's gateways). However the Istio project's focus is on application connectivity, mostly service-to-service within the cluster. By contrast, Tetrate Enterprise Gateway for Envoy is designed to be expose apps to the outside world; to be internet-facing and to handle requests from humans, not just services. This can be seen in the support for eg SSO using OIDC.

To get the best of both worlds -- Istio's mesh and TEG's advanced gateway features -- it's possible to use both together. This guide will talk you through setting that up, but be warned it's a little involved for now, so this guide is fairly advanced. We assume you're starting with a running Kubernetes cluster and have TEG installed. Being TEG-focussed, this guide assumes that cluster does not currently have Istio installed. If you do have Istio and are looking to add TEG, you can adjust the steps here to your needs.

Current caveats

Neither the Pod nor Service definitions for an app must configure ports as http2 (neither as a port name prefix, nor with the appProtocol field). This breaks connection between the TEG sidecar and the app sidecar.

Steps

Prepare TEG to Interoperate with Istio
Label TEG's namespace so that the data plane gets Istio sidecars. The control plane will too, but that's no bad thing.
```
kubectl label namespace envoy-gateway-system --overwrite=true istio-injection=enabled
```
The EG control plane Service is mis-defined; saying it takes plaintext traffic when actually it serves TLS. When Istio is handling traffic to the control plane this will actually matter, so patch it.
control-plane-tls.yaml
```
spec:
  ports:
    - port: 18000
      appProtocol: tls
```
```
kubectl patch service -n envoy-gateway-system envoy-gateway \
     --type strategic --patch-file control-plane-tls.yaml
```
We also need to tell the TEG sidecars not to process any traffic entering the gateway Envoys.
Normally, an app's sidecar would process traffic on the way in (auth, rate-limiting, etc), as well as on the way out (routing, mirroring, etc). But since the "app" in this case is an edge proxy, we don't want the sidecar to do its thing to traffic coming in from users, as there are several potential pitfalls:
- Strict mTLS would be unusable, as the Istio Envoys think they're sidecars not gateways, so they'd insist on istio-style mTLS from external clients.
- It wouldn't be possible to serve TLS from TEG without putting the sidecars in SNI routing mode on the inbound. It's also a performance optimisation, as TEG offers a superset of the sidecar's features.
The best way to not have the sidecars interfere with inbound traffic is to tell them not to even intercept it within the Pod; to not configure the iptables rules they'd normally use for that. This can be done with an annotation on the Pods: traffic.sidecar.istio.io/includeInboundPorts="". The default value for includeInboundPorts is "*", meaning all ports. The empty string means no ports.
In order to add an annotation, we use an EnvoyProxy resource as follows:
teg-sidecars-no-inbound.yaml
```
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: data-plane-sidecars
  namespace: envoy-gateway-system
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        pod:
          annotations:
            traffic.sidecar.istio.io/includeInboundPorts: ""
  routingType: Service
```
```
kubectl apply -f teg-sidecars-no-inbound.yaml
```
We then alter the GatewayClass for this instance of the TEG control plane to point to that EnvoyProxy resource; applying its settings to all created proxy Pods.
gtwcls-use-envoyproxy.yaml
```
spec:
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    namespace: envoy-gateway-system
    name: data-plane-sidecars
```
```
kubectl patch gatewayclass teg --patch-file gtwcls-use-envoyproxy.yaml --type merge
```
Install Istio
Using minimal profiles to deploy Istio avoids installing the Ingress Gateway:
```
istioctl install --set profile=demo
```
Refer to the getting started documentation to deploy Istio.
Restart TEG Control Plane
Now Istio is ready with its sidecar-injection webhook, we restart all the TEG control plane Pods, which will come back up with sidecars.
```
for d in envoy-gateway envoy-ratelimit teg-envoy-gateway teg-redis; \
    do kubectl rollout restart deployment -n envoy-gateway-system $d; \
    done
```

Deploy Test Apps

It's important this happens after Istio is installed, as they need sidecars too.

For example:

kubectl create namespace httpbin
kubectl label namespace httpbin --overwrite=true istio-injection=enabled
kubectl apply -n httpbin -f https://raw.githubusercontent.com/istio/istio/master/samples/httpbin/httpbin.yaml

Configure TEG

We now configure TEG to process traffic at the edge, in the normal way.

For example

apps-gateway.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: apps
  namespace: httpbin
spec:
  gatewayClassName: teg
  listeners:
    - name: http
      protocol: HTTP
      port: 80

kubectl apply -f apps-gateway.yaml

httpbin-route.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: httpbin
  namespace: httpbin
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: apps
  hostnames:
  - "www.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /httpbin/
    filters:
    - type: URLRewrite
      urlRewrite:
        hostname: httpbin.httpbin.svc.cluster.local
        path:
          type: ReplacePrefixMatch
          replacePrefixMatch: /
    backendRefs:
    - kind: Service
      name: httpbin
      port: 8000

kubectl apply -f httpbin-route.yaml

Set the $GATEWAY_ADDRESS
Choose the tab below that applies to your setup to configure $GATEWAY_ADDRESS that will set you up to try the
- For LB with External IP
- For LB with Hostname
- For Local with no LB - Port forwarding
We will set the $GATEWAY_ADDRESS to the external IP.
export GATEWAY_ADDRESS=$(kubectl get gateway/apps -n httpbin -o jsonpath='{.status.addresses[0].value}')
If your provider does not provide an ingress IP address, but rather a hostname (e.g., AWS) use the following to set the $GATEWAY_ADDRESS:
export GATEWAY_ADDRESS=$(kubectl get service -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-namespace=httpbin -o jsonpath='{.items[0].status.loadBalancer.ingress[0].hostname}')
Setup Port Forwarding on Local
Set $ENVOY_SERVICE variable
export ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=httpbin,gateway.envoyproxy.io/owning-gateway-name=apps -o jsonpath='{.items[0].metadata.name}')
Verify
echo $ENVOY_SERVICE
Configure port forwarding
kubectl -n envoy-gateway-system port-forward service/${ENVOY_SERVICE} 8899:80 &
For port-forwarding on Local with no loadbalancer
Set the $GATEWAY_ADDRESS
Assuming you set the port to 8899 from the instructions above, copy and paste the following.
export GATEWAY_ADDRESS=localhost:8899
Verify
```
echo $GATEWAY_ADDRESS
```
Check the output to ensure it is what you expect, it should be set.

Send Test Requests

Now when you’ve set the $GATEWAY_ADDRESS we are going to try it out!

CURL

curl -v -H Host:www.example.com http://$GATEWAY_URL/httpbin/get

Example Output

*   Trying 104.154.82.26:80...
* Connected to 104.154.82.26 (104.154.82.26) port 80
> GET /httpbin/get HTTP/1.1
> Host:www.example.com
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
< HTTP/1.1 200 OK
< server: envoy
< date: Tue, 30 Jul 2024 07:42:43 GMT
< content-type: application/json
< content-length: 282
< access-control-allow-origin: *
< access-control-allow-credentials: true
< x-envoy-upstream-service-time: 11
<
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "www.example.com",
    "User-Agent": "curl/8.7.1",
    "X-Envoy-Attempt-Count": "1",
    "X-Envoy-External-Address": "123.120.227.173"
  },
  "origin": "123.120.227.173",
  "url": "http://www.example.com/get"
}
* Connection #0 to host 104.154.82.26 left intact

Enable Strict mTLS
The above configuration will work just fine. However, it would also work if we had an Istio mesh with our app and a separate TEG without sidecars. In that case, there would be TLS between TEG and the app as the traffic crosses the boundary (consider this as "lifting TEG into the mesh").
To avoid accidentally configuring this incorrectly and as a best practice, enable strict mutual TLS and verify that everything still works correctly.
strict-mtls.yaml
```
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: "default"
  namespace: "istio-system"
spec:
  mtls:
    mode: STRICT
```
```
kubectl apply -f strict-mtls.yaml
```

Using TEG in Conjunction with an Istio Service Mesh

Steps

Prepare TEG to Interoperate with Istio

Install Istio

Restart TEG Control Plane

Deploy Test Apps

Configure TEG

Set the `$GATEWAY_ADDRESS`

Setup Port Forwarding on Local

Set `$ENVOY_SERVICE` variable

Verify

Configure port forwarding

For port-forwarding on Local with no loadbalancer

Set the `$GATEWAY_ADDRESS`

Verify

Send Test Requests

Enable Strict mTLS

Steps​

Prepare TEG to Interoperate with Istio​

Install Istio​

Restart TEG Control Plane​

Deploy Test Apps​

Configure TEG​

Set the $GATEWAY_ADDRESS​

Setup Port Forwarding on Local​

Set $ENVOY_SERVICE variable​

Verify​

Configure port forwarding​

For port-forwarding on Local with no loadbalancer​

Set the $GATEWAY_ADDRESS​

Verify​

Send Test Requests​

Enable Strict mTLS​

Steps

Prepare TEG to Interoperate with Istio

Install Istio

Restart TEG Control Plane

Deploy Test Apps

Configure TEG

Set the `$GATEWAY_ADDRESS`

Setup Port Forwarding on Local

Set `$ENVOY_SERVICE` variable

Verify

Configure port forwarding

For port-forwarding on Local with no loadbalancer

Set the `$GATEWAY_ADDRESS`

Verify

Send Test Requests

Enable Strict mTLS