Skip to main content
logoTetrate Enterprise Envoy Gateway (TEG)Version: v0.1.0

Using TEG in Conjunction with an Istio Service Mesh

  • Get a k8s cluster
  • Install TEG

Caveats

  • Neither the Pod nor Service definitions for the app can specify http2 for the port (neither as a port name prefix, nor with the appProtocol field). This breaks connection between the TEG sidecar and the app sidecar.

Steps

  1. Prepare TEG to Interoperate with Istio

    Label TEG's namespace so that the data plane gets Istio sidecars. The control plane will too, but that's no bad thing.

    kubectl label namespace envoy-gateway-system --overwrite=true istio-injection=enabled

    The EG control plane Service is mis-defined; saying it takes plaintext traffic when actually it serves TLS. When Istio is handling traffic to the control plane this will actually matter, so patch it.

    control-plane-tls.yaml
    spec:
    ports:
    - port: 18000
    appProtocol: tls
    kubectl patch service -n envoy-gateway-system envoy-gateway \
    --type strategic --patch-file control-plane-tls.yaml

    We also need to tell the TEG sidecars not to process any traffic entering the gateway Envoys.

    Normally, an app's sidecar would process traffic on the way in (auth, rate-limiting, etc), as well as on the way out (routing, mirroring, etc). But since the "app" in this case is an edge proxy, we don't want the sidecar to do its thing to traffic coming in from users, as there are several potential pitfalls:

    • Strict mTLS would be unusable, as the Istio Envoys think they're sidecars not gateways, so they'd insist on istio-style mTLS from external clients.
    • It wouldn't be possible to serve TLS from TEG without putting the sidecars in SNI routing mode on the inbound. It's also a performance optimisation, as TEG offers a superset of the sidecar's features.

    The best way to not have the sidecars interfere with inbound traffic is to tell them not to even intercept it within the Pod; to not configure the iptables rules they'd normally use for that. This can be done with an annotation on the Pods: traffic.sidecar.istio.io/includeInboundPorts="". The default value for includeInboundPorts is "*", meaning all ports. The empty string means no ports.

    In order to add an annotation, we use an EnvoyProxy resource as follows:

    teg-sidecars-no-inbound.yaml
    apiVersion: config.gateway.envoyproxy.io/v1alpha1
    kind: EnvoyProxy
    metadata:
    name: data-plane-sidecars
    namespace: envoy-gateway-system
    spec:
    provider:
    type: Kubernetes
    kubernetes:
    envoyDeployment:
    pod:
    annotations:
    traffic.sidecar.istio.io/includeInboundPorts: ""
    kubectl apply -f teg-sidecars-no-inbound.yaml

    We then alter the GatewayClass for this instance of the TEG control plane to point to that EnvoyProxy resource; applying its settings to all created proxy Pods.

    gtwcls-use-envoyproxy.yaml
    spec:
    parametersRef:
    group: config.gateway.envoyproxy.io
    kind: EnvoyProxy
    namespace: envoy-gateway-system
    name: data-plane-sidecars
    kubectl patch gatewayclass teg --patch-file gtwcls-use-envoyproxy.yaml --type merge
  2. Install Istio

    Left as an exercise to the reader. You probably want to use a profile that doesn't deploy the default Istio Ingress Gateway. They won't interfere, but it avoids confusion.

  3. Restart TEG Control Plane

    Now Istio is ready with its sidecar-injection webhook, we restart all the TEG control plane Pods, which will come back up with sidecars.

    for d in envoy-gateway envoy-ratelimit teg-envoy-gateway teg-redis; \
    do kubectl rollout restart deployment -n envoy-gateway-system $d; \
    done
  4. Deploy Test Apps

    It's important this happens after Istio is installed, as they need sidecars too.

    For example:

    kubectl label namespace default --overwrite=true istio-injection=enabled
    kubectl apply -f https://raw.githubusercontent.com/istio/istio/master/samples/httpbin/httpbin.yaml
  5. Configure TEG

    We now configure TEG to process traffic at the edge, in the normal way.

    For example

    apps-gateway.yaml
    apiVersion: gateway.networking.k8s.io/v1beta1
    kind: Gateway
    metadata:
    name: apps
    spec:
    gatewayClassName: teg
    listeners:
    - name: http
    protocol: HTTP
    port: 80
    kubectl apply -f apps-gateway.yaml
    httpbin-route.yaml
    apiVersion: gateway.networking.k8s.io/v1beta1
    kind: HTTPRoute
    metadata:
    name: httpbin
    spec:
    parentRefs:
    - group: gateway.networking.k8s.io
    kind: Gateway
    name: apps
    hostnames:
    - "www.example.com"
    rules:
    - matches:
    - path:
    type: PathPrefix
    value: /httpbin/
    backendRefs:
    - group: ""
    kind: Service
    name: httpbin
    port: 8000
    kubectl apply -f httpbin-route.yaml
    note

    We don't remove the routed path prefix here. This is important, because the later intra-mesh stage will need it to differentiate requests, just like this HTTPRoute did. It can be removed in that later stage, for apps which can't be "mounted" on any path other than /.

  6. Configure Istio to Transport Traffic Through the Mesh

    Now comes the crux of EG + Istio integration.

    EG is listening on the edge of the network, processing requests as a gateway, and then just dumping them "on the wire", aimed at the designated internal service, with no idea that there's anything more than a Layer 3 Kubernetes cluster network in the way.

    Likewise, Istio has attached sidecars to the TEG Pods, not knowing they're anything other than "normal" workloads in the mesh. Because TEG is acting as a gateway, it's receiving requests eg for www.example.com and putting them back on the wire still with Host: www.example.com. Because Istio's sidecars are precisely that - configured to be sidecars, not gateways (which Istio also supports), they will ignore the IP destination of the requests coming out of TEG (which are correct), and instead try to route to the HTTP host (as if the microservice they're sidecar for wants to call an internet API).

    With Istio in outbound-traffic-mode ALLOW_ANY, this request will get sent to the real www.example.com - which is not what we want. If instead this were a production deployment and the host were your real domain, say acme.com, the traffic would infinite loop back to TEG. With Istio in outbound-traffic-mode REGISTRY_ONLY, these requests get blocked as there's not a ServiceEntry declaring www.example.com (Istio returns a 502, which TEG forwards to the client).

    There are a couple of dirty solutions to this, which just about work. The most obvious of which is to have TEG re-write the HTTP hostname of the request to the name of the Service it's routing to. This is trivial with a filter in HTTPRoute, but it provides a bad user experience, and the new hostname has to be fully qualified (eg httpbin.default.svc.cluster.local) which reduces its portability.

    The correct solution is, having used HTTPRoute to configure routing through TEG, now configure further routing through the Istio mesh. This is done in the normal way with a VirtualService, but this VS must attach not to an Ingress Gateway, but to the special mesh Gateway which represents normal sidecars. Notice that this is where we can remove the path prefix, if desired.

    httpbin-mesh-vs.yaml
    apiVersion: networking.istio.io/v1beta1
    kind: VirtualService
    metadata:
    name: example-mesh
    spec:
    gateways: ["mesh"]
    hosts:
    - www.example.com
    http:
    - match:
    - uri: { prefix: /httpbin }
    rewrite:
    uri: /
    route:
    - destination:
    host: httpbin
    port: { number: 8000 }
    kubectl apply -f httpbin-mesh-vs.yaml
    info

    A ServiceEntry isn't actually needed, because SEs add entries to the (hidden) Istio service registry, which is only consulted if the traffic falls through on internal routes (which are precisely what a VS adds) and is thus determined to be external. Neither is a DestinationRule needed, because Istio's service registry entry for our httpbin Pod knows it has a sidecar, as it's under the control of the same mesh control plane.

    Easiest is to think about the three different resources thus

    • Gateway - punch port open. This would always exist with either TEG or Istio
    • HTTPRoute - configure routing and features at the edge, eg authn, ratelimit
    • VirtualService - configure routing and features in the mesh, eg mTLS, traffic shifting, traffic mirroring, fault injection

    Of course, we could have used the Gateway API (specifically another HTTPRoute) to configure the intra-mesh routing, but we wanted to make the distinction easy.

  7. Enable Strict mTLS

    The above configuration will work just fine. However, so would a configuration where we had an Istio mesh containing our app, and a separate TEG, without sidecars. The difference is that there would have been TLS between TEG and the app, as the traffic "jumped the gap" (think of our setup as "lifting TEG into the mesh").

    To avoid any chance of configuring this by accident, and as a general best practice, enable strict mutual TLS, and verify things still work.

    strict-mtls.yaml
    apiVersion: security.istio.io/v1beta1
    kind: PeerAuthentication
    metadata:
    name: "default"
    namespace: "istio-system"
    spec:
    mtls:
    mode: STRICT
    kubectl apply -f strict-mtls.yaml

Next Steps