External Rate Limiting with TLS Verification

Once you have configured an external rate limit server, you may want to secure the traffic to the rate limit service. TSB supports specifying TLS or mTLS parameters for securing communication to external rate limit servers. This document will show you how to configure TLS validation for an external rate limit server by adding CA certificate to the rate limiting configuration.

Before you get started, make sure you:
✓ Familiarize yourself with TSB concepts
✓ Install the TSB environment. You can use TSB demo for quick install
✓ Completed TSB usage quickstart. This document assumes you already created Tenant and are familiar with Workspace and Config Groups. Also you need to configure tctl to your TSB environment.
✓ Competed Setting Up an External Rate Limiting Server. This document will continue what you have done in Setting Up an External Rate Limiting Server. You will work in ext-ratelimit namespace, and should already have an Ingress Gateway with external rate limit properly configured

TLS certificate

To enable TLS for Ingress Gateway to rate limit service traffic, you must have a TLS certificate. This document assumes you already have TLS certificates which usually include server certificate and private key along with the CA as root certificate that will be used by the client.

This document assumes the presence of the following files. If you are using different file names, please change them accordingly:

File name	Description
ratelimit.crt	The server certificate
ratelimit.key	The certificate private key
ratelimit-ca.crt	The CA certificate

self signed certificate

For the purpose of example, you may opt to use a self-signed certificate. You may generate a self-signed certificate using the script show here, but make sure to adjust the input parameters accordingly.

Once you have the certificate files, create Kubernetes secret using server certificate and private key.

kubectl create secret tls -n ext-ratelimit ratelimit-certs \
  --cert=ratelimit.crt \
  --key=ratelimit.key

Deploy Rate Limit Service with TLS certificate

In this example you will use the Envoy rate limit service. The Envoy proxy sidecar acts as pass through proxy that will validate and terminate TLS before sending the request to the rate limit service.

Create a configuration file for Envoy with the following content as proxy-config-tls.yaml

# Envoyproxy configuration for different modes of TLS termination
# of external rate-limit server used for testing purposes

# Summary of various ports
#
# 10250 - admin port
# 18080 - insecure GRPC
# 18443 - TLS GRPC

admin:
  address:
    socket_address:
      address: 127.0.0.1
      port_value: 10250

static_resources:
  listeners:
    # Insecure GRPC listener
    - name: grpc-insecure
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 18080
      access_log:
        - name: envoy.access_loggers.file
          typed_config:
            '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
            path: /dev/stdout
      filter_chains:
        - filters:
            - name: envoy.filters.network.tcp_proxy
              typed_config:
                '@type': type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
                cluster: grpc_rlserver
                stat_prefix: grpc_insecure

    # Secured by TLS
    - name: grpc-simple-tls
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 18443
      access_log:
        - name: envoy.access_loggers.file
          typed_config:
            '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
            path: /dev/stdout
      filter_chains:
        - filters:
            - name: envoy.filters.network.tcp_proxy
              typed_config:
                '@type': type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy
                cluster: grpc_rlserver
                stat_prefix: grpc_simple_tls
          transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
              common_tls_context:
                tls_certificates:
                  - certificate_chain: {filename: /certs/tls.crt}
                    private_key: {filename: /certs/tls.key}
  clusters:
    - name: grpc_rlserver
      type: STRICT_DNS
      connect_timeout: 0.5s
      dns_refresh_rate: 3600s
      typed_extension_protocol_options:
        envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
          '@type': type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
          explicit_http_config:
            http2_protocol_options: {}
      load_assignment:
        cluster_name: grpc_rlserver
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: 127.0.0.1
                      port_value: 8081

Execute the following to store the configuration in Kubernetes as a ConfigMap.

kubectl create configmap -n ext-ratelimit ratelimit-proxy \
  --from-file=proxy-config-tls.yaml

You will need to deploy the rate limit service with an Envoy sidecar to terminate TLS. Create a file called ratelimit-tls.yaml with the following content.

kind: Service
metadata:
  name: ratelimit-tls
  namespace: ext-ratelimit
  labels:
    app: ratelimit-tls
spec:
  ports:
    - name: http
      port: 8080
      targetPort: 8080 # Doesn't go through Envoy
    - name: grpc-insecure
      port: 18080
      targetPort: 18080
    - name: grpc-tls
      port: 18443
      targetPort: 18443
  selector:
    app: ratelimit-tls
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ratelimit-tls
  namespace: ext-ratelimit
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ratelimit-tls
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: ratelimit-tls
    spec:
      containers:
        - name: envoy-proxy
          image: envoyproxy/envoy-alpine:v1.18.4
          imagePullPolicy: Always
          command:
            - '/usr/local/bin/envoy'
          args:
            - '--config-path /etc/envoy/config.yaml'
            - '--mode serve'
            - '--service-cluster ext-ratelimit'
          ports:
            - name: grpc-plaintext
              containerPort: 18080
            - name: grpc-tls
              containerPort: 18443
          volumeMounts:
            - name: proxy-config
              mountPath: /etc/envoy
            - name: proxy-certs
              mountPath: /certs
        - name: ratelimit
          image: envoyproxy/ratelimit:6f5de117 # 2021/01/08
          imagePullPolicy: Always
          command: ['/bin/ratelimit']
          env:
            - name: LOG_LEVEL
              value: debug
            - name: REDIS_SOCKET_TYPE
              value: tcp
            - name: REDIS_URL
              value: redis:6379
            - name: USE_STATSD
              value: 'false'
            - name: RUNTIME_ROOT
              value: /data
            - name: RUNTIME_SUBDIRECTORY
              value: ratelimit
          ports:
            - containerPort: 8080
            - containerPort: 8081
            - containerPort: 6070
          volumeMounts:
            - name: ratelimit-config
              mountPath: /data/ratelimit/config/config.yaml
              subPath: config.yaml
      volumes:
        - name: ratelimit-config
          configMap:
            name: ratelimit-config
        - name: proxy-certs
          secret:
            secretName: ratelimit-certs
        - name: proxy-config
          configMap:
            name: ratelimit-proxy

Then apply this using kubectl:

kubectl apply -f ratelimit-tls.yaml

Once you applied the new configuration, make sure that the ratelimit-tls service is running properly. Note that if you have followed the instructions from Setting Up an External Rate Limiting Server, you will also see ratelimit and redis services as well.

kubectl get pods -n ext-ratelimit

Output

NAME                             READY   STATUS    RESTARTS   AGE
ratelimit-d5c5b64ff-m87dt        1/1     Running   0          2h
ratelimit-tls-568c5cdc69-z82xf   2/2     Running   0          89s
redis-7d757c948f-42sxg           1/1     Running   0          2h

Enable TLS validation for rate limit server in Ingress Gateway

The ratelimit-tls service can now terminate TLS, but the Ingress Gateway must also be configured to validate the TLS connections.

First, create a ConfigMap named ratelimit-ca to store the CA information from ratelimit-ca.crt:

kubectl create configmap -n httpbin ratelimit-ca \
  --from-file=ratelimit-ca.crt

Then add the ratelimit-ca ConfigMap into the Ingress Gateway pod. To do this, you will need to edit the httpbin-ingress-gateway.yaml file and add an overlay that reads the ConfigMap you have created in the previous steps, then mount the configuration in the ingress gateway deployment.

apiVersion: install.tetrate.io/v1alpha1
kind: IngressGateway
metadata:
  name: httpbin-ingress-gateway
  namespace: httpbin
spec:
  kubeSpec:
    service:
      type: LoadBalancer
    overlays:
      - apiVersion: apps/v1
        kind: Deployment
        name: httpbin-ingress-gateway
        patches:
          - path: spec.template.spec.volumes[-1]
            value:
              name: ratelimit-ca
              configMap:
                name: ratelimit-ca
          - path: spec.template.spec.containers.[name:istio-proxy].volumeMounts[-1]
            value:
              name: ratelimit-ca
              mountPath: /etc/certs
              readOnly: true

Apply with kubectl to update existing ingress gateway

kubectl apply -f httpbin-ingress-gateway.yaml

Finally, update the Ingress Gateway configuration in ext-ratelimit-ingress-gateway.yaml and enable TLS validation:

apiVersion: gateway.tsb.tetrate.io/v2
kind: IngressGateway
metadata:
  name: httpbin-gateway # Need not be the same as spec.labels.app
  organization: tetrate
  tenant: tetrate
  group: httpbin-gateway
  workspace: httpbin
spec:
  workloadSelector:
    namespace: httpbin
    labels:
      app: httpbin-ingress-gateway # name of Ingress Gateway created for httpbin
  http:
    - name: httpbin
      hostname: 'httpbin.tetrate.com'
      port: 80
      routing:
        rules:
          - route:
              host: 'httpbin/httpbin.httpbin.svc.cluster.local'
              port: 8000
      rateLimiting:
        externalService:
          domain: 'httpbin-ratelimit'
          rateLimitServerUri: 'grpcs://ratelimit-tls.ext-ratelimit.svc.cluster.local:18443'
          failClosed: true
          tls:
            mode: SIMPLE
            files:
              caCertificates: /etc/certs/ratelimit-ca.crt
          rules:
            - dimensions:
                - requestHeaders:
                    headerName: ':path'
                    descriptorKey: request-path

And apply with tctl

tctl apply -f ext-ratelimit-ingress-gateway-tls.yaml

Testing

To verify that the setup is working, you can use the same testing steps as shown in the Testing steps for "Setting Up an External Rate Limiting Server