TEG Demo Install Guide
This guide is aimed at Engineers who are starting out with Kubernetes and the Gateway API and wish to try TEG and learn more about its features.
At the end of this guide you will have:
- TEG running in your cluster
httpbin
deployed into its own namespace- A dedicated gateway for Httpbin
- A setup ready for you to continue to learn and experiment with the features of TEG
If you are experienced with Kubernetes, the Gateway API and Envoy Gateway, feel free to head straight to the Installation Guide to dive deeper straight away.
The TEG demo installation chart comes with the necessary configurations to set up TEG and the required dependencies for rate limiting and observability.
The demo installation chart installs:
- TEG
- Redis for rate limiting
- Grafana for visualisations
- Loki as the log collection backend
- Tempo as the tracing backend
- Prometheus as metrics backend
- Demo app for testing traffic
Prerequisitesβ
There are two options for you to try the demo out:
- On a hosted cluster
- On a cluster on your local machine
Demo Setup on a Hosted Clusterβ
- A cluster you can administer and apply changes to
- Refer to the π Compatibility Matrix before proceeding
Demo Setup on your localβ
If you are trying this out on a local machine, make sure you are ready to get started.
This guide requires that you have the following installed:
Let's create a clusterβ
- Kind
- Minikube
kind create cluster
minikube start
Deploy the Demo Setupβ
In this section, we will:
- Install TEG
- Deploy
httpbin
- Set up a dedicated TEG gateway for
httpbin
- Send a request to
httpbin
via the gateway
Install TEGβ
Now we will install TEG using the teg-demo-helm chart.
helm install teg oci://docker.io/tetrate/teg-demo-helm \
--version v1.1.2 \
-n envoy-gateway-system --create-namespaceTo make it easy to kick the tires, TEG also comes distributed as a demonstration helm chart.
Next to TEG, the demonstration helm chart installs a full Prometheus + Grafana observability stack with ephemeral storage.
Check out what is running
kubectl get pod -n envoy-gateway-system
Example output
NAME READY STATUS RESTARTS AGE
envoy-default-eg-e41e7b31-7fd5d4b467-6tf29 2/2 Running 0 6m44s
envoy-gateway-7d8686df-mwz7f 1/1 Running 0 6m52s
envoy-httpbin-dedicated-gateway-c4239473-7645d8674b-sg9v7 2/2 Running 0 5m2s
envoy-ratelimit-7b7d5c56d9-jjplr 2/2 Running 1 (6m39s ago) 6m45s
fluent-bit-qdfxs 1/1 Running 0 6m53s
grafana-77b8b78748-nqc7m 1/1 Running 0 6m53s
loki-0 1/1 Running 0 6m52s
prometheus-86cf4c9969-l2nmd 2/2 Running 0 6m53s
teg-redis-7d56f4c49-x9b2b 1/1 Running 0 6m53s
teg-envoy-gateway-f5cfcd787-x2mk9 1/1 Running 0 6m53s
tempo-0 1/1 Running 0 6m52sDeploy an Appβ
We are going to deploy httpbin to the cluster in a namespace called httpbin.
1. Create Namespace
kubectl create namespace httpbin
2. Deploy sample
kubectl apply -n httpbin -f https://raw.githubusercontent.com/istio/istio/master/samples/httpbin/httpbin.yaml
3. Verify Deployment
kubectl get pod -n httpbin
You should see a httpbin pod running in the httpbin namespace.
It's name will look something like this:
httpbin-86b8ffc5ff-w2zlb
Deploy a Gatewayβ
- Apply from stdin
- Apply from file
In this example you will be applying configurations using stdin, for applying from file select the other tab above.
Apply the configβ
Copy and paste the following into your terminal.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: dedicated-gateway
namespace: httpbin
spec:
gatewayClassName: teg
listeners:
- name: http
protocol: HTTP
port: 80
EOFCreate a file for the configurationsβ
In this example you will using a file for the configurations of the Gateway, and apply changes from this file.
Create a file, in this example we use the file name:
dedicated-gateway.yaml
touch dedicated-gateway.yaml
infoCreate it in a location that is easy to reference from the Terminal.
You will notice that this tutorial uses a single file for the configuration you will be applying, we do not recommend that you use this practice outside a demo environment
Populate the config fileβ
This is the beginning of the config map that we will continue to build on through this guide.
Open the file and add the below yaml, my preferred terminal editor is
nano
but you might prefervim
. You may also open the file in a desktop editor such as VS Code.# This section defines the dedicated gateway
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: dedicated-gateway
namespace: httpbin
spec:
gatewayClassName: teg
listeners:
- name: http
protocol: HTTP
port: 80Apply the configβ
Note The file location may vary on your machine, make sure you have the correct path
kubectl apply -n httpbin -f dedicated-gateway.yaml
Verifyβ
kubectl get pods -n envoy-gateway-system \
-l gateway.envoyproxy.io/owning-gateway-namespace=httpbinConfigure the Gatewayβ
- Apply from stdin
- Apply from file
Apply configβ
cat <<EOF | kubectl apply -n httpbin -f -
apiVersion: gateway.networking.k8s.io/v1beta1
# This is a simple HTTPRoute
kind: HTTPRoute
metadata:
name: httpbin
namespace: httpbin
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: dedicated-gateway
rules:
- matches:
- path:
type: PathPrefix
value: /httpbin/
filters:
- type: URLRewrite
urlRewrite:
path:
type: ReplacePrefixMatch
replacePrefixMatch: /
backendRefs:
- group: ""
kind: Service
name: httpbin
port: 8000
EOFAdd config to fileβ
Update your config map file to have the additional routing logic included. Below is what the file should now look like:
# This section defines the dedicated gateway
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: dedicated-gateway
namespace: httpbin
spec:
gatewayClassName: teg
listeners:
- name: http
protocol: HTTP
port: 80
---
# This section includes the routing logic
apiVersion: gateway.networking.k8s.io/v1beta1
# This is a simple HTTPRoute
kind: HTTPRoute
metadata:
name: httpbin
namespace: httpbin
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: dedicated-gateway
rules:
- matches:
- path:
type: PathPrefix
value: /httpbin/
filters:
- type: URLRewrite
urlRewrite:
path:
type: ReplacePrefixMatch
replacePrefixMatch: /
backendRefs:
- group: ""
kind: Service
name: httpbin
port: 8000Apply Configβ
Donβt forget to apply the config.
kubectl apply -n httpbin -f dedicated-gateway.yaml
Check your Gatewayβ
π Letβs check your Gateway and letβs find out whether you have an LB or notΒ (if you donβt already know!)
kubectl get gateway -n httpbin
ADDRESS
will be populated if you have a LoadBalancer (LB) and if it isnβt, you do not have an LB, but donβt worry, this guide will help you work around that on your machine with port forwarding.If you have an LB - ADDRESS is populatedβ
You will see
PROGRAMMED = True
andADDRESS
populatedNAME CLASS ADDRESS PROGRAMMED AGE
dedicated-gateway teg 35.238.21.86 True 2m28sVerifyβ
kubectl get svc -n envoy-gateway-system \
-l gateway.envoyproxy.io/owning-gateway-namespace=httpbinOutput Example You should see an
EXTERNAL-IP
populated if this is working as expected that matches theADDRESS
outputted from thekubectl get gateway -n httpbin
command.NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
envoy-httpbin-dedicated-gateway-c4239473 LoadBalancer 10.0.7.31 35.238.21.86 80:31583/TCP 5m17sIf you do NOT have an LB - ADDRESS is not populatedβ
You will see
PROGRAMMED = False
and thatADDRESS
is not populated.NAME CLASS ADDRESS PROGRAMMED AGE
dedicated-gateway teg False 13mnoteThis is likely to happen on a local machine when you are using something like Kind or Minikube
Do not fret, follow the instructions below for to set up Port Forwarding for your local machine with no LB to try out Envoy Gateway on your local machine.
Set the
$GATEWAY_ADDRESS
βChoose the tab below that applies to your setup to configure
$GATEWAY_ADDRESS
that will set you up to try the- For LB with External IP
- For LB with Hostname
- For Local with no LB - Port forwarding
We will set the
$GATEWAY_ADDRESS
to the external IP.export GATEWAY_ADDRESS=$(kubectl get gateway/dedicated-gateway -n httpbin -o jsonpath='{.status.addresses[0].value}')
If your provider does not provide an ingress IP address, but rather a hostname (e.g., AWS) use the following to set the
$GATEWAY_ADDRESS
:export GATEWAY_ADDRESS=$(kubectl get service -n envoy-gateway-system -l gateway.envoyproxy.io/owning-gateway-namespace=httpbin -o jsonpath='{.items[0].status.loadBalancer.ingress[0].hostname}')
Setup Port Forwarding on Localβ
Set
$ENVOY_SERVICE
variableβexport ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=httpbin,gateway.envoyproxy.io/owning-gateway-name=dedicated-gateway -o jsonpath='{.items[0].metadata.name}')
Verifyβ
echo $ENVOY_SERVICE
Configure port forwardingβ
kubectl -n envoy-gateway-system port-forward service/${ENVOY_SERVICE} 8899:80 &
For port-forwarding on Local with no loadbalancerβ
Set the
$GATEWAY_ADDRESS
βAssuming you set the port to
8899
from the instructions above, copy and paste the following.export GATEWAY_ADDRESS=localhost:8899
Verifyβ
echo $GATEWAY_ADDRESS
Check the output to ensure it is what you expect, it should be set.
Try your TEG Demo Setupβ
Now when youβve set the
$GATEWAY_ADDRESS
we are going to try it out!CURL
curl -i http://${GATEWAY_ADDRESS}/httpbin/get
Example Output
HTTP/1.1 200 OK
server: envoy
date: Tue, 02 Apr 2024 15:48:29 GMT
content-type: application/json
content-length: 418
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 15
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "localhost:8899",
"Traceparent": "00-d9eaaa5ed02283c641802dd30b7c58c0-a4ec6f331ca7373f-01",
"Tracestate": "",
"User-Agent": "curl/8.4.0",
"X-Envoy-Expected-Rq-Timeout-Ms": "15000",
"X-Envoy-Internal": "true",
"X-Envoy-Original-Path": "/httpbin/get"
},
"origin": "10.244.0.29",
"url": "http://localhost:8899/get"
}TEG Demo is upβ
Congratulations! π
You now have the TEG Demo running and can start exploring policies and routing logic, as well as observe traffic and performance in Grafana.
Explore Policiesβ
First you configure policies and then you apply them to the Routes.
Below is an example on how to get started with a Rate-Limiting configuration and apply it to the route.
Configuring ratelimitingβ
What is it? First, we will configure a ratelimiting setting using BackendTrafficPolicy. This states that any endpoint subject to it will be limited to 1 request per second.
Configure Policyβ
- Apply from stdin
- Apply from file
Copy and paste the following into your terminal to apply the policy.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
namespace: httpbin
name: ratelimit-1hz
spec:
targetRef:
group: gateway.networking.k8s.io
kind: HTTPRoute
name: httpbin
rateLimit:
type: Global
global:
rules:
- limit:
requests: 1
unit: Second
EOFNow letβs add the config to the same file that we have been working on, it should look like this now:
# This section defines the dedicated gateway
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: dedicated-gateway
namespace: httpbin
spec:
gatewayClassName: teg
listeners:
- name: http
protocol: HTTP
port: 80
---
# This section includes the routing logic
apiVersion: gateway.networking.k8s.io/v1beta1
# This is a simple HTTPRoute
kind: HTTPRoute
metadata:
name: httpbin
namespace: httpbin
spec:
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: dedicated-gateway
rules:
- matches:
- path:
type: PathPrefix
value: /httpbin/
filters:
- type: URLRewrite
urlRewrite:
path:
type: ReplacePrefixMatch
replacePrefixMatch: /
backendRefs:
- group: ""
kind: Service
name: httpbin
port: 8000
---
# This section includes the rate limiting policy
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
namespace: httpbin
name: ratelimit-1hz
spec:
targetRef:
group: gateway.networking.k8s.io
kind: HTTPRoute
name: httpbin
rateLimit:
type: Global
global:
rules:
- limit:
requests: 1
unit: SecondLetβs apply the config
kubectl apply -n httpbin -f dedicated-gateway.yaml
Letβs try the RateLimiterβ
Letβs send too many requests and observe the output!
curl -i http://$GATEWAY_ADDRESS/httpbin/get ; echo β===β ; \
curl -i http://$GATEWAY_ADDRESS/httpbin/get ;Output example
HTTP/1.1 200 OK
server: envoy
date: Tue, 02 Apr 2024 16:43:36 GMT
content-type: application/json
content-length: 418
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 9
x-ratelimit-limit: 1, 1;w=1
x-ratelimit-remaining: 0
x-ratelimit-reset: 1
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "localhost:8899",
"Traceparent": "00-1e9bbc4981b753c19e5e2dc21350f166-0c07d3af5a6933c3-01",
"Tracestate": "",
"User-Agent": "curl/8.4.0",
"X-Envoy-Expected-Rq-Timeout-Ms": "15000",
"X-Envoy-Internal": "true",
"X-Envoy-Original-Path": "/httpbin/get"
},
"origin": "10.244.0.29",
"url": "http://localhost:8899/get"
}
β===β
HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
x-ratelimit-limit: 1, 1;w=1
x-ratelimit-remaining: 0
x-ratelimit-reset: 1
date: Tue, 02 Apr 2024 16:43:36 GMT
server: envoy
content-length: 0
Observe traffic and performanceβ
See stats in Grafanaβ
TEG bundles a fully loaded Prometheus + Grafana stack with the demo helm install so you can immediately test drive TEG's observability capabilities.
Set up port forwarding for Grafanaβ
To access the Grafana deployment you can establish a port-forward tunnel to the Grafana dashboard.
kubectl port-forward -n envoy-gateway-system deployment/grafana 3001:3000
Visit http://localhost:3001 and use the following credentials to login:
Username admin
Password admin
Go to one of the dashboards and check it out, try this one: [Envoy Pod Memory and CPU Usage][11]
For better visualizations, you can try running continuous traffic to the demo application by running the following in a separate process:
while true; do curl -i http://${GATEWAY_ADDRESS}/httpbin/get; sleep 5; done
Metrics Dashboardsβ
The Grafana deployments come configured with two dashboards:
Envoy Global Dashboardβ
The dashboard highlights aggregated metrics for all of your envoy proxy deployments, visualizing metrics:
- Uptime
- Resource consumption metrics
- Upstream and Downstream connection and request metrics
- Number of Active connections
- Healthy Endpoints
- HTTP latencies
- TCP bytes received and transmitted
Envoy Clusters Dashboardβ
This dashboard highlights metrics for clusters configured in your envoy proxy configuration, visualizing the metrics listed above for each cluster.
Logs visualizationβ
The demo chart configures the Loki data source with the Grafana dashboard. Users can dig into the access logs for the proxies by using the Explore tab in the Grafana deployment and selecting the Loki data source.
Tracing Visualizationβ
The demo chart also configures the Tracing data source with the Grafana dashboard. Users can dig into the access logs for the proxies by using the Explore tab in the Grafana deployment and selecting the Tempo data source.