Configuring Application Gateways Using OpenAPI Annotations
An Application in TSB represents a set of logical groupings of Services that are related to each other and expose a set of APIs that implement a complete set of business logic.
TSB can leverage OpenAPI annotations when configuring API runtime policies. In this document you will enable authorization via Open Policy Agent (OPA), as well as rate limiting through an external service. Each request will need to go through basic authorization, and for each valid user a rate limit policy will be enforced.
Before you get started, make sure you:
✓ Familiarize yourself with TSB concepts
✓ Familiarize yourself Open Policy Agent (OPA)
✓ Familiarize yourself Envoy external authorization and rate limit
✓ Install the TSB demo environment
✓ Deploy the Istio Bookinfo sample app
✓ Create a Tenant
Deploy httpbin
Service
Follow the instructions in this document to create the httpbin
service. Complete all steps in that document.
TSB Specific Annotations
The following extra TSB-specific annotations can be added to the OpenAPI specification in order to configure the API.
Annotation | Description |
---|---|
x-tsb-service | The upstream service name in TSB that provides the API, as seen in the TSB service registry (you can check with tctl get services ). |
x-tsb-cors | The CORS policy for the server. |
x-tsb-tls | The TLS settings for the server. If omitted, the server will be configured to serve plain text connections. The secretName field must point to an existing Kubernetes secret in the cluster. |
x-tsb-external-authorization | The OPA settings for the server. |
x-tsb-ratelimiting | The external rate limit server (e.g. envoyproxy/ratelimit settings. |
Configure the API
Create the following API definition in a file called httpbin-api.yaml
.
In this scenario you will only use one of the APIs (/get
) that the httpbin
service provides. If you want to use all of the httpbin
API you can get their OpenAPI specifications from this link.
apiversion: application.tsb.tetrate.io/v2
kind: API
metadata:
organization: <organization>
tenant: <tenant>
application: httpbin
name: httpbin-api
spec:
description: Httpbin OpenAPI
workloadSelector:
namespace: httpbin
labels:
app: httpbin-gateway
openapi: |
openapi: 3.0.1
info:
version: '1.0-oas3'
title: httpbin
description: An unofficial OpenAPI definition for httpbin
x-tsb-service: httpbin.httpbin
servers:
- url: https://httpbin.tetrate.com
x-tsb-cors:
allowOrigin:
- "*"
x-tsb-tls:
mode: SIMPLE
secretName: httpbin-certs
paths:
/get:
get:
tags:
- HTTP methods
summary: |
Returns the GET request's data.
responses:
'200':
description: OK
content:
application/json:
schema:
type: object
Apply using tctl
:
tctl apply -f httpbin-api.yaml
At this point, you should be able to send requests to the httpbin
Ingress Gateway.
Since you do not control httpbin.tetrate.com
, you will have to trick curl
into thinking that httpbin.tetrate.com
resolves to the IP address of the Ingress Gateway.
Obtain the IP address of the Ingress Gateway that you previously created using the following command.
kubectl -n httpbin get service httpbin-gateway -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
Execute the following command to send HTTP requests to the httpbin
service through the Tier-1 Gateway. Replace the gateway-ip
with the value you obtained in the previous step. You also need to pass the CA cert, which you should have created in the step to deploy the httpbin
service.
curl -I "https://httpbin.tetrate.com/get" \
-H "X-B3-Sampled: 1" \
--resolve "httpbin.tetrate.com:443:<gateway-ip>" \
--cacert httpbin.crt
You should see a successful HTTP response.
Authorization with OPA
Once the API is properly exposed via OpenAPI annotations, OPA can be configured against the API Gateway.
In this example you will create a policy that checks for basic authentication in the request header. If the user is authenticated, the user name should be added to the x-user
header so that it can be used by the rate limit service to enforce quota for each user later.
Configure OPA
Create the opa
namespace, where the OPA and its configuration will be deployed to:
kubectl create namespace opa
Create a file name openapi-policy.rego
:
package demo.authz
default allow = false
# username and password database
user_passwords = {
"alice": "password",
"bob": "password"
}
allow = response {
# check if password from header is same as in database for the specific user
basic_auth.password == user_passwords[basic_auth.user_name]
response := {
"allowed": true,
"headers": {"x-user": basic_auth.user_name}
}
}
basic_auth := {"user_name": user_name, "password": password} {
v := input.attributes.request.http.headers.authorization
startswith(v, "Basic ")
s := substring(v, count("Basic "), -1)
[user_name, password] := split(base64url.decode(s), ":")
}
Then create a ConfigMap
using the file you created:
kubectl -n opa create configmap opa-policy \
--from-file=openapi-policy.rego
Create the Deployment
and the Service
objects that use the above policy configuration in a file named opa.yaml
.
apiVersion: v1
kind: Service
metadata:
name: opa
namespace: opa
spec:
selector:
app: opa
ports:
- name: grpc
protocol: TCP
port: 9191
targetPort: 9191
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: opa
namespace: opa
spec:
replicas: 1
selector:
matchLabels:
app: opa
template:
metadata:
labels:
app: opa
name: opa
spec:
containers:
- image: openpolicyagent/opa:0.29.4-envoy-2
name: opa
securityContext:
runAsUser: 1111
ports:
- containerPort: 8181
args:
- 'run'
- '--server'
- '--addr=localhost:8181'
- '--diagnostic-addr=0.0.0.0:8282'
- '--set=plugins.envoy_ext_authz_grpc.addr=:9191'
- '--set=plugins.envoy_ext_authz_grpc.path=demo/authz/allow'
- '--set=decision_logs.console=true'
- '--ignore=.*'
- '/policy/openapi-policy.rego'
livenessProbe:
httpGet:
path: /health?plugins
scheme: HTTP
port: 8282
initialDelaySeconds: 5
periodSeconds: 5
readinessProbe:
httpGet:
path: /health?plugins
scheme: HTTP
port: 8282
initialDelaySeconds: 5
periodSeconds: 5
volumeMounts:
- readOnly: true
mountPath: /policy
name: opa-policy
volumes:
- name: opa-policy
configMap:
name: opa-policy
Then apply the manifest:
kubectl apply -f opa.yaml
Finally, open the httpbin-api.yaml
file that you created in a previous section, and add the x-tsb-external-authorization
annotation in the server
component:
...
servers:
- url: https://httpbin.tetrate.com
...
x-tsb-external-authorization:
uri: grpc://opa.opa.svc.cluster.local:9191
And apply the changes again:
tctl apply -f httpbin-api.yaml
Testing
To test, execute the following command, replacing the values for username, password, and gateway-ip accordingly.
curl -u <username>:<password> \
"https://httpbin.tetrate.com/get" \
--resolve "httpbin.tetrate.com:443:<gateway-ip>" \
--cacert httpbin.crt \
-s \
-o /dev/null \
-w "%{http_code}\n" \
-H "X-B3-Sampled: 1"
Username | Password | Status Code |
---|---|---|
alice | password | 200 |
bob | password | 200 |
<anything else> | <anything else> | 403 (*1) |
(*1) See documentation for more details
Rate limiting with external service
TSB supports internal and external mode for rate limiting. In this example you will deploy a separate Envoy rate limit service.
Configure Rate Limit
Create the ext-ratelimit
namespace, where the rate limit server and its configuration will be deployed to:
kubectl create namespace ext-ratelimit
Create a file name ext-ratelimit-config.yaml
. This configuration specifies that the user alice
should be rate limited to 10 requests/minute and user bob
should be limited to 2 requests/minute.
domain: httpbin-ratelimit
descriptors:
- key: x-user-descriptor
value: alice
rate_limit:
unit: minute
requests_per_unit: 10
- key: x-user-descriptor
value: bob
rate_limit:
unit: minute
requests_per_unit: 2
Then create a ConfigMap
using the file you created:
kubectl -n ext-ratelimit create configmap ext-ratelimit \
--from-file=config.yaml=ext-ratelimit-config.yaml
You now need to deploy Redis and envoyproxy/ratelimit
. Create a file called redis-ratelimit.yaml
with the following contents:
# Copyright Istio Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
####################################################################################
# Redis service and deployment
# Ratelimit service and deployment
####################################################################################
apiVersion: v1
kind: Service
metadata:
name: redis
namespace: ext-ratelimit
labels:
app: redis
spec:
ports:
- name: redis
port: 6379
selector:
app: redis
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis
namespace: ext-ratelimit
spec:
replicas: 1
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
spec:
containers:
- image: redis:alpine
imagePullPolicy: Always
name: redis
ports:
- name: redis
containerPort: 6379
restartPolicy: Always
serviceAccountName: ''
---
apiVersion: v1
kind: Service
metadata:
name: ratelimit
namespace: ext-ratelimit
labels:
app: ratelimit
spec:
ports:
- name: http-port
port: 8080
targetPort: 8080
protocol: TCP
- name: grpc-port
port: 8081
targetPort: 8081
protocol: TCP
- name: http-debug
port: 6070
targetPort: 6070
protocol: TCP
selector:
app: ratelimit
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: ratelimit
namespace: ext-ratelimit
spec:
replicas: 1
selector:
matchLabels:
app: ratelimit
strategy:
type: Recreate
template:
metadata:
labels:
app: ratelimit
spec:
containers:
- image: envoyproxy/ratelimit:6f5de117 # 2021/01/08
imagePullPolicy: Always
name: ratelimit
command: ['/bin/ratelimit']
env:
- name: LOG_LEVEL
value: debug
- name: REDIS_SOCKET_TYPE
value: tcp
- name: REDIS_URL
value: redis:6379
- name: USE_STATSD
value: 'false'
- name: RUNTIME_ROOT
value: /data
- name: RUNTIME_SUBDIRECTORY
value: ratelimit
ports:
- containerPort: 8080
- containerPort: 8081
- containerPort: 6070
volumeMounts:
- name: config-volume
mountPath: /data/ratelimit/config/config.yaml
subPath: config.yaml
volumes:
- name: config-volume
configMap:
name: ratelimit-config
If everything is successful, you should have a working rate limit server. The next step is to add an annotation for x-tsb-ratelimiting
to the OpenAPI object:
Next, update your OpenAPI spec by adding the following x-tsb-ratelimiting
annotation in OpenAPI server object
...
servers:
- url: https://httpbin.tetrate.com
...
x-tsb-external-ratelimiting:
domain: "httpbin-ratelimit"
rateLimitServerUri: "grpc://ratelimit.ext-ratelimit.svc.cluster.local:8081"
rules:
- dimensions:
- requestHeaders:
headerName: x-user
descriptorKey: x-user-descriptor
...
Testing
To test, execute the following command, replacing the values for username, password, and gateway-ip accordingly.
curl -u <username>:<password> \
"https://httpbin.tetrate.com/get" \
--resolve "httpbin.tetrate.com:443:<gateway-ip>" \
--cacert httpbin.crt \
-s \
-o /dev/null \
-w "%{http_code}\n" \
-H "X-B3-Sampled: 1"
First, try sending multiple requests using the username alice
and password password
. You should receive the status code 200
until the 10th request. After that you should receive 429
responses, until 10 minutes have passed.
Try the same with username bob
and password password
. The behavior should be identical, except that this time you should only be able to send 2 requests before starting to receive 429
responses.
Policy ordering
TSB does not currently support specifying explicit policy order.
Instead, the creation timestamp for the configurations will implicitly be used. Therefore, the execution order cannot be guaranteed if you specify the external authorization and rate limiting services in one go.
This is why in this document the external authorization and rate limiting configurations were applied in two separate steps, in this specific order. This way the authorization processing is performed before the rate limiting.