Resource Consumption and Capacity Planning

This document describes a conservative guideline for capacity planning of Tetrate Service Bridge (TSB) in Management and Control planes.

These parameters apply to production installations: TSB will run with minimal resources if you are using a demo-like environment.

disclaimer

The resource provisioning guidelines described in this document are very conservative.

Also please be aware that the resource provisioning described in this document are applicable to vertical resource scaling. Multiple replicas of the same TSB components do not share the load with each other, and therefore you cannot expect the combined resources from multiple components to have the same effect. Replicas of TSB components should only be used for high availability purposes only.

Recommended baseline production installation resource requirements

For a baseline installation of TSB with 100 clusters and 1000 services, the following resources are recommended.

To reiterate, the amount of memory described below are conservative. Also, the actual performance given by the number of vCPUs tend to fluctuate depending on your underlying infrastructure. You are advised to verify the results in your environment.

info

If you are using a Tetrate-hosted Management Plane, then the rows for the Management Plane and the XCP Central should be ignored

Component	vCPU #	Memory MiB
TSB server (tsb)	2	2048
TSB operator (tsb-operator-management-plane)	0.5	512
XCP Central (central)	2	2048
XCP operator (xcp-operator-central)	0.125	128
Front Envoy (envoy)	2	512
IAM (iam)	0.25	128
TSB UI (web)	0.125	256
OAP (oap)	4	10240
OTEL Collector (otel-collector)	0.25	512
MPC (mpc)	1	2048

Recommended scaling resource parameters

The TSB stack is mostly CPU-bound. Additional clusters registered with TSB via XCP increase the CPU utilization by ~4%.

The effect of additional registered clusters or additional deployed workload services on memory utilisation is almost negligible. Likewise, the effect of additional clusters or workloads on resource consumption of the majority of TSB components is mostly negligible, with the notable exceptions of TSB, XCP Central component, TSB UI and IAM.

note

Components that are part of the visibility stack (e.g. OTel/OAP, etc.) have their resource utilisation driven by requests, thus the resource scaling should follow the user request rate statistics. As a general rule of thumb, more than 1 vCPU is preferred. It is also important to notice that the visibility stack performance is largely bound by Elasticsearch performance.

Thus, we recommend vertically scaling the components by 1 vCPU for a number of deployed workflows:

Management Plane

info

If you are using a Tetrate-hosted Management Plane, then you can skip this section

Besides OAP, All components don't require any resource adjustment. Those components are architectured and tested to support very large clusters.

OAP in Management plane requires extra CPU and Memory ~ 100 millicores of CPU and 1024 MiB of RAM per every 1000 services. E.g. 4000 services aggregated in TSB Management Plane from all TSB clusters would require approximately 400 millicores of CPU and 4096 MiB of RAM in total.

Control Plane Resource Requirements

Following table shows typical peak resource utilization for TSB control plane with the following assumptions:

• 4,000 services with sidecars
• Traffic on entire cluster is 500 requests-per-second
• OAP trace sampling rate is 1% of the traffic
• Metric is captured for every request at every workload.

Note that average CPU utilization would be a fraction of the typical peak value.

Component	Typical Peak CPU (vCPU)	Typical Peak Memory (Mi)
Istiod (istiod)	1	2192
OAP (oap-deployment)	3	6144
XCP Edge (edge)	1	1024
XCP Operator (xcp-operator-edge)	1	256
TSB Control Plane Operator (tsb-operator-control-plane)	0.25	256
OTEL Collector (otel-collector)	0.25	256

Multiple Istio Revisions

Please be aware that when Isolation Boundary is configured with multiple boundaries in the ControlPlane configuration, a dedicated Istiod will be spawned for each revisions under an IsolationBoundary.

TSB/XCP Operator resource usage per Ingress Gateway

The following table shows the resources used by TSB Operator and Istio Operator per Ingress Gateways

Ingress Gateways	TSB Operator vCPU (m)	TSB Operator Mem (Mi)	XCP Operator vCPU (m)	XCP Operator Mem (Mi)
0	30m	120Mi	10m	75Mi
50	200m	165Mi	1121m	111Mi
100	300m	177Mi	1154m	138Mi
150	400m	191Mi	1582m	143Mi
200	450m	194MI	1675m	145Mi
250	475m	197Mi	1714m	171Mi
300	500m	198Mi	1733m	218Mi

Component resource utilization

The following tables will show how the different components of TSB scale with 4000 services and peaking with 60 rpm, this is divided by information from the Management Plane, and the Control Plane.

Management Plane

info

If you are using a Tetrate-hosted Management Plane, then skip this section

Services	Gateways	Traffic(rpm)	Central CPU(m)	Central Mem(Mi)	MPC CPU(m)	MPC Mem(Mi)	OAP CPU(m)	OAP Mem(Mi)	Otel CPU(m)	Otel Mem(Mi)	TSB CPU(m)	TSB Mem(Mi)
0	0	0 rpm	10m	74.3Mi	5m	75.1Mi	35m	408Mi	60m	200Mi	14m	93MI
400	2	60 rpm	21m	76.9Mi	15m	72.2Mi	156m	1650Mi	65m	220Mi	50m	95MI
800	4	60 rpm	25m	91.8Mi	24m	85.4Mi	170m	1800Mi	55m	225Mi	85m	102MI
1200	6	60 rpm	30m	109Mi	32m	92.1Mi	103m	1580Mi	75m	210Mi	213m	112MI
1600	8	60 rpm	50m	138Mi	44m	106Mi	114m	1930Mi	80m	240Mi	113m	120Mi
2000	10	60 rpm	55m	151Mi	41m	112Mi	180m	1750Mi	50m	220Mi	211m	135MI
2400	12	60 rpm	55m	170Mi	72m	126Mi	161m	1880Mi	85m	240Mi	227m	128Mi
2800	14	60 rpm	70m	180Mi	73m	137Mi	129m	2500Mi	95m	220Mi	275m	153Mi
3200	16	60 rpm	80m	198Mi	85m	137Mi	198m	2380Mi	65m	240Mi	382m	178Mi
3600	18	60 rpm	125m	211Mi	94m	146Mi	284m	2240Mi	120m	220Mi	625m	182Mi
4000	20	60 rpm	130m	221Mi	90m	151Mi	358m	2210Mi	110m	240Mi	508m	190Mi

note

IAM will peak at 5m/32Mi, LDAP at 1m/12Mi and XCP Operator at 3m and 23Mi

Control Plane

Services	Gateways	Traffic(rpm)	Edge CPU(m)	Edge Mem(Mi)	Istiod CPU(m)	Istiod Mem(Mi)	OAP CPU(m)	OAP Mem(Mi)	Otel CPU(m)	Otel Mem(Mi)
0	0	0 rpm	45m	77.3 MB	6m	110Mi	601m	2.05 GB	16m	74Mi
400	2	60 rpm	45m	194 MB	33m	182Mi	1011m	2.15 GB	18m	75Mi
800	4	60 rpm	110m	271 MB	35m	249Mi	1070m	3.11 GB	21m	85Mi
1200	6	60 rpm	125m	366 MB	68m	286Mi	2084m	4.78 GB	23m	90Mi
1600	8	60 rpm	138m	475 MB	84m	324Mi	2673m	5.79 GB	20m	75Mi
2000	10	60 rpm	156m	595 MB	84m	357Mi	2292m	8.12 GB	17m	85Mi
2400	12	60 rpm	235m	676 MB	98m	370Mi	2534m	8.42 GB	19m	85Mi
2800	14	60 rpm	289m	778 MB	334m	450Mi	3174m	9.02 GB	19m	75Mi
3200	16	60 rpm	425m	870 MB	243m	470Mi	2657m	9.32 GB	20m	80Mi
3600	18	60 rpm	533m	948 MB	130m	489Mi	2899m	9.95 GB	20m	100Mi
4000	20	60 rpm	713m	958 MB	138m	523Mi	3275m	10.1 GB	20m	110Mi

note

Metric Server will peak at 4m/24Mi, Onboarding Operator at 4m/24Mi, and XCP-Operator at 3m/22Mi

HPA (Horizontal Pod Autoscaling)

Management Plane

Component	HPA
TSB server (tsb)	No
TSB operator (tsb-operator-management-plane)	No
XCP Central (central)	No
XCP operator (xcp-operator-central)	No
Front Envoy (envoy)	Yes
IAM (iam)	No
TSB UI (web)	No
OAP (oap)	Yes
OTEL Collector (otel-collector)	Not supported - use vertical scaling
MPC (mpc)	No

Control Plane

Component	HPA
Istiod (istiod)	Yes, but autoscaling not recommended (workload is spiky); deploy replicas if needed
OAP (oap-deployment)	Yes
XCP Edge (edge)	No
XCP Operator (xcp-operator-edge)	No
TSB Control Plane Operator (tsb-operator-control-plane)	No
OTEL Collector (otel-collector)	Not supported - use vertical scaling