HPA using SkyWalking

Apache SkyWalking Cloud on Kubernetes (SWCK) provides an external metrics adapter from which the Kubernetes Horizontal Pod Autoscaling (HPA) controller can retrieve metrics from. Users may deploy SWCK adapter into the TSB control plane in order to fetch target metrics from the Observability Analysis Platform (OAP) service.

Before you get started, make sure you:
✓ Familiarize yourself with TSB concepts
✓ Install the TSB demo environment
✓ Deploy the Istio Bookinfo sample app

Verifying the SWCK metrics adapter

The SWCK adapter is responsible for managing the OAP component, among others.

The adapter should have been installed when you installed the TSB demo profile. To verify the adapter has been deployed successfully, check the that the corresponding pod has started correctly.

kubectl get po -n istio-system

Output

... <snip> ...
istio-system-custom-metrics-apiserver-7cdbb5bdbb-zmwh7   1/1     Running   0    5m54s

If for some reason the following resources are not being updated/generated correctly, try deleting them by hand. Deleting them should trigger the creation of a new pod with the up-to-date configurations:

• apiservice/v1beta1.external.metrics.k8s.io
• rolebinding/istio-system-custom-metrics-auth-reader (in kube-system namespace)

HPA Configuration

To enable a HorizontalPodAutoscaler that uses the SWCK adapter, you will need to setup your configuration using the External metrics type.

The External metrics type allows you to auto-scale your cluster based on any metric available in the OAP cluster. To use this feature, provide a metric block with a name and selector, and use the External metric type.

kind: HorizontalPodAutoscaler
metadata:
  name: productpage-hpa-external-metrics
spec:
- type: External
  external:
    metric:
      name: <metric_name>
      metricSelector:
        matchLabels:
          <label_key>: <label_value>
          ...
    target:
      ....

The metric_name should be the metric name generated by Observability Analysis Language (OAL), or other subsystems.

The label_key is the entity name of SkyWalking metrics. If label_value contains special characters other than ".", "-" and "_", you should leverage "byte" labels to encode them to hex bytes. service.str.<number> would represent the literal of label value, whereas service.byte.<number> could be used to represent special characters as hex bytes.

For example, if the service name is v1|productpage|bookinfo|demo, the matchLabels should look like the following:

matchLabels:
  "service.str.0":  "v1"
  "service.byte.1": "7c" # the hex byte of "|"
  "service.str.2":  "productpage"
  "service.byte.3": "7c"
  "service.str.4":  "bookinfo"
  "service.byte.5": "7c"
  "service.str.6":  "demo"

Note that byte labels only accept a single character. That means an input like || should be transformed to two entries consisting of "service.byte.0":"7c" and "service.byte.1":"7c", instead of service.byte.0:"7c7c".

The label_keys may contain entity names for service for the name of the service, instance for the service instance, endpoint for the name of the endpoint, and label, for the labels to query. i.e. to encode a service name you would use "service.str.<number>" or "service.byte.<number>", to encode an endpoint you would use "endpoint.str.<number>" and "endpoint.byte.<number>", and so forth.

A comprehensive example

Suppose you would like to to auto-scale your deployment such that there is always enough replicas that accept around 80 requests per minute.

Assuming that "productpage-v1" deployment in your "demo" cluster based on "service_cpm" metrics from OAP, and the service name was "v1|productpage|demo|-", your HPA manifest would look like the following:

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: productpage-hpa-external-metrics
spec:
  minReplicas: 1
  maxReplicas: 5
  metrics:
  - type: External
    external:
      metric:
        name: tsb.tetrate.io|service_cpm
        selector:
          matchLabels:
            "service.str.0":  "v1"
            "service.byte.1": "7c"
            "service.str.2":  "productpage"
            "service.byte.3": "7c"
            "service.str.4":  "bookinfo"
            "service.byte.5": "7c"
            "service.str.6":  "demo"
            "service.byte.7": "7c"
            "service.byte.8": "2d"
      target:
        type: AverageValue
        value: 80
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: productpage-v1

Once you apply the above manifest using kubectl apply -f, you should see the HPA created in your bookinfo namespace:

kubectl get hpa -n bookinfo

Output

NAME                               REFERENCE                         TARGETS   MINPODS   MAXPODS   REPLICAS   
...<snip>...
productpage-hpa-external-metrics   Deployment/productpage-v1         0/80      1         5         1        

And to test your application, use a tool like Hey to generate some load. Eventually you should see the HPA creating more replicas to handle the load:

kubectl get hpa -n bookinfo

Output

NAME                               REFERENCE                         TARGETS   MINPODS   MAXPODS   REPLICAS   
...<snip>...
productpage-hpa-external-metrics   Deployment/productpage-v1         2252/80   1         10        4  

The following script may help you get most if not all matchLabels printed out:

bash-5.2$ read -p "Enter the service name: " input_string; index=0; token=""; echo "matchLabels:"; for (( i=0; i<${#input_string}; i++ )); do c="${input_string:$i:1}"; if [[ "$c" =~ [a-zA-Z0-9._-] ]]; then token+="$c"; else if [[ -n "$token" ]]; then echo "  \"service.str.$index\": \"$token\""; ((index++)); token=""; fi; ascii=$(printf "%d" "'$c"); hex=$(printf "%02x" "$ascii"); echo "  \"service.byte.$index\": \"$hex\""; ((index++)); fi; done; if [[ -n "$token" ]]; then echo "  \"service.str.$index\": \"$token\""; fi
Enter the service name: v1|productpage|bookinfo|demo
matchLabels:
  "service.str.0": "v1"
  "service.byte.1": "7c"
  "service.str.2": "productpage"
  "service.byte.3": "7c"
  "service.str.4": "bookinfo"
  "service.byte.5": "7c"
  "service.str.6": "demo"
bash-5.2$ 

Please read the SWCK metrics adapter document for more details.