Service Metrics with PromQL

TSB embeds the Apache SkyWalking PromQL Service to expose service metrics, providing the same PromQL query language as the Prometheus server. Third-party systems or visualization platforms that already support PromQL (such as Grafana) can obtain metrics through PromQL Service in the TSB managed cluster.

Before you get started, make sure you:
✓ Familiarize yourself with TSB concepts
✓ Install the TSB demo environment
✓ Deploy the Istio Bookinfo sample app

The API Address

The PromQL API is provided by the oap (Observability Analysis Platform) service in the TSB Management Plane. It is exposed with the same address and authorization as the Web UI, and the base context path is promql.

${PROMQL_SERVICE_ADDRESS} = ${WEB_UI_ADDRESS}/promql

For example: if the Web UI address is https://localhost:8443, then the PromQL Service API address is https://localhost:8443/promql.

Querying From the API

Querying Metadata

Metadata is the information about the metrics, such as the service name, instance name, endpoint name, and labels.

SkyWalking's metadata exists in the following metrics(traffics):

• service_traffic
• instance_traffic
• endpoint_traffic

For example querying all the services in the TSB managed cluster(Bookinfo):

curl -H "X-B3-Sampled: 1" -k -u '${user}:${pass}' ${PROMQL_SERVICE_ADDRESS}/api/v1/series -d 'match[]=service_traffic{layer="MESH"}&start=1677479336&end=1677479636'

The result could be:

{
    "status": "success",
    "data": [
        {
            "__name__": "service_traffic",
            "service": "-|bookinfo-gateway|bookinfo|demo|-",
            "scope": "Service",
            "layer": "MESH"
        },
        {
            "__name__": "service_traffic",
            "service": "v2|reviews|bookinfo|demo|-",
            "scope": "Service",
            "layer": "MESH"
        },
        {
            "__name__": "service_traffic",
            "service": "v1|details|bookinfo|demo|-",
            "scope": "Service",
            "layer": "MESH"
        },
        {
            "__name__": "service_traffic",
            "service": "v1|ratings|bookinfo|demo|-",
            "scope": "Service",
            "layer": "MESH"
        },
        {
            "__name__": "service_traffic",
            "service": "v3|reviews|bookinfo|demo|-",
            "scope": "Service",
            "layer": "MESH"
        },
        {
            "__name__": "service_traffic",
            "service": "v1|reviews|bookinfo|demo|-",
            "scope": "Service",
            "layer": "MESH"
        },
        {
            "__name__": "service_traffic",
            "service": "v1|productpage|bookinfo|demo|-",
            "scope": "Service",
            "layer": "MESH"
        }
    ]
}

The following definitions apply to TSB Service Format Definition, subset|service name|namespace name|cluster|- Layer Definition, All the metrics in the TSB managed cluster are belong to the MESH layer.

Querying Metrics

For example querying the Service Avg Response Time metric for the v1|productpage|bookinfo|demo|- service in 5 minutes:

curl -H "X-B3-Sampled: 1" -k -u 'admin:admin' https://35.220.249.46:8443/promql/api/v1/query -d 'query=service_resp_time{service="v1|productpage|bookinfo|demo|-", layer="MESH"}[5m]'

The result could be:

{
    "status": "success",
    "data": {
        "resultType": "matrix",
        "result": [
            {
                "metric": {
                    "__name__": "service_resp_time",
                    "layer": "MESH",
                    "scope": "Service",
                    "service": "v1|productpage|bookinfo|demo|-"
                },
                "values": [
                    [
                        1680861600,
                        "24"
                    ],
                    [
                        1680861660,
                        "24"
                    ],
                    [
                        1680861720,
                        "27"
                    ],
                    [
                        1680861780,
                        "23"
                    ],
                    [
                        1680861840,
                        "24"
                    ],
                    [
                        1680861900,
                        "28"
                    ]
                ]
            }
        ]
    }
}

Please read the PromQL Service document for more details.

Build Grafana Dashboard

Config Data Source

1. In the data source config panel, chose the Prometheus type.
2. Set the URL to the OAP PromQL Service address.
3. Select Basic auth and fill in the User and Password.
4. Click the Save & Test button to test the connection.

Add Dashboard

When you complete the data source configuration, you can add a dashboard to visualize the metrics. The following example dashboards are a good starting point:

• Service Metrics dashboard file
• Service relation metrics dashboard file

Import one of these dashboards to Grafana; once sufficient telemetry has been collected, you will be able to chart activity as follows:

Please read the Use Grafana As The UI document for more details.