Configuring Application Access Logging

Envoy produces verbose logs at runtime by default to enable easy debugging. They can be split into two categories:

• Runtime Envoy logs: intended for platform teams to troubleshoot Envoy itself
• Request Access logs: per-request information similar to the Apache common log

In production deployments, the first should be disabled as they're verbose and produce a high volume of logs. However, in lower environments where logs are not retained, they should be maintained to facilitate debugging during application development. See our guide for Envoy Debug Logging for a full overview of this kind of logging.

The second is what we'll focus on in this guide, in particular how to configure those logs to include the information your existing systems need, and how to update the format of the output to make it easy for existing systems to ingest.

Envoy Access Logs

Envoy provides a ton of data at runtime that can be included in per-request log lines. That guide covers how to format the log string itself. By default, Envoy Gateway (and Envoy) ships with a log format string:

[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%"
%RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION%
%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%"
"%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%"

Which produces a log line at runtime that looks like:

[2016-04-15T20:17:00.310Z] "POST /api/v1/locations HTTP/2" 204 - 154 0 226 100 "10.0.35.28"
"nsq2http" "cc21d9b0-cf5c-432b-8c7e-98aeb7988cd2" "locations" "tcp://10.0.2.1:80"

Configuring the Envoy Gateway's Log Format

With an updated log format string in hand, we can update Envoy Gateway to use the new format. We cannot change the log format per application: we can only support custom log formats per EnvoyProxy. This means we can only support a single log configuration today.

To set that configuration, we use the telemetry.accessLog field in the EnvoyProxy. In this example, we'll set the value to a JSON formatted output, via the text logger. Envoy supports direct JSON logging, but in some cases this kind of JSON-like output via type: Text performs better than type: JSON logging.

apiVersion: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: do-not-use-example-envoyproxy-logging-config
  namespace: envoy-gateway-system
spec:
  telemetry:
    accessLog:
      settings:
        - format:
            type: Text
            text: |
              {"start_time":"%START_TIME%","method":"%REQ(:METHOD)%","x-envoy-origin-path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","protocol":"%PROTOCOL%","response_code":"%RESPONSE_CODE%","response_flags":"%RESPONSE_FLAGS%","response_code_details":"%RESPONSE_CODE_DETAILS%","connection_termination_details":"%CONNECTION_TERMINATION_DETAILS%","upstream_transport_failure_reason":"%UPSTREAM_TRANSPORT_FAILURE_REASON%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","duration":"%DURATION%","x-envoy-upstream-service-time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","x-forwarded-for":"%REQ(X-FORWARDED-FOR)%","user-agent":"%REQ(USER-AGENT)%","x-request-id":"%REQ(X-REQUEST-ID)%",":authority":"%REQ(:AUTHORITY)%","upstream_host":"%UPSTREAM_HOST%","upstream_cluster":"%UPSTREAM_CLUSTER%","upstream_local_address":"%UPSTREAM_LOCAL_ADDRESS%","downstream_local_address":"%DOWNSTREAM_LOCAL_ADDRESS%","downstream_remote_address":"%DOWNSTREAM_REMOTE_ADDRESS%","requested_server_name":"%REQUESTED_SERVER_NAME%","route_name":"%ROUTE_NAME%"}
          sinks:
          - file:
              path: /dev/stdout
            type: File

Note that if we're configuring the entire EnvoyProxy resource, we also need to set up the sinks for our logging. Envoy supports both output to files as well as to an OpenTelemetry agent. To change just the format, or to edit an existing EnvoyProxy resource, we can patch it instead.

caution

In the v0.1 release of TEG, TEG manages the single EnvoyProxy resource that is allowed. Rather than writing it directly, you need to edit the value that is instantiated to include the per-component logging you need. See the examples at the end for a guide on patching an existing EnvoyProxy to include debug logging.

Patch Existing Config

We can patch an existing EnvoyProxy rather than authoring the entire resource. For a complex configuration like access logging, this has the advantage of meaning we only need to write a portion of the config, rather than the entire object (assuming the default meets our needs - in the case of logging, printing to /dev/stdout). We're assuming the deployment name and namespace from the quickstart install, so update to your own as appropriate:

kubectl patch \
  EnvoyProxy teg-envoy-proxy-config -n envoy-gateway-system \
  --type json --patch '
- op: add
  path: "/spec/telemetry/accessLog/settings/0/format/text"
  value: |
    {"start_time":"%START_TIME%","method":"%REQ(:METHOD)%","x-envoy-origin-path":"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%","protocol":"%PROTOCOL%","response_code":"%RESPONSE_CODE%","response_flags":"%RESPONSE_FLAGS%","response_code_details":"%RESPONSE_CODE_DETAILS%","connection_termination_details":"%CONNECTION_TERMINATION_DETAILS%","upstream_transport_failure_reason":"%UPSTREAM_TRANSPORT_FAILURE_REASON%","bytes_received":"%BYTES_RECEIVED%","bytes_sent":"%BYTES_SENT%","duration":"%DURATION%","x-envoy-upstream-service-time":"%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%","x-forwarded-for":"%REQ(X-FORWARDED-FOR)%","user-agent":"%REQ(USER-AGENT)%","x-request-id":"%REQ(X-REQUEST-ID)%",":authority":"%REQ(:AUTHORITY)%","upstream_host":"%UPSTREAM_HOST%","upstream_cluster":"%UPSTREAM_CLUSTER%","upstream_local_address":"%UPSTREAM_LOCAL_ADDRESS%","downstream_local_address":"%DOWNSTREAM_LOCAL_ADDRESS%","downstream_remote_address":"%DOWNSTREAM_REMOTE_ADDRESS%","requested_server_name":"%REQUESTED_SERVER_NAME%","route_name":"%ROUTE_NAME%"}
'

Note that if you have multiple access loggers configured, you'll need to update the index used in the example. For the quickstart deployment, there is only a single access log configured, so we edit index zero (the 0 in path: "/spec/telemetry/accessLog/settings/0/format/text").

JSON-patch

We have to use patch --type json here as it supports changing individual fields rather than entire objects - we don't want to have to mess with log output location just to change format! However, json-patch operations require a special JSON Patch document. In short, it's a small JSON object like {"op": "replace", "path": "/some/path", "value": "updated value"}. Fortunately, the Kubernetes tooling works equally well with both JSON and YAML, so we can use the easier YAML format above rather than writing out JSON by hand.

See the guide for configuring Gateway deployments for more information on configuring the deployment and runtime behavior of Envoy.