Configuring Application Access Logging

Envoy produces verbose logs at runtime by default to enable easy debugging. They can be split into two categories:

• Runtime Envoy logs: intended for platform teams to troubleshoot Envoy itself
• Request Access logs: per-request information similar to the Apache common log

In production deployments, the first should be disabled as they're verbose and produce a high volume of logs. However, in lower environments where logs are not retained, they should be maintained to facilitate debugging during application development. See our guide for Envoy Debug Logging for a full overview of this kind of logging.

The second is what we'll focus on in this guide, in particular how to configure those logs to include the information your existing systems need, and how to update the format of the output to make it easy for existing systems to ingest.

Envoy Access Logs

Envoy provides a ton of data at runtime that can be included in per-request log lines. That guide covers how to format the log string itself. By default, Envoy Gateway (and Envoy) ships with a log format string:

[%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%"
%RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION%
%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%"
"%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%"

Which produces a log line at runtime that looks like:

[2016-04-15T20:17:00.310Z] "POST /api/v1/locations HTTP/2" 204 - 154 0 226 100 "10.0.35.28"
"nsq2http" "cc21d9b0-cf5c-432b-8c7e-98aeb7988cd2" "locations" "tcp://10.0.2.1:80"

Configuring the Envoy Gateway's Log Format

With an updated log format string in hand, we can update Envoy Gateway to use the new format. We cannot change the log format per application: we can only support custom log formats per EnvoyProxy. This means we can only support a single log configuration today.

To set that configuration, we use the telemetry.accessLog field in the EnvoyProxy. In this example, we'll set the value to a JSON formatted output, via the text logger.

apiVersion: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: do-not-use-example-envoyproxy-logging-config
  namespace: envoy-gateway-system
spec:
  telemetry:
    accessLog:
      settings:
        - format:
            type: JSON
            json:
              start_time: "%START_TIME%"
              method: "%REQ(:METHOD)%"
              "x-envoy-origin-path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
              protocol: "%PROTOCOL%"
              response_code: "%RESPONSE_CODE%"
              response_flags: "%RESPONSE_FLAGS%"
              response_code_details: "%RESPONSE_CODE_DETAILS%"
              upstream_transport_failure_reason: "%UPSTREAM_TRANSPORT_FAILURE_REASON%"
              request_headers_bytes: "%REQUEST_HEADERS_BYTES%"
              bytes_received: "%BYTES_RECEIVED%"
              bytes_sent: "%BYTES_SENT%"
              duration: "%DURATION%"
              upstream_duration: "%COMMON_DURATION(US_TX_BEG:US_RX_END:ms)%"
              "x-envoy-upstream-service-time": "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%"
              "x-forwarded-for": "%REQ(X-FORWARDED-FOR)%"
              "user-agent": "%REQ(USER-AGENT)%"
              "x-request-id": "%REQ(X-REQUEST-ID)%"
              ":authority": "%REQ(:AUTHORITY)%"
              authorization: "%REQ(AUTHORIZATION)%"
              upstream_host: "%UPSTREAM_HOST%"
              upstream_cluster: "%UPSTREAM_CLUSTER%"
              upstream_local_address: "%UPSTREAM_LOCAL_ADDRESS%"
              downstream_local_address: "%DOWNSTREAM_LOCAL_ADDRESS%"
              downstream_remote_address: "%DOWNSTREAM_REMOTE_ADDRESS%"
              requested_server_name: "%REQUESTED_SERVER_NAME%"
              route_name: "%ROUTE_NAME%"
          sinks:
          - file:
              path: /dev/stdout
            type: File

Note that if we're configuring the entire EnvoyProxy resource, we also need to set up the sinks for our logging. Envoy supports both output to files as well as to an OpenTelemetry agent.

See the guide for configuring Gateway deployments for more information on configuring the deployment and runtime behavior of Envoy.