Configuring Envoy Internal (Debug) Logs

Envoy produces verbose logs at runtime by default to enable easy debugging. They can be split into two categories:

• Runtime Envoy logs: intended for platform teams to troubleshoot Envoy itself
• Request Access logs: per-request information similar to the Apache common log

The first category is covered in our guide on application logging. The second is the focus of this guide.

Using the EnvoyProxy's logging section, we can configure how Envoy's internal components emit logs. This is very valuable for debugging Envoy's internal behavior when it's not doing what you expect. For example, we might turn up logging for some components to understand why our external authorization integration isn't working, or to log the quota bucket used for each request when we're rate limiting. Envoy divides logging up into components which each emit logs at a specific level.

In the guide on configuring Gateway deployments we described the EnvoyProxy resource and how to attach it to our GatewayClass to configure Envoys at runtime. We'll continue to work with that resource in this guide.

Envoy Logging Components

The source-of-truth for components is defined here in the Envoy codebase. To list a few notable components that are more frequently used:

• config — for insight into how Envoy is processing configuration, and config errors
• connection, conn_handler, udp — for insight into how TCP and UDP connections are being handled
• ext_authz — for insight into External Authorization configurations
• ext_proc — for insight into External Process integrations
• wasm — for insight into the WASM runtime and WASM process execution in Envoy
• grpc, http, http2, websocket, quic, quic_stream — for insight into gRPC, HTTP, and QUIC traffic
• jwt, rbac, oauth2 — for insight into authentication and authorization related functions

Components produce logs at different levels. level can be set to any of:

• debug
• info
• warn
• error

Log levels

Envoy supports the full list of:

• off
• trace
• debug
• info
• warning/warn
• error
• critical

for each log scope, but current EG validation limits us to only the four listed above.

Importantly, there's also the default pseudo-component that's used to configure all components globally. Typically, we'll specify both a default level (usually default: warn) when we turn up logging for specific components.

Logging Can Be Very Verbose!

Envoy internal logging can be very verbose, especially at the debug and trace levels — be wary of setting default: debug — and in general be wary of log retention policies when enabling verbose logging for chatty components. It can get expensive fast!

Setting Log Levels per Component

In the EnvoyProxy resource's logging section, we can fill in levels with component: level pairs:

apiVersion: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: do-not-use-example-envoyproxy-logging-config
  namespace: envoy-gateway-system
spec:
  logging:
    level:
      #default: warn
      http2: debug
      #jwt: trace
      #grpc: critical
      ext_authz: info
      ext_proc: error
      #wasm: off

caution

In the current release of TEG, TEG manages the singular EnvoyProxy resource that is allowed to exist. Rather than overwriting it, you need to edit the resource that already exists, to include the per-component logging you need. See the examples at the end for a guide on patching an existing EnvoyProxy to include debug logging.

Patch Existing Config

We can do the same, using a patch config instead, to modify an existing EnvoyProxy. We're assuming the deployment name and namespace from the quickstart install, so update to your own as appropriate:

kubectl patch \
  EnvoyProxy teg-envoy-proxy-config -n envoy-gateway-system \
  --type merge --patch '
spec:
  logging:
    level:
      #default: warn
      http2: debug
      #jwt: trace
      #grpc: critical
      ext_authz: info
      ext_proc: error
      #wasm: off
'

See the guide for configuring Gateway deployments for more information on configuring the deployment and runtime behavior of Envoy.