Elasticsearch Credentials

Requirements

• Kubernetes cluster 1.15 or newer
• Vault 1.3.1 or newer
• Vault Injector 0.3.0 or newer
• Tetrate Service Bridge 0.8.0 or newer
• Elasticsearch 6.8.8 with basic license

Setup Vault

This guide assumes that Vault and the Vault injector are already installed on your Kubernetes cluster. For more details on how to do this check the Vault documentation.

Startup Elasticsearch with Security Enabled

If you manage your own instance of Elasticsearch enable security and set the super-user password as environment variables. Here we set the password of super-user elastic to elastic.

- name: xpack.security.enabled
  value: "true"
- name: ELASTIC_PASSWORD
  value: elastic

Create a Role for Vault

First, using the super-user create a role that will allow Vault the minimum privileges by performing a POST to Elasticsearch.

curl \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"cluster": ["manage_security"]}' \
  http://<super-user-username>:<super-user-password>@<elastic-ip>:<elastic-port>/_xpack/security/role/vault

Next, create a user for Vault associated with that role.

curl \
  -X POST \
  -H "Content-Type: application/json" \
  -d @data.json \
  http://<super-user-username>:<super-user-password>@<elastic-ip>:<elastic-port>/_xpack/security/user/vault

The content of data.json in this example is:

{
  "password": "<vault-elastic-password>",
  "roles": ["vault"],
  "full_name": "Hashicorp Vault",
  "metadata": {
    "plugin_name": "Vault Plugin Database Elasticsearch",
    "plugin_url": "https://github.com/hashicorp/vault-plugin-database-elasticsearch"
  }
}

Now, an Elasticsearch user is configured and ready to be used by Vault.

Vault setup

Install Vault (customers are supposed to handle Vault on their side. Vault may not be installed in the Kubernetes cluster, but should be reachable from inside the Kubernetes cluster). The Vault Injector (agent-injector) have to be installed into the cluster and configured to inject sidecars. This is automatically done by the Helm chart v0.5.0+ which install Vault 0.12+ and Vault-Injector 0.3.0+. We assume here that Vault is installed in the tcc namespace.

Enable the database secrets engine in Vault.

vault secrets enable database
Success! Enabled the database secrets engine at: database/

By default, the secrets engine will enable at the name of the engine. To enable the secrets engine at a different path, use the -path argument.

Configure Vault with the plugin and connection information:

vault write database/config/<my-elasticsearch-database> \
  plugin_name="elasticsearch-database-plugin" \
  allowed_roles="internally-defined-role" \
  username=vault \
  password=<vault-elastic-password> \
  url=http://<elastic-ip>:<elastic-port> \

If using HTTPS connection, you have to provide the certificate, key and CA bundle (Root CA and Intermediates):

vault write database/config/my-elasticsearch-database \
    plugin_name="elasticsearch-database-plugin" \
    allowed_roles="internally-defined-role" \
    username=vault \
    password=myPa55word \
    url=https://localhost:9200  \
    ca_cert=es-bundle.ca \
    client_cert=es.crt \
    client_key=es.key

Configure a role that maps a name in Vault to a role definition in Elasticsearch.

vault write database/roles/internally-defined-role \
  db_name=<my-elasticsearch-database> \
  creation_statements='{"elasticsearch_role_definition": {"cluster":["manage_index_templates","monitor"],"indices":[{"names":["*"],"privileges":["manage","read","write"]}],"applications":[],"run_as":[],"metadata":{},"transient_metadata":{"enabled":true}}}' \
  default_ttl="1h" \
  max_ttl="24h"

Success! Data written to: database/roles/internally-defined-role

For validation of the configuration you can generate a new credential by reading from the /creds endpoint with the name of the role:

vault read database/creds/internally-defined-role
Key                Value
---                -----
lease_id           database/creds/internally-defined-role/jZHuJvZeEOvGJhfFixVcwOyB
lease_duration     1h
lease_renewable    true
password           A1a-SkZ9KgF7BJGn2FRH
username           v-token-internally-defi-UJT1IOSADK5fjXx0bRrE-1587043589

You can check in the Elasticsearch cluster to ensure the newly created credential is present:

curl -u "vault:myPa55word" -ks -XGET http://localhost:9200/_xpack/security/user/v-token-internally-defi-UJT1IOSADK5fjXx0bRrE-1587043589|jq '.'

  {
  "v-token-internally-defi-UJT1IOSADK5fjXx0bRrE-1587043589": {
    "username": "v-token-internally-defi-UJT1IOSADK5fjXx0bRrE-1587043589",
    "roles": [
      "v-token-internally-defi-UJT1IOSADK5fjXx0bRrE-1587043589"
    ],
    "full_name": null,
    "email": null,
    "metadata": {},
    "enabled": true
  }
}

Set up Kubernetes secret engine

Configure a policy named “database”, This is a very non-restrictive policy, and in a production setting, we should lock this down more.

vault policy write es-auth - <<EOF
path "database/creds/internally-defined-role" {
  capabilities = ["read"]
}
EOF
Success! Uploaded policy: es-auth

Configured Vault to enable access to the Kubernetes API. This example assumes that you are running commands in the Vault pod using kubectl exec. If not, you will need to find the right JWT Token, Kubernetes API URL (that Vault will use to connect to Kubernetes) and the CA certificate of the vaultserver service account:

vault auth enable kubernetes
vault write auth/kubernetes/config \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Attach our database policy to oap service account:

vault write auth/kubernetes/role/oap \
  bound_service_account_names=oap \
  bound_service_account_namespaces=tsb \
  policies=es-auth \
  ttl=1h

Attach our database policy to zipkin service account:

vault write auth/kubernetes/role/zipkin \
  bound_service_account_names=tsb-zipkin \
  bound_service_account_namespaces=tsb \
  policies=es-auth \
  ttl=1h

Inject Secrets into the Pod

If connecting to Elasticsearch using SSL (HTTPS), you will need to provide the Root CA and possible an Intermediate CA. This is done by creating a secret and adding that secret to the ManagementPlane:

# copy both Root CA and Intermediate CA into a bundle file
cat ca.crt intermediate.crt > bundle-ca.crt

# add the bundle to the es-certs secret
# the certificate must be in the ca.crt file
k create secret generic es-certs --from-file=ca.crt=bundle-ca.pem

To use Vault Agent Injector in combination with Elasticsearch, add the following deployment pod annotations and environment variables to the ManagementPlane custom resource.

spec:
  components:
    oap:
      kubeSpec:
        deployment:
          env:
            - name: SW_ES_SECRETS_MANAGEMENT_FILE
              value: /vault/secrets/credentials
          podAnnotations:
            vault.hashicorp.com/agent-inject: "true"
            vault.hashicorp.com/agent-init-first: "true"
            vault.hashicorp.com/agent-inject-secret-credentials: "database/creds/internally-defined-role"
            vault.hashicorp.com/agent-inject-template-credentials: |
              {{- with secret "database/creds/internally-defined-role" -}}
              user={{ .Data.username }}
              password={{ .Data.password }}
              trustStorePass=tetrate
              {{- end -}}
            vault.hashicorp.com/role: "oap"
    zipkin:
      kubeSpec:
        deployment:
          env:
            - name: ES_CREDENTIALS_FILE
              value: /vault/secrets/zipkin-credentials
            - name: DB_CREDENTIALS_FILE
              value: /vault/secrets/es-template-credentials
          podAnnotations:
            vault.hashicorp.com/agent-inject: "true"
            vault.hashicorp.com/agent-init-first: "true"
            vault.hashicorp.com/agent-inject-secret-zipkin-credentials: "database/creds/internally-defined-role"
            vault.hashicorp.com/agent-inject-template-zipkin-credentials: |
              {{- with secret "database/creds/internally-defined-role" -}}
                zipkin.storage.elasticsearch.username={{ .Data.username }}
                zipkin.storage.elasticsearch.password={{ .Data.password }}
              {{- end -}}
            vault.hashicorp.com/agent-inject-secret-es-template-credentials: "database/creds/internally-defined-role"
            vault.hashicorp.com/agent-inject-template-es-template-credentials: |
              {{- with secret "database/creds/internally-defined-role" -}}
                username={{ .Data.username }}
                password={{ .Data.password }}
              {{- end -}}
            vault.hashicorp.com/role: "zipkin"
        job:
          env:
            - name: CREDS_FILE
              value: /vault/secrets/credentials
          podAnnotations:
            vault.hashicorp.com/agent-pre-populate-only: "true"
            vault.hashicorp.com/agent-inject: "true"
            vault.hashicorp.com/agent-init-first: "true"
            vault.hashicorp.com/agent-inject-secret-credentials: "database/creds/internally-defined-role"
            vault.hashicorp.com/agent-inject-template-credentials: |
              {{- with secret "database/creds/internally-defined-role" -}}
              export ES_USERNAME={{ .Data.username }}
              export ES_PASSWORD={{ .Data.password }}
              {{- end -}}
            vault.hashicorp.com/role: "zipkin"

Control plane

The control plane also needs credentials for connecting to Elasticsearch. You will need to add the pod annotations for Vault injector into the OAP and Zipkin components in the control plane too. Below is a YAML snippet with the needed changes.

spec:
  components:
    oap:
      kubeSpec:
        deployment:
          env:
            - name: SW_ES_SECRETS_MANAGEMENT_FILE
              value: /vault/secrets/credentials
          podAnnotations:
            vault.hashicorp.com/agent-init-first: "true"
            vault.hashicorp.com/agent-inject: "true"
            vault.hashicorp.com/agent-inject-secret-credentials: database/creds/internally-defined-role
            vault.hashicorp.com/agent-inject-template-credentials: |
              {{- with secret "database/creds/internally-defined-role" -}}
              user={{ .Data.username }}
              password={{ .Data.password }}
              trustStorePass=tetrate
              {{- end -}}
            vault.hashicorp.com/role: oap
    zipkin:
      kubeSpec:
        deployment:
          env:
            - name: ES_CREDENTIALS_FILE
              value: /vault/secrets/zipkin-credentials
            - name: DB_CREDENTIALS_FILE
              value: /vault/secrets/es-template-credentials
          podAnnotations:
            vault.hashicorp.com/agent-init-first: "true"
            vault.hashicorp.com/agent-inject: "true"
            vault.hashicorp.com/agent-inject-secret-zipkin-credentials: database/creds/internally-defined-role
            vault.hashicorp.com/agent-inject-template-zipkin-credentials: |
              {{- with secret "database/creds/internally-defined-role" -}}
                zipkin.storage.elasticsearch.username={{ .Data.username }}
                zipkin.storage.elasticsearch.password={{ .Data.password }}
              {{- end -}}
            vault.hashicorp.com/role: zipkin

Debugging

You can add more debug info from Vault-Injector by adding the annotation: vault.hashicorp.com/log-level: trace