Istio CA

Requirements

• Kubernetes cluster 1.15 or newer
• Vault 1.3.1 or newer
• Vault Injector 0.3.0 or newer
• Tetrate Service Bridge 0.8.0 or newer

Setup Vault

This guide assumes that Vault and the Vault injector are already installed on your Kubernetes cluster. For more details on how to do this check the Vault documentation.

Create a CA Certificate

note

If you already have your own CA certificate and/or an intermediate CA certificate to use with Istio, skip to Adding Intermediate CA Certificate to Vault

Vault has a PKI Secret back-end that supports creation or management of CA Certificates. It is recommended users create an Intermediate CA certificate for Istio and keep the Root CA outside of Vault. Refer to the Vault documentation for more details.

Generate a self-signed root CA certificate in Vault using the Vault PKI back-end.

# Add audit trail in Vault
vault audit enable file file_path=/vault/vault-audit.log

# Enable PKI secret back-end
vault secrets enable pki

# Update PKI lease to 1 year
vault secrets tune -max-lease-ttl=8760h pki

# Create a Self Signed CA
vault write pki/root/generate/internal common_name=tetrate.io ttl=8760h

Intermediate CA for Istio

Now that we have a CA in Vault, we use it to create an intermediate CA for Istio. We re-use the pki Secret back-end but with a new path (istioca). This time we need to get the CA Key and will call the exported endpoint instead of internal:

# Enable PKI in a new path for intermediate CA
vault secrets enable --path istioca pki

# Update lease time to 5 years
vault secrets tune -max-lease-ttl=43800h istioca

# Create Intermediate CA cert and Key
vault write istioca/intermediate/generate/exported common_name="tetrate.io Intermediate Authority" ttl=43800h

Key                 Value
---                 -----
csr                 -----BEGIN CERTIFICATE REQUEST-----
MIICcTCCAVkCAQAwLDEqMCgGA1UEAxMhdGV0cmF0ZS5pbyBJbnRlcm1lZGlhdGUg
...
...
7HJEy22yCFvVcR+Gtf++iZG+w04E2ah99xrzb+NdWDgw6asBg7oJg/bJQoA4/Wb5
OX2jl0E=
-----END CERTIFICATE REQUEST-----
private_key         -----BEGIN RSA PRIVATE KEY-----
MIIEpQIBAAKCAQEA8Vmm2urwUHdAp1j3vs8aOqYGrDz3NwJbm6du+3WmgGHQ+sEC
...
...
yri2DiQWzwk3zIvzNSSbQdACPPeF90BLFW9L4xvN8D6gBxFL0wBa7GY=
-----END RSA PRIVATE KEY-----
private_key_type    rsa

Copy the Certificate Signing Request (CSR) into a local file istioca.csr (including the -----BEGIN CERTIFICATE REQUEST----- and -----END CERTIFICATE REQUEST-----). Copy the CA Key into a local file istioca.key. It is advised to keep a backup of this key in a secure place.

Now we can use the Vault CA to sign the CSR:

vault write pki/root/sign-intermediate csr=@istioca.csr format=pem_bundle ttl=43800h

Key              Value
---              -----
certificate      -----BEGIN CERTIFICATE-----
MIIDMDCCAhigAwIBAgIUGJgs6yFbK/eDW31RpdSiNeVQYvUwDQYJKoZIhvcNAQEL
...
...
PcPHltvhXDjckPK1jt9gpLMTaBhe9uu7Ve7b2OFv+mtAGiCEvALv+ddLa0GHrl3s
f2vq6A==
-----END CERTIFICATE-----
expiration       1618951159
issuing_ca       -----BEGIN CERTIFICATE-----
MIIDLDCCAhSgAwIBAgIUFVRq5X/cAOlDKl/h34VudUdSiMwwDQYJKoZIhvcNAQEL
...
...
d4SklUyQdxnW96IdkHSPWf46jh31tDWqco6LzmrxD4OmjSeLaf0zeErU1i41xAnW
-----END CERTIFICATE-----
serial_number    18:98:2...:f5

Adding Intermediate CA Certificate to Vault

We now need to put the signed Intermediate CA certificate into Vault.

vault write istioca/intermediate/set-signed certificate=@istioca.crt

Istio will also generate certificates based on the intermediate CA certificate and needs both the certificate and its key. While Vault can send the certificate using it's API, we have to take care of the key. To do so we make a copy of the key inside a Vault Secret for later:

vault kv put secret/istioca ca-key.pem=@istioca.key

Configure the Vault-Injector

Please refer to the Injector doc on Vault's Website.

Configured Vault to enable access to the Kubernetes API. This example assumes that you are running commands in the Vault pod using kubectl exec. If not, you will need to find the right JWT Token, Kubernetes API URL (that Vault will use to connect to Kubernetes) and the CA certificate of the vaultserver service account:

vault auth enable kubernetes
vault write auth/kubernetes/config \
  token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
  kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \
  kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

Now configure a Vault role that will match the Kubernetes Service Account of Istio. This role will be used by the Vault-Injector: In Vault, the role is named istiod and is bound to the istiod service account, istiod-service-account

vault write auth/kubernetes/role/istiod \
 bound_service_account_names=istiod-service-account \
 bound_service_account_namespaces=istio-system \
 policies=istioca \
 period=600s

Create the Policy that will allow the above mentioned role to access the PKI Secret back-end. This Policy enables Istio to read the key/values inside the secret secret/data/istioca, where we have stored the Key of the Intermediate CA. It also allows Istio to access the Intermediate CA and the CA chain (by using *) and finally the Root CA.

cat > policy.hcl <<EOF
path "secret/data/istioca" {
 capabilities = ["read", "list"]
}
path "istioca/cert/*" {
 capabilities = ["read", "list"]
}
path "pki/cert/ca" {
 capabilities = ["read", "list"]
}
EOF

vault policy write istioca /tmp/policy.hcl

Configure TSB ControlPlane CRD

Update your ControlPlane resource and add the specific istiod component configuration to use Vault:

apiVersion: install.tetrate.io/v1alpha1
kind: ControlPlane
metadata:
  name: controlplane
  namespace: tsb
spec:
  components:
    istio:
      kubeSpec:
        deployment:
          env:
            - name: ROOT_CA_DIR
              value: /etc/cacerts-vault
          podAnnotations:
            vault.hashicorp.com/agent-inject: "true"
            vault.hashicorp.com/agent-inject-secret-ca-cert.pem: istioca/cert/ca
            vault.hashicorp.com/agent-inject-secret-ca-key.pem: secret/data/istioca
            vault.hashicorp.com/agent-inject-secret-cert-chain.pem: istioca/cert/ca_chain
            vault.hashicorp.com/agent-inject-secret-root-cert.pem: pki/cert/ca
            vault.hashicorp.com/agent-inject-template-ca-cert.pem: |
              {{- with secret "istioca/cert/ca" -}}
              {{ .Data.certificate  }}
              {{- end }}
            vault.hashicorp.com/agent-inject-template-ca-key.pem: |
              {{- with secret "secret/data/istioca" -}}
              {{ index .Data.data "ca-key.pem" }}
              {{- end }}
            vault.hashicorp.com/agent-inject-template-cert-chain.pem: |
              {{- with secret "istioca/cert/ca_chain" -}}
              {{ .Data.certificate  }}
              {{- end }}
            vault.hashicorp.com/agent-inject-template-root-cert.pem: |
              {{- with secret "pki/cert/ca" -}}
              {{ "" | or ( .Data.certificate ) }}
              {{- end }}
            vault.hashicorp.com/role: istiod
            vault.hashicorp.com/secret-volume-path: /etc/cacerts-vault

Here, we add an environment variable ROOT_CA_DIR pointing Istio to the directory where we create the new CA files from Vault. Then we add the Vault-Injector annotations to create the certificates.

Annotations are composed of a secret from Vault and a template defining how to create the file on the disk. We create one file per certificate component (certificate, key, chain and root)

Troubleshooting

If something is wrong, like the istiod pods not starting, check in the logs:

• Pod is failing during Init phase check the Vault-Injector logs in the istiod pod

kubectl logs -n istio-system deployment/istiod -c vault-agent-init

• Pod is failing after Init

kubectl logs -n istio-system deployment/istiod -c vault-agent

• istiod process is crashing If the certificate is not working, the Vault-Injector will work but istiod will not be able to start. Check istiod logs:

kubectl logs -n istio-system deployment/istiod -c discovery