Using Amazon Private CA

Use an external CA to sign certs used by the TSE dataplane

The common implementation scenarios for Enterprises is to host the Root CA in AWS Private Certificate Authority (AWS PCA). This page demonstrates how Istio Data-path mTLS can leverage the Enterprise PKI served by AWS PCA.

Overview

The recommended scenario for Enterprise PKI setup in Istio is to issue an Intermediate CA for every EKS cluster. Istio will use those intermediate CAs to deploy leaf CAs for every istio-proxy and gateway. When inter-cluster communication happens, the validation of the other cluster's certificate is done using the common Root CA. If a cluster is compromised the intermediate CA for that cluster is revoked, but the rest of the security model is safe and doesn't require any modifications to keep it secure.

Use of the intermediate CAs by Istio

By default, TSE manages its own certificates. It generates a self-signed Root CA and distributes Intermediate CAs to each onboarded Workload cluster.

In this scenario, we will host the Root CA in AWS PCA and we will manually create Intermediate CAs and install them on each Workload cluster.

Implementation steps - the Management Plane

The Management Plane manages the root CA and generates Intermediate CAs for each cluster. We need to disable this behavior.

Disable TSE self-signed certificate distribution

To disable the propagation of the self-signed certificate secret, the ManagementPlane CR needs to be modified. The change is required only once. Before executing the command, make sure kubectl points to the kubernetes cluster that hosts TSE Management Plane and then patch the resource:

kubectl patch managementplanes.install.tetrate.io managementplane -n tse \
    --patch '{"spec":{"certIssuer":{"clusterIntermediateCAs":null}}}' \
    --type merge

Implementation steps - each Workload Cluster

The next section covers the exact steps to configure each Workload cluster:

1. Generate a Certificate Signing Request (CSR) for the cluster
2. Request then obtain the Intermediate CA
3. Acquire the Root CA
4. Disable certificate management on the workload cluster
5. Install the new Intermediate CA and Root CA on the workload cluster

The Workload cluster will then be able to issue short-lived workload certificates and validate authenticity of cross-cluster incoming calls.

Variables setup

The commands in this guide use repeating parameters and setting environment variables will help:

CLUSTER_NAME=<EKS Cluster name>
AWS_REGION=<AWS Region where EKS Cluster is placed>
CERT_AUTHORITY=<AWS PCA CA Arn>

Certificate Signing Request

A Certificate Signing Request (CSR) needs to be created first. The CSR will be used during next step to request the certificate from AWS Certificate Manager. During certificate creation the signing (private) key is created.

Please use the following script to create the CSR and the key:

#!/bin/bash
# Script to create self signed certificate

mkdir -p ${CLUSTER_NAME}

# Create openssl config file
cat <<EOF | envsubst > ${CLUSTER_NAME}/${CLUSTER_NAME}.cnf
[req]
default_bits       = 2048
prompt             = no
distinguished_name = req_distinguished_name
req_extensions     = san_reqext

[ req_distinguished_name ]
countryName         = US
stateOrProvinceName = California
organizationName    = Tetrate.io

[ san_reqext ]
subjectAltName      = @alt_names

EOF

openssl req -new \
   -nodes \
   -newkey rsa:2048 \
   -days 365 \
   -keyout ${CLUSTER_NAME}/${CLUSTER_NAME}-priv-key.pem \
   -out ${CLUSTER_NAME}/${CLUSTER_NAME}.csr \
   -subj "/C=US/ST=CA/O=Tetrateio/CN=${CLUSTER_NAME}-cluster"

Request Intermediate CA

The following command will request AWS Certificate Manager to issue Intermediate CA. If successful the certificate ARN is returned. The ARN will be stored in CERT_ARN variable. Please note that AWS Intermediate CA Template is referred in the command.

tip

If short-lived certificate mode is configured for AWS PCA - the certificate length have to be adjusted in the command below with the value of 7 days or less --validity Value=7,Type="DAYS"

CERT_ARN=$(aws acm-pca issue-certificate \
--certificate-authority-arn ${CERT_AUTHORITY} \
--template-arn "arn:aws:acm-pca:::template/SubordinateCACertificate_PathLen0/V1" \
--signing-algorithm "SHA256WITHRSA" \
--validity Value=1,Type="YEARS" \
--region ${AWS_REGION} \
--csr fileb://${CLUSTER_NAME}/${CLUSTER_NAME}.csr \
| jq -r .CertificateArn)

Validate that previous command successfully returned the value of Certificate ARN. If the command returns no value - please execute the previous step in sections (i.e. aws command, piping AWS command to jq)

echo $CERT_ARN

Obtaining Intermediate CA

The issued certificate needs to be obtained from AWS Certificate Manager using the AWS CLI Docs as follows:

aws acm-pca get-certificate \
  --certificate-arn ${CERT_ARN} \
  --certificate-authority-arn ${CERT_AUTHORITY} \
  --region ${AWS_REGION} \
  --output text | sed $'s/\t/\\\n/g' \
  > ${CLUSTER_NAME}/${CLUSTER_NAME}-cert.pem

Obtaining CA Root Certificate

The Istio cacerts secret requires CA Root Certificate. The certificate is used for validation of the workload certificates from other cluster. The command below is based on AWS CLI Docs:

aws acm-pca get-certificate-authority-certificate \
  --certificate-authority-arn ${CERT_AUTHORITY} \
  --region ${AWS_REGION} \
  --output text \
  > ${CLUSTER_NAME}/root-cert.pem

Disable TSE self-signed certificate in ControlPlane CR

Before executing the following command, make sure kubectl points to the kubernetes cluster that hosts TSE Control Plane. If the cluster was added to TSE before self-signed cacerts were disabled system-wise (in Management Plane CRD), the propagation of the self-signed certificate secret needs to be disabled by patching ControlPlane CRD:

Not necessary for new clusters

Note - this step is not required for new clusters, onboarded after the Management Plane change above

kubectl patch ControlPlane controlplane -n istio-system \
    --patch '{"spec":{"components":{"xcp":{"centralProvidedCaCert":false}}}}' \
    --type merge

The secret created by TSE might be still present, and would need to be cleared:

kubectl delete secret cacerts -n istio-system

Creating the secret

At this point, all required artifacts (Intermediate CA Certificate and Key, Root CA certificate and certificate chain) are collected. The TSE Controlplane is prepared. The following command will create the cacerts secret that is issued by AWS Certificate Manager.

kubectl create secret generic cacerts -n istio-system \
      --from-file=ca-cert.pem=${CLUSTER_NAME}/${CLUSTER_NAME}-cert.pem \
      --from-file=ca-key.pem=${CLUSTER_NAME}/${CLUSTER_NAME}-priv-key.pem \
      --from-file=root-cert.pem=${CLUSTER_NAME}/root-cert.pem \
      --from-file=cert-chain.pem=${CLUSTER_NAME}/${CLUSTER_NAME}-cert.pem

Istiod requires a restart after the secret is successfully created:

kubectl -n istio-system rollout restart deployment -l=app=istiod

tip

All pods that have istio-proxy injected and Istio ingress gateways will need to wait for the per-service attached short-lived (default is 24 hours) certificate to to expire and be re-issued. To make sure the applications and gateway leaf certificated are refreshed without any delay,the recommendation is to restart all pods. The restart will immediately re-issue the leaf certificate from the AWS PCA trust chain.

By now the Controlplane is using AWS PCA PKI Trust chain. The same procedure needs to be repeated in the every cluster.

To test cross-cluster communication we recommend to follow this guide.