Integrating Istio with AWS Private Certificate Authority (PCA)

Introduction

This guide offers detailed, step-by-step instructions to help you set up a custom Certificate Authority (CA) for Istio, leveraging AWS PCA's managed infrastructure and cert-manager's automation capabilities.

Architecture Overview

The integration involves the following components:

• AWS Private CA: A managed service that provides a secure CA hierarchy.
• cert-manager: Automates the issuance and renewal of certificates in Kubernetes.
• AWS PCA Issuer Plugin: Enables cert-manager to interact with AWS PCA.
• Istio: Configured to use the custom CA for issuing mTLS certificates within the service mesh.

Prerequisites

Before you begin, ensure you have the following:

• AWS Account: With permissions to manage AWS PCA and IAM roles.
• AWS PCA: Access to create and manage CAs.
• Tetrate Istio Subscription (TIS): Deployed in your Kubernetes cluster.
• Kubernetes Cluster: With cluster admin access.
• kubectl: Configured to interact with your cluster.
• Helm: Installed on your local machine.
• cert-manager: Version 1.0.0 or later.
• AWS CLI: Configured with appropriate permissions.
• IAM Roles and Policies: Ability to create IAM roles and policies.
• OpenSSL: For certificate inspection (optional).

Steps

Step 1: Install cert-manager

cert-manager simplifies certificate management in Kubernetes clusters.

1. Add the Jetstack Helm Repository:

   helm repo add jetstack https://charts.jetstack.io --force-update

2. Install cert-manager:

   helm install \
     cert-manager jetstack/cert-manager \
     --namespace cert-manager \
     --create-namespace \
     --version v1.15.3 \
     --set crds.enabled=true

3. Verify Installation:

   kubectl -n cert-manager get pods

   Ensure all cert-manager pods are in the Running state.

Step 2: Set Up AWS Private Certificate Authority

Create a CA hierarchy in AWS PCA to issue certificates for Istio.

1. Create a Root CA:

   • Navigate to the AWS PCA console.
   • Choose Create a private CA and select Root CA.
   • Provide the required information (e.g., Organization, Country).
   • Activate the CA by installing the self-signed certificate.

2. Create a Subordinate CA:

   • Choose Create a private CA and select Subordinate CA.
   • Provide the necessary details.
   • Install a certificate from the parent CA.
   • Note that when installing the subordinate certificate, do not specify the path length parameter as 0, otherwise a CA cannot be created for Istio.

3. Note the CA ARN:

   Record the Amazon Resource Name (ARN) of the subordinate CA; it will be used later.

   export ISTIO_CA_ARN="arn:aws:acm-pca:region:account-id:certificate-authority/CA_ID"

   Alternatively, if you have an existing CA hierarchy, you can use its ARN.

Step 3: Create IAM Policy and Role

Create an IAM policy and role to allow cert-manager to interact with AWS PCA.

1. Create an IAM Policy:

   cat <<EOF > AWSPCAIssuerPolicy.json
   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Action": [
           "acm-pca:DescribeCertificateAuthority",
           "acm-pca:GetCertificate",
           "acm-pca:IssueCertificate"
         ],
         "Resource": "${ISTIO_CA_ARN}"
       }
     ]
   }
   EOF
   
   aws iam create-policy \
     --policy-name AWSPCAIssuerPolicy \
     --policy-document file://AWSPCAIssuerPolicy.json

2. Create an IAM Role with OIDC Provider:

   • Enable OIDC Provider for your EKS cluster (if not already enabled):

     eksctl utils associate-iam-oidc-provider \
       --cluster your-cluster-name \
       --approve

   • Create the IAM Role:

     eksctl create iamserviceaccount \
       --name aws-pca-issuer \
       --namespace cert-manager \
       --cluster your-cluster-name \
       --attach-policy-arn arn:aws:iam::your-account-id:policy/AWSPCAIssuerPolicy \
       --approve \
       --override-existing-serviceaccounts

     This command creates a service account aws-pca-issuer in the cert-manager namespace with the necessary permissions.

Step 4: Install AWS PCA Issuer Plugin

Install the AWS PCA Issuer plugin to allow cert-manager to request certificates from AWS PCA.

1. Add the AWS PCA Issuer Helm Repository:

   helm repo add awspca https://cert-manager.github.io/aws-privateca-issuer
   helm repo update

2. Install the AWS PCA Issuer:

   helm install aws-pca-issuer awspca/aws-privateca-issuer \
     --namespace cert-manager \
     --set serviceAccount.create=false \
     --set serviceAccount.name=aws-pca-issuer \
     --wait

3. Verify Installation:

   kubectl -n cert-manager get pods

   Ensure the aws-pca-issuer pod is running.

Step 5: Configure the AWS PCA Issuer

Create an AWSPCAIssuer resource to define how cert-manager interacts with AWS PCA.

1. Create the Issuer Resource:

   kubectl create namespace istio-system
   cat <<EOF | kubectl apply -f -
   apiVersion: awspca.cert-manager.io/v1beta1
   kind: AWSPCAIssuer
   metadata:
     name: aws-pca-issuer-istio
     namespace: istio-system
   spec:
     arn: ${ISTIO_CA_ARN}
     region: your-aws-region
   EOF

   Replace your-aws-region with the AWS region where your PCA is located.

2. Verify the Issuer:

   kubectl -n istio-system describe awspcaissuer aws-pca-issuer-istio

   Ensure the issuer's status is Ready.

Step 6: Create the Istio CA Certificate

Use cert-manager to generate a CA certificate for Istio.

1. Create the Certificate Resource:

   cat <<EOF | kubectl apply -f -
   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: istio-ca
     namespace: istio-system
   spec:
     isCA: true
     commonName: istio-ca
     duration: 2160h # 90 days
     renewBefore: 360h # 15 days before expiration
     privateKey:
       algorithm: RSA
       size: 2048
     secretName: cacerts
     issuerRef:
       name: aws-pca-issuer-istio
       kind: AWSPCAIssuer
       group: awspca.cert-manager.io
   EOF

2. Verify Certificate Creation:

   kubectl -n istio-system get certificate istio-ca
   kubectl -n istio-system get secret cacerts

   Ensure the cacerts secret is created and contains the certificate data.

Step 7: Install Istio

Install Istio by referring to the TIS document.

Step 8: Deploy Sample Applications

Deploy sample workloads to test the mTLS communication with the custom CA.

1. Deploy the Bookinfo Application:

   kubectl label namespace default istio-injection=enabled
   kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml

2. Verify Application Pods:

   kubectl get pods

   Ensure all application pods are running with sidecar injection.

Step 9: Verify the Certificate Chain

Confirm that the pods' certificates are issued by AWS PCA.

1. Inspect the productpage pod’s certificate trust chain:

   istioctl -n default proxy-config secret deployment/productpage-v1 -o json | jq -r \
   '.dynamicActiveSecrets[0].secret.tlsCertificate.certificateChain.inlineBytes' | base64 --decode > chain.pem

2. Split the certificate chain:

   split -p "-----BEGIN CERTIFICATE-----" chain.pem cert-

   You will see five certificates named from cert-aa to cert-ae. The certificate trust chain is structured as cert-aa -> cert-ab -> cert-ac -> cert-ad, with the last certificate cert-ae being the root certificate, identical to cert-ad. Check the issuer information in the certificate details to ensure it aligns with your AWS PCA configuration.

   The following figure shows the certificates trust chain.

   

3. Match Serial Numbers:

   • Inspect the serial number from the certificate:

     openssl x509 -in cert-ad -noout -serial | cut -d'=' -f2 | tr -d '\n' | xargs -I {} echo "ibase=16; {}" | bc
     openssl x509 -in cert-ac -noout -serial | cut -d'=' -f2 | tr -d '\n' | xargs -I {} echo "ibase=16; {}" | bc

   • Compare it with the serial number of the certificate in AWS PCA console.

Troubleshooting Tips

• Check AWSPCAIssuer Status:

  kubectl -n istio-system describe awspcaissuer aws-pca-issuer-istio

• Inspect CertificateRequest Objects:

  kubectl -n istio-system get certificaterequests
  kubectl -n istio-system describe certificaterequest istio-ca-xxxxx

• Review Logs:

  • cert-manager:

    kubectl -n cert-manager logs deploy/cert-manager

  • AWS PCA Issuer:

    kubectl -n cert-manager logs deploy/aws-pca-issuer-aws-privateca-issuer