Rotating the IAM Signing Key

The IAM signing key is the key TSB uses to sign every JWT it issues. Rotating it — replacing the key while still accepting tokens signed by the one it replaces, until those tokens expire — can be done two ways:

• Automatic rotation, where the operator manages the whole process on a schedule you configure. This only works when the operator generates and owns the signing key itself (the default), in the iam-signing-key secret it creates.
• Manual rotation, where you generate the new key yourself and edit the ManagementPlane token issuer configuration by hand. This is the only option if you bring your own signing key in a secret of your own — for example a key backed by the tsb-certs certificate, or a custom secret referenced by signingKeysSecret.

Automatic rotation

Available from TSB 1.14.4000

Set rotationPeriod on the first issuer in spec.tokenIssuer.jwt.issuers to have the operator rotate the signing key on that schedule. Only the first issuer signs tokens, so it's the only one that can be rotated; rotation is disabled by default (no rotationPeriod means the key is never rotated and stays in use until you replace it by hand).

apiVersion: install.tetrate.io/v1alpha1
kind: ManagementPlane
metadata:
  name: managementplane
  namespace: tsb
spec:
  tokenIssuer:
    jwt:
      issuers:
        - name: https://demo.tetrate.io
          rotationPeriod: 2160h            # 90 days
          rotationPropagationDelay: 30m    # default; shown for clarity

Field	Description
rotationPeriod	How often the signing key is replaced. Measured from when the current key started signing, so lowering it below the current key's age starts a rotation immediately. Unset (the default) disables rotation.
rotationPropagationDelay	How long a newly generated key is published for validation only before it starts signing, so that everything validating tokens (IAM, XCP Central, onboarded clusters) has picked it up first. Defaults to 30m; increase it if your slowest validator takes longer than that to pick up a rotated secret.
validationKeys	The key within the signing-keys secret used to store keys that are valid for validation only, not signing. Defaults to validationKeys — most installs don't need to change it.

Each rotation the operator runs goes through four steps:

1. Once rotationPeriod has passed, the operator creates a new key, saves it into the secret as a pending key, and adds its public half to validationKeys.
2. It waits rotationPropagationDelay for every validator to pick up the new key from validationKeys, so nothing rejects a token signed by it the moment it starts signing.
3. It retires the current signing key into validationKeys — tagged with when it was retired and when it can be pruned — and promotes the pending key to be the new signing key.
4. On later rotation cycles, any retired key past its prune time (refreshExpiration, plus a one-day buffer, after it stopped signing — so no token it signed can still be presented) is dropped from validationKeys, keeping the key set from growing forever.

Because rotation depends on retiring the outgoing key into validationKeys, it's rejected for the deprecated HMAC algorithms (HS256, HS384, HS512), whose keys can't be published for validation — use one of the RSA or ECDSA algorithms instead.

Signing key algorithm and size

spec.tokenIssuer.jwt.issuers[].algorithm selects the signing algorithm (RSA, RSA-PSS or ECDSA; see the full list). For the RSA-based algorithms, rsaKeySize sets the key size the operator generates — 2048 (the default), 3072 or 4096. It's ignored for the ECDSA algorithms, which take their curve from the algorithm itself (ES256 → P-256, ES384 → P-384, ES512 → P-521).

spec:
  tokenIssuer:
    jwt:
      issuers:
        - name: https://demo.tetrate.io
          algorithm: RS256
          rsaKeySize: 3072

Changing algorithm or rsaKeySize only takes effect the next time the operator generates a key for that issuer — it doesn't rotate an existing key on its own. Combine it with rotationPeriod, or a manual rotation below, to actually pick up the change.

ES384 and ES512 keys generated before this fix

A management plane that already runs an ES384 or ES512 issuer may be holding a P-256 key: earlier versions of the operator generated P-256 for every ECDSA algorithm, instead of the P-384/P-521 curve those algorithms require under RFC 7518. TSB itself accepted the resulting tokens, but third-party verifiers that enforce the curve rejected them.

The operator only (re)generates a key when the signing key secret doesn't already exist, so this isn't fixed automatically. To pick up the correct curve, delete the iam-signing-key secret in the management plane namespace and let the operator recreate it — this invalidates every token signed with the previous key.

Manual rotation

Use this procedure when you manage the signing key(s) yourself, in a secret the operator doesn't own — for example when migrating off the key backed by the tsb-certs certificate, or maintaining your own signing-keys secret.

Fetching the current signing key

First of all you need to retrieve the configuration for the token issuer with:

kubectl -n tsb get managementplane managementplane

And find the token issuer configuration. In this example, the following issuer is configured:

tokenIssuer:
  jwt:
    expiration: 3600s
    issuers:
    - name: https://demo.tetrate.io
      signingKey: tls.key
    refreshExpiration: 2592000s
    signingKeysSecret: tsb-certs
    tokenPruneInterval: 3600s

This indicates that the current signing key is the one in the tls.key entry in the tsb-certs secret. We can retrieve it and save it for later as follows:

kubectl get secret -n tsb tsb-certs -o jsonpath='{.data.tls\.key}' | base64 -d >/tmp/old-key.pem

Generating a new secret for the signing keys

The next thing is to generate a new secret with the new IAM signing key. This can be done in many ways, but in this example you'll create a new RSA key. Please refer to the supported key algorithms list for further details about the supported keys.

openssl genrsa -out /opt/iam-key.pem 2048
cat /opt/iam-key.pem

Output

-----BEGIN RSA PRIVATE KEY-----
MIIEpgIBAAKCAQEA+KdhvSZBExMHlaWo7MdKA8Ku55iu/y4FwMPixitjs/DUgaQ5
1AVHyuWcV576qMi1pZwFGbx72sU+oMS4BHr8JNv5a1DwwCKdidD89aAWeL5gmCdB
1gh5qrIBohvQQ5clnQnl7PXYauDohy9U5sIWzrZ1222sweYHVhD7A1Hd7864faR4
103xP/kyvT3b2kBauAXiLQoqFT7Uk0eR/uiJmjkl8lBFt/s3ApChRytxjxiDZiGW
x6Hw9rfEcgzu0gvpJpntCHY9WrdSO1YyXbWJ2C/59OwRkhqO1UOsl7QlHrWGGGYD
9CiGPahYhSt1qq01Dk6ievJQGv16Sd2Rv+rbNwIDAQABAoIBAQCqfOGX9k2yDV8q
7P3o8y+9alPQObDrCBwrsmOfqopfCyY5iWeZBtHVvR84OKn25j8dwN8CaWimdI1f
X+IoOEb/4s+eFE4t/s3ze5alt1EREr9aM7iBTyhUsF5MTzO51D2W8f1zPpFXnsPw
RLS6z6MhspsWi5ljDRxEl7nz6cL5M3LujW/bQMk/uG8noA2mRCywGij/6tEzytR9
+h7y0A2QU36YF6yS/amOyP+3LgpycyV6LMercABgnPUse7iLDWGg+uxPBTts78oS
b1YGe20cSTDrfrDHkgYXuUKRiI7blH9+VDgLR4iZHYSdr+8cZ8zxCKKGzbW7UClP
hNZ+nb9xAoGBAP620Azi+OplI1nLUetPm1X0VfZeMKg8w3gsw4DxECiKF9y5PPje
7E8DgQLl99fRNKGJoNCbdC5c6cwZv0MIiC2qyhTsaNiGZt+kGx3KcVQtjBu0sIyA
YFmWYNFbIkcu3W7ugVrLk74u2BPN8YQMVse2sa9ODWE9ZL2A0mBIqUe5AoGBAPno
vKanUg5Djk3CJPjOaDmUr9RS9Jiou/EdCWKjHwER8nNSQU/f/YKC0h8CGdiUA1HD
Jj353Rn2bSkB2DO3S64jkOr5GmXZIf5G8GCMIBkRlGHZtZoOUlWYZyitv34wuf91
e/T+50mvt3KWdvvgiG3CUpCs5sagccKJGTJYp9JvAoGBALbl6IDIXjpZQ0gQEhOo
xv6ygyN0QPYdI7LgWcX100d42WeZ76k40XBvMK03Gn9y7prr63i/l25PM2ZmOotU
zgwUriTWGPcZkzcVbI84taXfStL+LSPGbukFbSIHkZaRlVk5k9LxiXYvxuJ5p+nM
vmeLzQz3O+5OGk9k+CtBIaSpAoGBALZBIIvdjL8wT3Cv/OyjA2my4QRUt2M5806l
YXnZArxyDUJDI7SP4z8yDvFkQ9sqHr2bN6GNPs03ZWa5nKYisAPAlmh24OSUFPFv
ZNDUgHgn1PIDpyhB95PLALiu9e+es5b1ZEBJQf4AMyZTS1Tn7Dc3t6UhI3CKBEze
VUzdUQ7rAoGBAJ9y76IWic7PIBbstNOq0ejsq3iMEoH/fn84lYwMDEzRLV3Y+HvQ
mu69O2h7ud88ozXJntC0VTv2nU1cKpiMHq3jZ0vxNmJomd7wKxwunKAZj8GJczhm
8T+O1c682fgu4YPysGJw35j/oGed0pEKXhBMMJh/X8HPmBcujHZYXDy0
-----END RSA PRIVATE KEY-----

Then you will generate a new secret that contains the new key and the OLD key as well:

kubectl -n tsb create secret generic iam-signing-keys \
    --from-file=new.key=/opt/iam-key.pem \
    --from-file=old.key=/tmp/old-key.pem

This would create a secret like the following one, containing the old and the new key:

kubectl -n tsb get secret iam-signing-keys -o yaml

Output

apiVersion: v1
data:
  new.key: [...]
  old.key: [...]
kind: Secret
metadata:
  creationTimestamp: "2022-12-14T15:56:46Z"
  name: iam-signing-keys
  namespace: tsb
  resourceVersion: "3378979"
  uid: 54cea82b-4505-49bb-a12e-fe6f5fbee1de
type: Opaque

Updating the Management Plane to use the new keys

Once the secret with all the IAM signing keys has been created, all that is needed is to update the tokenIssuer in the ManagementPlane CR or Helm values accordingly. In our example, it would be as follows:

tokenIssuer:
  jwt:
    expiration: 3600s
    issuers:
    - name: https://newissuer.tetrate.io
      algorithm: RS256
      signingKey: new.key
    - name: https://demo.tetrate.io
      algorithm: RS256
      signingKey: old.key
    refreshExpiration: 2592000s
    signingKeysSecret: iam-signing-keys
    tokenPruneInterval: 3600s

The changes needed are:

• Update the signingKeysSecret to use the newly created secret containing the two keys.
• Declare two issuers, one for each key in the new secret. The first issuer in the list is the one that will be used to sign new JWT tokens, so make sure you put the new key first if you want it to be used to sign the new JWT tokens. The rest of the issuer's keys will be only used for token verification.

note

It is important that you choose a different issuer (it can be any string) for the new key and that you keep the old issuer for the old key. Otherwise token verification will not work.

Once the token issuer information has been updated in the ManagementPlane, new tokens will be issued with the new key, and old tokens will still be accepted. Once all tokens have been migrated and the old key is not needed anymore, the old issuer can be removed from the issuers list and the old key can be removed from the secret as well.