Rotating Keys, Tokens and Certificates

Keys, tokens and certificates should be replaced on a schedule and immediately
when one may have leaked. This guide shows how to replace each of them in
Mockarty, which ones can be replaced without stopping anything, and what a
replacement does to the data and the connections that depend on it.

Take a full backup before rotating an encryption key — see
Backup and Disaster Recovery.

TLS certificates

Mockarty reads its certificates from files: the HTTPS certificate
(HTTPS_CERT_FILE, HTTPS_KEY_FILE), the certificate of the port runners and
resolvers connect to, and the certificate of the mock broker ports. To rotate a
certificate, replace the certificate and key files. Mockarty notices the change
within about ten seconds and uses the new certificate for new connections —
no restart. Connections that are already open keep the old one until they
reconnect.

Write the new key first and the certificate last, or replace both files at once
(on Kubernetes, updating the certificate Secret does this). If Mockarty reads
the files while only one of them is new, it keeps serving the previous
certificate and tries again on the next connection.

Rotating the certificate authority for runner connections

When runners and resolvers present client certificates, the admin node trusts
the certificate authority listed in its client CA file. Rotate the authority
without breaking a connection:

1. Put both the old and the new authority into the client CA file on the admin
   node. It is re-read automatically.
2. Issue new client certificates from the new authority and roll them out to
   runners and resolvers. They pick up a replaced certificate on their next
   connection.
3. When no client uses a certificate from the old authority, remove it from the
   client CA file.

Rotate the authority that runners use to check the admin node the same way: add
the new authority to the runners’ trust file and restart the runners first, then
switch the admin node’s certificate.

The personal-data encryption key

When MOCKARTY_PII_ENCRYPTION_KEY is set, Mockarty encrypts personal data and
stored credentials with it. Rotate it without downtime:

1. Generate a new key: openssl rand -base64 32.

2. Set the new key as MOCKARTY_PII_ENCRYPTION_KEY and move the current one to
   MOCKARTY_PII_ENCRYPTION_KEY_PREVIOUS. Several previous keys can be listed,
   separated by commas.

3. Restart or roll the admin nodes. New data is encrypted with the new key;
   data written earlier stays readable through the previous key.

4. Re-encrypt the stored data with the new key, using the same settings as the
   server:

   mockarty --pii-rotate --pii-rotate-dry-run   # count first, change nothing
   mockarty --pii-rotate
   

     users.email: re-encrypted 1840, already current 12
     users.full_name: re-encrypted 7, already current 0
   Total: re-encrypted 1847, already current 12, refused 0, unreadable 0.
   Every stored value is under the current key. You can remove MOCKARTY_PII_ENCRYPTION_KEY_PREVIOUS.
   

   The command can run while Mockarty is working; it changes a value only if it
   still holds exactly what was read.

5. When the summary says every value is under the current key, remove
   MOCKARTY_PII_ENCRYPTION_KEY_PREVIOUS and restart.

Three results need attention:

• refused — the database does not allow those rows to change. Permanent
  audit-log entries keep the key they were written with, so keep the previous
  key configured for as long as you keep those entries.
• unreadable — no configured key opens those values: they were written with
  a key older than the previous ones. Add that key to
  MOCKARTY_PII_ENCRYPTION_KEY_PREVIOUS and run the command again.
• changed while running — Mockarty changed those values while the command
  was working, so the command left them alone. Run it again; keep the previous
  key until a run reports every value under the current key.

Keep every key you have used in your secret store with the dates it was in use:
restoring an old backup needs the key of that time.

The lookup pepper

MOCKARTY_PII_HMAC_PEPPER makes encrypted email addresses searchable. It cannot
be changed on an installation that already has users: with a new pepper,
existing accounts can no longer be found by email. Choose it once, store it with
your encryption keys, and treat a leak as you would a leaked database backup.

The chaos-cluster credential key

CHAOS_ENCRYPTION_KEY protects the Kubernetes credentials of connected chaos
clusters. Stored credentials cannot be moved to a new key: after changing it,
connect the clusters again so their credentials are saved with the new key.

API tokens

API tokens can overlap, so replacing one never breaks an automation:

1. Create a new token in Admin Panel > API Tokens, with an expiry date.
2. Put the new token into your CI system or scripts.
3. When nothing uses the old token any more, revoke it. Its owner is notified.

Revoke a token at once if it may have leaked; requests with it stop working
immediately on every node.

Runner tokens

The token a runner connects with expires 90 days after it was issued or last
rotated. Replace it before that, in Admin Panel > Integrations:

• Rotate Token issues a new token for the same connection and invalidates the old
  one immediately. Runners using the old token stop working until they get the
  new one, so use it when the runners can be updated right away or when the
  token has leaked.
• Without interruption: create a second connection with the same settings,
  move the runners to its token one by one, then delete the old connection.

The licence key

To replace the licence, load the new licence in the licence panel of the
administration panel. It takes effect immediately — no restart. See
Licensing Model.