Seal & Unseal Runbook

This runbook covers the procedures for operating a DynamoDB-backed chamber under the kms_auto-static custody model. For the state diagram, transition rules, and design model, see the Seal State Machine. For request and response schemas, see the API Reference.

The seal/unseal flow only applies when using the DynamoDB key source.

How the Chamber Unseals

There are no operator ceremony steps. The chamber is fully self-managing:

• First boot on a fresh table — if a ceremony is configured, the chamber auto-inits: generates the master key, Shamir-splits it, KMS-wraps each share, self-tests (decrypt + re-combine + commitment verify) before writing anything durable, then writes the MASTER_KEY row and a root token. The replica is unsealed without any operator action.
• Every subsequent boot — the chamber reads the MASTER_KEY row and reconstructs the master key from its KMS-wrapped shares (KMS auto-unseal). Again, no operator action required.
• Emergency stop only — the operator seal command is break-glass. Use it when you need to halt signing fleet-wide immediately. Recovery requires a manual IAM step and a restart.

Prerequisites

Confirm before deploying:

• containment-chamber binary is installed.
• The signer’s IAM role has kms:Decrypt and kms:GenerateDataKey on all ceremony KMS keys.
• The signer state DynamoDB table exists and the IAM role has read/write access.
• The DynamoDB key-source table exists (for the validator keys).
• On non-Nitro deployments: the ceremony: block is present in the config file.
• On Nitro: the ceremony static is compiled in and the KMS key policies admit the enclave’s PCR0 measurement.

Use env:VAR_NAME for sensitive flags so secrets do not appear in process listings:

export CC_AUTH_TOKEN="cc_token_..."

containment-chamber operator status \
  --auth-token "env:CC_AUTH_TOKEN" \
  --signer-url http://localhost:9000 \
  --allow-plaintext-signer

Ceremony Configuration

Non-Nitro: ceremony: Config Block

Add the ceremony: block to the signer config YAML. The block is the source of truth for custody parameters on non-Nitro deployments.

ceremony:
  generation: 1
  kms_threshold: 2          # Shamir M — minimum shares needed to reconstruct the master key
  kms_keys:                  # N logical KMS keys; master key is split M-of-N across them
    - arns:
        - "arn:aws:kms:us-east-1:123456789012:key/aaaaaaaa-0000-0000-0000-aaaaaaaaaaaa"
        - "arn:aws:kms:us-west-2:123456789012:key/aaaaaaaa-0000-0000-0000-aaaaaaaaaaaa-mrk"
    - arns:
        - "arn:aws:kms:us-east-1:123456789012:key/bbbbbbbb-0000-0000-0000-bbbbbbbbbbbb"
    - arns:
        - "arn:aws:kms:us-east-1:123456789012:key/cccccccc-0000-0000-0000-cccccccccccc"
  retired_kms_keys: []       # Old keys kept decryptable across a rotation; empty in steady state
  root_token_recipients:     # age X25519 public keys; the root-token bootstrap is encrypted to any of these
    - "age1qz2…"

Each entry in kms_keys is one logical (potentially multi-Region) key. List the primary-region ARN first, then any replica-region ARNs. A minimum of kms_threshold entries must be reachable for the chamber to unseal.

Note

A Nitro build rejects the ceremony: block — the compiled-in PCR0-measured static is authoritative. Presence of this block in a Nitro config file is a boot error.

Nitro: Compiled-In Static

On Nitro, the ceremony parameters are baked into the binary at build time and bound to the enclave’s PCR0 measurement. No config block is needed or allowed. See Nitro Enclave Deployment for how to build and configure the ceremony static.

First Boot

Start the signer:

containment-chamber --config /etc/containment-chamber/config.yaml

On first boot against a fresh table, the chamber:

1. Finds no MASTER_KEY row.
2. Runs auto-init: generates the master key, Shamir-splits it, KMS-wraps each share.
3. Self-tests: decrypts the shares via KMS and verifies the commitment before writing anything.
4. Atomically writes the MASTER_KEY row plus a root token (age-encrypted to root_token_recipients) stored in ROOT_TOKEN_BOOTSTRAP.
5. Transitions to unsealed.

The chamber is ready when status shows unsealed. No operator action is required.

Check Status

containment-chamber operator status \
  --auth-token env:CC_AUTH_TOKEN \
  --signer-url http://localhost:9000 \
  --allow-plaintext-signer

The response includes the current seal state (unsealed or sealed) and the ceremony generation. A healthy signer shows unsealed.

Emergency Seal and Recovery

Seal (Break-Glass)

Sealing stops all signing fleet-wide. Use only in a genuine emergency.

containment-chamber operator seal \
  --auth-token env:CC_AUTH_TOKEN \
  --signer-url http://localhost:9000 \
  --allow-plaintext-signer

This command:

1. Writes the HMAC-stamped SEAL_OVERRIDE row to DynamoDB.
2. Transitions the local replica to Sealed.
3. Zeroizes the master key from memory.
4. Revokes all auth tokens.

Every other replica’s background watcher observes the latch on its next tick and seals the same way. The seal is cluster-wide and restart-surviving — a replica restart without removing the latch comes back Sealed.

Danger

There is no unseal command. Recovery requires a manual IAM action and a full fleet restart.

Recovery from Sealed

1. Investigate and resolve the cause of the emergency seal before proceeding.

2. Delete the SEAL_OVERRIDE row using an IAM principal with dynamodb:DeleteItem on the state table:

   aws dynamodb delete-item \
     --table-name <signer-state-table> \
     --key '{"pk": {"S": "SEAL_OVERRIDE"}}'

3. Restart every replica. Each replica boots, finds no SEAL_OVERRIDE latch, and auto-unseals from the existing MASTER_KEY row.

4. Confirm recovery:

   containment-chamber operator status \
     --auth-token env:CC_AUTH_TOKEN \
     --signer-url http://localhost:9000 \
     --allow-plaintext-signer

   Expect unsealed.

Caution

The IAM role the signer itself uses must not have dynamodb:DeleteItem on pk = SEAL_OVERRIDE. Only a break-glass IAM principal (human operator or a tightly-scoped incident-response role) should hold that permission. This separation is what makes the seal non-self-reversible.

Rotation

Rotation is declarative — there are no rotate CLI commands. Change the ceremony static and redeploy; the background watcher reconciles the MASTER_KEY row automatically after unseal.

Rotate KMS Keys or Threshold

1. Edit the ceremony: block (non-Nitro) or rebuild the binary with a new CEREMONY static (Nitro):

   • Bump generation by 1.
   • Update kms_keys and/or kms_threshold with the new values.
   • Move the old kms_keys entries into retired_kms_keys — they must remain decryptable by KMS until every replica has reconciled.

   ceremony:
     generation: 2                  # bumped from 1
     kms_threshold: 2
     kms_keys:
       - arns: ["arn:aws:kms:us-east-1:123456789012:key/new-key-1"]
       - arns: ["arn:aws:kms:us-east-1:123456789012:key/new-key-2"]
       - arns: ["arn:aws:kms:us-east-1:123456789012:key/new-key-3"]
     retired_kms_keys:
       - arns: ["arn:aws:kms:us-east-1:123456789012:key/old-key-1"]
       - arns: ["arn:aws:kms:us-east-1:123456789012:key/old-key-2"]
       - arns: ["arn:aws:kms:us-east-1:123456789012:key/old-key-3"]
     root_token_recipients:
       - "age1qz2…"

2. Redeploy replicas in a rolling fashion.

3. After each replica unseals, its watcher compares the static’s ceremony_config_hash against the MASTER_KEY row. When it detects the generation bump, it re-wraps the (unchanged) master key under the new key set in a single DynamoDB transaction — proving the new keys can decrypt before retiring the old wraps.

4. Once all replicas are running the new generation, remove retired_kms_keys from the config and do a final redeploy to bump to the clean state.

The signer stays unsealed and serving throughout the entire rotation. No signing downtime.

Rotate Root-Token Recipients

To change who can decrypt the root-token bootstrap:

1. Update root_token_recipients in the ceremony static and bump generation.
2. Redeploy. The watcher reconcile pass detects the changed recipient set and re-creates the root token (age-encrypting the new ciphertext to the updated recipient list) inside the same DynamoDB transact that re-wraps the KMS shares.

The old root token is revoked atomically as part of the reconcile commit.

Root Token Re-Create

The root token is created once at auto-init and stored as an age-encrypted ciphertext in the ROOT_TOKEN_BOOTSTRAP DynamoDB row. To decrypt it, use the age private key matching one of the root_token_recipients public keys:

# Download the ciphertext and decrypt with age
aws dynamodb get-item \
  --table-name <signer-state-table> \
  --key '{"pk": {"S": "ROOT_TOKEN_BOOTSTRAP"}}' \
  --query 'Item.ciphertext.B' \
  --output text | base64 -d | age --decrypt -i ~/.age/identity.txt

If the root token is lost or the ROOT_TOKEN_BOOTSTRAP row is absent, re-create by:

1. Deleting the ROOT_TOKEN_BOOTSTRAP row:

   aws dynamodb delete-item \
     --table-name <signer-state-table> \
     --key '{"pk": {"S": "ROOT_TOKEN_BOOTSTRAP"}}'

2. Restarting the replicas. On the next unseal, the watcher reconcile pass detects the missing bootstrap row and re-creates the root token, age-encrypting it to the current root_token_recipients.

Caution

The re-created root token is a new token. Any system holding the old root token value will need to re-decrypt from ROOT_TOKEN_BOOTSTRAP using the appropriate age private key.

Attestation (Nitro)

On Nitro deployments, retrieve the NSM attestation document to verify enclave measurements:

containment-chamber operator attestation \
  --auth-token env:CC_AUTH_TOKEN \
  --signer-url https://enclave.example.internal:9000

The attestation document includes PCR0 (the enclave image measurement that custody parameters are bound to). See Nitro Enclave Operations for PCR verification and measurement-pinning procedures.

Troubleshooting

Symptom	Action
Signing returns 503	Check operator status; the signer may be sealed or auto-init has not completed.
Signer process fails to start (boot error)	A stateful deployment requires a ceremony and a reachable state table. Verify the ceremony: block is present and well-formed (non-Nitro) or compiled in (Nitro); confirm the IAM role has KMS and DynamoDB access; check boot logs for the specific error.
Status shows sealed	A SEAL_OVERRIDE latch is present. Follow the Recovery from Sealed procedure.
KMS decrypt denied	Check IAM role policy and KMS key policy; on Nitro, verify the enclave’s PCR0 is admitted in the KMS condition.
requires a custody ceremony, but none resolved	A stateful chamber needs a ceremony. Add a ceremony: block (non-Nitro) or compile it into the EIF via CHAMBER_CEREMONY_FILE (Nitro). Auto-init does not require a DynamoDB key store.
Watcher reconcile fails repeatedly	Check signer logs for reconcile failed; usually a transient KMS or DynamoDB error — will retry on the next watcher tick (H5). Ensure retired_kms_keys still includes all old key ARNs until reconcile completes.

For broader diagnostics, see Troubleshooting.

Disaster Recovery

If all KMS keys are lost and the MASTER_KEY row cannot be decrypted, the master key is unrecoverable through the normal path. The recovery path is the age-encrypted mnemonic backup created during key generation, if backup recipients are configured. See DynamoDB Key Source.

Next Steps

• Auth Policies & Tokens — create management tokens scoped to your operations after the chamber is unsealed
• Production Hardening — restrict the seal endpoint and set up audit logging
• Upgrading — safe version bumps that preserve the existing MASTER_KEY row