Architecture Overview

Containment Chamber is an Ethereum remote signer designed to protect validator keys while remaining compatible with existing Web3Signer clients. It can run as a standard Linux process or inside an AWS Nitro Enclave, where decrypted validator keys and reconstructed master-key material stay inside enclave memory. Every signing request is checked against authorization rules, network boundaries, and EIP-3076 slashing protection before a BLS signature is produced.

Validator-key custody follows the kms_auto-static model: the chamber master key is Shamir-split across N AWS KMS keys; each share is individually KMS-wrapped and stored in the MASTER_KEY DynamoDB row. There are no operator passphrases, no quorum handshakes, and no manual unseal steps. The key set, threshold, and root-token recipients are static parameters — compiled into the binary for Nitro builds (PCR0-measured), or supplied in a trusted ceremony: config block for standard deployments.

It has two deployment modes:

• Standard mode — runs as a normal Linux process or container. The ceremony parameters live in the ceremony: config block and are trusted because the host is trusted. This is the direct Web3Signer replacement path for bare metal, Docker, Kubernetes, and conventional TLS deployments.
• Nitro Enclave mode — runs the signer inside an AWS Nitro Enclave. The parent EC2/Kubernetes pod only forwards bytes over vsock; decrypted validator keys and the reconstructed master key stay inside enclave memory. RA-TLS lets clients verify the enclave measurements before sending signing or operator traffic. The ceremony parameters are compiled into the enclave image and PCR0-measured — the binary itself is the trust anchor.

The signing pipeline is intentionally the same in both modes: backpressure, auth policy evaluation, network guard, anti-slashing, BLS signing, and key lookup. Nitro changes the trust boundary around that pipeline; it does not change what the pipeline does.

Standard Mode

direction: down

vc: "Validator Clients"
bp: "Backpressure\n(concurrency · timeout · load shedding)"
auth: "Auth Policy Check\n(HMAC token · scope · key binding)"
signer: "EthereumSigner\n(network guard → anti-slashing → BLS sign)"
keystore: "In-Memory Keystore"

key_sources: "Key Sources (boot)" {
  fs: "Filesystem\n(EIP-2335 keystores)"
  ddb_keys: "DynamoDB Key Source\n(master-key encrypted)"
}

chamber: "Chamber" {
  master_key: "Master Key\n(memory — Unsealed only)"
  watcher: "Background Watcher\n(SEAL_OVERRIDE · reconcile)"
}

ddb: "DynamoDB\n(MASTER_KEY row · validator keys · state)"
kms: "AWS KMS\n(custody key set)"
metrics: "Metrics\n(port 3000)"

vc -> bp: "signing request"
bp -> auth
auth -> signer: "authorized"
signer -> keystore: "key lookup"
key_sources.fs -> keystore: "load at boot"
key_sources.ddb_keys -> keystore: "load at boot"
chamber.master_key -> key_sources.ddb_keys: "decrypt validator keys"
ddb -> chamber.master_key: "MASTER_KEY row\n(KMS-wrapped shares)"
kms -> chamber.master_key: "KMS decrypt\n(auto-unseal)"
chamber.watcher -> ddb: "observe latch · reconcile"
signer -> metrics: "counters / histograms"

vc.style.fill: "#F6F8FA"
vc.style.stroke: "#6B7280"
vc.style.font-color: "#170206"
bp.style.fill: "#F6F8FA"
bp.style.stroke: "#6B7280"
bp.style.font-color: "#170206"
auth.style.fill: "#F6F8FA"
auth.style.stroke: "#6B7280"
auth.style.font-color: "#170206"
signer.style.fill: "#EEF2FF"
signer.style.stroke: "#4338CA"
signer.style.font-color: "#170206"
keystore.style.fill: "#EEF2FF"
keystore.style.stroke: "#4338CA"
keystore.style.font-color: "#170206"
key_sources.style.fill: "#F6F8FA"
key_sources.style.stroke: "#6B7280"
chamber.style.fill: "#FFF7ED"
chamber.style.stroke: "#C2410C"
ddb.style.fill: "#ECFDF5"
ddb.style.stroke: "#059669"
ddb.style.font-color: "#170206"
kms.style.fill: "#ECFDF5"
kms.style.stroke: "#059669"
kms.style.font-color: "#170206"
metrics.style.fill: "#F6F8FA"
metrics.style.stroke: "#6B7280"
metrics.style.font-color: "#170206"

Standard mode exposes the Web3Signer-compatible API directly over HTTP or file-based TLS. Validator clients send signing requests to Containment Chamber, which routes them through:

• Backpressure — load shedding, concurrency limits, and timeouts protect the runtime under overload.
• Auth policy evaluation — HMAC-hashed tokens are checked against route scope, validator key, and signing operation.
• EthereumSigner — the network guard rejects wrong-genesis requests, anti-slashing runs before every signature, and BLS signing only proceeds after the key lookup succeeds.
• Key sources — filesystem and DynamoDB load keys at boot; DynamoDB can refresh keys; Key Manager API imports are memory-only runtime key imports.
• Anti-slashing backends — PostgreSQL, SQLite, or DynamoDB provide check_and_update() before signing.

This is the lowest-friction path when the host is already trusted enough to hold decrypted validator keys in process memory.

Nitro Enclave Mode

direction: down

parent: "EC2 Parent / Kubernetes Pod (untrusted)" {
  proxy_in: "enclave-proxy\n(ingress vsock)"
  proxy_eg: "vsock-proxy\n(egress to AWS)"
  bootstrap: "config bootstrap\n+ log forwarding"
}

enclave: "Nitro Enclave (trusted)" {
  ratls: "RA-TLS Listener\n(PCR-bound ephemeral cert)"
  pipeline: "Signing Pipeline\n(same as Standard mode)"
  nsm: "NSM\n(attestation docs · CSPRNG seed)"
  attest_kms: "AttestedKmsClient\n(KMS decrypt with attestation)"
  state: "MasterKeyHolder\n+ DynamoDB state backend"
}

kms_aws: "AWS KMS\n(RecipientAttestation policy)"
ddb_aws: "AWS DynamoDB"

parent.proxy_in -> enclave.ratls: "vsock (RA-TLS verified)"
enclave.ratls -> enclave.pipeline
enclave.pipeline -> enclave.state
enclave.state -> enclave.attest_kms: "unwrap KMS shares"
enclave.attest_kms -> kms_aws: "attested decrypt"
enclave.state -> ddb_aws: "read / write (via egress)"
enclave.nsm -> enclave.ratls: "attestation doc in cert"
parent.proxy_eg -> kms_aws: "network forwarding"
parent.proxy_eg -> ddb_aws: "network forwarding"

parent.style.fill: "#FEF2F2"
parent.style.stroke: "#DC2626"
enclave.style.fill: "#F0FDF4"
enclave.style.stroke: "#16A34A"
kms_aws.style.fill: "#ECFDF5"
kms_aws.style.stroke: "#059669"
kms_aws.style.font-color: "#170206"
ddb_aws.style.fill: "#ECFDF5"
ddb_aws.style.stroke: "#059669"
ddb_aws.style.font-color: "#170206"

Nitro mode wraps the same signer pipeline in a stricter trust boundary:

• The EC2 parent instance / Kubernetes pod is untrusted. It runs nitro-cli, ingress enclave-proxy, egress vsock-proxy processes, config bootstrap, and log forwarding.
• The enclave is trusted. It holds decrypted validator keys, the reconstructed master key, and auth token secrets. The NSM generates attestation documents and seeds the CSPRNG at boot.
• Custody parameters are PCR0-measured. The CEREMONY static compiled into the enclave image specifies which KMS ARNs may decrypt shares. A rebuild with different custody parameters produces a different PCR0 and does not satisfy KMS key policies — every config change is observable and auditable.
• AttestedKmsClient binds each KMS decrypt to a valid NSM attestation document. The parent host cannot substitute plaintext responses or pre-compute decrypts for a different binary.

Nitro mode is the differentiated deployment path versus Web3Signer: a compromised parent host can observe traffic timing and move bytes, but cannot read enclave memory, plaintext validator keys, the reconstructed master key, or KMS plaintext shares. See Nitro Enclave Overview for the full deployment model.

Signing Path

The signing path is the same in Standard and Nitro Enclave mode:

• The validator client sends a Web3Signer-compatible signing request.
• Backpressure enforces concurrency limits and timeout budgets, protecting against overload.
• Auth policy evaluation verifies the HMAC-hashed token against the route scope, the specific validator key, and the signing operation type.
• Network guard rejects requests for a genesis fork hash that does not match the configured network.
• Anti-slashing calls check_and_update(), which atomically checks and records slashing-protection data. The signature is produced only after this succeeds.
• BLS signing proceeds only after the key lookup succeeds.

Nitro changes where this path runs, not what the path does. In Nitro mode, the parent instance forwards traffic over vsock while the signing path executes inside the enclave.

Key Sources

Every validator BLS keypair in memory is tagged with one of three sources: Filesystem, DynamoDB, or Memory. The taxonomy distinguishes how the key got there, not where it is now.

Boot-time backends

Two configured backends load keys when the signer starts:

• Filesystem (KeySource::Filesystem) — Web3Signer-compatible EIP-2335 keystores with local password files, or raw hex files, read from one or more directories. Read-only. No background refresh.
• DynamoDB (KeySource::DynamoDb) — keys stored encrypted (under the chamber master key) in a DynamoDB table. Refreshed at a configurable interval. Only available when the DynamoDB key source and state backend are configured.

Runtime key import

Keys can also be added after startup through HTTP APIs. Runtime imports can either stay memory-only or be persisted to DynamoDB:

• Key Manager API (POST /eth/v1/keystores) — Web3Signer-compatible import for EIP-2335 keystores. These keys are memory-only and vanish on restart. Validator-client-facing. Gated by http.key_manager_api.enabled.
• Chamber Key Management API — operator-facing DynamoDB-backed verbs (generate, import, lifecycle). Persisted keys survive restart and are encrypted under the master key. Gated per-verb: chamber.keys.{generate,import,lifecycle}.enabled.

Chamber Key Management API

A separate operator-facing surface tied to the DynamoDB key source:

• POST /api/v1/chamber/keys/generate — derive new validator keys (BIP-39 mnemonic → EIP-2333 → BLS), optionally back up the mnemonic age-encrypted to one or more public keys.
• POST /api/v1/chamber/keys/import — import EIP-2335 keystores into DynamoDB-backed storage.
• POST /api/v1/chamber/keys/{pubkey}/lifecycle — set per-key lifecycle state (Active, DrainSignatures, etc.).
• GET /api/v1/chamber/keys — list all DynamoDB-backed keys and their lifecycle states.

Each verb is gated by its own flag: chamber.keys.{generate,import,lifecycle}.enabled. List is always mounted. See DynamoDB + KMS for the full surface and request shapes.

Collision precedence

When the same pubkey arrives from more than one source, first loaded wins. Boot-time backends load in registration order (filesystem before dynamodb), so a filesystem-supplied key shadows a same-pubkey DynamoDB key. Runtime imports against an already-loaded pubkey are reported back to the caller as Duplicate (HTTP 200 with per-key status) and the original entry is preserved untouched.

Custody and State Machine

When the DynamoDB key source is configured, the signer operates a two-state custody machine driven entirely by the background watcher. The master key only exists in memory in the Unsealed state — all other states reject signing requests.

direction: down

unsealed: "Unsealed\nsigning enabled"
sealed: "Sealed — terminal\nsigning disabled"

unsealed.style.fill: "#CAF2E6"
unsealed.style.stroke: "#13A477"
unsealed.style.font-color: "#170206"
sealed.style.fill: "#FEEC8C"
sealed.style.stroke: "#D35F0A"
sealed.style.font-color: "#170206"

unsealed -> sealed: "break-glass seal\n(writes SEAL_OVERRIDE latch)" {style.stroke: "#D35F0A"}

The two states are:

• Unsealed — master key is reconstructed in memory; signing is enabled; the background watcher runs reconcile on each tick.
• Sealed — break-glass terminal: a SEAL_OVERRIDE latch row is present in DynamoDB. No key material is resident. There is no Sealed → Unsealed transition — recovery requires a manual IAM DeleteItem on the SEAL_OVERRIDE row followed by a restart of every replica.

Filesystem-only deployments bypass the custody machine entirely — they have no state backend and run statelessly.

Auto-init

On first boot with a ceremony configured and no MASTER_KEY row:

1. Generate a 32-byte random master key (NSM-seeded CSPRNG on Nitro).
2. Shamir-split into N shares, one per custody KMS key.
3. KMS-wrap each share under its custody key (attested on Nitro, using RecipientAttestation).
4. Pre-write self-test: KMS-decrypt all shares, Shamir-combine, verify the master-key commitment. Any failure — including a KMS key policy that has not yet admitted this PCR0 — aborts cleanly with no row written and is safe to retry.
5. Age-encrypt the root management token to the root_token_recipients list.
6. Atomic put_ceremony: write the MASTER_KEY row and the ROOT_TOKEN_BOOTSTRAP row in a single DynamoDB transaction.
7. Transition to Unsealed.

On an init race (two replicas both seeing an empty table), the loser’s conditional write conflicts. It discards its candidate key and reconstructs from the winner’s row via KMS auto-unseal. The fleet converges on one master key — no split-brain.

Auto-unseal

On boot with an existing MASTER_KEY row:

1. Read the MASTER_KEY row from DynamoDB.
2. KMS-decrypt at least kms_threshold shares (each attested on Nitro).
3. Shamir-combine the plaintext shares to reconstruct the master key.
4. Verify the master-key commitment (master_key_hash) stored in the row — detects substituted-share attacks.
5. Verify the share-binding HMAC (share_hmac) — a master-key-keyed HMAC over each share’s share_id and every wrap’s (kms_key_arn, blob), preventing an attacker with DynamoDB write access from renumbering or splicing shares.
6. Transition to Unsealed.

Background watcher

After unsealing, the watcher runs on every tick:

1. SEAL_OVERRIDE check — reads the latch row first; if present and HMAC-verified against the live master key, seals the local replica. Other replicas observe the same row on their own tick — the seal is fleet-wide within one watcher interval and survives restarts.
2. Reconcile (when Unsealed) — compares the MASTER_KEY row’s ceremony_config_hash + generation against the static. If the static is ahead in generation, re-wraps shares under the new KMS key set and re-creates the root token when the recipient set changed, all in a single generation-gated DynamoDB transaction.

Rotation is declarative: change the ceremony: config (or rebuild the Nitro image), bump generation, add old keys to retired_kms_keys, and redeploy. The watcher reconciles automatically after each replica unseals. There are no rotate CLI commands.

For the full state diagram, transition rules, and operational procedures, see Seal State Machine.

TLS and RA-TLS

When TLS is configured, the signer runs a second listener on port 9443 (HTTPS) alongside the HTTP listener on port 9000. Only the HTTPS listener serves the API routes; the HTTP listener is reduced to health probes (/upcheck, /healthcheck) so Kubernetes checks don’t need TLS. Prometheus scraping uses the separate metrics listener on port 3000, which is always plain HTTP.

File mode reads a PEM certificate and private key from disk and reloads them on a configurable poll interval. It works with any PKI: self-signed certs, Let’s Encrypt, cert-manager, or Vault PKI. Cert rotation happens without a restart.

RA-TLS mode (remote attestation TLS) generates an ephemeral X.509 certificate and binds its public key hash (SPKI) into an NSM attestation document embedded in the cert’s X.509 extension. This gives connecting clients cryptographic proof that:

• The server is running in a genuine AWS Nitro Enclave
• The enclave image has the expected PCR measurements (validated by the client’s RA-TLS verifier)
• The TLS private key was generated inside the enclave and never left it

Certs rotate automatically every rotation_interval_seconds. Active sessions complete on the old cert; new connections receive the fresh one. No restart needed.

The TOFU model (Trust On First Use): the first connection records the enclave’s PCR measurements. Future connections reject certificates from enclaves with different PCR values, even if those certs carry valid attestations from a different binary.

For setup instructions and the dual-listener port layout, see RA-TLS.

Next Steps

• Security Model — threat assumptions and what the signer protects against
• Production Hardening — lock down network exposure, tokens, file permissions, and audit logging
• Docker Deployment — deploy the architecture you just read about