Troubleshooting

This guide covers the most common errors you’ll encounter when operating Containment Chamber, with clear causes and fixes for each.

Diagnostic Commands

Before diving into specific errors, these commands help narrow down the problem:

# Is the signer running?
curl http://localhost:9000/upcheck

# Is the anti-slashing backend ready?
curl http://localhost:9000/healthcheck

# Which keys are loaded? Add Authorization if auth is enabled.
curl http://localhost:9000/api/v1/eth2/publicKeys

# State-backed deployments: inspect seal state and loaded-key counts.
containment-chamber operator status \
  --auth-token env:CC_AUTH_TOKEN \
  --signer-url http://localhost:9000 \
  --allow-plaintext-signer

# Verbose logging
RUST_LOG=containment_chamber=debug containment-chamber server -c config.yaml

Note

When TLS is enabled, :9000 is health-only. Use the HTTPS listener, usually https://localhost:9443, for /publicKeys, signing, chamber, auth, and key-management APIs.

HTTP 412 — Slashing Protection Triggered

Cause: EIP-3076 anti-slashing refused the signing request because it would produce a slashable message (double vote or surround vote).

This is expected behavior. The signer is protecting your validators from being slashed.

Danger

Do not retry the same request, disable slashing protection, or delete the slashing database. The 412 means the protection system is working correctly.

Common triggers:

• The same validator key is active in another validator client or signer
• Multiple signer instances are not sharing the same production anti-slashing backend
• A validator client replayed a duty with different signing data after a crash or failover
• Importing a slashing protection database from the wrong source or validator

What to check:

• Ensure only one active validator-client/signer path can use each validator key
• Confirm all horizontally scaled signers share PostgreSQL or the same single-region DynamoDB anti-slashing table
• If migrating validators, export the slashing DB from the previous signer and import it before starting
• See Anti-Slashing Protection for backend configuration

HTTP 404 — Key Not Found

Cause: The requested public key isn’t loaded. The signer doesn’t have the keystore for this validator.

Fix:

1. Verify the key is actually loaded:

   curl http://localhost:9000/api/v1/eth2/publicKeys

2. Check your key_sources.filesystem.paths configuration points to the correct directory
3. Verify file permissions: the keystore files must be readable by the service user
4. Check startup logs for load errors (wrong password, corrupt keystore, bad filename)
5. For DynamoDB-backed keys, confirm the key status is active; inactive keys are not loaded into the signer by default

For encrypted keystores: each .json keystore file needs a matching YAML descriptor and .password file alongside it. The YAML descriptor’s keystorePasswordFile field must point to the correct password file. See the Key Formats guide for the expected directory structure.

HTTP 400 — Bad Request

Cause: The signing request was rejected before any signature was attempted. Two cases produce a 400:

• Unsupported signing request — the request type is not one Containment Chamber implements, or the body does not match the expected schema for that type.
• Network mismatch — the validator client is configured for a different Ethereum network than the signer. The client-facing body is the generic network mismatch; the signer logs the expected vs received fork details server-side and intentionally does not disclose its configured network to the client.

Fix:

1. Confirm the validator client uses a supported Web3Signer operation — see Validator Clients.
2. Verify the signer and validator client target the same network — see Networks and Forks.

HTTP 401 — Unauthorized

Cause: The request arrived without an authorization header, with a malformed/unsupported authorization scheme, or while the state-backed auth evaluator is still sealed. In stateless mode, unauthenticated requests are rejected unless a static_auth.anonymous policy allows the route. State-backed deployments cannot use static_auth; create client tokens with the operator CLI instead.

Fix: Add the token to your request as an Authorization: Bearer <token> header:

curl -H "Authorization: Bearer your-token-here" http://signer:9000/api/v1/eth2/publicKeys

For state-backed deployments, first confirm the chamber is unsealed:

containment-chamber operator status \
  --auth-token env:CC_AUTH_TOKEN \
  --signer-url http://signer:9000 \
  --allow-plaintext-signer

See Auth Policies for full configuration details.

HTTP 403 — Forbidden

Cause: A bearer token was present, but it is invalid, expired, unrecognized, or the associated policy denies this specific request. The key may not be in the policy’s allowed keys, the operation may be blocked by the policy rules, or the route’s scope may not be permitted.

Fix:

1. Check which policy the token is bound to:

   containment-chamber operator auth token lookup \
     --accessor cc-client-... \
     --token env:CC_ROOT_TOKEN \
     --signer-url http://signer:9000 \
     --allow-plaintext-signer

2. Review the policy rules:

   containment-chamber operator auth policy get \
     --name validators \
     --token env:CC_ROOT_TOKEN \
     --signer-url http://signer:9000 \
     --allow-plaintext-signer

3. Verify the requested public key is permitted by the policy
4. Verify the operation and scope are allowed by the policy rules

See Auth Policies for the full policy behavior matrix.

HTTP 503 — Service Unavailable

Cause: The signer is not currently able to serve the request. Common causes include:

• The chamber is sealed or auto-init has not yet completed
• The state watcher is stale and signing fails closed
• The signing request concurrency limit is saturated
• The anti-slashing backend is unavailable, unhealthy, or failing row-integrity checks
• The master key became unavailable during a DynamoDB-backed operation

Fix: First check liveness, readiness, and chamber state:

curl http://localhost:9000/upcheck
curl http://localhost:9000/healthcheck
containment-chamber operator status \
  --auth-token env:CC_AUTH_TOKEN \
  --signer-url http://localhost:9000 \
  --allow-plaintext-signer

If the chamber is sealed or has not reached Unsealed, see Chamber State Issues below. If /healthcheck reports the anti-slashing backend as unhealthy, fix that backend before retrying signing duties.

If the signer is unsealed and backends are healthy, increase the signing limits in your config:

signing:
  queue_buffer_size: 4000
  max_concurrent_jobs: 2000

If you’re consistently hitting 503s, you may be running too many validators for a single instance. Consider scaling horizontally with a shared PostgreSQL anti-slashing backend.

Chamber State Issues

The chamber auto-inits directly to Unsealed on every boot (auto-init on fresh table, KMS auto-unseal on warm table). The only transition away from Unsealed is → Sealed via the break-glass latch. There is no Sealed → Unsealed transition — recovery from Sealed requires a restart.

Auto-Init Failed (Signer Not Reaching Unsealed)

Cause: Boot could not resolve a custody ceremony or complete auto-init / auto-unseal, so the process fails to start — it exits with an error rather than serving (there is no non-operational running state).

Common sub-causes:

• Missing ceremony: block (non-Nitro) — the YAML config does not include a ceremony: section, so the signer has no custody parameters and cannot auto-init. Add the ceremony: block with kms_keys, kms_threshold, root_token_recipients, and generation.
• Nitro build with a ceremony: block — Nitro builds reject the config-file ceremony: block; custody parameters must be compiled in. Remove the block and redeploy with the ceremony baked into the binary.
• State table unreachable — the signer cannot reach the DynamoDB state table. Check IAM permissions, VPC endpoints, and the signer_state.backend.table name.
• KMS keys unreachable at init — auto-init writes KMS-wrapped Shamir shares. If any key in kms_keys is unreachable, init will fail. See KMS Auto-Unseal Failures below.
• Threshold exceeds key count — kms_threshold must be ≤ the number of entries in kms_keys.

Fix: Correct the ceremony config or table/IAM issue and restart the signer. On a fresh table the signer will auto-init on the next boot attempt.

Note

Auto-init runs a recovery self-test before writing the MASTER_KEY row. If the self-test fails (KMS decrypt fails, Shamir recombination mismatch, or commitment check fails), boot aborts with no row written rather than persisting a bad master key; a restart retries once the cause is fixed.

Stuck in Sealed (Break-Glass Latch)

Cause: A SEAL_OVERRIDE row is present in the state table (pk attribute = SEAL_OVERRIDE). This latch is written by containment-chamber operator seal and persists across restarts. Every replica’s background watcher observes it and seals; the signer refuses to auto-unseal while it exists.

Recovery procedure:

1. Confirm the latch is present:

   aws dynamodb get-item \
     --table-name YOUR_STATE_TABLE \
     --key '{"pk": {"S": "SEAL_OVERRIDE"}}'

2. Delete the latch row (requires IAM dynamodb:DeleteItem on the state table — not the signer’s runtime role):

   aws dynamodb delete-item \
     --table-name YOUR_STATE_TABLE \
     --key '{"pk": {"S": "SEAL_OVERRIDE"}}'

3. Restart every replica. On the next boot, the signer will find no latch and will auto-unseal from the MASTER_KEY row.

Danger

There is no HTTP unseal endpoint. The latch can only be cleared by an IAM principal with explicit dynamodb:DeleteItem access to the state table — separate from the signer’s runtime role.

KMS Auto-Unseal Failures

Cause: The signer cannot reconstruct the master key from KMS-wrapped Shamir shares. This blocks both auto-init (fresh table) and auto-unseal (existing MASTER_KEY row).

Common sub-causes:

• KMS key unreachable — network connectivity to the AWS KMS endpoint is broken, a VPC endpoint is misconfigured, or the key is in a different region than the signer.
• IAM permission denied — the signer’s IAM role lacks kms:Decrypt on one or more custody keys. Check CloudWatch or the signer’s error logs for AccessDeniedException.
• KMS key disabled or deleted — the key was disabled, scheduled for deletion, or deleted from the KMS console. Check key status in the KMS console for each ARN in ceremony.kms_keys.
• Attestation policy mismatch (Nitro) — the KMS key policy includes a kms:RecipientAttestation condition and the enclave’s PCR0 measurement does not match. Verify the compiled-in ceremony matches the PCR0 value in the key policy.
• Threshold not met — auto-unseal requires at least kms_threshold KMS keys to be reachable and decryptable. If fewer than the threshold are available, unseal fails. Ensure enough keys are healthy across regions.
• HMAC verification failure — the share-binding HMAC stored in MASTER_KEY does not match what the signer computed. This indicates the master key row was tampered with or the wrong ceremony key set is configured.

What to check:

1. Run containment-chamber operator status to confirm the signer has not reached unsealed
2. Check signer logs for kms, auto_unseal, or auto_init error messages
3. Verify each key ARN in ceremony.kms_keys is accessible:

   aws kms describe-key --key-id arn:aws:kms:REGION:ACCOUNT:key/KEY_ID

4. Verify the signer’s IAM role has kms:Decrypt on each key
5. For Nitro builds, confirm PCR0 in the attestation matches the key policy condition

KMS ARN-Gate Refusals

Cause: The signer refused to use a KMS key because its ARN (or the ARN it resolved to) is not in the ceremony key set. The signer validates that every key it calls is one of the configured custody keys; a key that was removed from ceremony.kms_keys (or retired_kms_keys) is rejected even if IAM would allow the call.

Fix: Ensure the ARNs in ceremony.kms_keys (and ceremony.retired_kms_keys during a rotation rollout) exactly match the KMS key ARNs the signer holds shares for. For multi-Region keys, each replica ARN must be listed; aliases are not accepted.

Keys Not Loading

Keystores are loaded at startup and failures are logged per file. Run with debug logging to see exactly what is happening:

RUST_LOG=containment_chamber=debug containment-chamber server -c config.yaml

Common causes:

• key_sources.filesystem.paths directory doesn’t exist or isn’t readable
• Encrypted keystores missing their password file
• Password file is empty after trimming, or contains the wrong password
• Keystore JSON is malformed or uses an unsupported format
• File permissions too restrictive for the service user
• DynamoDB-backed keys exist but have a status that is not included in key_sources.dynamodb.status_filter

Quick checklist:

1. Does the directory exist? ls -la /path/to/keystores/
2. Are keystore files readable? Check ownership and permissions
3. For encrypted keystores, is the password file present and correct?
4. Do the logs show any errors during key loading?
5. For DynamoDB, do active keys appear on Chamber Key Management list responses?

Startup Failures

Containment Chamber validates all configuration at startup and prints clear error messages to stderr.

Common startup-blocking misconfigurations:

• static_auth set together with signer_state — the two are mutually exclusive. Remove static_auth for a state-backed deployment, or drop signer_state for a stateless one.
• anti_slashing.backend: noop without --allow-unsafe-noop-anti-slashing — the noop backend is refused at startup unless you pass the explicit override flag. Never use noop in production.
• Unresolved env:VAR_NAME reference — a config value uses the env: prefix but the named environment variable is unset.
• DynamoDB key source without signer_state — persistent DynamoDB keys require the signer-state backend because the master key and chamber state live there.
• RA-TLS without a TEE backend — production RA-TLS mode requires a TEE attestation provider; use tls.mode: file for traditional TLS outside an enclave.
• ceremony: block on a Nitro build — Nitro builds compile the ceremony in at PCR0-measurement time and reject the config-file block at startup. Remove the block.

Policy and token validation does not happen at startup. Policies and tokens are created at runtime through the Auth API, and invalid input (bad scope, unknown operation, malformed public key, over-long identifier) is rejected there with HTTP 400, not at boot.

Environment variable issues:

• Tokens using env:VAR_NAME syntax fail if the variable isn’t set
• Environment variables use CONTAINMENT_ prefix with __ for nesting (e.g., CONTAINMENT_ANTI_SLASHING__BACKEND)

Next Steps

• Configuration Reference — every option, env var, and CLI flag
• Auth Policies & Tokens — fix policy syntax and token format errors
• Observability — set up structured logging before the next incident

Still Stuck?

If none of the above matches your issue:

1. Run with RUST_LOG=containment_chamber=debug and check the full log output
2. Verify your config file parses correctly by starting with a minimal configuration
3. Check the Configuration Reference for all available options