Production Hardening

Harden networking, process isolation, memory handling, auth, and observability for production

A remote signer holds your validator private keys. Every layer of your deployment should reflect that responsibility. This guide covers practical hardening steps you can apply today.

Network Exposure

Your signing port should never be reachable from the public internet. The right bind address depends on where the process runs:

Bare metal / VM: bind to loopback for same-host validator clients, or to a specific private interface for remote validator clients.
Docker / Kubernetes: bind inside the container or pod to 0.0.0.0 so the Docker port mapping or Kubernetes Service can reach it, then restrict exposure with host firewalls, private Services, NetworkPolicy, and cloud load-balancer settings.

Bare metal same-host example:

server:
  listen_address: "127.0.0.1"
  listen_port: 9000

metrics:
  listen_address: "127.0.0.1"
  listen_port: 3000

Kubernetes and Docker example:

server:
  listen_address: "0.0.0.0"
  listen_port: 9000

metrics:
  listen_address: "0.0.0.0"
  listen_port: 3000

Network controls to apply:

Port 9000 (signing API): allow only your validator client IPs
Port 3000 (metrics): allow only your monitoring system (Prometheus, Grafana, etc.)
Block all other inbound traffic to these ports

Token Security

If you’re running multiple clients against one signer, configure auth policies with per-client tokens.

Key rules for tokens:

State-backed auth tokens are generated at runtime and persisted only as HMAC-SHA256 hashes in DynamoDB.
State-backed token secrets are returned once at creation and never stored in plaintext.
Stateless static_auth token secrets live in config, but should use env:VAR_NAME so the clear-text secret comes from your secrets manager or runtime environment.
Static token secrets must be at least 16 characters. API-created tokens are prefixed (cc_token_... or cc_root_...) and generated by the server.
Prefer short-lived client tokens over broad long-lived management tokens.
Bind client tokens to source CIDRs with --bound-cidrs when validator-client egress IPs are stable.
Store management tokens in a secrets manager.

See Auth Policies & Tokens for the policy model and API Reference for request schemas.

File Permissions

Tight file permissions prevent other users on the system from reading your keys or config.

# Config file: owner read/write only
sudo chmod 600 /etc/containment-chamber/config.yaml

# Keystores directory: owner only
sudo chmod 700 /var/lib/containment-chamber/keystores

# Individual keystore files
sudo chmod 600 /var/lib/containment-chamber/keystores/*.json

# Ensure correct ownership
sudo chown -R containment-chamber:containment-chamber \
  /etc/containment-chamber \
  /var/lib/containment-chamber

The SQLite slashing protection database is created with 0600 permissions automatically.

systemd Hardening

On Linux with systemd, the service unit can enforce additional isolation. These directives are already included in the bare metal guide:

[Service]
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadOnlyPaths=/
ReadWritePaths=/var/lib/containment-chamber

This prevents the process from gaining new privileges, restricts filesystem access to what it actually needs, and isolates its /tmp.

Docker Security

When running in Docker, apply the principle of least privilege:

docker run \
  --user 1000:1000 \
  --read-only \
  --tmpfs /tmp \
  --cap-drop ALL \
  --cap-add IPC_LOCK \
  -v ./config.yaml:/config.yaml:ro \
  -v ./keystores:/keystores:ro \
  -v ./data:/data \
  ghcr.io/unforeseen-consequences/containment-chamber:latest \
  server -c /config.yaml

What each flag does:

--user 1000:1000 runs as a non-root user
--read-only makes the container filesystem immutable
--tmpfs /tmp provides a writable scratch space
--cap-drop ALL removes ambient Linux capabilities
--cap-add IPC_LOCK lets the binary’s file capability activate mlockall, preventing key material from being paged to swap
:ro mounts config and keystores as read-only

If your runtime forbids IPC_LOCK, the signer still starts, but logs a warning that memory locking could not be enabled.

Kubernetes Security

For Kubernetes, keep the application listener on 0.0.0.0 inside the pod and make the boundary private outside the pod:

config:
  server:
    listen_address: "0.0.0.0"
  metrics:
    listen_address: "0.0.0.0"

service:
  type: ClusterIP

netpolicies:
  ingress:
    enabled: true
    allowedNamespaces:
      - consensus-layer

The Helm chart is designed for a restricted container security context while preserving memory locking:

securityContext:
  readOnlyRootFilesystem: true
  allowPrivilegeEscalation: false
  capabilities:
    drop:
      - ALL
    add:
      - IPC_LOCK

Use an internal load balancer only when validator clients run outside the cluster. Avoid public Services or Ingress for the signing API.

Seccomp Syscall Filter

On Linux, you can restrict the signer to a minimal syscall allowlist using the kernel’s seccomp BPF filter. This limits what a code execution vulnerability can do — even if an attacker achieves arbitrary code execution, they can’t call execve, ptrace, or other dangerous syscalls.

server:
  seccomp: true  # opt-in, Linux only. Default: false

If the filter fails to apply (e.g., the kernel doesn’t support it or the process lacks CAP_SYS_ADMIN), the signer logs a warning and continues without the filter rather than refusing to start.

Canary Keys

Canary keys are designated validator public keys that should never sign in normal operation. When a canary key signs, the signer logs a warning and increments the containment_canary_signing_total metric. Signing proceeds normally — canary keys don’t block requests.

canary_keys:
  - "0x1234..."
  - "0x5678..."

Use canary keys to detect unauthorized access. If an attacker can submit signing requests, they’ll likely try to sign with whatever keys are loaded. A canary key that suddenly appears in your metrics is a strong signal that something is wrong.

Audit Logging

All security-relevant events are logged with target: "audit". This lets you route audit events to a separate sink — a SIEM, a write-once log store, or a separate file — without changing the rest of your logging configuration.

Events logged to the audit target include:

State transitions — seal machine state changes (e.g., Unsealed → Sealed)
Signing requests — every signing attempt, including the key and operation type
Seal operations — when the signer is sealed, and by whom

To capture audit events separately, configure your tracing subscriber to route the audit target:

# Include audit events at info level alongside normal logs
RUST_LOG=containment_chamber=info,audit=info

# Audit-only (suppress all other logs)
RUST_LOG=off,audit=info

In production, pipe JSON logs to a log aggregator and filter on "target":"audit" to build an audit trail.

Built-in Protections

Containment Chamber includes several protections that activate automatically:

Memory zeroization: private keys are zeroed from memory when they’re no longer needed
Core dump protection (Linux): the process marks itself as non-dumpable at startup, preventing key material from leaking into core dumps
Memory locking (Linux): mlockall(MCL_CURRENT | MCL_FUTURE) is attempted at startup so resident pages are not swapped to disk; grant IPC_LOCK in container runtimes so this succeeds
Token hashing: state-backed authentication tokens are HMAC-SHA256 hashed at creation time and persisted as hashes; stateless static_auth tokens are hashed into memory at boot
Constant-time comparison: token validation uses constant-time comparison to prevent timing attacks

Core dump protection, zeroization, token hashing, and constant-time comparison require no configuration. Memory locking is automatic when the process has the required IPC_LOCK capability or equivalent OS limit.

KMS Custody Key Hardening

The kms_auto-static model stores Shamir key shares in DynamoDB and reconstructs the master key via KMS. IAM configuration is the primary confidentiality defense; the ceremony ARN allowlist, master-key commitment, and share-binding HMAC are the integrity layer on top.

Custody KMS key policies

Every KMS key listed in the ceremony: block (or the Nitro PCR0-measured static) must require kms:RecipientAttestation on Decrypt and GenerateDataKey. Without this condition the parent role can decrypt shares outside the enclave and reconstruct the master key without attestation.

Minimum condition for each custody key policy:

{
  "Effect": "Allow",
  "Principal": { "AWS": "arn:aws:iam::123456789012:role/enclave-role" },
  "Action": ["kms:Decrypt", "kms:GenerateDataKey"],
  "Resource": "*",
  "Condition": {
    "StringEqualsIgnoreCase": {
      "kms:RecipientAttestation:PCR0": "<your-pcr0-hex>"
    }
  }
}

Do not grant kms:Encrypt without the attestation condition. Encrypt cannot be attestation-gated (it produces ciphertext, not a secret), so it must be limited to the enclave role only via IAM. The parent needs no kms:Encrypt right; reconcile re-encrypts shares using GenerateDataKey, which is attestation-gated.

See AWS KMS Key Policy for a complete policy reference.

DynamoDB table policy: SEAL_OVERRIDE deny

The parent role (the IAM role the EC2 instance or pod uses) must be explicitly denied write access to the SEAL_OVERRIDE item. Without this deny, a compromised parent can write a garbage SEAL_OVERRIDE row; the watcher will detect the HMAC mismatch (failing the freshness tick rather than sealing), but the 503 outage persists until the row is removed.

Add a Deny statement to the DynamoDB resource policy or an SCP:

{
  "Effect": "Deny",
  "Principal": { "AWS": "arn:aws:iam::123456789012:role/parent-role" },
  "Action": ["dynamodb:PutItem", "dynamodb:DeleteItem"],
  "Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/containment-chamber-state",
  "Condition": {
    "ForAllValues:StringEquals": {
      "dynamodb:LeadingKeys": ["SEAL_OVERRIDE"]
    }
  }
}

The enclave role still needs PutItem on SEAL_OVERRIDE to write the latch when the operator issues containment-chamber operator seal.

Least-privilege summary

Role	KMS actions	DynamoDB actions
Enclave role	`Decrypt` + `GenerateDataKey` (attested), `DescribeKey`	Full access to state and key tables
Parent role	None on custody keys	Read-only on state table; deny `PutItem`/`DeleteItem` on `SEAL_OVERRIDE`

Pinning security-critical config (Nitro)

The ceremony locks custody parameters into PCR0. The same build-time mechanism can lock other security-critical config leaves so a compromised parent cannot override them over the vsock config channel — most importantly the mnemonic-backup recipients (chamber.keys.backup.recipients), which an unpinned parent could redirect to its own age key to exfiltrate every generated mnemonic. A pinned leaf rejects a conflicting runtime value at boot (fail-closed). See Pinning configuration into the image.

Checklist

A quick reference for production deployments:

Next Steps

Auth Policies & Tokens — create policies and tokens after hardening the network
Observability — set up Prometheus metrics and audit-log routing
Troubleshooting — diagnostic patterns for when hardening breaks something

Production Hardening

Network Exposure

Token Security

File Permissions

systemd Hardening

Docker Security

Kubernetes Security

Seccomp Syscall Filter

Canary Keys

Audit Logging

Built-in Protections

KMS Custody Key Hardening

Custody KMS key policies

DynamoDB table policy: SEAL_OVERRIDE deny

Least-privilege summary

Pinning security-critical config (Nitro)

Checklist

Next Steps

Overview

Getting Started

Operate

Key Sources

Security

Nitro Enclave

Deployment

Reference

Internals