Upgrade and Rollback

Containment Chamber ships as a single static binary or Docker image. Upgrades normally replace the binary/image and restart the process. SQL anti-slashing backends initialize missing tables on startup; DynamoDB tables must already exist and row-format changes are versioned in the data model.

Before You Upgrade

1. Check the changelog for breaking changes, especially storage-format, config, auth, or seal-state changes.
2. Back up your current binary or record the exact Docker/Helm image tag you’re running.
3. Back up anti-slashing state:
   • PostgreSQL: take a pg_dump or provider snapshot.
   • SQLite: stop the signer, then copy the SQLite file.
   • DynamoDB: enable point-in-time recovery or take an on-demand backup of the anti-slashing table.
4. For DynamoDB-backed deployments, also back up the validator-key table and signer-state table.
5. Save the current config and root/operator tokens needed to inspect the upgraded signer after restart.

Upgrade Steps

Bare Metal / systemd

# Stop the service
sudo systemctl stop containment-chamber

# Download new binary
curl -L -o containment-chamber \
  https://github.com/unforeseen-consequences/containment-chamber/releases/latest/download/containment-chamber-linux-amd64

chmod +x containment-chamber

# Keep the old binary for rollback
sudo cp /usr/local/bin/containment-chamber /usr/local/bin/containment-chamber.bak

# Install the new binary
sudo install -m 755 containment-chamber /usr/local/bin/

# Restart
sudo systemctl start containment-chamber
sudo systemctl status containment-chamber

Stop the service, replace the binary, and restart.

Current release artifacts publish containment-chamber-linux-amd64. Check the release notes before assuming another architecture is available.

Docker

# Docker Compose (recommended)
docker compose pull && docker compose up -d

# Standalone docker run
docker pull ghcr.io/unforeseen-consequences/containment-chamber:latest
docker stop containment-chamber
docker rm containment-chamber
docker run -d --name containment-chamber \
  -p 9000:9000 -p 3000:3000 \
  -v ./config.yaml:/config.yaml:ro \
  -v ./keystores:/keystores:ro \
  ghcr.io/unforeseen-consequences/containment-chamber:latest \
  server -c /config.yaml

Pull the new image and recreate the container. Use Docker Compose where possible; the standalone commands show the equivalent manual sequence.

Tip

Pin to a specific version tag (e.g. v1.2.0) instead of latest for production. This prevents accidental upgrades when containers restart.

Kubernetes / Helm

helm upgrade containment-chamber oci://ghcr.io/unforeseen-consequences/charts/containment-chamber \
  --namespace validators \
  -f values.yaml

kubectl -n validators rollout status deployment/containment-chamber

Use the same values.yaml you used for install, with only the image/chart values you intend to change. If you run Nitro Enclave mode, confirm the new EIF measurements and KMS policy updates before rolling the deployment.

After restarting, confirm the signer is healthy:

curl http://localhost:9000/upcheck
curl http://localhost:9000/healthcheck
curl http://localhost:9000/api/v1/eth2/publicKeys

For state-backed deployments, also confirm the seal state:

containment-chamber operator status \
  --auth-token env:CC_AUTH_TOKEN \
  --signer-url http://localhost:9000 \
  --allow-plaintext-signer

Note

When TLS is enabled, :9000 is health-only. Use the HTTPS listener, usually https://localhost:9443, for /publicKeys, signing, chamber, auth, and key-management APIs.

Zero-Downtime Upgrades

Containment Chamber has a 25-second graceful shutdown drain. When the process receives SIGTERM, it stops accepting new connections but finishes all in-flight signing requests before exiting. This means brief upgrades with a single instance won’t lose requests that are already being processed.

For true zero-downtime, run multiple instances behind a load balancer:

1. Configure your load balancer or orchestrator readiness check to use /healthcheck, not /upcheck.
2. Start the new instance and wait for it to pass health checks.
3. Stop the old instance. The 25-second drain ensures in-flight requests complete.
4. Remove the old instance from the load balancer pool.

/upcheck is liveness only. It returns 200 while the process is alive, including while the signer is sealed. /healthcheck returns ready only after a signer is present and the anti-slashing backend health poller succeeds.

Caution

SQLite anti-slashing uses exclusive file locking. You cannot run two instances against the same SQLite database simultaneously. For multi-instance or zero-downtime upgrades, use PostgreSQL or DynamoDB anti-slashing and make sure all instances can see the same validator keys.

Rollback

If something goes wrong after upgrading, first check the changelog for storage-format changes. Rolling back the binary is usually enough for config or runtime issues, but storage-format changes may require restoring the matching database or DynamoDB table backup.

Bare Metal

sudo systemctl stop containment-chamber
sudo mv /usr/local/bin/containment-chamber.bak /usr/local/bin/containment-chamber
sudo systemctl start containment-chamber

Docker

# Roll back to a specific version
docker compose down
# Update image tag in docker-compose.yml to the previous version
docker compose up -d

Or pull the previous tag directly:

docker pull ghcr.io/unforeseen-consequences/containment-chamber:v1.1.0

Kubernetes / Helm

helm rollback containment-chamber \
  --namespace validators

kubectl -n validators rollout status deployment/containment-chamber

If the chart values changed between releases, inspect the rollback revision with helm history containment-chamber --namespace validators before applying it.

After rollback, check /healthcheck, confirm the seal state, and verify validator clients reconnect before considering the incident closed.

Tip

Always test upgrades on a staging environment or testnet before applying them to your production validators.

Next Steps

• Production Hardening — verify your post-upgrade deployment is still locked down
• Observability — check metrics and alerts after the new version starts
• Troubleshooting — if the signer behaves unexpectedly after upgrade