Nitro Enclave Operations

Release and PCR0 Rotation

PCR0, exposed in KMS policies as ImageSha384, changes whenever the EIF changes. For normal upgrades, keep this workflow separate from first-time deployment: AWS Setup explains how to create the signing certificate and initial measurements, while this page covers rotating an already-deployed enclave release.

With reproducible builds, each key holder can independently verify the new PCR0 before approving it in their KMS key policy.

1. Build the new Nitro pod image and capture PCR0:

   just docker-enclave \
     signing_cert.pem \
     signing_key.pem \
     ghcr.io/unforeseen-consequences/containment-chamber:v1.5.0-nitro \
     custody/ceremony.ron custody/pinned.ron custody/config-signers.ron

   The recipe builds the enclave image, signs the EIF, embeds it in the parent pod image, and prints the PCR0 value to approve in KMS.

2. Send PCR0 to all key holders — each holder independently verifies by rebuilding the same source and comparing PCR0:

   # Verifiers need the same source checkout (incl. committed RON files); no signing key needed.
   RON_ARGS=()
   [ -f custody/ceremony.ron ]        && RON_ARGS+=(--build-arg CEREMONY_FILE=custody/ceremony.ron)
   [ -f custody/pinned.ron ]          && RON_ARGS+=(--build-arg PINNED_FILE=custody/pinned.ron)
   [ -f custody/config-signers.ron ]  && RON_ARGS+=(--build-arg SIGNERS_FILE=custody/config-signers.ron)
   docker build -f Dockerfile.enclave --target enclave-image \
     --build-arg SOURCE_DATE_EPOCH="$(git log -1 --pretty=%ct)" \
     --build-arg VERGEN_GIT_SHA="$(git rev-parse HEAD)" \
     --build-arg VERGEN_GIT_DESCRIBE="$(git describe --always --dirty 2>/dev/null || git rev-parse HEAD)" \
     "${RON_ARGS[@]}" -t verify:local .
   
   nitro-cli build-enclave \
     --docker-uri verify:local \
     --output-file verify.eif
   
   nitro-cli describe-eif --eif-path verify.eif | jq -r '.Measurements.PCR0'
   # Must match the PCR0 you received

3. Each key holder updates their KMS key policy (add new PCR0, keep old for zero-downtime):

   terraform apply \
     -var 'enclave_pcr0_hashes=["NEW_PCR0", "OLD_PCR0"]'

4. Deploy the new enclave image:

   helm upgrade containment-chamber \
     oci://ghcr.io/unforeseen-consequences/charts/containment-chamber \
     --reuse-values \
     --set image.tag=v1.5.0-nitro

5. Remove the old PCR0 after all instances are updated:

   terraform apply \
     -var 'enclave_pcr0_hashes=["NEW_PCR0"]'

PCR8 Rotation (Signing Certificate)

PCR8 changes only when you rotate the EIF signing certificate. This is rare and should not be part of routine release upgrades.

1. Generate a new signing keypair and certificate.
2. Extract the new PCR8 hash with nitro-cli pcr --signing-certificate signing_cert.pem.
3. Build and sign a new EIF with the new certificate.
4. Add the new PCR8 to every Shamir-share KMS key policy, keeping the old PCR8 during the transition.
5. Deploy the new enclave image.
6. Remove the old PCR8 from every Shamir-share KMS key policy after all pods run the new image.

Debug Mode

Debug mode breaks attestation

In debug mode, the NSM produces attestation documents with all PCR values set to zero. KMS will reject these documents if your key policy requires specific PCR values. Never use debug mode in production.

Debug mode is blocked by the entrypoint when PRODUCTION=true:

# This will fail with FATAL error:
kubectl set env deployment/containment-chamber \
  ENCLAVE_DEBUG=1 PRODUCTION=true

For development, omit PRODUCTION=true:

kubectl set env deployment/containment-chamber ENCLAVE_DEBUG=1
# Then attach to the enclave console:
kubectl exec -it <pod> -- sh -c \
  'nitro-cli console --enclave-id "$(nitro-cli describe-enclaves | jq -r ".[0].EnclaveID")"'

Log Forwarding (vsock)

Tracing events emitted inside the enclave are forwarded over vsock to a listener in the parent pod, which writes them to the pod’s stdout. The parent and enclave logs share the same tracing-subscriber format — there is no [enclave] prefix; events are interleaved.

• Vsock port: 7000 (wire-protocol constant, not configurable).
• Parent listener: containment-chamber-enclave-proxy vsock-stdout is started by scripts/enclave-entrypoint.sh before nitro-cli run-enclave.
• Backpressure: the enclave-side writer is non-blocking with bounded buffering (tracing-appender). When the vsock connection is unavailable or backed off, individual events are dropped and counted by containment_enclave_log_events_dropped_total. Connection retries use exponential backoff (100 ms → 5 s) with ±20% jitter.
• Shutdown flush: a WorkerGuard is held for the lifetime of the server so that buffered events are flushed before exit.

kubectl logs <pod> -f
# Tracing events from both the parent process (proxies, init) and the enclave
# stream to stdout in the same format. Filter enclave-only spans by the fields
# emitted from inside (e.g. enclave_id, attestation_*).

# Verify drops are zero (or steady) under load:
kubectl port-forward svc/containment-chamber-metrics 3000:3000
curl -s http://localhost:3000/metrics | grep enclave_log_events_dropped_total

If containment_enclave_log_events_dropped_total is climbing, investigate vsock saturation, the listener’s stdout consumer (kubelet), or the parent CPU budget. The metric is always exposed regardless of whether the nitro feature is built.

Troubleshooting

Enclave fails to start

kubectl exec -it <pod> -- nitro-cli describe-enclaves
# Check: is the enclave in "running" state?
# If empty: nitro-cli run-enclave failed — check pod logs
kubectl logs <pod>

KMS attestation rejected

Symptoms: enclave starts but fails to reconstruct master key. Check:

1. PCR0 in KMS key policy matches the deployed EIF
2. PCR8 in KMS key policy matches the signing certificate
3. The signer IAM role has permission to call KMS

# Extract PCR values from the running enclave
kubectl exec -it <pod> -- nitro-cli describe-enclaves | jq '.[0].Measurements'

vsock connectivity issues

# Check egress proxies are running
kubectl exec -it <pod> -- sh -c "ps | grep '[v]sock-proxy'"

# Check which endpoints the entrypoint rendered and started
kubectl logs <pod> | grep 'spawning vsock-proxy'

Entropy seeding failure

If the enclave fails to initialize the NSM device (logs show failed to initialize NSM device — is /dev/nsm available? or initializing NSM — is /dev/nsm available?), the /dev/nsm device is not available. Verify:

• The node has Nitro Enclaves enabled in the launch template
• The device plugin DaemonSet is running on the node
• The pod has aws.ec2.nitro/nitro_enclaves: "1" in resource limits

Next Steps

• Enclave Deployment — Helm values and vsock ingress/egress configuration
• RA-TLS — attested TLS and client verification
• Observability — Prometheus metrics and structured logging for enclave workloads