Cluster Node Decommission

Decommissioning removes an agent node from the cluster: it revokes the node's mTLS certificate, removes the node from the registry, and propagates the CRL update to all remaining nodes. This runbook covers the normal path (reachable node), the force path (dead or unreachable node), and the required pre-work when customer accounts are hosted on the node being removed.

Pre-conditions

• You are logged in as server_admin or super_admin.
• You know whether the node is reachable. If it is not reachable and you need to remove it anyway, you will use force=true (see below).
• If customer accounts are hosted on this node: you must migrate them before decommissioning, or the saga will refuse. Do not start the decommission until account migration is complete. See "Migrating accounts off the node" below.
• No other decommission operations are in progress on the cluster.

Procedure

Step 1: Check for hosted accounts

Before triggering decommission, confirm whether any accounts are hosted on this node.

Admin -> Servers -> click the node -> Hosted accounts.

If the list is non-empty, complete the account migration steps first.

Step 2: Navigate to the decommission dialog

Admin -> Servers -> click the node -> Decommission.

Step 3: Review the drain option

The dialog presents a "Drain workflows before decommissioning" checkbox.

• Checked (default): the saga waits for all in-flight workflows on this node to finish before revoking the certificate. This is the safe choice when the node is still running and you want a clean handoff.
• Unchecked: the saga proceeds immediately. Any in-flight workflows on the node will be interrupted. The workflow engine records them as failed; compensations run where possible.

For a planned decommission with advance notice, leave the checkbox checked. For an emergency removal of a misbehaving node, uncheck it.

Step 4: Confirm by typing the hostname

The dialog requires you to type the node's hostname exactly to confirm. This prevents accidental decommission from a misclick.

Step 5: Watch the saga

The decommission saga runs four steps:

1. Drain (if selected): waits for active workflow runs on the node to finish. Timeout is 15 minutes; if workflows are still running after that, the saga pauses and alerts you.
2. Revoke cert: calls mtls.Authority.RevokeNode, adds the certificate to the CRL.
3. Remove from registry: deletes the node record from the cluster state.
4. CRL propagation: the updated CRL is published via NATS and received by all remaining nodes within a few seconds.

Step 6: Shut down the agent (if reachable)

After the saga completes, stop the daemon on the decommissioned node to prevent it from attempting reconnection:

sudo systemctl stop hostcraftd
sudo systemctl disable hostcraftd

Once the node's cert is in the CRL, the panel and all peer nodes will refuse connections from it. Stopping the daemon avoids noisy reconnection errors in the logs.

Decommissioning an unreachable node (force mode)

If the node is dead, partitioned, or otherwise unreachable, the drain step cannot run and the cert push cannot be confirmed. Use force=true to skip these checks.

Via the UI: the decommission dialog shows a "Force remove (node is unreachable)" option when the panel cannot reach the node. Check this and proceed.

Via the API:

curl -X POST https://panel.example.com/api/v1/cluster/nodes/<node-id>/decommission \
  -H "Authorization: Bearer $HOSTCRAFT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"force": true}'

With force=true:

• The drain step is skipped.
• The cert is revoked without waiting for agent acknowledgement.
• Any in-flight workflow runs on the node are marked failed.

Note: if the dead node later comes back online, its daemon will attempt to connect to the panel and be refused (cert is in the CRL). The operator on that machine will see connection-refused errors and should stop and disable the daemon.

Migrating accounts off the node

The saga refuses to revoke the certificate while accounts are still registered to the node, unless force=true is used (which can leave those accounts inaccessible). Migrate accounts first.

For each account on the node:

Step 1: Create a backup on the source node

hostcraftd backup run --account=<account-id> --repo=<backup-repo>

Wait for the backup to complete and verify it:

hostcraftd backup verify --account=<account-id> --snapshot=latest

Step 2: Restore on the target node

On the target node (or via the panel pointed at the target node):

curl -X POST https://panel.example.com/api/v1/restore/run \
  -H "Authorization: Bearer $HOSTCRAFT_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "account_id": <account-id>,
    "source_snapshot": "<snapshot-id>",
    "target_node": "<target-node-id>"
  }'

The restore.run workflow handles file restore, database restore, DNS, mail, and SSL on the target node.

Step 3: Verify the account on the target node

hostcraftd site verify --account=<account-id> --node=<target-node-id>

Confirm HTTP probes pass, mailboxes authenticate, and DNS resolves correctly.

Step 4: Update the account's node assignment

Once restore is verified, update the account's node assignment in the panel so future workflow runs target the new node:

Admin -> Accounts -> the account -> Settings -> Node -> change to target node -> Save.

Step 5: Repeat for all accounts

Repeat steps 1 to 4 for every account hosted on the node being decommissioned. Only then proceed with the decommission saga.

Verification

After the saga completes:

1. The node no longer appears in Admin -> Servers.
2. The node's certificate appears in the CRL:

   hostcraftd cluster crl list | grep <node-hostname>

3. If the agent daemon is still running on the decommissioned machine, its connection attempts are refused:

   # On the decommissioned node (optional check)
   journalctl -u hostcraftd -n 50 | grep "certificate\|refused\|revoked"

4. All migrated accounts serve correctly from their new node.