Cluster Certificate Rotation

Each agent node in a HostCraft cluster authenticates to the panel over mTLS using a certificate issued by the panel's internal CA. This runbook covers rotating that certificate: when to do it, the four-step saga the panel runs, what happens during the 10-day grace window, and how to recover if rotation fails.

When to rotate

Scheduled rotation (30 days before expiry): The panel flags certificates approaching expiry in Admin -> Servers. The UI pins a warning to the node row 30 days out. You should rotate before the warning becomes an alert (14 days out).

Emergency rotation (immediate): If you suspect the certificate private key has been compromised (server breach, key file exposed in logs, unauthorized access), rotate immediately without waiting for a scheduled window.

Pre-conditions

• You are logged in as server_admin or super_admin.
• The agent node is reachable from the panel (the rotation saga pushes the new cert to the agent).
• If the agent is not reachable, see "If push to agent fails" below before starting.
• There are no in-progress workflow runs on the node that you cannot afford to interrupt. Check Admin -> Servers -> the node -> Active workflows.

Procedure

Step 1: Navigate to the node

Admin -> Servers -> click the node name.

Step 2: Open the rotation dialog

Click "Rotate certificate". A dialog opens explaining the four saga steps:

1. issue_new_cert: the panel's mTLS CA issues a new certificate for this node's selector.
2. push_to_agent: the panel pushes the new certificate to the agent's hostcraftd instance over the existing mTLS connection.
3. await_acknowledgement: the agent confirms it has written the new cert and can present it.
4. retire_old_cert: the panel marks the old certificate as retired in the cluster_cert_rotations ledger. The old cert remains accepted for 10 days.

Step 3: Confirm

Click "Rotate" in the dialog. The saga starts. You can watch progress in the workflow panel at the bottom of the node detail page.

The full rotation typically completes in under 30 seconds on a reachable agent.

During the 10-day grace window

After retire_old_cert completes, both the old certificate and the new certificate are valid. The agent presents the new certificate for outgoing connections. The panel accepts the old certificate on any inbound connection during the grace window.

This grace period exists so that:

• Automated processes that cached the old certificate have time to pick up the new one.
• A failed rotation can be diagnosed and retried without immediately severing the agent connection.

The grace window does not extend if you start a second rotation. Starting a new rotation immediately retires the previous grace-period cert.

Verification

After the rotation completes:

1. In Admin -> Servers, the node should show a green "Connected" status.
2. In Admin -> Servers -> the node -> Certificate, the issued-at date should reflect today.
3. Check the cluster_cert_rotations ledger:
   • Via the UI: Admin -> Servers -> the node -> Certificate history.
   • The most recent entry should show status: success.
4. Run a health probe against the agent:

   hostcraftd cluster probe --node=<hostname>

   A successful probe confirms the agent is accepting connections on the new cert.

If push to agent fails

The saga fails at push_to_agent if the agent is unreachable (network partition, daemon stopped, etc.).

If the agent is reachable but the daemon is stopped: start the daemon on the agent, then re-run the rotation from the dialog. The saga resumes from where it failed.

If the agent can pull the cert manually: on the agent node, run:

sudo hostcraftd cert refresh

This command pulls the pending certificate from the panel and installs it. After it completes, return to the panel and confirm the rotation in the workflow panel.

Note: hostcraftd cert refresh is the intended recovery path. For v1, if the command is not yet available in your build, copy the new certificate file to the agent manually and restart the daemon:

# On the agent node
sudo systemctl stop hostcraftd
# Copy the new cert from the panel (the file path is shown in the rotation dialog)
sudo cp /tmp/hostcraftd-node.pem /etc/hostcraft/cluster/node.pem
sudo systemctl start hostcraftd

Then confirm the rotation in the panel UI.

If the agent is permanently unreachable (dead node): do not complete the rotation. Instead, decommission the node with force=true. See the Cluster Node Decommission runbook.

Rollback

If the rotation completed but you need to revert (for example, the new cert was issued with the wrong attributes):

1. Navigate to Admin -> Servers -> the node -> Certificate history.
2. Find the previous certificate's selector.
3. Revoke the new certificate:
   • Via the API: POST /api/v1/cluster/nodes/<id>/cert/revoke with {"selector": "<new-cert-selector>"}.
   • This calls mtls.Authority.RevokeNode internally and adds the cert to the CRL.
4. Re-issue a certificate from the previous selector:
   • Via the API: POST /api/v1/cluster/nodes/<id>/cert/reissue with {"from_selector": "<prev-selector>"}.

The CRL update propagates to all nodes via NATS within a few seconds.

If you are not sure which selector to use, the cluster_cert_rotations ledger entries include selector, issued_at, and retired_at for each rotation.