Troubleshooting

Start every investigation with sudo hostcraftd doctor. It runs the full probe suite and points directly at failing components. If doctor exits 0, the daemon and its managed services are healthy; look at the application layer instead.

sudo hostcraftd doctor

Exit code 0 means all probes passed. Non-zero means at least one probe failed. The output identifies which probe failed and why. Work through the failures top to bottom.

Daemon refuses to start

Symptoms: systemctl status hostcraftd shows Active: failed, the unit exits immediately, no panel is reachable.

Step 1: Read the log.

journalctl -u hostcraftd -n 100

The daemon logs the exact failure reason before exiting. The most common causes are below.

Missing session_secret in config:

Log line: config: session_secret is required and must be at least 32 bytes

Fix: open /etc/hostcraft/config.yaml and add or correct the session_secret field. Generate a suitable value:

openssl rand -hex 32

Database file permissions:

Log line: state: open /var/lib/hostcraft/state.db: permission denied

Fix:

sudo chown hostcraft:hostcraft /var/lib/hostcraft/state.db
sudo chmod 600 /var/lib/hostcraft/state.db

If the file does not exist at all, the daemon creates it on first start. A missing file is not an error; a permission-denied on the parent directory is. Check:

ls -la /var/lib/hostcraft/
sudo chown -R hostcraft:hostcraft /var/lib/hostcraft/

Port 7700 already in use:

Log line: api: listen tcp 0.0.0.0:7700: bind: address already in use

Find and stop the conflicting process:

sudo ss -tlnp | grep 7700
# Note the PID from the output
sudo kill <pid>

If port 7700 must be changed, update HOSTCRAFT_LISTEN_ADDR in /etc/hostcraft/config.yaml and restart.

Invalid config.yaml:

Log line: config: unknown field "xyz" or config: yaml: unmarshal errors

The daemon refuses to start on a config schema violation. Open the config file and correct the offending field. Validate the file manually:

sudo hostcraftd config validate

Caddy fails to obtain a Let's Encrypt certificate for the panel hostname

Symptoms: the panel is accessible by IP but not by hostname; browser shows a certificate error; journalctl -u hostcraftd shows ACME errors.

Step 1: Confirm the DNS A record exists.

dig panel.example.com A +short
# Must return the server's public IP

If it does not resolve, add the A record and wait for propagation before trying again.

Step 2: Confirm ports 80 and 443 are reachable from the internet.

Let's Encrypt requires an HTTP-01 challenge on port 80 (or a TLS-ALPN-01 challenge on 443). If a firewall is blocking these ports, the challenge will fail.

# From an external machine
curl -I http://panel.example.com/.well-known/acme-challenge/test
# Expecting 404 (not a firewall timeout)

If the port is blocked, open it at the server firewall and any upstream firewall (cloud provider security groups, etc.):

sudo nft add rule inet filter input tcp dport { 80, 443 } accept

Step 3: Confirm the panel hostname matches what is in the config.

The ACME certificate is issued for the hostname in hostcraft.hostname in config.yaml. If the config has a different hostname than the DNS record, the challenge will fail with a domain-mismatch error.

grep hostname /etc/hostcraft/config.yaml

Correct the hostname and restart the daemon to trigger a fresh certificate request.

Step 4: Check Let's Encrypt rate limits.

If you have been issuing certificates for the same domain in quick succession (e.g. repeated reinstalls during testing), you may have hit the 5 duplicate-certificate per week limit. The ACME error log will say "too many certificates already issued". Wait for the rate limit window to reset (up to 7 days) or use a staging ACME URL for testing.

nginx -t fails after vhost create

Symptoms: a new site was just created via the panel, and the site is not serving; nginx -t shows a syntax error.

The configwriter always runs nginx -t before activating a new config and reverts on failure. If you are seeing a vhost that fails -t, the configwriter reverted it but the underlying template may have a bug, or a snippet path is wrong.

Step 1: Run nginx -t manually and read the error.

sudo nginx -t 2>&1

Step 2: Check the config snapshot.

The configwriter keeps a snapshot of every version:

ls /var/lib/hostcraft/configs/<domain>/
# You will see: current, previous, and timestamped snapshots
cat /var/lib/hostcraft/configs/<domain>/current

Compare the current config to the previous working config. The snippet path is the most common mismatch.

Step 3: Fix the snippet path.

nginx vhosts include shared snippets from /etc/nginx/hostcraft-snippets/. If a snippet was installed after the vhost was created, or if a PHP version changed, the snippet path in the vhost may reference a file that does not exist.

ls /etc/nginx/hostcraft-snippets/
# Confirm the snippet the vhost is referencing is present

If the snippet is missing, re-run the service install for that PHP version:

hostcraftd service install php-fpm --version=8.3

Then re-render the vhost config via the panel (Sites -> the site -> Config -> Re-render).

PHP-FPM pool not starting

Symptoms: a site returns 502; the FPM pool for the account is not listed in systemctl list-units | grep fpm.

Step 1: Check the per-pool systemd unit.

Each PHP-FPM pool runs as a separate systemd service. The unit name follows the pattern hostcraftd-fpm-<version>-acct_<id>.service:

systemctl status hostcraftd-fpm-8.3-acct_42.service
journalctl -u hostcraftd-fpm-8.3-acct_42.service -n 50

Step 2: Check for AppArmor denials.

If the pool starts but immediately exits, AppArmor may be blocking a required path. Check:

sudo dmesg | grep -i apparmor | tail -20
sudo journalctl -k | grep -i apparmor | tail -20

A denial looks like: apparmor="DENIED" operation="open" profile="hostcraftd-fpm-8.3-acct_42" name="/var/lib/hostcraft/...".

The AppArmor profile is generated by the daemon. If a path was added after the profile was generated, reload the profile:

sudo apparmor_parser -r /etc/apparmor.d/hostcraftd-fpm-*

Or, regenerate the profile via the panel: Sites -> the site -> Security -> Regenerate AppArmor profile.

Step 3: Check pool config syntax.

php8.3-fpm --test --fpm-config /etc/php/8.3/fpm/pool.d/acct_42.conf

If the config has an error, the configwriter snapshot will show the last working version. Re-render from the panel to attempt recovery.

Mail not delivering

Step 1: Check the PTR record.

Most receiving servers will reject mail from an IP without a PTR record. Confirm:

dig -x <server-ip> +short
# Must return a hostname, and that hostname must forward-resolve back to the IP

Step 2: Check RBL listings.

hostcraftd mail rbl-check --ip=<server-ip>

If listed, see IP Warmup: RBL Recovery for the delisting procedure.

Step 3: Check DKIM signing.

hostcraftd mail dkim-check --domain=example.com

Output shows whether the DKIM key in DNS matches the key in Rspamd's signing config. A mismatch causes DKIM=fail on receiving servers, which triggers spam filtering.

If there is a mismatch, rotate the DKIM key: Mail -> DKIM -> Rotate for the affected domain. After rotation, confirm the DNS record has propagated before testing again.

Step 4: Check the Postfix queue.

postqueue -p
hostcraftd mail queue

Deferred messages show the rejection reason from the receiving server. That reason is the most direct clue to what is wrong.

Backup failing

Step 1: Check the error.

hostcraftd backup runs --account=<account-id> --limit=5

The run log shows the exact restic error.

Step 2: Check the restic password file permissions.

restic requires the password file to be readable only by the daemon user:

ls -la /etc/hostcraft/backup-repos/
# Password files should be mode 600, owner hostcraft
sudo chmod 600 /etc/hostcraft/backup-repos/*.password
sudo chown hostcraft:hostcraft /etc/hostcraft/backup-repos/*.password

Step 3: Check backend connectivity.

# Test S3 connectivity (replace with your bucket and region)
hostcraftd backup test-backend --repo=<repo-name>

If the backend is unreachable, check network connectivity, firewall rules, and S3/B2 credentials in the repo config.

Step 4: Check disk space.

restic needs temporary scratch space during backup. Check available space:

df -h /var/lib/hostcraft/
df -h /tmp/

If disk space is critically low, free space before retrying.

WP vulnerability scan returning empty results

Symptoms: WP Toolkit shows no vulnerabilities found, but you expect there to be results; or the scan page is blank after running.

Step 1: Check the vulnerability log.

cat /var/log/hostcraft/wpvuln.log | tail -50

Look for rate-limit errors. The WPVulnDB free tier allows a limited number of API calls per day. If the log shows 429 Too Many Requests, the rate limit has been hit.

Step 2: Free-tier rate limit reached.

If your server has many WordPress installs, the free-tier API quota may be exhausted before all sites are scanned. Options:

• Upgrade to a WPVulnDB API key with higher limits and add it to config.yaml under wptoolkit.wpvuln_api_key.
• Reduce scan frequency: WP Toolkit -> Settings -> Scan schedule.
• The scan will resume the next day when the quota resets.

Cluster agent cannot join the panel

Symptoms: hostcraftd join command fails, or the node appears in "Pending" state and never becomes "Connected".

Step 1: Check the CA fingerprint.

The join token embeds a fingerprint of the panel's CA certificate. If the CA was rotated after the token was generated, the fingerprint will not match.

Generate a new join token from the panel (Settings -> Cluster -> Add node) and retry the join.

Step 2: Check clock skew.

mTLS certificates have a validity window. If the agent's clock is more than 5 minutes ahead of or behind the panel's clock, the certificate validation fails.

# On the agent
timedatectl
# Check "System clock synchronized: yes"
# If not:
sudo timedatectl set-ntp true
sudo systemctl restart systemd-timesyncd

Step 3: Check port connectivity.

The agent needs outbound TCP to the panel on port 7700 (REST API) and port 4222 (NATS). From the agent:

nc -zv panel.example.com 7700
nc -zv panel.example.com 4222

If either port is blocked, open it at the firewall on both the agent and the panel.

"Permission denied" on a panel action

Symptoms: a panel user clicks an action and sees a permission-denied error. The action appears to be available in the UI but returns a 403.

Step 1: Check the user's role.

Admin -> Users -> the user -> Role. Confirm the role has the required permission for the action. The permission required is logged in the audit log.

Step 2: Check the audit log.

hostcraftd audit search --actor=<user-email> --limit=20

The audit log records every permission-denied with the specific permission constant that was missing (e.g. web.site.delete). This is the most direct way to identify the gap.

Step 3: For sub-users, check the preset.

Sub-users operate under a preset that limits their permissions even if their role would otherwise allow an action. Admin -> Accounts -> the account -> Sub-users -> the sub-user -> Preset.

Confirm the preset includes the permission being requested. If it does not, update the preset or switch the sub-user to a broader preset.

Note: admin-only actions (account deletion, factory reset, license management) always deny non-admin roles regardless of preset. This is by design and cannot be overridden without changing the user's role.

Decoding doctor command output

hostcraftd doctor runs each module health check and prints one line per module:

[<module>] <level> <detail>

<level> is healthy, degraded, or unhealthy. Exit code 0 means no module reported unhealthy; degraded modules are warnings. The [security] line aggregates the WAF, CrowdSec, ClamAV, FIM and network monitor with a per-component breakdown in the detail.

Module	Common cause	Fix
nginx	Service stopped or a rendered config failed validation	nginx -t, then systemctl restart nginx
phpfpm	A PHP-FPM pool is down	systemctl restart php<ver>-fpm
caddy	Panel TLS front not running	journalctl -u caddy -n 100
security	A security component is down	Restart the named component, e.g. systemctl restart crowdsec