Proxmox Runner Disk Cleanup

Install, check and remove the self-cleaning setup on the two Linux GitHub Actions runner VMs on thb2: gh-runner-1 (VMID 101) and gh-runner-2 (VMID 111). See Desktop Dev Hosts for the guests themselves.

Use this runbook when:

• Rolling the cleanup out to a runner VM, or updating it after a change to the script
• Checking what is installed on a VM and in which mode
• The runner disk Slack alert fires, or a job fails with “Runner disk full”
• Removing the cleanup from a VM

What it does

One script, run from your laptop, installs the following on a VM:

Hooks run as runner and are wired into each slot’s .env between # BEGIN proxmox-runner-cleanup and # END proxmox-runner-cleanup. Timers run as root. Files land in /usr/local/lib/proxmox-runner/, /opt/runner-hooks/proxmox-runner-job-{started,completed}.sh, /etc/proxmox-runner/ and /etc/systemd/system/proxmox-runner-*.

Ownership: what counts as a slot’s leftovers

The reaper only touches containers and networks it can tie to a slot:

1. Runner label. The Actions runner labels everything it creates with the first 6 hex characters of the sha256 of the lowercased contents of the slot’s .runner file. The cleanup computes the same value.
2. CI label convention. A script that runs docker create or docker run itself (the runner does not label those) should add --label com.thehelperbees.ci.runner=$RUNNER_NAME. Containers with that label are reaped by the slot of that name.
3. Compose working directory. Compose containers whose com.docker.compose.project.working_dir is under the slot’s _work/ are reaped. A compose project’s networks and volumes are removed only when no container anywhere still carries that project label.

Never reaped: the nf-sync-lab compose project (it runs from inside a gh-runner-1 slot’s _work) and anything labelled com.thehelperbees.lifecycle=persistent. Label long-lived containers that must live on a runner VM this way.

Profiles

gh-runner-1 runs the thb2 lab on the same Docker daemon as its CI slots, with named volumes and stopped containers that must survive. The shared profile therefore never runs a VM-wide docker image, builder, volume or container prune. docker volume prune --all is never used anywhere.

Known gap on gh-runner-1: service-container volumes are not reclaimed. The Actions runner removes services: containers (postgres, redis and other images that declare a VOLUME) with docker rm --force and no -v, so each such job leaves one anonymous volume per service behind, after the job-completed hook can see it. Once its container is gone, an anonymous volume carries no label tying it to a slot, and on gh-runner-1 it cannot be told apart from anonymous volumes the lab created (99 dangling ones were there on 2026-10-28, of unknown origin). The ci profile reclaims them with docker volume prune; the shared profile cannot do so safely, so they accumulate on gh-runner-1, and the disk alert is the backstop. For scale, gh-runner-2 gathered 86 of them (6.5 GiB) in five days before cleanup existed; gh-runner-1 has 4 active slots instead of 18. Two ways to close it: jobs that use services: on the pool add options: --tmpfs /var/lib/postgresql/data (postgres) or --tmpfs /data (redis) so no volume is ever created, or the lab’s owner confirms which dangling volumes are the lab’s and the rest are removed by hand.

Thresholds

Prerequisites

• Office VPN. Connect WireGuard as WireGuard-Server-1-AJ. The guests are only reachable through thb2.
• Root SSH to thb2 (192.168.2.6). The script drives each VM with ssh root@192.168.2.6 qm guest exec <vmid>, so the VM needs no SSH key of its own.
• A clean known_hosts for thb2. The Mac’s ~/.ssh/known_hosts usually holds a stale entry for 192.168.2.6 and the connection is refused. Build a file from the cluster’s copy and pass it with --known-hosts:

# On thb2 (web UI shell or an already-trusted session), print the host key line(s)
cat /etc/pve/nodes/thb2/ssh_known_hosts

# On your laptop, save them to a file, e.g. ~/thb2_known_hosts, then:
./zig/zig build scripts -- proxmox_runner_cleanup check --known-hosts ~/thb2_known_hosts

Rollout

Roll out one VM at a time, gh-runner-2 first because it has the simpler profile and the more urgent problem.

1. Render and review (offline). Writes each host’s bundle and rendered config; touches nothing:

   ./zig/zig build scripts -- proxmox_runner_cleanup render --out /tmp/prc
   ./zig/zig build scripts -- proxmox_runner_cleanup render --out /tmp/prc --host gh-runner-2 --mode observe

2. Check the current state (read-only):

   ./zig/zig build scripts -- proxmox_runner_cleanup check --host gh-runner-2 --known-hosts ~/thb2_known_hosts

3. Install in observe mode. Everything is installed and the timers start at once. Idle slots get the hooks immediately; slots busy with a job show as restart pending until you rerun install (see Busy slots). Cleanup only logs “would …” lines and the guard never fails a job. Alerts still fire:

   ./zig/zig build scripts -- proxmox_runner_cleanup install --host gh-runner-2 --mode observe \
     --slack-webhook-file ~/runner-disk-webhook.txt --known-hosts ~/thb2_known_hosts

4. Read the logs for about a day (see Logs). Confirm the “would” decisions only name leftovers from finished jobs, and nothing you want to keep.

5. Switch to enforce:

   ./zig/zig build scripts -- proxmox_runner_cleanup install --host gh-runner-2 --mode enforce \
     --slack-webhook-file ~/runner-disk-webhook.txt --known-hosts ~/thb2_known_hosts

6. Repeat steps 2 to 5 for --host gh-runner-1. Extra care there: also confirm the lab containers and nf-sync-lab never appear in the “would remove” lines.

Optional on install: --snitch-url <URL> makes the nightly run ping a Dead Man’s Snitch check, so a dead timer pages instead of going unnoticed.

Busy slots and restart pending

A slot only picks up the new hooks when its actions.runner.* service restarts. Install restarts idle slots and reports busy ones as restart pending. Rerun the same install command once they are idle; it is idempotent. check also lists pending restarts.

Slack webhook

The disk alert posts to a Slack incoming webhook. Create one for #devops (Slack app settings, Incoming Webhooks), save the URL to a local file, and pass it with --slack-webhook-file. It is stored on the VM at /etc/proxmox-runner/slack-webhook-url, readable only by root.

CI jobs on these VMs have passwordless sudo, so a job can read that file. That is acceptable only because a webhook can do nothing but post to one channel. Never put a bot token or any broader Slack credential on these VMs. Without a webhook the alert only logs.

Logs

• In the job log. Hook output appears in the “Set up runner” step (job-started hook) and the “Complete runner” step (job-completed hook) of each job.
• Timers and alert, on the VM:

journalctl -u proxmox-runner-cleanup-hourly -u proxmox-runner-cleanup-nightly -u proxmox-runner-disk-alert --since "1 day ago"
systemctl list-timers 'proxmox-runner-*'

Reach the VM through thb2: ssh root@192.168.2.6, then qm guest exec 111 -- bash -c '<command>' (use 101 for gh-runner-1). Observe-mode decisions are the lines starting with would:.

When the alert fires or a job fails on disk

1. check --host <vm> for df, the mode, timer states and per-slot container counts.
2. If the mode is observe, nothing is being deleted. Switch to enforce once the logs look right.
3. Look at the nightly and hourly journal for errors. Run the service by hand on the VM if needed: systemctl start proxmox-runner-cleanup-nightly.
4. If it is still full, find what is big (docker system df, du -xh --max-depth=2 /home/runner). On gh-runner-1 check the lab (/opt/thb2, /srv/thb2) before blaming CI; the cleanup does not touch it.

Uninstall

Per VM, as root (via qm guest exec). Detach the slots before deleting the hook files, otherwise a slot that still points at a missing hook fails its next job at “Set up runner”.

1. Stop and disable the timers:

   systemctl disable --now proxmox-runner-cleanup-hourly.timer proxmox-runner-cleanup-nightly.timer proxmox-runner-disk-alert.timer

2. In every slot’s .env (/home/runner/actions-runner*/.env), delete the managed block from # BEGIN proxmox-runner-cleanup through # END proxmox-runner-cleanup:

   sed -i '/^# BEGIN proxmox-runner-cleanup/,/^# END proxmox-runner-cleanup/d' /home/runner/actions-runner*/.env

3. Restart the idle actions.runner.* services so they drop the hook variables. Busy slots keep the old hooks until they restart, so restart them after their job finishes and do not continue until every slot has been restarted.

4. Remove the units and files:

   rm -f /etc/systemd/system/proxmox-runner-cleanup-{hourly,nightly}.{service,timer} \
         /etc/systemd/system/proxmox-runner-disk-alert.{service,timer} \
         /etc/tmpfiles.d/proxmox-runner*.conf
   rm -rf /usr/local/lib/proxmox-runner /etc/proxmox-runner /var/lib/proxmox-runner
   rm -f /opt/runner-hooks/proxmox-runner-job-started.sh /opt/runner-hooks/proxmox-runner-job-completed.sh
   systemctl daemon-reload

Run check afterwards to confirm nothing is left.