SKYHOUSE.dev Journal

Why: An adversarial review of our alerting redesign predicted that the monitoring scripts would pile up and hang under exactly the conditions they exist to detect. We checked, and it was true of all six of them.

The prediction was specific: a watchdog that calls docker inspect with no timeout will hang when the Docker daemon is saturated — which is precisely the failure it was written to catch — and because cron keeps firing it, instances accumulate. We went looking for counter-evidence and found none. Every one of the six cron watchers ran with no flock and no timeout, and the shared Telegram transport that all of them call had no network timeout either. During the 2026-08-03 incident, docker-log-guard.sh was firing every 15 minutes and rdtclient-watchdog.sh every 30, straight into a wedged daemon.

1. What was actually wrong

                          flock   timeout
cpu_watchdog.sh             0        0
dns-monitor.sh              0        0
rdtclient-watchdog.sh       0        0    ← untimed `docker inspect` / `docker logs`
docker-log-guard.sh         0        0
sonarr-queue-cleanup.sh     0        0
security-patch-check.sh     0        0

The nightly backup scripts were the exception — their 2026-06-02 and 2026-07-30 rewrites already carry an internal flock guard and a 6h timeout. That existing pattern is what the rest have now been brought up to.

2. The Telegram transport was worse than unbounded

/usr/local/bin/telegram_notify.sh ran curl -s with no --max-time and no --connect-timeout, redirected all output to /dev/null, and never checked the exit status. Two consequences: a network stall would hang whichever cron job called it (none of which had timeouts), and a failed alert was indistinguishable from a delivered one — the exact silent-failure mode the whole alerting redesign is meant to eliminate.

Rewritten with --connect-timeout 5 --max-time 15, two retries capped at 30s total, HTTP status captured, and a delivery audit log at /var/log/telegram_notify.log recording every send and every failure. Verified with a live send returning http=200.

3. The guards

Applied at the crontab level rather than inside each script — uniform, reviewable in one place, and no edits to seven working scripts:

*/5  flock -n /run/lock/cpu_watchdog.lock         timeout 120  cpu_watchdog.sh
*/5  flock -n /run/lock/sonarr-queue-cleanup.lock timeout 180  sonarr-queue-cleanup.sh
*/5  flock -n /run/lock/dns-monitor.lock          timeout 120  dns-monitor.sh
*/30 flock -n /run/lock/rdtclient-watchdog.lock   timeout 600  rdtclient-watchdog.sh
*/15 flock -n /run/lock/docker-log-guard.lock     timeout 300  docker-log-guard.sh   (root)
0 4  flock -n /run/lock/plex-check.lock           timeout 120  plex-check
0 5  flock -n /run/lock/security-patch-check.lock timeout 1800 security-patch-check.sh
30 3 flock -n /run/lock/arr-config-backup.lock    timeout 3600 arr-config-backup.sh

flock -n means a run that finds the previous one still going exits immediately instead of stacking. Locks live in /run/lock (tmpfs), so they clear on reboot, which is the correct semantics for a stale lock. Timeouts are sized to each job's real work — generous enough never to fire in normal operation, tight enough to bound a hang.

4. A latent bug found on the way

cpu_watchdog.sh had STATE_DIR="/var/tmp/cpu_watchdog|immich|handbrake" — at some point the process allowlist string had been pasted into the state-directory line. It never crashed, because mkdir -p happily created a directory with pipes in its name and both state files landed there consistently, so the cooldown and suspect-tracking logic worked. It was simply writing to a nonsense path, and the intended /var/tmp/cpu_watchdog sat stale beside it. Corrected, state migrated, malformed directory removed.

5. An accidental demonstration

Running dns-monitor.sh once as a smoke test produced this:

2026-07-19T23:05:02 WARN: not resynced 19m after change
2026-08-04T00:35:02 INFO: resynced to 97.113.247.79 after 21710m

21,710 minutes is just over 15 days. The script had been correct the entire time and had the answer in hand at every 5-minute run; it simply had no mechanism to say so more than once. That is the gap the alerting redesign exists to close, stated more clearly than any design document has managed. [See 2026-08-04 apex-cname-flattening for the underlying fix.]

Nothing new is being monitored — the same watchers simply can no longer hang, stack up, or silently fail to deliver.