SKYHOUSE.dev Journal

Why: Three nagging problems surfaced together — reboots still paused on a SMART-related hang despite the 06-01 smartd fix, the plex1 nightly backup had been silently failing, and a public-IP change "updated DNS" yet the server wasn't findable. Each turned out to have a clean root cause once chased down.

Follow-up to yesterday's reboot fixes. Three independent issues, three fixes.

1. The reboot pause was netdata, not smartd

Yesterday I took the flaky Kingston-on-a-JMicron-USB-bridge (/dev/sdf) out of smartd. But the boot still stalled, and the kernel log showed the reason: 6 UAS aborts per boot on sdf, each an ATA command pass-through (a1 80 / a1 82) — a SMART read — hanging 30–60s before the USB layer gave up. Something else was still SMART-probing the bridge.

That something was netdata. Its go.d smartctl collector runs smartctl (as root, via the ndsudo setuid helper) against every disk smartctl --scan finds — including the bridge — and the JMicron can't service those commands.

Action: added a device_selector: '!/dev/sdf* *' to /etc/netdata/go.d/smartctl.conf so the collector skips the bridge, mirroring the smartd exclusion. After restart, netdata shows live SMART charts for all 7 internal disks (sda–sde, sdg, sdh) but none for sdf, and zero new UAS aborts. The selector is name-based, so it needs a re-check after any drive swap that reshuffles /dev/sd*. Prior config saved as smartctl.conf.bak-20260602.

2. The plex1 backup was blocked by immich's root-owned library

The nightly backup-plex1.sh runs as the plex user, but immich_server runs as root and writes its photo library to /media/plex1/Photos/immich/data with root-only permissions. So rsync hit opendir … failed: Permission denied (13), exited 23, skipped the --delete pass, and fired a failure alert every night. Everything else on plex1 backed up fine — only immich's library was unreadable.

Action: the rsync now runs as root via a fixed-argument wrapper, gated by a narrow NOPASSWD sudoers grant — the same least-privilege pattern as the read-only diagnostics drop-ins:

• /usr/local/sbin/backup-plex1-rsync.sh — root-owned, not writable by plex, takes no arguments; it just execs the one fixed rsync -a --delete /media/plex1/ /media/plex1_backup/.
• /etc/sudoers.d/plex-backup-plex1 — plex ALL=(root) NOPASSWD: /usr/local/sbin/backup-plex1-rsync.sh (validated with visudo -c before install). Because it's NOPASSWD, the 04:30 cron works with no password and no warm lease; because the wrapper is unparameterised, the grant can't be repurposed to rsync arbitrary paths as root.
• backup-plex1.sh still runs as plex and keeps the flock guard, the 6h timeout, and the Telegram-on-failure logic; only the rsync line changed to call the wrapper via sudo -n.

Verified root reads the immich subtree with no permission errors, and kicked off a full run through the new path. Destination copies now preserve source ownership (root for immich, plex for everything else).

3. DDNS: two updaters, one of them a permanent no-op

A public-IP change (97.113.230.103 → 97.113.239.113) got a "DNS updated" alert, but the server wasn't reliably findable. Pulling the live Dynadot record set showed the domain is cleanly structured — an apex A record plus a * wildcard CNAME, so updating the single apex record is enough and every subdomain (plex, www, …) follows it. The "not findable" was just resolver/TTL propagation lag (TTL 600; different public resolvers were mid-propagation). But the deeper finding was that two DDNS mechanisms were running and only one worked:

• ddns-go (docker) works correctly — its log shows Updated domain skyhouse.dev successfully, and Dynadot's authoritative apex was indeed the new IP. It does an idempotent "set apex = current IP."
• The custom ip-watch.sh + dynadot-update.py cannot, by design. Its strategy was "find an A record equal to the old IP and swap it." ddns-go wins the race and sets the apex to the new IP first, so when the custom script runs there's no record matching the old IP → it no-ops, reports success, and advances its state. It had effectively never done anything. (Even alone it's fragile: any drift in its idea of the "old" IP makes it silently no-op — the opposite of how a DDNS client should behave.)

Action: consolidated on ddns-go as the sole DNS updater and demoted ip-watch.sh to alerts-only — it still detects an IP change and sends the Telegram heads-up, but no longer calls Dynadot (no more racing no-op API calls or misleading "✅ updated" messages). dynadot-update.py is left in place but unused (handy as a manual idempotent fixer if ever needed). ddns-go manages skyhouse.dev with a 600s TTL; the wildcard CNAME means that's all it needs to touch.

Net effect: reboots no longer stall on the dead USB bridge, the plex1 backup (immich included) completes and the nightly false-failure alerts stop, and DNS now has exactly one updater that actually works.