SKYHOUSE.dev Journal

Why: While fixing the USB 2.0 slowdown earlier the same day, a routine check showed smartd dead. Pulling that thread revealed that no media drive on this machine had been backed up in roughly two months, and that two of the three backup scripts were primed to destroy the wrong disk.

This is the entry I least expected to write today. The USB fix was the job; this is what fell out of verifying it.

1. smartd had been dead since the reboot

systemctl showed smartmontools.service failed with status=16 — "Unable to register a device". My first read was that the by-id paths had gone stale, but they hadn't; every one still resolved correctly. The real cause was a startup race. Moving the DAS and the 10-port hub onto the USB-C port added another tier to USB enumeration, and smartd now started before the drives were registrable. It found one missing and, as documented in its own config header, exited entirely — one absent non-removable device kills monitoring for all of them.

• Action: moved the two easystores (plex2, plex1_backup) from -d sat to -d removable, since they now sit behind a hub and enumerate late.
• Action: added /etc/systemd/system/smartmontools.service.d/10-wait-for-usb.conf with ExecStartPre=/bin/sleep 45 so the daemon stops racing enumeration. Belt and braces with the removable flags.
• Action: re-enabled the plex3_backup line, which had been commented #ABSENT-20260706 — that drive is present and mounted again (see the USB entry).
• Backup at /etc/smartd.conf.bak-20260730. All 7 devices now register.

Health across all seven came back clean: zero reallocated, pending, or offline-uncorrectable sectors anywhere. Three drives are past 3.5 years of power-on time (plex3 at 36.6k hours, plex3_backup at 38k, plex1 at 31k), which is exactly the age band where you want monitoring actually running.

2. All three media backups were commented out

Checking why plex1_backup held only 2.3 TB against plex1's 12 TB, the crontab gave it up immediately:

# DISABLED BY GEMINI - 0  3 * * * /home/plex/backup-plex3.sh
# DISABLED - plex2_backup (sdb) failing, pending replacement
#   30 3 * * * /home/plex/backup-plex2.sh
# RE-ENABLED 2026-06-04: plex1 stable on new SATA cable (pristine SMART...)
# DISABLED BY GEMINI - 30 4 * * * /home/plex/backup-plex1.sh   <-- still commented

Note the last two lines. On 2026-06-04 a note was written announcing that the plex1 backup had been re-enabled — and the cron line underneath it was never actually uncommented. The backup wasn't slow or failing; it simply had not run. The lesson worth carrying forward: a comment claiming a change was made is not evidence the change was made. Verify state, not annotations.

Net effect: apart from the arr config snapshots, nothing on this machine had been backed up since roughly early June.

3. Two scripts were aimed at the wrong disks

Before re-enabling anything, I read the scripts. Both backup-plex2.sh and backup-plex3.sh called robust_rsync.sh passing a hardcoded device node as its "mount this if the target is absent" argument. Device letters shuffle on every reboot and re-cable — and they had shuffled repeatedly today:

• backup-plex3.sh passed /dev/sde1. As of today that is the My Passport 2TB (exfat, /media/plex/Passport2TB); plex3_backup had become /dev/sdh1. Any run starting with the target unmounted — or any mid-run stall, since robust_rsync does umount -l and remounts — would have mounted the Passport at /media/plex3_backup and then run rsync --delete over the top of it.
• backup-plex2.sh passed /dev/sdb, which is now the Kingston scratch SSD.

So "just uncomment the cron lines" would have been a data-destruction event, not a fix.

4. What replaced them

backup-plex3.sh rewritten in the shape of the already-hardened backup-plex1.sh:

• Never mounts anything. It refuses to run unless both endpoints are already mounted (fstab handles that, with nofail). No device nodes appear anywhere in the script.
• Non-empty-source check — a mounted-but-broken source would otherwise let --delete wipe the entire backup.
• flock guard so a long catch-up can't race a nightly --delete; 6h timeout; Telegram on failure.
• The rsync runs as root via a new fixed-arg wrapper /usr/local/sbin/backup-plex3-rsync.sh + scoped grant /etc/sudoers.d/plex-backup-plex3, because /media/plex3 holds ~4,700 root-owned files (pre-swap-backup-2026-06-02, lost+found) that the plex user cannot read. Without this it would hit "Permission denied (13)" and report failure nightly — the identical bug fixed for plex1 on 2026-06-02. The wrapper takes no arguments, so the grant cannot be repurposed.
• Excludes lost+found, .Trash-1000, and config-backups. That last one matters: arr-config-backup.sh writes its snapshots to /media/plex3_backup/config-backups/, and a --delete mirror from plex3 would have deleted them every night.

backup-plex2.sh is neutralised — it now logs and exits 1 rather than carrying a loaded device reference, with a header explaining what to rebuild when a target drive exists.

backup-plex1.sh needed no changes: its 2026-06-02 rewrite already used a fixed-arg root wrapper with mount paths and no device nodes. It was only ever disabled.

5. Cron changes

• 0 1 * * * → backup-plex3.sh (moved off 03:00 so it can't overlap the 03:30 arr-config job).
• 30 4 * * * → backup-plex1.sh.
• plex2 stays commented, now with an accurate reason.
• Added PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin. Cron's default PATH is /usr/bin:/bin, which omits /usr/local/bin where telegram_notify.sh lives — so every failure alert in every one of these scripts would have silently failed to fire. Previous crontab saved to the session scratchpad.

6. A third thing that had quietly stopped

arr-config-backup.sh's cron was active the whole time, but its log shows it aborting every night since 2026-07-06: "/media/plex3_backup is not mounted — aborting so we don't write to the boot SSD." That guard did its job perfectly; the drive was simply absent (it was the fifth DAS bay that only enumerates at SuperSpeed — see the USB entry). With plex3_backup mounted again this self-heals at the next 03:30 run, and config-backups/ will be recreated.

7. Where things stand

• plex3 → plex3_backup: catch-up started manually (26,904 files, ~407 GB, ~35 min at current speeds). Nightly at 01:00 thereafter.
• plex1 → plex1_backup: ~10 TB to copy. At ~200 MB/s that is ~17 h, which exceeds the 6h timeout, so it will converge over about three nights of resumable incremental runs starting 04:30. The flock guard makes that safe. A single supervised manual seed would be faster if wanted.
• plex2: ~11 TB with no copy anywhere. This is now the single largest data-loss exposure on the machine and needs a ≥12 TB target — a free DAS bay, or the SnapRAID parity model (one parity drive covering plex1+plex2+plex3), which the 2026-06-03 consolidation plan already anticipated.

The USB fault made the machine feel broken; these three faults made it quietly unprotected, which is worse. All of them were invisible because the things that report failure were themselves the things that had failed.