SKYHOUSE.dev Journal

Why: The root SSD was sitting at 95% full and Damien is about to start a chain of drive upgrades (new boot SSD, a DAS, replacement backup HDDs). Before spending money we wanted to (a) buy breathing room on root, (b) give Claude a few read-only hardware-inspection tools for the upcoming disk work, and (c) finally fix the plex1 nightly backup, which had been silently failing for weeks.

This was a planning session that turned into three concrete pieces of work. The headline: the plex1 backup failure was never a dying drive — it was an accidental NTFS filesystem plus a too-short timeout. The drive is healthy; the job design was broken.

1. Root disk cleanup (95% → 77%)

Root (/dev/sda3, 109 GB) was at 98 GB used / 95%. Almost all the reclaimable space was Docker churn from the steady stream of *arr / Immich / Portainer updates:

• Action: docker system prune -a -f — removed unused images + build cache. All 21 containers were running, so their images were protected; only genuinely-unused layers went. Freed ~17 GB on disk.
• Action: cleared ~/.cache (regenerable) — freed ~1.9 GB. journalctl was already lean (~200 MB); the rest of /var/log is nginx/backup logs worth keeping.

Net: 98 GB → 79 GB, 95% → 77%, ~25 GB free. This removes the urgency from the boot-SSD swap — it can now happen on Damien's timeline, not as an emergency. Confirmed afterward that all 21 containers were still healthy (Immich included).

2. Read-only hardware-diagnostics sudo drop-in

This session hit a gap: sudo du wasn't in the allowlist, so Claude couldn't see into root-owned dirs like /var/lib/docker. With a month of disk/hardware work ahead, we added a third sudoers drop-in for strictly read-only inspection.

• Action: created /etc/sudoers.d/claude-readonly-hardware granting NOPASSWD for du, smartctl -a/-x/-H/-i/--scan, dmidecode, blkid, and fdisk -l.
• Installed via visudo -cf validation then atomic install -m 0440, so a syntax error couldn't lock sudo.
• The governing rule held: NOPASSWD only covers commands that cannot mutate state or destroy data. Anything that acts on disks (parted, hdparm, mkfs, mount) was deliberately left out and still requires the warm sudo -v lease. This keeps the claude-readonly boundary meaningful. Companion to the 2026-05-22 sudo-lease setup.

3. The plex1 backup fix — NTFS was the culprit, not the drive

The nightly backup-plex1.sh had been failing for weeks. The assumption was a timeout or a dying drive. SMART told a different story: sdh (the plex1_backup target, a USB-connected WD Easystore external) is spotless — 0 reallocated, 0 pending, 0 uncorrectable sectors. The real causes were two:

• Accidental NTFS. The drive carried its original WD Easystore NTFS filesystem and was never reformatted for Linux. Under rsync -av, the -a flag tries to preserve Unix ownership/permissions, which NTFS (via ntfs-3g) can't store — so rsync saw a metadata mismatch on every file every run and re-copied endlessly, never converging. (NTFS also can't even store some media filenames containing : * ? etc.)
• 1-hour timeout. The script's timeout 3600 killed rsync mid-pass every night (the logs ended with rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20)), so an 11 TB initial baseline could never complete.

Fix, executed under a warmed sudo lease after triple-confirming /dev/sdh1 = plex1_backup:

• Action: umount + mkfs.ext4 -L plex1_backup -m 0 /dev/sdh1 (the -m 0 drops the 5% root-reserved blocks — pointless on a backup drive, reclaims ~700 GB usable).
• Action: updated /etc/fstab to the new ext4 UUID (70d95e9e-…), type ext4, dropped the stale x-gvfs-show flag to match the other ext4 backup entries. Backup of fstab at /etc/fstab.bak-2026-05-31.
• Ownership snag (caught live): a fresh ext4 root dir is owned root:root, so the first rsync (running as plex) failed to create a single directory. Rather than move the job to root cron, we matched the working plex3 pattern — chown the destination to plex:plex and run as plex. On a backup copy, the handful of UID-525/root-owned source files becoming plex-owned is harmless, and it needs zero infrastructure changes.
• Action: launched the initial 11 TB seed detached (setsid + flock), running ~195 MB/s over USB, ETA ~13–14h. Wrapped in flock /tmp/backup-plex1.lock so the 4:30 AM cron skips cleanly instead of colliding with the seed.
• Action: hardened backup-plex1.sh — added the same flock guard, raised timeout 1h→6h, and dropped -v so the nightly log stops accumulating full file listings. Steady-state incrementals on ext4 finish in minutes.

4. Two things worth remembering

• Docker app state all lives on root. Every container's config + database is on the OS SSD (/home/*, /var/lib/docker, /.config/prowlarr, Vaultwarden at /data/compose/10/vw-data); only bulk content (Photos, Movies, TV, Music, downloads) is bind-mounted out to the media drives. This is why a whole-disk clone is the clean boot-SSD migration — and an argument against splitting OS across two small disks.
• The root drive has no backup. The media drives have local mirrors, but the OS SSD — holding the Vaultwarden vault, Immich DB, and all app configs — is mirrored nowhere. If that ancient Corsair dies before the swap, it's all gone. A nightly tar of /home + Docker volumes to a media drive would close it. Logged as pending.

Net effect: root has 25 GB of headroom again, Claude can read SMART/firmware data for the upcoming drive work, and the plex1 backup is converging for the first time in weeks — on a healthy drive that just needed the right filesystem.