SKYHOUSE.dev Journal

Why: The boot swap + drive shuffle left the SMART monitoring keyed to /dev/sdX letters that no longer meant what they used to — and, it turned out, had silently crashed smartd entirely. We wanted all drive monitoring keyed to stable serials, the secondary swap off a spinning media disk and onto the scratch SSD, and the swapped-in plex3_backup drive brought online — ideally without touching hardware.

This started as "clean up the monitoring drift" and turned into a small incident: a hot-swap on the shared USB toaster had thrown transient ext4 errors on plex3, and smartd had been dead since the morning boot. We fixed all of it live, by serial, with no reboot and no cable-pulling.

1. smartd was dead — and rebuilt by-id

Discovery: smartmontools.service had been failed since the 20:49 boot — it exits (status 16) if any listed device is absent, and the 2026-06-03 boot swap had left a stale Corsair line for the removed old boot SSD. So SMART monitoring was off entirely. Rebuilt /etc/smartd.conf:

• Removed the retired drives' lines (old Corsair boot, dead WD WUH721414/9RHHSV3L).
• Added the new boot SSD (SSV8) and the scratch Kingston, all by /dev/disk/by-id/ata-* serial paths with -d sat.
• The detachable toaster drive plex3_backup (ST10000DM005/WP001CYM) uses -d removable so smartd won't exit when it's unplugged — and when present, removable still auto-detects it as [SAT] and monitors full ATA SMART. Best of both worlds for a drive that gets swapped.
• Result: all 7 present drives monitored by serial; smartd active again.

2. netdata: monitor-all instead of a name-based exclusion

netdata's go.d smartctl collector had device_selector: '!/dev/sdf* *' — added to dodge the JMicron USB-NVMe bridge that used to hang. After the drive shuffle, /dev/sdf had become plex2, so the filter was wrongly hiding a real media drive. Since the JMicron enclosure is now empty (the Kingston moved to internal SATA), there is nothing left to exclude. Changed /etc/netdata/go.d/smartctl.conf to device_selector: '* *' (monitor everything). netdata only scans present devices, so this is inherently shuffle-proof. Confirmed it now charts all drives by serial.

3. Toaster recovered with a "virtual replug" (no hardware touched)

plex3_backup had been physically swapped into the Sabrent dual-SATA USB toaster, but the system still showed the old dead WD in that slot — USB-to-SATA bridges cache the drive identity and don't re-read it on a hot-swap. Rather than pull the USB cable (which would also bounce plex3, sharing the same bridge), we re-enumerated it in software:

• Confirmed the toaster is USB 1-1, alone on Bus 001 — the easystore drives (plex2, plex1_backup) are on Bus 002, so the reset couldn't touch them.
• Quiesced plex3 first (stopped Plex + music-assistant, swapoff, unmounted) so the bridge reset couldn't corrupt it.
• Virtual replug: echo 1-1 > /sys/bus/usb/drivers/usb/unbind then …/bind. The bridge re-probed both slots → the ST10000 plex3_backup appeared, the dead WD vanished.

4. plex3 ext4 errors were transient, not real corruption

The earlier hot-swap had thrown bad extra_isize ext4 errors on plex3 (Plex was reading it mid-bridge-reset). An e2fsck -f -y on the unmounted drive found the filesystem structurally clean — only two trivial extent-tree optimizations, and it did not find the bad inode. Conclusion: the "corruption" was garbage read off the wire during the USB reset, not on-disk damage. FS state is back to clean; plex3 and plex3_backup remounted.

5. Secondary swap moved to the scratch SSD

Moved the 32 GB secondary swap off plex3 (a spinning media disk) onto the scratch SSD: created /media/scratch/swapfile_extra, mkswap/swapon (priority −3, after the 4 GB primary), fstab entry sw,nofail so a dead scratch disk can't break boot, and deleted the old /media/plex3/swapfile_extra (freed 32 GB on plex3). Faster swap, off the media drive.

6. fstab housekeeping

Commented out the stale /media/plex2_backup entry (it pointed at the dead WD's UUID 134aed68) so that retired drive can never auto-remount there. findmnt --verify is clean (only the benign "swapfile is a regular file" notices).

Net effect: all drive monitoring is now keyed to serials and survives letter shuffles, smartd is alive again after being silently dead all day, the secondary swap rides the scratch SSD, and the toaster drive was recovered and its filesystem cleared — all live, by-id, with zero hardware intervention.