SKYHOUSE.dev Journal

Why: Every reboot sat for ~30 seconds on a screen full of red [FAILED] lines — a smartmontools start timeout plus three media filesystems that refused to unmount. It looked alarming and wasted time on every boot, and the smartd failure meant SMART monitoring was silently off for every disk.

The reboot screen was actually showing two unrelated problems that happen to land on the same console — one on the boot side, one on the shutdown side. I tracked down both and fixed them in one change-set.

1. smartmontools timed out on the JMicron USB bridge

/etc/smartd.conf used a blanket DEVICESCAN, so smartd probed every block device — including /dev/sdf, the Kingston SSD ("128GB SSD2") sitting behind a JMicron USB-NVMe bridge (usb-JMicron_Tech_DD564198838A1). The boot log told the whole story:

• smartd: Device: /dev/sdf [USB NVMe JMicron], opened … then a 90-second gap … then smartmontools.service: start operation timed out. Terminating.

JMicron USB bridges are notorious for hanging on SMART pass-through commands. Because the unit is Type=notify, smartd never sent its "ready" signal, so systemd killed it after 90s and marked it [FAILED] — and SMART monitoring was effectively disabled for all disks, not just the USB one.

Action: I replaced DEVICESCAN with explicit per-disk lines for the 7 internal SATA disks, named by stable by-id (model_serial) so the config survives /dev/sd* reordering. A DEFAULT -a -n standby -m root -M exec … line carries the shared options; each disk line is just the by-id path with -d sat. The USB-bridged drive is simply never listed. I validated the new file out-of-band first with smartd -c /tmp/smartd.conf.new -d -q onecheck — all 7 disks opened instantly and reported SMART-capable — before installing it. After systemctl restart smartmontools the service came up active (running) with no delay. Original saved to /etc/smartd.conf.bak-20260601.

2. Three filesystems were busy at shutdown

/media/plex1, /media/plex2 and /media/plex1_backup were busy when systemd tried to unmount them, so it waited out the stop timeout (~30s) and then printed [FAILED]. systemd simply didn't know which services depended on which disks. The holders break down as:

• /media/plex1 & /media/plex2 — docker containers (radarr, rdtclient-movies, immich_server on plex1; sonarr, rdtclient on plex2) bind-mount these, and Plex keeps its TMPDIR on plex2 plus libraries on both.
• /media/plex1_backup — held only by the nightly backup-plex1.sh rsync (04:30 cron). No persistent service owns it; it only hangs if a reboot lands mid-rsync.

(/media/plex3 never failed because nothing heavy holds it — confirmed in a clean prior shutdown log.)

Action — the two service-held mounts: added RequiresMountsFor=/media/plex1 /media/plex2 drop-ins to docker.service and plexmediaserver.service (as separate 10-requiresmountsfor.conf files, leaving Plex's existing override.conf untouched). RequiresMountsFor creates a Requires= + After= on the mount units, so on shutdown those services are fully stopped before the disks unmount. Verified live: both services now show After=media-plex1.mount media-plex2.mount.

Action — the backup-only mount: since nothing persistent holds /media/plex1_backup, there's nothing to order against. Instead I bounded its unmount with a TimeoutSec=10 drop-in on media-plex1_backup.mount — a rare mid-backup reboot now fails fast (10s) instead of hanging the console for 30.

3. What this means for the upcoming drive swaps

With the plex2_backup drive (sdb) degraded and a boot-SSD replacement already on order, it's worth being explicit about which of these configs are swap-sensitive:

• /etc/smartd.conf IS serial-bound. Each line names a drive by model_serial. After replacing any drive you must update its line with the new by-id name (ls -l /dev/disk/by-id/ | grep ata-), or smartd silently stops monitoring the new disk and warns about the missing old one. A prominent reminder block lives at the top of the file.
• The systemd unmount drop-ins are path-based and swap-safe. RequiresMountsFor=/media/plex2 tracks the mount point, not the device. Swapping a drive — or the planned "promote plex2_backup's replacement to plex2, demote old plex2 to backup" shuffle — needs no change here, as long as the mount-point paths (/media/plex1, /media/plex2, …) stay the same. Only /etc/fstab (the UUID) needs editing, which is mandatory for the drive to mount at all.

Net effect: reboots are now quiet and fast — smartd starts instantly with SMART live on all 7 internal disks, and the media filesystems unmount cleanly because their consumers stop first.