SKYHOUSE.dev Journal

Maintaining the Cloud Fortress

Capping every media mount's unmount timeout so a wedged drive can't hang shutdown

Why: A reboot threw a scary console error — Dependency failed for media-plex3_backup.mount / Timed out waiting for device — and, more to the point, when a USB drive wedges it stalls shutdown for the full 90s default. Damien wants maintenance reboots to be as quick as possible, so we bounded the worst case.

The reported error turned out to be two separate things, only one of which actually costs time. Investigating both led to extending an existing fix (the 2026-06-01 plex1_backup unmount cap) across all the media mounts.

1. The boot error is cosmetic; the real cost is on shutdown

The console message was a boot-time mount failure: the USB-attached plex3_backup drive (a Seagate ST10000DM005 in the external toaster) didn't enumerate within systemd's 90s device timeout on the 2026-06-04 boot, so media-plex3_backup.mount reported Dependency failed. But because every media mount already carries nofail in /etc/fstab, systemd does not order the mount before local-fs.target and does not wait for it — boot proceeded normally (measured: local-fs.target @674ms, total userspace ~8s). The line is noise, not delay.

The genuine delay was on the shutdown side. The journal showed a real 90s stall on 2026-06-04:

When the cheap USB→SATA bridge wedges, systemd waits the full DefaultTimeoutStopSec (90s) trying to unmount cleanly before force-killing it. That's the reboot delay worth eliminating.

2. One timeout governs both directions on a mount unit

For .mount units there is no separate start/stop timeout — a single TimeoutUSec bounds both the mount and the unmount operation. So shortening it caps the shutdown unmount attempt. This is exactly the lever the 2026-06-01 work pulled for media-plex1_backup.mount (a TimeoutSec=10 drop-in, because its only holder is the nightly rsync — nothing to order against with RequiresMountsFor). That one mount was already at 10s; the other six were still at the 90s default.

Topology check (USB vs internal SATA, by sysfs path — ID_BUS reports ata even for the USB drives, so it can't be trusted here):

3. The change: a 10s unmount cap on every media mount

I added the same drop-in as plex1_backup to the five remaining media mounts. Each file is /etc/systemd/system/media-<name>.mount.d/10-timeout.conf containing a dated comment plus:

Applied to plex2, plex3, plex3_backup (the USB drives) and plex1, scratch (internal SATA, capped too for uniform worst-case bounding — the host's internal SATA links have also been flapping in the run-up to the DAS migration). Followed by systemctl daemon-reload, which updates the live TimeoutUSec without remounting — zero disruption to running drives or in-flight reads. Verified all six media mounts now report TimeoutUSec=10s and stayed active.

Net effect: no single wedged drive can stall a reboot for more than ~10s — systemd gives it 10s, then force-terminates and moves on. Like the original, these drop-ins are path-based, so a drive swap that keeps the mount point needs no change here.

4. This carries forward to the DAS

The drives are USB-bridged today. The incoming Mediasonic H82-SU3S3 8-bay enclosure supports both USB 3.0 and eSATA: over USB it's the same kernel mass-storage path, so the wedge-on-unmount behaviour can persist; eSATA would be deterministic but needs a host port/HBA with port-multiplier (FIS) support. Either way these systemd caps apply unchanged, so the worst-case shutdown stays bounded regardless of how the DAS ends up connected.

The trade-off: a 10s cap means a drive genuinely mid-write at shutdown gets force-unmounted → an unclean unmount → ext4 journal replay on next mount. For read-mostly media and cron-driven backup targets that's a non-issue — the same trade-off already accepted on plex1_backup.

← Back to Admin Hub