SKYHOUSE.dev Journal

Why: The server had been falling over for days — couldn't reboot, then hung opening Docker containers, then hung on Plex, then stopped responding entirely. Damien pulled the DAS to get a stable boot before a trip; on return we sat down to find the actual failing hardware and bring the box back to life in a durable way.

The symptom Damien saw — hangs at boot/shutdown with systemd-udevd/modprobe timeouts and a dead local keyboard/mouse — had a working hypothesis: a failing drive inside the USB DAS array was crashing the motherboard's USB controller at POST, and Linux then booted blind to the whole storage pool, cascading into fstab/Docker/Plex mount hangs. The firsthand log evidence refined that picture in one important way and let us fix it cleanly.

1. What the logs actually said

The box had been stable for 3+ days on the current boot (DAS unplugged since it left). The boot list told the story: a two-week-stable boot ended the evening of Jul 1, followed by eleven short boots in ~2 hours — the crash storm.

• The DAS is a JMicron JMS583 bridge running UAS (152d:0583, "JMicron External", scsi host12: uas). It enumerated at 16:25:26 Jul 1 and the hang cascade began at 16:29. UAS on JMS583 bridges is a notoriously flaky combo that floods/resets the USB bus under load — that is the "it polluted all my USB ports so the keyboard died" symptom.
• When storage stopped responding, filesystem unmounts hung forever (cgroup_writeback_umount → generic_shutdown_super, tasks stuck in D-state 122s+). That is what wedged reboot, then Docker, then Plex. The short boots were the kernel hanging during storage probe (modprobe/udev-worker in D-state).
• Correction to the hypothesis: the live offending drive was not in the DAS. With the DAS already unplugged, sde was still present and still timing out every command every 5 minutes ([sde] tag#0 timing out command, waited 60s). sde was a failing WD "My Passport" portable drive (1058:0748) on USB port 1-6 — directly attached, separate from the DAS and from the two healthy easystore desktop drives. Pulling DAS bays would never have fixed it; the My Passport had to come out.

So there were two independent USB-storage faults: the JMS583/UAS DAS (crash storm, already disconnected) and a dying My Passport (still wedging the running kernel).

2. Isolating the hardware

Damien physically pulled the WD My Passport; the kernel's hung sde host cleared instantly (no software echo 1 > .../delete needed). The Sabrent toaster drives (plex3 / plex3_backup) were already disconnected from when the DAS was added, so nothing else was left to remove. The DAS stays unplugged pending a deliberate test session.

3. Bringing back a partial stack without plex1

/media/plex1 lived on the now-disconnected DAS, so it no longer mounts. Both docker.service and plexmediaserver.service carry RequiresMountsFor=/media/plex1 /media/plex2 drop-ins, which left both services stuck inactive waiting on the missing mount. We chose a partial restore: run everything that only needs plex2, keep the plex1-dependent services down.

• Root-disk guard first. /media/plex1 exists as an empty directory on the root SSD. If a container bind-mounts it while unmounted, Docker silently writes to root — and rdtclient-movies would dump TorBox movie downloads there and re-fill/re-crash the box. Guarded with sudo chattr +i /media/plex1 so any such write fails loudly instead of leaking to root.
• Action: edited both 10-requiresmountsfor.conf drop-ins (docker + plex) to RequiresMountsFor=/media/plex2 only, with an inline RESTORE note. daemon-reload, then started both services. Docker and Plex came up; Plex answers HTTP 200 on :32400/identity and serves plex2 content (plex1/plex3 libraries show "unavailable", no crash).
• Action: on Docker start, the whole fleet auto-started via restart policy. Immediately set --restart=no and stopped the six plex1-dependent containers: radarr, rdtclient-movies, and the immich_* set (immich_server's UPLOAD_LOCATION=/media/plex1/Photos). The chattr +i guard proved itself — rdtclient-movies exited 255 unable to create its download dir on root, and root usage never moved (194 GB).

Running (plex2-safe): sonarr, rdtclient, prowlarr, flaresolverr, plus homeassistant, music-assistant, npm, portainer, vaultwarden, uptime-kuma and the rest. Held down (need plex1): radarr, rdtclient-movies, immich.

4. Reviving smartd

smartd (unit smartmontools.service) was dead — it exits with status 16 the moment a listed non-removable device is absent, and its config still named the DAS Seagate plex1 (ata-ST14000NM001G...ZL2BG3W8). So the box had zero disk-health monitoring.

• Action: commented out the three currently-absent drives in /etc/smartd.conf (Seagate plex1, and the two Sabrent-toaster drives WD120 plex3 / ST10000 plex3_backup), prefixing each with #ABSENT-20260706. Restarted smartd.
• smartd is now active and monitoring 4 devices: boot SSD (SSV8), scratch (Kingston), and both 14 TB easystores — plex2 (2CGXPDRN) and plex1_backup (2CJBWWZL). Confirmed the easystore USB bridges pass SMART through cleanly via -d sat (both drives read healthy, ~20°C).

5. Home Assistant's HubZ stick (cp210x)

homeassistant exited 255 on start: its /dev/serial/by-id/usb-Silicon_Labs_HubZ_... device paths were missing. Root cause: the cp210x driver never loaded (only bare usbserial was present), so no /dev/ttyUSB* nodes existed for the already-enumerated HubZ controller (10c4:8a2a) — udev almost certainly missed the modalias autoload during the USB-controller instability.

• Action: sudo modprobe cp210x → it bound the controller, created ttyUSB0/1 and the stable HubZ serial-by-id symlinks. Started homeassistant — up.
• Action: made it persistent with /etc/modules-load.d/cp210x.conf so it survives the next (cold) boot.

6. On the fstab "nofail" armor

All /media/plex1|2|3 (and backups) already carry nofail — that next-step was already done. Worth recording though: nofail did not prevent this outage. The failure wasn't a mount timeout; it was the USB controller crashing at POST plus the UAS storage layer hanging — both upstream of anything fstab/systemd can armor against. The real fix was removing the bad drive. A soft reboot does not clear a crashed USB controller; only a full cold boot (pull power, hold power button ~10s to drain caps) does.

7. DAS recovery — UAS quirk + reseating the drives (same session)

With hands on the machine we brought the DAS back the safe way. The root fix for the crash-flooding was to force the JMS583 bridge out of UAS and onto plain usb-storage:

• Action: applied the quirk live (no reboot) — echo 152d:0583:u | sudo tee /sys/module/usb_storage/parameters/quirks — and persisted it in /etc/modprobe.d/das-jms583-uas.conf (options usb-storage quirks=152d:0583:u). The u flag (US_FL_IGNORE_UAS) makes the uas driver reject the device so usb-storage claims it.
• Damien also reseated every drive before reconnecting — several had worked loose in their bays when the enclosure was moved the week before. Loose seating alone would explain a lot of the drop-outs.
• Result: plugged in the loaded DAS (no reboot) — local input stayed alive (no USB flood), all hosts bound usb-storage (not uas), and zero timeouts/resets/I-O errors. Four drives enumerated clean, all SMART PASSED: sde Seagate Exos 14TB = plex1 (auto-mounted, UUID matches), sdf WD 12TB = plex3 (auto-mounted), plus a blank Toshiba 6TB (sdg) and an old WD 2TB NTFS "DATA" (sdh) — both healthy, neither set up for anything, left alone. The expected 10TB plex3_backup was not in the enclosure. Conclusion: there was never a bad drive inside the DAS — the outage was UAS flooding + loose seating + the separate (now-removed) My Passport.
• Action: restored all six plex1 containers (radarr, rdtclient-movies, immich_*) — all up, 0 restarts, endpoints return HTTP 200. smartd re-armed for plex1 + plex3 using -d removable (not -d sat) so it monitors them when attached but survives a DAS detach instead of exiting 16. Now monitoring 6 devices.

8. Operating model — DAS is a post-boot hotplug (never boot with it attached)

Because a POST-stage USB crash is unrecoverable even over SSH (Linux never boots, so ssh -p 2222 can't save us), the DAS is treated as a deliberate post-boot attach, never present at cold boot. Docker and Plex are kept requiring only /media/plex2 (both drop-ins) so the box always boots clean without the DAS; the plex1 containers are kept restart=no so a DAS-less boot doesn't crash-loop them against the guarded empty mountpoint (/media/plex1 stays chattr +i — harmless while the real drive is mounted over it, protective when it isn't).

• Attach flow: boot clean → confirm up → hotplug DAS → run /home/plex/das-up.sh (verifies plex1 is the right disk by UUID, then starts the six containers).
• Detach flow: run /home/plex/das-down.sh first (stops the containers, unmounts plex1/plex3 cleanly) → then physically detach. Avoids dirty-ext4 / journal-replay on the next attach.
• SSH (2222) has no media-mount dependency and comes up regardless of the DAS — the remote lifeline for any Linux-stage (not POST) trouble.

9. Retiring the "no-boot" rule — hardening for always-attached

The post-boot-hotplug model in section 8 was a conservative default while the failure was still unexplained. But a NAS should survive a power-loss cold boot with no human in the loop, and a sub-month-old enclosure shouldn't need permanent workarounds — both fair points from Damien. Since the two most likely POST/early-boot USB-controller killers are now gone (the dead My Passport that hung enumeration, and UAS flooding), we hardened for the DAS staying attached across reboots:

• Action: put the UAS quirk on the kernel command line — GRUB_CMDLINE_LINUX="usb-storage.quirks=152d:0583:u" in /etc/default/grub (backup grub.bak-20260706), update-grub'd. This suppresses uas before any module loads, so it applies even during the earliest boot-time probe of the DAS (the modprobe.d file can be too late for a device present at boot).
• Action: created /etc/systemd/system/plex1-stack.service — BindsTo=media-plex1.mount, WantedBy=media-plex1.mount. When /media/plex1 mounts (boot or hotplug) it runs das-up.sh to start the containers; when it unmounts it stops them. So a cold boot with the DAS attached brings the whole stack up with no manual step, and a power outage self-heals. Containers stay restart=no (the service owns their lifecycle); nofail + the TimeoutSec=10 mount caps mean a dead DAS still never blocks boot or hangs shutdown.
• Still pending: one validation reboot with the DAS attached, done physically-present — the only thing software can't test is a true BIOS/POST-stage USB crash. If it boots clean, the "no-boot" rule is fully retired. If it still hangs at POST even with the Passport gone and UAS off, that isolates the fault to BIOS/firmware or a defective enclosure (within the return window) → BIOS USB settings (legacy-USB / XHCI hand-off) or an RMA.

Net effect: the box is fully back — Plex, the complete media stack, immich, smart-home, web and monitoring all live off a now-stable DAS in mass-storage mode; disk-health monitoring across all six present drives; and the DAS is being promoted from "hotplug-only workaround" to a normal always-attached device via an early-boot UAS quirk + mount-bound auto-start, gated on one physically-present validation reboot.