SKYHOUSE.dev Journal

Why: Downloading anything in a browser had become unusable — 2 to 5 minutes just to get the "where do you want to save this?" dialog to appear, more multi-minute stalls while navigating it, and downloads that silently refused to complete after all that.

The symptom pointed at the browser, but the browser was innocent. The whole machine was freezing on I/O roughly 39% of every wall-clock second, and the reason was that the new DAS — the one bought specifically to fix our storage problems — had been quietly negotiating a USB 2.0 link since the day it was plugged in. We found it, fixed it, and got a 6× throughput improvement plus a bay we didn't know was missing.

1. From "slow downloads" to "the kernel is wedged"

The first useful number was pressure-stall information, which is much more diagnostic than load average:

• /proc/pressure/io showed full avg300=35.80. The full line means every runnable task on the box was blocked on I/O — not "a disk is busy," but "nothing can run." A third of the time.
• The usb-storage kernel thread (PID 883) was sitting in D state (uninterruptible), along with two flush-8:64 kworkers. Major:minor 8:64 is sde, which was /media/plex3.
• iostat showed sde pinned at 99% utilization while moving only ~33 MB/s. A 12 TB helium drive should do six times that.

That last pair is the tell: 99% busy at terrible throughput means the bottleneck is the transport, not the platter.

2. Root cause — a USB 2.0 link, not a USB 2.0 cable

The DAS bridge reported /sys/bus/usb/devices/1-2/speed = 480. Four drives (later five) were sharing a single 480 Mbit/s pipe. Two compounding factors made it far worse than the raw bandwidth suggests:

• max_sectors_kb was 120 on the DAS drives versus 1024 on the two easystores — every I/O chopped into 120 KB pieces.
• queue_depth = 1. Plain usb-storage (BOT) allows exactly one command in flight, so each 120 KB chunk costs a full USB round-trip.

We assumed a bad cable. It was the wall socket. Damien swapped the DAS and one easystore between sockets, each keeping its own cable, which turned out to be a clean controlled experiment:

• DAS moved to the easystore's socket → 5000 Mbit, hdparm reads 200.94 MB/s (was ~33).
• easystore moved to the DAS's old socket → dropped to 480 Mbit, 42.4 MB/s.
• Identical easystore model on a good socket, for reference: 202 MB/s.

Same drive, same cable, 42 vs 202 MB/s depending only on which hole it sits in. That rear socket carries no SuperSpeed pairs at all.

3. Why this is easy to miss: one socket, two logical ports

Worth writing down because it cost us time. On xHCI, a physical socket wired for USB 3 enumerates as two ports — a USB 2.0 half on Bus 001 and a SuperSpeed half on Bus 002. A device that shows up on Bus 001 is therefore not necessarily in the wrong socket; it may be in a good socket whose SuperSpeed link failed to train. The way to tell them apart is whether the socket has a Bus 002 twin at all. Ours didn't — it is a USB 2.0-only socket by design, not a broken USB 3 one.

The controller exposes 7 SuperSpeed root ports, of which only 2 are in use. There is no shortage of fast ports; we were simply using a slow one.

4. Fixing the UAS guard before touching the cable

Moving to SuperSpeed was not safe to do blind. The 2026-07-06 outage was caused by UAS flooding, and the guard we added then was usb-storage.quirks=152d:0583:u on the kernel cmdline. Checking the live hardware, the bridge in this enclosure enumerates as 152d:0567 (JMS567) — so that quirk had never matched anything. It appeared to be working only because USB 2.0 never offers UAS in the first place. Going to SuperSpeed would have re-armed the exact failure mode.

• Action: backed up /etc/default/grub to /etc/default/grub.bak.20260730, then extended the list to usb-storage.quirks=152d:0583:u,152d:0567:u (keeping the old ID in case that enclosure ever returns), ran update-grub, and verified the string landed in /boot/grub/grub.cfg.
• Action: before rebooting, stopped Nautilus and ran sync to drain 571 MB of dirty pages — over a 480 Mbit link an un-drained writeback queue turns a reboot into a multi-minute unmount hang.

After the reboot the DAS came up on the SuperSpeed port bound to usb-storage, not uas. The quirk holds.

5. What was actually consuming the bandwidth

The link was the ceiling, but something had to hit it. Per-process I/O sampling found Nautilus reading ~17 MB/s and writing ~17 MB/s continuously — generating thumbnails for a Google Takeout dump at /media/plex3/Downloads/takeout-20260720.../Google Photos/. That one background job consumed the entire USB 2.0 budget, and because all DAS bays share a single usb-storage thread, it stalled /media/plex1 too.

Killing it alone dropped full avg10 from 38.88% to 1.52% before any hardware change — which confirmed the diagnosis on the spot.

The browser connection: Chrome's default_directory was /media/plex3/Downloads — the saturated drive — and the GTK file chooser stat()s the download directory plus every bookmark before it can draw. Three bookmarks lived behind the blocked thread. Hence a five-minute dialog.

6. A fifth bay we didn't know was missing

At SuperSpeed the bridge enumerated five drives instead of four. The newcomer is a Seagate ST10000DM005-3AW101 (serial WP001CYM), which mounted itself at /media/plex3_backup — it had been sitting in /etc/fstab with nofail the entire time, failing silently. At 480 Mbit the JMS567 could not enumerate all five bays reliably. We had assumed that drive was outside the DAS; it never was.

This also means the #ABSENT-20260706 comment on the plex3_backup line in /etc/smartd.conf is now wrong and should be re-enabled (noted below as outstanding).

7. Results

• DAS throughput: ~33 MB/s → 201 MB/s (6×).
• System I/O pressure full avg10: 38.88% → 0.08%.
• max_sectors_kb on all DAS drives: 120 → 1024.
• D-state processes: 3 → 0.
• UAS still disabled — drives bound to usb-storage.
• Recovered /media/plex3_backup (10 TB) that had been silently absent since at least 2026-07-06.

This reboot also closes the validation item left open on 2026-07-06: the box was rebooted with the DAS attached and came up clean, with no POST hang and no USB crash flood. The "post-boot hotplug only" restriction on the DAS can be retired.

8. New tool: das-check.sh

Added /home/plex/das-check.sh (companion to the existing das-up.sh / das-down.sh). One command prints the active quirk string, every USB mass-storage device's negotiated link speed with a loud flag on anything that fell back to 480 Mbit, each disk's driver and max_sectors_kb, current I/O pressure, and any blocked processes. It also shouts if anything binds uas. This is the first thing to run after any DAS re-cabling or reboot.

The bcdUSB discriminator — the single most useful trick from this session. When every socket is unlabelled and identical, /sys/bus/usb/devices/<port>/version tells you what a device is capable of, independent of what it currently negotiated:

• ver=2.10 (or 3.x) — the device carries a BOS descriptor advertising SuperSpeed. It can do USB 3.
• ver=2.00 or 1.10 — genuinely USB 2.0 / low-speed silicon. Nothing to fix.
• ver=2.10 + speed=480 is the money signature — capable but not trained, meaning a USB 2.0-only socket or a USB 2.0 upstream cable.

This is what separates "this device is slow" from "this device is being made slow", and it works without unplugging anything. das-check.sh now prints the version column and classifies every device on this basis. Corollary worth remembering: a USB 2.0 device can never map your sockets for you — with no SuperSpeed pins it enumerates on Bus 001 in a fast socket and a slow one alike, so probing requires a device that reports 2.10 or better and a known-good USB 3 cable. Validate the probe in a known-fast socket before trusting its verdict elsewhere.

Note: devices at 1-1.x legitimately show 480 Mbit — they sit downstream of a hub that is itself linked at 480, so they are capped regardless of their own capability. das-check.sh labels these separately rather than flagging them as faults.

9. Outstanding

• plex1_backup is still stuck at 480 Mbit / 42 MB/s — and fast sockets turn out to be scarce. The controller reports 7 SuperSpeed root ports, but only 2 are wired to reachable rear sockets (both now occupied by the DAS and the plex2 easystore); the rest are unexposed or unconnected front-panel headers. Every rear socket is black — this OEM board does no colour coding — and moving the easystore to the one other free socket left it at 480. The trap: a USB 2.0 device cannot map the sockets for you. A USB 2.0 hub has no SuperSpeed pins, so it enumerates on Bus 001 whether it sits in a fast socket or a slow one — meaning one of the two USB 2.0 hubs currently plugged in (TerraMaster 4-port on 1-1, Huasheng on 1-7) may be squatting in a SuperSpeed socket invisibly. Added /home/plex/usb-portmap.sh to map them physically: run it, then move one known USB 3 device socket to socket and watch for the SUPERSPEED verdict. Escalation if no third fast socket exists: check whether the case's front-panel USB 3 header is actually connected to the board; then a powered USB 3.x hub (5 Gbps ≈ 400–500 MB/s shared, ample for 2–3 HDDs — and exactly what the 2026-06-03 consolidation plan already specified); then shucking the easystores into DAS bays. Low urgency: plex1_backup is an rsync target hit once nightly at 03:00 and is not behind the DAS bridge, so unlike the original fault it cannot stall the system — it is merely slow. Do move it to a fast socket before any full re-seed, though (14 TB at 42 MB/s is ~90 hours).
• Re-enable the plex3_backup line in /etc/smartd.conf (drop the #ABSENT-20260706 comment) now that the drive is present and mounted — use -d removable to stay detach-safe.
• The 10-port powered hub (1-1) is USB 3-capable but also stuck at 480. Discovered via the bcdUSB trick below — it reports ver=2.10 and its internal chips are Realtek RTS5411 (a genuine USB 3.0 hub controller), so like the DAS it is capable-but-untrained: slow socket, or a USB 2.0 upstream cable. Its product string reads "4-Port USB 2.0 Hub", which is a red herring — that is just what a USB 3 hub's USB 2.0 companion half calls itself. If it can be trained, the best layout on a 2-fast-socket board is DAS + this hub, with both easystores hanging off the hub (5 Gbps ≈ 400–500 MB/s shared — a 10× improvement over the 480 Mbit plex1_backup is on now, and no shucking required). The separate 4-port hub on 1-8 reports ver=2.00 — genuinely USB 2.0 silicon, correctly used for keyboard / Zigbee stick / receiver, no action needed.
• Label the USB 2.0-only rear sockets physically so nothing fast lands in them again — every socket on this board is black, with no colour coding.
• Check whether the case's front-panel USB 3 header is actually cabled to the motherboard. Five SuperSpeed root ports are unaccounted for, and an unconnected front-panel block is the usual explanation — two free fast ports if so.

The DAS was never the problem, and neither was its cable — it spent its whole life so far breathing through a USB 2.0 straw, and we found a lost 10 TB drive by widening it.