SKYHOUSE.dev Journal

Maintaining the Cloud Fortress

August 18, 2026

Wi-Fi Rate Collapse, and Finally Instrumenting the Router

Why: Every Wi-Fi speaker in the house started glitching at once, a new iPhone couldn't hold a connection one room from the router, and sitting right next to a multi-gigabit router it pulled 40 Mbps. Damien wanted diagnostics before rebooting, on the reasonable theory that a reboot would fix it and teach us nothing.

It would have. The reboot would also have thrown away the evidence, because the fault only exists after the router has been up a long time. What we found was that a chunk of the 5 GHz clients were transmitting at 6–15 Mbps with an excellent signal — a rate-control collapse, not a coverage problem. Along the way we finally gave ourselves real access to the router, which we should have done years ago.

1. Ruling out the boring explanations

The server's wired path was spotless: 0.34 ms to the router, 3.7 ms to the internet, 0% loss, 2.5 GbE link, zero errors or drops across 533 M packets. Multicast was 7 pkt/s out of 139 pkt/s total, so no mDNS/Cast broadcast storm. ISP, modem, cabling and server were all clean.

One measurement deserves a caveat we got wrong at first: sampling enp2s0 counters showed an idle uplink, and we briefly read that as "nothing on the network is saturating the link." It isn't. On a switched LAN the server's NIC only sees its own traffic plus broadcast/multicast — unicast between the router and other clients never reaches its port. That measurement rules out the server as a hog and nothing else.

We also wanted an RF scan and couldn't get one. The server's Intel AC3168 (wlp3s0) is hard-blocked — /sys/class/rfkill/rfkill0/hard = 1, a BIOS-level disable no software can clear. Bluetooth on the same combo card is unblocked, so the card is alive. Enabling it needs a reboot into BIOS, and it's only a 1×1 AC card anyway: useful as a channel scanner, useless as a throughput reference.

2. Using ping as an air-quality meter

With no RF access, we measured the air indirectly: the last hop to a wireless client goes over the radio, so per-device RTT from a wired host reports the state of the medium. Pinging all 29 LAN devices split them cleanly. Three answered in under a millisecond (router, Hue bridge, a Lutron device) — those are wired. Everything else ranged from 5 ms to 3189 ms, with three devices at 100% loss. A local ping across the house should be 2–5 ms.

3. A correlation that was real, and an explanation that wasn't

Damien had just set up a 5-speaker Google Home group. Stopping it dropped the worst device from 1441 ms to 4.9 ms, another from 784 ms to 18 ms, another from 555 ms to 6.1 ms — with the wired router control reading 0 ms in every sample, so it wasn't measurement drift.

We initially explained this as 2.4 GHz airtime saturation and recommended splitting SSIDs, moving speakers to 5 GHz, and narrowing the 2.4 GHz channel. All of that was already configured — Damien runs separate SSIDs per band, the 5 GHz-capable speakers were already on 5 GHz, 2.4 GHz was already at 20 MHz, and legacy_rates was already disabled. The advice was confidently wrong, and it stayed wrong until we got actual data off the router. Worth remembering: a plausible mechanism plus a real correlation is still a guess.

4. Getting into the router

Browser automation against the admin UI failed (the extension lost its connection, and the UI is a JS SPA that tells curl nothing). SSH was the better path anyway. I generated a dedicated keypair on the server and Damien installed the public half himself, so his admin password never passed through the session:

The router is a GL.iNet BE9300 (Flint 3) — Qualcomm IPQ5332, tri-band Wi-Fi 7, firmware 4.9.0. Both the context dump and the cheat sheet had it recorded as v4.8.4; both are now corrected.

5. The driver fights you

The Wi-Fi stack is Qualcomm's proprietary qcawificfg80211, not vanilla mac80211, and the standard tools quietly lie:

6. The actual fault

The airtime survey killed the congestion theory outright: 2.4 GHz was 47% busy, 5 GHz was 2% busy and 6 GHz 1%. The bands the speakers live on were essentially idle — yet wlan1 logged 25 disassociations in 90 minutes.

The station table explained it. Comparing TX rate against signal strength:

At −49 dBm on VHT80 a client should negotiate 400–900 Mbps. Several were pinned around 6 Mbps — roughly a hundredfold collapse, concentrated in the Google/Nest units and the Sunitec speaker modules.

That reconciles the contradiction. Idle, those clients only beacon, so the survey reads a misleading 2% busy. Put them under real load — five speakers streaming synced audio — and the same stuck clients must move continuous traffic at 6 Mbps, burning about 100× the airtime per byte, and the band falls over. The speaker group didn't cause the fault; it was the first workload heavy enough to expose it.

Contributing factors worth noting: 20 of 25 wireless clients sit on the single 5 GHz radio while 6 GHz carries one client at 1.3 Gbps (band steering is disabled, so nothing rebalances); the router had 29 days uptime with the stuck stations associated for ~713 hours straight; 5 GHz runs EHT160 with a mixed legacy client fleet; and all five thermal zones idle at 71 °C against a 75 °C fan trigger.

7. So this is cheaper next time

Rediscovering all of the above from scratch is the real cost, so it is now written down:

8. A measurement trap worth writing down

Before rebooting we nearly published a much bigger claim than the evidence supported. Two snapshots eleven minutes apart showed most of the "collapsed" stations recovering on their own, with no intervention — because wlanconfig ... list sta reports the last-used TX rate, not a capability ceiling. An idle speaker shows 6 Mbps simply because its last frame was a small keepalive.

The fix is to force traffic and include a known-good control station, then read the table immediately. Done that way the picture held up, but it shrank: four devices stayed pinned at 7–8 Mbps under sustained load while the control climbed from 390 to 780 Mbps. Real, but four devices — not the dozen the idle snapshot implied. Never diagnose rate collapse from an idle reading.

9. The reboot — and a prediction that was wrong

Damien approved a reboot on one condition: that it not trigger the public-IP change he sees whenever the modem loses power. We checked the mechanics — WAN MAC statically pinned at 94:83:c4:c5:26:dc, no cloning or randomisation, so udhcpc would re-request the same lease from the same MAC — and concluded a router-only reboot should keep the address, since the modem stays registered upstream.

That was wrong. The IP changed anyway: 97.113.247.79 → 97.126.151.251, with the MAC unchanged. This ISP simply does not honour lease continuity across a DHCP re-request. The reasoning was sound in general and untested here, and it should have been flagged as an assumption rather than a reassurance. Rebooting this router changes the public IP. Plan for it.

The safety net worked exactly as designed, which is the happier half of the story. gl_ddns updated cw526dc.glddns.com within about two minutes, and because all three apexes have been flattened CNAME → cw526dc.glddns.com since 2026-08-04, skyhouse.dev, botaa.org and room101.com all followed with no manual edits — verified HTTP 200 with valid TLS from both 1.1.1.1 and 8.8.8.8, Plex's 32400 reachable, room101's Cloudflare MX intact. The same event before the flattening is precisely what left room101 serving a dead IP for 16 days.

10. Result

Measured under identical forced-traffic load, before and after:

Whole-LAN latency: the worst device fell from 1441 ms avg / 3189 ms max to 13 ms / 36 ms, the three devices at 100% loss all came back, and there is no packet loss anywhere. Honest caveat: it did not return to the 2–5 ms measured during a quiet pre-reboot window — it settled around 12–41 ms avg with twenty clients freshly re-associated. That wants re-measuring once things settle.

11. Still open

One device (ac:67:84:89:9e:fb) is still at 6 Mbps after the reboot — device-specific, worth isolating on its own. The bigger question is whether the rates decay again over the coming days; a daily snapshot would turn that into a TX-rate-versus-uptime curve and, if it recurs, a GL.iNet bug report with a reproducible timeline. Untested beyond that: dropping 5 GHz from EHT160 to 80 MHz, moving some of the 20 clients onto the idle 6 GHz radio, and watching thermals (71 °C idle against a 75 °C fan trigger) under sustained load.

The reboot fixed most of it — but running it blind would have destroyed the evidence, left the measurement trap undiscovered, and taught us nothing about why it will happen again.

← Back to Admin Hub