SKYHOUSE.dev Journal

Why: The RAM is confirmed faulty (see the RAM entry), but replacing DDR4 that has roughly doubled in price since it was bought is a last resort. Before committing to that, we needed to know one thing: is the fault localized to a few cells we can permanently bench, or is the whole array becoming fragile? That decides whether "blacklist the bad pages and keep running" is a real strategy or wishful thinking.

The test had to run inside Linux, because taking Plex offline for six hours on a weekend was not acceptable. That constrains what can be measured — more on that below — but it answers the localization question directly.

1. The test

• stressapptest, 46 GB of the 64 GB (the most an in-OS test can safely claim), 8 of 12 threads, six hours.
• The three known-bad pages were already runtime-offlined (soft_offline_page; HardwareCorrupted: 12 kB), so the allocator could not hand them to the test — important for reading the result.
• Temperature was sampled every 30 s for the whole run.
• Midway, by hand, a box fan was added to the open case — splitting the run into a hot phase and a cooled phase, an accidental but useful controlled comparison.

2. The result

669,247,296 MB moved at ~31 GB/s — about 638 TB of write/read/compare — 0 hardware incidents, 0 errors, Status: PASS. And it passed identically in both thermal regimes:

• Hot phase (no fan, 3.3 h): package temp averaged 89 °C, peaked 92 °C — hotter than either memtest run that produced errors. Zero errors.
• Cooled phase (fan, 2.7 h): averaged 79 °C, peaked 85 °C. Zero errors.
• The load included stressapptest's deliberate synchronized power-spike transients (all threads idle, then slam to full load together) to hunt voltage-droop failures. Nothing.

3. What it means — and what it doesn't

The defect is localized, not spreading. The rest of the array stayed perfect through 638 TB at 92 °C. If the sticks were broadly degrading under heat, that hot phase would have lit up; it didn't. That is the single most decision-relevant fact of the whole investigation, and it is what makes a blacklist-based bridge a genuine option rather than a hope.

But the result must be read with three honest limits, or it will be overtrusted:

1. The known-bad cells were never tested here — they were offlined. "PASS" means "the other 46 GB is clean," not "the RAM is healthy."
2. ~16 GB was unreachable. An in-OS test cannot touch memory the kernel and services already hold. A weak cell could hide there; only a full offline memtest86+ reaches it.
3. Pure retention failures cannot be reproduced from software — the memory controller refreshes every 7.8 µs regardless of what any program does. This test stresses pattern/coupling failures well, which is what most of the memtest errors were, but a retention-only weak cell could pass here and still be marginal.

So the honest verdict is strong support for the blacklist strategy, not proof. Upgrading support to proof needs two more things: a targeted probe of the known cells when cooled, and eventually a full cooled memtest86+ of the entire address space including the OS-held region.

4. Tooling built

• ~/bin/overnight-memtest.sh — the in-OS runner used here (stressapptest + a temperature sampler), so long memory tests never cost Plex uptime again.
• ~/bin/offline-bad-pages.sh — runtime soft_offline_page of the three known-bad pages; takes effect with no reboot (HardwareCorrupted: 12 kB confirms).
• ~/bin/apply-memmap.sh — makes the exclusion permanent via two memmap=2M$<addr> reservations (4 MB of 64 GB), self-verifying that the $ survives GRUB quoting and that usb-storage.quirks is preserved.
• ~/bin/cell-probe + ~/bin/run-cell-probe.sh — a targeted tester that maps only the two reserved 2 MB windows via uncached /dev/mem and hammers the exact failing words to 64-bit granularity, so the blacklist can be narrowed and the known cells can be checked against temperature while Plex stays up. Requires the memmap reservation to be active first.

5. Still to do

• One reboot to carry the BIOS cooling changes (Package C-State → Auto, CPU fan curve → Performance) and apply-memmap.sh; then confirm /proc/cmdline and that both regions read Reserved in /proc/iomem.
• Run run-cell-probe.sh cooled — does the known defect still fire?
• A hardware fix for cooling regardless of RAM outcome: the case has no chassis fans at all, and only two SSDs plus a hot CPU live inside it (every spinning disk is in the USB DAS, contributing no internal heat). Repaste + at least one exhaust fan.
• Eventually, a full cooled memtest86+ to clear the ~16 GB this test could not reach. Now usable interactively, since the keyboard works once the tool is running even though it is dead at the GRUB menu — press F1 for its address-range config to test narrow windows fast.

Net: the machine is now known to have a small, fixed set of bad cells rather than failing memory at large — and every byte of that was learned without taking Plex down.