SKYHOUSE.dev Journal

Why: A corrupt Postgres page in Immich's asset_exif table had been documented that morning as the top-priority fix, complete with a recommended repair. We sat down to run that repair. It turned out the page was fine, a second and much less pleasant problem was hiding behind it, and the recommended repair would have destroyed good data.

This one is worth reading for the method as much as the outcome. Two separate storage "corruptions" surfaced on the same morning, on two different drives on two different controllers. One was real. One was not. Telling them apart is what led us to the actual fault, which was in neither drive.

1. Two corruptions, one morning

The starting point was a note in the photo-migration handoff: one 8 KB page in asset_exif failed its checksum, pg_dump failed on that table, and so the nightly Immich DB backup was broken. The scope had been measured carefully — 24 of the 25 largest tables scanned clean, about 44 of 334,997 rows at risk — and two recovery paths were documented. The recommended fix was zero_damaged_pages = on followed by VACUUM FULL.

Before touching it I ran the checks the note said had never been run. Two things came back that weren't in the handoff at all:

• immich_postgres was reporting unhealthy. Immich's healthcheck counts pg_stat_database.checksum_failures, so this had been visible in docker ps the whole time.
• /media/plex1 — the drive holding all 164,000 photos — was in state clean with errors with 122 filesystem errors and climbing. Nobody had noticed because its Errors behavior is Continue: it never remounts read-only, it just keeps going and writes a counter into the superblock. It had never been fsck'd since the filesystem was created in December 2022.

2. The Postgres page that was never broken

The original diagnosis was reproducible — a forced heap scan errored every time. But a reproducible error is not the same as damaged media, and the test that had been run couldn't tell the difference between bad bytes on disk and good bytes misread. Postgres verifies a page's checksum after reading it into memory, so a fault anywhere along that path looks identical to a bad page.

The way to separate them is to force the read to come off the platter, which means clearing both caches:

docker stop immich_postgres                          # clears shared_buffers
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches     # clears the OS page cache
docker start immich_postgres
docker exec immich_postgres psql -U postgres -d immich \
  -c "SET enable_indexonlyscan=off; SET enable_indexscan=off; SET enable_bitmapscan=off;
      SELECT count(*) FROM asset_exif;"

334,997 rows. Zero checksum failures. pg_dump -t public.asset_exif then finished in 2.6 seconds, exit 0, all 334,997 rows. The failure counter has sat frozen at 6 ever since, last incremented at 15:13.

The page on disk had always been fine. The nightly backup was never permanently broken — it would have worked the next time it ran. We zeroed nothing and restored nothing. Had we run the documented fix, zero_damaged_pages would have blanked a perfectly good 8 KB page and thrown away roughly 44 real rows to solve a problem that did not exist.

Before any of this I took a safety dump excluding the suspect table (--exclude-table-data=public.asset_exif), which succeeded and verified clean at 335,004 asset rows. It turned out to be unnecessary, which is the correct outcome for a safety net.

3. What was actually damaged on plex1

Having just learned not to trust an uncorroborated checksum error, I re-tested plex1 the same cold way. It failed the same way it had before: the zeroed inodes still read as bad type, debugfs still refused to open the volume, and ls on an affected directory still returned Bad message and logged fresh kernel errors. That damage is genuinely on disk.

Scope, measured rather than assumed: 19 corrupt inodes plus 31 block groups with bad block-bitmap checksums. I resolved all 19 to paths, and every one lives under Photos/thumbs/e00fa930-.../ — Immich thumbnails. Sixteen are empty thumbnail directories created between 07:47:01 and 07:47:30, three minutes before the morning's 07:50 reboot, while Immich was mass-generating thumbnails for 70,000 freshly imported photos. Three are inodes wiped to Links: 0.

The question that actually matters is whether any originals were hit. Two independent checks say no:

• A find across library/ and upload/ for anything written between 07:00 and 07:55 returns zero files. Only thumbnails were in flight.
• I SHA-1'd the one real photo whose thumbnail died. It matched its asset.checksum in Postgres exactly. The original is intact.

So the damage is confined to derived data that Immich can regenerate on demand.

4. Finding the common cause

At this point we had file data misread on the internal SATA SSD and file metadata corrupted on a USB disk, on the same morning, on different controllers. That pattern is worth taking seriously, so we went looking for what they share.

What we ruled out first:

• The drives. Both SMART-clean — zero reallocated sectors, zero pending, zero offline-uncorrectable, zero CRC errors on the plex1 disk. The drives returned exactly what had been written to them, which points at the writing, not the storing.
• An unclean shutdown. The 07:50 reboot was orderly — systemd-shutdown synced filesystems and unmounted cleanly.
• The DAS. Tempting, given its history (see 2026-07-06 and 2026-07-30), but /media/plex3 sits on the same JMicron JMS567 bridge and is completely clean. So is plex3_backup. A shared-transport fault should not spare two of three volumes behind it.

That left memory, and memory is not monitored here because the RAM is non-ECC — there are no EDAC counters to consult. So I installed memtester and gave it 28 GB of the 48 GB free. It failed in under a minute, and kept failing:

offset 0x294fb2240   bit 43   byte lane 5   set
offset 0x294fb2d40   bit 44   byte lane 5   cleared
offset 0x354a3ef40   bit 43   byte lane 5   cleared
offset 0xbc479f0     bit 43   byte lane 5   set
offset 0x294fb2d40   bit 44   byte lane 5   cleared
   + "possible bad address line at offset 0x294fb2d40"

The pattern is the finding. Every single failure is a one-bit error on bit 43 or bit 44, and both of those live in byte lane 5 — across four different addresses spanning the whole 28 GB, produced by five unrelated test patterns. Random soft errors scatter across all 64 bits and all eight lanes. A fault that confines itself to two adjacent bits in one lane is a specific physical path: one DRAM chip, or a pair of adjacent data lines.

Contributing factor, almost certainly: the RAM is 2×32 GB non-ECC G.Skill DDR4 running at 3600 MT/s against an SMBIOS-rated 2600. Signal integrity degrades on particular data lines first, which is exactly the shape of what we're seeing — and it explains why memory that has been fine for years chose this week to misbehave, right after the heaviest sustained write the machine has ever done (a 195 GB import at load ~29).

One fault, both symptoms: it corrupted plex1's metadata on the way to disk, where it stuck, and the Postgres page on the way back from disk, where it didn't.

5. What we did, and what we deliberately didn't

• Action: took a verified safety dump of the Immich DB to /media/scratch/immich-repair-20260821/ (on the internal SATA controller, off the DAS), gzip-tested, 335,004 asset rows.
• Action: disproved the Postgres page corruption with a cold-cache read, and confirmed pg_dump works. No repair was needed or performed.
• Action: installed memtester and characterised the memory fault down to the failing bits and byte lane.
• Action: made the GRUB menu actually reachable. It was GRUB_TIMEOUT_STYLE=hidden with GRUB_TIMEOUT=1 — a one-second window on a UEFI box, which in practice means you cannot get to the memory test at all. Now menu / 10, so the menu appears on its own and waits. GRUB_DEFAULT=0 is unchanged, so doing nothing still boots Ubuntu; the USB usb-storage.quirks cmdline survived update-grub intact (checked). Backups: /etc/default/grub.bak-20260821-memtest, /media/scratch/immich-repair-20260821/grub.cfg.bak-20260821. Regenerating also added a UEFI Firmware Settings entry, which is a convenient way into BIOS to test the XMP question without knowing the vendor key.
• Deliberately not done: the e2fsck on plex1. It rewrites metadata across a 12.7 TB volume, and doing that through a known-bad byte lane is how a thumbnail problem becomes a catastrophe. It waits for the RAM.
• Deliberately not done: the planned Immich migration to /media/plex3. That copies ~1.1 TB of irreplaceable originals with the express intent of freeing the source afterwards. It is the single worst thing to run on suspect memory, and it also would not have escaped the DAS — plex3 is on the same bridge as plex1.
• Deliberately left red: immich_postgres still reports unhealthy. Its healthcheck counts cumulative checksum failures, so it cannot self-clear; SELECT pg_stat_reset(); will clear it, and that can wait until the RAM is fixed. An accurate warning light is worth more than a tidy dashboard.

6. The GRUB menu we could see but not steer

Making the GRUB menu visible (timeout 1 → 10, style hidden → menu) turned out to be only half the problem. On the first real attempt the menu appeared, counted down, and booted Ubuntu anyway — GRUB was not receiving any keystrokes. Neither a Logitech Unifying receiver nor Bluetooth worked, and both were plugged straight into rear motherboard ports, so hub depth was not the explanation. The wired Keychron K3 enumerates as an Apple Aluminium Keyboard (05ac:024f), which firmware is inconsistent about claiming at POST.

Rather than keep guessing at BIOS USB settings we could not reach without a working keyboard, we removed the need to press anything. GRUB's next_entry mechanism sets the default for exactly one boot and then clears itself, and this machine's generated grub.cfg already honours it unconditionally in the 00_header block — independently of GRUB_DEFAULT, which stays 0. grub-reboot only writes that variable, so no regeneration and no config edit were needed.

• Added /home/plex/bin/arm-memtest.sh. One command arms a single keyboard-free boot into memtest86+; --disarm cancels it.
• It refuses to arm if grub-probe reports the grubenv on lvm/diskfilter, where GRUB cannot clear next_entry — on a box whose keyboard does not work in GRUB, a sticky entry would mean booting memtest forever. Here grubenv is on plain ext4 (/dev/sda3), so it clears normally.
• It also greps the entry title out of grub.cfg before arming, since GRUB silently ignores a title that does not match.
• Exit is by holding the power button — safe, because memtest86+ mounts nothing and writes to no disk.

For the BIOS step that still lies ahead (dropping XMP to the rated speed), OsIndicationsSupported has bit 0 set, so systemctl reboot --firmware-setup boots straight into ASRock setup with no keypress. That solves getting in; navigating still needs the keyboard to be recognised.

7. Still open

• Confirm the RAM independently. memtest86+ 7.00 is already installed with a GRUB entry ("Memory test"), so it is one reboot away. Our fault reproduces in under a minute, so a ~12-minute run should be decisive and will name the DIMM slot. After that, the cheap experiment is dropping XMP to the rated speed and re-testing — if it goes clean, it's the overclock, not dead silicon.
• Then e2fsck -fy /dev/disk/by-uuid/73130bac-9c4c-42d1-90bd-ef9838716e87. (Amended 2026-08-21 22:30: this line originally read /dev/sdh1. By that evening plex1 had moved to /dev/sdd1 and /dev/sdh1 was a different, NTFS disk. Device letters on this box rotate across reboots — always address these volumes by UUID.) Better news than expected on timing: only 2,222,410 of 427,261,952 inodes are in use (1%), and e2fsck skips uninitialised inode tables, so this should run 15–45 minutes rather than all night.
• Then regenerate the affected Immich thumbnails — one job, derived data.
• das-down.sh needs whisparr added to its CONTAINERS list. It has a /media/plex1 bind mount and would have blocked the unmount that the fsck depends on.
• The plex3 migration stays parked until the memory is trustworthy.

Net effect: the "highest priority" corruption turned out not to exist, the repair we were about to run would have destroyed good rows, and the real fault was two bits in one byte lane of non-ECC RAM — found only because two unrelated drives failed on the same morning and that coincidence was treated as evidence rather than bad luck.