SKYHOUSE.dev Journal

Why: We were about to hand over a runbook telling Damien to run e2fsck -fy /dev/sdh1 on a machine where, four hours later, /dev/sdh1 was a different disk with a different filesystem on it. Checking the mounts before writing the instruction is the only reason this is a journal entry and not an incident report.

The RAM investigation earlier today (The Corruption That Wasn’t) ended with one pending destructive operation: an offline e2fsck on /media/plex1, deferred until the memory is trustworthy. Both STATE.md and that journal entry recorded the command with a literal device node, /dev/sdh1, because that is what plex1 was when we measured it that morning.

Damien rebooted four times that evening trying to reach the GRUB menu. When we re-checked the mounts afterwards, plex1 had moved.

1. What actually moved

Every media drive on the box was renumbered. Not one of the letters in server-context.md §7 survived:

• /media/plex1: documented sdc, measured this morning as sdh1, now sdd1.
• /media/plex2: documented sdg, now sdj1.
• /media/plex3: documented sdd, now sde1.
• /media/plex1_backup: documented sdh, now sdi1.
• /media/plex3_backup: documented sde, now sdf1.

And the specific hazard: /dev/sdh1 is now a 1.8 TB NTFS disk labelled DATA — a completely different drive from the one the pending e2fsck was written against.

We got lucky in one respect. e2fsck would have refused an NTFS volume outright (“Bad magic number in super-block”), so this particular stale letter was self-limiting. That is luck, not design: five of the letters in the rotation point at real ext4 filesystems, and e2fsck -fy against the wrong one of those answers “yes” to every repair prompt on a filesystem it has misidentified.

2. This is the third instance, not the first

The uncomfortable part is that the server already had this failure mode documented twice, and we still wrote a third one:

• backup-plex3.sh passed a hardcoded /dev/sde1 to robust_rsync.sh as its “mount if absent” argument. By 2026-07-30 that node was the My Passport 2 TB, so an unmounted start would have mounted the Passport at /media/plex3_backup and rsync --deleted plex3 over it. Rewritten to never mount anything. [journal]
• backup-plex2.sh held a stale /dev/sdb reference after that node became the Kingston scratch SSD. Neutralised the same day.

Both were caught by audit rather than by failure, which is fortunate but not a control. The pattern is stable enough to name: this box has five USB/DAS-bridged drives whose enumeration order is not deterministic, so any /dev/sdX written down is a time bomb with an unknown fuse.

3. What we changed

• STATE.md — the pending fsck now reads e2fsck -fy /dev/disk/by-uuid/73130bac-9c4c-42d1-90bd-ef9838716e87. Its existing “don’t trust sdX” note had itself gone stale (it asserted plex1 “is now sdh1”) and was rewritten to name no current letter, since that is the only form that cannot rot.
• The RAM journal entry — amended in place with a dated inline note. Normally the journal is append-only, but this session knew the line was wrong, which is exactly the case the supersession policy carves out.
• server-context.md §7 — the section was organised entirely by device letter and every row was wrong. Rather than discard the per-drive history attached to each bullet, we put an authoritative mountpoint → UUID → label table at the head of the section and marked the bullets’ letters as names-at-the-time.
• cheatsheet.html — new quirk entry with the UUID for plex1 and the resolve-at-use-time command.
• The photo_import project CLAUDE.md — added to the “read first” block, so a future session inherits the warning before it proposes anything.

The rule going in everywhere: address volumes as /dev/disk/by-uuid/…, and resolve at use time with findmnt -no SOURCE,UUID /media/plex1. A letter is a label, never an address.

4. The method note worth keeping

This was found by running findmnt before writing an instruction, not by testing anything clever. The general form: re-measure the identifiers in a destructive command at the moment you issue it, not at the moment you learned them. Today’s other lesson was that a checksum failure proves only that the bytes you compared did not match; this one is its sibling — a device path proves only what was true at the time you looked.

No data was lost. The pending e2fsck remains deferred behind the RAM question; it is simply now aimed at the right disk.