SKYHOUSE.dev Journal

Why: We had agreed, out loud and repeatedly, that no large writes should touch the backup drives while the RAM fault (see the RAM entry) was unresolved. Then at 04:30 backup-plex1.sh fired on schedule and started a rsync --delete mirror into our only copy of the photo library — while a memory stress test was deliberately cooking the DIMMs. The agreement was real; it just wasn't written anywhere cron could read it.

During the overnight stressapptest run, Damien asked a sharp question — essentially the inversion heuristic applied pre-emptively: "is a backup running right now, during the six-hour test we know has a high chance of flipping bits?" It was.

1. What was actually running

• backup-plex1.sh (user cron 30 4 * * *) had started at 04:30 and was ~1.5 h in.
• It runs, as root via /usr/local/sbin/backup-plex1-rsync.sh, the command rsync -a --delete /media/plex1/ /media/plex1_backup/ — a mirror, meaning anything on the backup absent from the source gets deleted.
• The source, /media/plex1, is the filesystem with known ext4 corruption. The destination is the only backup of the photo library.
• Concurrently, stressapptest was holding 46 GB under load and had driven the CPU package (and with it the on-die memory controller) to a sustained 89–92 °C — hotter than either memtest run.

2. Why this was the wrong thing to be running

Three independent risks stacked: (a) a --delete mirror can propagate damage into the good copy if the source reads wrong; (b) the source filesystem is already known-corrupt; (c) the RAM through which every byte passed was being deliberately stressed and heated, lowering the retention margin of cells we have not yet found and offlined. This is the precise combination the standing constraint existed to prevent. It ran anyway because the constraint lived only in conversation.

3. What actually happened (little, this time)

We stopped it (pkill of the wrapper and the root rsync) and checked. The damage was minimal, mostly by luck:

• Zero deletions were logged across the whole run. rsync was failing to read the known-corrupt thumbnail directory (e00fa930…, "Bad message (74)"), and an unreadable source directory makes rsync skip it — it does not delete the destination copy. All 38 log lines were that same read failure.
• The kill was clean: rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20). rsync writes to a temp name and renames on completion, so an interrupted run leaves a stray temp file, never a corrupted destination file.
• /media/plex1_backup was intact afterward: 12 TB used, 1.7 TB free, no space swing.
• The three known-bad RAM pages were already offlined (HardwareCorrupted: 12 kB), so rsync's data was not landing on the cells we had identified.

So the safety net survived. But "survived because rsync happened to choke on the corruption before --delete could act" is luck, not a control.

4. What we changed

• Commented out both mirror-backup cron jobs in the plex user crontab — backup-plex1.sh (30 4) and backup-plex3.sh (0 1) — each prefixed # DISABLED-20260822 (RAM fault: no large writes to backup drives until resolved). Crontab backed up first to /media/scratch/immich-repair-20260821/crontab.bak-20260822-0558.
• Left stressapptest running — it writes to no disk and detects miscompares in RAM, so it was never the hazard; the concurrent disk backup was.
• Recommended (not yet applied): add --max-delete=1000 to both mirror scripts as a circuit breaker, so a future run can never delete more than a sane number of files without aborting — a guard that would have made this incident structurally impossible regardless of cron timing or RAM state.

Both cron jobs and the --max-delete guard are recorded in the RAM-recovery reactivation checklist in STATE.md, so backups come back deliberately rather than by being forgotten back on.

5. The lesson

An operational constraint that is only spoken is not a constraint — it is a hope. cron does not read STATE.md, journal entries, or the decision we reached three hours earlier. If a rule matters, it has to be encoded where the machine enforces it: a disabled cron line, a lockfile, a --max-delete ceiling, a guard clause that checks a flag. This is the mirror image of the device-letter near-miss — there, a written-down fact went stale; here, an agreed rule was never written down at all. Both failures are the same shape: the system's real behaviour diverging from the model in someone's head.

No data lost. The catch came from asking "what is the worst thing that could be running right now?" before it backfired, rather than after — which is the whole value of the question.