SKYHOUSE.dev Journal

Why: Twenty minutes into the alert rewrite, Damien got “❌ DNS still wrong 45 min after the IP change” — from a script that had been rewritten one second earlier, about a DNS state that had been correct for 45 minutes.

A parallel session diagnosed it while this one kept working, and the diagnosis held up under independent checking. It was a false positive with an unusually instructive cause, and it exposed two genuine bugs that had nothing to do with the race.

1. Bash does not slurp a script

The timeline is exact:

• 15:35:01.758 — the refactor rewrote dns-monitor.sh in place, 3804 → 4330 bytes.
• 15:35:01–02 — cron fired dns-monitor.sh (*/5) in that same second.
• 15:35:02 — the alert went out, followed by syntax error near unexpected token `fi' at line 97.

Bash reads a script incrementally and tracks a byte offset into the file. It does not read the whole thing into memory first. The file grew by 526 bytes underneath a shell that was already partway through it, so its next read resumed at an offset that no longer meant what it had a moment earlier — and it executed a mix of old and new code.

Three pieces of evidence pin it down, and all three were re-verified here:

• The file passes bash -n cleanly now, yet that run died of a syntax error. A file that parses cannot fail to parse; a file being rewritten mid-read can.
• The alert has no entry in /var/log/telegram_notify.log. The new tg() routes through telegram_notify.sh, which logs every send. So the message left through the old private tg() that POSTs straight to api.telegram.org — old function, new state, one process.
• The old tg() only fires when TELEGRAM_BOT_TOKEN is set, which only happens via the old . "$KEYS_FILE" line the new file does not contain. That line ran too.

dns-monitor.changedat was still 1787349002 — 14:50:02 — so the IP-changed block did not re-run, yet age computed to exactly (15:35:02 − 14:50:02)/60 = 45. The arithmetic is verified; the precise line bash landed on is inference and not recoverable.

2. The alert thread could never be closed

This one was live, and it is independent of the race — any false alarm strands the same way.

The resync branch is gated on synced == 0, and it is the only thing that ever clears WARNED_FILE. State on disk was synced=1 and warned=1, so every subsequent run took the success path, found synced==1, and said nothing. Damien was holding a red alert that would never get a follow-up until the next IP change happened to reset the flag.

Under the house rule that a ✅ exists to close a thread you already saw, a warning that cannot be retracted is a bug, not a nuisance. There is now an elif [[ "$warned" == "1" ]] arm that sends the retraction and clears the flag. It fired on the first run after install:

• ✅ DNS is correct — retracting the earlier warning

3. A 35-minute WAN blackout nobody was told about

Between 14:15 and 14:45, get_pubip failed on seven consecutive runs — all five sources, three retries each. That is the WAN being down, and it is the single best early warning that an IP change is coming, since one usually follows the reconnect. The only trace was WARN: could not determine public IP; skipping in a log nobody reads.

dns-monitor.sh now keeps a consecutive-failure counter in ~/.cache/dns-monitor.pubipfails, alerts once on crossing PUBIP_ALERT_AFTER=6 (30 minutes), and sends a ✅ when the route comes back — the same shape dns-apex-check.sh already uses for SKIP_ALERT_AFTER.

4. Removing the failure class

The refactor touched 13 scripts in 12 minutes. dns-monitor.sh lost the coin flip; dns-apex-check.sh missed its 15:30 run by 112 seconds and got away with it. Two fixes, in order of how much they actually help:

• Atomic install — the real fix. ~/bin/install-script.sh writes a temp file in the destination directory, syntax-gates it (bash -n or py_compile, plus shebang and empty-file guards), inherits the target's mode, then rename(2)s it into place. Rename is atomic: a shell already executing the old file keeps its open inode and finishes cleanly on the old bytes, while every new execve gets the new file whole. Nothing ever sees a half-written script. Every change in this entry was installed with it.
• Parse guard — defence in depth. The executable body of the frequent bash cron scripts now lives in main(), called on the last line. Bash must parse the entire function before running any of it, so a file caught mid-write either runs one consistent version or fails to parse and runs nothing — it converts arbitrary half-execution into all-or-nothing. Applied to the four widest windows: dns-monitor.sh and cpu_watchdog.sh (*/5), sonarr-queue-cleanup.sh (*/5), dns-apex-check.sh (*/15), plus rdtclient-watchdog.sh (*/30). Each was run end-to-end afterwards.

The Python cron scripts were never exposed. CPython reads and compiles a module in full before executing a line of it, so arr-pipeline-check.py and its siblings are structurally immune to this. That is worth knowing: it halves the surface, and it is why the guard was applied to bash only.

A false alarm about DNS turned out to be a true alarm about how this box edits its own running code — and the two bugs it flushed out were both real, both independent of it, and both silent until something went wrong at exactly the wrong second.