SKYHOUSE.dev Journal

Why: The WAN IP changed at 14:20 and the site went ~half-unreachable for over an hour — the "auto-redirection" we'd built reacted (Telegram alert fired) but nothing fixed anything, and when I dug in, the supposed manual fix turned out to be impossible too.

This entry reverses the headline decision of 2026-06-03 (apex ANAME → glddns). That design was based on a wrong premise; today's IP change exposed it. I converted the apexes back to a literal A record that the API can actually manage, wired up real remediation (not just alerting), and added a second, independent watchdog so two different triggers can each kick off a repair.

1. What actually broke

At 14:20 the public IP went 97.113.239.113 → 174.21.187.113. The router's own DDNS name cw526dc.glddns.com tracked it correctly within seconds. But skyhouse.dev stayed broken for ~half the internet for over an hour. The smoking gun:

• Dynadot ns1 served the dead old IP indefinitely. ns2 was correct; ns1 was stuck. Public resolvers got whichever they happened to cache, so ~50% of fresh lookups went to the dead IP.
• The "force a re-sync" fix is a fiction here. Dynadot does not increment the SOA serial on edits (it was frozen at 2026060203 on both nameservers even while they served different data), so a lagging replica has no trigger to reload.
• The documented manual fix couldn't run at all. dynadot-update.py --force calls set_dns2, and set_dns2 cannot write an ANAME record — its accepted types are only a, aaaa, cname, forward, txt, mx, stealth, email. Every attempt returned invalid record type. So while the apex was an ANAME, no script — automated or manual — could touch DNS.

2. Root cause: the ANAME's "staticness" was the bug, not the feature

The 2026-06-03 entry praised the ANAME because "the Dynadot record never changes, so there's nothing for ns1 to get wrong." That's exactly backwards. An ANAME isn't static at the IP level — Dynadot resolves glddns.com server-side and hands clients a plain A, caching that result per-backend. When the IP changed, the registrar record didn't change, so ns1 had no reload trigger and kept serving its cached old resolution forever. The 6/3 "verified 8/8 correct" only proved the zone was consistent at switch-time; it never proved ANAME re-resolution stays consistent across an actual IP change. It doesn't.

Proof in the act: the moment I pushed a real record edit (ANAME→A), ns1 got its trigger and converged within minutes. The earlier room101.com "ANAME worked, so ANAME is what saves us" conclusion was the same trap — one domain happening to look fine (its ns1 replica happened to be current) is luck, not evidence. Today botaa.org's ns1 was coincidentally correct at the same moment skyhouse.dev's was an hour stale.

3. The fix: literal A on the API-writable domains

Converted the apex from ANAME → cw526dc.glddns.com to a literal A → <current IP> (plus the unchanged * wildcard CNAME → apex) via set_dns2:

• skyhouse.dev — converted, TTL 120. ✅
• botaa.org — converted, TTL 120. ✅
• room101.com — blocked. It has Dynadot "Email Settings" configured, so set_dns2 refuses (Cannot set DNS … since this domain has existing "Email Settings"). It remains an ANAME and is therefore still vulnerable. To convert it you must first disable Email Settings in the Dynadot web UI (and decide whether you still want room101 email forwarding).

The trade-off we knowingly give up: the router no longer auto-feeds the apex. So the IP must now be pushed on change — which is the whole point, because a push is what gives ns1 a reload trigger.

4. New processes — two independent triggers, real remediation

The old setup was detect-and-alert only. Now there are two independent reactors, either of which initiates a repair, plus hardened IP detection:

• /home/plex/bin/lib-pubip.sh — shared public-IP fetcher: 5 sources × 3 retries. Fixes the repeated could not determine public IP (got '<empty>') blind spells in the ip-watch log on 6/6–6/7 that nearly made us miss the change.
• /home/plex/bin/dns-push.sh <domain> <ip> [ttl] — deterministic apex setter (A + wildcard CNAME). Replaces the broken dynadot-update.py --force path. Safe only for domains whose whole record set is {apex A, * CNAME} — i.e. skyhouse.dev and botaa.org; it would drop MX/TXT if any were added, so extend it first if that changes.
• ip-watch.sh (cron */5) — now actually pushes the new A to all managed domains on a change (was relying on the retired ddns-go), then launches the verifier. Reacts to IP change.
• dns-watchdog.sh (cron 2-59/5, NEW) — independent of ip-watch. Samples ns1/ns2 (3× each, to debounce the flapping) against the true current IP; on drift it re-asserts the A and sends one ⚠️ (then ✅ on recovery). Reacts to authoritative-DNS drift — the exact failure ip-watch is blind to, since the IP was already correct while ns1 lagged.
• ip-recovery-verify.sh — now re-asserts the record every pass while waiting (each push is a fresh reload nudge), and its timeout alert reports ns1/ns2/public side-by-side and points at dns-push.sh / the Cloudflare runbook instead of the dead ddns-go advice.

5. The honest limit — and why Cloudflare is still the real fix

Even after converting to A, I watched both Dynadot nameserver pools flap between the old and new IP for a while (some backend machines in each pool hadn't reloaded). Re-asserting nudges them and ns2 recovers fast, but we cannot force Dynadot's pools to converge on demand — there's simply no API lever for it. So this setup gives automatic, hands-off recovery bounded by Dynadot's own (minutes-to-tens-of-minutes, occasionally worse) convergence, which is a massive improvement over "stuck until a human logs in." But the only thing that eliminates per-nameserver split-brain entirely is a single anycast authority. The durable fix remains the Cloudflare migration (same flattened-CNAME pattern, a real DDNS API, plus free email forwarding which would also unblock room101). Damien is resistant to adding another company in the path; this A-record self-healing is the agreed interim until he's ready.

Net effect: the apex is a literal A the API can manage; an IP change is now auto-pushed, and a second watchdog independently catches and re-asserts authoritative-DNS drift — so a recurrence repairs itself instead of waiting for a human, within the bounds of Dynadot's flaky nameservers.