SKYHOUSE.dev Journal

Why: After a public-IP change, the site and all subdomains were intermittently unreachable for hours — and the Telegram alert just said "updating DNS" with no follow-up on whether it actually recovered. This is the recurring DNS/IP failure mode; the goal here was to make an IP change a fully hands-off event with a clear "you're back up" (or "needs you") signal.

Builds on the earlier DDNS consolidation. Several compounding causes, fixed in layers.

1. What was actually wrong

• High TTL (600s) meant every IP change left stale answers cached across the internet for ~10 min.
• The server resolved its own domain via Google DNS (8.8.8.8), hardcoded in netplan. Google kept serving the dead old IP, so the box itself half-failed to reach its own services.
• One of Dynadot's authoritative nameservers (ns1) was serving a stale record on a fraction of queries — re-poisoning resolver caches even after the record was correct everywhere else.
• The alert had no recovery confirmation — no way to know if/when the site was truly back.
• Reassuring non-finding: the * wildcard CNAME means subdomains always follow the apex — they never needed their own update mechanism.

2. Short TTL + forced nameserver re-sync

Added --ttl and --force flags to /home/plex/bin/dynadot-update.py, then re-pushed the records idempotently (same apex A + wildcard CNAME) with TTL 600 → 120. The re-push bumped the SOA serial, forcing Dynadot to re-propagate to both nameservers; the shorter TTL means future changes clear in ~2 min instead of ~10. ddns-go's own configured TTL was also set to 120 so it doesn't revert on its next update. (Caveat logged: ddns-go's config lives inside the container with no host bind-mount, so a container recreate would wipe the TTL setting and the Dynadot API key — worth giving it a volume later.)

3. The server now resolves its own domains over the LAN

• Dropped 8.8.8.8 from netplan (/etc/netplan/90-NM-df544b33-…​.yaml) so the box uses only the router (192.168.1.1) for DNS — no more stale-Google answers. Backup at .bak-20260602.
• Extended the /etc/hosts split-horizon (a 3-subdomain entry already existed) to the apex + all NPM subdomains → 192.168.1.136. The server now reaches every one of its own services directly over the LAN, fully independent of the public IP, public DNS, and any resolver caching. (No wildcards in /etc/hosts — new NPM subdomains must be added to that line.)

4. A real recovery notifier

New /home/plex/bin/ip-recovery-verify.sh, launched (detached) by ip-watch.sh the moment an IP change is seen. It polls until the new IP is genuinely live to the public — public DNS propagated (1.1.1.1 + 8.8.8.8) AND the site loads from outside (check-host.net, ≥2 external nodes returning 200) — then sends a Telegram ✅ "back up and publicly reachable". If it doesn't recover within 40 min it sends a ⚠️ with concrete manual steps (router port-forward → 192.168.1.136, check ddns-go, check Dynadot). The initial alert now sets expectations: "nothing for you to do — I'll confirm when it's verified back up, or tell you if it needs a manual look."

5. The IP-change flow now (all automated)

1. ISP changes the public IP.
2. ddns-go (polls ~5 min) updates the Dynadot apex A record (TTL 120). Wildcard CNAME carries every subdomain.
3. Dynadot propagates to its nameservers; public resolvers refresh within ~2 min of TTL expiry.
4. Router port-forward (80/443 → 192.168.1.136) is LAN-side and IP-independent; NPM, certs, services untouched.
5. ip-watch.sh (polls 5 min) sends the heads-up and launches the verifier, which sends the ✅ when externally confirmed.
6. LAN/server access never blips — it uses /etc/hosts → LAN IP throughout.

Manual steps in the normal case: none. The only hands-on scenarios are the ⚠️ path (e.g., the ISP reset the router's port-forward) or a Dynadot nameserver that won't converge (registrar-side — a support ticket).

Net: an IP change is now a hands-off event with a verified "you're back up" signal, the server never loses sight of its own services, and propagation windows shrank from ~10 min to ~2.