SKYHOUSE.dev Journal

Maintaining the Cloud Fortress

August 4, 2026

The DNS Self-Healing We Documented Was Never Actually Built

Why: While auditing the server's monitoring and alerting systems for a redesign, we checked whether the "IP changed and was never remediated" complaint had a real cause — and found room101.com had been serving a dead IP to the public internet for 16 days.

The 2026-06-07 Cloudflare migration was supposed to end public-IP churn forever: point each apex at the router's DDNS hostname via a flattened CNAME, and Cloudflare follows the IP automatically with no scripts and no credentials. The journal, the cheat sheet, and the context dump all describe that as the current architecture. It was not. All three apexes were still static A records, and had been the whole time.

1. What we found

Querying the authoritative Cloudflare nameservers directly, with the local /etc/hosts split-horizon bypassed:

The public IP changed on 2026-07-19 at 22:45. Two zones were updated manually in the Cloudflare dashboard that night; the third was missed. Nothing caught it for 16 days, because the only thing watching was dns-monitor.sh, and its entire record of the incident is two lines:

2026-07-19T22:45:01 INFO: IP changed: 174.21.187.113 -> 97.113.247.79
2026-07-19T23:05:02 WARN: not resynced 19m after change

One Telegram message, which did not name the failing domain, and no follow-up ever. The /etc/hosts split-horizon meant the site loaded perfectly from inside the house the entire time.

2. The fix

Damien created a scoped Cloudflare API token (Zone:DNS:Edit + Zone:Zone:Read, later widened to Workers and Email Routing) stored at /home/plex/.cloudflare.env, mode 600. Each apex A record was replaced in place with a flattened CNAME:

PUT /client/v4/zones/<zone>/dns_records/<record>
{"type":"CNAME","name":"<domain>","content":"cw526dc.glddns.com",
 "ttl":1,"proxied":false}

Cloudflare's apex CNAME flattening returns a synthesised A in responses, which is what keeps this legal alongside room101.com's MX records — a CNAME at the apex would otherwise conflict with them under RFC 1034.

Rollback record (pre-change state, all proxied=false, ttl=1):

3. Verification

All checks run against external resolvers, deliberately bypassing /etc/hosts:

4. What this corrects in the documentation

This is the more useful lesson. server-context.md contained the contradiction in plain sight: §4 described the flattening as done while §10 correctly listed it as pending — "Damien to do in dashboard." §10 was right. The cheat sheet and the 2026-06-07 journal entry inherited §4's version, and from then on every reader (human and AI) believed the server had self-healing DNS that did not exist.

Both documents were corrected in this change-set. The stale origin IP 174.21.187.113, which appears throughout the docs as "the origin," has been updated to reflect that there is no longer a hardcoded origin at all — the apexes track the router.

The general lesson: a planned step recorded as complete is worse than one recorded as pending, because it silently removes itself from the backlog. Where a document says a change was made, it should say how it was verified.

5. The API token, and its limits

This is a deliberate reversal of a prior architectural decision — the Cloudflare design specifically avoided holding any DNS write credential on the box. The token exists for operator-initiated infrastructure work (this change, future Workers deployment, Email Routing). Policy: no automated path gets it. No cron job, no watchdog, and no self-healing remediation script may hold or use this token. DNS self-healing needs no credential at all now — that is the entire point of the flattening — so any future script that wants DNS write access should be treated as a design smell first and a requirement second.

Three domains that each needed a manual dashboard edit after every IP change now need none, and the one that had been dark since July 19 is serving again.

← Back to Admin Hub