SKYHOUSE.dev Journal

Why: Damien kept receiving “Radarr pipeline flowing — 0 item(s) in queue, none stalled” at moments when nothing was happening, and wanted to know what it meant. It turned out to mean nothing at all — and answering it properly meant auditing all 53 message bodies this server can send.

We inventoried every Telegram template across all three transports, then rewrote them on a single rule: the glyph answers exactly one question — does this need you? ❌ you act, ⚠️ automation is on it and a follow-up is coming, ✅ it is over. Severity lives in the words; the icon only routes your attention. Every message is now the same three lines: glyph + scope + object, then what is happening, then one → next step.

1. The message that meant nothing

The one Damien asked about was a recovery notice, and the mechanism behind it is worth recording because it silently destroyed the diagnostics we most needed.

When arr-pipeline-check.py found a stall, it pushed status=down to Uptime Kuma with a message naming the offending titles. Kuma files a down push as a PENDING beat, and PENDING does not notify. It then runs its own retry beat on retryInterval — and since that interval (15 min) was shorter than the probe's cron period (30 min), Kuma's beat usually landed first, found no new push, and wrote its own DOWN reading “No heartbeat in the time window”. Whichever arrived first won the race:

• 15 of 23 pipeline DOWN alerts in 30 days said “No heartbeat in the time window” — naming nothing.
• The other 8 carried the real diagnosis, purely by timing luck.
• Then the next probe run pushed up, and that produced the cheerful “0 items in queue” text — the only part of the whole exchange that reliably arrived.

Kuma could not be templated out of this, because it can only ever forward the pusher's msg string. So the alerting moved into the probe, which already had the classification, and the Kuma monitor became a dead-man's switch: pushed up on every run, including runs that find a stall, so its single DOWN condition is silence from the probe. Same split already chosen for Job · stranded-requests, and for the same reason.

2. Turning the classification into a glyph

The probe already knew the difference between a stall you must act on and one the stack will clear — it just had nowhere to say it. TorBox's verdict maps directly onto the question:

• not-at-provider → ❌ TorBox never accepted the grab; blocklist it so the arr picks another release.
• provider-ready → ❌ cached and finished, and the arr still has not imported. This one is ours.
• provider-incomplete → ⚠️ an uncached grab holding a slot; the slot guard and hourly cleanup handle it.
• Stall clears → ✅ closing the thread that a ❌ or ⚠️ opened.

Alerts fire once per downloadId and the ✅ only exists as a follow-up, so a persistent stall never nags and a resolved one never leaves you wondering.

3. The downloading notice, for free

Damien asked for “Radarr downloading [filename]”, and the obvious build was a Connect entry in each arr firing a custom script — container-side scripts, a spool directory, a flusher. Then it turned out the probe was already keeping, per downloadId, a first_seen timestamp used to measure stall persistence. A downloadId with no prior state IS a grab that just started. No webhook, no listener, no config inside the arrs — about twenty lines in a script that already had the state, the titles and a way to send.

Coalescing came free with it: one run sees all the new IDs at once, so an RSS sweep that grabs seven produces one message — “✅ Radarr downloading — Poser.2022.1080p.WEBRip / +6 more just started · 7 items in queue” — instead of seven texts. First sighting of a service seeds silently, the same rule arr-stranded-check.py uses, so switching this on did not announce the standing queue as a burst of new downloads.

4. Moving the probe to every five minutes

A download notice half an hour late is useless, so the cron went from */30 to */5. Two things had to be true first.

• Cost. Measured before changing anything: both arr queue calls answer in ~1.5 ms. The TorBox lookup is the only expensive part at ~105 ms, and it is now lazy — consulted only on a run that already found something stalled. A healthy five-minute tick makes no external call at all, so this cannot earn a 429.
• No-progress detection had to survive. That check compares sizeleft against a sample at least 20 minutes old; a faster cron would have quietly disabled it by always comparing against a five-minute-old sample. The progress sample is now carried forward untouched until MIN_PROGRESS_WINDOW has really elapsed, so the check is cadence-independent.

5. The false alarm we caught in the act

Mid-audit, at 14:50, the ISP changed the public IP (97.126.138.197 → 97.113.224.65). Cloudflare resynced all three apexes in under a minute and dns-monitor.sh reported it correctly. But during the blip dns-apex-check.sh could not resolve the router's DDNS name, and it treated that as a reason to skip silently — on the reasoning, written in good faith in its own comments, that Kuma's heartbeat grace would catch a sustained outage. What Kuma actually did was send three messages reading “No heartbeat in the time window”, naming no domain and no cause, for an event that had already fixed itself.

So that script got the same treatment: it now sends one Telegram covering all three domains, pushes its Kuma monitors up as dead-man's switches, and on an unresolvable target pushes a reason rather than nothing — escalating to a ⚠️ only if the target stays unresolvable for 45 minutes.

6. The rest of the surface

• telegram_notify.sh — the 🖥️ plex: header is gone. One server, so it never carried information, and it pushed the part that does off the first line of a phone notification.
• Backups — the failure body was tail -n 50 of the rsync log: the least actionable payload on the box, the template most likely to hit the 4000-character truncation, and it never stated how many files failed. A new lib-rsync-report.sh counts the unreadable paths and names the first one. The plex1 6h timeout is ⚠️, not ❌ — that job is known to run long, and only a genuinely missed night is your problem.
• security-patch-check.sh — it sent whenever any of five buckets was non-empty, including “known issue, no upstream fix” and “muted”: the two most likely to be non-empty on any day. That is a daily text whose every line ends in “nothing to do”. Only the three actionable buckets can open a message now; the quiet two ride along when one already exists.
• rdtclient-watchdog.sh — the pre-wedge alert ran to ~1,400 characters including a two-line SQL statement. It is now four lines, and its glyph follows the outcome: ⚠️ when auto-salvage acted, ❌ when it declined or is switched off and the queue stays blocked.
• dns-monitor.sh — the last live script still POSTing to api.telegram.org through its own tg(), so it missed the HTML escaping, truncation guard and send log added on 2026-08-09. Now on the shared sender. Two transports remain instead of three.
• plex-check — an update notice is ❌, which is the clearest test of the rule: nothing is broken, but nothing happens until you type the command.
• Also rewritten: docker-log-guard.sh, media-stack-update.sh, arr-remediation-ledger.py, rdtclient-stuck-alert.py, arr-stranded-check.py, cpu_watchdog.sh.

7. The silence rule

Nothing sends unless it needs you, is being worked on, closes a thread you already saw, or is a download starting. A healthy queue, a clean patch scan, a resolving domain: all silent. “Nothing to report” is only ever a follow-up, never an opener.

53 message bodies audited, three transports down to two, and the one alert that used to arrive reliably — the one that said nothing was wrong — is now the one that never fires.