SKYHOUSE.dev Journal

Why: Damien's longest-running complaint is "I request media and it never arrives, even though I can see it's cached and waiting at TorBox" — and every monitor we had reported the pipeline perfectly healthy the entire time, because every individual component was healthy.

This is steps 2 and 3 of the monitoring redesign: an end-to-end pipeline probe, and the capture-and-runbook layer that turns an alert from a noun into a verb.

1. The structural blind spot

Uptime monitoring asks "is X reachable?". Sonarr is up. Radarr is up. Both rdt-client instances are up and healthy. TorBox answers. Every check passes — and nothing moves. The failure is that items enter the download queue and never leave it, which is invisible to reachability checks by construction. Nothing on this box watched the flow.

The first exploratory query into Sonarr's queue found the symptom sitting there in the open: 76 queue records, with items added 2026-07-31 still "downloading" and frozen at exactly 50%, estimated completion four days in the future. Radarr's queue was more explicit still — The.Shape.of.Water.2017 added 2026-07-29 with sizeleft == size (zero bytes moved in six days), alongside items carrying "The download is stalled with no connections" and "qBittorrent is reporting an error".

All of that was already exposed by the arr APIs. Nothing was reading it.

2. The probe, and what "stalled" means

~/bin/arr-pipeline-check.py (cron */30) reads both queues and groups records by downloadId — a season pack appears as one record per episode, so without grouping a single stuck pack looks like eight separate failures. One live example collapsed 8 records into 1 fault.

It reports and never acts. Removing and blocklisting is sonarr-queue-cleanup.sh's job. Conflating "tell me" with "fix it" is how you end up unable to trust either.

3. Three defects found by testing against live data

The first version over-reported badly, and each fix is worth recording because each is a general trap:

• statusMessages are not errorMessage. The first run flagged "Not an upgrade for existing movie file" as a stall. That is Radarr working correctly and declining an import on purpose. Only errorMessage now counts as a fault; statusMessages are checked against a benign-pattern list first.
• Unknown age bypassed the persistence filter. Many queue records carry no added field at all, and the original logic treated "no age" as "old enough", so a transient error alerted on first sighting. The probe now records its own first-sighting timestamp, so every category must survive at least one full interval before it counts. Sonarr's stuck Klepper entries were correctly silent on their first pass as a result.
• "No progress" was measured against the previous run regardless of when that was. Running the probe twice 26 seconds apart reported 12 in-flight downloads as stuck — of course nothing moved in 26 seconds. A MIN_PROGRESS_WINDOW of 20 minutes now gates that verdict. The same run dropped from 12 false hits to 1 genuine errored item.

The lesson generalises: a probe tuned only against a healthy system will be tuned wrong. All three defects were invisible until it ran against real, messy, live data.

3b. The finding that inverted the premise

Damien supplied a TorBox API key mid-session, which let the probe cross-reference each stalled item against the provider. The arr's downloadId is the torrent infohash, so the join is exact. Two operational notes: TorBox sits behind a WAF that 403s urllib's default User-Agent (the same request via curl succeeds — a confusing failure worth remembering), and rdt-client's own API needs a session login at POST /api/Authentication/Login.

The result contradicts the assumption the complaint was built on. TorBox holds 24 torrents — 21 cached, 1 completed, 2 incomplete. Cross-referencing every stalled queue item:

Sonarr — all stalled items      NOT IN TORBOX
Radarr — 6 of 7 stalled items   NOT IN TORBOX
Radarr — 1df48075…              in TorBox: state=incomplete, present=False

Nothing was cached and waiting. These items never reached TorBox at all — rdt-client failed to add them, and they sat in the arr queues as ghosts for days. The one exception is present but unfinished: an uncached grab burning a slot, which is the known cached-only enforcement gap. The 21 cached torrents are the ones that worked.

So the correct fix is blocklist-and-re-search, not repairing an import path — the opposite of what "it's sitting right there and never arrives" implies. The probe now classifies every stalled item as not-at-provider, provider-incomplete, or provider-ready, and leads the alert with the breakdown, because that verdict is what decides which fix applies. A provider-ready item would be the genuine "cached but not importing" case; there are currently zero, and that absence is itself the finding.

TorBox is queried once per run, and a lookup failure degrades gracefully to the previous behaviour rather than failing the probe.

4. ops-capture — the verb

An alert saying "backup failed" is a noun. What is needed is which endpoint, why, what the log said, and the commands that fix it. Worse, most of these failures are transient in their evidence: by the time anyone looks, the queue has churned and the container has restarted. The diagnosis has to be captured at the moment of the alert or it is gone.

~/bin/ops-capture.py takes a profile (dns, backup, media, docker, disk, system), runs a fixed set of read-only commands each individually timeout- and size-bounded, and writes a self-contained bundle to ~/ops/incidents/<timestamp>-<profile>/ containing meta.json, the captured output, and a copy of the relevant runbook. The path goes into the alert. Then investigating is:

claude "read /home/plex/ops/incidents/20260804T023302-media and tell me what to do"

...with no re-describing the situation, which was the original motivation for this entire redesign.

Deliberately not web-served. An earlier design rendered these into the web tree behind basic auth. Dropped on review: capture output can contain API keys, redaction-by-deny-list is a weak primary control, and one Nginx alias mistake exposes everything. This is a single-operator, terminal-first box, so the bundle lives at a local path read over SSH. Redaction is still applied, but as defence-in-depth behind a stronger control — the profiles simply never run commands that emit secrets (no docker inspect, no config dumps; status output and log tails only). Verified on a live bundle: no 32-hex API keys, no Telegram token, no leaked arr keys — and no redaction markers either, because there was nothing to redact.

Capture happens on the transition into failure, never on every run. A stall persisting for days would otherwise mint a bundle every 30 minutes and bury the one that matters. Verified: one bundle on transition, none on the repeat while still down.

Two safety properties, both because a capture runs precisely when the system is already unwell: every command is bounded at 20s (a hang in docker logs is itself recorded as a finding — it means the daemon is saturated), and the tool refuses to run at all when the root disk has under 500 MB free. Writing a diagnostic bundle must not be the thing that tips a full-disk incident over.

5. Runbooks

~/ops/runbooks/{media,dns,backups}.md, each section anchored so alerts can point at it — the pipeline monitors emit runbook: media#stalled-queue. Fixed shape: Symptom / What it means / Check / Fix / Escalate. The backups runbook also carries the standing risks that have no monitor because there is no job to watch: plex2's 13 TB with no copy, the root disk with no backup, nothing off-site.

6. Current state

Kuma now runs 12 monitors — 4 original HTTP checks plus 8 new push monitors (3 DNS apex, 3 job dead-man's switches, 2 pipeline). It started the day at 5, one of which pointed at a service retired in June and was deleted. The two new pipeline monitors went DOWN immediately on real findings:

Pipeline · Sonarr queue   DOWN   1 stalled of 62 queued
Pipeline · Radarr queue   DOWN   5 stalled of 10 queued
   The.Shape.of.Water.2017    no-progress   6d4h
   Edtv.1999.1080p.BluRay     errored       4d5h   "stalled with no connections"
   Las Corrientes (2025)      errored       3d14h  "qBittorrent is reporting an error"

These are real and pre-existing — the probe did not create them, it revealed them. Some have been sitting in the queue since 2026-07-29.

The pipeline has been quietly failing for at least six days while every monitor on the box reported it healthy. Now it says so, names the titles, and leaves a bundle to read.