SKYHOUSE.dev Journal

Why: Damien woke up to a wall of Telegram messages about the Sonarr queue — Down, Up, Down, Up — and asked whether Sonarr was configured differently from Radarr, because he only wants to hear about things that need action.

It was not configured differently. The two pipeline monitors were byte-for-byte identical. Sonarr was louder because it was the one that was working, and three separate defects had to line up to produce that result. Fixing them turned 12 messages a day into approximately zero, without removing a single real alert.

1. A DOWN and an UP, 93 milliseconds apart

The overnight sample looked like this: Down — No heartbeat in the time window at 12:30, Up — pipeline flowing at 12:30, and again at 4:00, 5:00, 5:30. Kuma's own heartbeat table gave the mechanism away immediately:

12:00:04.848  DOWN  No heartbeat in the time window
12:00:04.941  UP    Sonarr pipeline flowing — 0 items in queue

Ninety-three milliseconds. Nothing was wrong at either moment.

The push monitor's interval was 1800s and the cron that feeds it runs every 1800s. Zero margin. The probe takes one to six seconds depending on how quickly TorBox answers, so its push routinely landed a beat after the window closed. Kuma declared the heartbeat missed, the push arrived, Kuma cleared it — two Telegram messages per occurrence.

2. The grace period that did not exist

That should have been survivable, because provision-kuma.py declared a generous grace on every push monitor. It had no effect. Kuma consults retryInterval only while it is retrying, and every monitor was created with maxretries=0 — no retries, so the first missed window is an immediate DOWN and a notification. The "27h backup grace" and "35-minute DNS grace" recorded in the August 4 entry were decorative. Nothing had tested them, because nothing had missed a heartbeat yet.

The corrected model, now written into the provisioner:

Grace is therefore interval + retryInterval × maxretries, which the provisioner now prints so the intent stays legible. Live values: DNS apex 40 min, pipeline 70 min, nightly jobs 27h — the last one now genuinely 27h rather than nominally.

A second defect surfaced while fixing this: the provisioner only ever created monitors. Anything already present was reported as = exists and skipped, so the desired state in the file and the live state in Kuma were free to disagree forever — and did. It now diffs the fields it owns and updates them, printing each change. Deliberately narrow: asserting every field would stomp a maintenance window or a manual pause on the next run.

3. Sonarr was noisy because Sonarr was healthy

This is the part that inverts the premise. sonarr-queue-cleanup.sh has run every 5 minutes since May, so Sonarr's queue is empty, so its monitor sits UP — and every window miss produces a DOWN and an UP. Two messages.

radarr-queue-cleanup.py was written on August 4 and never added to cron. So Radarr never self-healed, sat continuously DOWN with four genuinely stalled items, and because resendInterval=0 means "say it once", it said it once and went quiet. Radarr felt calm because it was broken and staying broken. Of Sonarr's 12 alerts in the preceding 24 hours, 11 were the timing artifact and 1 was the nightly config backup. Sonarr contributed 20 of the 32 alerting beats in the whole week.

4. Telling a planned outage from a real one

The remaining message was Sonarr API unreachable: [Errno 104] Connection reset by peer at 3:30 AM — which is exactly when arr-config-backup.sh stops sonarr, radarr, prowlarr and both rdt-clients to take a consistent SQLite snapshot. A true observation about a deliberate outage, which is the definition of an alert worth suppressing.

Two independent mechanisms, because either alone is too fragile:

• A maintenance flag. The backup writes /run/lock/arr-stack-maintenance around the stop; the probe and both cleanups skip entirely while it is set. It carries an until= deadline, because a flag left behind by a killed backup would otherwise mute the pipeline monitor permanently — a far worse failure than the noise it prevents. Its EXIT trap now also catches INT and TERM: bash only runs an EXIT trap on a signal if that signal is trapped too, and cron wraps the job in timeout.
• Two strikes for an unreachable arr. A single failed API read is almost always a restart or a blip. It now takes two consecutive readings (~30 min) to report DOWN, and below that gate the probe pushes nothing rather than pushing "up" — claiming health it has not verified would be a lie, whereas silence is honest and Kuma's 70-minute grace is the backstop.

5. Reporting the fixing, not the fixed

Damien's actual request was narrower than "less noise": tell me about things needing attention, and about problems the system is actively trying to fix, and whether those efforts worked. Routine self-healing should be silent — that is the system working — but two things about it genuinely need a human, and neither was visible before:

• A loop. The same episode or film is remediated over and over. Every individual cleanup succeeds, so nothing errors, so nothing alerts — and the item still never arrives. That means every candidate release is failing, which points at the indexer, the provider, or the import path rather than at any one release.
• A failure. The cleanup call itself failed, so the pipeline has stopped self-healing. sonarr-queue-cleanup.sh could not report this at all: it piped its DELETE to /dev/null and never checked the status, so a broken self-heal looked exactly like a working one.

~/bin/arr-remediation-ledger.py records every remediation and speaks only on those two. The key detail is what it counts: the episode or movie id, never the release name. Blocklisting works precisely by swapping one release for another, so a release-keyed counter would never see the loop it exists to detect.

It also had to learn what "one attempt" means. Radarr's first live run cleared three dead Freeway releases in nine seconds, which the ledger counted as three failed attempts and immediately alerted on — a fair description of the situation, arrived at by faulty arithmetic. Remediations of the same key within 4 hours now collapse into one round, so the count means "rounds we have tried and failed", which is the only unit under which "three" means anything.

6. A removal that reports success and does not stick

Scheduling the Radarr cleanup exposed something worth recording: it logged Removed: 5, and four of the five items were back in the queue with identical IDs ten seconds later. This is the already-documented root cause from 2026-08-04 — the arrs rebuild their queues from the download client, and rdt-client still holds permanently-errored records — but the cleanup had no idea, and cheerfully claimed a success it had not achieved.

It now re-reads the queue after the pass and logs DID NOT STICK per item, naming rdt-client as the surviving holder. This does not alert: an item returning is remediation not sticking, and the ledger's loop detection already converts repeated rounds into exactly one message. What was fixed here is the log lying about what happened.

Still open: the rdt-client records themselves. bin/rdtclient-cleanup.py exists and is still manual-only.

7. Verification

• Provisioner: 8 monitors updated, then a re-run reported 0 changes planned, 8 already correct — idempotent. Settings confirmed directly in kuma.db.
• Maintenance flag honoured by all three consumers; an expired flag correctly failed open and did not suppress (the probe ran and reported the real Radarr stall).
• Strike gate, in isolation against a dead port: first miss silent, second miss DOWN with the message naming both, recovery resetting the counter to 0.
• Ledger: five remediations of one key in one pass → one round, no alert. Three rounds spread over 20h → exactly one alert, and the fourth suppressed by cooldown. Failure alert fires once, second suppressed.
• Sonarr's record extraction re-verified against synthetic records: stale entries in with episode keys, actively-downloading excluded, too-fresh excluded, unparseable-age still included via the queue-id fallback (the pre-existing behaviour).
• All edited scripts pass bash -n / py_compile. Crontab backed up to ~/.cache/plex.crontab.bak-20260805 before the edit.

Three defects, one symptom: the monitor that shouted loudest was the one with nothing wrong. It is quiet now, and the two arrs finally behave the same way as each other.