SKYHOUSE.dev Journal

Maintaining the Cloud Fortress

August 4, 2026

Kuma Stops Being a Mystery Background Process

Why: Uptime Kuma had been running for months with five monitors, one of them pointed at a service retired in June, and Damien could not have said what it watched or recalled ever hearing from it. It was ~15% of a capable tool sitting idle while we were about to hand-write the alert manager it already is.

This is the first implementation step of the monitoring redesign. The governing idea, arrived at after two adversarial design reviews: Kuma is not the product — it is the memory. The intelligence stays in small purpose-built scripts, which is where it already lives on this box and where it demonstrably works. What those scripts have always lacked is state: they fire once into Telegram and forget, so nothing clears, nothing repeats, and nothing notices when a watcher itself dies. Kuma supplies exactly that and nothing else.

1. Kuma was never broken — it was unconfigured

Before changing anything we checked why it had been silent. Its Telegram notification was already configured, already default, already pointing at the same chat every other alert on this box uses, and all five monitors were wired to it. The heartbeat history explains the silence entirely:

monitor            up beats   down beats
Plex                 2798          7
Nginx                2794          3
Music Discovery      2798          7
Plex (Remote)        2800          8

It monitors four things that essentially never fail. It had nothing to say. The problem was coverage, not plumbing — which meant new monitors would inherit working notifications for free.

2. What was built

3. Why push monitors rather than Kuma's native checks

Kuma natively answers "is X reachable?". Almost none of this server's real failures are reachability failures. room101.com resolved perfectly for 16 days — to the wrong address. The nightly plex1 backup runs, reports success, and quietly fails to converge. Media requests stall while every component in the pipeline reports healthy. Those need local knowledge, so the check stays in a script and Kuma receives only the verdict.

The apex check is the clearest case. It must query an external resolver: this box's /etc/hosts split-horizon maps every domain to the LAN IP, so anything resolving locally looks perfect while the public internet gets a dead address. That is precisely how the July outage hid. Had this check existed on 2026-07-19, it would have caught the miss within 15 minutes instead of 16 days.

4. Tuned against notification fatigue, deliberately

Both design reviews independently warned that the real failure mode of a monitoring system is the operator muting it. Kuma has no geometric backoff — only a flat "resend every N minutes" — so resendInterval is set to 0: notify once, do not nag. The backup monitors use a 24h expected heartbeat with a 27h grace, because a backup running long is not yet a backup that failed. Kuma's maintenance windows are the snooze mechanism for a known-open issue awaiting a part (the dead plex2 backup drive being the obvious case).

⚠ Correction (2026-08-05): the grace periods described in this section did not exist. They were set via retryInterval with maxretries=0, and Kuma consults retryInterval only while retrying — with no retries, the first missed heartbeat is an instant DOWN plus a notification. Combined with the pipeline monitors' interval being set equal to their pusher's cron period, this made flapping certain and produced 11 false alerts in a single night. Fixed in 2026-08-05 alert-noise-and-remediation §2, which also explains the correct model.

5. Two gotchas worth recording

Kuma 2.x breaks the client library. uptime-kuma-api 1.2.1 targets Kuma 1.x, and Kuma 2.3.2 added monitor.conditions as a NOT NULL column the library never sends. Every add_monitor died with SQLITE_CONSTRAINT: NOT NULL constraint failed: monitor.conditions. Existing rows store '[]', so a one-function shim injecting conditions: [] fixes it. This is exactly the socket.io version-coupling fragility raised in review — and also the argument for keeping that coupling in a provisioning-time script: it broke loudly, at the one moment someone was watching, rather than silently rotting as a runtime dependency.

Push tokens cannot be preset by the client library — Kuma generates them. So the provisioner reads them back after creation and writes /home/plex/ops/push-tokens.env (mode 600), which the checker scripts source. Tokens therefore live in exactly one place: a monitor deleted and recreated in the UI mints a new token, and the fix is re-running the provisioner rather than hand-editing shell scripts. (Reading back immediately also needs a short retry — the client caches monitors from socket events and ones created seconds earlier may not have landed.)

6. Monitoring must not be able to cause an outage

kuma_push returns 0 on every failure path — missing token file, missing token, Kuma down, network stall — and its curl is bounded at 5s connect / 10s total. A monitoring hook that fails a backup job is worse than no monitoring. Nothing is lost by swallowing these errors, because the dead-man's-switch on the Kuma side is what catches a job that stops reporting. Verified explicitly: bogus token → exit 0, missing token file → exit 0.

The arr-config-backup.sh abort path (exit 1 when /media/plex3_backup is unmounted) deliberately does not ping. No ping is the correct signal there, and the grace period converts that silence into an alert on its own.

7. Verification performed

Worth knowing: Kuma sends via its own internal Telegram client, not through our hardened /usr/local/bin/telegram_notify.sh, and it does not log notification dispatch at default verbosity. So Kuma-originated alerts will not appear in /var/log/telegram_notify.log — there are now two independent delivery paths to the same chat, and only the script-originated one is audited.

Six monitors where there were five, but for the first time they watch things that actually break — and a job that stops reporting is now itself an alert.

← Back to Admin Hub