SKYHOUSE.dev Journal

Why: Yesterday's pipeline rewiring traced back to a Docker reorg that emptied a config dir, which made Prowlarr mint a fresh API key and silently orphan every synced indexer. Damien asked the obvious follow-up — "make it not happen" — so we built the actual insurance: a backup we can restore from.

First I checked whether the API key could just be pinned declaratively in the compose. It can't — I tested a throwaway Prowlarr with PROWLARR__APIKEY set and the running API rejected that key (401) while still using the random one written to config.xml (200). The Servarr apps read their key only from config.xml, so the real protection isn't pinning the key — it's not losing the config dir, and being able to restore it if you do. (Pinning wouldn't have saved the indexers, download clients, or quality profiles anyway — those all live in the same dir.)

1. What's protected, and where it lands

All settings for the five-container stack live in plex-owned bind mounts on the boot SSD: /home/sonarr/data, /home/radarr/data, /home/prowlarr/data, /home/rdtclient/data, /home/plex/rdtclient-movies/db — i.e. each app's config.xml plus its SQLite DBs. The backup tars those to a different spindle, /media/plex3_backup/config-backups/, so a boot-disk failure can't take the backups with it. Bulk media is deliberately excluded — it's mirrored separately by the nightly plex1/plex3 rsyncs.

2. The backup script

/home/plex/bin/arr-config-backup.sh, nightly at 03:30 via the plex crontab, logging to /home/plex/.cache/arr-config-backup.log.

• Consistency by brief stop: it docker stops the five containers (~15s), which makes SQLite checkpoint its WAL into the main .db on clean shutdown, then tars, then restarts. A trap guarantees the stack comes back even if tar fails. The payoff for that 15s of off-peak downtime is that restore is a plain tar x — no per-database surgery while you're stressed.
• Mount guard: it refuses to run if /media/plex3_backup isn't a mountpoint, so it can never silently fill the boot SSD by writing under an absent mount.
• Lean archive: excludes the regenerable MediaCover artwork cache, the apps' own in-dir Backups, logs, and SQLite side-files — ~2.9 GB of source compresses to a ~40 MB archive. Keeps the newest 14.

3. The restore script

/home/plex/bin/arr-config-restore.sh — the thing you actually reach for at 2am.

• arr-config-restore.sh with no args lists available snapshots and does nothing.
• arr-config-restore.sh latest (or a specific filename) restores, after a y/N confirm.
• Before overwriting anything it saves a pre-restore-<ts>.tar.gz of the current state — so even a wrong restore is reversible. It stops the stack, clears stale WAL/SHM side-files so the restored .db isn't shadowed by an old write-ahead log, extracts over /, re-asserts plex:plex ownership, and restarts.
• It prints the one manual follow-up: if indexers look off afterward, run Prowlarr → Settings → Apps → Sync App Indexers (the keys are consistent again, so it self-heals).

4. Verified end-to-end

Ran the backup once by hand: 40 MB archive, ~11s of downtime, and it contains every config.xml plus all five databases (sonarr.db, radarr.db, prowlarr.db, both rdtclient.db). The stack came back healthy and the restore script's list mode reads the snapshot cleanly.

Net effect: the media-download stack's settings are now snapshotted nightly to a separate drive and restorable in one command — so the next time a Docker mishap empties a config dir, recovery is a restore, not another evening of re-deriving every mismatch.