Cloud Fortress Control Center
Domains
Containers
A six-hour in-OS stress test — 638 TB of read/write/compare — passed with zero errors, holding clean at 92 °C and again cooled to 79 °C, all while Plex stayed up. The finding that matters: the fault is localized to a few benched cells, not a whole array going fragile, which makes "blacklist and keep running" a real option. With three honest caveats about what an in-OS test can't reach, and the targeted tooling built to close them.
We had agreed: no large writes to the backup drives while the RAM fault is open. Then at 04:30 backup-plex1.sh fired on schedule and started an rsync --delete mirror into our only copy of the photo library — during a memory stress test cooking the DIMMs to 92 °C. Caught by asking "what's the worst thing running right now?" before it backfired. No data lost, mostly by luck; both backup crons now disabled. The lesson: a rule that lives only in conversation is a hope, not a constraint.
/dev/sdX Nearly Cost Us a DiskA pending e2fsck -fy /dev/sdh1 was written down in the morning. By evening /dev/sdh1 was a different disk with a different filesystem, and every one of the five media drives had renumbered — including all of server-context.md §7. The server had already documented two near-misses of exactly this shape and we still wrote a third. Everything now addresses volumes by UUID.
The morning’s top-priority fix was a corrupt Postgres page in Immich’s asset_exif. A cold-cache re-read showed the page on disk was perfect — and the documented repair would have blanked ~44 good rows to fix nothing. The real damage was on /media/plex1, which had been running clean with errors in silence; the real cause was neither drive but two bits in byte lane 5 of non-ECC RAM overclocked to 3600. No originals lost.
A “DNS still wrong” alert arrived from a script that had been rewritten one second earlier, about a DNS state that had been correct for 45 minutes. Bash tracks a byte offset rather than slurping a file, so cron caught it mid-write and ran a mix of old and new code. Fixed with atomic install and a parse guard — and the false alarm flushed out two real bugs: a warning that could never be retracted, and a 35-minute WAN blackout nobody was told about.
The recurring “Radarr pipeline flowing — 0 items in queue” text turned out to be a recovery notice for an alarm that had said nothing: Uptime Kuma files a down push as PENDING and then overwrites the diagnosis with “No heartbeat in the time window” — 15 times out of 23. Audited all 53 message bodies and rewrote them on one rule: the glyph answers only “does this need you?” Alerting moved off Kuma into the scripts that hold the classification, and the probe’s own state file turned out to already know when a download starts.
Six scripts left in ~/bin after the June Cloudflare migration were still shaped exactly like live tooling, and carried eight Telegram templates that could never fire — one of which still told you to re-push an A record to a Dynadot nameserver we no longer use. Moved to ~/bin/retired/ with the rollback path intact, and corrected every living document that pointed at them.
Four complaints about the Movies queue turned out to be three causes and one dropdown: rdt-client was receiving Radarr’s removal requests and discarding them, which is the whole reason Movies held 94 records against TV’s 13. Underneath it, two dead magnets from one bad indexer were holding both paid TorBox slots — and a full slot makes TorBox queue new adds, which rdt-client misreads as failure, which is where the nameless DIFF_ISSUE records come from. Cleared 58 stranded records and added the one alert that actually needs a human.
A 263 GB folder written off as a duplicate and queued for deletion turned out to be four-fifths absent from Immich — Flickr originals, not the Google Takeout everyone assumed. Imported 70,239 of them content-verified, then retired 783 redundant copies under a rule that ranks on pixels rather than the _o suffix, because the suffix lies.
A week of Radarr and rdt-client alerts turned out to be one disease: TorBox flaps, returns its status page as HTML, rdt-client crash-loops parsing it, and every “remove” call between TorBox, rdt-client and Radarr silently fails with nothing recording that it needs retrying. Scheduled the cleanup that was written 16 days ago and never wired up, cleared 16 dead records and a movie that was really an 877 MB Windows executable, and fixed the seeding bug that made one backlog arrive as 51 alerts.
Whole-house Wi-Fi glitching traced to 5 GHz clients stuck at 7 Mbps under load despite excellent signal. Reboot took the worst from 7 Mbps to 585 and 3189 ms to 36 — and changed the public IP, which the flattened apexes self-healed. Left behind root SSH, a snapshot tool and a full router reference.
Left a failed request alone to see what the system would do about it, and the answer was nothing, forever, silently — Radarr has no scheduled search for missing movies, and RSS only carries newly published releases. 55 titles were in that state. Added failure retry, a rule that makes this class of stall detectable at all, and a daily check that alerts on newly stranded requests without ever going permanently red.
TorBox's checkcached answers for arbitrary infohashes, so "is this instantly available?" is knowable before grabbing — which collapses the manual grab-wait-retry loop into one query. Built a cached-first picker and a library scanner; the scanner's first run offered 220 GB of "savings" that were mostly quality downgrades, because Radarr labels much of the library Unknown and the resolution helper defaulted those to 1080p.
Refusing BR-DISK promoted Remux-2160p (82 GB median) to the top of the profile, so the fix for one bad default handed us a worse one. Set size caps and the first custom formats this server has ever had, capped simultaneous transcodes (they were unlimited, and past the GPU's ceiling Plex silently falls back to CPU), and confirmed the GTX 1050 Ti is genuinely doing HEVC decode.
Home-LAN SSH to skyhouse.dev refused: the GL.iNet router doesn't support NAT hairpin/loopback, so use the LAN IP when home. Also traced a fail2ban ban of 192.168.1.1 (the router itself) to a dev-tool SSH push that used the local username instead of the server's — unbanned it and added a permanent ignoreip exception.
rdt-client wedged for 16 hours on a Blu-ray disc rip whose BDMV/BACKUP/ tree produced duplicate database paths — one bad row rolling back every tick's transaction, 121,112 exceptions, nothing moving. The monitoring caught it perfectly and the alert naming the cause could never be delivered: its own SQL remediation snippet contained <id>, which Telegram's HTML parser rejects. Rewrote the transport to escape, cap, and log the reason — and found the setting that stops rdt-client hoarding finished records.
Swap looked full a day after the Immich import, but nothing was actively swapping — it was stale pages from idle desktop apps that Linux never pages back. Chasing it found the real issue: the 32 GB scratch-SSD swapfile had been commented out of /etc/fstab, so the box was running on 4 GB. Restored it, drained the root swapfile 3.6 GiB → 67 MiB.
Reconciled Ellis's and Damien's Google Takeout exports into Immich — 135,202 and 94,305 assets, six files lost to pre-existing corruption, ~300 GB of SSD reclaimed. Documented the two ways immich-go reports Errors: 0 while silently skipping files: it abandons the rest of an archive on a bad CRC, and from-google-photos drops any file lacking sidecar JSON into a "pending" count that a re-run won't fix.
Cleared the last rdt-client orphans and proved the loop closed — arr-level removals finally stuck. Then worked out why the arrs dumped ~36 duplicate films: a bulk import that couldn't parse legacy filenames, a folder template keyed on a year that moves, the same rip published under two names, and a pile of Damien's own Blu-ray rips. Built tv-duplicate-audit.py because Plex has no duplicate detection for TV at all — it found 400 duplicated episodes and 228.9 GB safe to delete.
Sonarr's queue monitor sent 12 alerts in a day and 11 were a timing artifact — its heartbeat window exactly equalled the cron period that fed it, and the declared grace never existed because Kuma ignores retryInterval when maxretries=0. Radarr only seemed calmer because its cleanup was never scheduled, so it sat permanently DOWN. Fixed both, taught the probe to recognise the nightly backup's planned outage, and added a remediation ledger that stays silent while self-healing works and speaks when it loops or fails.
Two on-demand tools for the 5-hour token blocks: warmclaude pre-starts the block clock before a work session, and the /span-sessions skill lets long unattended jobs checkpoint, park at block exhaustion, and self-resume after the reset. Now publicly installable at skyhouse.dev/span-kit/.
Cleared 37 permanently-errored rdt-client records so the arrs stop re-serving downloads that can never finish. Also the sharpest self-inflicted mistake of the project: setting all three delete flags to "false" for safety made the tool a no-op that reported success, because the flags specify what to delete rather than adding extras.
Built Radarr and TorBox cleanup tools with hard path-safety gates, ran them, watched the queue drop from 10 to 1 — and hours later the same items were back. rdt-client holds 46 permanently-errored records that the arrs faithfully re-display, which is the real reason requested media never arrives.
Every monitor said the media pipeline was healthy because every component was — while items sat in the queue for six days. A probe that watches the flow found them immediately, plus ops-capture and runbooks so an alert says what to do instead of just what broke.
Uptime Kuma had run for months watching five things that never fail — it wasn't broken, just unconfigured. Now it holds the state our one-off scripts never could: three apex-correctness checks via an external resolver, and dead-man's switches on the nightly backups so a job that stops reporting is itself an alert.
A parked-work record, not a change. Pricing an off-site backup ran into the realisation that restored databases are useless without the 30 TB of content they index — so the real problem is mirroring the DAS, which needs its own project. Measurements, options, and costs recorded; a boot-drive mirror carved out as the smaller, higher-priority piece.
A review predicted our monitoring scripts would pile up and hang under exactly the conditions they detect. All six were unguarded, and the shared Telegram transport had no network timeout and never checked whether a send succeeded. Now wrapped in flock + timeout, with a delivery audit log.
room101.com had been serving a dead IP to the public for 16 days. The June migration's apex CNAME flattening — described as complete in one section of the context dump and correctly as pending in another — had never been done on any of the three domains, so every IP change was a manual dashboard edit. Now genuinely self-healing.
Downloads failing and the browser crawling looked like /media/plex2 again, but the drive was fine at 192 MB/s — a single wedged database row had rdt-client throwing 8 stack traces per second into a 29 GB unbounded Docker log, pinning dockerd at 136% CPU and saturating the root disk.
smartd had been dead since the reboot, all three media backup crons were commented out (one behind a note claiming it had been re-enabled), and two backup scripts held stale device nodes that now pointed at the My Passport and the scratch SSD. Repaired, hardened, and restarted.
Browser downloads stalling for minutes traced to the DAS negotiating a 480 Mbit link on a USB 2.0-only rear socket; fixed the never-matching UAS quirk, moved to SuperSpeed, and went 33 → 201 MB/s while recovering a missing 10 TB bay.
Updated Plex to 1.43.3.10828 via manual .deb (Pass/beta builds don't come through apt), migrated the stale apt repo to repo.plex.tv so future public releases track normally, and added Lidarr + Whisparr (V3/Eros) to the media-downloads Portainer stack alongside a re-pull of Sonarr/Radarr/Prowlarr/rdtclient.
Traced the multi-day crash storm to two independent USB-storage faults: the JMicron JMS583/UAS DAS flooding the USB bus, and a dying WD My Passport (sde) that hung the kernel every 5 min. Pulled the My Passport, then fully recovered the DAS by forcing the JMS583 out of UAS (usb-storage.quirks=152d:0583:u) and reseating loose drives — all four DAS drives enumerated clean and SMART-passed (no bad drive was ever inside it). plex1 + plex3 remounted, the complete media stack + immich restored (0 restarts, HTTP 200). Also revived smartd and fixed Home Assistant's HubZ stick (cp210x). New rule: the DAS is a post-boot hotplug (never attached at cold boot, where a POST crash is unrecoverable) — managed by das-up.sh/das-down.sh.
A new webapp for customizing and exporting print-ready daily calendars was stood up at photocalendar.skyhouse.dev. Added a corresponding tile to the main server index with a matching style and calendar icon.
After Dynadot's ns1 served the dead IP for 3+ hours and couldn't be forced, we flipped all three domains' nameservers to Cloudflare (via the Dynadot API), set every record to DNS-only (grey cloud) to dodge the proxy 100 MB/video limits, and rebuilt room101's email on Cloudflare Email Routing (verified working). Replaced the whole Dynadot-era script pile with one token-free notifier dns-monitor.sh (🌐 IP-changed / ✅ resynced / ⚠️ alerts). Last manual step for full self-heal: flatten each apex to a CNAME → cw526dc.glddns.com. Accelerating ISP IP churn is now background noise.
An IP change left Dynadot's ns1 serving the dead old IP for over an hour while everything else was correct — because the ANAME apex never changes, ns1 had no reload trigger, and set_dns2 can't even write an ANAME so the "force re-sync" fix was impossible. Reverted skyhouse.dev + botaa.org apexes to a literal A (room101 blocked by Email Settings), wired ip-watch.sh to actually push the new IP, added an independent dns-watchdog.sh cron that detects ns1/ns2 drift and re-asserts, hardened public-IP detection, and made the verifier re-assert while it waits. Honest limit: Dynadot's pools still flap; Cloudflare remains the durable fix.
New Portainer stack home-assistant runs Home Assistant (:8123) and Music Assistant (:8095), both on network_mode: host so mDNS/UPnP discovery works — which meant pointing the ha./ma.skyhouse.dev proxy hosts at the host LAN IP (not a container name), enabling WebSockets, and pasting a hardened location block for MA's auth handshake. HA needed trusted_proxies + use_x_forwarded_for to stop 400-ing the proxy, and its hardcoded homeassistant: YAML block deleted to hand timezone/URL config back to the UI. Closed out by passing the HubZ Zigbee/Z-Wave stick through to HA and onboarding a first IKEA light via the Philips Hue hub.
A reboot showed Dependency failed for media-plex3_backup.mount — but that boot-time line is cosmetic (nofail keeps it from blocking boot). The real cost was a 90s shutdown stall when the cheap USB→SATA bridge wedged on unmount (Unmounting timed out. Terminating.). Since one TimeoutUSec governs both mount and unmount on a .mount unit, we extended the 2026-06-01 plex1_backup fix to every media mount: a TimeoutSec=10 drop-in on plex2, plex3, plex3_backup, plex1, and scratch, applied live via daemon-reload (no remount). Now no wedged drive can stall a reboot more than ~10s, and it carries forward to the incoming DAS whether it lands on USB or eSATA.
Made the stack's compose current and redeploy-safe. The deployed v7 already matched reality; the only drift was bitmagnet — a leftover from an older revision, crash-looping 594× against a missing postgres and unused by Prowlarr — so we docker rm -f'd it (no redeploy needed). Saved a maintained host-side copy of the stack at /home/plex/docs/media-downloads-stack.yml (it previously existed only inside Portainer's volume), dropping the obsolete version: key and normalising a volume; validated with docker compose config. Kept images on :latest to match Damien's redeploy-to-update workflow.
Built the insurance against yesterday's settings loss. First confirmed the *arr API key can't be pinned in the compose (tested: PROWLARR__APIKEY is ignored, the app only reads config.xml), so the real fix is backup + restore. New bin/arr-config-backup.sh runs nightly at 03:30: briefly stops the five containers for a consistent SQLite snapshot, tars sonarr/radarr/prowlarr/rdtclient(-movies) config to /media/plex3_backup/config-backups/ (~40 MB, keep 14), and refuses to run if the dest drive is unmounted. Paired bin/arr-config-restore.sh lists snapshots and restores in one command, saving a pre-restore snapshot first so it's reversible. Verified end-to-end.
A Docker error left the *arr stack a mess of mismatches: TV rdt-client was set to remove torrents on finish and reported the Windows-default C:\Downloads path; remote path mappings were keyed to the wrong host; Sonarr had self-blocklisted 9 releases from days of failed imports; Radarr was reaching rdt-client via flaky host-hairpin; and the oldest synced indexers still carried Prowlarr's old API key (regenerated by the rebuild) → 401s. Fixed each via the apps' APIs (rdt-client via stop/edit/start), added Prowlarr download clients, repaired a dead collection root folder, and resynced indexers. All three apps now report clean health and test green.
Found smartd silently dead since boot (stale Corsair line) — rebuilt /etc/smartd.conf fully by-id (all 7 drives by serial; plex3_backup as -d removable so it survives unplugs). Switched netdata's smartctl collector from a fragile /dev/sdX exclusion to monitor-all. Recovered the swapped-in plex3_backup with a software "virtual replug" of the USB toaster (USB-SATA bridges don't hot-swap), e2fsck'd plex3 (errors were transient bus glitches, not corruption), moved the 32 GB secondary swap onto the scratch SSD, and tidied the retired plex2_backup fstab entry — all live, by-id, no reboot.
Fresh SATA cables fixed the morning's flapping — plex1 is stable (zero link resets) and the 128 GB Kingston is back. Confirmed the retired plex2_backup WD is genuinely dead (5,483 offline-uncorrectable sectors, even through the toaster). Built the Kingston into a 117 GB scratch disk at /media/scratch, moved Plex's transcode temp onto it (downloads deliberately left on the media drives for hardlink locality), and reverted the protective measures: Plex unmasked, plex1 mirror re-enabled.
plex1's internal-SATA link is flapping again (drive SMART-pristine; failures are link-layer) and its ext4 wedged mid-I/O. Masked Plex so it stops retrying the indexer, and paused the nightly plex1→plex1_backup rsync — both reversible — until a physical cable/port/power fix or the 6/8 DAS migration. The unhealthy rdtclient-movies container is downstream of this, not a config bug.
Replaced the 12-year-old Corsair boot SSD with a 1TB SSD via a quiesced live ddrescue clone (0 errors), grew root online to 938 GB, re-seated plex1, and retired both the Corsair and the dying WD plex2_backup. The 128 GB scratch SSD dropped on a loose cable — pending re-seat.
Plan for the incoming Mediasonic 8-bay DAS: shuck the easystore externals (frees USB ports), consolidate the hot "toaster" drives, run SnapRAID+mergerfs parity (parity ≥ largest data drive, ideally on internal SATA), LUKS bays for a neighbor's offsite backup, and a USB-3.x-powered-hub port layout. Includes the 3.3V-pin shuck gotcha + burn-in checklist.
Pointed the skyhouse.dev + botaa.org apexes at the router's built-in DDNS name (static ANAME → cw526dc.glddns.com, like room101.com) and retired ddns-go — the static records eliminate Dynadot's ns1 split-brain entirely (verified: ns1 went 8/8 correct instantly).
Ready-to-execute plan (not yet done) to move DNS hosting to Cloudflare's anycast nameservers if Dynadot keeps split-braining — with full record inventory, ddns-go reconfig, and rollback. Also documents botaa.org joining the ddns-go pipeline and room101.com's self-tracking via the router's glddns DDNS.
Made an IP change hands-off: TTL 600→120, forced a Dynadot nameserver re-sync, stopped the server using stale Google DNS, pointed all subdomains at the LAN IP in /etc/hosts, and added a notifier that confirms via Telegram when the site is verified publicly back up (or flags manual steps).
Prep for replacing the ancient Corsair boot SSD: a full config/DB safety backup to plex3, a disk-layout snapshot, and a step-by-step offline-clone runbook (ddrescue, the duplicate-UUID trap, grow root, EFI) — including the BENFEI=flaky-JMicron enclosure caveat.
Stopped netdata from SMART-probing the flaky USB bridge (the real reboot stall); fixed the plex1 backup, which was blocked by immich's root-owned library, via a narrow root rsync wrapper; and consolidated DNS onto ddns-go after finding the custom updater was a racing no-op.
Killed the ~30s of red [FAILED] on every reboot: replaced smartd's blanket DEVICESCAN (which hung on a JMicron USB bridge) with explicit by-id disk lines, and added systemd dependencies so docker/Plex stop before their media disks unmount.
Freed ~19 GB on root (95%→77%), added a read-only hardware-diagnostics sudo drop-in for the coming drive work, and fixed the weeks-failing plex1 backup — the target was accidentally NTFS, not a dying drive; reformatted to ext4 and hardened the script.
Purged a self-sustaining download loop from rdt-client + TorBox, and fixed the queue-cleanup cron so dead grabs get blocklisted instead of endlessly re-grabbed.
Fixed root cause of "downloads complete but never import": RDTClient was deleting completed torrent records before Sonarr could poll them. Added weekly queue cleanup cron for edge cases.
Deployed a second RDTClient instance so Sonarr uses a strict cache-only bouncer while Radarr gets an open door for live torrent slots — each downloading directly to its own media drive for instant atomic moves.
Recursively fixed ownership and permissions for the TV and Movie libraries to resolve Sonarr import failures (UnauthorizedAccessException).
Fixed a volume-mount crashloop and redeployed, but the tool targets the Immich 1.x API and is incompatible with our 2.7.5 — retired it in favor of Immich's built-in duplicate detection.
Pinned Portainer to 2.42.0, deployed sonarr/radarr/rdtclient via the API to bypass the phantom-Network-Error UI bug, and documented the surviving cosmetic quirks.
Disabled nginx response buffering for the Portainer host so streamed stack-deploy progress reaches the browser instead of triggering a 499 client abort.
Added a permanent read-only sudo allowlist and session-scoped credential lease so Claude can run approved sudo commands without copy-paste friction.
Added a nightly check to alert when a new Next.js update is available for the music-discovery app.
Removed retired proxy hosts and orphaned TLS certificates via the NPM web UI.
Removed obsolete botaa Docker networks and retained btst_default.
Disabled OpenVPN, restricted NPM admin port 81 to LAN, and archived the obsolete plexupdate script.
Archived 19 stale files and one-off scripts from /home/plex and the web root.
Fixed a silent crash in ip-watch.sh caused by bash set -e and arithmetic evaluation.
Audit snapshot of the four-container Immich stack and its NPM proxy to photos.skyhouse.dev.
Audit snapshot of three standalone containers: LAN-only password vault, cross-device file drop, and shared photo album.
Audit snapshot of ip-watch.sh, dynadot-update.py, and the ddns-go container — including a known set -e bug that stalls state-file writes.
Audit snapshot of Netdata, cpu_watchdog, telegram_notify, and fail2ban — plus the note that Uptime Kuma is currently down.
Audit snapshot of the eight-drive array, the three nightly backup scripts, robust_rsync.sh, and the no-offsite-backup gap.
New scripts to update Plex on demand and send a Telegram alert when a new version is available.
Adding a graphical interface for Docker container orchestration to the Admin Hub.
Relocating administrative tools to a password-protected environment.
Streamlining Plex, security hardening, and a fresh coat of paint for the portal.