Cloud Fortress Control Center
Traced the multi-day crash storm to two independent USB-storage faults: the JMicron JMS583/UAS DAS flooding the USB bus, and a dying WD My Passport (sde) that hung the kernel every 5 min. Pulled the My Passport, then fully recovered the DAS by forcing the JMS583 out of UAS (usb-storage.quirks=152d:0583:u) and reseating loose drives — all four DAS drives enumerated clean and SMART-passed (no bad drive was ever inside it). plex1 + plex3 remounted, the complete media stack + immich restored (0 restarts, HTTP 200). Also revived smartd and fixed Home Assistant's HubZ stick (cp210x). New rule: the DAS is a post-boot hotplug (never attached at cold boot, where a POST crash is unrecoverable) — managed by das-up.sh/das-down.sh.
A new webapp for customizing and exporting print-ready daily calendars was stood up at photocalendar.skyhouse.dev. Added a corresponding tile to the main server index with a matching style and calendar icon.
After Dynadot's ns1 served the dead IP for 3+ hours and couldn't be forced, we flipped all three domains' nameservers to Cloudflare (via the Dynadot API), set every record to DNS-only (grey cloud) to dodge the proxy 100 MB/video limits, and rebuilt room101's email on Cloudflare Email Routing (verified working). Replaced the whole Dynadot-era script pile with one token-free notifier dns-monitor.sh (🌐 IP-changed / ✅ resynced / ⚠️ alerts). Last manual step for full self-heal: flatten each apex to a CNAME → cw526dc.glddns.com. Accelerating ISP IP churn is now background noise.
An IP change left Dynadot's ns1 serving the dead old IP for over an hour while everything else was correct — because the ANAME apex never changes, ns1 had no reload trigger, and set_dns2 can't even write an ANAME so the "force re-sync" fix was impossible. Reverted skyhouse.dev + botaa.org apexes to a literal A (room101 blocked by Email Settings), wired ip-watch.sh to actually push the new IP, added an independent dns-watchdog.sh cron that detects ns1/ns2 drift and re-asserts, hardened public-IP detection, and made the verifier re-assert while it waits. Honest limit: Dynadot's pools still flap; Cloudflare remains the durable fix.
New Portainer stack home-assistant runs Home Assistant (:8123) and Music Assistant (:8095), both on network_mode: host so mDNS/UPnP discovery works — which meant pointing the ha./ma.skyhouse.dev proxy hosts at the host LAN IP (not a container name), enabling WebSockets, and pasting a hardened location block for MA's auth handshake. HA needed trusted_proxies + use_x_forwarded_for to stop 400-ing the proxy, and its hardcoded homeassistant: YAML block deleted to hand timezone/URL config back to the UI. Closed out by passing the HubZ Zigbee/Z-Wave stick through to HA and onboarding a first IKEA light via the Philips Hue hub.
A reboot showed Dependency failed for media-plex3_backup.mount — but that boot-time line is cosmetic (nofail keeps it from blocking boot). The real cost was a 90s shutdown stall when the cheap USB→SATA bridge wedged on unmount (Unmounting timed out. Terminating.). Since one TimeoutUSec governs both mount and unmount on a .mount unit, we extended the 2026-06-01 plex1_backup fix to every media mount: a TimeoutSec=10 drop-in on plex2, plex3, plex3_backup, plex1, and scratch, applied live via daemon-reload (no remount). Now no wedged drive can stall a reboot more than ~10s, and it carries forward to the incoming DAS whether it lands on USB or eSATA.
Made the stack's compose current and redeploy-safe. The deployed v7 already matched reality; the only drift was bitmagnet — a leftover from an older revision, crash-looping 594× against a missing postgres and unused by Prowlarr — so we docker rm -f'd it (no redeploy needed). Saved a maintained host-side copy of the stack at /home/plex/docs/media-downloads-stack.yml (it previously existed only inside Portainer's volume), dropping the obsolete version: key and normalising a volume; validated with docker compose config. Kept images on :latest to match Damien's redeploy-to-update workflow.
Built the insurance against yesterday's settings loss. First confirmed the *arr API key can't be pinned in the compose (tested: PROWLARR__APIKEY is ignored, the app only reads config.xml), so the real fix is backup + restore. New bin/arr-config-backup.sh runs nightly at 03:30: briefly stops the five containers for a consistent SQLite snapshot, tars sonarr/radarr/prowlarr/rdtclient(-movies) config to /media/plex3_backup/config-backups/ (~40 MB, keep 14), and refuses to run if the dest drive is unmounted. Paired bin/arr-config-restore.sh lists snapshots and restores in one command, saving a pre-restore snapshot first so it's reversible. Verified end-to-end.
A Docker error left the *arr stack a mess of mismatches: TV rdt-client was set to remove torrents on finish and reported the Windows-default C:\Downloads path; remote path mappings were keyed to the wrong host; Sonarr had self-blocklisted 9 releases from days of failed imports; Radarr was reaching rdt-client via flaky host-hairpin; and the oldest synced indexers still carried Prowlarr's old API key (regenerated by the rebuild) → 401s. Fixed each via the apps' APIs (rdt-client via stop/edit/start), added Prowlarr download clients, repaired a dead collection root folder, and resynced indexers. All three apps now report clean health and test green.
Found smartd silently dead since boot (stale Corsair line) — rebuilt /etc/smartd.conf fully by-id (all 7 drives by serial; plex3_backup as -d removable so it survives unplugs). Switched netdata's smartctl collector from a fragile /dev/sdX exclusion to monitor-all. Recovered the swapped-in plex3_backup with a software "virtual replug" of the USB toaster (USB-SATA bridges don't hot-swap), e2fsck'd plex3 (errors were transient bus glitches, not corruption), moved the 32 GB secondary swap onto the scratch SSD, and tidied the retired plex2_backup fstab entry — all live, by-id, no reboot.
Fresh SATA cables fixed the morning's flapping — plex1 is stable (zero link resets) and the 128 GB Kingston is back. Confirmed the retired plex2_backup WD is genuinely dead (5,483 offline-uncorrectable sectors, even through the toaster). Built the Kingston into a 117 GB scratch disk at /media/scratch, moved Plex's transcode temp onto it (downloads deliberately left on the media drives for hardlink locality), and reverted the protective measures: Plex unmasked, plex1 mirror re-enabled.
plex1's internal-SATA link is flapping again (drive SMART-pristine; failures are link-layer) and its ext4 wedged mid-I/O. Masked Plex so it stops retrying the indexer, and paused the nightly plex1→plex1_backup rsync — both reversible — until a physical cable/port/power fix or the 6/8 DAS migration. The unhealthy rdtclient-movies container is downstream of this, not a config bug.
Replaced the 12-year-old Corsair boot SSD with a 1TB SSD via a quiesced live ddrescue clone (0 errors), grew root online to 938 GB, re-seated plex1, and retired both the Corsair and the dying WD plex2_backup. The 128 GB scratch SSD dropped on a loose cable — pending re-seat.
Plan for the incoming Mediasonic 8-bay DAS: shuck the easystore externals (frees USB ports), consolidate the hot "toaster" drives, run SnapRAID+mergerfs parity (parity ≥ largest data drive, ideally on internal SATA), LUKS bays for a neighbor's offsite backup, and a USB-3.x-powered-hub port layout. Includes the 3.3V-pin shuck gotcha + burn-in checklist.
Pointed the skyhouse.dev + botaa.org apexes at the router's built-in DDNS name (static ANAME → cw526dc.glddns.com, like room101.com) and retired ddns-go — the static records eliminate Dynadot's ns1 split-brain entirely (verified: ns1 went 8/8 correct instantly).
Ready-to-execute plan (not yet done) to move DNS hosting to Cloudflare's anycast nameservers if Dynadot keeps split-braining — with full record inventory, ddns-go reconfig, and rollback. Also documents botaa.org joining the ddns-go pipeline and room101.com's self-tracking via the router's glddns DDNS.
Made an IP change hands-off: TTL 600→120, forced a Dynadot nameserver re-sync, stopped the server using stale Google DNS, pointed all subdomains at the LAN IP in /etc/hosts, and added a notifier that confirms via Telegram when the site is verified publicly back up (or flags manual steps).
Prep for replacing the ancient Corsair boot SSD: a full config/DB safety backup to plex3, a disk-layout snapshot, and a step-by-step offline-clone runbook (ddrescue, the duplicate-UUID trap, grow root, EFI) — including the BENFEI=flaky-JMicron enclosure caveat.
Stopped netdata from SMART-probing the flaky USB bridge (the real reboot stall); fixed the plex1 backup, which was blocked by immich's root-owned library, via a narrow root rsync wrapper; and consolidated DNS onto ddns-go after finding the custom updater was a racing no-op.
Killed the ~30s of red [FAILED] on every reboot: replaced smartd's blanket DEVICESCAN (which hung on a JMicron USB bridge) with explicit by-id disk lines, and added systemd dependencies so docker/Plex stop before their media disks unmount.
Freed ~19 GB on root (95%→77%), added a read-only hardware-diagnostics sudo drop-in for the coming drive work, and fixed the weeks-failing plex1 backup — the target was accidentally NTFS, not a dying drive; reformatted to ext4 and hardened the script.
Purged a self-sustaining download loop from rdt-client + TorBox, and fixed the queue-cleanup cron so dead grabs get blocklisted instead of endlessly re-grabbed.
Fixed root cause of "downloads complete but never import": RDTClient was deleting completed torrent records before Sonarr could poll them. Added weekly queue cleanup cron for edge cases.
Deployed a second RDTClient instance so Sonarr uses a strict cache-only bouncer while Radarr gets an open door for live torrent slots — each downloading directly to its own media drive for instant atomic moves.
Recursively fixed ownership and permissions for the TV and Movie libraries to resolve Sonarr import failures (UnauthorizedAccessException).
Fixed a volume-mount crashloop and redeployed, but the tool targets the Immich 1.x API and is incompatible with our 2.7.5 — retired it in favor of Immich's built-in duplicate detection.
Pinned Portainer to 2.42.0, deployed sonarr/radarr/rdtclient via the API to bypass the phantom-Network-Error UI bug, and documented the surviving cosmetic quirks.
Disabled nginx response buffering for the Portainer host so streamed stack-deploy progress reaches the browser instead of triggering a 499 client abort.
Added a permanent read-only sudo allowlist and session-scoped credential lease so Claude can run approved sudo commands without copy-paste friction.
Added a nightly check to alert when a new Next.js update is available for the music-discovery app.
Removed retired proxy hosts and orphaned TLS certificates via the NPM web UI.
Removed obsolete botaa Docker networks and retained btst_default.
Disabled OpenVPN, restricted NPM admin port 81 to LAN, and archived the obsolete plexupdate script.
Archived 19 stale files and one-off scripts from /home/plex and the web root.
Fixed a silent crash in ip-watch.sh caused by bash set -e and arithmetic evaluation.
Audit snapshot of the four-container Immich stack and its NPM proxy to photos.skyhouse.dev.
Audit snapshot of three standalone containers: LAN-only password vault, cross-device file drop, and shared photo album.
Audit snapshot of ip-watch.sh, dynadot-update.py, and the ddns-go container — including a known set -e bug that stalls state-file writes.
Audit snapshot of Netdata, cpu_watchdog, telegram_notify, and fail2ban — plus the note that Uptime Kuma is currently down.
Audit snapshot of the eight-drive array, the three nightly backup scripts, robust_rsync.sh, and the no-offsite-backup gap.
New scripts to update Plex on demand and send a Telegram alert when a new version is available.
Adding a graphical interface for Docker container orchestration to the Admin Hub.
Relocating administrative tools to a password-protected environment.
Streamlining Plex, security hardening, and a fresh coat of paint for the portal.