Reference card for skyhouse.dev
The "what is what / where is what" reference for this server. For "something is wrong,
what do I do" see the Troubleshooting Guide;
for full prose orientation see /home/plex/docs/server-context.md.
Verified 2026-05-20.
| Item | Value |
|---|---|
| Host / LAN IP | plex — 192.168.1.136 (static DHCP reservation) |
| SSH | port 2222 (not 22) — ssh -p 2222 plex@skyhouse.dev |
| Topology | ISP modem (bridge mode) → GL.iNet router (192.168.1.1) → server |
| Router | GL.iNet BE9300 (Flint 3) — Qualcomm IPQ5332, tri-band Wi-Fi 7, fw 4.9.0 (OpenWrt 23.05-SNAPSHOT). Cheat sheet said v4.8.4 until 2026-08-18. Root SSH from server: ssh -i ~/.ssh/router_ed25519 root@192.168.1.1. Full docs: /home/plex/docs/router-reference.md — read before debugging any Wi-Fi issue. Wi-Fi driver is Qualcomm proprietary: iw ... station dump returns NOTHING, use wlanconfig <if> list sta; iw ... survey dump does work. Port-forwards live in /etc/config/port_forward, not firewall. SSIDs: SH24 (2.4/ch1), SkyHouse+SkyHouseWork5 (5G/ch36/EHT160), SkyHouse6+SkyHouseWork6 (6G/ch5/EHT320). Band steering + MLO + SQM all off. Idles at 71°C against a 75°C fan trigger. [Journal: 2026-08-18 wifi-rate-collapse] |
| Router port-forwards | WAN 80, 443, 32400, 2222 → 192.168.1.136 (same ports) |
| Emergency hostname | cw526dc.glddns.com — GL.iNet DDNS, always tracks current WAN IP |
| DNS | Cloudflare anycast NS (luciane/remy.ns.cloudflare.com) for skyhouse.dev, botaa.org, room101.com (registered at Dynadot). All records DNS-only (grey cloud). Apex = flattened CNAME → cw526dc.glddns.com (router DDNS) on all three — genuinely done 2026-08-04, so IP changes self-heal with no scripts and no hardcoded origin IP; * wildcard → apex. (Before 2026-08-04 all three were static A records despite docs claiming otherwise — room101 served a dead IP for 16 days.) room101 email = Cloudflare Email Routing. dns-monitor.sh notifies only. [Journal: 2026-08-04 apex-cname-flattening] |
Red = forwarded to the internet. Green = LAN / localhost only.
| Port | Service | Scope |
|---|---|---|
| 80 / 443 | Nginx Proxy Manager — HTTP / HTTPS for every subdomain | Public |
| 81 | NPM admin UI — manages all proxy hosts & certs | LAN only (UFW-restricted) |
| 32400 | Plex Media Server (main) | Public (Plex remote access) |
| 32401 | Plex (local control) | localhost |
| 2222 | SSH / SFTP | Public |
| 9443 / 9000 | Portainer — the web UI that manages all Docker containers (9443 = HTTPS, preferred) | LAN only |
| 2283 | Immich (photos) | LAN; public via photos.skyhouse.dev |
| 3000 | Pairdrop (file sharing) | LAN; public via share.skyhouse.dev |
| 5001 | Sharedmoments | LAN; public via btst.skyhouse.dev |
| 8000 / 8001 | Music Discovery — client / API server | LAN; public via music.skyhouse.dev |
| 3001 | Uptime Kuma (status page) | LAN; public via status.skyhouse.dev |
| 19999 | Netdata (system monitoring) | LAN; public via monitor.skyhouse.dev |
| 8989 | Sonarr (TV PVR) | LAN only (external access not yet wired — see 2026-05-28 journal) |
| 7878 | Radarr (Movie PVR) | LAN only (external access not yet wired) |
| 8686 | Lidarr (Music PVR) | Public via lidarr.skyhouse.dev (basic auth, bypassed for LAN IPs) — 2026-07-17 journal |
| 6969 | Whisparr (V3/Eros, adult movie PVR) | Public via whisparr.skyhouse.dev (basic auth, bypassed for LAN IPs) — 2026-07-17 journal |
| 6500 | rdt-client (Real-Debrid client) | LAN only |
| 8123 | Home Assistant (host-networked) | LAN; public via ha.skyhouse.dev |
| 8095 | Music Assistant (host-networked) | LAN; public via ma.skyhouse.dev |
| 9876 | ddns-go UI — legacy, slated for retirement | LAN only |
| 53 | systemd-resolved (local DNS stub) | localhost |
| internal | Vaultwarden :80, immich Postgres :5432, immich Redis :6379, npm-db MariaDB :3306, npm-php :9000 | Docker network only (no host port) |
Authoritative hostname → backend, as configured in Nginx Proxy Manager.
| Hostname | Backend |
|---|---|
| skyhouse.dev | static site — /home/plex/www (legacy sites served as path subfolders, e.g. /randoma) |
| plex.skyhouse.dev | 192.168.1.136:32400 |
| photos.skyhouse.dev | 192.168.1.136:2283 (Immich) |
| music.skyhouse.dev | 192.168.1.136:8000 (/api → :8001) |
| vault.skyhouse.dev | vaultwarden:80 — LAN-allowlisted, not public |
| share.skyhouse.dev | 192.168.1.136:3000 (Pairdrop) |
| btst.skyhouse.dev | 192.168.1.136:5001 (Sharedmoments) |
| monitor.skyhouse.dev | 192.168.1.136:19999 (Netdata) |
| status.skyhouse.dev | 192.168.1.136:3001 (Uptime Kuma) |
| router.skyhouse.dev | https://192.168.1.1 (GL.iNet admin) |
| portainer.skyhouse.dev | https://192.168.1.136:9443 (Portainer) |
| npm.skyhouse.dev | http://127.0.0.1:81 (NPM admin โ LAN only) |
| sonarr.skyhouse.dev | 192.168.1.136:8989 (Sonarr) |
| radarr.skyhouse.dev | 192.168.1.136:7878 (Radarr) |
| rdtclient-tv.skyhouse.dev | 192.168.1.136:6500 (RDTClient TV — cache-only, for Sonarr) |
| rdtclient-movies.skyhouse.dev | 192.168.1.136:6501 (RDTClient Movies — open slots, for Radarr) |
| prowlarr.skyhouse.dev | 192.168.1.136:9696 (Prowlarr — indexer manager) |
| lidarr.skyhouse.dev | 192.168.1.136:8686 (Lidarr — Music PVR, library at /media/plex3/Music) |
| whisparr.skyhouse.dev | 192.168.1.136:6969 (Whisparr V3/Eros — adult movie PVR, library at /media/plex1) |
| ha.skyhouse.dev | 192.168.1.136:8123 (Home Assistant — host IP, not a container name; WebSockets on) |
| ma.skyhouse.dev | 192.168.1.136:8095 (Music Assistant — host IP; WebSockets on + hardened Advanced location block for its auth handshake) |
| room101.com | static — /home/plex/www/room101 |
| botaa.org / share.botaa.org / smut.botaa.org | static / Pairdrop / Plex (group connect URL) |
| Container | Purpose | Compose dir |
|---|---|---|
| npm-app-1 / npm-db-1 / npm-php-1 | Nginx Proxy Manager + MariaDB + PHP-FPM | /home/plex/npm |
| immich_server / _postgres / _machine_learning / _redis | Immich photo platform (4-container stack) | /home/plex/immich-app |
| music-discovery-client / -server | Next.js music app + API | /home/plex/music_discovery |
| uptime-kuma | Service uptime monitoring | /home/plex/uptime-kuma |
| vaultwarden | Password vault (Bitwarden-compatible) | standalone docker run |
| pairdrop | Cross-device file sharing | standalone |
| sharedmoments | Shared photo album | standalone (network btst_default) |
| portainer | Docker management web UI — pinned to portainer-ee:2.42.0 after May 27 :latest drift (journal) | standalone |
| prowlarr | Indexer manager for Sonarr & Radarr | portainer stack, port 9696 |
| lidarr | Music PVR — library at /media/plex3/Music, config at /home/lidarr/data (journal) | portainer stack, port 8686 |
| whisparr | Adult movie PVR, V3/Eros build (ghcr.io/hotio/whisparr:v3 — different env convention than the linuxserver images: UMASK/WEBUI_PORTS) — library at /media/plex1 (whole drive, root folders set inside its own UI, same as Radarr), config at /home/whisparr/data (journal) | portainer stack, port 6969 |
| sonarr | TV PVR — library at /media/plex2/TV, config at /home/sonarr/data. After the split-rdtclient migration, root folder inside container = /data/TV (journal) | portainer stack, port 8989 |
| radarr | needs /media/plex1 (hotplug DAS); restart=no — started via das-up.sh after attaching the DAS. Movie PVR — library at /media/plex1/Movies, config at /home/radarr/data. After split, root folder = /data/Movies; download client reaches rdt-client by Docker service name on the internal port (rdtclient-movies:6500, not host-IP:6501) (journal, rewiring, plex1 offline) | portainer stack, port 7878 |
| rdtclient | RDTClient TV (for Sonarr; intended cache-only but not actually gated — see 2026-05-29 journal) — downloads to /media/plex2/torbox_downloads, db at /home/rdtclient/db (journal) | portainer stack, port 6500 |
| rdtclient-movies | needs /media/plex1 (hotplug DAS); restart=no — started via das-up.sh after attaching the DAS. RDTClient Movies (open slots, for Radarr) — downloads to /media/plex1/torbox_downloads, db at /home/plex/rdtclient-movies/db. Compose at /home/plex/rdtclient-movies/docker-compose.yml (journal, plex1 offline) | docker compose (outside Portainer), port 6501 |
| homeassistant | Home Assistant — network_mode: host, port 8123. Config at /opt/appdata/homeassistant/config. HubZ Zigbee/Z-Wave stick passed through via two /dev/serial/by-id paths → ttyUSB0/ttyUSB1. Also mounts /etc/localtime + /run/dbus. [journal] | portainer stack home-assistant, port 8123 |
| music-assistant | Music Assistant — network_mode: host, port 8095. Data at host /home/music-assistant → container /data; library /media/plex3/Music mounted ro. Caps: SYS_ADMIN, DAC_READ_SEARCH, apparmor:unconfined. [journal] | portainer stack home-assistant, port 8095 |
| ddns-go | Legacy DDNS updater — slated for retirement | standalone |
plexmediaserver · netdata · fail2ban · smartmontools · ssh (port 2222) · docker / containerd. (openvpn was disabled in the 2026-05 cleanup.) Note: plexmediaserver is temporarily masked as of 2026-06-04 (plex1 SATA link wedged); systemctl unmask plexmediaserver to restore. [journal]
| What | Path |
|---|---|
| Web root | /home/plex/www |
| Server journal & these docs | /home/plex/www/serverjournal/ |
| Plex config / library DB | /var/lib/plexmediaserver/Library/Application Support/Plex Media Server/ |
| Docker data root | /var/lib/docker (on the OS disk) |
| NPM config & certs | /home/plex/npm/data/, /home/plex/npm/letsencrypt/ |
| Maintenance scripts | /usr/local/bin/ and /home/plex/bin/ and /home/plex/*.sh |
| Logs | /var/log/plex-backup.log, ~/.cache/dns-monitor.log, ~/.cache/nextjs-check.log |
| Secrets (locations only) | /etc/telegram_notify.conf, ~/.messaging-keys, /etc/netdata/health_alarm_notify.conf, /home/plex/npm/data/.htpasswd |
| Orientation docs | /home/plex/docs/ — server-context.md, server-audit-2026-05.md, server-runbook.md, disk-cutover-guide.md |
| Audit archive (movable junk) | /home/plex/archive/2026-05-cleanup/ |
plex)โ Jobs below are shown logically. As of 2026-08-04 every watcher actually runs as flock -n /run/lock/<name>.lock timeout <N> <script> — before that none of the six had a lock or a timeout, so a wedged Docker daemon left instances stacking every 5 min. Backups were already guarded internally. telegram_notify.sh also had no curl timeout and never checked its exit code (a failed alert looked identical to a delivered one); it now logs every send to /var/log/telegram_notify.log. Crontab backups: ~/.cache/plex.crontab.bak-20260804, /root/root.crontab.bak-20260804. [Journal: 2026-08-04 watcher-hardening]
| When | Job | Purpose |
|---|---|---|
*/5 * * * * | cpu_watchdog.sh | High-CPU alerting |
*/5 * * * * | bin/dns-monitor.sh | Cloudflare-era DNS notifier (๐ IP-changed / โ resynced / โ ๏ธ); no DNS writes |
*/5 * * * * | bin/arr-pipeline-check.py | Media pipeline probe โ stalled items in the Sonarr/Radarr queues, and new downloads, โ Telegram. The only check that watches whether media actually moves. 2026-08-05: skips entirely while /run/lock/arr-stack-maintenance is set (the 03:30 config backup stops the stack on purpose), and needs 2 consecutive unreachable readings before reporting DOWN โ one blip is silent. 2026-08-21: moved */30โ*/5 and does its own alerting โ it sends โ/โ ๏ธ/โ
via telegram_notify.sh, announces downloads that just started, and pushes Kuma always up so that monitor is a dead-man's switch only. TorBox lookup is now lazy (only on a run that found a stall). [journal] [noise fix] [alert rewrite] |
5 * * * * | bin/rdtclient-cleanup.py --apply --include-arr-referenced --include-idle | rdt-client stale-record cleanup (scheduled 2026-08-20; script written 2026-08-04 and unwired until then). THE MISSING HALF of the Radarr cleanup below — the arrs rebuild their queues from the download client, so an arr-layer delete alone just reappears on the next sync with the same queue id. Seven Joy.Ride.2021 records resurrected that way for 9 days while the hourly Radarr cleanup logged DID NOT STICK every run. Runs at :05, twelve minutes ahead of :17, so the client layer is clean before the arr layer re-syncs. --include-arr-referenced is required, not optional: without it the layers deadlock protecting each other. Both real gates remain — record must be errored and ≥24h old. โ Now defers entirely while any arr is downloading or awaiting import, and caps at 20 deletions per run (2026-08-21): rdt-client's delete endpoint takes ~32 s per call, so an unbounded backfill spent half an hour hammering the service Radarr polls every minute for completed-download handling. Blind spot CLOSED 2026-08-21 by --include-idle, which also sweeps the two shapes it used to ignore — records rdt-client reports Finished, and records stalled with no seeders — once older than 3 days and absent from every arr queue in a healthy state. That last qualifier is the point: a stalled record is in the arr queue because the client holds it, so plain presence cannot count as proof of life. [journal] [2026-08-21] |
17 * * * * | bin/radarr-queue-cleanup.py --apply | Radarr queue self-heal (2026-08-05) โ the counterpart to Sonarr's */5 cleanup, which had existed since 2026-05-29 while Radarr had none, so Radarr's monitor sat DOWN for days. Hourly (not */5): it is the only ops script that deletes. Offset to :17 so it never lands on the probe (:00/:30) or the 03:30 backup. [journal] |
8,23,38,53 * * * * | bin/torbox-slot-guard.py --apply | TorBox slot guard (2026-08-21). A magnet with a dead swarm never fails on TorBox — it sits in checking with size=-1 and eta=8640000 (100 days), holding one of the three paid slots for ever. With slots full TorBox queues new adds; rdt-client reads “queued” as a failed add, retries, gets DIFF_ISSUE: Download already queued and parks a nameless dead record — so one dead magnet poisons every grab behind it. Evicts only when active + unfinished + 0 progress + 0 seeds + 0 peers + >30 min, and separately flushes orphaned add-queue entries. Four times an hour because a jam costs every download queued behind it. [journal] |
12,42 * * * * | bin/rdtclient-stuck-alert.py | Terminal-failure alert — the only media job that texts you (2026-08-21). Fires when retryCount ≥ TorrentRetryAttempts (rdt-client has stopped trying) and either TorBox still holds the release cached (recoverable by hand; nothing else will ever retry it) or the error is unrecognised. Everything the stack self-heals stays silent — the first draft fired on 17 records of which 16 were slot-jam residue. Runs inside the 180-min DeleteOnError window so there is time to act. Dedup state: ~/.cache/rdtclient-stuck-alert.json (--reset to clear). [journal] |
45 5 * * * | bin/media-stack-update.sh | Media stack auto-update (2026-08-21). rdt-client has NO self-update — the banner only compares version strings against GitHub, and :latest is resolved at pull time, so a container serves its build for ever until re-pulled (ours ran v2.0.140 from 2026-07-17 while v2.0.142 shipped). Runs Watchtower --run-once --cleanup over the eight media-downloads containers by explicit name — immich, home-assistant, vaultwarden, npm and portainer are deliberately out of scope. Why a wrapper and not Watchtower's own scheduler: Docker creates a missing bind-mount source as an empty directory, so recreating radarr with the DAS detached would point it at a library of zero films and let it reconcile. The wrapper proves /media/plex1/Movies and /media/plex2/TV are mounted and hold >10 entries (mounted alone is not enough), and refuses while /run/lock/arr-stack-maintenance is held by the 03:30 backup. Silent unless something updated or was refused. Log: ~/.cache/media-stack-update.log. [journal] |
40 4 * * * | bin/torbox-cleanup.py --apply | TorBox retention backstop (scheduled 2026-08-21; written 2026-08-04 and unwired until then — the same gap rdtclient-cleanup sat in). Deletes finished TorBox torrents >6 days old that no arr is tracking. With FinishedAction now removing on import this should usually find nothing. [journal] |
*/15 * * * * | bin/dns-apex-check.sh | Apex correctness โ Kuma push monitors (2026-08-04). Asserts each apex resolves via an external resolver to the same IP as cw526dc.glddns.com. Local resolution is useless here โ /etc/hosts split-horizon hid a 16-day room101 outage. [journal] |
0 1 * * * | backup-plex3.sh | โ DISABLED 2026-08-22 (RAM fault: no large writes to backup drives until resolved; re-enable after). [Journal: 2026-08-22 backup-cron] rsync plex3 → plex3_backup. RE-ENABLED + REWRITTEN 2026-07-30 (was commented out; old version passed a stale device node and would have mounted the My Passport as the target). Moved to 01:00 so it can't overlap the 03:30 arr-config job. [journal] |
| disabled | backup-plex2.sh | โ No target drive exists — /media/plex2 (~11 TB) has NO COPY ANYWHERE. Script neutralised 2026-07-30 (it refuses to run) because it held a stale /dev/sdb reference, which is now the scratch SSD. Needs a ≥12 TB target or SnapRAID parity. [journal] |
30 4 * * * | backup-plex1.sh | โ DISABLED 2026-08-22 (RAM fault; re-enable after, and add --max-delete=1000 as a circuit breaker). [Journal: 2026-08-22 backup-cron] rsync plex1 → plex1_backup. RE-ENABLED 2026-07-30 — it had been commented out since June despite a crontab note claiming it was re-enabled on 2026-06-04, so plex1_backup held 2.3 TB against plex1's 12 TB. Script itself was already sound. First convergence takes ~3 nights (~10 TB at 6h/night). [journal] |
30 3 * * * | bin/arr-config-backup.sh | *arr + rdt-client config snapshot (settings/API-keys/DBs) → /media/plex3_backup/config-backups/, keep 14. Stops the stack ~15s for a consistent SQLite copy. Restore with bin/arr-config-restore.sh. 2026-08-05: writes /run/lock/arr-stack-maintenance around the stop so the pipeline probe and both queue cleanups treat the outage as planned. The flag carries a 15-min deadline โ a killed backup can never mute monitoring for good. [journal] [flag] |
0 4 * * * | plex-check | Plex version alert |
0 5 * * * | nextjs-check | Next.js version alert |
โ 2026-08-21 — /media/plex1 is mounted clean with errors and needs an offline e2fsck. 19 corrupt inodes + 31 bad block-bitmap groups, all confined to Immich thumbnails (regenerable); no originals affected — verified by SHA-1 against asset.checksum and by confirming zero originals were written during the corruption window. It has run silently for hours because its Errors behavior is Continue, so it never remounts read-only — check with sudo dumpe2fs -h /dev/<plex1> | grep -E "Filesystem state|FS Error count", not by waiting for a failure. Never fsck'd since the fs was created in 2022. The fsck is blocked on the RAM fault (ยง11) — rewriting metadata across 12.7 TB through bad memory is how a thumbnail problem becomes a catastrophe. Expect 15–45 min, not hours: only 2.2 M of 427 M inodes are in use. Add whisparr to das-down.sh first — it holds a plex1 bind mount and will block the unmount. [Journal: 2026-08-21]
| Device | Mount | Role |
|---|---|---|
1 TB SSD "SSV8" (AA0โฆ2477) | / + /boot/efi | OS disk — replaced the ancient Corsair via live ddrescue clone 2026-06-03; root ext4 grown to 938 GB (UUID 879bcc42-โฆ). [journal] |
| sdc1 (14 TB) | /media/plex1 | Media — ~92% full |
| sdh1 (14 TB, ext4) | /media/plex1_backup | Backup of plex1 (USB ext.) — reformatted NTFS→ext4 2026-05-31, UUID 70d95e9e-โฆ. [journal] |
| sdg1 (14 TB) | /media/plex2 | Media |
| (removed) | Dead drive retired — WD 9RHHSV3L (5,483 offline-uncorrectable, confirmed 2026-06-04 even via toaster) physically removed; backup-plex2.sh cron stays disabled (no target). [journal] | |
| sdd1 (12 TB) | /media/plex3 | Media. (No longer holds swapfile_extra — moved to the scratch SSD 2026-06-04.) |
| sde1 (10 TB) | /media/plex3_backup | Backup of plex3 |
Kingston 128 GB SSD (203S10Q4T73Z) | /media/scratch | Scratch disk (internal SATA, ext4 117 GB, UUID 46b67b3d-โฆ) — hosts Plex transcode temp (/media/scratch/transcode). Disposable: ~9.8 yr-old SSD, nothing irreplaceable here. nofail. [journal] |
โ Device letters shuffle — identify drives by serial/UUID, never /dev/sdX. The sdc/sdd/sde/sdg/sdh labels in the table above are historical and have already moved twice. As of 2026-07-30 the DAS (JMS567 bridge) runs at SuperSpeed with 5 populated bays — plex1 (ST14000NM001G/ZL2BG3W8), plex3 (WD120EDBZ/5QGWXSKF), plex3_backup (ST10000DM005/WP001CYM), plus an unassigned Toshiba 6 TB (HDWE160) and WD 2 TB (WD20EARS). The two WD easystores (plex2, plex1_backup) are separate USB externals. Health check: ~/das-check.sh (link speeds, max_sectors_kb, UAS binding, I/O pressure); attach/detach helpers remain ~/das-up.sh / ~/das-down.sh (das-down FIRST, then unplug). [Journal: 2026-07-30]
Swap: /swapfile 4 GB (OS disk) + /media/scratch/swapfile_extra 32 GB (scratch SSD, sw,nofail) — 35 GB total. Moved off plex3 → scratch 2026-06-04. Media drives mount by UUID in /etc/fstab with nofail. [Journal: 2026-06-04]
โ 2026-08-06: the scratch swap line had been commented out in /etc/fstab — the box was silently running on 4 GB, not 35 GB. Restored (backup /etc/fstab.bak-20260806-swap). Post-reboot check: swapon --show must list both files; if only /swapfile appears, the scratch entry has been disabled again. Note neither entry sets pri=, so after a reboot the root SSD is the preferred swap target and scratch is overflow only. [Journal: 2026-08-06]
SMART monitoring (rebuilt by-id 2026-06-04): /etc/smartd.conf lists every present drive explicitly by serial (/dev/disk/by-id/ata-<model>_<serial> -d sat, no DEVICESCAN) — boot SSD, scratch SSD, plex1, plex2, plex3, plex1_backup. The detachable toaster drive plex3_backup uses -d removable (auto-detects as SAT when present, and lets smartd survive it being unplugged). โ smartd EXITS (status 16) if a non-removable listed device is absent — so when you retire or move a drive you MUST drop/adjust its by-id line, or smartd dies silently at next boot (exactly what a stale Corsair line did after the 6/3 boot swap). netdata's smartctl collector (/etc/netdata/go.d/smartctl.conf) now monitors everything present (device_selector: '* *', no /dev/sdX exclusion) — the JMicron bridge that once hung it is gone, and netdata only scans present devices, so it's shuffle-proof. Backups: /etc/smartd.conf.bak-20260604, /etc/netdata/go.d/smartctl.conf.bak-20260604. [Journal: 2026-06-04]
| Script | Does |
|---|---|
warmclaude <time> [-h lead] | Claude session preheater โ schedules a tiny Haiku ping (default 2h) before a planned work session so the 5h token block resets mid-session. -n dry-run, -l list. Transient systemd user timers (survive logout โ linger enabled 2026-08-05); log ~/.claude/warmclaude.log. Kit + /span-sessions skill: ~/claude-span-kit/. Published publicly (unlisted, no auth, no secrets) at https://skyhouse.dev/span-kit/ โ install anywhere with curl -fsSL https://skyhouse.dev/span-kit/install.sh | bash; re-publish after edits with ~/claude-span-kit/publish.sh (bump VERSION first) [journal] (2026-08-05) |
sudo plex-update | Download + install latest Plex Media Server |
plex-check / nextjs-check | Version checks — Telegram alert if behind |
cpu_watchdog.sh | Alerts on sustained high CPU by non-allowlisted procs |
telegram_notify.sh | Shared Telegram-send helper used by the others. 2026-08-09: HTML-escapes the body by default โ it sends parse_mode=HTML, and a single <, > or & in arbitrary text makes Telegram reject the WHOLE message with a 400. That silently killed the rdt-client pre-wedge alert for days, because its own remediation snippet contained <id>. Callers composing deliberate markup set TELEGRAM_HTML=1 (only arr-remediation-ledger.py does). Also caps at 4000 chars and logs the API's error description to /var/log/telegram_notify.log. 2026-08-21: house style. The ๐ฅ๏ธ plex: header is gone (one server โ it never carried information and pushed the glyph off line one). Every message is now glyph + scope + object, then what is happening, then one โ next step; the glyph answers only does this need you? [journal] [house style] |
bin/arr-pipeline-check.py | Media pipeline probe (cron */5) โ watches the FLOW: items that enter the arr queues and never leave. Classifies errored/no-start/no-progress/import-blocked, names the titles, reports only (never touches the queue). Since 2026-08-21 it also alerts directly (โ = you act, โ ๏ธ = being handled, โ
= cleared or a download starting) and detects new downloads from its own state file โ a downloadId with no prior record IS a grab that just began, which is why this needed no webhook. Supports --dry-run. Keys: ~/.arr-keys.env (2026-08-04, rewritten 2026-08-21) |
bin/ops-capture.py | Incident capture โ ops-capture.py <dns|backup|media|docker|disk|system> --reason "..." writes a bundle to ~/ops/incidents/ with output + runbook. Not web-served; bounded commands; refuses to run under 500 MB free (2026-08-04) |
ops/runbooks/*.md | Anchored runbooks (media#stalled-queue, dns#apex-mismatch, backups#job-silent) โ referenced in alert text, copied into capture bundles (2026-08-04) |
bin/dns-apex-check.sh | Apex-vs-DDNS correctness check feeding Kuma push monitors; external resolver, dual-resolver fallback (2026-08-04) |
bin/lib-kuma.sh | Sourced helper: kuma_push <TOKEN_VAR> <up|down> <msg>. Returns 0 on every failure path โ must never fail the job it monitors (2026-08-04) |
ops/provision-kuma.py | Idempotent Kuma monitor provisioning from code (--dry-run default; venv at ops/venv; creds ~/.kuma.env). Emits ops/push-tokens.env. 2026-08-05: now CONVERGES existing monitors (it used to treat "exists" as "correct", so a wrong setting could never be pushed out) and computes grace as heartbeat + retry ร retries. โ Kuma applies retryInterval only while retrying โ with maxretries=0 a declared grace period does not exist, which is what made every push monitor flap. [journal] |
bin/immich-flickr-dedupe.py | Flickr size-variant de-duplication (2026-08-21, dry-run by default; --apply to retire). โ Never filter on the _o suffix — it lies. A 1024px _b render re-uploaded to Flickr later earns a NEW photo id and exports as that upload's _o, so a 680x1024 copy and the true 4288x2848 original both end _o. Groups on the first 8+ digit Flickr id, ranks on pixels then bytes. Albums are the majority case: 426 of 783 retirable assets were in albums — the keeper joins the loser's albums BEFORE the loser is retired. Soft-delete to the 30-day trash; full CSV audit in ~/.cache/. First run: 783 retired, 764 album additions, 0 failures. Only sees Flickr-vs-Flickr; cross-source copies need bin/immich-nearmiss-dupes.py. [journal] |
bin/arr-cached-search.py | Cached-first release picker (2026-08-10, report-only; --apply to grab). TorBox's checkcached answers for arbitrary infohashes, so "is it instantly available?" is known BEFORE grabbing. Seeders are the wrong signal on debrid โ a 1-seeder cached release beats a 30-seeder uncached one. Fails closed if the cache check errors. [journal] |
bin/arr-upgrade-scan.py | Library scanner (2026-08-10, report-only). Default: files a cached smaller copy could replace. --upgrade: sub-1080p titles worth improving (8 for ~3 GB, some negative cost). No --apply by design โ profile Any has upgradeAllowed=false and a smaller file is not an "upgrade", so replacement is delete-then-repick (--script emits the worklist). [journal] |
bin/tv-duplicate-audit.py | TV duplicate audit (2026-08-05, read-only, on demand). Plex has duplicate detection for movie libraries only, so a duplicated episode is invisible in the UI. Finds SPLIT SERIES (one show under two folders โ Bad Sisters vs Bad Sisters (2022)), DUPLICATE EPISODES, and folders Sonarr cannot see. Marks which copy Sonarr TRACKS โ deleting the tracked one leaves the episode missing and a monitored series re-grabs it. Ignores Featurettes/Deleted Scenes (they carry episode codes; counting them reported 737 instead of 400). Baseline report: /home/plex/docs/tv-duplicate-audit-20260805.txt. [journal] |
bin/arr-remediation-ledger.py | Remediation bookkeeping (2026-08-05). Both queue cleanups record every fix against the episode/movie (not the release โ blocklisting swaps releases, so a release-keyed count never sees the loop). Routine self-healing is silent; it texts only when the same title needs remediating 3 rounds in 7 days (โฅ4h apart, so one multi-release pass counts once), or when the cleanup call itself fails. arr-remediation-ledger.py report for the last 7 days. [journal] |
bin/wifi-device-stress.shbin/cast-network-test.pybin/run-arch-ab2.shbin/ma-cli.py | Wi-Fi / multi-room audio diagnostics (2026-08-18, on demand). wifi-device-stress.sh floods each wireless client in turn and reports its loss plus the collateral latency inflicted on others — use it to find devices that poison the medium (needs sudo). The cast scripts A/B individual streams against Cast groups using ~/.venv-cast (pychromecast); run-arch-ab2.sh is the one to use (interleaved + repeated, since the idle baseline drifts several ms). Full writeup: /home/plex/docs/multiroom-audio-reference.md. [journal] |
bin/router-snapshot.sh | Router health snapshot (2026-08-18, read-only, no cron — run on demand). SSHes to the BE9300 with ~/.ssh/router_ed25519 and dumps identity, thermals, WAN, per-radio airtime survey, per-station TX-rate vs RSSI, wireless/steering/MLO/SQM config, port-forwards and recent Wi-Fi log events to /home/plex/docs/router-snapshots/<date>.txt. First tool to run for any Wi-Fi complaint. [journal] |
bin/dns-monitor.sh | Cloudflare-era notifier: Telegram on IP-change + resync confirm; no DNS writes (2026-06-07). 2026-08-21: on the shared sender, body in main() (parse guard), a retraction arm so a warning raised outside the resync window can be closed (it could dangle for ever before), and a consecutive-failure counter that reports a WAN blackout after 30 min — a 35-minute outage went entirely unreported that day. [journal] Largely redundant since 2026-08-04 (apexes self-heal); its โ ๏ธ never names the failing domain and never repeats โ slated for replacement by an external check. A scoped Cloudflare API token now exists at ~/.cloudflare.env (600) for operator-initiated work only โ no cron or watchdog may use it. |
bin/lib-pubip.sh | Hardened multi-source public-IP fetch (used by dns-monitor) |
bin/install-script.sh | โ USE THIS TO EDIT ANY CRON SCRIPT. Writes a temp file beside the target, syntax-gates it (bash -n / py_compile + shebang + empty-file guard), inherits the mode, then rename(2)s it into place. Bash tracks a byte offset through a script rather than slurping it, so rewriting a live one in place lets a running shell resume at a stale offset and execute a mix of old and new code — which sent a false DNS alarm on 2026-08-21. Rename is atomic: a running shell keeps its old inode. install-script.sh <src> <dest> [journal] |
bin/lib-rsync-report.sh | Turns rsync output into one actionable line โ how many files failed and which failed first. Sourced by backup-plex{1,3}.sh, which used to send tail -n 50 of the log as the alert body (2026-08-21) |
| Dynadot-era (RETIRED, rollback-only): | Moved out of ~/bin on 2026-08-21 → ~/bin/retired/: ip-watch.sh, ip-recovery-verify.sh, dns-watchdog.sh, cf-migration-verify.sh, dns-push.sh, dynadot-update.py. None scheduled or sourced since 2026-06-07; they carried 8 dead Telegram templates. Restore steps in ~/bin/retired/README.md. Only useful if reverting NS to Dynadot. [journal] |
backup-plex{1,3}.sh | Nightly media mirroring. Both now run their rsync as root via fixed-arg wrappers (/usr/local/sbin/backup-plex{1,3}-rsync.sh, sudo NOPASSWD) so they can read root-owned trees (immich on plex1; pre-swap-backup-2026-06-02 on plex3). Both guard on mountpoint for both endpoints and never mount by device node. plex3 excludes config-backups so the mirror can't delete the arr snapshots. robust_rsync.sh is no longer used by any enabled job — it auto-mounts a device node passed as an argument, which is what made the old scripts dangerous. [journal] |
bin/pre-swap-config-backup.sh | One-shot (run manually) pre-boot-disk-swap safety backup: DB dumps (immich PG, NPM MariaDB, vaultwarden) + all configs/certs/compose//etc → /media/plex3/pre-swap-backup-<date>/. See swap runbook |
bin/arr-config-backup.sh | Nightly (03:30) consistent snapshot of the media-download stack's config — sonarr/radarr/prowlarr/rdtclient(-movies) settings, API keys, and SQLite DBs → /media/plex3_backup/config-backups/arr-config-<ts>.tar.gz (~40 MB, keep 14). Briefly stops the stack so WAL is checkpointed; refuses to run if the dest drive isn't mounted. [journal] |
bin/arr-config-restore.sh | Restore the above after a settings loss. arr-config-restore.sh (no args) lists snapshots; arr-config-restore.sh latest or โฆ <file> restores (prompts y/N). Saves a pre-restore-<ts>.tar.gz of current state first, so the restore is itself reversible. Then in Prowlarr: Settings → Apps → Sync App Indexers if indexers look off. |
lock_ssh.sh / unlock_ssh.sh | Disable / enable SSH password auth (key-only toggle) |
Updating apps, generally (2026-07-17): two different procedures depending on where the app lives.
repo.plex.tv/deb, see Quirks). Either sudo apt update && sudo apt upgrade plexmediaserver, or sudo plex-update (hits Plex's own download API, checksum-verifies, then dpkg -is — same effect, doesn't need apt)..deb from the browser (needs the Plex Pass login), then sudo dpkg -i /path/to/plexmediaserver_<version>_amd64.deb. No script for this side — it's manual by design on Plex's end.media-downloads Portainer-stack apps (prowlarr, sonarr, radarr, rdtclient, rdtclient-movies, lidarr, whisparr): all track :latest, so there's no version to bump in the compose file itself. To update: (if structure is changing) edit /home/plex/docs/media-downloads-stack.yml first, then in Portainer go to Stacks → media-downloads → Editor, paste that file's contents, tick "Re-pull image", and click Update the stack. Skipping the re-pull tick just redeploys with whatever image is already cached locally — no version change. This is also how new services (like lidarr/whisparr) get added to the stack. [journal]Copy-paste to confirm the server is healthy — e.g. after a reboot or the disk swap. Each line prints OK / a status when good.
systemctl is-active plexmediaserver netdata fail2ban ssh smartmontools
for c in npm-app-1 immich_server music-discovery-client music-discovery-server \
vaultwarden pairdrop sharedmoments portainer uptime-kuma ddns-go; do
printf '%-26s ' "$c"; docker inspect -f '{{.State.Status}}' "$c" 2>/dev/null || echo MISSING
done
curl -sf http://localhost:2283/api/server/ping # Immich -> {"res":"pong"}
for m in / /media/plex1 /media/plex2 /media/plex3 /media/scratch \
/media/plex1_backup /media/plex3_backup; do # plex2_backup drive retired 6/4
printf '%-24s ' "$m"; mountpoint -q "$m" && echo OK || echo "NOT MOUNTED"
done
find /var/log/plex-backup.log -mtime -2 | grep -q . && echo "backups fresh"
find ~/.cache/dns-monitor.log -mtime -1 | grep -q . && echo "dns notifier fresh"
curl -s ifconfig.me ; echo # current public IP dig +short skyhouse.dev @1.1.1.1 # should match the line above
| Account | Detail |
|---|---|
| Dynadot | Registrar + DNS for skyhouse.dev, room101.com, botaa.org (the dsm-iii/dsm-3 domains exist; hosting retired) |
| GL.iNet router | 192.168.1.1 — firmware v4.8.4 |
| Telegram bot | @skyhouse_server_bot — all alert delivery. Credentials in the 3 secret files listed in §5; rotating means updating all three. |
| Plex.tv | account damienmjones |
memtester produced 6 failures and every one is a single-bit error on bit 43 or 44, both in byte lane 5, across four addresses and five unrelated test patterns. Random soft errors scatter across all 64 bits; a fault confined to two adjacent bits in one lane is a specific physical path (one DRAM chip or a pair of DQ lines). The RAM is non-ECC 2×32 GB G.Skill Ripjaws V (F4-3600C18-32GVK, SK Hynix dies, dual-rank) running its rated XMP profile, DDR4-3600 CL18 at 1.35 V — correctly applied, and in the conservative Gear2. The out-of-spec part is on the CPU side: Rocket Lake officially supports DDR4-3200, and four ranks at 3600 is the heaviest load an 11th-gen IMC can be given — so there are no EDAC counters and nothing will ever warn you. This one fault explains both storage symptoms that day: it corrupted plex1's metadata on the way to disk (persisted) and a Postgres page on the way back from disk (transient). Do not run the plex1 e2fsck or the Immich→plex3 migration until it is resolved. Confirm with memtest86+ 7.00, already installed (GRUB entry “Memory test”, /boot/memtest86+x64.efi) — the fault reproduces in under a minute, so ~12 min is decisive and names the DIMM slot. Then test whether dropping XMP to rated speed clears it before buying anything. [Journal: 2026-08-21]docker stop immich_postgres (clears shared_buffers) → sync; echo 3 | sudo tee /proc/sys/vm/drop_caches → restart → forced heap scan (enable_indexonlyscan/indexscan/bitmapscan=off). A documented “corrupt page” in Immich's asset_exif read back 334,997 rows with zero failures this way, and pg_dump finished in 2.6 s — the page had always been fine. The recommended repair (zero_damaged_pages + VACUUM FULL) would have blanked ~44 perfectly good rows to fix nothing. Conversely plex1's damage did survive the same cold re-read, which is exactly how the real fault was told apart from the phantom. [Journal: 2026-08-21]immich_* containers are restart: no by design (so a DAS-less boot doesn't crash-loop them against an empty mountpoint). Run ~/das-up.sh — it verifies plex1 by UUID before starting anything. Forgetting this leaves Immich down silently. [Journal: 2026-07-06]GRUB_TIMEOUT_STYLE=hidden / GRUB_TIMEOUT=1, which on this UEFI box made the menu effectively unreachable — you could not get to memtest86+ or UEFI Firmware Settings without perfect timing on Esc. Now menu / 10. GRUB_DEFAULT=0 unchanged, so doing nothing still boots Ubuntu normally, and the usb-storage.quirks cmdline survives. Revert with sudo sed -i 's/^GRUB_TIMEOUT_STYLE=menu/GRUB_TIMEOUT_STYLE=hidden/; s/^GRUB_TIMEOUT=10/GRUB_TIMEOUT=1/' /etc/default/grub && sudo update-grub. Backup: /etc/default/grub.bak-20260821-memtest. But visible ≠ steerable: the menu appears and counts down, yet GRUB receives no USB keystrokes on this box (tested with a wired Keychron K3 and a Logitech Unifying receiver, both directly on rear ports). Use sudo ~/bin/arm-memtest.sh instead — see the next entry. [Journal: 2026-08-21]dockerd hit 136% CPU driving 987 reads/sec on sda, and everything touching the root disk โ including browser downloads โ became slow and fragile while the media drives sat idle and fast. /etc/docker/daemon.json now sets max-size 50m / max-file 3, but log options apply at container CREATION, not start โ all 28 existing containers still report opts=map[] and must be recreated (compose up --force-recreate / Portainer stack redeploy) before they are actually protected. Triage: sudo du -sh /var/lib/docker/containers/*/ | sort -rh | head; truncate a runaway with sudo truncate -s 0 <path>-json.log (safe while running). Diagnostic tell: docker stats or docker logs timing out means the daemon is saturated, not that the container is busy. [Journal: 2026-08-03]SQLite Error 19: UNIQUE constraint failed: Downloads.TorrentId, Downloads.Path repeating in TorrentRunner.Tick() โ the whole download pipeline stops and the DB stops being written. Live DB is /home/rdtclient/data/rdtclient.db (not /home/rdtclient/db, which is a stale June copy). Find the culprit: the torrent with many Downloads rows while every other pending one has zero. Now monitored + auto-salvaged: ~/bin/rdtclient-watchdog.sh (plex cron, */30) checks both instances for the pre-wedge precondition โ torrent not complete, RdProgress=100, โฅ1 file row unfinished โ which is visible days before the loop starts. It calls ~/bin/rdtclient-stuckfiles.py, which resolves the opaque fakedl index to a real filename via Torrents.RdFiles (the 2026-08-03 culprit was a 50 KB .srt subtitle, not a video), then closes stale orphan rows so the torrent finalises and the queue unblocks. Guards: orphan must be >6h old, torrent must already have completed rows, DB backed up before any write, read-only unless --remediate. Manual fallback = back up the DB, then DELETE FROM Downloads + DELETE FROM Torrents for that TorrentId. Blast radius is bounded separately by ~/bin/docker-log-guard.sh (ROOT cron, */15). Recorded paths are torbox.app/fakedl/... URLs, so this touches no local files and nothing on the remote. [Journal: 2026-08-03]QBittorrent.TorrentsDelete switches on Integrations:Default:FinishedAction only — UI path Settings → qBittorrent / *darr → Finished action. On Do nothing the removal is received, logged as Not removing torrents or files, and silently discarded; nothing errors and nothing warns. Movies sat like that for a month and accumulated 94 records against TV's 13. The trap: setting Provider:Default:FinishedAction instead does nothing at all for arr grabs — TorrentsAddMagnet hardcodes FinishedAction = None on every record it creates, so an arr-added torrent always reads finishedAction: 0 no matter what the UI shows. Current values on both instances: FinishedAction=3 (Remove From Client — 3 not 1 so the TorBox copy survives a failed import) with FinishedActionDelay=60 min (an arr that loses its download-client record mid-import loses the import), MinFileSize=50 (MB — stops rdt-client fetching 30-byte subtitle stubs and tripping TorBox 429s; one YTS release has 37 files), TorrentLifetime=180, DeleteOnError=180 (min). All four apply only to NEWLY added torrents — they are copied onto each record at add time, so changing them never fixes existing records; that is what rdtclient-cleanup.py --include-idle is for. Settings backups: /home/plex/ops/rdt-settings-backups/. [Journal: 2026-08-21]torrents/mylist and queued/getqueued are cached and can list deleted items for minutes — a live run deleted 12 things, every call returned success, and the verify step reported nothing had changed. Always append bypass_cache=true to any TorBox GET (getqueued?type=torrent showed 10 entries; with bypass_cache=true, 0). Deletes also return 500 DATABASE_ERROR fairly often while succeeding, so never trust a delete response — only a fresh re-read. Both torbox-slot-guard.py and torbox-cleanup.py now do this. [Journal: 2026-08-21]/proc/pressure/io full avg300=35.8) and made browser download dialogs take 2–5 minutes to open. Moving it to a SuperSpeed socket took it from 33 → 201 MB/s. Diagnosis rule: on xHCI a USB 3 socket enumerates twice — a USB 2.0 half on Bus 001 and a SuperSpeed half on Bus 002; a socket with no Bus 002 twin is USB 2.0-only by design, not broken. The controller reports 7 SuperSpeed root ports but only 2 are wired to reachable rear sockets (the rest are unexposed/front-panel headers) — so fast sockets are genuinely scarce, and all the rear sockets are black (no colour coding on this OEM board). You cannot map them from software using a USB 2.0 device — a USB 2.0 hub or keyboard has no SuperSpeed pins and lands on Bus 001 either way, so a slow device may be squatting in a fast socket invisibly. Map them physically with ~/usb-portmap.sh: run it, then move one known USB 3 device socket to socket and watch for the SUPERSPEED verdict — validate the probe in a known-fast socket first, or a USB 2.0 cable on the probe will make every socket look slow. The bcdUSB discriminator: cat /sys/bus/usb/devices/<port>/version — 2.10+ means the device can do SuperSpeed, 2.00/1.10 means it is genuinely USB 2.0 silicon. ver=2.10 + speed=480 = capable but not trained → slow socket or USB 2.0 upstream cable. That one field separates "is slow" from "is being made slow" without unplugging anything, and das-check.sh now classifies every device on it. Run ~/das-check.sh after any re-cabling or reboot — it flags any drive that fell back to 480 Mbit, shows max_sectors_kb (want 1024, not 120), and shouts if anything binds uas. [Journal: 2026-07-30]Chassis Fan 1: N/A and Chassis Fan 2: N/A — the only fan in the system is the CPU cooler (1666 RPM, Standard curve) plus the GPU's own. Consequences: the CPU idles at 50–67 °C and reached 83 °C under moderate load with the case open and a box fan blowing into it, while the GTX 1050 Ti sat at 28 °C (its own fan) — cool case air, hot CPU, i.e. heat is not leaving the cooler. The DIMMs sit in dead air and have no thermal sensor (no JC-42 over SMBus, confirmed via ~/bin/dimm-thermals.sh), so the memory controller cannot know they are hot and cannot raise the refresh rate to compensate. Also raising idle temps: Package C State Support: Disabled. [Journal: 2026-08-21]sudo ~/bin/arm-memtest.sh (added 2026-08-21). Because GRUB does not take keyboard input here, the menu is useless for selecting anything. GRUB's next_entry variable sets the default for exactly one boot and then clears itself, and this machine's grub.cfg 00_header honours it regardless of GRUB_DEFAULT (which stays 0). grub-reboot writes only that variable — no update-grub, no config edit. So: sudo ~/bin/arm-memtest.sh then sudo reboot, touch nothing, read the screen, hold the power button to exit (safe — memtest mounts nothing and writes to no disk), power on → normal Ubuntu. Cancel before rebooting with --disarm. The script refuses to arm if grubenv sits on LVM/MDRAID, where the entry would go sticky and loop forever. [Journal: 2026-08-21]systemctl reboot --firmware-setup. OsIndicationsSupported has bit 0 set on this ASRock H570M-ITX/ac (AMI P1.70), so this reboots straight into UEFI setup — no Del/F2 timing. Getting in is solved; navigating still needs the keyboard to be recognised at POST. [Journal: 2026-08-21]/dev/sdX letters rotate across reboots — never put one in a command or a script. /media/plex1 has been sdf1, then sdh1, then sdd1 (2026-08-21 22:20, within hours) — and by then /dev/sdh1 was a different, NTFS disk. This server has already had two near-misses of exactly this shape (backup-plex3.sh holding /dev/sde1 after it became the Passport; backup-plex2.sh holding /dev/sdb after it became the scratch SSD). Address volumes as /dev/disk/by-uuid/… and resolve at use time with findmnt -no SOURCE,UUID /media/plex1. plex1 = 73130bac-9c4c-42d1-90bd-ef9838716e87. [Journal: 2026-08-21 device letters]usb-storage.quirks=152d:0583:u,152d:0567:u. The original 0583-only entry from 2026-07-06 never matched this hardware and only appeared to work because USB 2.0 doesn't offer UAS at all — going to SuperSpeed without adding 0567 would have re-armed the July crash-flood. Verify with cat /sys/module/usb_storage/parameters/quirks; drives must bind usb-storage, never uas. Backup: /etc/default/grub.bak.20260730. [Journal: 2026-07-30]/media/plex3_backup (Seagate ST10000DM005, WP001CYM) silently never mounted despite being in fstab with nofail. It is present and mounted at SuperSpeed. [Journal: 2026-07-30]/var/lib/docker) — it is a top cause of root-disk fill.plex-check's cron logs to /var/log/plex-check.log, but the plex user may not be able to create files in /var/log — that cron job's logging (or the job itself) may be silently failing. Pending fix./etc/sudoers.d/claude-audit-tty sets Defaults:plex timestamp_type=global so sudo -v in any terminal warms a ~15 min lease for Claude's bash subshell. /etc/sudoers.d/claude-readonly grants NOPASSWD for read-only diagnostics (journalctl, systemctl status, ufw status, ss, lsof, log reads). Mutating commands still require either a password or a warm lease. A third drop-in /etc/sudoers.d/claude-readonly-hardware (added 2026-05-31) adds NOPASSWD for strictly read-only disk/hardware diagnostics (du, smartctl -a/-x/-H/-i/--scan, dmidecode, blkid, fdisk -l) to support the disk-upgrade work. See 2026-05-22 + 2026-05-31 entries. Do not remove these files — the "audit-tty" name is legacy./media/plex2_backup currently has no drive — the degraded WD (9RHHSV3L) was SMART-confirmed dead and retired 2026-06-04; backup-plex2.sh cron stays disabled until a replacement backup target exists (likely a DAS bay). [Journal: 2026-06-04]docker.service and plexmediaserver.service have RequiresMountsFor=/media/plex1 /media/plex2 drop-ins so they stop before those disks unmount at shutdown; every media mount (plex1, plex2, plex3, plex1_backup, plex3_backup, scratch) now has a 10s TimeoutSec drop-in (/etc/systemd/system/media-*.mount.d/10-timeout.conf) so a wedged USB bridge can't hang shutdown for the 90s default — it force-terminates the unmount after 10s. plex1_backup got this 2026-06-01; the other five on 2026-06-06. These are path-based — a drive swap that keeps the same mount point needs no change here. [Journal: 2026-06-01, 2026-06-06]skyhouse.dev, botaa.org, room101.com) use Cloudflare anycast nameservers (luciane/remy.ns.cloudflare.com); they stay registered at Dynadot but Dynadot no longer hosts DNS. Every record is DNS-only (grey cloud) โ origin 174.21.187.113 (NOT proxied โ the free tier's 100 MB upload cap + no-video-streaming ToS would break Immich/Pairdrop/Plex-ish use). Self-healing: each apex is a flattened CNAME โ cw526dc.glddns.com (the GL.iNet router's DDNS, ~30s TTL) so an IP change is tracked by the router and followed by Cloudflare automatically โ no scripts, no API token, no split-brain. * wildcard โ apex. room101 email = Cloudflare Email Routing (MX route1/2/3.mx.cloudflare.net + SPF + DKIM). Automation: bin/dns-monitor.sh (cron */5) is notify-only (๐ changed / โ
resynced / โ ๏ธ). Dynadot-era scripts retired (rollback-only). Rollback: set NS back to ns1/ns2.dyna-ns.net at Dynadot (zones still exist). [Journal: 2026-06-07 cloudflare-done]/etc/hosts → 192.168.1.136 (apex + all NPM subdomains), and uses only the router (192.168.1.1) for DNS (8.8.8.8 removed from netplan). So LAN/server access is immune to public-IP churn and resolver caching. Add new NPM subdomains to that /etc/hosts line (no wildcards there). [Journal: 2026-06-02]/etc/sudoers.d/plex-backup-plex1 lets plex run the fixed-arg root rsync wrapper /usr/local/sbin/backup-plex1-rsync.sh NOPASSWD (so the plex1 backup can read immich's root-owned files). Unparameterised wrapper = the grant can't rsync arbitrary paths. [Journal: 2026-06-02]smut.botaa.org pointing at Plex is intentional — it's a connect URL for a private group.network_mode: host (stack home-assistant) so mDNS/UPnP device discovery works. Consequences: they have no Docker DNS name, so their NPM proxy hosts must target the host LAN IP 192.168.1.136:8123 / :8095 (not a container name), with WebSockets enabled. MA also needs a hardened Advanced location block (forwards X-Forwarded-Host + the WS Upgrade/Connection headers) or its login handshake fails. HA trusts the proxy via trusted_proxies (172.16.0.0/12 covers all Docker bridges + 192.168.1.0/24) in /opt/appdata/homeassistant/config/configuration.yaml — without it HA returns 400 Bad Request. Because both proxy targets hardcode 192.168.1.136, a host LAN-IP change breaks ha./ma. — keep the host on a static DHCP reservation. [journal]