Server Cheat Sheet

1. Network & access

2. Port map

Red = forwarded to the internet. Green = LAN / localhost only.

3. Reverse-proxy (subdomain) map

Authoritative hostname → backend, as configured in Nginx Proxy Manager.

4. Service inventory

Docker containers (23)

systemd services (non-stock)

plexmediaserver · netdata · fail2ban · smartmontools · ssh (port 2222) · docker / containerd. User units (systemctl --user, linger on): inspiredby-web (port 8440) + inspiredby-worker.timer (hourly; drains its task queue for up to 50 min, budget = 250 network-using tasks) + inspiredby-refresh.timer (Sun 04:00) + inspiredby-reader.timer (hourly; Claude Haiku via claude -p under Damien's login reads 100 Wikipedia articles/h; INSPIREDBY_LLM_READER=off / _PER_HOUR in .env) [journal] — PAUSED 2026-10-07 (INSPIREDBY_LLM_READER=off, timer stopped: ~1M tokens/100 pages on a Pro plan); hosts that send a long Retry-After are blocked in ~/inspiredby/.cache/blocked_hosts.json (delete an entry to lift it early) [journal]; Spotify matching capped at 1 task/run, 10/day, plus site play buttons that look songs up on demand (/api/spotify, 120/h site-wide); Brave Search counter in ~/inspiredby/.cache/brave_usage.json (run.py brave --remaining N to correct); overnight reader test scheduler/usage-test.sh full [journal], code /home/plex/inspiredby — secrets in ~/inspiredby/.env (600: Last.fm, Spotify client shared with music-discovery, cookie key); public POST /suggest writes only to ~/inspiredby/submissions/; photos in ~/inspiredby/media/photos/; DB is WAL (back up with sqlite3 .backup) — admin mode: direct LAN http://192.168.1.136:8440/admin, editor-pass cookie, or editor Spotify id (never proxied IPs: NPM trusts X-Real-IP from CDN ranges) [journal] [journal] (deployed from ~/dev/inspiredby via scheduler/deploy.sh; start with its CLAUDE.md) [journal]; yt-plex — yt-dlp → Plex queue app, port 8430, code at /home/plex/yt-plex. v2 (2026-09-29): login, libraries and Plex/Radarr/Sonarr connections live in the app DB (data/ytplex.db, edit via Settings); no .env; yt-dlp self-updates daily in-app into data/ytdlp-lib (the old 50 5 * * * pip cron is retired). Also packaged as a Docker image (yt-plex:latest, built locally). Published at skyhouse.dev/yt-plex/ (/home/plex/www/yt-plex/). Lost password: YTP_RESET_PASSWORD=… in data/.env + restart. [journal] [journal v2] [journal] (openvpn was disabled in the 2026-05 cleanup.) Note: plexmediaserver is temporarily masked as of 2026-06-04 (plex1 SATA link wedged); systemctl unmask plexmediaserver to restore. [journal]

5. Key paths

6. Cron schedule (user plex)

⚠ Jobs below are shown logically. As of 2026-08-04 every watcher actually runs as flock -n /run/lock/<name>.lock timeout <N> <script> — before that none of the six had a lock or a timeout, so a wedged Docker daemon left instances stacking every 5 min. Backups were already guarded internally. telegram_notify.sh also had no curl timeout and never checked its exit code (a failed alert looked identical to a delivered one); it now logs every send to /var/log/telegram_notify.log. Crontab backups: ~/.cache/plex.crontab.bak-20260804, /root/root.crontab.bak-20260804. [Journal: 2026-08-04 watcher-hardening]

7. Storage & drives

⚠ 2026-08-21 — /media/plex1 is mounted clean with errors and needs an offline e2fsck. 19 corrupt inodes + 31 bad block-bitmap groups, all confined to Immich thumbnails (regenerable); no originals affected — verified by SHA-1 against asset.checksum and by confirming zero originals were written during the corruption window. It has run silently for hours because its Errors behavior is Continue, so it never remounts read-only — check with sudo dumpe2fs -h /dev/<plex1> | grep -E "Filesystem state|FS Error count", not by waiting for a failure. Never fsck'd since the fs was created in 2022. The fsck is blocked on the RAM fault (§11) — rewriting metadata across 12.7 TB through bad memory is how a thumbnail problem becomes a catastrophe. Expect 15–45 min, not hours: only 2.2 M of 427 M inodes are in use. Add whisparr to das-down.sh first — it holds a plex1 bind mount and will block the unmount. [Journal: 2026-08-21] Scope established 2026-09-02: all 19 EBADMSG paths are directories under /media/plex1/Photos/thumbs/ — Immich thumbnails, i.e. regenerable derived data. No originals/movies/metadata affected. Damage is frozen: identical 31 block groups across 08-22 → 09-02 (8d20h uptime, zero new errors with the memmap= blacklist live). ext4 quarantines those groups, costing ~4.2 GB of capacity rather than corrupting writes.

⚠ Device letters shuffle — identify drives by serial/UUID, never /dev/sdX. The sdc/sdd/sde/sdg/sdh labels in the table above are historical and have already moved twice. As of 2026-07-30 the DAS (JMS567 bridge) runs at SuperSpeed with 5 populated bays — plex1 (ST14000NM001G/ZL2BG3W8), plex3 (WD120EDBZ/5QGWXSKF), plex3_backup (ST10000DM005/WP001CYM), plus an unassigned Toshiba 6 TB (HDWE160) and WD 2 TB (WD20EARS). The two WD easystores (plex2, plex1_backup) are separate USB externals. Health check: ~/das-check.sh (link speeds, max_sectors_kb, UAS binding, I/O pressure); attach/detach helpers remain ~/das-up.sh / ~/das-down.sh (das-down FIRST, then unplug). [Journal: 2026-07-30]

Swap: /swapfile 4 GB (OS disk) + /media/scratch/swapfile_extra 32 GB (scratch SSD, sw,nofail) — 35 GB total. Moved off plex3 → scratch 2026-06-04. Media drives mount by UUID in /etc/fstab with nofail. [Journal: 2026-06-04]
⚠ 2026-08-06: the scratch swap line had been commented out in /etc/fstab — the box was silently running on 4 GB, not 35 GB. Restored (backup /etc/fstab.bak-20260806-swap). Post-reboot check: swapon --show must list both files; if only /swapfile appears, the scratch entry has been disabled again. Note neither entry sets pri=, so after a reboot the root SSD is the preferred swap target and scratch is overflow only. [Journal: 2026-08-06]

SMART monitoring (rebuilt by-id 2026-06-04): /etc/smartd.conf lists every present drive explicitly by serial (/dev/disk/by-id/ata-<model>_<serial> -d sat, no DEVICESCAN) — boot SSD, scratch SSD, plex1, plex2, plex3, plex1_backup. The detachable toaster drive plex3_backup uses -d removable (auto-detects as SAT when present, and lets smartd survive it being unplugged). ⚠ smartd EXITS (status 16) if a non-removable listed device is absent — so when you retire or move a drive you MUST drop/adjust its by-id line, or smartd dies silently at next boot (exactly what a stale Corsair line did after the 6/3 boot swap). netdata's smartctl collector (/etc/netdata/go.d/smartctl.conf) now monitors everything present (device_selector: '* *', no /dev/sdX exclusion) — the JMicron bridge that once hung it is gone, and netdata only scans present devices, so it's shuffle-proof. Backups: /etc/smartd.conf.bak-20260604, /etc/netdata/go.d/smartctl.conf.bak-20260604. [Journal: 2026-06-04]

8. Maintenance scripts

Updating apps, generally (2026-07-17): two different procedures depending on where the app lives.

• Plex, public releases: apt now tracks it correctly (repo migrated to repo.plex.tv/deb, see Quirks). Either sudo apt update && sudo apt upgrade plexmediaserver, or sudo plex-update (hits Plex's own download API, checksum-verifies, then dpkg -is — same effect, doesn't need apt).
• Plex, Pass/beta releases: apt can never get these — Plex only serves beta builds through the signed-in web UI's own updater. When it says "Update available … install manually," download the offered .deb from the browser (needs the Plex Pass login), then sudo dpkg -i /path/to/plexmediaserver_<version>_amd64.deb. No script for this side — it's manual by design on Plex's end.
• The media-downloads Portainer-stack apps (prowlarr, sonarr, radarr, rdtclient, rdtclient-movies, lidarr, whisparr): all track :latest, so there's no version to bump in the compose file itself. To update: (if structure is changing) edit /home/plex/docs/media-downloads-stack.yml first, then in Portainer go to Stacks → media-downloads → Editor, paste that file's contents, tick "Re-pull image", and click Update the stack. Skipping the re-pull tick just redeploys with whatever image is already cached locally — no version change. This is also how new services (like lidarr/whisparr) get added to the stack. [journal]

9. Verification checklist

Copy-paste to confirm the server is healthy — e.g. after a reboot or the disk swap. Each line prints OK / a status when good.

Services

systemctl is-active plexmediaserver netdata fail2ban ssh smartmontools
for c in npm-app-1 immich_server music-discovery-client music-discovery-server \
         vaultwarden pairdrop sharedmoments portainer uptime-kuma ddns-go yt-plex-demo; do
  printf '%-26s ' "$c"; docker inspect -f '{{.State.Status}}' "$c" 2>/dev/null || echo MISSING
done
curl -sf http://localhost:2283/api/server/ping        # Immich -> {"res":"pong"}

Mounts & freshness

for m in / /media/plex1 /media/plex2 /media/plex3 /media/scratch \
         /media/plex1_backup /media/plex3_backup; do  # plex2_backup drive retired 6/4
  printf '%-24s ' "$m"; mountpoint -q "$m" && echo OK || echo "NOT MOUNTED"
done
find /var/log/plex-backup.log -mtime -2 | grep -q . && echo "backups fresh"
find ~/.cache/dns-monitor.log -mtime -1 | grep -q . && echo "dns notifier fresh"

External reachability

curl -s ifconfig.me ; echo                # current public IP
dig +short skyhouse.dev @1.1.1.1          # should match the line above

10. Accounts & externals

11. Quirks & gotchas

• ⚠⚠ THE RAM IS FAULTY (2026-08-21) — this gates any large write. memtester produced 6 failures and every one is a single-bit error on bit 43 or 44, both in byte lane 5, across four addresses and five unrelated test patterns. Random soft errors scatter across all 64 bits; a fault confined to two adjacent bits in one lane is a specific physical path (one DRAM chip or a pair of DQ lines). The RAM is non-ECC 2×32 GB G.Skill Ripjaws V (F4-3600C18-32GVK, SK Hynix dies, dual-rank) running its rated XMP profile, DDR4-3600 CL18 at 1.35 V — correctly applied, and in the conservative Gear2. The out-of-spec part is on the CPU side: Rocket Lake officially supports DDR4-3200, and four ranks at 3600 is the heaviest load an 11th-gen IMC can be given — so there are no EDAC counters and nothing will ever warn you. This one fault explains both storage symptoms that day: it corrupted plex1's metadata on the way to disk (persisted) and a Postgres page on the way back from disk (transient). Do not run the plex1 e2fsck or the Immich→plex3 migration until it is resolved. Confirm with memtest86+ 7.00, already installed (GRUB entry “Memory test”, /boot/memtest86+x64.efi) — the fault reproduces in under a minute, so ~12 min is decisive and names the DIMM slot. Then test whether dropping XMP to rated speed clears it before buying anything. [Journal: 2026-08-21]
• A checksum error does NOT prove the disk is bad — re-test cold before you “repair” anything (2026-08-21). It proves only that the bytes you compared did not match. Postgres verifies a page checksum after reading it into memory, so a RAM fault is indistinguishable from a corrupt page until you force the read to come off the platter. The test that separates them: docker stop immich_postgres (clears shared_buffers) → sync; echo 3 | sudo tee /proc/sys/vm/drop_caches → restart → forced heap scan (enable_indexonlyscan/indexscan/bitmapscan=off). A documented “corrupt page” in Immich's asset_exif read back 334,997 rows with zero failures this way, and pg_dump finished in 2.6 s — the page had always been fine. The recommended repair (zero_damaged_pages + VACUUM FULL) would have blanked ~44 perfectly good rows to fix nothing. Conversely plex1's damage did survive the same cold re-read, which is exactly how the real fault was told apart from the phantom. [Journal: 2026-08-21]
• ⚠ Do NOT trust the ✅ … cleared Telegram alert — measured 0 for 4 (2026-08-27). The pipeline probe fires cleared when an item stops matching the stall criteria, not when it is actually imported — and a cleanup job deleting the record satisfies that condition. rdtclient-cleanup (:05) or radarr-queue-cleanup (:17) abandoning a download reads as success on the next poll. Of the seven ✅ Radarr cleared messages ever sent, the four that name a title were all false — one imported 5 h later, one 12 h later, two never; the other three use the aggregate form which drops titles and cannot be audited at all. Verify by hand: check the film has hasFile: true, or look for a downloadFolderImported event in Radarr history near the alert time. The ❌ stalled alerts are reliable and fire a consistent 2 h 03 m after the grab. Rewrite in progress — full reconstruction and method in /home/plex/projects/arr/05-messaging-handoff.md. [Journal: 2026-08-27]
• A healthy mountpoint does not mean the containers can see it. Docker resolves a bind mount once, at container start. If the DAS is not mounted yet, Docker creates the source as an empty directory and the container reads that directory for the rest of its life — mounting the real filesystem over the top on the host changes nothing for it. On 2026-08-27 radarr had started 33 seconds ahead of the mount and spent two days with an empty /data while mountpoint /media/plex1 said true, docker ps said healthy, and the files sat complete on disk. The tell is the import error naming a path you can plainly ls — it reads as permissions and is not. Diagnose: docker exec radarr ls /data (empty vs. the host) or compare /proc/<pid>/mountinfo device numbers. Fix: docker restart, never docker start — start is a no-op on a running container and does not re-resolve the mount, which is exactly why das-up.sh reported success on every run. Now caught automatically every ten minutes by bin/das-mount-check.py. [Journal: 2026-08-27]
• Immich never restarts itself after a reboot. All four immich_* containers are restart: no by design (so a DAS-less boot doesn't crash-loop them against an empty mountpoint). Run ~/das-up.sh — it verifies plex1 by UUID before starting anything. Forgetting this leaves Immich down silently. [Journal: 2026-07-06]
• SSH is on 2222, never 22. Set in the drop-in /etc/ssh/sshd_config.d/10-skyhouse.conf (not the main sshd_config), which lock_ssh.sh / unlock_ssh.sh now edit. [Journal: 2026-10-03]
• ⚠ After any Ubuntu release upgrade, re-check these (2026-10-03). The 26.04 upgrade (1) replaced sshd_config (fixed for good by the drop-in), (2) disabled every third-party apt repo (Plex, Docker, netdata, Chrome — check ls /etc/apt/sources.list.d/ for .disabled / Enabled: no; without the Plex repo, plex-check quietly stops seeing updates), and (3) switched sudo to sudo-rs, which rejects our timestamp_type=global and wildcard rules and so breaks the Claude lease. Fix: sudo update-alternatives --set sudo /usr/bin/sudo.ws, then sudo -v again (a sudo-rs timestamp doesn't carry over). The upgrade leaves the old config next to the new one as *.ucf-old / *.dpkg-old; find them with find /etc \( -name '*.ucf-*' -o -name '*.dpkg-*' \). [Journal: 2026-10-03]
• The GRUB menu is now visible for 10 s at boot (changed 2026-08-21). It was GRUB_TIMEOUT_STYLE=hidden / GRUB_TIMEOUT=1, which on this UEFI box made the menu effectively unreachable — you could not get to memtest86+ or UEFI Firmware Settings without perfect timing on Esc. Now menu / 10. GRUB_DEFAULT=0 unchanged, so doing nothing still boots Ubuntu normally, and the usb-storage.quirks cmdline survives. Revert with sudo sed -i 's/^GRUB_TIMEOUT_STYLE=menu/GRUB_TIMEOUT_STYLE=hidden/; s/^GRUB_TIMEOUT=10/GRUB_TIMEOUT=1/' /etc/default/grub && sudo update-grub. Backup: /etc/default/grub.bak-20260821-memtest. But visible ≠ steerable: the menu appears and counts down, yet GRUB receives no USB keystrokes on this box (tested with a wired Keychron K3 and a Logitech Unifying receiver, both directly on rear ports). Use sudo ~/bin/arm-memtest.sh instead — see the next entry. [Journal: 2026-08-21]
• ⚠ Docker container logs are UNCAPPED on all existing containers (2026-08-03). A stuck row in rdt-client's SQLite made its TaskRunner throw ~8 stack traces/second; the container's JSON log reached 29 GB, dockerd hit 136% CPU driving 987 reads/sec on sda, and everything touching the root disk — including browser downloads — became slow and fragile while the media drives sat idle and fast. /etc/docker/daemon.json now sets max-size 50m / max-file 3, but log options apply at container CREATION, not start — all 28 existing containers still report opts=map[] and must be recreated (compose up --force-recreate / Portainer stack redeploy) before they are actually protected. Triage: sudo du -sh /var/lib/docker/containers/*/ | sort -rh | head; truncate a runaway with sudo truncate -s 0 <path>-json.log (safe while running). Diagnostic tell: docker stats or docker logs timing out means the daemon is saturated, not that the container is busy. [Journal: 2026-08-03]
• rdt-client wedges on a duplicate download row. Signature: SQLite Error 19: UNIQUE constraint failed: Downloads.TorrentId, Downloads.Path repeating in TorrentRunner.Tick() — the whole download pipeline stops and the DB stops being written. Live DB is /home/rdtclient/data/rdtclient.db (not /home/rdtclient/db, which is a stale June copy). Find the culprit: the torrent with many Downloads rows while every other pending one has zero. Now monitored + auto-salvaged: ~/bin/rdtclient-watchdog.sh (plex cron, */30) checks both instances for the pre-wedge precondition — torrent not complete, RdProgress=100, ≥1 file row unfinished — which is visible days before the loop starts. It calls ~/bin/rdtclient-stuckfiles.py, which resolves the opaque fakedl index to a real filename via Torrents.RdFiles (the 2026-08-03 culprit was a 50 KB .srt subtitle, not a video), then closes stale orphan rows so the torrent finalises and the queue unblocks. Guards: orphan must be >6h old, torrent must already have completed rows, DB backed up before any write, read-only unless --remediate. Manual fallback = back up the DB, then DELETE FROM Downloads + DELETE FROM Torrents for that TorrentId. Blast radius is bounded separately by ~/bin/docker-log-guard.sh (ROOT cron, */15). Recorded paths are torbox.app/fakedl/... URLs, so this touches no local files and nothing on the remote. [Journal: 2026-08-03]
• rdt-client: ONE setting decides whether the arrs can remove anything — and it is not the one you would pick. When Radarr/Sonarr ask rdt-client to remove a download, QBittorrent.TorrentsDelete switches on Integrations:Default:FinishedAction only — UI path Settings → qBittorrent / *darr → Finished action. On Do nothing the removal is received, logged as Not removing torrents or files, and silently discarded; nothing errors and nothing warns. Movies sat like that for a month and accumulated 94 records against TV's 13. The trap: setting Provider:Default:FinishedAction instead does nothing at all for arr grabs — TorrentsAddMagnet hardcodes FinishedAction = None on every record it creates, so an arr-added torrent always reads finishedAction: 0 no matter what the UI shows. Current values on both instances: FinishedAction=3 (Remove From Client — 3 not 1 so the TorBox copy survives a failed import) with FinishedActionDelay=60 min (an arr that loses its download-client record mid-import loses the import), MinFileSize=50 (MB — stops rdt-client fetching 30-byte subtitle stubs and tripping TorBox 429s; one YTS release has 37 files), TorrentLifetime=180, DeleteOnError=180 (min). All four apply only to NEWLY added torrents — they are copied onto each record at add time, so changing them never fixes existing records; that is what rdtclient-cleanup.py --include-idle is for. Settings backups: /home/plex/ops/rdt-settings-backups/. [Journal: 2026-08-21]
• TorBox's API serves stale reads and lies about deletes. torrents/mylist and queued/getqueued are cached and can list deleted items for minutes — a live run deleted 12 things, every call returned success, and the verify step reported nothing had changed. Always append bypass_cache=true to any TorBox GET (getqueued?type=torrent showed 10 entries; with bypass_cache=true, 0). Deletes also return 500 DATABASE_ERROR fairly often while succeeding, so never trust a delete response — only a fresh re-read. Both torbox-slot-guard.py and torbox-cleanup.py now do this. [Journal: 2026-08-21]
• ⚠ One rear USB socket is USB 2.0-only — never put a media drive in it (2026-07-30). The DAS spent weeks negotiating 480 Mbit there, which froze the whole box on I/O ~39% of the time (/proc/pressure/io full avg300=35.8) and made browser download dialogs take 2–5 minutes to open. Moving it to a SuperSpeed socket took it from 33 → 201 MB/s. Diagnosis rule: on xHCI a USB 3 socket enumerates twice — a USB 2.0 half on Bus 001 and a SuperSpeed half on Bus 002; a socket with no Bus 002 twin is USB 2.0-only by design, not broken. The controller reports 7 SuperSpeed root ports but only 2 are wired to reachable rear sockets (the rest are unexposed/front-panel headers) — so fast sockets are genuinely scarce, and all the rear sockets are black (no colour coding on this OEM board). You cannot map them from software using a USB 2.0 device — a USB 2.0 hub or keyboard has no SuperSpeed pins and lands on Bus 001 either way, so a slow device may be squatting in a fast socket invisibly. Map them physically with ~/usb-portmap.sh: run it, then move one known USB 3 device socket to socket and watch for the SUPERSPEED verdict — validate the probe in a known-fast socket first, or a USB 2.0 cable on the probe will make every socket look slow. The bcdUSB discriminator: cat /sys/bus/usb/devices/<port>/version — 2.10+ means the device can do SuperSpeed, 2.00/1.10 means it is genuinely USB 2.0 silicon. ver=2.10 + speed=480 = capable but not trained → slow socket or USB 2.0 upstream cable. That one field separates "is slow" from "is being made slow" without unplugging anything, and das-check.sh now classifies every device on it. Run ~/das-check.sh after any re-cabling or reboot — it flags any drive that fell back to 480 Mbit, shows max_sectors_kb (want 1024, not 120), and shouts if anything binds uas. [Journal: 2026-07-30]
• ⚠ THE CASE HAS NO CHASSIS FANS (found 2026-08-22). BIOS H/W Monitor reports Chassis Fan 1: N/A and Chassis Fan 2: N/A — the only fan in the system is the CPU cooler (1666 RPM, Standard curve) plus the GPU's own. Consequences: the CPU idles at 50–67 °C and reached 83 °C under moderate load with the case open and a box fan blowing into it, while the GTX 1050 Ti sat at 28 °C (its own fan) — cool case air, hot CPU, i.e. heat is not leaving the cooler. The DIMMs sit in dead air and have no thermal sensor (no JC-42 over SMBus, confirmed via ~/bin/dimm-thermals.sh), so the memory controller cannot know they are hot and cannot raise the refresh rate to compensate. Also raising idle temps: Package C State Support: Disabled. [Journal: 2026-08-21]
• Boot memtest86+ with no keyboard: sudo ~/bin/arm-memtest.sh (added 2026-08-21). Because GRUB does not take keyboard input here, the menu is useless for selecting anything. GRUB's next_entry variable sets the default for exactly one boot and then clears itself, and this machine's grub.cfg 00_header honours it regardless of GRUB_DEFAULT (which stays 0). grub-reboot writes only that variable — no update-grub, no config edit. So: sudo ~/bin/arm-memtest.sh then sudo reboot, touch nothing, read the screen, hold the power button to exit (safe — memtest mounts nothing and writes to no disk), power on → normal Ubuntu. Cancel before rebooting with --disarm. The script refuses to arm if grubenv sits on LVM/MDRAID, where the entry would go sticky and loop forever. [Journal: 2026-08-21]
• Into the BIOS without a keypress: systemctl reboot --firmware-setup. OsIndicationsSupported has bit 0 set on this ASRock H570M-ITX/ac (AMI P1.70), so this reboots straight into UEFI setup — no Del/F2 timing. Getting in is solved; navigating still needs the keyboard to be recognised at POST. [Journal: 2026-08-21]
• ⚠ /dev/sdX letters rotate across reboots — never put one in a command or a script. /media/plex1 has been sdf1, then sdh1, then sdd1 (2026-08-21 22:20, within hours) — and by then /dev/sdh1 was a different, NTFS disk. This server has already had two near-misses of exactly this shape (backup-plex3.sh holding /dev/sde1 after it became the Passport; backup-plex2.sh holding /dev/sdb after it became the scratch SSD). Address volumes as /dev/disk/by-uuid/… and resolve at use time with findmnt -no SOURCE,UUID /media/plex1. plex1 = 73130bac-9c4c-42d1-90bd-ef9838716e87. [Journal: 2026-08-21 device letters]
• DAS UAS quirk — the bridge is a JMS567, not a JMS583 (2026-07-30). Kernel cmdline is now usb-storage.quirks=152d:0583:u,152d:0567:u. The original 0583-only entry from 2026-07-06 never matched this hardware and only appeared to work because USB 2.0 doesn't offer UAS at all — going to SuperSpeed without adding 0567 would have re-armed the July crash-flood. Verify with cat /sys/module/usb_storage/parameters/quirks; drives must bind usb-storage, never uas. Backup: /etc/default/grub.bak.20260730. [Journal: 2026-07-30]
• The DAS is a 5-bay-populated enclosure and may now stay attached across reboots (2026-07-30). The 2026-07-06 "post-boot hotplug only" restriction is retired — the validation reboot with the DAS attached passed clean (no POST hang, no USB flood). At 480 Mbit the bridge could only enumerate 4 of 5 bays, which is why /media/plex3_backup (Seagate ST10000DM005, WP001CYM) silently never mounted despite being in fstab with nofail. It is present and mounted at SuperSpeed. [Journal: 2026-07-30]
• Docker data lives on the small OS disk (/var/lib/docker) — it is a top cause of root-disk fill.
• FIXED 2026-09-10: plex-check's cron used to log to /var/log/plex-check.log, but the plex user can't create files in root:syslog /var/log — the whole redirect aborted the command before the version check or Telegram alert ever ran, every day, for as long as the job existed. Now logs to /home/plex/.cache/plex-check.log, confirmed working (alert fires). This is how Plex sat at 1.43.3.10896 for 3.5 weeks while 1.43.4.10903 was available. [Journal: 2026-09-10]
• Claude Code sudo lease (permanent): two sudoers drop-ins make Claude-assisted sudo workable. Requires classic sudo (/usr/bin/sudo → sudo.ws) — sudo-rs, the 26.04 default, rejects timestamp_type (2026-10-03). /etc/sudoers.d/claude-audit-tty sets Defaults:plex timestamp_type=global so sudo -v in any terminal warms a ~15 min lease for Claude's bash subshell. /etc/sudoers.d/claude-readonly grants NOPASSWD for read-only diagnostics (journalctl, systemctl status, ufw status, ss, lsof, log reads). Mutating commands still require either a password or a warm lease. A third drop-in /etc/sudoers.d/claude-readonly-hardware (added 2026-05-31) adds NOPASSWD for strictly read-only disk/hardware diagnostics (du, smartctl -a/-x/-H/-i/--scan, dmidecode, blkid, fdisk -l) to support the disk-upgrade work. See 2026-05-22 + 2026-05-31 entries. Do not remove these files — the "audit-tty" name is legacy.
• /media/plex2_backup currently has no drive — the degraded WD (9RHHSV3L) was SMART-confirmed dead and retired 2026-06-04; backup-plex2.sh cron stays disabled until a replacement backup target exists (likely a DAS bay). [Journal: 2026-06-04]
• Reboot unmount ordering: docker.service and plexmediaserver.service have RequiresMountsFor=/media/plex1 /media/plex2 drop-ins so they stop before those disks unmount at shutdown; every media mount (plex1, plex2, plex3, plex1_backup, plex3_backup, scratch) now has a 10s TimeoutSec drop-in (/etc/systemd/system/media-*.mount.d/10-timeout.conf) so a wedged USB bridge can't hang shutdown for the 90s default — it force-terminates the unmount after 10s. plex1_backup got this 2026-06-01; the other five on 2026-06-06. These are path-based — a drive swap that keeps the same mount point needs no change here. [Journal: 2026-06-01, 2026-06-06]
• DNS / IP tracking (as of 2026-06-07 — DNS now on Cloudflare; supersedes all prior Dynadot designs): All three domains (skyhouse.dev, botaa.org, room101.com) use Cloudflare anycast nameservers (luciane/remy.ns.cloudflare.com); they stay registered at Dynadot but Dynadot no longer hosts DNS. Every record is DNS-only (grey cloud) → origin 174.21.187.113 (NOT proxied — the free tier's 100 MB upload cap + no-video-streaming ToS would break Immich/Pairdrop/Plex-ish use). Self-healing: each apex is a flattened CNAME → cw526dc.glddns.com (the GL.iNet router's DDNS, ~30s TTL) so an IP change is tracked by the router and followed by Cloudflare automatically — no scripts, no API token, no split-brain. * wildcard → apex. room101 email = Cloudflare Email Routing (MX route1/2/3.mx.cloudflare.net + SPF + DKIM). Automation: bin/dns-monitor.sh (cron */5) is notify-only (🌐 changed / ✅ resynced / ⚠️). Dynadot-era scripts retired (rollback-only). Rollback: set NS back to ns1/ns2.dyna-ns.net at Dynadot (zones still exist). [Journal: 2026-06-07 cloudflare-done]
• The server resolves its own domains via /etc/hosts → 192.168.1.136 (apex + all NPM subdomains), and uses only the router (192.168.1.1) for DNS (8.8.8.8 removed from netplan). So LAN/server access is immune to public-IP churn and resolver caching. Add new NPM subdomains to that /etc/hosts line (no wildcards there). [Journal: 2026-06-02]
• An extra sudoers drop-in /etc/sudoers.d/plex-backup-plex1 lets plex run the fixed-arg root rsync wrapper /usr/local/sbin/backup-plex1-rsync.sh NOPASSWD (so the plex1 backup can read immich's root-owned files). Unparameterised wrapper = the grant can't rsync arbitrary paths. [Journal: 2026-06-02]
• smut.botaa.org pointing at Plex is intentional — it's a connect URL for a private group.
• Home Assistant & Music Assistant run network_mode: host (stack home-assistant) so mDNS/UPnP device discovery works. Consequences: they have no Docker DNS name, so their NPM proxy hosts must target the host LAN IP 192.168.1.136:8123 / :8095 (not a container name), with WebSockets enabled. MA also needs a hardened Advanced location block (forwards X-Forwarded-Host + the WS Upgrade/Connection headers) or its login handshake fails. HA trusts the proxy via trusted_proxies (172.16.0.0/12 covers all Docker bridges + 192.168.1.0/24) in /opt/appdata/homeassistant/config/configuration.yaml — without it HA returns 400 Bad Request. Because both proxy targets hardcode 192.168.1.136, a host LAN-IP change breaks ha./ma. — keep the host on a static DHCP reservation. [journal]