Server Cheat Sheet

1. Network & access

2. Port map

Red = forwarded to the internet. Green = LAN / localhost only.

3. Reverse-proxy (subdomain) map

Authoritative hostname → backend, as configured in Nginx Proxy Manager.

4. Service inventory

Docker containers (21)

systemd services (non-stock)

plexmediaserver · netdata · fail2ban · smartmontools · ssh (port 2222) · docker / containerd. (openvpn was disabled in the 2026-05 cleanup.) Note: plexmediaserver is temporarily masked as of 2026-06-04 (plex1 SATA link wedged); systemctl unmask plexmediaserver to restore. [journal]

5. Key paths

6. Cron schedule (user plex)

7. Storage & drives

Swap: /swapfile 4 GB (OS disk) + /media/scratch/swapfile_extra 32 GB (scratch SSD, sw,nofail) — 35 GB total. Moved off plex3 → scratch 2026-06-04. Media drives mount by UUID in /etc/fstab with nofail. [Journal: 2026-06-04]

SMART monitoring (rebuilt by-id 2026-06-04): /etc/smartd.conf lists every present drive explicitly by serial (/dev/disk/by-id/ata-<model>_<serial> -d sat, no DEVICESCAN) — boot SSD, scratch SSD, plex1, plex2, plex3, plex1_backup. The detachable toaster drive plex3_backup uses -d removable (auto-detects as SAT when present, and lets smartd survive it being unplugged). ⚠ smartd EXITS (status 16) if a non-removable listed device is absent — so when you retire or move a drive you MUST drop/adjust its by-id line, or smartd dies silently at next boot (exactly what a stale Corsair line did after the 6/3 boot swap). netdata's smartctl collector (/etc/netdata/go.d/smartctl.conf) now monitors everything present (device_selector: '* *', no /dev/sdX exclusion) — the JMicron bridge that once hung it is gone, and netdata only scans present devices, so it's shuffle-proof. Backups: /etc/smartd.conf.bak-20260604, /etc/netdata/go.d/smartctl.conf.bak-20260604. [Journal: 2026-06-04]

8. Maintenance scripts

9. Verification checklist

Copy-paste to confirm the server is healthy — e.g. after a reboot or the disk swap. Each line prints OK / a status when good.

Services

systemctl is-active plexmediaserver netdata fail2ban ssh smartmontools
for c in npm-app-1 immich_server music-discovery-client music-discovery-server \
         vaultwarden pairdrop sharedmoments portainer uptime-kuma ddns-go; do
  printf '%-26s ' "$c"; docker inspect -f '{{.State.Status}}' "$c" 2>/dev/null || echo MISSING
done
curl -sf http://localhost:2283/api/server/ping        # Immich -> {"res":"pong"}

Mounts & freshness

for m in / /media/plex1 /media/plex2 /media/plex3 /media/scratch \
         /media/plex1_backup /media/plex3_backup; do  # plex2_backup drive retired 6/4
  printf '%-24s ' "$m"; mountpoint -q "$m" && echo OK || echo "NOT MOUNTED"
done
find /var/log/plex-backup.log -mtime -2 | grep -q . && echo "backups fresh"
find ~/.cache/ip-watch.log    -mtime -1 | grep -q . && echo "ddns log fresh"

External reachability

curl -s ifconfig.me ; echo                # current public IP
dig +short skyhouse.dev @1.1.1.1          # should match the line above

10. Accounts & externals

11. Quirks & gotchas

• SSH is on 2222, never 22.
• Docker data lives on the small OS disk (/var/lib/docker) — it is a top cause of root-disk fill.
• plex-check's cron logs to /var/log/plex-check.log, but the plex user may not be able to create files in /var/log — that cron job's logging (or the job itself) may be silently failing. Pending fix.
• Claude Code sudo lease (permanent): two sudoers drop-ins make Claude-assisted sudo workable. /etc/sudoers.d/claude-audit-tty sets Defaults:plex timestamp_type=global so sudo -v in any terminal warms a ~15 min lease for Claude's bash subshell. /etc/sudoers.d/claude-readonly grants NOPASSWD for read-only diagnostics (journalctl, systemctl status, ufw status, ss, lsof, log reads). Mutating commands still require either a password or a warm lease. A third drop-in /etc/sudoers.d/claude-readonly-hardware (added 2026-05-31) adds NOPASSWD for strictly read-only disk/hardware diagnostics (du, smartctl -a/-x/-H/-i/--scan, dmidecode, blkid, fdisk -l) to support the disk-upgrade work. See 2026-05-22 + 2026-05-31 entries. Do not remove these files — the "audit-tty" name is legacy.
• /media/plex2_backup currently has no drive — the degraded WD (9RHHSV3L) was SMART-confirmed dead and retired 2026-06-04; backup-plex2.sh cron stays disabled until a replacement backup target exists (likely a DAS bay). [Journal: 2026-06-04]
• Reboot unmount ordering: docker.service and plexmediaserver.service have RequiresMountsFor=/media/plex1 /media/plex2 drop-ins so they stop before those disks unmount at shutdown; every media mount (plex1, plex2, plex3, plex1_backup, plex3_backup, scratch) now has a 10s TimeoutSec drop-in (/etc/systemd/system/media-*.mount.d/10-timeout.conf) so a wedged USB bridge can't hang shutdown for the 90s default — it force-terminates the unmount after 10s. plex1_backup got this 2026-06-01; the other five on 2026-06-06. These are path-based — a drive swap that keeps the same mount point needs no change here. [Journal: 2026-06-01, 2026-06-06]
• DNS / IP tracking (as of 2026-06-07 — DNS now on Cloudflare; supersedes all prior Dynadot designs): All three domains (skyhouse.dev, botaa.org, room101.com) use Cloudflare anycast nameservers (luciane/remy.ns.cloudflare.com); they stay registered at Dynadot but Dynadot no longer hosts DNS. Every record is DNS-only (grey cloud) → origin 174.21.187.113 (NOT proxied — the free tier's 100 MB upload cap + no-video-streaming ToS would break Immich/Pairdrop/Plex-ish use). Self-healing: each apex is a flattened CNAME → cw526dc.glddns.com (the GL.iNet router's DDNS, ~30s TTL) so an IP change is tracked by the router and followed by Cloudflare automatically — no scripts, no API token, no split-brain. * wildcard → apex. room101 email = Cloudflare Email Routing (MX route1/2/3.mx.cloudflare.net + SPF + DKIM). Automation: bin/dns-monitor.sh (cron */5) is notify-only (🌐 changed / ✅ resynced / ⚠️). Dynadot-era scripts retired (rollback-only). Rollback: set NS back to ns1/ns2.dyna-ns.net at Dynadot (zones still exist). [Journal: 2026-06-07 cloudflare-done]
• The server resolves its own domains via /etc/hosts → 192.168.1.136 (apex + all NPM subdomains), and uses only the router (192.168.1.1) for DNS (8.8.8.8 removed from netplan). So LAN/server access is immune to public-IP churn and resolver caching. Add new NPM subdomains to that /etc/hosts line (no wildcards there). [Journal: 2026-06-02]
• An extra sudoers drop-in /etc/sudoers.d/plex-backup-plex1 lets plex run the fixed-arg root rsync wrapper /usr/local/sbin/backup-plex1-rsync.sh NOPASSWD (so the plex1 backup can read immich's root-owned files). Unparameterised wrapper = the grant can't rsync arbitrary paths. [Journal: 2026-06-02]
• smut.botaa.org pointing at Plex is intentional — it's a connect URL for a private group.
• Home Assistant & Music Assistant run network_mode: host (stack home-assistant) so mDNS/UPnP device discovery works. Consequences: they have no Docker DNS name, so their NPM proxy hosts must target the host LAN IP 192.168.1.136:8123 / :8095 (not a container name), with WebSockets enabled. MA also needs a hardened Advanced location block (forwards X-Forwarded-Host + the WS Upgrade/Connection headers) or its login handshake fails. HA trusts the proxy via trusted_proxies (172.16.0.0/12 covers all Docker bridges + 192.168.1.0/24) in /opt/appdata/homeassistant/config/configuration.yaml — without it HA returns 400 Bad Request. Because both proxy targets hardcode 192.168.1.136, a host LAN-IP change breaks ha./ma. — keep the host on a static DHCP reservation. [journal]