Server Troubleshooting Guide

Did you just copy something large into /home/plex? Is a big Plex transcode or a media-processing job running right now? Those are self-inflicted, not failures.

df -h /
sudo du -xhd1 /     | sort -h | tail -12   # biggest dirs on the OS disk
sudo du -xhd1 /var  | sort -h | tail -12
docker system df                          # image / build-cache bloat
journalctl --disk-usage

On this server the usual culprits, in order: Docker (/var/lib/docker is on the OS disk — ~21 GB of images plus a multi-GB build cache), /var/log, the systemd journal, old kernels (3 are installed), and the Plex transcoder's temp files if they land on /.

docker image prune              # delete unused images
docker builder prune            # delete build cache (safe; rebuilds slower once)
sudo journalctl --vacuum-size=200M
sudo apt clean                  # clear downloaded .deb cache
sudo apt autoremove --purge     # remove old kernels & orphan packages

Avoid bare docker system prune -a — it also deletes images not tied to a running container, forcing big re-pulls. The durable fix is the larger primary disk: see /home/plex/docs/disk-cutover-guide.md.

If the growth has no explanation — not Docker, not logs, not media — a dropped payload or attack logging is possible. Run §1.

A job legitimately in progress will use scratch space — that is expected. The concern is space that stays consumed with nothing running.

df -h / /tmp
du -sh /tmp/* 2>/dev/null | sort -h | tail
ls -lt /tmp

Plex transcodes to its Transcoder temporary directory (Plex settings → Transcoder, or TranscoderTempDirectory in Preferences.xml); if that points at /, a 4K transcode can eat many GB. A crashed job can also leave orphaned temp files behind.

Delete orphaned temp files that no live process owns. Finish or stop a stuck job. Longer term: point Plex's transcoder temp at a media drive with room (one of the multi-TB /media/plex* mounts), and lean on the planned bigger OS disk — expanding scratch space is the whole reason for the swap.

Did you start something heavy that simply isn't on the watchdog's allowlist — a manual ffmpeg, a big rsync, an immich library re-scan? If so it is benign; the fix is to add it to ALLOWLIST in /usr/local/bin/cpu_watchdog.sh.

top                                   # instantaneous; find the PID
ps -eo pid,ppid,user,%cpu,etime,comm --sort=-%cpu | head
ps -ef --forest                        # who is the parent?
docker stats --no-stream               # per-container CPU

Identify the process, its user, and whether it lives in a container. The March cryptominer showed up exactly here: sustained high CPU by a process the watchdog did not recognize, running inside music-discovery-client.

If you cannot account for the process — unfamiliar name, odd user, running from a container — go straight to §1. A cryptominer is the single most likely cause of an unexplained sustained-CPU alert on a public-facing server.

free -h
ps aux --sort=-%mem | head -12
docker stats --no-stream

Benign: the desktop Chrome browser routinely holds several GB; many simultaneous Plex transcodes and immich machine-learning runs are memory-hungry by design. Closing browser tabs or letting jobs finish resolves it.

An unrecognized process high in the memory list — run §1.

# 1. Public IP vs what DNS says (the recurring failure mode):
curl -s ifconfig.me ; echo
dig +short skyhouse.dev @1.1.1.1
# 2. Is the reverse proxy up? (80/443 reach nothing if not)
docker ps | grep npm-app-1

If the public IP and the DNS A-record disagree, DNS is stale — the once-every-few-months event. The DDNS pipeline (ip-watch.sh) should fix it; if it hasn't, correct the record at Dynadot manually. The router is also reachable directly at the always-current hostname cw526dc.glddns.com.

If DNS is correct but it's still dead from outside: confirm the router port-forwards (WAN 80, 443, 32400, 2222 → 192.168.1.136) in the GL.iNet admin; confirm the modem is still in bridge mode (a modem reset can re-enable its router and double-NAT you, killing all inbound); and confirm the server still holds its DHCP-reserved 192.168.1.136.

Low risk here — this is almost always plumbing. But if DNS records were changed and you didn't change them, check the Dynadot account for unauthorized access.

SSH is on 2222, not 22. A deploy tool defaulting to 22 will silently fail. ssh -p 2222 plex@skyhouse.dev.

# Did fail2ban ban your deploy host? (a few bad auths = 24h ban)
sudo fail2ban-client status sshd
# Is password auth currently off? (lock_ssh.sh sets this)
sudo sshd -T | grep -i passwordauthentication

The most common non-obvious cause: a deploy script retried with a wrong key/password, tripped fail2ban (3 strikes), and got the source IP banned for 24h. The second: lock_ssh.sh was run, so password auth is off and a password-based deploy now needs a key.

sudo fail2ban-client set sshd unbanip <YOUR-IP>
/home/plex/unlock_ssh.sh        # re-enable password auth, if needed

Also rule out: public IP changed (the deploy targets the hostname → §6), or sshd itself is down (systemctl is-active ssh from the console).

Port 2222? · fail2ban ban on your current IP (test from a different network, or check from the console)? · password auth turned off by lock_ssh.sh and you have no key enrolled? · sshd stopped? · the whole server down (§10)?

This server has a physical desktop (keyboard + monitor). From that local console you can always log in and run /home/plex/unlock_ssh.sh, sudo fail2ban-client set sshd unbanip <IP>, or sudo systemctl restart ssh — no network required. You cannot lock yourself out permanently.

Map the failing site to its backend:

# containerized service:
docker ps -a | grep <name>             # Up, or Exited?
docker logs --tail 50 <name>
# systemd service (Plex, netdata):
systemctl status <unit>

docker start <name>                            # or, from its compose dir:
cd /home/plex/<dir> && docker compose up -d
sudo systemctl restart <unit>                  # for Plex / netdata

If it crashes again immediately: read the logs, check disk space (§2). If the affected service is music-discovery-client, recall its incident history — run §1 before assuming it's a benign crash.

From a LAN device: does ping 192.168.1.136 answer? If no — the box is off, crashed, or off the network: check power, the network cable, and the router. If ping answers but services don't — treat as a stack problem (Docker daemon down? → check from the console). Check uptime for an unexpected reboot.

Most services auto-start (Docker restart: unless-stopped, systemd units). Once it's back, run the verification checklist on the Cheat Sheet to confirm every service and mount came back. Note: the media drives mount with nofail, so the system boots even if a drive is missing — verify mounts explicitly.