SKYHOUSE.dev Journal

Why: Documenting active monitoring services that pre-date the journal — captured during the 2026-05 audit so the cheat sheet has a citation target and the current alerting gaps are on record.

The server uses a layered approach to health monitoring: a commercial-grade time-series metrics daemon for system and container stats, a custom cron watchdog for CPU abuse, a shared Telegram helper for all alert delivery, and fail2ban for SSH brute-force protection. Uptime Kuma, which previously handled service-level availability checks, is currently down. This entry documents the state of all four layers as found during the audit.

1. Netdata

Netdata runs as a native systemd service (netdata.service, enabled, active) and listens on port 19999. It is exposed externally via NPM proxy host 5 (5.conf) at https://monitor.skyhouse.dev, cert npm-8. Telegram alerting is configured in /etc/netdata/health_alarm_notify.conf (root-readable, mode 0640); the bot token is stored there — refer to the file directly for the actual value. Netdata auto-updates daily via a symlink in /etc/cron.daily/netdata-updater pointing to /usr/libexec/netdata/netdata-updater.sh.

2. cpu_watchdog.sh

/usr/local/bin/cpu_watchdog.sh (root-owned) runs every five minutes via the plex user crontab. It checks for processes consuming 50% or more CPU for a sustained period and fires a Telegram alert via telegram_notify.sh for any confirmed offender. An allowlist excludes expected high-CPU processes: Plex, Immich, and Handbrake. The audit noted a minor cosmetic bug: the STATE_DIR variable contains literal pipe characters in the path name, creating an oddly-named directory in /var/tmp/ — this does not affect function. Watchdog output goes to stderr only (not captured to a log file).

3. telegram_notify.sh

/usr/local/bin/telegram_notify.sh (root-owned) is the shared Telegram delivery helper consumed by cpu_watchdog.sh, the backup scripts, ip-watch.sh, and plex-check. It sources its bot token and chat ID from /etc/telegram_notify.conf (root:plex, mode 0640). The same bot token is also stored in /home/plex/.messaging-keys and /etc/netdata/health_alarm_notify.conf — three locations means rotating the token requires touching all three files.

4. fail2ban

fail2ban runs as a native systemd service (fail2ban.service, enabled, active). One jail is configured: sshd, watching port 2222 via the journald backend. Policy: maxretry=3, bantime=24h. The action chain includes a custom Telegram action, so ban events result in a Telegram notification in addition to the iptables drop. As of the audit, 11 IPs were banned — indicating real abuse traffic, not phantom triggers. No jails exist for NPM HTTP auth failures or Plex.

5. Uptime Kuma — currently down

Uptime Kuma previously provided service-level monitoring (Plex local/remote, NPM, Music Discovery, DDNS-Go) with 60-second check intervals and Telegram alerting. As of the 2026-05 audit, the container is completely gone — docker compose ps -a against /home/plex/uptime-kuma/docker-compose.yml returns no rows. The Compose file still exists. NPM proxy host 6 (6.conf) still routes status.skyhouse.dev → port 3001, but nothing listens there. Service-level availability monitoring is currently silently off. Decision pending: revive or retire.

The net alerting picture: system metrics and SSH brute-force are well-covered; service availability (is Plex actually responding?) has no active monitor until Uptime Kuma is either revived or replaced.