SKYHOUSE.dev Journal

Why: Documenting the pre-existing backup pipeline and disk layout — captured during the 2026-05 audit so the cheat sheet has a citation target and the known gaps are on record.

The server hosts a large media library spread across six spinning hard drives plus the root SSD. Three nightly backup scripts mirror the primary Plex media directories to dedicated backup drives. This entry records the layout, the scripts, and the gaps the audit surfaced — none of which were changed during this session.

1. Eight-drive layout

All drives are mounted via /etc/fstab by UUID with nofail,x-gvfs-show options. The array as of the audit:

• sda — 120 GB Corsair SSD; / (root). 77% used. Also holds /swapfile (primary swap).
• sdb — 14 TB WDC (WUH721414ALE6L4); /media/plex2_backup. 76% used. Degraded: 52 reallocated sectors, 8 pending, 5483 offline-uncorrectable. SMART overall still PASSED. Flagged in cron as failing / pending replacement.
• sdc — 14 TB Seagate (ST14000NM001G); /media/plex1. 92% used — highest utilization in the array.
• sdd — 12 TB WDC (WD120EDBZ); /media/plex3. 27% used. Also holds /media/plex3/swapfile_extra (secondary swap — goes away if sdd is unmounted).
• sde — 10 TB Seagate (ST10000DM005); /media/plex3_backup. 32% used. Healthy.
• sdf2 — 457 GB (USB/eMMC, no SMART); /media/plex/Timeshift. 35% used. Timeshift snapshot target.
• sdg — 14 TB WDC (WD140EDGZ); /media/plex2. 76% used. UDMA_CRC_Error_Count=5927 — transient bus errors, watch.
• sdh — 14 TB WDC (WD140EDGZ); /media/plex1_backup. 86% used. Only NTFS drive in the array (fuseblk mount). All SMART counters healthy.

2. Backup scripts

Three scripts run nightly from the plex user crontab. All send a Telegram notification on any non-zero exit via /usr/local/bin/telegram_notify.sh.

• /home/plex/backup-plex3.sh — runs at 0 3 * * *. Calls sudo /home/plex/robust_rsync.sh to mirror /media/plex3 → /media/plex3_backup. Recent runs (2026-05-14 through 2026-05-19) all completed with SUCCESS.
• /home/plex/backup-plex1.sh — runs at 30 4 * * *. Uses plain rsync (not robust_rsync.sh) to mirror /media/plex1 → /media/plex1_backup. The 2026-05-19 run logged rsync error: code 23 (some files/attrs not transferred) — non-fatal but indicates attribute-permission mismatches worth periodically reviewing.
• /home/plex/backup-plex2.sh — disabled in cron (the crontab line is commented out with a note pointing at sdb's degraded state). Would mirror /media/plex2 → /media/plex2_backup via sudo robust_rsync.sh. Should remain disabled until sdb is replaced.

3. robust_rsync.sh

/home/plex/robust_rsync.sh is a retry-on-stall rsync wrapper owned by root. It is invoked by the backup scripts via sudo, which is permitted by /etc/sudoers.d/plex-backup (plex ALL=(ALL) NOPASSWD: /home/plex/robust_rsync.sh). The wrapper adds stall detection and automatic retry logic on top of vanilla rsync, making it more resilient to temporary drive hiccups than a plain rsync call.

4. Known gaps

Two gaps were noted during the audit and are recorded here for follow-up:

• No restore procedure documented. The runbook covers backup runtime but there is no written procedure for "how to restore from a _backup drive." This should be added before any disk-swap operation.
• No off-site backup. All backup drives are physically in the same chassis as the source drives. A fire, flood, or catastrophic PSU failure would take source and backup simultaneously. The runbook §8 acknowledges this as a known gap. No cloud or off-site destination is currently configured.

The backup story is solid for hardware-failure scenarios within the array, but has zero off-site redundancy — worth resolving before the next disk swap forces any attention on this hardware.