SKYHOUSE.dev Journal

Why: The cross-device flapping from earlier today (plex1 wedge + the 128 GB Kingston dropping off the bus) all pointed at the marginal internal SATA cabling, not the drives. New cables arrived, so we re-seated everything and confirmed the fix — then completed the deferred scratch-disk build and reverted the protective measures.

Damien swapped in fresh SATA cables and re-seated the internal drives. Everything that had been misbehaving came back clean, which retroactively confirms the diagnosis: the host's cabling/power path was the fault, the drives were fine. With the hardware stable, we set up the long-deferred scratch SSD, moved Plex's transcode churn onto it, and undid the morning's protective lockdown.

1. Cable fix confirmed — plex1 stable, Kingston back

• plex1 (Seagate ST14000NM001G / ZL2BG3W8): SMART still pristine (Reallocated 0, Pending 0, Offline-uncorrectable 0, CRC 0, Command-timeout 0) and — the part that matters — zero ata link resets / I/O errors in the kernel log since the boot, where before it was flapping ata5 up/down. Filesystem mounted clean. The new cable held.
• 128 GB Kingston (SNVP325S2128GB / 203S10Q4T73Z): back on the bus and visible again — it had previously dropped with no kernel error, the signature of a loose power/data lead rather than a signal fault. New cable, stable link (SATA_Phy_Error_Count 0).
• This closes out the host-vs-drive question from the morning incident: it was the host (cabling), now remediated.

2. Old plex2_backup WD — confirmed genuinely dead (different connection, same verdict)

Damien re-attached the retired plex2_backup drive (WD WUH721414ALE6L4 / 9RHHSV3L) through the Sabrent USB "toaster" — a completely different connection path — as a final sanity check on whether it was really failing or just badly connected. The verdict is unambiguous and is not a connection artifact:

• Offline_Uncorrectable: 5,483 — sectors the drive physically cannot read back.
• Current_Pending_Sector: 8, Reallocated_Sector_Ct: 52 — active, ongoing media failure.
• The "SMART overall: PASSED" header is the usual lie; the attribute table is damning. Only the 28 UDMA-CRC errors are link-related noise.

It stays retired. (Housekeeping note: through the toaster it auto-mounts at /media/plex2_backup via its old label — umount before pulling so nothing writes to it.)

3. Kingston set up as a dedicated scratch disk → /media/scratch

SMART on a real SATA link: old but clean — 0 reallocated sectors, 0 bad clusters, 0 phy errors, PASSED. The caveat is age: ~86,150 power-on hours (≈9.8 years) and 467 unexpected-power-loss events. That's fine for scratch precisely because scratch data is disposable; if it dies we lose nothing and swap it. Rule: nothing irreplaceable lives here.

• Wiped (was an old NTFS "128GB SSD2" partition), fresh GPT, single ext4 partition labelled scratch, formatted with -m 0 to reclaim the 5% root reserve (full 117 GB usable).
• fstab entry mounts by UUID (46b67b3d-bd85-4877-b231-066d4461201e) at /media/scratch with nosuid,nodev,nofail,x-systemd.device-timeout=10 — a dead scratch disk can never stall boot.
• Subdirs transcode/, downloads/, tmp/ created, owned by plex.

4. Plex transcode temp moved onto the scratch SSD

Plex previously had no TranscoderTempDirectory set, so transcodes landed in its config dir on the boot SSD. Set TranscoderTempDirectory="/media/scratch/transcode" in Preferences.xml (edited while Plex was stopped so it wouldn't be overwritten on exit; ownership preserved as plex:plex). Transcode write-churn now lands on the dedicated SSD instead of the boot disk or a media HDD.

Download staging was deliberately NOT moved. The rdtclient containers stage to /media/plex1/torbox_downloads and /media/plex2/torbox_downloads, co-located with the library so Sonarr/Radarr imports are instant same-filesystem renames. Moving staging to scratch would turn every import into a slow cross-filesystem copy, pile write-wear on a 10-year-old SSD, and risk filling 117 GB with a single large release. Transcode (transient, small, disposable) is the right tenant for scratch; downloads are not.

5. Protective measures reverted

With plex1 confirmed stable, the two reversible measures from the morning incident were undone (the third, nofail on plex1's fstab entry, was already in place and stays):

• Plex unmasked & started: systemctl unmask plexmediaserver → active, enabled, listening on :32400.
• plex1 → plex1_backup nightly mirror re-enabled: uncommented the 30 4 * * * backup-plex1.sh line in the plex crontab.

(The backup-plex2.sh cron stays disabled — its target was the dead WD, now physically retired.)

Net effect: the internal SATA flapping is resolved by re-cabling, the spare Kingston is now a working 117 GB scratch disk hosting Plex transcode, the dead plex2_backup WD is confirmed and retired, and the morning's protective lockdown is fully lifted with all services back to normal.