SKYHOUSE.dev Journal

Why: plex1's internal-SATA link is flapping again (it was re-seated during the 6/3 boot swap, but the cable/port/power path is still marginal). Its ext4 wedged mid-I/O, and Plex was endlessly retrying to index the unresponsive drive. We took reversible protective measures to stop all further I/O against the flapping disk until it can be physically fixed and/or moved to the DAS arriving 6/8.

This started as a "move the *arr stack into a Portainer stack" session and turned into a disk investigation. The headline finding: the drive is healthy; the connection isn't. Nothing here is a Docker or config bug.

1. Diagnosis — flapping link, not a dying drive

• SMART is pristine on plex1 (Seagate ST14000NM001G / ZL2BG3W8): Reallocated 0, Pending 0, Offline-uncorrectable 0, CRC 0, PASSED, 32°C. The drive is not degrading.
• Kernel log = textbook marginal connection: ata5 link up/down, downshift to 1.5 Gbps, qc timeout … cmd 0xec (IDENTIFY), revalidation failed, eventual disable device → detaching. The drive's own internal error log: No Errors Logged — the failures are link-layer, never reached the platters.
• Wedged state: jbd2/sdd1-8 (ext4 journal) plus the rdtclient-movies startup chown -R are stuck in uninterruptible D state; they can't be killed and will clear only on reboot. This is also why, in the same session, a container "wouldn't die" and required killing the containerd-shim — a D-state process on the dead disk.
• It's intermittent, not dead: the 4:30 AM mirror appears to have completed quietly (plex1 was readable then); the hard wedge came later when the container chown hit a link drop. Flapping = marginal cable/power.
• Pattern across devices points at the host, not the drives: plex1 (re-seated 6/3, flapping again), the 128 GB Kingston scratch SSD (dropped off the bus with no kernel error → power/port, not signal), and even the plex3 USB mirror logging "STALL DETECTED → Resetting drive connection."

2. Plex masked

Stopped plexmediaserver.service (it went to a defunct/zombie state — one worker thread is stuck in D on plex1), then masked it so its Restart=always policy can't relaunch the indexer into the wedged disk after a reboot.

• Action: sudo systemctl mask plexmediaserver — reverse with sudo systemctl unmask plexmediaserver once plex1 is on a stable connection.

3. plex1 → plex1_backup nightly mirror paused

Commented out the 30 4 * * * backup-plex1.sh line in the plex crontab so the nightly rsync -a --delete can't run against a flapping source while unattended.

• Backup integrity was already protected: the wrapper (/usr/local/sbin/backup-plex1-rsync.sh) uses --delete but not --ignore-errors, so rsync auto-skips the delete pass on any source I/O error (seen firing in the 6/2 log: "IO error encountered -- skipping file deletion"). /media/plex1_backup is intact and current as of 6/3 and is the trusted copy.
• Action: crontab line prefixed with a dated # DISABLED comment; re-enable by uncommenting after plex1 is stable.

4. Downstream symptom (not a separate bug)

The rdtclient-movies container (now part of the consolidated media-downloads Portainer stack) shows unhealthy and Radarr gets "Connection refused" — purely because its LinuxServer chown -R /data startup step hangs on the wedged plex1 and the app never binds :6500. No YAML fix applies; it will start in seconds once plex1 is responsive. (The TV rdtclient, on plex2, is healthy; its DB was relocated to /home/rdtclient/data and re-keyed during the session.)

5. Next steps (physical, when on-site)

• Cord evaluation: isolate by changing one variable at a time — new data cable (same port) → different port (same cable) → different power lead off any splitter. Watch SMART attr 199 (UDMA CRC) and dmesg link resets. Suspect the shared SATA-power "cord chain" given multiple drives misbehave.
• Before remount: run e2fsck -f on the unmounted plex1 (detached mid-write).
• Before any reboot: add nofail to plex1's fstab entry (or physically disconnect it) so a flapping disk can't hang boot.
• DAS (arrives 6/8): moving plex1 (and the Kingston) into it is both the fix and the definitive diagnostic — if they behave there, the machine's internal SATA path is condemned.
• To revert this entry's measures: systemctl unmask plexmediaserver + uncomment the cron line, once plex1 is on a stable connection.

Net effect: all I/O against the flapping plex1 is halted and made reboot-safe, with zero data risk — the drive's media is healthy and its backup is intact — pending a physical cable/port/power fix or the DAS migration.