SKYHOUSE.dev Journal

Why: Automatic downloads had stopped working and manual browser downloads had gone slow, fragile, and interruption-prone. The symptom pointed at /media/plex2, which had just been through the USB overhaul — so the obvious suspicion was that the storage problem had come back.

It hadn't. plex2 was blameless, and the actual culprit was a single database row that had been wedged since July 18.

1. Ruling out the drive first

Given the history it was worth being thorough before believing anything else. Every storage measurement came back healthy:

• plex2 write 192 MB/s, read 179 MB/s (plex1 264 MB/s for comparison — plex2 trails only because it is 89% full).
• A real 100 MB HTTP download written straight to /media/plex2: 105 MB/s, indistinguishable from plex3 (111) and the root disk (112).
• Internet throughput 111 MB/s; DNS resolving; WAN IP and Cloudflare records in agreement (the apex CNAME self-healing had correctly followed an IP change to 97.113.247.79).
• All USB links still SuperSpeed, max_sectors_kb 1024, IO pressure 0.00, no D-state processes, inodes 1% used.
• /media/plex2/torbox_downloads was empty, last touched July 30 — nothing had reached the disk in four days.

Storage, filesystem, and network were all fine. The failure was upstream of all of them.

2. The actual fault: a crash loop at 8 Hz

docker logs rdtclient was full of one repeating exception:

SQLite Error 19: 'UNIQUE constraint failed: Downloads.TorrentId, Downloads.Path'
  at RdtClient.Data.Data.TorrentData.UpdateComplete(...)
  at RdtClient.Service.Services.TorrentRunner.Tick()

56,712 occurrences in two hours — about eight per second, throwing and retrying on the next tick, forever. ⚠️ Corrected 2026-08-04 (while assembling the upstream bug report, from the app's own logs rather than the docker log): the loop throws from two call sites per tick, not one, and the dominant one is not the insert originally described here. Tick():521 → UnrestrictLink → Downloads.UpdatePath (DownloadData.cs:145) is an UPDATE — the unrestrict call returns a path already held by a sibling row of the same torrent — at 1,038 hits, alongside Tick():773 → UpdateComplete (TorrentData.cs:236) at 1,039 in the same window. Detection and remediation are unaffected, since both key on the database state rather than the call site, but the original attribution would have sent a future session hunting the wrong function. Nothing in the download pipeline could progress, and rdtclient.db had not been written since 03:30 that morning because every transaction rolled back.

3. Why a container bug broke the browser

This is the part worth remembering. Each failure wrote a multi-kilobyte stack trace, and it went to two unbounded logs, both on the root disk:

• Docker's JSON log for the container — which had reached 29 GB. Docker's logging was configured with json-file and no size limit whatsoever.
• rdt-client's own application log at /home/rdtclient/data/rdtclient_*.log, rotating a fresh 5 MB file every ~4 minutes — roughly 1.8 GB/day.

Servicing that, dockerd was burning 136% CPU and driving 987 reads/sec at 122 MB/s against sda. Both docker stats and docker logs timed out at 120 s, which was itself the tell that the daemon was saturated.

Chrome's profile, download history, and the entire desktop session live on sda. So the media drives were idle and fast while anything touching the root disk was fighting the daemon for it. The symptom appeared on plex2; the contention was on sda. A storage-shaped complaint whose cause was three layers away.

4. The one row

The wedge was torrent 02D88676 — Lost.Girl.2010.Complete.Seasons.1.to.5, added July 18:

• 232 Downloads rows — every other pending torrent had zero, because they were all queued behind it.
• RdStatus 4, RdProgress 100 — the remote side considered it finished.
• 231 of 232 rows already Completed. One orphan remained, and that single row was what UpdateComplete kept colliding with.

Critically, the content had downloaded fine: 135 GB and 115 video files sitting in /media/plex2/TV/Lost Girl (2010), already imported. The recorded paths were https://torbox.app/fakedl/... URLs, so deleting the database record touched neither local files nor the remote.

5. What was done

• Stopped rdtclient. Immediate: dockerd 136% → idle, sda 987 reads/sec → 0, CPU idle 86% → 97%.
• Truncated the 29 GB JSON log. Root disk 73% → 69%, reclaiming 29 GB.
• Repaired the database. Backed up to rdtclient.db.bak-20260803-213811, verified integrity_check ok, then deleted torrent 02D88676 and its 232 Downloads rows in one transaction. 71→70 torrents, 355→123 downloads, integrity still ok.
• Set a daemon-wide log cap in /etc/docker/daemon.json (max-size 50m, max-file 3) — the file had been an empty {}. Restarted the daemon; all containers returned.
• Restarted rdtclient: zero constraint errors, container healthy, JSON log back to 8 KB.

6. ⚠️ The log cap is only half applied

Worth being precise about, because it is easy to assume otherwise: daemon.json log options apply at container creation, not at start. Verified after the daemon restart — all 28 existing containers still report opts=map[], i.e. no limit. Restarting them does not make them inherit the new default; each must be recreated (docker compose up --force-recreate, or a Portainer stack redeploy) before it is actually protected.

So: anything created from now on is safe, and every container currently running is still capable of doing exactly this again. Recreating 28 containers — including immich's postgres and Home Assistant — is a deliberate operation for a quiet moment, not a drive-by. A lighter interim option is a cron job that truncates any container log over a threshold.

7. Residual

• rdt-client now logs occasional TorBoxClient-torbox_client_handler//Timeout resilience events — a few per minute, down from a burst at startup. Remote-side or rate-limiting; worth watching but not blocking.
• The 11 torrents that were queued behind the wedge should now process.
• plex2 is at 89% and plex1 at 95%; ext4 degrades past ~90%.

8. Countermeasures (added same day)

First, the finding that shapes everything else: rdt-client cannot be configured out of this. Its DownloadRetryAttempts (3), TorrentRetryAttempts (1), DeleteOnError (0) and TorrentLifetime (0) all govern failed downloads. The wedge is an unhandled DbUpdateException thrown inside the background service before any retry logic applies, so no setting reaches it. General:LogLevel is already Error and these stack traces are Error-level, so the volume can't be reduced without going dark. Detection and mitigation must live outside the container.

Two layers were added, deliberately separated by role:

• /home/plex/bin/docker-log-guard.sh — bounds the damage. ROOT crontab, every 15 min. Truncates any container json log reaching 500 MB, warns if all logs together pass 2 GB, and alerts by Telegram. Deliberately failure-agnostic: it doesn't care which container misbehaves or why. This matters because the daemon-wide log cap only protects containers created after this change (see §6), and because the next runaway won't be the one we predicted. Thresholds and paths are env-overridable with a DRY_RUN=1 mode so the truncation path can be exercised against a scratch directory rather than taken on trust — both paths were verified before scheduling.
• /home/plex/bin/rdtclient-watchdog.sh — catches it early, and now repairs it. Plex crontab, every 30 min, covering both instances (separate databases: tv at /home/rdtclient/data/, movies at /home/plex/rdtclient-movies/db/). Two checks. A — pre-wedge: looks for the precondition, days before the loop starts. B — wedged: error-rate backstop; more than 50 errors in 10 min means wedged, not unlucky. Calibration is comfortable: ~13 errors/10 min in normal operation versus ~4,800 during the wedge. When it fires it recognises the UNIQUE-constraint signature and includes the exact SQL to find the culprit torrent. It alerts once per episode and sends a recovery notice when the rate returns to normal. It does not auto-repair — deleting rows from a live database unattended is a worse risk than the wedge itself, and the log guard already prevents the damage while it waits for a human.

9. What actually triggered it — not what anyone assumed

Two working assumptions were wrong; the pre-repair backup settled both.

Wrong assumption 1: that finalising by hand caused it. Damien pushed back — the manual copy went to a path rdt-client knew nothing about. He was right. Of 232 Downloads rows, exactly one was incomplete:

Path:        https://torbox.app/fakedl/57169452/47
Added:       2026-07-25 04:05   <- seven days AFTER the torrent (07-18)
RetryCount:  1
DownloadStarted / DownloadFinished / Completed:  all empty

One member failed; rdt-client retried it a week later, leaving one incomplete row. The provider meanwhile reported the torrent 100% complete (RdStatus 4), so UpdateComplete fired and tried to insert a row for index 47 that already existed. Structural, not bad luck — any multi-file torrent with one unfetchable member reproduces it. That makes tv the exposed instance (multi-season packs) and movies far less so.

Wrong assumption 2: that a corrupt file inside a zip was being re-extracted. There was no archive and no unpacking. The torrent held 77 .mp4, 154 .srt, 1 .txt — discrete files, zero archives. Resolving index 47 through Torrents.RdFiles (the provider's file list, stored as JSON) shows the file that took the pipeline down for 16 days was a ~50 KB English subtitle. (The index may be the file's Id or its array position — these differ, as the array is not Id-sorted. Both candidates are .srt files under Subs/, so the conclusion holds either way; the tooling reports both.)

10. Automatic salvage — the behaviour actually wanted

The goal: download everything possible, move what can be moved into position, and say something about the file that failed. Most of that already worked — 231 of 232 files downloaded fine. The broken part was finalisation: the torrent never reaches complete, so FinishedAction never fires and the queue jams behind it.

/home/plex/bin/rdtclient-stuckfiles.py does both jobs. It resolves opaque fakedl indices to real filenames (a failed .srt is a shrug; a failed .mkv wants a Sonarr re-grab), and with --remediate closes stale orphan rows so the torrent finalises and the queue unblocks. The watchdog runs it with salvage on (AUTO_SALVAGE=1) and reports the action over Telegram.

Guards matter, because the pre-wedge shape also matches a torrent legitimately mid-download (provider at 100%, files still arriving):

• the orphan row must be older than 6 hours (the reference case was 9 days);
• the torrent must already have ≥1 completed row, proving it is genuinely partial rather than new;
• the database is copied to a timestamped backup before any write;
• read-only by default — opens mode=ro unless --remediate is passed, and offers --dry-run.

Tested against a throwaway copy of the pre-repair database through all four paths: dry-run inert, age guard refusing, real salvage closing the orphan (232/232, backup written), clean re-scan afterwards.

One honest limitation: that rdt-client finalises cleanly after the rows close could not be verified without reproducing a live wedge. The database side is proven; the application's reaction is not. If salvage ever fails to unblock things, the fallback is deleting the torrent record outright — what actually worked on 2026-08-03. Every write is backed up, so either path is reversible.

Workflow note. Torrents also exposes IncludeRegex / ExcludeRegex for per-torrent file filtering — useful for pre-emptively skipping a member known to be bad.

The lesson to carry: when a symptom names a disk that has recently been worked on, measure that disk first and then keep going. Here the storage was exonerated in five minutes, and the real cause was a single row of bookkeeping that had been quietly amplifying itself into 29 GB since mid-July.