SKYHOUSE.dev Journal

Why: Damien asked to look at recent Radarr errors. The pipeline had been dead for sixteen hours, the monitoring had detected it correctly — and the one alert that named the cause had been physically incapable of reaching him since the day it was written.

1. The wedge, with an exact cause this time

rdtclient-movies's TaskRunner had been throwing since 2026-08-08 17:00: 121,112 UNIQUE constraint failed: Downloads.TorrentId, Downloads.Path exceptions in 48 hours, about 5,380 every ten minutes, with zero bytes moved.

The culprit was Dont.Worry.Darling.2022.1080p.BluRay.AVC.TrueHD.7.1.Atmos-FGT — a full Blu-ray disc structure, 160 files, including a BDMV/BACKUP/ tree that mirrors the main one.

A correction, made the same day. The first version of this entry claimed rdt-client had "resolved the same file twice, once by identifier and once by position," citing the watchdog's [by Id] / [by position] output. That was a misreading of the tool's own caveat. Those labels are the diagnostic's two candidate interpretations of the <n> in torbox.app/fakedl/<tid>/<n> — it has never been established whether that is the file's Id or its array index, so both guesses are printed for the same row. Reading a stated uncertainty as a finding is a particularly easy mistake to make when the wrong reading is a tidier story.

What the pre-deletion database backup actually shows, checked afterwards: the provider's file list was clean — 160 entries, all selected, Ids contiguous 0–159, no duplicate Ids and no duplicate paths. So the collision does not come from the provider's data. It happens during the unrestrict step (Cannot unrestrict link appears alongside every exception): rows have their Path rewritten to a resolved URL, and two rows ended up assigned the same one. The precise mechanism inside rdt-client is not established from the outside, and this entry no longer pretends otherwise.

What is established is the shape of the trigger: large multi-file torrents. The 2026-08-03 case was a 232-file pack; this one is a 160-file disc rip. Ordinary single-file releases do not do this.

Why one bad row kills everything: each tick processes every torrent and commits the lot in a single transaction. The rollback discards the whole transaction, including all the correct work done for every other torrent in that pass. Then the timer fires again, on unchanged data, into the same collision. It cannot even drop the offending torrent by itself, because recording that removal is another write inside the same doomed transaction. The only exits are from outside. That is the whole meaning of "wedged": not one file stuck somewhere, but one unprocessable row holding the loop hostage so nothing else can complete either.

A signature worth memorising: an item sitting at exactly 50% in Radarr means the provider finished and the local transfer never began — rdt-client averages provider progress with local progress. The 2026-08-04 entry noted items "frozen at exactly 50%" without explaining it; this is the explanation. Full disc rips are the trigger to watch: a BDMV/BACKUP/ tree is practically built to produce duplicate paths.

1b. The blind spot: dead on arrival

The watchdog detected this correctly at 17:00 on the 8th and kept detecting it every 30 minutes for sixteen hours. Auto-salvage declined to act every single time — and was right to, by its own rules. rdtclient-stuckfiles.py requires the torrent to have ≥1 COMPLETED row before it will close orphan rows, precisely so it never touches a download that is legitimately still arriving.

Don't Worry Darling had 0 of 160 completed. That guard is correct at minute five and useless at hour sixteen: provider says 100%, nothing has ever completed, and half a day has passed is not a slow start, it is dead.

So that shape is now its own classification — [DEAD ON ARRIVAL], meaning zero completed rows and the oldest orphan older than --doa-hours (default 12). Salvage still refuses, correctly: there is nothing on disk to finalise, and closing the rows would tell rdt-client a file arrived that never did. Instead the report names the condition, prints the age, and gives the exact removal command, because removing a torrent record so the arr re-grabs a different release is a bigger decision than closing an orphan row and stays a human's call. Replayed against the pre-deletion backup, it produces:

CD3C85A4  Dont.Worry.Darling.2022.1080p.BluRay...  [DEAD ON ARRIVAL]
    160 of 160 files never completed, oldest row 17.7h old

The watchdog runs this per instance, so the TV instance gets the same detection for free — disc rips are rarer there, but the failure is not movie-specific and the coverage should not be either.

2. The alert that could never have arrived

The watchdog did its job. It detected the pre-wedge state at 17:00 on the 8th and fired. The delivery audit log — added 2026-08-04 for exactly this purpose — shows what happened next:

SEND-FAILED rc=0 http=400  subject=rdt-client pre-wedge: rdtclient-movies
sent        http=200       subject=rdt-client WEDGED: rdtclient-movies
SEND-FAILED rc=0 http=400  subject=Docker log guard: runaway log truncated

Damien got "WEDGED". He never got pre-wedge — the alert that names the torrent and lists the stuck files. telegram_notify.sh sends with parse_mode=HTML and interpolated the body raw, and the pre-wedge template contains a SQL remediation snippet:

sqlite3 $db "DELETE FROM Downloads WHERE TorrentId LIKE '<id>%';"

Telegram reads <id> as an unknown start tag and rejects the entire message. The most diagnostic alert on this box was undeliverable from the moment it was written, and its own remediation instructions were what made it so.

Two wrong hypotheses, corrected by testing rather than reasoning. First guess: the message exceeded Telegram's 4096-character limit. Measured — 1,040 characters. Second guess: the -> arrows in the body. Reconstructed the real payload and sent it: HTTP 200, delivered fine. Only then did grepping the template for <tag> patterns find <id>, and a minimal reproduction gave the answer verbatim:

{"ok":false,"error_code":400,"description":"Bad Request: can't parse
 entities: Unsupported start tag \"id\" at byte offset 71"}

That description had been available on every failed send since 2026-08-04 and was being discarded by -o /dev/null. The script logged that a send failed but never why — a smaller instance of the same silent-failure shape it exists to eliminate.

3. The transport, rewritten

• The body is HTML-escaped by default (& first, or it double-escapes its own output). Every watchdog on this box emits arbitrary text — file paths, log excerpts, process names — and none of it can be trusted as markup.
• TELEGRAM_HTML=1 opts out, for the one caller that deliberately composes markup (arr-remediation-ledger.py, which interpolates only titles and paths it controls). Everything else stays escaped.
• Truncation at 4000 characters, on a character boundary rather than a byte one — slicing a multi-byte glyph in half would itself produce a 400.
• The API's error description is logged. Captured greedily to the last quote, because the description embeds escaped quotes (Unsupported start tag \"id\") and a lazy match clips it exactly where the useful part starts — the tag name is the only bit that tells you what to fix.

Verified live: the exact payload that had been failing since the 8th now returns sent http=200; the TELEGRAM_HTML=1 path still renders bold and code; and a deliberately malformed message fails with reason=Bad Request: can't parse entities: Unsupported start tag \"unclosed\" at byte offset 49 in the audit log.

4. Why nothing ever self-cleaned

Damien asked whether he should have to clear rdt-client records by hand. He should not, and the reason we have been doing it manually for a week is one setting:

Provider:Default:FinishedAction = 0    ("No Action") — on BOTH instances
   0 = No Action
   1 = Remove Torrent From Client And Provider
   2 = Remove Torrent From Provider
   3 = Remove Torrent From Client

The arrs download through the Provider profile, so that is the one that governs. Both instances were set to take no action, which is precisely why finished torrents accumulated until someone deleted them. (The other profiles were partly set to 1 — TV's Integrations/Gui/Watch, Movies' Watch — so this was configured once and the profile that matters was missed.)

Set to 3 (Remove From Client) with a 60-minute FinishedActionDelay on both. Rationale: 3 rather than 1 keeps the cached copy at TorBox, so a failed import is still recoverable; the delay preserves the arr's import window, because an arr that loses its download-client record mid-import loses track of the import. Provider-side retention stays a separate concern for torbox-cleanup.py.

Correction, 2026-08-21: this setting never fired, and the reasoning above picked the wrong profile. Arr grabs do not use the Provider profile — QBittorrent.TorrentsAddMagnet hardcodes FinishedAction = None on every record it creates, so an arr-added torrent's finished action is always 0 whatever the UI shows. The only setting that has any effect is Integrations:Default:FinishedAction, read when the arr asks for a removal. Movies had it on "Do nothing" and was silently discarding every request Radarr sent. See 2026-08-21 — The Setting That Threw Away Every Removal.

5. Refusing the trigger outright — the fix that beats detecting it

Everything above detects and recovers from the wedge. Damien asked the better question: why accept raw disc rips at all, when no machine here can turn one into a streamable file? The answer is that the arrs already have a name for exactly this thing, and Radarr was configured to prefer it.

Dont.Worry.Darling.2022.1080p.BluRay.AVC.TrueHD.7.1.Atmos-FGT
    parsed quality: BR-DISK  (source=bluray, res=1080)

Radarr classified it correctly. The problem was the profile:

That asymmetry is the whole reason this is a movies-only failure. And the ranking made it worse: BR-DISK sat above Remux-2160p, i.e. top of the profile. Radarr was not grabbing disc rips by accident — it had been told they were the best available thing.

Cost of refusing it: essentially nothing. Remux-1080p and Remux-2160p stay allowed — single-file, lossless, streamable — so the highest usable quality is untouched. Only two movies held BR-DISK files. On the tooling side, ffmpeg is present but HandBrakeCLI, mkvmerge and makemkvcon are not; ffmpeg cannot read .mpls playlists or handle seamless branching, so extracting a main title from a BDMV tree is manual per-disc work that nothing in this pipeline can automate. Forty gigabytes for a file no machine here can play.

BR-DISK is now disallowed in every Radarr profile (verified by reading the profiles back, not by trusting the API's 202; the cutoff was checked first — Bluray-480p, id 20, not BR-DISK, id 22, so the profile stays valid). This removes the trigger class rather than detecting it after the fact.

An unplanned demonstration. Deleting the two BR-DISK files and rescanning produced hasFile=False on both — Radarr refused to adopt the legacy .mp4 sitting in each folder, because Harry Potter and the Chamber of Secr.mp4 carries a truncated name with no year and no quality tag. That is precisely the unparseable-legacy-filename mechanism documented on 2026-08-05, reproducing itself on demand: both titles are now monitored, missing, with a video file in the folder — the "primed to duplicate" state, created as a direct side effect of a cleanup.

6. Verification

• Culprit record deleted via the rdt-client API with deleteData only (TorBox and local files untouched): 38 → 37 torrents, 0 rows remaining.
• Container restarted. The 440 errors in the first two minutes were a startup burst as rdt-client re-walked its torrent list, not a persistent fault — zero in every window since.
• Both pipelines UP, both queues empty, 0 errored records in either instance (movies 37, TV 6).
• Watchdog: pre-wedge check clean, recovered (0 errs/10m).
• Settings read back from the SQLite file, not trusted from the API's 200 — this app returns 200 unconditionally, a lesson from 2026-08-04.

Known wrinkle: the watchdog's error-rate window is 10 minutes, so a deliberate restart re-triggers a WEDGED alert for one cycle before rolling off. It did so here at 10:07 and cleared on the next run.

The monitoring was right about everything. It detected the wedge within the hour, identified the exact torrent and the exact colliding files, and wrote a remediation snippet — and that snippet contained an angle bracket, so the message was rejected before it ever left the building.