SKYHOUSE.dev Journal

Why: While auditing what was filling /media/plex1 we looked at Photos/Damien — 263 GB that both the DR audit and Damien's own plan had written off as a duplicate of what Immich already held, and which was queued for deletion. It wasn't a duplicate. 80% of it had never been imported at all.

The 2026-08-04 DR audit left an explicit open question: "Damien/ (259 G) may duplicate library/ if the Google Takeout was imported into Immich — check before seeding." This is that check, and the answer was the uncomfortable one.

1. Four-fifths of it was missing

First pass matched filenames against Immich's originalFileName: 18.6% present. That number is not trustworthy on its own — the 2026-08-05 entry says exactly why ("this filename isn't in the database never proves an asset is missing"), because Immich deduplicates on content and immich-go renames on collision. So it was re-tested against content:

• asset.checksum was first confirmed to be a plain SHA-1 of the file bytes, by hashing files Immich already held and matching them to their rows.
• A 60-file random sample then hashed at 23% present / 77% absent — consistent with the filename pass, so the gap was real rather than a naming artifact.
• A full pass over all 92,575 jpgs settled it: 22,328 present, 70,247 absent, 208.9 GB.

Worth noting the two methods disagreed by ~4,200 files in the direction that matters: content matching found more already present than filenames did. Had we acted on the filename number we would have re-uploaded thousands of duplicates.

2. It was Flickr, not Google Takeout

The premise everyone had been working from was wrong in a second way. These are Flickr exports — _o.jpg suffixes and numeric Flickr photo ids — not Google Takeout. That is why the August Takeout import never covered them: it was never pointed at them. Two unextracted Flickr archives were also sitting in the folder, and data-download-193.zip turned out to be 39 of 41 files absent, including 8 videos and 4 PNGs — the only videos in the whole set, never seen by any previous run because nobody had unzipped it.

3. The import

Staged by hardlink into /media/plex1/_flickr-ingest — 195 GiB of files at zero bytes of disk cost, with the originals untouched and unreachable by the importer. Proven on 20 files first (20 uploaded, Postgres +20, all re-confirmed by hash) before committing to the rest.

immich-go upload from-folder, which is the documented fix for sidecar-less files. 2h20m at ~450 files/min:

• uploaded successfully 70,239 · server asset upgraded 9 · discarded local duplicate 23 — totalling exactly the 70,271 processed.
• Errors 0, Discarded-as-failure 0, Pending 0. Pending is the counter that hides the silent-skip failure mode; it stayed at zero.
• Postgres went 94,610 → 164,849, a delta of exactly 70,239 — matching the tool's own count, and measured against the asset table rather than /api/assets/statistics, per the 2026-08-09 correction.

Then the check that does not take immich-go's word for anything: every staged file re-hashed and looked up in asset. 70,271 verified, 70,271 present, 0 missing.

4. 3,840 photos have no date, and never did

An honest defect worth recording. 3,840 of the imported files (5.5%) carry no EXIF date whatsoever — exiftool returns empty for both DateTimeOriginal and CreateDate; Flickr stripped it. Immich did the only thing available and fell back to file mtime, so all 3,840 now sit on 2023-03-08, the day the Flickr export was generated.

This is not an import bug — the date is genuinely not in the file, and any tool would land the same way. It is recoverable only from Flickr's separate metadata archives (JSON per photo, carrying the true date taken), which are not on this server. Some filenames encode dates (7-jan-11-1148_21524162825_o.jpg) but only some, so nothing was guessed. The other ~66,000 imported onto correct dates spanning 1983–2018.

5. The de-duplication, and why the obvious rule was wrong

Damien's rule: for a Flickr photo held at several sizes, keep the original and drop the compressed copies — unless a copy is in an album, in which case promote the original into that album instead.

The obvious implementation is "keep the _o". It deletes the wrong file.

seaf-2011_5762196700_o.jpg                  4288x2848  4.35 MB   <- the real original
5762196700_8b1ae20a19_b_21537057631_o.jpg    680x1024  0.29 MB   <- a 1024px copy
5762196700_8b1ae20a19_b.jpg                  680x1024  0.11 MB

Two of those end in _o. What happened is that the 1024px _b render was re-uploaded to Flickr years later, earned a new photo id, and was exported as the "original" of that second upload. The suffix is telling the truth about the wrong thing. So bin/immich-flickr-dedupe.py groups on the first Flickr id in the name — always the original photo's — and ranks on actual pixels, then bytes. (The id must be 8+ digits: a 7-digit floor matches camera filenames like P1110255.JPG and silently groups unrelated photos.)

Albums were the majority of the work, not an edge case: 426 of 783 retirable assets were in at least one album, so plain deletion would have quietly shrunk them. The keeper is added to each of the loser's albums first, and retirement only proceeds if that succeeded.

Result: 783 retired, 764 album additions, 0 failures. Verified afterwards across every audit row — 0 album-integrity failures, 0 keepers wrongly trashed. Retirement is a soft delete into Immich's 30-day trash, and every decision is in a CSV with both asset ids, resolutions, byte sizes and album ids, so a single decision can be reversed rather than only the whole batch.

6. What this pass cannot see

Filename matching only ever finds Flickr-vs-Flickr. 73,723 of the assets carry no Flickr id at all — the Google Photos and iPhone ones — so a compressed copy that arrived through Google Photos as IMG_4472.JPG is structurally invisible to it, and the 783 recovered only 2.1 GB. This pass is library hygiene, not a space saving.

The tool for the rest already exists: bin/immich-nearmiss-dupes.py, which queries the CLIP embeddings already in smart_search for the band just above Immich's own conservative 0.01 cutoff — precisely the "same picture, re-encoded and resized" case. That is where the real volume of compressed copies will be.

7. The source directory, deleted

Later the same day, with the import reconciled and the de-duplication verified, Damien gave the go-ahead to remove the source. Two checks ran first. A final 500-file spot check by content hash (0 absent), and — because the server had been rebooted in the meantime and the device node had moved from sdf1 to sdh1 — a UUID check that /media/plex1 was even the disk we thought it was. It was: 73130bac-9c4c-42d1-90bd-ef9838716e87.

The staging directory had to go too, and this is the part that is easy to get wrong. The 70,247 staged files were hardlinks to the same inodes as the originals, so deleting Photos/Damien on its own would have freed nothing. Both paths had to be removed before a single byte came back. (Files that had been staged showed links=2; the 22,328 already in Immich, never staged, showed links=1.)

Result: 751 GB → 1009 GB free, 258 GB reclaimed, plex1 back from 94% to 92%. The two original Flickr archives were kept aside at /media/plex3/Downloads/flickr-export-reference/ — 611 MB, retained purely so the structure of the next export can be compared against a known one.

Net effect: 70,239 photographs and videos that existed only as loose files on a disk earmarked for deletion are now in the library, content-verified; 783 redundant copies retired without losing a single album placement; a 263 GB directory that was about to be deleted turned out to be four-fifths irreplaceable; and once it genuinely was redundant, it was removed for 258 GB back.