Skip to content

gh-342 — Repeat-fingerprint asset filter (kill IPN-watermark false figures)

  • GitHub issue: #342 ("figure extraction returns decorative page borders")
  • Status: implemented (code fix + tests + read-only audit script). Cleanup/backfill of already-persisted bad assets is deferred to a separate, human-run post-canary step.

Problem

Every curriculum PDF from the government institute IPN carries a copyright watermark/background on every page. For a text-only curriculum with no real figures, the figure-extraction step treats that recurring watermark (and decorative page borders — the original #342 report) as figures. The pipeline then runs a gpt-4o vision call per bogus candidate, wasting tokens, and persists the results into assets + chunks, polluting search_chunks_hybrid retrieval.

Root cause

app/services/asset_extractor.py judged "is this a figure" from geometry + xref identity only, and the two extraction paths had asymmetric, evadable guards:

  • Raster path dropped an image whose xref appeared on > MAX_REPEAT_PAGES (5) pages. A watermark that the PDF producer re-embeds with a distinct xref on every page has an xref-count of 1 per page, so it never tripped the filter.
  • Vector path (_add_vector_candidates) had no repeat filter at all — a drawn watermark or border was rendered on every page with zero dedup.
  • The MAX_PAGE_AREA (0.85) magic number let margin-inset full-page backgrounds and edge-hugging border bands slip through.
  • There was no text-only short-circuit: a document with zero real figures still produced N candidates, N vision calls, and N persisted rows.

Fix (implemented)

All filtering lives inside AssetExtractor.extract() over a whole-document view, so every caller (ingestion, admin preview, admin persist, backfill) inherits it — one runtime path.

  1. Content-based repeat-fingerprint filter (both paths). An 8×8 average hash (aHash) computed with fitz alone (no new dependencies — no Pillow/imagehash/numpy). Candidates are bucketed by Hamming distance ≤ HAMMING_TOL (4); a bucket spanning more than max(MAX_REPEAT_PAGES, ASSET_REPEAT_PAGE_FRACTION × total_pages) pages is dropped. This catches the distinct-xref-per-page watermark and #342 borders. The legacy shared-xref count is kept as a fast path for logos that share one xref.
  2. Page-dominating coverage rule (_is_page_dominating) replaces the 0.85 magic number: drop if coverage ≥ ASSET_MAX_PAGE_COVERAGE (0.80), or near-full in both axes (≥ 0.92 × 0.92), or a full-width/height edge band (≥ 0.95 on one axis, ≤ 0.10 on the other — the classic decorative border).
  3. Text-only short-circuit. After filtering, if fewer than ASSET_MIN_DISTINCT_FIGURES (1) figures survive, extract() returns []. AssetService.persist returns a text_only summary without any vision call, embedding, or DB write.
  4. Restructure: the whole document is scanned before truncation; max_assets is applied to the survivors last, so a small preview limit can never corrupt the repeat counts. Vector clusters are fingerprinted at a tiny DPI and only rendered at full DPI if they survive — the vector path gets cheaper.

New tunables in app/core/config.py: ASSET_REPEAT_PAGE_FRACTION, ASSET_MAX_PAGE_COVERAGE, ASSET_MIN_DISTINCT_FIGURES (safe defaults; operator-tunable without redeploy logic changes).

Admin preview (POST /admin/references/{id}/assets/preview) now returns a filter_summary (total_pages, raw_candidates, dropped_repeat, dropped_coverage, kept, verdict) so an admin sees why a text-only document yields zero figures instead of a bare empty list.

Token-cost impact

  • Ingestion: a text-only IPN document previously burned up to INGEST_ASSET_LIMIT (10) gpt-4o vision calls of pure garbage per doc → now 0 (the dominant case for this corpus).
  • Preview: up to 20 gpt-4o calls per dry-run on a text-only doc → 0.
  • Fingerprinting adds no API cost (local fitz rasterization + stdlib hash), well within the backend container's 1g envelope.
  • Retrieval quality improves — bogus watermark figure-chunks stop surfacing in hybrid search.

Cleanup / backfill plan (DEFERRED — human-run, post-canary; NOT in this PR)

The code fix stops new pollution. Existing bad rows in assets/chunks are handled separately, after canary validation, by a human:

  • Step A — Audit (read-only). scripts/audit_bad_assets.py re-runs the fixed extractor per reference and flags persisted assets the new filter would drop (plus stored-data heuristics: page-dominating bbox, description repeated across many pages). Emits a JSON report. No writes. Canary reference f170a1a4-837f-40ae-bf23-60407f198f7a should light up fully. This script is included in this PR; running it is deferred.
  • Step B — Delete confirmed bad rows. Deleting from assets cascades to the paired figure-chunks (chunks.asset_id is ON DELETE CASCADE). App-side batched supabase.table("assets").delete().in_("id", bad_ids) (no migration), or a coordinated forward-only SQL in the migration/ project. Snapshot rows before deleting; scope to the audited asset_id list only — never a blanket reference_id wipe. Not run in this PR.
  • Step C — Re-backfill. scripts/backfill_assets.py (already resumable). Text-only references now short-circuit and persist nothing. Not run in this PR.
  • Idempotency check: re-running the audit after cleanup reports zero flags.

Rollout order

  1. Land the code fix + tests + audit script (this PR) → merge auto-deploys to the VPS.
  2. Validate on the canary via the admin preview endpoint (expect verdict: text_only, watermark count in dropped_repeat, zero vision calls).
  3. Run the read-only audit → human review of the flagged set.
  4. Delete confirmed bad rows → re-backfill → re-audit (expect zero flags).

Tests

tests/services/test_asset_extractor.py, tests/services/test_asset_service.py, tests/routers/test_admin_assets.py — see the implementation record for the full list.