Skip to content

gh-342 — Repeat-fingerprint asset filter (implementation record)

1. Issue reference

  • GitHub issue: #342
  • Issue title: figure extraction returns decorative page borders (extended: full-page IPN copyright watermark on text-only curriculum PDFs)
  • Issue type: bug (retrieval pollution + wasted vision tokens)

2. Summary

  • What changed: figure/asset extraction now filters recurring page-level watermarks/borders with a content-based (aHash) repeat-fingerprint applied to both the raster and vector paths, plus a page-dominating coverage rule and a text-only short-circuit. Text-only documents no longer trigger any gpt-4o vision call or DB write.
  • Why: IPN curriculum PDFs carry a copyright watermark on every page; the old geometry+xref filters treated it (and #342 decorative borders) as figures, burning a vision call per bogus candidate and polluting assets/chunks and hybrid retrieval.

3. Initial repo state

  • AssetExtractor.extract dropped shared-xref repeats (> MAX_REPEAT_PAGES) and images over MAX_PAGE_AREA (0.85); the vector path had no repeat filter; no text-only short-circuit.
  • AssetService.persist always ran vision per candidate and persisted anything described.

4. Plan doc referenced

  • Plan doc: docs/95_plans/gh-342-repeat-fingerprint-asset-filter.md
  • Plan updated during implementation: no material change from the architect's design.

5. Decisions taken

Decision Reason Alternative rejected
aHash via fitz + stdlib No new dependency (frugal box) Pillow / imagehash / numpy
Filter inside extract() One runtime path; every caller inherits it Filtering per-caller
max(floor, fraction×pages) repeat threshold Majority-of-pages catches watermarks; floor protects genuine few-page recurrences Fixed page count only
Keep legacy shared-xref fast path Distinct-xref watermark needs the fingerprint; shared-xref logo is cheaper to drop by count Replace it entirely
Text-only short-circuit in persist Guarantees zero vision/embed/DB on empty extract Rely on the empty loop alone
Cleanup/backfill deferred Deleting persisted rows is irreversible; must be human-run post-canary Auto-delete in this PR

6. Files changed

File Change summary
app/services/asset_extractor.py Repeat-fingerprint (aHash) + coverage rule on both paths; whole-doc collect, truncate last; deferred vector render; extract_with_report() + ExtractionReport; __init__ reads tunables
app/services/asset_service.py Text-only short-circuit: empty extract → text_only summary, no vision/embed/persist
app/api/routers/admin.py Preview returns filter_summary via extract_with_report()
app/core/config.py ASSET_REPEAT_PAGE_FRACTION, ASSET_MAX_PAGE_COVERAGE, ASSET_MIN_DISTINCT_FIGURES
scripts/audit_bad_assets.py New READ-ONLY audit of persisted bad assets (#342 cleanup step A)
docs/95_plans/gh-342-repeat-fingerprint-asset-filter.md Plan doc

7. Migrations / schema changes

  • None. No schema or RPC change for the core fix.

8. Tests

Added / updated (all mock-based, no network): - tests/services/test_asset_extractor.py: - test_drops_margin_inset_background — ~0.82-coverage background now dropped (old 0.85 kept it) - test_drops_full_width_border_band — edge-hugging decorative band dropped (#342) - test_drops_repeating_watermark_distinct_xref — the IPN bug: distinct xref per page dropped, genuine figure survives - test_drops_repeating_watermark_shared_xref — regression guard for the legacy shared-xref path - test_repeating_vector_cluster_dropped — identical drawn cluster on every page dropped (vector path had no filter) - test_genuine_figure_recurring_under_threshold_survives — a real figure on 2 pages is not over-filtered - test_text_only_document_short_circuits — watermark-only doc → [], verdict=text_only - test_max_assets_truncates_after_filtering — cap applied to survivors; repeat counts over the whole doc - tests/services/test_asset_service.py: - test_persist_short_circuits_on_text_only_document — asserts vision NOT called, no embeddings, no assets/chunks writes - tests/routers/test_admin_assets.py: - preview tests updated to mock extract_with_report and assert the filter_summary payload

9. Deferred (human-run, post-canary — NOT in this PR)

  • Running scripts/audit_bad_assets.py against prod.
  • Deleting the flagged assets rows (cascades to figure-chunks).
  • Re-running scripts/backfill_assets.py, then re-auditing for zero flags.