Skip to content

Plan — Operational hardening (health, SSRF, auth leaks, job claiming, pins)

Scope: five production-readiness fixes in the rag backend, no schema changes.

Problems

  1. /health is a static stub. It returns 200 unconditionally, so a pod that cannot reach Supabase or publish usage.recorded billing events to RabbitMQ keeps receiving traffic — and silently drops charges.
  2. Unpinned critical dependencies. requirements.txt leaves supabase and aio-pika floating while the scraper pins them; a rebuild can pull an untested major version.
  3. SSRF via references.pdf_source. Admin endpoints (and ingestion) fetch arbitrary URLs from the database with redirects enabled — a crafted row can make the backend request internal services (Kong, host Postgres, metadata endpoints).
  4. Auth errors echo internals. get_current_user returns str(e) from Supabase/GoTrue failures to the client (detail=f"Authentication failed: {e}").
  5. Ingestion job claiming is select-then-run. The drain loop and /admin/jobs/dispatch select the oldest queued job and rely on execute_job's embedded conditional update; losers of the race idle a full cycle, and the claim isn't reusable or observable.

Approach

  1. Liveness vs readiness. /health (+ /health/live) stays a dependency-free liveness probe (Docker healthcheck and deploy curl it). New /health/ready runs concurrent, timeout-bounded checks — a PostgREST round-trip and a RabbitMQ publisher ping() — and returns 503 when either fails so the pod leaves rotation. No broker configured counts as failure only in production.
  2. Pin supabase==2.31.0 and aio-pika>=9.4,<10 — the scraper's tested specifiers.
  3. Allowlist fetcher. New app/services/pdf_fetcher.py: scheme must be http(s), host must match PDF_SOURCE_ALLOWED_HOSTS (default koutoubi.mr, subdomains included). The Supabase host is NOT allowlisted wholesale — in prod SUPABASE_URL is the internal Kong gateway, which also fronts GoTrue/PostgREST — only Storage presigned URLs on the exact configured origin (scheme + host + port, /storage/v1/object/sign/ prefix) pass. Private/loopback/link-local IP literals and localhost-style names are rejected outright (defense-in-depth). Redirects are followed manually with every hop re-validated. Admin endpoints return 400 on violation; ingestion fails the job.
  4. Generic auth errors. Log the underlying exception server-side, return "Authentication failed" to the client.
  5. Claim-before-execute. IngestionService.claim_job() is the single atomic claim (conditional queued -> parsing update; Postgres row locking guarantees one winner) and records the claiming worker_id in the audit trail. The drain loop and dispatch endpoint claim from a small candidate batch and fall through to the next job when a claim is lost. A FOR UPDATE SKIP LOCKED claim-next RPC (+ claimed_by/claimed_at columns) needs a migration in the migration repo — follow-up there; rag is the sole owner of the ingestion_jobs lifecycle (scraper stops inserting into it in a separate task).

Out of scope by design: the in-memory rate limiter stays a best-effort backstop (documented in code); the gateway throttler is the enforcement point.