Implementation — Operational hardening
Plan: 95_plans/operational-hardening.md
What shipped
app/services/health.py(new) —check_supabase()(PostgRESTreferencesprobe in an executor),check_rabbitmq()(publisherping(); unconfigured broker fails only in prod),readiness()gathering both with 5s timeouts.app/api/router.py—/healthand/health/liveare liveness-only (unchanged 200 body, so the Docker healthcheck and deploy probe keep working);/health/readyreturns 200/503 with per-check statuses. Check details are logged, not exposed.app/events/publisher.py—UsagePublisher.ping()verifies connection/channel and drops the cached exchange on a dead channel so the next publish re-establishes it.requirements.txt—supabase==2.31.0,aio-pika>=9.4,<10(scraper's specifiers).app/services/pdf_fetcher.py(new) — SSRF-safefetch_pdf(): http(s) only, host allowlist (PDF_SOURCE_ALLOWED_HOSTSsetting, defaultkoutoubi.mr). The Supabase origin is admitted only for Storage presigned URLs — exact scheme + host + port match againstSUPABASE_URLand the/storage/v1/object/sign/path prefix — because in prod that origin is the internal Kong gateway (GoTrue/PostgREST live there too). Private/loopback/link-local IP literals and localhost-style names are rejected outright as defense-in-depth. Manual redirect following (max 5) re-validates each hop; a redirect without a Location header raises an explicitHTTPStatusError. Used by all three admin PDF endpoints (400 on violation, generic 502 on download failure with the cause logged server-side) and the ingestion pipeline (job fails withPDF download failed: ...).app/core/auth.py— token-verification failures log the real cause viaour_logsand return a generic 401"Authentication failed"; no internal exception text reaches clients.app/services/ingestion.py— newclaim_job(job_id): atomic conditionalqueued -> parsingupdate, audit entry recordsworker_id(host:pid).execute_job(..., claimed=False)claims unless the caller already won.app/services/background_jobs.py— the drain loop selects up to 5 queued candidates and claims before executing, falling through when another instance wins a claim.app/api/routers/admin.py—/admin/jobs/dispatchclaims before scheduling the background task (claimed=True), trying up to 5 candidates.app/core/middleware.py— documented that the in-memory rate limiter is a per-process backstop only; the gateway throttler is the enforcement point (deliberately not extended).
Notes / follow-ups
- A
FOR UPDATE SKIP LOCKEDclaim-next RPC withclaimed_by/claimed_atcolumns requires a migration (migration repo) — the conditional-update claim already guarantees a single winner; the RPC would only let losers skip contended rows server-side. - Scraper stops inserting into
ingestion_jobsin a separate task (rag owns the lifecycle).
Verification
pytest: 282 passed (36 new: pdf_fetcher allowlist/redirect cases, health liveness +
readiness matrix, auth generic-error cases, claim/dispatch/drain race cases). ruff check
clean.