Skip to content

Issue #26 Implementation Doc Stub

1. Issue reference

  • GitHub issue: #26
  • Issue title: [Backend][Ingestion] Orchestrate reference_id handoff from scrape storage
  • Issue type: feature
  • Milestone: Backend RAG M2 - Scrape to Ingest Pipeline

2. Summary

  • What this issue changed: TBD
  • Why the change was needed: connect discovered scrape records to the ingestion path through stable reference_id handoff.

3. Initial repo state

  • Relevant behavior before implementation: the current release todo marks scrape-to-ingest handoff as a separate execution step.
  • Known constraints or gaps at start: there is no dedicated plan doc under docs/plans/ tied to this handoff workflow yet.

4. Plan doc referenced

  • Plan doc path: TBD - create a reference handoff orchestration plan under docs/plans/
  • Plan status at implementation start: TBD
  • Was the plan updated during implementation?: TBD
  • If yes, what changed in the plan?: TBD

5. Decisions taken

Decision Reason Alternative rejected
Stub only Reserve the implementation record before code changes start. Capturing handoff details only after code lands.

6. Files changed

File Change summary
docs/implementation/issue-26-orchestrate-reference-id-handoff-from-scrape-storage.md Created implementation doc stub.

7. Migrations / schema changes

  • Migration files: none yet
  • Schema changes: none yet
  • Data backfill or manual steps: none yet
  • Rollback notes: not applicable yet

8. API changes

Surface Change Compatibility impact
None yet No implementation details recorded yet. None

9. Tests added or updated

Test file or suite Change
None yet No implementation work has started.

10. Risks / caveats

  • Handoff semantics should be documented before any ingestion state machine or DB workflow changes are merged.

11. Follow-up work

  • Create the plan doc that defines the handoff contract.
  • Replace placeholders with actual workflow, DB, and API details during implementation.

12. Final repo state

  • Relevant behavior after implementation: not implemented yet; stub created to anchor the issue.
  • Remaining limitations: plan, implementation, and validation details are still pending.

13. Docs updated

Doc path Update summary
docs/implementation/issue-26-orchestrate-reference-id-handoff-from-scrape-storage.md Added initial implementation doc stub.

14. Update — scheduled handoff consumer (feat/handoff-consumer)

IngestionHandoffService.orchestrate() previously had zero callers outside its own tests; queued ingestion_handoffs rows were only ever created, never consumed. This update wires it into the APScheduler:

  • app/services/ingestion_handoff.py — new claim_handoff(handoff_id) (atomic conditional queued -> running UPDATE, same pattern as IngestionService.claim_job; lost race = skip) and consume_queued(limit=10) (select oldest queued rows, skip exhausted attempts, claim, then orchestrate() each claimed row into an ingestion_jobs + ingestion_audit record).
  • app/services/background_jobs.py — consume_queued_ingestion_handoffs() runs every 60s (handoff_consumer, max_instances=1), executing the sync consumer in an executor; never raises into the scheduler.
  • Contract: scraper PR #183 removes the scraper's direct ingestion_jobs writes and must not merge before this ships. Failed handoffs are intentionally re-queued by the scraper on its next scrape of the same payload — see docs/60_ingestion/handoff-lifecycle.md.
  • Validation: end-to-end test (tests/services/test_handoff_to_ready_pipeline.py) proves a queued handoff row ends as a ready ingestion job against a stateful in-memory Supabase fake; unit tests cover claim atomicity, lost-race skip, and exhausted-attempt skip.

Addendum — stale-running recovery

A handoff claimed queued -> running was permanently stranded if the process restarted before orchestration's terminal update (consume_queued selects queued only; the scraper only re-queues failed). New stale_handoff_reaper scheduler job (every 5m) resets rows stuck in running > STALE_HANDOFF_TIMEOUT_MINUTES (default 15) back to queued (reason_code=requeued_stale_running), leaving attempt_count untouched.