Skip to content

Reasoning-grounded synthesis (Issue #31)

Summary

The final answer synthesis stage now uses a dedicated OpenAI reasoning model (OPENAI_REASONING_MODEL, default o4-mini) with explicit grounding and safety constraints.

Prompt contract

app/services/llm.py::generate_answer now enforces this synthesis contract:

  1. Use only facts grounded in retrieved context snippets.
  2. Cross-language AND multi-snippet synthesis are grounded: the snippets may be in a different language than the required answer language, and the full answer may need facts drawn from several different snippets. Translating, paraphrasing, or combining facts FROM the snippets — across languages or across multiple snippets — into one coherent answer is expected behavior, never a reason to refuse (F4 fix, rounds 1–2 — see ../95_plans/arabic-grounded-generation.md).
  3. Include source citations (e.g., [1], [2]) for factual claims.
  4. If the needed information is genuinely absent from the snippets (in any language), explicitly return uncertainty instead of guessing.
  5. Write the entire answer in the requested language and script (Arabic script for ar, Latin for fr/en); standard math notation, numbers, and [n] citations are exempt; no words or characters from any other language or script. Student-requested format/style is preserved.

For exercise-type questions, the model is further constrained to provide guided hints rather than full final solutions.

Guardrails

1) Empty-context fallback

If context is empty or set to NO_CONTEXT_FOUND, answer generation bypasses model calls and returns a deterministic uncertainty response in the target language.

2) Hallucination guard

When retrieved context includes citation IDs, final non-streaming answers are post-validated: - if no citations appear in the answer, or - if the answer includes citation IDs that are not present in the retrieved context, and - the answer is not already an uncertainty response,

then the response is replaced with an uncertainty-safe fallback plus available citation IDs.

Citation parsing accepts compact ([1]), comma-separated ([1, 2]), and range ([1-2]) reference notation to reduce false negatives while still rejecting unknown IDs.

3) Language safety

If the final answer drifts from requested output language (fr/ar/en), a translation guard translates it back while preserving: - citation markers ([n]) - math notation - bullets/paragraph structure

Tests

Added/updated tests in tests/services/test_llm.py: - empty-context behavior without model call - hallucination guardrail behavior (missing citations + out-of-context citation IDs) - translation-out language safety behavior for Arabic, French, and English targets - existing baseline non-stream generation