Plan: Dynamic Configuration System Implementation

Date: 2026-02-21 Status: Draft / For Review Purpose: Implement a system to tune and A/B test all AI parameters (retrieval, reasoning, costs) via database-driven configuration profiles with automatic fallback to code defaults.

---

1. Executive Summary

This plan enables "hot-tuning" of the AI Teacher. Instead of hardcoded constants, the system will look up a configuration profile assigned to the user. If a parameter is found, it is used; otherwise, it falls back to the existing Settings in app/core/config.py. Crucially, it includes a "Self-Healing" mechanism that reverts to defaults if a custom parameter (e.g., a misspelled model name) causes an execution error.

---

2. Phase A: Database Schema (Supabase)

Step 1: Create Configuration Tables

We need a table to store the profiles and a way to link them to users.

Migration: 20260221000024_dynamic_config_tables.sql

-- Store configuration profiles
CREATE TABLE IF NOT EXISTS app_configurations (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name TEXT UNIQUE NOT NULL,
    description TEXT,
    settings JSONB NOT NULL DEFAULT '{}',
    is_active BOOLEAN DEFAULT true,
    created_at TIMESTAMPTZ DEFAULT now(),
    updated_at TIMESTAMPTZ DEFAULT now()
);

-- Link profiles to users
ALTER TABLE profiles 
ADD COLUMN IF NOT EXISTS config_id UUID REFERENCES app_configurations(id);

-- Insert a 'Default Experimental' profile for testing
INSERT INTO app_configurations (name, description, settings)
VALUES ('experimental_v1', 'Initial tuning test', '{
    "retrieval_top_k": 25,
    "rerank_top_n": 5,
    "reasoning_model": "gpt-4o",
    "temperature": 0.2
}');

---

3. Phase B: Config Manager Service

Step 1: Create ConfigManager

A new service to fetch, cache, and merge settings.

File: app/services/config_manager.py * get_user_config(user_id): Fetches the merged dictionary of settings. * Logic: 1. Get config_id from profiles. 2. If config_id is NULL, return empty dict (full fallback). 3. Load settings from app_configurations. 4. Merge with app/core/config.py as the base.

---

4. Phase C: Service Integration (The Fallback Pattern)

Step 1: Update RetrievalPipeline

Inject ConfigManager and use it to override top_k and rerank_n.

Step 2: Update LLM Service (Reasoning)

Implement the "Execution Failure" safety net.

Example Logic:

def generate_answer(query, context, config_params):
    model = config_params.get("reasoning_model", settings.OPENAI_CHAT_MODEL)
    try:
        # Attempt with custom model
        return call_openai(model=model, ...)
    except Exception as e:
        our_logs.error(f"Config parameter {model} failed: {e}. Falling back to default.")
        # Automatic fallback to hardcoded default
        return call_openai(model=settings.OPENAI_CHAT_MODEL, ...)

---

5. Phase D: Agent Integration

TeacherAgent Node Update

The check_wallet node (which runs first) will be renamed to initialize_session. 1. Verify wallet/tier (as before). 2. Fetch user configuration via ConfigManager. 3. Store the configuration dict in the TeacherState.

---

6. Implementation Roadmap (A-Z)

Phase	Task	File/Action
A	Create SQL Migration	db/migrations/20260221000024_...
B	Implement ConfigManager	app/services/config_manager.py
C	Add Config Fetching to Agent	app/agents/teacher_agent.py
D	Update Retrieval Pipeline to use Dynamic Params	app/services/retrieval_pipeline.py
E	Update LLM Service with "Self-Healing" Logic	app/services/llm.py
F	Integration Test (Assign bad config and verify fallback)	tests/test_dynamic_config.py

---

7. Success Criteria

1. Zero Downtime: If a bad model name is put in the DB, the system logs it and answers anyway using the default model.
2. No Code Change Tuning: Changing retrieval_top_k in the DB immediately affects the next chat message for that user.
3. Traceability: usage_logs should record which config_id was used for the request.