167 lines
6.9 KiB
Markdown
167 lines
6.9 KiB
Markdown
# CLAUDE.md - Technical Notes for LLM Council
|
|
|
|
This file contains technical details, architectural decisions, and important implementation notes for future development sessions.
|
|
|
|
## Project Overview
|
|
|
|
LLM Council is a 3-stage deliberation system where multiple LLMs collaboratively answer user questions. The key innovation is anonymized peer review in Stage 2, preventing models from playing favorites.
|
|
|
|
## Architecture
|
|
|
|
### Backend Structure (`backend/`)
|
|
|
|
**`config.py`**
|
|
- Contains `COUNCIL_MODELS` (list of OpenRouter model identifiers)
|
|
- Contains `CHAIRMAN_MODEL` (model that synthesizes final answer)
|
|
- Uses environment variable `OPENROUTER_API_KEY` from `.env`
|
|
- Backend runs on **port 8001** (NOT 8000 - user had another app on 8000)
|
|
|
|
**`openrouter.py`**
|
|
- `query_model()`: Single async model query
|
|
- `query_models_parallel()`: Parallel queries using `asyncio.gather()`
|
|
- Returns dict with 'content' and optional 'reasoning_details'
|
|
- Graceful degradation: returns None on failure, continues with successful responses
|
|
|
|
**`council.py`** - The Core Logic
|
|
- `stage1_collect_responses()`: Parallel queries to all council models
|
|
- `stage2_collect_rankings()`:
|
|
- Anonymizes responses as "Response A, B, C, etc."
|
|
- Creates `label_to_model` mapping for de-anonymization
|
|
- Prompts models to evaluate and rank (with strict format requirements)
|
|
- Returns tuple: (rankings_list, label_to_model_dict)
|
|
- Each ranking includes both raw text and `parsed_ranking` list
|
|
- `stage3_synthesize_final()`: Chairman synthesizes from all responses + rankings
|
|
- `parse_ranking_from_text()`: Extracts "FINAL RANKING:" section, handles both numbered lists and plain format
|
|
- `calculate_aggregate_rankings()`: Computes average rank position across all peer evaluations
|
|
|
|
**`storage.py`**
|
|
- JSON-based conversation storage in `data/conversations/`
|
|
- Each conversation: `{id, created_at, messages[]}`
|
|
- Assistant messages contain: `{role, stage1, stage2, stage3}`
|
|
- Note: metadata (label_to_model, aggregate_rankings) is NOT persisted to storage, only returned via API
|
|
|
|
**`main.py`**
|
|
- FastAPI app with CORS enabled for localhost:5173 and localhost:3000
|
|
- POST `/api/conversations/{id}/message` returns metadata in addition to stages
|
|
- Metadata includes: label_to_model mapping and aggregate_rankings
|
|
|
|
### Frontend Structure (`frontend/src/`)
|
|
|
|
**`App.jsx`**
|
|
- Main orchestration: manages conversations list and current conversation
|
|
- Handles message sending and metadata storage
|
|
- Important: metadata is stored in the UI state for display but not persisted to backend JSON
|
|
|
|
**`components/ChatInterface.jsx`**
|
|
- Multiline textarea (3 rows, resizable)
|
|
- Enter to send, Shift+Enter for new line
|
|
- User messages wrapped in markdown-content class for padding
|
|
|
|
**`components/Stage1.jsx`**
|
|
- Tab view of individual model responses
|
|
- ReactMarkdown rendering with markdown-content wrapper
|
|
|
|
**`components/Stage2.jsx`**
|
|
- **Critical Feature**: Tab view showing RAW evaluation text from each model
|
|
- De-anonymization happens CLIENT-SIDE for display (models receive anonymous labels)
|
|
- Shows "Extracted Ranking" below each evaluation so users can validate parsing
|
|
- Aggregate rankings shown with average position and vote count
|
|
- Explanatory text clarifies that boldface model names are for readability only
|
|
|
|
**`components/Stage3.jsx`**
|
|
- Final synthesized answer from chairman
|
|
- Green-tinted background (#f0fff0) to highlight conclusion
|
|
|
|
**Styling (`*.css`)**
|
|
- Light mode theme (not dark mode)
|
|
- Primary color: #4a90e2 (blue)
|
|
- Global markdown styling in `index.css` with `.markdown-content` class
|
|
- 12px padding on all markdown content to prevent cluttered appearance
|
|
|
|
## Key Design Decisions
|
|
|
|
### Stage 2 Prompt Format
|
|
The Stage 2 prompt is very specific to ensure parseable output:
|
|
```
|
|
1. Evaluate each response individually first
|
|
2. Provide "FINAL RANKING:" header
|
|
3. Numbered list format: "1. Response C", "2. Response A", etc.
|
|
4. No additional text after ranking section
|
|
```
|
|
|
|
This strict format allows reliable parsing while still getting thoughtful evaluations.
|
|
|
|
### De-anonymization Strategy
|
|
- Models receive: "Response A", "Response B", etc.
|
|
- Backend creates mapping: `{"Response A": "openai/gpt-5.1", ...}`
|
|
- Frontend displays model names in **bold** for readability
|
|
- Users see explanation that original evaluation used anonymous labels
|
|
- This prevents bias while maintaining transparency
|
|
|
|
### Error Handling Philosophy
|
|
- Continue with successful responses if some models fail (graceful degradation)
|
|
- Never fail the entire request due to single model failure
|
|
- Log errors but don't expose to user unless all models fail
|
|
|
|
### UI/UX Transparency
|
|
- All raw outputs are inspectable via tabs
|
|
- Parsed rankings shown below raw text for validation
|
|
- Users can verify system's interpretation of model outputs
|
|
- This builds trust and allows debugging of edge cases
|
|
|
|
## Important Implementation Details
|
|
|
|
### Relative Imports
|
|
All backend modules use relative imports (e.g., `from .config import ...`) not absolute imports. This is critical for Python's module system to work correctly when running as `python -m backend.main`.
|
|
|
|
### Port Configuration
|
|
- Backend: 8001 (changed from 8000 to avoid conflict)
|
|
- Frontend: 5173 (Vite default)
|
|
- Update both `backend/main.py` and `frontend/src/api.js` if changing
|
|
|
|
### Markdown Rendering
|
|
All ReactMarkdown components must be wrapped in `<div className="markdown-content">` for proper spacing. This class is defined globally in `index.css`.
|
|
|
|
### Model Configuration
|
|
Models are hardcoded in `backend/config.py`. Chairman can be same or different from council members. The current default is Gemini as chairman per user preference.
|
|
|
|
## Common Gotchas
|
|
|
|
1. **Module Import Errors**: Always run backend as `python -m backend.main` from project root, not from backend directory
|
|
2. **CORS Issues**: Frontend must match allowed origins in `main.py` CORS middleware
|
|
3. **Ranking Parse Failures**: If models don't follow format, fallback regex extracts any "Response X" patterns in order
|
|
4. **Missing Metadata**: Metadata is ephemeral (not persisted), only available in API responses
|
|
|
|
## Future Enhancement Ideas
|
|
|
|
- Configurable council/chairman via UI instead of config file
|
|
- Streaming responses instead of batch loading
|
|
- Export conversations to markdown/PDF
|
|
- Model performance analytics over time
|
|
- Custom ranking criteria (not just accuracy/insight)
|
|
- Support for reasoning models (o1, etc.) with special handling
|
|
|
|
## Testing Notes
|
|
|
|
Use `test_openrouter.py` to verify API connectivity and test different model identifiers before adding to council. The script tests both streaming and non-streaming modes.
|
|
|
|
## Data Flow Summary
|
|
|
|
```
|
|
User Query
|
|
↓
|
|
Stage 1: Parallel queries → [individual responses]
|
|
↓
|
|
Stage 2: Anonymize → Parallel ranking queries → [evaluations + parsed rankings]
|
|
↓
|
|
Aggregate Rankings Calculation → [sorted by avg position]
|
|
↓
|
|
Stage 3: Chairman synthesis with full context
|
|
↓
|
|
Return: {stage1, stage2, stage3, metadata}
|
|
↓
|
|
Frontend: Display with tabs + validation UI
|
|
```
|
|
|
|
The entire flow is async/parallel where possible to minimize latency.
|