Add Addie feedback and training data system #428
Merged
+1,263
−13
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The action buttons in the thread view (Create Docs Issue, Add to Perspectives, Suggest Rule Change) were referencing an undefined variable `currentConversationData`. The thread view sets `currentThreadData` but never set `currentConversationData`. - Add `currentConversationData` variable declaration - Set `currentConversationData = data` in viewThread() - Update all action handlers to use `thread_id` instead of `conversation_id` since thread data uses that field name 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Build a feedback system for Addie threads to enable evaluation of rule changes. This captures training data about what responses work well and what doesn't. Features: - Per-message 5-star rating with auto-save on click - Per-message feedback tags (accurate, helpful, missing_info, etc.) - Per-message notes with expandable detail panel - Thread-level feedback with thumbs up/down + stars - Thread-level notes text area for overall impressions - Display of existing feedback inline with messages - All feedback saves to existing API endpoints The feedback panel appears when viewing any thread and attaches ratings to assistant messages. This creates a database of labeled examples for evaluating Addie's performance when rules change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Build a comprehensive training data system for evaluating Addie's performance and testing rule changes. ## Execution Diagnostics - Add timing breakdown fields to thread messages (system_prompt_ms, total_llm_ms, total_tool_ms, processing_iterations) - Store cache performance (tokens_cache_creation, tokens_cache_read) - Track active_rule_ids per message - Update bolt-app to capture all execution metadata from Claude response - Add collapsible "Execution Details" panel in thread message view ## Claude Diagnosis - Add POST /api/admin/addie/threads/:id/diagnose endpoint - Claude analyzes conversation and provides: - Response quality assessment - What worked well / could be improved - Suggested rule changes - Training data quality label - "Diagnose with Claude" button in thread actions panel - Includes admin feedback notes as context for analysis ## Evaluation Dashboard - New "Evaluation" tab in admin UI - Summary stats: total responses, rated, avg rating, positive/negative - Feedback tags distribution - Low-rated threads list for quick review - GET /api/admin/addie/feedback/summary endpoint - Filterable by date range (7/30/90 days) ## Database Migration (066) - Add timing columns to addie_thread_messages - Add active_rule_ids array column with GIN index - Create addie_execution_analysis view for debugging - Create addie_feedback_summary view for dashboard 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix migration error: use jsonb_array_elements_text instead of unnest for JSONB feedback_tags - Drop and recreate views to handle column name changes - Add Anthropic import at module level instead of dynamic import - Validate ANTHROPIC_API_KEY before using diagnosis endpoint - Sanitize feedback parameter to prevent prompt injection - Add proper error handling for Claude API rate limits - Validate and clamp days parameter (1-365 range) - Fix XSS risk by escaping thread_id in onclick handlers 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Resolve import conflict by keeping both Anthropic and getAddieBoltApp imports - Rename migration 066 to 067 to avoid conflict with 066_organization_domains.sql 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
currentConversationDatavariable)Changes
Bug Fix
currentConversationDatavariableFeedback System
Execution Diagnostics
Claude Diagnosis
/api/admin/addie/threads/:id/diagnoseendpointEvaluation Dashboard
Code Quality Fixes
jsonb_array_elements_textfor JSONB feedback_tagsTest plan
/admin-addie.html🤖 Generated with Claude Code