Implement comprehensive LLM monitoring and metrics collection system #21
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add comprehensive monitoring capabilities for LLM interactions across all AI providers (OpenAI, Anthropic, Ollama) with context window analysis, performance tracking, and debug logging. ## Features Added ### Core Metrics Collection - Token usage tracking (input, output, total, efficiency ratio) - Context window utilization monitoring with status classification - Performance metrics (response time, tokens per second) - Debug logging with configurable levels (basic, detailed, debug) ### Context Window Analysis - Real-time context utilization calculation - Smart status classification (optimal < 70%, warning 70-90%, critical > 90%) - Optimization suggestions (summarization, pruning) - Console warnings when approaching context limits ### Multi-Provider Integration - Integrated metrics collection across OpenAI, Anthropic, and Ollama clients - Provider-specific context limits and token extraction - Unified metrics interface with provider abstraction ### Configuration System - New MonitoringConfig class with validation - Configurable monitoring levels and debug log file paths - Context warnings and performance metrics toggles ### Architecture - MetricsCollector class for centralized metrics management - Structured dataclasses for metrics, context analysis, and token usage - Global configuration with get_metrics_collector() and configure_metrics() - Comprehensive test suite with 15 test cases ## Implementation Details **Files Modified:** - nova/core/ai_client.py: Added metrics integration to all AI providers - nova/models/config.py: Added MonitoringConfig class **Files Added:** - nova/core/metrics.py: Core metrics collection system - tests/unit/test_metrics.py: Comprehensive test suite **Configuration:** ```yaml monitoring: enabled: true level: "debug" # basic, detailed, debug debug_log_file: "~/.nova/debug.log" context_warnings: true performance_metrics: true ``` All tests pass and code follows project quality standards. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
This PR implements a comprehensive LLM monitoring and metrics collection system for Nova that provides real-time visibility into AI interactions across all supported providers (OpenAI, Anthropic, Ollama).
Key Features
Technical Implementation
Core Components:
MetricsCollector: Central metrics management with configurable detail levelsLLMMetrics: Comprehensive interaction metrics dataclassContextAnalysis: Context window analysis with optimization suggestionsTokenUsage: Token counting and efficiency trackingIntegration:
Configuration:
Files Changed
Testing
Test plan
🤖 Generated with Claude Code