-
Notifications
You must be signed in to change notification settings - Fork 227
More Eval Scenarios #13402
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
More Eval Scenarios #13402
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This pull request adds additional evaluation scenarios for testing AI agent tool usage, enhancing the test coverage for Azure SDK CLI evaluation capabilities. The changes include a new tool mock for linking namespace approval issues, new test scenarios, and improvements to the chat completion helper to ensure complete conversation history capture.
Key changes:
- New
LinkNamespaceApprovalIssuetool mock and corresponding evaluation scenario - Three new evaluation scenarios for TypeSpec project validation workflows
- Updated
GetModifiedTypespecProjectsmock to includetargetBranchparameter - Improved chat completion logic to capture final assistant messages
- Increased test parallelism from 5 to 10 concurrent tests
- Enhanced error messages in the tool input evaluator
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
ToolMocks/ToolMocks.cs |
Registers the new LinkNamespaceApprovalIssue tool mock |
ToolMocks/LinkNamespaceApprovalIssue.cs |
New tool mock for linking namespace approval issues to release plans |
ToolMocks/GetModifiedTypespecProjects.cs |
Adds targetBranch parameter to mock call arguments |
Scenarios/General/Evaluate_LinkNamespaceApprovalIssue.cs |
New test scenario for the namespace approval link functionality |
Scenarios/AzureRestApiSpecs/Evaluate_GetModifiedTypespecProjects.cs |
New test scenario for listing modified TypeSpec projects |
Scenarios/AzureRestApiSpecs/Evaluate_CheckPublicRepoThenValidate.cs |
New test scenario combining public repo check with validation |
Scenarios/AzureRestApiSpecs/Evaluate_CheckPublicRepo.cs |
New test scenario for checking TypeSpec project location |
Helpers/ChatCompletion.cs |
Adds final assistant message to conversation history for completeness |
Helpers/AssemblyInfo.cs |
Increases test parallelism level from 5 to 10 |
Evaluators/ExpectedToolInputEvaluator.cs |
Enhances error message to include tool name for better debugging |
TestData/example2.json |
Minor clarification added to user request text |
Uh oh!
There was an error while loading. Please reload this page.