Skip to content

Conversation

@jeo02
Copy link
Member

@jeo02 jeo02 commented Dec 19, 2025

  • Added a couple more eval scenarios
  • Resolves Optional Tool Call #13455 allowing optional parameters so we don't need to rely on the prompt for not running x tool.
  • Left the prompts as is since they do run quicker if less tools are called, but saying "don't run verify setup" for example is no longer needed.
  • Resolves Evaluation Scenarios #12307

@jeo02 jeo02 requested a review from a team as a code owner December 19, 2025 19:42
Copilot AI review requested due to automatic review settings December 19, 2025 19:42
@github-actions github-actions bot added the azsdk-cli Issues related to Azure/azure-sdk-tools::tools/azsdk-cli label Dec 19, 2025
@jeo02 jeo02 requested a review from maririos December 19, 2025 19:46
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request adds additional evaluation scenarios for testing AI agent tool usage, enhancing the test coverage for Azure SDK CLI evaluation capabilities. The changes include a new tool mock for linking namespace approval issues, new test scenarios, and improvements to the chat completion helper to ensure complete conversation history capture.

Key changes:

  • New LinkNamespaceApprovalIssue tool mock and corresponding evaluation scenario
  • Three new evaluation scenarios for TypeSpec project validation workflows
  • Updated GetModifiedTypespecProjects mock to include targetBranch parameter
  • Improved chat completion logic to capture final assistant messages
  • Increased test parallelism from 5 to 10 concurrent tests
  • Enhanced error messages in the tool input evaluator

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
ToolMocks/ToolMocks.cs Registers the new LinkNamespaceApprovalIssue tool mock
ToolMocks/LinkNamespaceApprovalIssue.cs New tool mock for linking namespace approval issues to release plans
ToolMocks/GetModifiedTypespecProjects.cs Adds targetBranch parameter to mock call arguments
Scenarios/General/Evaluate_LinkNamespaceApprovalIssue.cs New test scenario for the namespace approval link functionality
Scenarios/AzureRestApiSpecs/Evaluate_GetModifiedTypespecProjects.cs New test scenario for listing modified TypeSpec projects
Scenarios/AzureRestApiSpecs/Evaluate_CheckPublicRepoThenValidate.cs New test scenario combining public repo check with validation
Scenarios/AzureRestApiSpecs/Evaluate_CheckPublicRepo.cs New test scenario for checking TypeSpec project location
Helpers/ChatCompletion.cs Adds final assistant message to conversation history for completeness
Helpers/AssemblyInfo.cs Increases test parallelism level from 5 to 10
Evaluators/ExpectedToolInputEvaluator.cs Enhances error message to include tool name for better debugging
TestData/example2.json Minor clarification added to user request text

@jeo02 jeo02 requested a review from praveenkuttappan January 6, 2026 19:41
@jeo02 jeo02 enabled auto-merge (squash) January 6, 2026 21:49
@jeo02 jeo02 merged commit 807acc7 into Azure:main Jan 7, 2026
20 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

azsdk-cli Issues related to Azure/azure-sdk-tools::tools/azsdk-cli

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Optional Tool Call Evaluation Scenarios

3 participants