Create initial vally experiments for Azure MCP tool effectiveness.#3015
Draft
LarryOsterman wants to merge 5 commits into
Draft
Create initial vally experiments for Azure MCP tool effectiveness.#3015LarryOsterman wants to merge 5 commits into
LarryOsterman wants to merge 5 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Adds an initial Vally-based evaluation harness under servers/Azure.Mcp.Server/tests/Vally to measure Azure MCP tool effectiveness via baseline vs treatment experiment variants, with first experiments targeting Event Hubs.
Changes:
- Introduces a discovery-based PowerShell runner (
Invoke-VallyEval.ps1) that buildsazmcp, provisions per-area resources, runs Vally experiments, and compares baseline/treatment results. - Adds initial Event Hubs experiments (
eventhub-get,namespace-get) plus provisioning/teardown scripts. - Updates spelling dictionary and Azure MCP Server changelog to reflect the new harness.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 11 comments.
Show a summary per file
| File | Description |
|---|---|
| servers/Azure.Mcp.Server/tests/Vally/README.md | Documents the Vally evaluation framework and how to run/add experiments. |
| servers/Azure.Mcp.Server/tests/Vally/Invoke-VallyEval.ps1 | Runner script: builds azmcp, discovers experiments, provisions/tears down resources, runs Vally experiments, and reports deltas. |
| servers/Azure.Mcp.Server/tests/Vally/eventhubs/New-EventHubsResources.ps1 | Provisions Event Hubs test resources used by the experiments. |
| servers/Azure.Mcp.Server/tests/Vally/eventhubs/Remove-EventHubsResources.ps1 | Tears down Event Hubs test resources after experiments. |
| servers/Azure.Mcp.Server/tests/Vally/eventhubs/eventhub-get.experiment.yaml | Defines baseline vs treatment variants for the Event Hub “get” scenario. |
| servers/Azure.Mcp.Server/tests/Vally/eventhubs/eventhub-get.eval.yaml | Shared eval spec (stimuli + outcome graders) for Event Hub “get”. |
| servers/Azure.Mcp.Server/tests/Vally/eventhubs/namespace-get.experiment.yaml | Defines baseline vs treatment variants for the namespace “get” scenario. |
| servers/Azure.Mcp.Server/tests/Vally/eventhubs/namespace-get.eval.yaml | Shared eval spec (stimuli + outcome graders) for namespace “get”. |
| servers/Azure.Mcp.Server/tests/Vally/.gitignore | Ignores Vally output artifacts under the new test folder. |
| servers/Azure.Mcp.Server/CHANGELOG.md | Notes the addition of the Vally harness and initial Event Hubs experiment(s). |
| .vscode/cspell.json | Adds Vally-/eval-related words to the spelling dictionary. |
| @@ -0,0 +1,148 @@ | |||
| #!/bin/env pwsh | |||
| @@ -0,0 +1,73 @@ | |||
| #!/bin/env pwsh | |||
Comment on lines
+6
to
+7
| Provisions the Azure Event Hubs resources used by the vally "get Event Hub" | ||
| evaluation (eventhubs/eval.yaml). |
Comment on lines
+54
to
+59
| The **treatment** (`eventhub-get.eval.yaml`) additionally uses the | ||
| [`tool-calls`](https://microsoft.github.io/vally/reference/graders/) grader to | ||
| assert that: | ||
|
|
||
| - the agent **selects and invokes** `eventhubs_eventhub_get`, and | ||
| - it does **not** invoke a destructive Event Hubs tool (e.g. `*_delete`). |
Comment on lines
+66
to
+68
| `eventhub-get.eval.baseline.yaml` is the **control**: it reuses the exact same | ||
| prompts but omits the `environment.mcpServers` block, so the agent runs with only | ||
| its built-in tools. |
Comment on lines
+168
to
+175
| # Only run the treatments; skip the baseline/control runs | ||
| ./Invoke-VallyEval.ps1 -SkipBaseline | ||
|
|
||
| # Override the model and show full agent output | ||
| ./Invoke-VallyEval.ps1 -Model claude-opus-4.6 -Verbose | ||
|
|
||
| # Run one explicit spec (its baseline is derived automatically) | ||
| ./Invoke-VallyEval.ps1 -EvalSpec ./eventhubs/eventhub-get.eval.yaml |
Comment on lines
+54
to
+58
| $exists = az group exists --name $ResourceGroup @subArgs 2>$null | ||
| if ($exists -ne 'true') { | ||
| Write-Info "Resource group '$ResourceGroup' does not exist. Nothing to delete." | ||
| return | ||
| } |
…ports can be generated without running eval
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Create initial vally experiment framework for measuring effectiveness of MCP tools relative to the baseline.
This PR adds a new script named "Invoke-VallyEval.ps1" in
servers/Azure.Mcp.Server\tests\Vallywhich invokes vally experiments for each tool experiment in thetests\Vallydirectory.See the
README.mdfile intests\Vallyfor more information on the structure of the test framework and how the pieces all fit together.The evaluations are based on the prompts kept in
servers/Azure.Mcp.Server/docs/e2eTestPrompts.mdfile.GitHub issue number?
[Link to the GitHub issue this PR addresses]Pre-merge Checklist
servers/Azure.Mcp.Server/README.mdand/orservers/Fabric.Mcp.Server/README.mddocumentationREADME.mdchanges running the script./eng/scripts/Process-PackageReadMe.ps1. See Package READMEToolDescriptionEvaluatorand obtained a score of0.4or more and a top 3 ranking for all related test promptsconsolidated-tools.jsonbreaking-changelabelservers/Azure.Mcp.Server/docs/azmcp-commands.md./eng/scripts/Update-AzCommandsMetadata.ps1to update tool metadata inazmcp-commands.md(required for CI)servers/Azure.Mcp.Server/docs/e2eTestPrompts.mdcrypto mining, spam, data exfiltration, etc.)/azp run mcp - pullrequest - liveto run Live Test Pipeline