Add script to validate existence of E2E prompt tool. Update const used in ToolListCommand#3034
Add script to validate existence of E2E prompt tool. Update const used in ToolListCommand#3034conniey wants to merge 8 commits into
Conversation
conniey
requested review from
chidozieononiwu,
g2vinay,
jongio,
msalaman,
saikoumudi,
tmeschter and
vukelich
|
Azure Pipelines: Successfully started running 1 pipeline(s). There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
Pull request overview
Adds CI-side validation to ensure documented E2E tool-selection prompts reference real, runtime-available MCP tools, and tightens tool name tokenization to rely on the shared separator constant.
Changes:
- Added
Test-ToolSelectionPrompts.ps1to build servers and validatee2eTestPrompts.mdtool names againsttools list --name-only. - Updated
Analyze-Code.ps1to run the new prompt validation step during analysis. - Updated
ToolsListCommandto useCommandFactory.Separatorwhen constructing surfaced tool names; updated tool-length validation to consume--name-onlyoutput.
Invoking Livetests
Copilot submitted PRs are not trustworthy by default. Users with write access to the repo need to validate the contents of this PR before leaving a comment with the text /azp run mcp - pullrequest - live. This will trigger the necessary livetest workflows to complete required validation.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| eng/scripts/Test-ToolSelectionPrompts.ps1 | New script that builds servers and validates prompt tool names against runtime tool inventory. |
| eng/scripts/Analyze-Code.ps1 | Wires the new prompt validation into the analysis workflow. |
| eng/scripts/Test-ToolNameLength.ps1 | Switches tool enumeration to tools list --name-only and simplifies length checks. |
| eng/scripts/Test-ToolId.ps1 | Removes an unused variable. |
| core/Microsoft.Mcp.Core/src/Areas/Tools/Commands/ToolsListCommand.cs | Uses the shared tool-name separator constant when generating surfaced tool IDs. |
| if ($LASTEXITCODE -ne 0) { | ||
| Write-Host "❌ E2E tool prompt validation failed." | ||
| } else { | ||
| Write-Host "✅ E2E tool prompt validation did not detect any issues." | ||
| } |
| # Run tool prompt validation | ||
| & "$PSScriptRoot/Test-ToolSelectionPrompts.ps1" | ||
|
|
| # Use the build infrastructure - New-BuildInfo.ps1 and Build-Code.ps1 | ||
| $buildInfoPath = "$RepoRoot/.work/build_info.json" | ||
| $buildOutputPath = "$RepoRoot/.work/build" |
| # Build the servers | ||
| $platformName = Get-PlatformName | ||
| & "$RepoRoot/eng/scripts/Build-Code.ps1" -BuildInfoPath $buildInfoPath -PlatformName $platformName | ||
|
|
| if ($ServerName) { | ||
| Write-Host "Validating tool name length for $ServerName" | ||
| } else { | ||
| Write-Host "Validating tool name length for all servers" | ||
| } |
RickWinter
left a comment
RickWinter
left a comment
There was a problem hiding this comment.
This PR adds an E2E prompt tool-existence validation script and updates the tool-list constant. The new gate can report success without checking any prompt when every server is skipped. That execution path must fail before merge.
| } | ||
| } | ||
|
|
||
| if ($violationsCount -eq 0) { |
There was a problem hiding this comment.
When every server is skipped because tools list fails or returns no names, $violationsCount remains zero and this exits successfully. The new CI gate can pass without validating any prompt. Track successful validations and fail when none run, as Test-ToolNameLength.ps1 does.
What does this PR do?
Separator(in case it is changed in the future)GitHub issue number?
Pre-merge Checklist
servers/Azure.Mcp.Server/README.mdand/orservers/Fabric.Mcp.Server/README.mddocumentationREADME.mdchanges running the script./eng/scripts/Process-PackageReadMe.ps1. See Package READMEToolDescriptionEvaluatorand obtained a score of0.4or more and a top 3 ranking for all related test promptsconsolidated-tools.jsonbreaking-changelabelservers/Azure.Mcp.Server/docs/azmcp-commands.md./eng/scripts/Update-AzCommandsMetadata.ps1to update tool metadata inazmcp-commands.md(required for CI)servers/Azure.Mcp.Server/docs/e2eTestPrompts.mdcrypto mining, spam, data exfiltration, etc.)/azp run mcp - pullrequest - liveto run Live Test Pipeline