Tool Evaluations: Add multi-variant experiment runner with JSON summary output

[LarryOsterman]

Summary

Update the ToolDescriptionEvaluator experiment runner to support running evaluations comparing an MCP server variant against a baseline, and produce a structured JSON summary file for each run.

Variants

The three variants to evaluate (in separate runs, since namespace and consolidated are mutually exclusive server modes) are:

1. Baseline — No MCP server tools loaded. Measures how well prompts match without any tool descriptions present (establishes a noise floor).
2. Namespace mode — MCP server started with --mode namespace. Tools are grouped by service namespace (e.g., one storage tool that routes internally). Uses namespace-prompts.json and namespace-tools.json.
3. Consolidated mode — MCP server started with --mode single. A single azure tool handles all routing. Uses consolidated-prompts.json and consolidated tool definitions.

Requirements

Experiment Runner Updates

• Add a --variant CLI flag (required) that accepts namespace or consolidated.
• Each run automatically includes a baseline comparison (no tools loaded).
• For each variant, the runner should:
  1. Load the appropriate tools JSON (none for baseline, namespace-tools.json for namespace, consolidated tools for consolidated).
  2. Load the corresponding prompts JSON (prompts.json for baseline, namespace-prompts.json for namespace, consolidated-prompts.json for consolidated).
  3. Run the existing embedding-based evaluation pipeline.
  4. Collect per-prompt results (expected tool, actual top match, rank, confidence score).

JSON Summary Output

After the variant and baseline evaluations complete, emit a JSON summary file (e.g., experiment-namespace.json or experiment-consolidated.json) with the following structure:

{
  "timestamp": "2026-07-14T15:00:00Z",
  "server": "Azure",
  "area": "storage",
  "variant": "namespace",
  "results": {
    "baseline": {
      "toolCount": 0,
      "promptCount": 42,
      "topChoiceSuccessRate": 0.0,
      "avgConfidence": 0.0,
      "confidenceDistribution": { "veryHigh": 0, "high": 0, "good": 0, "fair": 0, "acceptable": 0, "low": 42 }
    },
    "namespace": {
      "toolCount": 28,
      "promptCount": 42,
      "topChoiceSuccessRate": 0.85,
      "avgConfidence": 0.72,
      "confidenceDistribution": { "veryHigh": 5, "high": 12, "good": 10, "fair": 8, "acceptable": 5, "low": 2 }
    }
  },
  "comparison": {
    "topChoiceSuccessRateDelta": 0.85,
    "avgConfidenceDelta": 0.72
  }
}

Existing Infrastructure to Leverage

• eng/tools/ToolDescriptionEvaluator/ — existing evaluator with embedding pipeline
• eng/tools/ToolDescriptionEvaluator/prompts/ — already contains prompts.json, namespace-prompts.json, consolidated-prompts.json
• eng/tools/ToolDescriptionEvaluator/scripts/Generate-GroupedPromptsJson.ps1 — generates namespace/consolidated prompt mappings
• servers/Azure.Mcp.Server/docs/e2eTestPrompts.md — source of per-tool prompts (~62 service areas)
• eng/tools/ToolDescriptionEvaluator/scripts/Run-ToolDescriptionEvaluator.ps1 — existing run script

Acceptance Criteria

• --variant namespace runs namespace evaluation against baseline
• --variant consolidated runs consolidated evaluation against baseline
• Each variant uses the correct tools and prompts files
• A structured JSON file is written summarizing variant and baseline results with comparison deltas
• Existing --area filtering works with the new variant runner
• Existing markdown/text output modes are preserved (JSON summary is additive)
• --ci mode continues to work with graceful credential handling