[EPIC] Live-Test Reliability and Environment Stabilization

[conniey]

Problem statement

Azure MCP live validation currently mixes product defects, shared-framework regressions, test-data failures, environment drift, flakiness, and high infrastructure cost. Individual CI incidents expose these problems, but resolving one run does not establish a reliable validation system or prevent the same failure classes from returning.

Vision

Live-test results are actionable and repeatable. Failures clearly distinguish product defects from test or environment problems, shared regressions are detected across toolsets, fixtures remain healthy, and the cost of live validation is intentional and measurable.

Current work

• Marketplace: PricingType enum deserialization throws JsonException (product list returns 500) #3029, fix Marketplace enum deserialization failures
• FunctionApp: validation message renders '----' instead of '--' (parameter-validation live tests fail) #3030, correct shared validation-message rendering
• Aks: parameter-validation / graceful-handling live tests failing #3031, stabilize AKS parameter-validation and graceful-handling tests
• Cosmos: live tests fail due to empty containers (missing seed data) #3032, provide deterministic Cosmos test data
• [BUG] Should_list_appconfig_accounts live test fails to find appconfig account #1949, stabilize App Configuration account discovery
• Flaky Kusto CI validation live tests #520, eliminate Kusto live-test flakiness
• Kusto live tests are expensive #151, reduce or justify Kusto live-test cost

Goals (in scope)

• Classify live-test failures as product, framework, fixture, environment, or infrastructure failures
• Detect cross-tool regressions caused by shared command and validation behavior
• Make test data and deployed fixtures deterministic and diagnosable
• Reduce flaky, redundant, and unnecessarily expensive live coverage
• Preserve useful failure artifacts and ownership signals from CI runs
• Define when a scenario requires live validation instead of recorded playback

Non-goals (out of scope)

• Completing the recorded-test migration, which is tracked by the recorded-test epic
• Treating every isolated product defect as a permanent child of this epic
• Masking failures through retries or relaxed assertions without identifying the cause

Success criteria

• Live-test failures identify the responsible product, framework, fixture, or environment layer
• Shared regressions are detected consistently across affected toolsets
• Known flaky and environment-dependent suites have deterministic setup and cleanup
• Live-test cost and execution time have documented baselines
• Resolved failures leave regression coverage or an explicit accepted-risk decision

Dependencies

• Shared live-test framework and Azure test-resource infrastructure
• Recorded-test guidance for scenarios that do not require live validation
• CI diagnostics, retained artifacts, and service-team ownership

[vcolin7]

A couple more related issues for Kusto: #520, #1143