Add DSPM for AI tests (MT.1172-MT.1176)#1703
Conversation
Up to standards ✅🟢 Issues
|
There was a problem hiding this comment.
Pull request overview
Adds Microsoft Purview DSPM for AI governance coverage to Maester by introducing five new MT.1152–MT.1156 tests (Copilot audit ingestion, sensitivity labels, insider risk, DLP, retention) along with their remediation and website documentation, config registration, test harness, and module exports.
Changes:
- Add 5 new Purview-focused Maester PowerShell tests (MT.1152–MT.1156) with detailed markdown result output.
- Add remediation markdown pages and Docusaurus website docs for each new test.
- Register the tests in
maester-config.json, add a consolidated Pester suite, and export the functions inMaester.psd1.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| website/docs/tests/maester/MT.1152.md | Website doc for unified audit log ingestion prerequisite test. |
| website/docs/tests/maester/MT.1153.md | Website doc for published file-scoped sensitivity labels prerequisite test. |
| website/docs/tests/maester/MT.1154.md | Website doc for Insider Risk “Risky AI usage” policy test. |
| website/docs/tests/maester/MT.1155.md | Website doc for Copilot-scoped DLP policy test. |
| website/docs/tests/maester/MT.1156.md | Website doc for Copilot-scoped retention policy test. |
| tests/maester-config.json | Adds MT.1152–MT.1156 entries to the test catalog (ID/severity/title). |
| tests/Maester/Purview/Test-MtPurviewAi.Tests.ps1 | New Pester suite invoking the 5 Purview AI tests. |
| powershell/public/maester/purview/Test-MtPurviewAiAuditLogIngestion.ps1 | Implements MT.1152 (unified audit log ingestion enabled). |
| powershell/public/maester/purview/Test-MtPurviewAiAuditLogIngestion.md | Remediation/details page for MT.1152. |
| powershell/public/maester/purview/Test-MtPurviewAiSensitivityLabelsForFiles.ps1 | Implements MT.1153 (published file-scoped sensitivity labels). |
| powershell/public/maester/purview/Test-MtPurviewAiSensitivityLabelsForFiles.md | Remediation/details page for MT.1153. |
| powershell/public/maester/purview/Test-MtPurviewAiInsiderRiskPolicy.ps1 | Implements MT.1154 (Risky AI usage Insider Risk policy enabled). |
| powershell/public/maester/purview/Test-MtPurviewAiInsiderRiskPolicy.md | Remediation/details page for MT.1154. |
| powershell/public/maester/purview/Test-MtPurviewAiDlpPolicy.ps1 | Implements MT.1155 (Copilot-targeted DLP policy enabled, non-simulation). |
| powershell/public/maester/purview/Test-MtPurviewAiDlpPolicy.md | Remediation/details page for MT.1155. |
| powershell/public/maester/purview/Test-MtPurviewAiRetentionPolicy.ps1 | Implements MT.1156 (Copilot-targeted retention policy enabled). |
| powershell/public/maester/purview/Test-MtPurviewAiRetentionPolicy.md | Remediation/details page for MT.1156. |
| powershell/Maester.psd1 | Exports the 5 new test functions from the module. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
MT.1153 (Test-MtPurviewAiSensitivityLabelsForFiles): Cross-reference Get-LabelPolicy output to verify a file-scoped label is actually included in a published policy, not merely that *some* file-scoped label exists alongside *some* published policy. Adds a third row to the result table showing the published file-scoped label count. MT.1154 (Test-MtPurviewAiInsiderRiskPolicy): Narrow the 'cmdlet unavailable / not licensed' skip to CommandNotFoundException and matching 'is not recognized' / 'is unavailable' messages. All other exceptions now fall back to -SkippedBecause Error -SkippedError so transient service errors, throttling, and module/session issues are no longer masked as licensing skips.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Address Copilot follow-up review on PR maester365#1703: A label policy is now treated as 'published' only when Mode is 'Enforce' AND (when the Enabled property is exposed by the SCC schema) Enabled is true. The previous '-or' logic could classify a disabled Enforce-mode policy or an enabled non-Enforce-mode policy as published, which is inconsistent with the in-code comment and remediation docs.
|
@copilot resolve the merge conflicts in this pull request |
SamErde
added
enhancement
|
Thanks for putting this together — these DSPM for AI checks are a really useful addition, and the overall PR shape looks good: helper functions are exported, companion docs and website docs are present, the test IDs are wired through config, and the targeted PowerShell/Pester validation I ran did not show syntax or analyzer issues. I found two areas that I think should be tightened before merge so the checks do not false-fail tenants that are configured through the current Microsoft Purview surfaces:
Also, GitHub currently reports the PR as conflicting with Thanks again for the contribution — once the policy-surface detection is broadened, these look much closer to being merge-ready. |
SamErde
left a comment
SamErde
left a comment
There was a problem hiding this comment.
See previous comment. Thanks! 🙂
Adds five Maester tests covering Microsoft Purview Data Security Posture Management for AI: - MT.1172 Unified audit log ingestion is enabled for AI activity - MT.1173 Sensitivity labels are published for files used by Microsoft 365 Copilot - MT.1174 Insider Risk Management policy for Risky AI usage is enabled - MT.1175 DLP policy is configured for the Microsoft 365 Copilot location - MT.1176 Retention policy is configured for the Microsoft Copilot location MT.1175 detects Copilot DLP coverage across the multiple shapes exposed by Get-DlpCompliancePolicy: MicrosoftCopilotLocation, Workload (MicrosoftCopilot), Locations, and EnforcementPlanes (CopilotExperiences). MT.1176 queries both retention surfaces -- legacy Get-RetentionCompliancePolicy and the newer Get-AppRetentionCompliancePolicy (User:M365Copilot) -- via the Get-MtExo cache wrapper. Passes if either surface protects Copilot. Test IDs originally proposed as MT.1152-MT.1156 were renumbered to MT.1172-MT.1176 to avoid collision with the Defender antivirus metadata batch added upstream in 7ebade5.
OfirGavish
changed the title
|
Thanks for the careful review @SamErde — addressed both points and pushed Renumbered to MT.1172-MT.1176 MT.1175 — broader Copilot DLP detection
MT.1176 — dual retention surface
Companion |
The build-validation Pester test "Should contain Write-Verbose logging" in powershell/tests/functions/Common.Tests.ps1 requires every exported public function to contain at least one Write-Verbose call. Four of the five new DSPM for AI tests were missing this, causing 4 failures across ubuntu-latest, macOS-latest and windows-latest runners. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Thanks for the PR @OfirGavish — really nice set of DSPM for AI tests! 🎉 The
That test enforces a project-wide convention that every exported public function must contain at least one Since
That should turn the build-validation checks green. Feel free to amend / squash the commit into your branch however you prefer — and thanks again for the contribution! |
# Conflicts: # tests/maester-config.json
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds five new Maester Purview AI governance test cmdlets (audit log ingestion, sensitivity labels for files, insider risk policy, DLP policy, retention policy) with corresponding Markdown docs and website pages, registers them in module exports, test config, and a new Pester test suite, plus adds two new Get-MtExo request routes. ChangesPurview AI Governance Tests
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Test as Test-MtPurviewAiRetentionPolicy
participant Exo as Get-MtExo
participant SC as Security & Compliance
participant Detail as Add-MtTestResultDetail
Test->>SC: Check SecurityCompliance connectivity
Test->>Exo: Get-RetentionCompliancePolicy
Exo-->>Test: legacy policies or 401/403/error
Test->>Exo: Get-AppRetentionCompliancePolicy
Exo-->>Test: app policies, missing cmdlet, or 401/403/error
Test->>Test: Match Copilot targets and mark Surface
Test->>Detail: Add-MtTestResultDetail -Result
Detail-->>Test: recorded
Test-->>Test: return bool / $null
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@powershell/public/maester/purview/Test-MtPurviewAiRetentionPolicy.ps1`:
- Around line 30-32: The public cmdlet Test-MtPurviewAiRetentionPolicy is
missing a Write-Verbose call immediately after the param() block, which will
cause the repo-wide Common.Tests.ps1 check to fail. Add a Write-Verbose
statement right after param() on line 32 to follow the same pattern used in
other new cmdlets in the repository.
In
`@powershell/public/maester/purview/Test-MtPurviewAiSensitivityLabelsForFiles.md`:
- Around line 10-14: The pass criteria in the documentation for
Test-MtPurviewAiSensitivityLabelsForFiles is ambiguous about whether a
File-scoped label must be part of a published/enforced policy or can exist in
any state. Revise the second bullet point to explicitly clarify that the
File-scoped label must not only exist with the File scope, but must also be
included in a published/enforced label policy, making it clear that an
unpublished File-scoped label alone does not satisfy the pass condition.
In `@website/docs/tests/maester/MT.1173.md`:
- Around line 12-16: The pass criteria documentation in the final sentence is
ambiguous about the relationship between published policies and file-scoped
labels. Update the wording of the sentence beginning with "The test passes when
at least one label policy is published..." to clarify that at least one label
with the File scope must be present within a published policy, rather than
implying that published policies and file-scoped labels can exist independently.
This ensures the documentation accurately reflects that the file-scoped label
requirement must be met as part of an actual published policy.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 1cae768b-b436-4f4d-a337-adbdf01b34fa
📒 Files selected for processing (19)
powershell/Maester.psd1powershell/public/cisa/exchange/Get-MtExo.ps1powershell/public/maester/purview/Test-MtPurviewAiAuditLogIngestion.mdpowershell/public/maester/purview/Test-MtPurviewAiAuditLogIngestion.ps1powershell/public/maester/purview/Test-MtPurviewAiDlpPolicy.mdpowershell/public/maester/purview/Test-MtPurviewAiDlpPolicy.ps1powershell/public/maester/purview/Test-MtPurviewAiInsiderRiskPolicy.mdpowershell/public/maester/purview/Test-MtPurviewAiInsiderRiskPolicy.ps1powershell/public/maester/purview/Test-MtPurviewAiRetentionPolicy.mdpowershell/public/maester/purview/Test-MtPurviewAiRetentionPolicy.ps1powershell/public/maester/purview/Test-MtPurviewAiSensitivityLabelsForFiles.mdpowershell/public/maester/purview/Test-MtPurviewAiSensitivityLabelsForFiles.ps1tests/Maester/Purview/Test-MtPurviewAi.Tests.ps1tests/maester-config.jsonwebsite/docs/tests/maester/MT.1172.mdwebsite/docs/tests/maester/MT.1173.mdwebsite/docs/tests/maester/MT.1174.mdwebsite/docs/tests/maester/MT.1175.mdwebsite/docs/tests/maester/MT.1176.md
…rify file-scoped label pass criteria in MT.1173 docs
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
|
@OfirGavish, I just reviewed this again and saw something I should have noticed earlier: MT.1172 I'm hesitant to add [more] redundant checks, which can add up to make the whole scan take longer. Also, your description of For now, I would like to suggest:
I'm also going to start a separate conversation about potential ways to handle overlapping and redundant tests+documentation in case there might be value in some scenarios. Thanks! |
…ditLogIngestion and broaden description Addresses SamErde's feedback on PR maester365#1703. The unified audit log is not AI-specific - it is the tenant-wide foundation for Purview Audit, eDiscovery, Insider Risk, Communication Compliance, and DSPM for AI. Removing 'Ai' from the function name and rewriting the description/docs so the test's purpose is clear to anyone regardless of whether they care about Copilot specifically. - Renamed powershell/public/maester/purview/Test-MtPurviewAiAuditLogIngestion.{ps1,md} - Updated .SYNOPSIS/.DESCRIPTION/.EXAMPLE/.LINK and Write-Verbose in the PS1 - Rewrote result markdown (pass and fail branches) to reference Purview broadly - Updated remediation MD and website MT.1172.md to describe UAL as the general Purview foundation with AI as one of many downstream consumers - Updated Pester test invocation and Because clause - Updated maester-config.json title - Updated Maester.psd1 FunctionsToExport
|
Thanks for the thoughtful review, @SamErde — and for linking discussion #1898. You're absolutely right on both counts: the AI framing was misleading and the function name suggested a much narrower scope than what the test actually checks. I've pushed ecf2be5 to address the rename + description feedback: Renamed (via git mv so history is preserved): Test-MtPurviewAiAuditLogIngestion.ps1 → Test-MtPurviewAuditLogIngestion.ps1 Updated references in Maester.psd1 (FunctionsToExport), tests/maester-config.json (title), and the Pester It block / -Because clause. On the redundancy with Test-MtCisaAuditLog / Test-MtCisAuditLogSearch — completely fair, and #1898 is the right place to think about it holistically. I left MT.1172 in place for this PR because it slots into the DSPM for AI test suite (MT.1172–MT.1176) as the prerequisite check the whole Purview AI story depends on, and gives users running only the Purview scope a self-contained result without pulling in the CIS/CISA baselines. Happy to remove it in a follow-up commit here (or as a separate PR tied to #1898) if you'd prefer the leaner surface now — just say the word. |
|
Thanks, sounds great! We'll leave the "redundant" CISA and CIS tests as-is for the sake of benchmark purity. I'm on the road for the whole day and will review again when I get a chance! 🤘 |
Up to standards ✅🟢 Issues
|
Summary
Adds 5 Microsoft Purview Data Security Posture Management (DSPM) for AI governance tests covering Microsoft 365 Copilot oversharing, audit, and retention controls. Reserved in #697.
Test-MtPurviewAiAuditLogIngestionTest-MtPurviewAiSensitivityLabelsForFilesTest-MtPurviewAiInsiderRiskPolicyTest-MtPurviewAiDlpPolicyTest-MtPurviewAiRetentionPolicyWhy
Maester currently has solid coverage for Copilot Studio agent governance (MT.1113-1122) and Intune controls (MT.1123, MT.1147-1151), but has no tests covering the Microsoft Purview DSPM for AI controls that govern Microsoft 365 Copilot itself. These 5 tests close that gap by checking the most impactful DSPM-for-AI prerequisites and protective controls a tenant should have configured before broadly enabling Copilot.
What each test does
InsiderRiskScenario/Name/PolicyTemplatefor forward compatibility.Get-DlpCompliancePolicy:MicrosoftCopilotLocation,Workload(MicrosoftCopilot),Locations, andEnforcementPlanes(e.g.CopilotExperiences).Get-RetentionCompliancePolicyand the newerGet-AppRetentionCompliancePolicy(User:M365Copilot) — via theGet-MtExocache wrapper. Passes if either surface protects Copilot.Files
powershell/public/maester/purview/Test-MtPurviewAi*.ps1(5 functions)powershell/public/maester/purview/Test-MtPurviewAi*.md(5 companion remediation pages)website/docs/tests/maester/MT.117[2-6].md(5 website docs)tests/Maester/Purview/Test-MtPurviewAi.Tests.ps1(consolidated Pester suite)tests/maester-config.json(5 new entries)powershell/Maester.psd1(5 new functions added toFunctionsToExport)powershell/public/cisa/exchange/Get-MtExo.ps1(addedRetentionCompliancePolicy+AppRetentionCompliancePolicyto the cached commands hashtable so MT.1176 can route through the standardGet-MtExowrapper)Conventions
All tests follow the standard Maester pattern:
Test-MtConnectionguard,Get-MtLicenseInformationwhere applicable (MT.1175 only - DLP requires Exo DLP licence),try/catchwithNotAuthorized/Errorskips,Add-MtTestResultDetailwith Well done. prefix on pass,%TestResult%markdown table with risk callout on detail tests. EXO/SCC cmdlets are routed throughGet-MtExo -Request <CmdletSuffix>so they participate in$__MtSession.ExoCache.Validation
Invoke-ScriptAnalyzer -Severity Warning,Errorclean on all 6 new PS1 files.maester-config.jsonparses (5 new entries verified).Test-ModuleManifestsucceeds.Test-MtPurviewAiSensitivityLabelsForFileshas[SuppressMessage('PSUseSingularNouns', ...)]with justification (matches existing pattern fromTest-MtAIAgentMcpTools).True(PASS) on the test tenant; MT.1174 + MT.1175 + MT.1176 returnedFalsewith correctly-rendered detail markdown showing the empty-state row.Notes for reviewers
Get-SPOTenant, but Maester does not connect to the SharePoint Management Shell. Pivoted to checking the actual prerequisite (published file-scoped sensitivity labels) viaGet-Label+Get-LabelPolicy.SkippedBecause Custom(with reason) since there is no specificNotLicensedIRMenum value inAdd-MtTestResultDetail. Happy to add one in a follow-up if preferred.MicrosoftCopilotLocation,Workload,Locations,EnforcementPlanes).Get-RetentionCompliancePolicyandGet-AppRetentionCompliancePolicyper @SamErde's review, with graceful fallback if the app-retention cmdlet isn't available in the connected SCC session. Each policy is tagged with aSurfacecolumn (Legacy/App) in the result detail.Summary by CodeRabbit
New Features
Documentation
Tests