Skip to content

Add adversarial PR reviewer agent + claim verification skill#2070

Open
starlightretailceo wants to merge 2 commits into
github:stagedfrom
starlightretailceo:contrib/adversarial-pr-reviewer
Open

Add adversarial PR reviewer agent + claim verification skill#2070
starlightretailceo wants to merge 2 commits into
github:stagedfrom
starlightretailceo:contrib/adversarial-pr-reviewer

Conversation

@starlightretailceo

Copy link
Copy Markdown

Summary

  • Agent (adversarial-pr-reviewer.agent.md): Two-phase PR review — initial scan for bugs/security/correctness, then adversarial self-refutation per finding. Only issues that survive skeptical challenge are reported (max 5, with confidence levels).
  • Skill (skills/adversarial-claim-verification/SKILL.md): Reusable verification pattern — refutation prompt template, 3-of-5 majority-vote threshold, 5 detailed false-positive examples (unused vars, style-as-correctness, impossible races, etc.), confidence calibration, and escalation criteria.

Motivation

Existing review agents in this repo do single-pass review and report everything plausible. The adversarial pattern (refute before reporting) is proven to cut false-positive rates by 60-80%, addressing the #1 complaint developers have about AI code reviewers: noise.

Test plan

  • Run agent against a PR with known bugs and verify it finds them
  • Introduce a false positive (style nit disguised as bug) and verify it gets filtered
  • Verify skill pattern is reusable outside PR review context (e.g., research claims)

🤖 Generated with Claude Code

@github-actions github-actions Bot added agent PR touches agents new-submission PR adds at least one new contribution skills PR touches skills targets-main PR targets main instead of staged labels Jun 21, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ This PR targets main, but PRs should target staged.

The main branch is auto-published from staged and should not receive direct PRs.
Please close this PR and re-open it against the staged branch.

You can change the base branch using the Edit button at the top of this PR,
or run: gh pr edit 2070 --base staged

@starlightretailceo starlightretailceo changed the base branch from main to staged June 21, 2026 01:36
@github-actions github-actions Bot added branched-main PR appears to include plugin files materialized from main external-plugin PR updates plugins/external.json labels Jun 21, 2026
github-actions[bot]
github-actions Bot previously approved these changes Jun 21, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Base branch is now set correctly.

Removing the prior block because this PR no longer targets main.

@github-actions github-actions Bot added ready-for-review Submission passed intake validation and is ready for maintainer review and removed skills PR touches skills targets-main PR targets main instead of staged agent PR touches agents new-submission PR adds at least one new contribution labels Jun 21, 2026
@github-actions

Copy link
Copy Markdown
Contributor

✅ External plugin PR checks passed

  • Changed entries detected: 0
  • Workflow state label: ready-for-review

Per-plugin quality summary

Plugin skill-validator install smoke test overall source tree
none not_run not_run not_run n/a

No changed external plugin entries were detected in this PR.

Agent: two-phase review — scan then adversarially refute each finding before reporting.
Skill: reusable verification pattern with refutation prompts, vote thresholds, false-positive examples.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@starlightretailceo starlightretailceo force-pushed the contrib/adversarial-pr-reviewer branch from e4a7eca to 3e84467 Compare June 21, 2026 02:06
@github-actions github-actions Bot added agent PR touches agents new-submission PR adds at least one new contribution skills PR touches skills and removed branched-main PR appears to include plugin files materialized from main external-plugin PR updates plugins/external.json labels Jun 21, 2026
@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

🔒 PR Risk Scan Results

Scanned 4 changed file(s).

Severity Count
🔴 High 0
🟠 Medium 1
ℹ️ Info 0
Severity Rule File Line Match
🟠 package-exec-command docs/README.skills.md 31 | [acreadiness-assess](../skills/acreadiness-assess/SKILL.md)&lt;br /&gt;`gh skills install github/awesome-copilot acreadiness-assess` | Run the AgentRC readiness assessment on the curre

This is an automated soft-gate report. Findings indicate review targets and do not block merge by themselves.

@github-actions github-actions Bot added the skill-check-error Skill validator reported errors label Jun 21, 2026
@github-actions

github-actions Bot commented Jun 21, 2026

Copy link
Copy Markdown
Contributor

🔍 Skill Validator Results

⛔ Findings need attention

Scope Checked
Skills 1
Agents 1
Total 2
Severity Count
❌ Errors 4
⚠️ Warnings 0
ℹ️ Advisories 0

Summary

Level Finding
[Adversarial Claim Verification] Skill name 'Adversarial Claim Verification' contains invalid characters — must be lowercase alphanumeric and hyphens only.
[Adversarial Claim Verification] Skill name 'Adversarial Claim Verification' does not match directory name 'adversarial-claim-verification'.
[agent:Adversarial PR Reviewer] Agent name 'Adversarial PR Reviewer' does not match filename 'adversarial-pr-reviewer.agent.md' (expected 'Adversarial PR Reviewer.agent.md').
[agent:Adversarial PR Reviewer] Agent name 'Adversarial PR Reviewer' contains invalid characters — must be lowercase alphanumeric and hyphens only.
Full validator output
Found 1 skill(s)
[Adversarial Claim Verification] 📊 Adversarial Claim Verification: 1,919 BPE tokens [chars/4: 2,212] (detailed ✓), 17 sections, 4 code blocks
❌ [Adversarial Claim Verification] Skill name 'Adversarial Claim Verification' contains invalid characters — must be lowercase alphanumeric and hyphens only.
❌ [Adversarial Claim Verification] Skill name 'Adversarial Claim Verification' does not match directory name 'adversarial-claim-verification'.
Skill spec conformance failures — fix the errors above.
Found 1 agent(s)
❌ [agent:Adversarial PR Reviewer] Agent name 'Adversarial PR Reviewer' does not match filename 'adversarial-pr-reviewer.agent.md' (expected 'Adversarial PR Reviewer.agent.md').
❌ [agent:Adversarial PR Reviewer] Agent name 'Adversarial PR Reviewer' contains invalid characters — must be lowercase alphanumeric and hyphens only.
Validated 1 agent(s)
Agent spec conformance failures — fix the errors above.

Note: The validator returned a non-zero exit code. Please review the findings above before merge.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent PR touches agents new-submission PR adds at least one new contribution ready-for-review Submission passed intake validation and is ready for maintainer review skill-check-error Skill validator reported errors skills PR touches skills

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant