feat(audit): proposal security audit module (Claude + Tenderly) [BE-213]#1275
Open
cristianizzo wants to merge 17 commits into
Open
feat(audit): proposal security audit module (Claude + Tenderly) [BE-213]#1275cristianizzo wants to merge 17 commits into
cristianizzo wants to merge 17 commits into
Conversation
Prototype CLI that loads a proposal from the DB, runs a Tenderly simulateFull to capture the call trace, then invokes Claude Code CLI with a structured prompt to produce a security audit with risk level, findings, and recommendations. Run: yarn audit:proposal <network> <pluginAddress> <proposalIndex> Output: pretty terminal report by default; AUDIT_OUTPUT=json for raw. Tracks: BE-213
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a prototype CLI module to audit Aragon DAO proposals by fetching proposal context from Mongo, simulating execution via Tenderly, and sending a structured prompt to Claude CLI for JSON-formatted security findings.
Changes:
- Introduces versioned audit prompt (
prompt.md) and a prompt builder with<untrusted>wrapping/neutralization. - Adds CLI entrypoint to load proposal/plugin/settings, run Tenderly full simulation, call Claude CLI, and render a terminal report.
- Adds
yarn audit:proposalscript to run the CLI viats-node.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| src/modules/audit/promptBuilder.ts | Loads the prompt template, wraps untrusted context, and interpolates placeholders. |
| src/modules/audit/prompt.md | Defines the audit instructions + strict JSON output schema and placeholders for context injection. |
| src/modules/audit/audit.ts | Implements the CLI flow: args → Mongo fetch → Tenderly simulateFull → prompt → Claude CLI → pretty/json output. |
| package.json | Adds audit:proposal script for running the new CLI module. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Adds POST /v2/proposal/:id/audit backed by an aragon-gateway queue consumer, persists results to the Proposal collection, and gates concurrent runs with an auditRunning flag plus a stale-lock TTL so a crashed worker cannot orphan the lock.
cristianizzo
force-pushed
the
feat/BE-213
branch
from
April 27, 2026 15:15
6fdcc06 to
3fa3c04
Compare
Removes the spawn-based Claude CLI dependency in favour of the official SDK. Splits prompt.md into a cacheable system block (rules + schema + risk categories) and a per-request user template, with prompt caching on the system block. Adds AUDIT_MODEL / AUDIT_MAX_TOKENS config knobs and replaces the stale CLAUDE_BIN setting. Estimates per-audit cost from SDK usage including cache-read discounts.
Replaces the in-tree CLI entrypoint with a manual test in test/manual that calls AuditRunner.run directly, prints the structured audit, and exits. Strips Markdown ```json fences before JSON.parse so audits no longer fail when the model wraps output in a code block.
Switches PromptBuilder.build from multi-pass split/join to a single-
pass regex replace. The previous implementation re-scanned the
already-mutated output on every replacement, so a placeholder string
("{{TENDERLY}}") embedded in untrusted input could be substituted with
real Tenderly data on a later pass. Adds a regression test covering
that injection vector.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 20 out of 21 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Use AUDIT.TIMEOUT_MS + buffer for the RabbitMQ RPC wait so audits taking up to the SDK timeout do not get prematurely cancelled by the default 60s queue timeout. - Strip the model-output slice from the JSON-parse error message so proposal/Tenderly content cannot leak through gateway error logs. - Drop simulationId from IProposalAudit — it was always null and never persisted; tenderlyUrl already encodes the simulation reference. - Sanitize the gateway audit-job error log: emit only the error name and a 200-char message snippet instead of the full err.message.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 20 out of 21 changed files in this pull request and generated 3 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- releaseAudit now requires the same claimToken (== auditStartedAt returned by claimForAudit) so a worker whose lock was reclaimed by another instance after the TTL cannot clobber the new claimant when it eventually finishes. Adds runValidators to the same updateOne so malformed audit payloads are rejected at write time. - Threads the claimToken through ProposalController.auditProposal. - Clarifies in code (with an Anthropic docs link) that cache_control: ephemeral on the system block IS the prompt-caching enable knob, not a disabler. - Test additions: stale-token release must not clear a fresh lock; controller release-call assertions updated for the new signature.
The replace callback's `?? match` branch was unreachable — the regex is built from Object.keys(replacements) so every match is guaranteed to be a key. Removing it brings branches coverage back over the 90% threshold without weakening behaviour.
Adds the missing branch test where claimForAudit fails and the re-fetched proposal has flipped to executed.status=true between the initial read and the claim attempt — the controller must surface proposalAuditNotAllowed (not proposalAuditInProgress).
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 20 out of 21 changed files in this pull request and generated 6 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Validate the model's audit payload at runtime (riskLevel ∈ allowed set, summary non-empty string, findings + recommendations shape) before treating it as IProposalAudit. - releaseAudit now clears the lock even when persisting the audit fails validation, so an invalid model response can't pin the proposal at "in progress" until the stale-lock TTL. - Controller no longer rethrows raw RabbitMQ/transport errors — surfaces proposalAuditFailed and logs internally instead. - Broaden the wrapUntrusted regex to neutralize whitespace / attribute variants of <untrusted> tags (e.g. </untrusted >), so an attacker can't break out of the JSON sandbox by smuggling a non-canonical closing tag. - estimateCost is now keyed by the configured AUDIT.MODEL prefix and returns null for unknown models instead of silently mis-pricing. - Gateway handler tests look the audit handler up by queue name instead of relying on registration order (getCall(15)).
Adds focused tests for every validateAudit failure path (non-object, empty/missing summary, invalid riskLevel, malformed findings array, non-array / non-string recommendations, non-numeric actionIndex coercion) plus estimateCost behaviour (null for unknown models, opus-rate variant). Also covers the new releaseAudit branch where validator-rejected audit payloads must still clear the lock.
findWithEntityId uses an explicit $project allowlist, so the new proposal.audit field was being stored but stripped from GET /v2/proposals/:id responses — meaning the FE could never tell that an audit had been generated. Adds `audit: 1` to the projection and surfaces an optional `audit` field on IProposalsResponse.
6 tasks
- Fetch last 10 proposals from same plugin with their audits - Pass title, actions, executed status, and audit findings to Claude - Add historical pattern detection as mandatory cross-check - Bump prompt version to 3
… + governance checks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an end-to-end proposal security audit flow to backend2.
POST /v2/proposal/:id/auditreturns the cached audit if one exists, otherwise runs a fresh audit (Tenderly simulation + Anthropic SDK) and persists it on the Proposal document.audit.proposalRabbitMQ queue consumed byaragon-gateway(mirrors the existingcanCreateProposalRPC pattern).claimForAuditlock withauditStartedAtTTL — concurrent calls for the same proposal get 409proposalAuditInProgress; orphaned locks from a crashed worker are reclaimable afterAUDIT_STALE_LOCK_MS.@anthropic-ai/sdk(no Claude CLI binary in the image). The static instructions / output schema / risk categories live in a cacheable system prompt; only the dynamic<untrusted>proposal context is sent per-request, so subsequent audits within 5 min hit prompt cache (~70% cheaper).AUDIT.*config section (env-var driven) replaces the prototype's looseprocess.env.*reads.Linear: BE-213
API behaviour
IProposalAuditIProposalAuditproposalAuditNotAllowedproposalAuditInProgressproposalAuditFailedproposalNotFoundServer env vars (set on
aragon-gateway)Required:
AUDIT_ANTHROPIC_API_KEYOptional (defaults shown):
AUDIT_MODEL—claude-sonnet-4-5AUDIT_MAX_TOKENS—4096AUDIT_TIMEOUT_MS—300000AUDIT_STALE_LOCK_MS—600000Existing:
TENDERLY_*,MONGO_*,RABBITMQ_URI.aragon-apineeds no new env vars — it only enqueues the RPC message.Manual test
yarn audit:proposal <network> <pluginAddress> <proposalIndex>— calls the runner directly (skips API + queue), prints the structured audit JSON. Useful for validating Mongo + Tenderly + SDK end-to-end without standing up the services.Example real-world demo (executed treasury proposal — should be flagged):
Expected:
riskLevel: medium,descriptionMismatchfinding (proposal title says "Swap to Fiat" but actions are bare USDC transfers), ~$0.20–0.30 first call, ~19s.Tests
test/unit/modules/audit/runner.spec.ts,test/unit/modules/audit/promptBuilder.spec.ts(incl. placeholder-injection regression), plus extensions to existing controller/router/model/gateway specs covering the new audit paths.Test plan
AUDIT_ANTHROPIC_API_KEYon the deployedaragon-gatewayservice before mergeproposal.auditproposalAuditNotAllowedproposalAuditInProgress