diff --git a/README.md b/README.md index f228ba9..a27aa81 100644 --- a/README.md +++ b/README.md @@ -130,6 +130,8 @@ script. With neither, it reports `no_gate` with exit code 2. An unconfigured gat | `proofloop runner run --plan --budget-usd 100` | Run an append-only, budgeted task plan under `.proofloop/runner/runs//`. | | `proofloop runner resume --run-id latest` | Resume a runner after a crash; stale `running` tasks are requeued. | | `proofloop runner status --run-id latest [--json]` | Inspect durable runner state and ledger paths. | +| `proofloop transfer-check sample --capability --seed ` | Deterministic stratified browser-lane sample from capability results (failures included by design). | +| `proofloop transfer-check gate --capability --browser ` | Two-layer agreement gate: exit 0 agreed / 1 diverged / 2 unusable (fail-closed). | | `proofloop mcp` | Start the optional read-only MCP server. | | `proofloop gate [--check]` | Run configured checks or `npm test`; exit 0 pass, 1 fail, 2 unusable. | | `proofloop hooks install\|uninstall\|status` | Install/remove/status Claude Code Stop, PreToolUse, and PostToolUse hooks. | @@ -165,6 +167,51 @@ npx proofloop tooluse verify --contract tooluse-contract.json The verifier is fail-closed: a deny-list cannot be certified from an empty or missing log, and server-pinned names mean `mcp__evil__X` cannot impersonate `mcp__composio__X`. +## Two-layer certification: the transfer gate + +Running every benchmark task through a real browser is slow and expensive, so the +two-layer model (`two-layer-certification-v1`) splits certification: + +- the **capability lane** runs ALL tasks through the live agent harness (headless, cheap); +- the **browser lane** replays a small stratified sample through the real production UI. + +The split saves ~95% of cost and wall-clock, but it opens a gaming surface: the harness +lane could take shortcuts the product path does not have (memory-mode, bypassed tool +contracts) and nothing would notice. `proofloop transfer-check` closes that hole by +treating the browser sample as an **agreement test on the capability lane** -- the +anti-cheat doctrine's in-app transfer rule made mechanical. Divergence fails closed. + +```bash +# 1. Sample which capability results the browser lane must replay. +# Pass the commit SHA as the seed: same seed + same input = byte-identical sample, +# so the agent cannot re-roll the dice until it gets a sample it likes. +npx proofloop transfer-check sample --capability capability-results.json --seed "$(git rev-parse HEAD)" > transfer-sample.json + +# 2. Run the sampled tasks through the real UI, record browser receipts, then gate. +npx proofloop transfer-check gate --capability capability-results.json --browser browser-receipts.json +``` + +Both lanes accept either a receipts JSON array (`[{ taskId, model, family?, pass }]`) +or a runner events ledger (`.proofloop/runner/runs//ledger.jsonl`; label ledger +rows with `--model`). + +**Why failures must be sampled.** Verifying capability-lane failures is as important as +verifying passes: a failure that passes in the browser exposes a harness bug (the lane is +under-reporting); a pass that fails in the browser exposes a harness shortcut (the claim +is inflated). The sampler therefore reserves at least `ceil(N/3)` slots per family for +failures whenever failures exist, and the gate refuses (exit 2, cherry-pick guard) a +browser set that paired with zero capability failures -- overridable only by an explicit +`--allow-no-failure-overlap`, which prints a loud warning into the output. + +**Claim language rule.** On agreement the gate prints exactly what was proven: +capability verified through the harness, plus transfer verified on N seeded pairs -- +never "all tasks browser-verified". Exit codes: `0` agreement >= `--min-agreement` +(default 0.9) with overlap >= `--min-overlap` (default 5); `1` divergence, with each +disagreeing pair labeled by direction (capability-pass/browser-fail = suspected harness +shortcut; capability-fail/browser-pass = suspected harness under-reporting); `2` +unusable evidence (unreadable inputs, thin overlap, or cherry-picked sample). Zero +evidence never exits 0. + ## Scope This package is the portable core: gate, refuse-fake-done hooks, expected-tool-use contracts, diff --git a/dist/cli.js b/dist/cli.js index 787df72..6917e95 100644 --- a/dist/cli.js +++ b/dist/cli.js @@ -14,6 +14,7 @@ exports.runCli = runCli; * proofloop ci install github write the GitHub Actions gate workflow * proofloop prompt print the one-prompt kickoff * proofloop this-repo [--goal ...] [--write-runner-plan] [--run] + * proofloop transfer-check sample|gate two-layer certification agreement gate * proofloop manifest|docs|template|workflow|ui|resume|report|charts|mcp * * Exit codes are per-command (documented at each case). Zero runtime deps. @@ -30,6 +31,7 @@ const proofloopToolUse_1 = require("./proofloopToolUse"); const mcp_1 = require("./mcp"); const project_1 = require("./project"); const runner_1 = require("./runner"); +const transferCheck_1 = require("./transferCheck"); exports.MCP_SERVER_RUNNING = -999; /** Parse `--flag`, `--flag value`, `--flag=value`, and positionals. */ function parseArgs(argv) { @@ -92,6 +94,7 @@ function usage() { " report latest [--json] latest gate report", " charts latest write local JSON/SVG proof charts", " runner run|resume|status durable append-only task runner with budget and resume", + " transfer-check sample|gate two-layer certification: seeded browser sample + agreement gate", " mcp start the optional read-only MCP server", " prompt print the one-prompt kickoff", " this-repo [--goal ] [--write-runner-plan] [--run]", @@ -157,6 +160,8 @@ function runCli(argv) { return runChartsCommand(positional[1], root); case "runner": return runRunnerCommand(positional[1], options, root); + case "transfer-check": + return (0, transferCheck_1.runTransferCheckCommand)(positional[1], options, root); case "mcp": (0, mcp_1.startMcpServer)({ root }); return exports.MCP_SERVER_RUNNING; diff --git a/dist/index.d.ts b/dist/index.d.ts index a790b11..e07fe40 100644 --- a/dist/index.d.ts +++ b/dist/index.d.ts @@ -18,4 +18,5 @@ export * from "./project"; export * from "./mcp"; export * from "./runner"; export * from "./layeredPlan"; +export * from "./transferCheck"; export { runCli } from "./cli"; diff --git a/dist/index.js b/dist/index.js index c2e5150..9ec5f00 100644 --- a/dist/index.js +++ b/dist/index.js @@ -35,5 +35,6 @@ __exportStar(require("./project"), exports); __exportStar(require("./mcp"), exports); __exportStar(require("./runner"), exports); __exportStar(require("./layeredPlan"), exports); +__exportStar(require("./transferCheck"), exports); var cli_1 = require("./cli"); Object.defineProperty(exports, "runCli", { enumerable: true, get: function () { return cli_1.runCli; } }); diff --git a/dist/prompt.d.ts b/dist/prompt.d.ts index 2245032..fdf655e 100644 --- a/dist/prompt.d.ts +++ b/dist/prompt.d.ts @@ -6,5 +6,5 @@ * mentioned here is a real command. */ /** The set of top-level commands the package CLI implements. */ -export declare const PACKAGE_COMMANDS: readonly ["init", "doctor", "gate", "hooks", "tooluse", "ci", "manifest", "docs", "template", "workflow", "ui", "resume", "report", "charts", "runner", "mcp", "prompt", "this-repo"]; +export declare const PACKAGE_COMMANDS: readonly ["init", "doctor", "gate", "hooks", "tooluse", "ci", "manifest", "docs", "template", "workflow", "ui", "resume", "report", "charts", "runner", "transfer-check", "mcp", "prompt", "this-repo"]; export declare function proofloopKickoffPrompt(): string; diff --git a/dist/prompt.js b/dist/prompt.js index ed15b29..2ad975b 100644 --- a/dist/prompt.js +++ b/dist/prompt.js @@ -26,6 +26,7 @@ exports.PACKAGE_COMMANDS = [ "report", "charts", "runner", + "transfer-check", "mcp", "prompt", "this-repo", diff --git a/dist/transferCheck.d.ts b/dist/transferCheck.d.ts new file mode 100644 index 0000000..40dbe6c --- /dev/null +++ b/dist/transferCheck.d.ts @@ -0,0 +1,87 @@ +export declare const TRANSFER_SAMPLE_SCHEMA = "proofloop-transfer-sample-v1"; +export type TransferLaneResult = { + taskId: string; + model: string; + family: string; + pass: boolean; +}; +export type TransferLaneRead = { + results: TransferLaneResult[]; + warnings: string[]; + source: "receipts" | "runner-ledger"; +}; +export type TransferSamplePair = { + taskId: string; + model: string; + family: string; + capabilityPass: boolean; +}; +export type TransferSamplePlan = { + schema: typeof TRANSFER_SAMPLE_SCHEMA; + seed: string; + pairs: TransferSamplePair[]; +}; +export type TransferDisagreementDirection = "capability-pass-browser-fail" | "capability-fail-browser-pass"; +export declare const TRANSFER_DIRECTION_LABELS: Record; +export type TransferDisagreement = { + taskId: string; + model: string; + family: string; + capabilityPass: boolean; + browserPass: boolean; + direction: TransferDisagreementDirection; + label: string; +}; +export type TransferGateEvaluation = { + status: "agreed" | "diverged" | "unusable"; + /** Present when status === "unusable". */ + reason?: string; + overlap: number; + matches: number; + agreementRatio: number; + minAgreement: number; + minOverlap: number; + capabilityFailuresTotal: number; + pairedCapabilityFailures: number; + disagreements: TransferDisagreement[]; + warnings: string[]; +}; +export declare function readTransferLaneResults(path: string, options?: { + ledgerModel?: string; +}): TransferLaneRead; +/** + * Deterministic stratified sample of the capability lane for browser + * certification. Same seed + same input = byte-identical plan (the per-family + * PRNG stream is `fnv1a(seed|family)`, so adding a family never changes + * another family's picks). If a family has capability failures, at least + * ceil(perFamily/3) of its sample slots are failures (capped by how many + * failures exist): failures must transfer too, or the failure path of the + * harness is never checked against the product. + */ +export declare function buildTransferSample(capability: readonly TransferLaneResult[], options: { + seed: string; + perFamily?: number; +}): TransferSamplePlan; +export declare function evaluateTransferGate(capability: readonly TransferLaneResult[], browser: readonly TransferLaneResult[], options?: { + minAgreement?: number; + minOverlap?: number; + allowNoFailureOverlap?: boolean; +}): TransferGateEvaluation; +/** + * The doctrine claim line printed on exit 0 -- deliberately scoped language. + * The browser lane certified a stratified sample, so the only honest claim is + * "capability lane verified + transfer verified on N seeded pairs", never + * "all tasks browser-verified". + */ +export declare function transferClaimLine(evaluation: TransferGateEvaluation): string; +type TransferCheckIo = { + log?: (line: string) => void; + logError?: (line: string) => void; +}; +/** + * `proofloop transfer-check sample|gate`. Exit codes: + * sample: 0 wrote/printed the plan, 2 unusable input or bad flags. + * gate: 0 agreed, 1 diverged, 2 unusable (fail-closed). + */ +export declare function runTransferCheckCommand(sub: string | undefined, options: Record, root: string, io?: TransferCheckIo): number; +export {}; diff --git a/dist/transferCheck.js b/dist/transferCheck.js new file mode 100644 index 0000000..9a88a5b --- /dev/null +++ b/dist/transferCheck.js @@ -0,0 +1,485 @@ +"use strict"; +Object.defineProperty(exports, "__esModule", { value: true }); +exports.TRANSFER_DIRECTION_LABELS = exports.TRANSFER_SAMPLE_SCHEMA = void 0; +exports.readTransferLaneResults = readTransferLaneResults; +exports.buildTransferSample = buildTransferSample; +exports.evaluateTransferGate = evaluateTransferGate; +exports.transferClaimLine = transferClaimLine; +exports.runTransferCheckCommand = runTransferCheckCommand; +/** + * `proofloop transfer-check` -- the two-layer certification AGREEMENT GATE. + * + * WHY THIS EXISTS: Proof Loop is adopting a two-layer certification model + * ("two-layer-certification-v1", the vocabulary shared with layeredPlan.ts): + * + * - a headless CAPABILITY lane runs ALL benchmark tasks through the live + * agent harness (cheap, no browser); + * - a BROWSER lane runs a small stratified sample through the real + * production UI. + * + * The split saves ~95% of cost and wall-clock, BUT it creates a new gaming + * surface: the harness lane could take shortcuts the product path does not + * have (memory-mode, bypassed tool contracts) and nothing would notice. + * transfer-check makes the anti-cheat doctrine's IN-APP TRANSFER rule + * mechanical: browser samples are an agreement test ON the capability lane, + * and divergence fails closed. + * + * Direction matters, so disagreements are labeled distinctly: + * - capability-pass / browser-fail => suspected harness shortcut or + * product-path break -- the capability claim is suspect. + * - capability-fail / browser-pass => suspected harness bug or env gap -- + * the capability lane is under-reporting. + * + * Verifying failures is as important as verifying passes: a failure that + * passes in the browser exposes a harness bug; a pass that fails in the + * browser exposes a harness shortcut. The sampler therefore MUST include + * capability-lane failures, and the gate refuses (exit 2, cherry-pick guard) + * a browser set that dodged every capability failure. + * + * Subcommands: + * proofloop transfer-check sample --capability --seed + * [--per-family 5] [--model