From eea0bea3f9b19087474fcd311d3941f474b43bdc Mon Sep 17 00:00:00 2001 From: homen Date: Sun, 5 Jul 2026 13:38:30 -0700 Subject: [PATCH] feat(transfer-check): two-layer certification agreement gate -- makes the headless/browser split un-gameable The two-layer certification model (headless capability lane runs ALL benchmark tasks cheaply; browser lane runs a stratified sample on the real prod UI) saves ~95% of cost/wall-clock but opens a new gaming surface: the headless harness could take shortcuts the product path doesn't have (memory-mode, bypassed tool contracts) and nothing would notice. `proofloop transfer-check` closes that -- it is the anti-cheat doctrine's IN-APP TRANSFER rule made mechanical. - `transfer-check sample --capability --per-family N --seed `: deterministic hash-seeded stratified sampler (pass a commit SHA so neither human nor agent can cherry-pick easy tasks -- same seed + input = byte-identical sample, tested). MUST include capability-lane failures (>= ceil(N/3) per family that has them): a "failure" that passes in the browser exposes a harness bug; a "pass" that fails in the browser exposes a harness shortcut -- both directions are evidence. - `transfer-check gate --capability --browser [--min-agreement 0.9] [--min-overlap 5]`: joins on taskId+model, computes agreement. Exit 0 = agreed (prints the doctrine claim verbatim: "Capability verified through the live agent harness; production browser path verified by stratified UI certification... This is NOT an all-tasks-browser-verified claim"); exit 1 = diverged, with a per-pair table labeling DIRECTION distinctly (capability-pass/browser-fail = "suspected harness shortcut or product-path break -- capability claim suspect"; capability-fail/browser-pass = "suspected harness bug or env gap -- capability lane under-reporting"); exit 2 = fail-closed (overlap below min, unreadable input, or CHERRY-PICK GUARD: capability has failures but the browser paired set has zero of them -> refuse unless --allow-no-failure-overlap, which warns loudly and certifies "including 0 capability-failures"). Never exit 0 on zero evidence. - Readers accept a generic receipts JSON array OR the merged runner.ts events ledger (.proofloop/runner/runs//ledger.jsonl -- real path, verified by round-tripping a real `proofloop runner run`); ledger rows carry no model field so --model labels them (fail-closed: mismatched models -> zero overlap -> exit 2). Deliberately COMPLEMENTARY to the merged layeredPlan.ts (the two-layer PLAN generator): this is the AGREEMENT GATE, not a second plan surface. 16 new scenario tests (agreement pass/exactly-threshold, both divergence directions, thin-overlap exit 2, cherry-pick guard + override, seeded determinism, failure-inclusion, real ledger round-trip, duplicate rejection); 81/81 total green; tsc strict. Independently re-verified live: deterministic sample with 5 failures included, exit 0/1/2 all correct, both divergence labels, cherry-pick guard firing. Registered in PACKAGE_COMMANDS (grep-asserted honesty) + README "Two-layer certification: the transfer gate" section documenting the claim-language rule (never "all tasks browser verified" unless literal). Weakest spots documented: --model labels both lanes for ledger inputs (use receipts format for cross-model); byte-identity depends on stable JSON key order (determinism test guards it); the gate certifies agreement of inputs given -- fabricated browser receipts are out of scope (mitigations are the SHA seed + runner-ledger source, not receipt provenance). Co-Authored-By: Claude Fable 5 --- README.md | 47 +++ dist/cli.js | 5 + dist/index.d.ts | 1 + dist/index.js | 1 + dist/prompt.d.ts | 2 +- dist/prompt.js | 1 + dist/transferCheck.d.ts | 87 ++++++ dist/transferCheck.js | 485 ++++++++++++++++++++++++++++++ src/cli.ts | 6 + src/index.ts | 1 + src/prompt.ts | 1 + src/transferCheck.ts | 577 ++++++++++++++++++++++++++++++++++++ tests/transferCheck.test.ts | 369 +++++++++++++++++++++++ 13 files changed, 1582 insertions(+), 1 deletion(-) create mode 100644 dist/transferCheck.d.ts create mode 100644 dist/transferCheck.js create mode 100644 src/transferCheck.ts create mode 100644 tests/transferCheck.test.ts diff --git a/README.md b/README.md index f228ba9..a27aa81 100644 --- a/README.md +++ b/README.md @@ -130,6 +130,8 @@ script. With neither, it reports `no_gate` with exit code 2. An unconfigured gat | `proofloop runner run --plan --budget-usd 100` | Run an append-only, budgeted task plan under `.proofloop/runner/runs//`. | | `proofloop runner resume --run-id latest` | Resume a runner after a crash; stale `running` tasks are requeued. | | `proofloop runner status --run-id latest [--json]` | Inspect durable runner state and ledger paths. | +| `proofloop transfer-check sample --capability --seed ` | Deterministic stratified browser-lane sample from capability results (failures included by design). | +| `proofloop transfer-check gate --capability --browser ` | Two-layer agreement gate: exit 0 agreed / 1 diverged / 2 unusable (fail-closed). | | `proofloop mcp` | Start the optional read-only MCP server. | | `proofloop gate [--check]` | Run configured checks or `npm test`; exit 0 pass, 1 fail, 2 unusable. | | `proofloop hooks install\|uninstall\|status` | Install/remove/status Claude Code Stop, PreToolUse, and PostToolUse hooks. | @@ -165,6 +167,51 @@ npx proofloop tooluse verify --contract tooluse-contract.json The verifier is fail-closed: a deny-list cannot be certified from an empty or missing log, and server-pinned names mean `mcp__evil__X` cannot impersonate `mcp__composio__X`. +## Two-layer certification: the transfer gate + +Running every benchmark task through a real browser is slow and expensive, so the +two-layer model (`two-layer-certification-v1`) splits certification: + +- the **capability lane** runs ALL tasks through the live agent harness (headless, cheap); +- the **browser lane** replays a small stratified sample through the real production UI. + +The split saves ~95% of cost and wall-clock, but it opens a gaming surface: the harness +lane could take shortcuts the product path does not have (memory-mode, bypassed tool +contracts) and nothing would notice. `proofloop transfer-check` closes that hole by +treating the browser sample as an **agreement test on the capability lane** -- the +anti-cheat doctrine's in-app transfer rule made mechanical. Divergence fails closed. + +```bash +# 1. Sample which capability results the browser lane must replay. +# Pass the commit SHA as the seed: same seed + same input = byte-identical sample, +# so the agent cannot re-roll the dice until it gets a sample it likes. +npx proofloop transfer-check sample --capability capability-results.json --seed "$(git rev-parse HEAD)" > transfer-sample.json + +# 2. Run the sampled tasks through the real UI, record browser receipts, then gate. +npx proofloop transfer-check gate --capability capability-results.json --browser browser-receipts.json +``` + +Both lanes accept either a receipts JSON array (`[{ taskId, model, family?, pass }]`) +or a runner events ledger (`.proofloop/runner/runs//ledger.jsonl`; label ledger +rows with `--model`). + +**Why failures must be sampled.** Verifying capability-lane failures is as important as +verifying passes: a failure that passes in the browser exposes a harness bug (the lane is +under-reporting); a pass that fails in the browser exposes a harness shortcut (the claim +is inflated). The sampler therefore reserves at least `ceil(N/3)` slots per family for +failures whenever failures exist, and the gate refuses (exit 2, cherry-pick guard) a +browser set that paired with zero capability failures -- overridable only by an explicit +`--allow-no-failure-overlap`, which prints a loud warning into the output. + +**Claim language rule.** On agreement the gate prints exactly what was proven: +capability verified through the harness, plus transfer verified on N seeded pairs -- +never "all tasks browser-verified". Exit codes: `0` agreement >= `--min-agreement` +(default 0.9) with overlap >= `--min-overlap` (default 5); `1` divergence, with each +disagreeing pair labeled by direction (capability-pass/browser-fail = suspected harness +shortcut; capability-fail/browser-pass = suspected harness under-reporting); `2` +unusable evidence (unreadable inputs, thin overlap, or cherry-picked sample). Zero +evidence never exits 0. + ## Scope This package is the portable core: gate, refuse-fake-done hooks, expected-tool-use contracts, diff --git a/dist/cli.js b/dist/cli.js index 787df72..6917e95 100644 --- a/dist/cli.js +++ b/dist/cli.js @@ -14,6 +14,7 @@ exports.runCli = runCli; * proofloop ci install github write the GitHub Actions gate workflow * proofloop prompt print the one-prompt kickoff * proofloop this-repo [--goal ...] [--write-runner-plan] [--run] + * proofloop transfer-check sample|gate two-layer certification agreement gate * proofloop manifest|docs|template|workflow|ui|resume|report|charts|mcp * * Exit codes are per-command (documented at each case). Zero runtime deps. @@ -30,6 +31,7 @@ const proofloopToolUse_1 = require("./proofloopToolUse"); const mcp_1 = require("./mcp"); const project_1 = require("./project"); const runner_1 = require("./runner"); +const transferCheck_1 = require("./transferCheck"); exports.MCP_SERVER_RUNNING = -999; /** Parse `--flag`, `--flag value`, `--flag=value`, and positionals. */ function parseArgs(argv) { @@ -92,6 +94,7 @@ function usage() { " report latest [--json] latest gate report", " charts latest write local JSON/SVG proof charts", " runner run|resume|status durable append-only task runner with budget and resume", + " transfer-check sample|gate two-layer certification: seeded browser sample + agreement gate", " mcp start the optional read-only MCP server", " prompt print the one-prompt kickoff", " this-repo [--goal ] [--write-runner-plan] [--run]", @@ -157,6 +160,8 @@ function runCli(argv) { return runChartsCommand(positional[1], root); case "runner": return runRunnerCommand(positional[1], options, root); + case "transfer-check": + return (0, transferCheck_1.runTransferCheckCommand)(positional[1], options, root); case "mcp": (0, mcp_1.startMcpServer)({ root }); return exports.MCP_SERVER_RUNNING; diff --git a/dist/index.d.ts b/dist/index.d.ts index a790b11..e07fe40 100644 --- a/dist/index.d.ts +++ b/dist/index.d.ts @@ -18,4 +18,5 @@ export * from "./project"; export * from "./mcp"; export * from "./runner"; export * from "./layeredPlan"; +export * from "./transferCheck"; export { runCli } from "./cli"; diff --git a/dist/index.js b/dist/index.js index c2e5150..9ec5f00 100644 --- a/dist/index.js +++ b/dist/index.js @@ -35,5 +35,6 @@ __exportStar(require("./project"), exports); __exportStar(require("./mcp"), exports); __exportStar(require("./runner"), exports); __exportStar(require("./layeredPlan"), exports); +__exportStar(require("./transferCheck"), exports); var cli_1 = require("./cli"); Object.defineProperty(exports, "runCli", { enumerable: true, get: function () { return cli_1.runCli; } }); diff --git a/dist/prompt.d.ts b/dist/prompt.d.ts index 2245032..fdf655e 100644 --- a/dist/prompt.d.ts +++ b/dist/prompt.d.ts @@ -6,5 +6,5 @@ * mentioned here is a real command. */ /** The set of top-level commands the package CLI implements. */ -export declare const PACKAGE_COMMANDS: readonly ["init", "doctor", "gate", "hooks", "tooluse", "ci", "manifest", "docs", "template", "workflow", "ui", "resume", "report", "charts", "runner", "mcp", "prompt", "this-repo"]; +export declare const PACKAGE_COMMANDS: readonly ["init", "doctor", "gate", "hooks", "tooluse", "ci", "manifest", "docs", "template", "workflow", "ui", "resume", "report", "charts", "runner", "transfer-check", "mcp", "prompt", "this-repo"]; export declare function proofloopKickoffPrompt(): string; diff --git a/dist/prompt.js b/dist/prompt.js index ed15b29..2ad975b 100644 --- a/dist/prompt.js +++ b/dist/prompt.js @@ -26,6 +26,7 @@ exports.PACKAGE_COMMANDS = [ "report", "charts", "runner", + "transfer-check", "mcp", "prompt", "this-repo", diff --git a/dist/transferCheck.d.ts b/dist/transferCheck.d.ts new file mode 100644 index 0000000..40dbe6c --- /dev/null +++ b/dist/transferCheck.d.ts @@ -0,0 +1,87 @@ +export declare const TRANSFER_SAMPLE_SCHEMA = "proofloop-transfer-sample-v1"; +export type TransferLaneResult = { + taskId: string; + model: string; + family: string; + pass: boolean; +}; +export type TransferLaneRead = { + results: TransferLaneResult[]; + warnings: string[]; + source: "receipts" | "runner-ledger"; +}; +export type TransferSamplePair = { + taskId: string; + model: string; + family: string; + capabilityPass: boolean; +}; +export type TransferSamplePlan = { + schema: typeof TRANSFER_SAMPLE_SCHEMA; + seed: string; + pairs: TransferSamplePair[]; +}; +export type TransferDisagreementDirection = "capability-pass-browser-fail" | "capability-fail-browser-pass"; +export declare const TRANSFER_DIRECTION_LABELS: Record; +export type TransferDisagreement = { + taskId: string; + model: string; + family: string; + capabilityPass: boolean; + browserPass: boolean; + direction: TransferDisagreementDirection; + label: string; +}; +export type TransferGateEvaluation = { + status: "agreed" | "diverged" | "unusable"; + /** Present when status === "unusable". */ + reason?: string; + overlap: number; + matches: number; + agreementRatio: number; + minAgreement: number; + minOverlap: number; + capabilityFailuresTotal: number; + pairedCapabilityFailures: number; + disagreements: TransferDisagreement[]; + warnings: string[]; +}; +export declare function readTransferLaneResults(path: string, options?: { + ledgerModel?: string; +}): TransferLaneRead; +/** + * Deterministic stratified sample of the capability lane for browser + * certification. Same seed + same input = byte-identical plan (the per-family + * PRNG stream is `fnv1a(seed|family)`, so adding a family never changes + * another family's picks). If a family has capability failures, at least + * ceil(perFamily/3) of its sample slots are failures (capped by how many + * failures exist): failures must transfer too, or the failure path of the + * harness is never checked against the product. + */ +export declare function buildTransferSample(capability: readonly TransferLaneResult[], options: { + seed: string; + perFamily?: number; +}): TransferSamplePlan; +export declare function evaluateTransferGate(capability: readonly TransferLaneResult[], browser: readonly TransferLaneResult[], options?: { + minAgreement?: number; + minOverlap?: number; + allowNoFailureOverlap?: boolean; +}): TransferGateEvaluation; +/** + * The doctrine claim line printed on exit 0 -- deliberately scoped language. + * The browser lane certified a stratified sample, so the only honest claim is + * "capability lane verified + transfer verified on N seeded pairs", never + * "all tasks browser-verified". + */ +export declare function transferClaimLine(evaluation: TransferGateEvaluation): string; +type TransferCheckIo = { + log?: (line: string) => void; + logError?: (line: string) => void; +}; +/** + * `proofloop transfer-check sample|gate`. Exit codes: + * sample: 0 wrote/printed the plan, 2 unusable input or bad flags. + * gate: 0 agreed, 1 diverged, 2 unusable (fail-closed). + */ +export declare function runTransferCheckCommand(sub: string | undefined, options: Record, root: string, io?: TransferCheckIo): number; +export {}; diff --git a/dist/transferCheck.js b/dist/transferCheck.js new file mode 100644 index 0000000..9a88a5b --- /dev/null +++ b/dist/transferCheck.js @@ -0,0 +1,485 @@ +"use strict"; +Object.defineProperty(exports, "__esModule", { value: true }); +exports.TRANSFER_DIRECTION_LABELS = exports.TRANSFER_SAMPLE_SCHEMA = void 0; +exports.readTransferLaneResults = readTransferLaneResults; +exports.buildTransferSample = buildTransferSample; +exports.evaluateTransferGate = evaluateTransferGate; +exports.transferClaimLine = transferClaimLine; +exports.runTransferCheckCommand = runTransferCheckCommand; +/** + * `proofloop transfer-check` -- the two-layer certification AGREEMENT GATE. + * + * WHY THIS EXISTS: Proof Loop is adopting a two-layer certification model + * ("two-layer-certification-v1", the vocabulary shared with layeredPlan.ts): + * + * - a headless CAPABILITY lane runs ALL benchmark tasks through the live + * agent harness (cheap, no browser); + * - a BROWSER lane runs a small stratified sample through the real + * production UI. + * + * The split saves ~95% of cost and wall-clock, BUT it creates a new gaming + * surface: the harness lane could take shortcuts the product path does not + * have (memory-mode, bypassed tool contracts) and nothing would notice. + * transfer-check makes the anti-cheat doctrine's IN-APP TRANSFER rule + * mechanical: browser samples are an agreement test ON the capability lane, + * and divergence fails closed. + * + * Direction matters, so disagreements are labeled distinctly: + * - capability-pass / browser-fail => suspected harness shortcut or + * product-path break -- the capability claim is suspect. + * - capability-fail / browser-pass => suspected harness bug or env gap -- + * the capability lane is under-reporting. + * + * Verifying failures is as important as verifying passes: a failure that + * passes in the browser exposes a harness bug; a pass that fails in the + * browser exposes a harness shortcut. The sampler therefore MUST include + * capability-lane failures, and the gate refuses (exit 2, cherry-pick guard) + * a browser set that dodged every capability failure. + * + * Subcommands: + * proofloop transfer-check sample --capability --seed + * [--per-family 5] [--model