feat(transfer-check): two-layer certification agreement gate#9
Open
HomenShum wants to merge 1 commit into
Open
feat(transfer-check): two-layer certification agreement gate#9HomenShum wants to merge 1 commit into
HomenShum wants to merge 1 commit into
Conversation
… the headless/browser split un-gameable The two-layer certification model (headless capability lane runs ALL benchmark tasks cheaply; browser lane runs a stratified sample on the real prod UI) saves ~95% of cost/wall-clock but opens a new gaming surface: the headless harness could take shortcuts the product path doesn't have (memory-mode, bypassed tool contracts) and nothing would notice. `proofloop transfer-check` closes that -- it is the anti-cheat doctrine's IN-APP TRANSFER rule made mechanical. - `transfer-check sample --capability <file> --per-family N --seed <REQUIRED>`: deterministic hash-seeded stratified sampler (pass a commit SHA so neither human nor agent can cherry-pick easy tasks -- same seed + input = byte-identical sample, tested). MUST include capability-lane failures (>= ceil(N/3) per family that has them): a "failure" that passes in the browser exposes a harness bug; a "pass" that fails in the browser exposes a harness shortcut -- both directions are evidence. - `transfer-check gate --capability <cap> --browser <browser> [--min-agreement 0.9] [--min-overlap 5]`: joins on taskId+model, computes agreement. Exit 0 = agreed (prints the doctrine claim verbatim: "Capability verified through the live agent harness; production browser path verified by stratified UI certification... This is NOT an all-tasks-browser-verified claim"); exit 1 = diverged, with a per-pair table labeling DIRECTION distinctly (capability-pass/browser-fail = "suspected harness shortcut or product-path break -- capability claim suspect"; capability-fail/browser-pass = "suspected harness bug or env gap -- capability lane under-reporting"); exit 2 = fail-closed (overlap below min, unreadable input, or CHERRY-PICK GUARD: capability has failures but the browser paired set has zero of them -> refuse unless --allow-no-failure-overlap, which warns loudly and certifies "including 0 capability-failures"). Never exit 0 on zero evidence. - Readers accept a generic receipts JSON array OR the merged runner.ts events ledger (.proofloop/runner/runs/<id>/ledger.jsonl -- real path, verified by round-tripping a real `proofloop runner run`); ledger rows carry no model field so --model labels them (fail-closed: mismatched models -> zero overlap -> exit 2). Deliberately COMPLEMENTARY to the merged layeredPlan.ts (the two-layer PLAN generator): this is the AGREEMENT GATE, not a second plan surface. 16 new scenario tests (agreement pass/exactly-threshold, both divergence directions, thin-overlap exit 2, cherry-pick guard + override, seeded determinism, failure-inclusion, real ledger round-trip, duplicate rejection); 81/81 total green; tsc strict. Independently re-verified live: deterministic sample with 5 failures included, exit 0/1/2 all correct, both divergence labels, cherry-pick guard firing. Registered in PACKAGE_COMMANDS (grep-asserted honesty) + README "Two-layer certification: the transfer gate" section documenting the claim-language rule (never "all tasks browser verified" unless literal). Weakest spots documented: --model labels both lanes for ledger inputs (use receipts format for cross-model); byte-identity depends on stable JSON key order (determinism test guards it); the gate certifies agreement of inputs given -- fabricated browser receipts are out of scope (mitigations are the SHA seed + runner-ledger source, not receipt provenance). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the headless/browser split un-gameable. The two-layer model (headless capability lane runs ALL benchmark tasks cheaply; browser lane samples the real prod UI) saves ~95% of cost/wall-clock but opens a gaming surface: the headless harness could take shortcuts the product path lacks (memory-mode, bypassed tool contracts) and nothing would notice.
proofloop transfer-checkis the anti-cheat doctrine's IN-APP TRANSFER rule made mechanical.sample --seed <REQUIRED>: deterministic hash-seeded stratified sampler (pass a commit SHA → neither human nor agent can cherry-pick easy tasks; same seed+input = byte-identical, tested). Must include capability-lane failures — a "failure" that passes in the browser exposes a harness bug; a "pass" that fails exposes a harness shortcut.gate: joins on taskId+model. Exit 0 = agreed (prints the doctrine claim verbatim, "…NOT an all-tasks-browser-verified claim"); exit 1 = diverged with both directions labeled distinctly (harness-shortcut vs harness-bug); exit 2 = fail-closed (thin overlap, unreadable, or cherry-pick guard: capability has failures but the browser set dodges all of them). Never exit 0 on zero evidence.runner.tsevents ledger (real path round-trip-verified).layeredPlan.ts(the two-layer plan generator) — this is the agreement gate, not a second plan surface.Test plan
--modellabels both lanes for ledger inputs (use receipts format for cross-model); byte-identity depends on stable JSON key order (determinism test guards it); the gate certifies agreement of the inputs given — fabricated browser receipts are out of scope (mitigations are the SHA seed + runner-ledger source)🤖 Generated with Claude Code