Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,8 @@ script. With neither, it reports `no_gate` with exit code 2. An unconfigured gat
| `proofloop runner run --plan <file> --budget-usd 100` | Run an append-only, budgeted task plan under `.proofloop/runner/runs/<runId>/`. |
| `proofloop runner resume --run-id latest` | Resume a runner after a crash; stale `running` tasks are requeued. |
| `proofloop runner status --run-id latest [--json]` | Inspect durable runner state and ledger paths. |
| `proofloop transfer-check sample --capability <file> --seed <sha>` | Deterministic stratified browser-lane sample from capability results (failures included by design). |
| `proofloop transfer-check gate --capability <file> --browser <file>` | Two-layer agreement gate: exit 0 agreed / 1 diverged / 2 unusable (fail-closed). |
| `proofloop mcp` | Start the optional read-only MCP server. |
| `proofloop gate [--check]` | Run configured checks or `npm test`; exit 0 pass, 1 fail, 2 unusable. |
| `proofloop hooks install\|uninstall\|status` | Install/remove/status Claude Code Stop, PreToolUse, and PostToolUse hooks. |
Expand Down Expand Up @@ -165,6 +167,51 @@ npx proofloop tooluse verify --contract tooluse-contract.json
The verifier is fail-closed: a deny-list cannot be certified from an empty or missing log, and
server-pinned names mean `mcp__evil__X` cannot impersonate `mcp__composio__X`.

## Two-layer certification: the transfer gate

Running every benchmark task through a real browser is slow and expensive, so the
two-layer model (`two-layer-certification-v1`) splits certification:

- the **capability lane** runs ALL tasks through the live agent harness (headless, cheap);
- the **browser lane** replays a small stratified sample through the real production UI.

The split saves ~95% of cost and wall-clock, but it opens a gaming surface: the harness
lane could take shortcuts the product path does not have (memory-mode, bypassed tool
contracts) and nothing would notice. `proofloop transfer-check` closes that hole by
treating the browser sample as an **agreement test on the capability lane** -- the
anti-cheat doctrine's in-app transfer rule made mechanical. Divergence fails closed.

```bash
# 1. Sample which capability results the browser lane must replay.
# Pass the commit SHA as the seed: same seed + same input = byte-identical sample,
# so the agent cannot re-roll the dice until it gets a sample it likes.
npx proofloop transfer-check sample --capability capability-results.json --seed "$(git rev-parse HEAD)" > transfer-sample.json

# 2. Run the sampled tasks through the real UI, record browser receipts, then gate.
npx proofloop transfer-check gate --capability capability-results.json --browser browser-receipts.json
```

Both lanes accept either a receipts JSON array (`[{ taskId, model, family?, pass }]`)
or a runner events ledger (`.proofloop/runner/runs/<runId>/ledger.jsonl`; label ledger
rows with `--model`).

**Why failures must be sampled.** Verifying capability-lane failures is as important as
verifying passes: a failure that passes in the browser exposes a harness bug (the lane is
under-reporting); a pass that fails in the browser exposes a harness shortcut (the claim
is inflated). The sampler therefore reserves at least `ceil(N/3)` slots per family for
failures whenever failures exist, and the gate refuses (exit 2, cherry-pick guard) a
browser set that paired with zero capability failures -- overridable only by an explicit
`--allow-no-failure-overlap`, which prints a loud warning into the output.

**Claim language rule.** On agreement the gate prints exactly what was proven:
capability verified through the harness, plus transfer verified on N seeded pairs --
never "all tasks browser-verified". Exit codes: `0` agreement >= `--min-agreement`
(default 0.9) with overlap >= `--min-overlap` (default 5); `1` divergence, with each
disagreeing pair labeled by direction (capability-pass/browser-fail = suspected harness
shortcut; capability-fail/browser-pass = suspected harness under-reporting); `2`
unusable evidence (unreadable inputs, thin overlap, or cherry-picked sample). Zero
evidence never exits 0.

## Scope

This package is the portable core: gate, refuse-fake-done hooks, expected-tool-use contracts,
Expand Down
5 changes: 5 additions & 0 deletions dist/cli.js
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ exports.runCli = runCli;
* proofloop ci install github write the GitHub Actions gate workflow
* proofloop prompt print the one-prompt kickoff
* proofloop this-repo [--goal ...] [--write-runner-plan] [--run]
* proofloop transfer-check sample|gate two-layer certification agreement gate
* proofloop manifest|docs|template|workflow|ui|resume|report|charts|mcp
*
* Exit codes are per-command (documented at each case). Zero runtime deps.
Expand All @@ -30,6 +31,7 @@ const proofloopToolUse_1 = require("./proofloopToolUse");
const mcp_1 = require("./mcp");
const project_1 = require("./project");
const runner_1 = require("./runner");
const transferCheck_1 = require("./transferCheck");
exports.MCP_SERVER_RUNNING = -999;
/** Parse `--flag`, `--flag value`, `--flag=value`, and positionals. */
function parseArgs(argv) {
Expand Down Expand Up @@ -92,6 +94,7 @@ function usage() {
" report latest [--json] latest gate report",
" charts latest write local JSON/SVG proof charts",
" runner run|resume|status durable append-only task runner with budget and resume",
" transfer-check sample|gate two-layer certification: seeded browser sample + agreement gate",
" mcp start the optional read-only MCP server",
" prompt print the one-prompt kickoff",
" this-repo [--goal <text>] [--write-runner-plan] [--run]",
Expand Down Expand Up @@ -157,6 +160,8 @@ function runCli(argv) {
return runChartsCommand(positional[1], root);
case "runner":
return runRunnerCommand(positional[1], options, root);
case "transfer-check":
return (0, transferCheck_1.runTransferCheckCommand)(positional[1], options, root);
case "mcp":
(0, mcp_1.startMcpServer)({ root });
return exports.MCP_SERVER_RUNNING;
Expand Down
1 change: 1 addition & 0 deletions dist/index.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,5 @@ export * from "./project";
export * from "./mcp";
export * from "./runner";
export * from "./layeredPlan";
export * from "./transferCheck";
export { runCli } from "./cli";
1 change: 1 addition & 0 deletions dist/index.js
Original file line number Diff line number Diff line change
Expand Up @@ -35,5 +35,6 @@ __exportStar(require("./project"), exports);
__exportStar(require("./mcp"), exports);
__exportStar(require("./runner"), exports);
__exportStar(require("./layeredPlan"), exports);
__exportStar(require("./transferCheck"), exports);
var cli_1 = require("./cli");
Object.defineProperty(exports, "runCli", { enumerable: true, get: function () { return cli_1.runCli; } });
2 changes: 1 addition & 1 deletion dist/prompt.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,5 @@
* mentioned here is a real command.
*/
/** The set of top-level commands the package CLI implements. */
export declare const PACKAGE_COMMANDS: readonly ["init", "doctor", "gate", "hooks", "tooluse", "ci", "manifest", "docs", "template", "workflow", "ui", "resume", "report", "charts", "runner", "mcp", "prompt", "this-repo"];
export declare const PACKAGE_COMMANDS: readonly ["init", "doctor", "gate", "hooks", "tooluse", "ci", "manifest", "docs", "template", "workflow", "ui", "resume", "report", "charts", "runner", "transfer-check", "mcp", "prompt", "this-repo"];
export declare function proofloopKickoffPrompt(): string;
1 change: 1 addition & 0 deletions dist/prompt.js
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ exports.PACKAGE_COMMANDS = [
"report",
"charts",
"runner",
"transfer-check",
"mcp",
"prompt",
"this-repo",
Expand Down
87 changes: 87 additions & 0 deletions dist/transferCheck.d.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
export declare const TRANSFER_SAMPLE_SCHEMA = "proofloop-transfer-sample-v1";
export type TransferLaneResult = {
taskId: string;
model: string;
family: string;
pass: boolean;
};
export type TransferLaneRead = {
results: TransferLaneResult[];
warnings: string[];
source: "receipts" | "runner-ledger";
};
export type TransferSamplePair = {
taskId: string;
model: string;
family: string;
capabilityPass: boolean;
};
export type TransferSamplePlan = {
schema: typeof TRANSFER_SAMPLE_SCHEMA;
seed: string;
pairs: TransferSamplePair[];
};
export type TransferDisagreementDirection = "capability-pass-browser-fail" | "capability-fail-browser-pass";
export declare const TRANSFER_DIRECTION_LABELS: Record<TransferDisagreementDirection, string>;
export type TransferDisagreement = {
taskId: string;
model: string;
family: string;
capabilityPass: boolean;
browserPass: boolean;
direction: TransferDisagreementDirection;
label: string;
};
export type TransferGateEvaluation = {
status: "agreed" | "diverged" | "unusable";
/** Present when status === "unusable". */
reason?: string;
overlap: number;
matches: number;
agreementRatio: number;
minAgreement: number;
minOverlap: number;
capabilityFailuresTotal: number;
pairedCapabilityFailures: number;
disagreements: TransferDisagreement[];
warnings: string[];
};
export declare function readTransferLaneResults(path: string, options?: {
ledgerModel?: string;
}): TransferLaneRead;
/**
* Deterministic stratified sample of the capability lane for browser
* certification. Same seed + same input = byte-identical plan (the per-family
* PRNG stream is `fnv1a(seed|family)`, so adding a family never changes
* another family's picks). If a family has capability failures, at least
* ceil(perFamily/3) of its sample slots are failures (capped by how many
* failures exist): failures must transfer too, or the failure path of the
* harness is never checked against the product.
*/
export declare function buildTransferSample(capability: readonly TransferLaneResult[], options: {
seed: string;
perFamily?: number;
}): TransferSamplePlan;
export declare function evaluateTransferGate(capability: readonly TransferLaneResult[], browser: readonly TransferLaneResult[], options?: {
minAgreement?: number;
minOverlap?: number;
allowNoFailureOverlap?: boolean;
}): TransferGateEvaluation;
/**
* The doctrine claim line printed on exit 0 -- deliberately scoped language.
* The browser lane certified a stratified sample, so the only honest claim is
* "capability lane verified + transfer verified on N seeded pairs", never
* "all tasks browser-verified".
*/
export declare function transferClaimLine(evaluation: TransferGateEvaluation): string;
type TransferCheckIo = {
log?: (line: string) => void;
logError?: (line: string) => void;
};
/**
* `proofloop transfer-check sample|gate`. Exit codes:
* sample: 0 wrote/printed the plan, 2 unusable input or bad flags.
* gate: 0 agreed, 1 diverged, 2 unusable (fail-closed).
*/
export declare function runTransferCheckCommand(sub: string | undefined, options: Record<string, string | boolean>, root: string, io?: TransferCheckIo): number;
export {};
Loading
Loading