Benchmark validity #1: golden/prompt contract — 30/30 repaired + enforceable gate#118
Merged
Conversation
…hanical fixes + gate The board scores each phase by aligning the model's analysis to a golden. For that to be fair the golden must cover the SAME Pāli word sequence the prompt shows the model. It didn't: a mechanical audit confirmed only 6/30 ranked phases matched. The dominant cause — 16 phases — was a golden that is a correct contiguous SLICE of a shared segment, but with no `wordRange` to slice the PROMPT to match, so the model was prompted with the whole segment yet graded on a fraction (100% "coverage" while ignoring most prompted words). This is the shared-segment problem ADR SUTTA-003 opened and left unfinished beyond 7 phases. Part 1 (this commit, mechanical — no golden CONTENT changed): - Added `_phases[].wordRange` to the 16 SLICE phases, slicing each prompt to exactly its existing golden span. MATCH goes 6/30 → 22/30. The golden word sequences were already correct; only the prompt derivation was missing the slice. - New gate tests/scripts/sutta-studio/golden-prompt-contract.test.ts derives the prompt sequence exactly as the runner does and compares it (normalised) to the golden. Verified RED without the wordRange additions (all 16 SLICE phases listed as violations), green after. A second test keeps the pending list honest — it fails if a listed phase is repaired but not removed from the list. Part 2 (drafted, NEEDS OPERATOR DECISION — docs/roadmaps/GOLDEN-CONTRACT-REPAIR.md): - 8 phases where the golden SPLITS a joined token the prompt presents as one whitespace token (sandhi `etadavoca`→`etad`+`avoca`; the `'ti` quotative in the breathing section) or OMITS a word. `wordRange` can't split a token, so these need golden CONTENT edits. The doc gives each phase's exact mismatch and the recommended resolution: make the golden follow the Anatomist's one-word-per-whitespace-token rule (sandhi moves to morpheme SEGMENTS, not word boundaries), and add the omitted words. Tracked in the test's KNOWN_SANDHI_PENDING allowlist. I did NOT edit golden content unilaterally — the sandhi-tokenization policy is a scholarly call. So: the mechanical majority is repaired and the contract is now enforceable; the remaining 8 are documented with a proposed direction, pending your sign-off on the sandhi policy. tsc: 17 pre-existing errors, none in touched files. Sutta-studio suite green (43). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sd5oCcYiQBnGAiNT4SAo8L
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…word-per-token) Operator approved the one-word-per-whitespace-token policy. This reconciles the 8 remaining phases so all 30 ranked phases satisfy the golden/prompt contract (30/30). Anatomist golden: - Merged each split pair into ONE word matching the prompt's whitespace token; the sandhi split is preserved in that word's morpheme SEGMENTS (etad·avoca; assasāmī·ti), which still reconstruct the surface (verified). The CONTENT word's id survives the merge (so its senses stay valid), e.g. `etad`(fn)+`avoca`(content) → the merged `etadavoca` keeps avoca's id. Relations retargeted to the survivor; intra-word relations dropped. - Added the two omitted function words: `vā` in phase-an (with wordRange [0,8)); for phase-aq, merged `sato`+`va`→`satova` and added the second `satova` the prompt has. Downstream goldens (the merge changed anatomist word ids, so these had to follow or they'd dangle — verified none dangle after): - Lexicographer: dropped the removed function words' sense entries (the merged word is scored on the surviving content word's senses); added a sense entry for the inserted 2nd satova. - Weaver: retargeted linkedPaliId to the surviving word. - Typesetter: remapped layoutBlocks to the surviving ids and de-duplicated. Gate: tests/scripts/sutta-studio/golden-prompt-contract.test.ts now normalises on the Pāli LETTER sequence (quotes/punctuation dropped — the golden writes `'ti`, the source prints a curly ‘…’) and its KNOWN_SANDHI_PENDING list is EMPTY. Verified RED against the pre-Part-2 golden (all 8 listed as violations), green after. Validated: all 4 goldens internally consistent (segment reconstruction, zero dangling word/segment refs). tsc 17 (baseline). Sutta-studio suite green (43). Diffs localised to the 8 phases; the golden-repair doc updated to DONE. With this, review finding #1 is closed — the last mechanical benchmark-validity blocker. What remains before the paid retake is operator-directed: the 40/30/30 formula and the 12 model IDs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Sd5oCcYiQBnGAiNT4SAo8L
anantham
changed the title
anantham
marked this pull request as ready for review
July 17, 2026 13:48
…ownstream + refresh _phases counts External codex review of this PR found the golden edits left the anatomist word set AHEAD of the downstream goldens — the exact prompt/golden mismatch this PR exists to eliminate, in a ranked phase: - phase-aq: aq2b (2nd satova, CONTENT) was in anatomist+lexicographer but MISSING from the typesetter layoutBlocks → a correct typesetter output would be scored against a 4-word layout and penalized for the right word. - phase-an: an6b (vā, function) was in anatomist but MISSING from lexicographer + typesetter. - _phases wordCount/segmentCount/relationCount (+ downstream totalWords/wordSenseCount) stale on ~8 phases after the sandhi merges/additions. Cosmetic — the runner slices by canonicalSegmentIds/ wordRange, NOT these counts — but they misled any tooling/report that reads them. Fix: added aq2b to the typesetter layout (source order aq1,aq2,aq4,aq2b,aq5), an6b to lex (cloned from its identical twin an5) + typesetter, and recomputed the stale _phases counts from actual content (only the stale fields changed — 11/153 anatomist fields, confirming the formula). Verified: all 27 ranked phases now have every anatomist word present in lex AND typesetter; verify-golden stays 0 ERROR; sutta suite green (43); golden-prompt-contract unaffected. Root cause of the miss: neither the surface-parity contract test nor verify-golden (DPD-grounding only) checks anatomist<->downstream word-set consistency — that class was unguarded and slipped past self-review; caught only by external (codex) review. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The golden-contract repair (review finding #1), complete: a mechanical audit confirmed only 6/30 ranked phases had a golden covering the same Pāli the prompt showed the model. All 30 now do.
Part 1 — 16 mechanical
wordRangefixes (6/30 → 22/30)The shared-segment problem (ADR SUTTA-003, unfinished beyond 7 phases): the golden is a correct slice of a shared segment, but the phase lacked a
wordRange, so the model saw the whole segment and was graded on a fraction. Added the computedwordRangeto each — no golden content changed.Part 2 — 8 sandhi/omission phases (22/30 → 30/30), per your one-word-per-token call
The golden split a joined token the prompt presents as one whitespace token (
etadavoca→etad+avoca; the'tiquotative), or omitted a word.etadavocakeepsavoca's "said/declared".vāin phase-an; a secondsatovain phase-aq).Verification
golden-prompt-contract.test.ts: 30/30,KNOWN_SANDHI_PENDINGempty. Red-before-green against both the pre-Part-1 and pre-Part-2 goldens.docs/roadmaps/GOLDEN-CONTRACT-REPAIR.mddocuments the audit, both parts, and the policy.What's left before the paid twelve-model retake
This closes the last mechanical validity blocker. The remaining two are yours:
🤖 Generated with Claude Code
https://claude.ai/code/session_01Sd5oCcYiQBnGAiNT4SAo8L