feat(v1): resume dashboard counts the whole run, not just resumed rollouts

[hallerite]

What

On --resume, the live eval dashboard counted only the re-run rollouts and ignored the ones already kept on disk. This makes it reflect the whole run.

The runner hands the dashboard only the owed (missing + errored) rollouts on resume, so:

• Before — the counter's numerator and denominator covered only that resumed slice. The bar started empty and filled to full, but "full" only ever meant the re-run rollouts, and the reward / err / breakdown reflected just those.
• After — the kept rollouts seed the counter and scores. The bar starts already partly filled at the kept count and fills to the true total, and the headline reward / err share / reward-metric breakdown / usage / time all fold in the kept on-disk rows.

Concretely, a 100-rollout run resumed with 40 kept / 60 re-run:

	start	end	denominator	reward covers
before	0/60	60/60	remaining (60)	60 resumed only
after	40/100	100/100	all (100)	all 100

So the counter goes from 0/remaining to done/all — it starts at 40/100 instead of 0/60.

How

Rather than keep score aggregates around, reload the kept traces once and treat them as finished rollouts (per review discussion):

• After resume.rewrite_results leaves only the kept (good) rows on disk, resume.load_kept reloads them as finished Traces — via read_traces + WireTask, the same offline-load path replay uses (no runtime, no taskset Task type needed).
• The runner passes them to the dashboard as finished; Progress folds them into its done list (done = finished + this session's done) and adds them back into the total. Reward, err, and the per-component / usage / time breakdown then fall out of the existing code with no special-casing — format_mean and _breakdown are untouched.
• Only the --rich dashboard loads the traces (skipped otherwise); non-resume runs pass an empty list and are unchanged.
• Behavioral note: the usage/time breakdown now covers the whole run, not just this session. Token/cost totals are summed over the kept rollouts too (so they include their original generation spend); each timing phase is averaged over the whole run (the kept rollouts pull the average). Both were session-scoped before. Happy to gate back to session-only if reviewers prefer.

Net +55/−11 across 3 files (vs. ~+236 for the earlier score-aggregate version).

Testing

Drove the resume path offline (no model needed): built real Traces — with rewards, metrics, per-node usage, and timing — for a prior run, wrote them in the on-disk format, then ran the actual plan → rewrite_results → load_kept and rendered Progress. Observed session-only → whole-run:

row	before (session-only)	after (kept folded in)
counter	1/2	4/5
reward	0.00	0.75 (mean of [1,1,1,0])
metric len	3.00	1.50 (mean of [0,1,2,3])
usage	100/20 · $0.0010	400/80 · $0.0040 (summed)
time	generation 6s	generation 3s (avg of [2,2,2,6])

This also confirms load_kept round-trips the full trace (nodes/usage/timing), not just scores.

• ruff check + ruff format clean; ty check reports no new diagnostics vs. main (load_kept mirrors replay's existing Trace[WireTask, state_cls(...)] pattern).
• Existing tests/v1 unit suite passes (53, non-e2e).
• Not run: a full end-to-end --resume against a live model (no endpoint available here); the runner wiring that calls load_kept and passes finished to the dashboard is exercised only by the piecewise test above, not a live run.

🤖 Generated with Claude Code

Note

Include kept rollouts from resumed runs in eval dashboard counts and means

• Introduces a Baseline dataclass in resume.py that aggregates non-errored kept rollouts (count, reward sum, per-key component sums); resume.plan now returns this alongside keep offsets and owed counts.
• Updates format_mean to accept base_sum/base_n kwargs, folding prior rollouts into both the clean mean and the parenthesized global mean.
• Updates the dashboard's Progress and _breakdown components in eval.py to accept and apply the Baseline, so counts, error rates, rewards, and per-key metrics reflect the entire run rather than only the current session.
• Behavioral Change: resumed run dashboards now show cumulative statistics; callers that do not pass base_sum/base_n are unaffected.

^{Macroscope summarized 3c764e2.}

---

Note

Low Risk
Display and aggregation-only for the resume dashboard path; format_mean defaults preserve existing callers, and persisted results logic is unchanged aside from reading reward/metric dicts when planning keeps.

Overview
On --resume, the rich eval dashboard now treats kept on-disk rollouts as part of the run for progress and scoring stats, instead of showing only the rollouts re-executed in the current session.

resume.plan returns a new Baseline (count, summed headline reward, per-@reward/@metric sums) built while scanning results.jsonl for rows to keep. The runner passes that baseline into the dashboard. format_mean accepts optional base_sum/base_n so error-corrected and parenthesized global means include those kept rows. Progress shows kept+done / kept+total, and reward/err/breakdown reward-metric rows use the combined denominators. Usage and time in the breakdown stay session-only (no token/time re-derivation from disk). Non-resume runs are unchanged (empty baseline).

Adds tests/v1/test_resume_baseline.py for format_mean baseline folding and plan aggregation (including group-scored incomplete groups).

^{Reviewed by Cursor Bugbot for commit 3c764e2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[macroscopeapp]

Approvability

Verdict: Needs human review

1 blocking correctness issue found. This PR modifies dashboard runtime behavior for resumed runs, and there is an unresolved review comment identifying a correctness issue where usage/time metrics would incorrectly display historical data rather than session-scoped data as intended.

^{You can customize Macroscope's approvability policy. Learn more.}