add Verifiers v1 support to Prime CLI#758
Conversation
xeophon
changed the title
xeophon
requested review from
JannikSt,
JohannesHa,
burnpiro,
d42me,
kcoopermiller and
willccbb
as code owners
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 518d3d9d3a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a3d41abef6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
mikasenghaas
left a comment
mikasenghaas
left a comment
There was a problem hiding this comment.
we get a weird merge of typer cli (prime cli) and pydantic config (vf/ prl). imo broader question if we want to adopt pydantic config everywhere. would prob need some changes for look + feel, and subcommands to work etc. maybe a nicer fusion is possible tho
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2499cbbbc9
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- Pass `@ config.toml` to the v1 CLI as two tokens; the pinned prime-pydantic-config only reads a root config file in that form, so the single-token `@config.toml` failed with "Unrecognized arguments". - Local v1 evals no longer probe hub owners or install remote envs prime-side: references (bare names, org/name[@Version]) pass through verbatim and verifiers resolves them (`ensure_installed`). Prime only pre-installs local ./environments checkouts and keeps platform metadata for the results upload. Hosted evals and the v0 --save-results path keep the existing resolution (v0 does not self-install). - Config peek accepts both `taskset.id` and the legacy top-level `id` (verifiers' EvalConfig.is_legacy rule), so v1-format configs run v0 envs through the legacy bridge. - Declare prime-pydantic-config[toml]>=0.3.0.dev86 directly, matching the verifiers constraint.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7b9f799232
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- Resume: only reuse the X-Prime-Eval-Env-Display header as the upload slug when it has slug shape; display values like "name (local - ahead of org/name)" contain a slash but are not slugs. - Results conversion: use vf's WireTrace alias; split prompt/completion at the first assistant turn; only the final branch carries the trace reward (earlier branches have no per-step reward, matching v0). - Fail fast with a pointed message when an eval config is in the old v0 format (env_id / [[eval]]) instead of surfacing a v1 parse error.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f696288f9b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- A bare --dry-run (no value or followed by another flag) now skips the inference preflight, matching the v1 parser's bool-flag semantics. - The --save-results path now skips the automatic upload when the local checkout is ahead of its tracked upstream, mirroring the v1 path, so unpublished local code is never attributed to the stale upstream env.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 56c29ebd34
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…l rename Hosted evals: - prime eval --hosted <v1 config.toml> (top-level [taskset]) submits the platform's raw `environments` payload; published envs keep environment_ids - hosted CLI flags collapse to one explicit-fields parse (no double argparse) Sandbox / v0 compatibility (verifiers no longer ships the v0 evaluator): - the frozen v0 argv surface (--save-results marker) and transitional [[eval]] TOMLs (incl. the platform's group_size dialect) convert onto the v1 CLI via verifiers.v1.cli.eval.compat; v0 env ids ride the legacy auto-detect, hub-slug taskset ids are installed and pinned to local names - --disable-tui/--debug map to --rich false; --disable-tui stays in the help so the platform's TUI probe keeps matching Command surface: - prime eval <env> / @ config.toml is the canonical command; bare and --help invocations show the eval help with subcommands appended; `prime eval run` remains a hidden alias for hosted-eval sandboxes
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: bd55bbc26e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
The platform's script generator moves to v1-native invocations (pinned toolchain), so prime no longer parses or converts v0 argv: - utils/verifiers_v0_cli.py deleted; transitional TOMLs convert inside verifiers at its ingestion boundary, v0 env ids ride the legacy auto-detect, and hub-slug ids in v1 configs are installed and pinned into a config copy (_pin_config_env) - hosted override flags speak the shared v1 dialect, parsed by verifiers' own EvalConfig (-n/-r/-m/--sampling.* and their aliases); endpoint/env payloads live in the config file - --save-results/--disable-tui/--debug/--sampling-args and the other v0-only flags are gone; unknown flags fail the v1 parse with a clean error
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4f63efe836
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- a flag-like --resume "value" falls through to the normal invocation so verifiers reports the missing directory instead of prime treating the flag as an output dir - config-driven validate pins hub env refs into a local config copy via _pin_config_env, matching the eval path
|
Re the typer/pydantic-config fusion (review from Jun 23): partially converged since. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7a812a7c3d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ides - _prime_runtime_env injects PRIME_API_KEY / PRIME_INFERENCE_URL / PRIME_TEAM_ID into every verifiers subprocess now that verifiers reads env vars only; an explicit --api-key-var removes the injected key - hosted override parsing accepts legacy --header pairs alongside v1 --client.headers and folds both into the submission payload - import build_extra_headers from its v1 home (verifiers.v1.cli.eval.compat)
There was a problem hiding this comment.
💡 Codex Review
Preserve --env-dir-path for hosted runs
When a direct hosted run points at a local checkout outside the default, e.g. prime eval my-env --hosted --env-dir-path /tmp/envs, this variable is never populated from passthrough_args; the target built below therefore passes None into _resolve_hosted_environment, which falls back to ./environments. That makes the resolver miss the selected checkout/metadata and either fail the hosted run or attach it to a stale default checkout. Please parse/pop --env-dir-path before hosted override parsing and forward it to the target config.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
build_extra_headers now takes (headers, header) directly, merge_sampling_args dropped its prefer_existing_keys knob, and PROVIDER_CONFIGS moved from the deleted verifiers.scripts.eval to verifiers.v1.cli.eval.compat (the guarded provider-URL fallback had been silently dead since that module was removed).
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 613a9e0cf4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… eval view - verifiers_bridge: pop the frozen v0 eval flags out of argv and convert them into a written v1 config (build_v1_eval_config), so the platform sandbox's legacy command line keeps working; --debug/--disable-tui map to --no-rich - module map realigned to what verifiers ships (gepa -> verifiers.scripts.gepa; install/build/setup/tui dropped, prime env build removed) - eval view: back-translate native v1 trace rows to the legacy sample shape at read time (legacy_row) and discover native outputs/<name>/<uuid> run dirs; env/model come from metadata.json instead of dir-name parsing - metadata.json env falls back to the config's env for resume runs - cleanup: merged help loader, shared _require_api_key, dead lab-app helpers, single-use wrappers, stale removed-era comments
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 52460ff9c2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
1. endpoint aliases: don't force Prime URL or run preflight (mirrors v0 gepa guard) 2. bare env names: probe_remote=True so installed Hub envs get upstream_slug for upload 3. hosted --client.base-url/--client.api-key-var: forward into overrides (were silently dropped) 4. compact @path: strip leading @ before hosted config loading 5. discover_local_eval_runs: find runs in custom output_dirs (no env--model in dir name) 6. v1 hosted [taskset] without num_tasks: default num_examples to -1 (all tasks), not 5 7. hosted --env-dir-path: pop before pydantic_cli and thread into bare-env resolution
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 98c90b3. Configure here.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 98c90b32be
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
- Persist an upload-guard marker in the run's client headers so --resume applies the same auto-upload skip as the original invocation (config-driven and unpublished-local runs no longer upload after a resume). - Strip the v0 -s save-results short flag in the v0 argv conversion path so it is not misread as v1 shuffle. - Convert transitional TOML configs up front so prime's client/preflight/env defaulting reads v1 shape, and skip Prime preflight for custom credentials. - Detect explicit hosted --client.* flags from raw args instead of the parsed model (which reports defaults as set), and forward --env-args / --extra-env-kwargs to hosted overrides. - Print hosted log lines markup-safe.
Removes the test files and test functions this PR introduced (test_eval_v0_args.py, test_validate.py, test_verifiers_plugin_modules.py, plus PR-added tests in test_eval_billing/test_hosted_eval/test_eval_help/ test_lab_view). Pre-existing tests keep their v1 adaptations.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: f30da534bd
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| "rollouts_per_example": _first_config_value( | ||
| raw, "group_size", "rollouts_per_example", "num_rollouts" | ||
| ), |
There was a problem hiding this comment.
When a raw v1 hosted TOML omits the rollout count, this leaves rollouts_per_example absent, so the later hosted merge falls back to HOSTED_RUN_DEFAULT_ROLLOUTS_PER_EXAMPLE (3). v1 EvalConfig defaults num_rollouts to 1, so a config with only [taskset] runs three times as many rollouts when hosted as it does locally; set the v1 target default to 1 when none of group_size / rollouts_per_example / num_rollouts is present.
Useful? React with 👍 / 👎.
| "pool", | ||
| "timeout", | ||
| "retries", | ||
| "sampling", |
There was a problem hiding this comment.
For a v1 hosted TOML that includes [sampling] (for example temperature or token limits), classifying sampling as part of the raw environment means HostedEvalConfig.sampling_args stays unset and the platform eval_config never receives those run-level sampling settings. The same config is honored locally by v1 EvalConfig, but hosted runs will use default sampling (or send an unexpected environment field); map raw["sampling"] into the hosted sampling args instead of the environment payload.
Useful? React with 👍 / 👎.
2 participants
Overview
Adds Verifiers v1 support to the Prime CLI while preserving compatibility with legacy and hosted evaluation flows. Verifiers is the source of truth for environment and config resolution; the Prime CLI's own machinery is scoped to hosted evaluations and results upload.
What changed
prime eval runandprime env initthrough the Verifiers v1 entrypoints@ config.tomlto the v1 CLI as two tokens — the only root-config form prime-pydantic-config parses — so bothprime eval @ config.tomlandprime eval @config.tomlworkorg/name[@version]references pass through verbatim (hub ids install on demand viaensure_installed); prime no longer probes hub owners or installs remote envs on the v1 path, and only pre-installs local./environmentscheckouts plus keeps platform metadata for uploads[taskset] id(v1 taskset) or top-levelid(legacy v0 env via the v1 bridge)--save-resultslegacy path on the existing v0 resolution (the v0 evaluator does not self-install hub environments)verifiers>=0.1.15.dev371from PyPI instead of a Git source, and declaresprime-pydantic-config[toml]>=0.3.0.dev86directly (the same constraint verifiers uses)exclude-newerpolicyuv syncanduv runworkflow for Python 3.11–3.13User impact
Local evaluations use the Verifiers v1 CLI by default and resolve environments exactly as
uv run evaldoes.prime eval @ config.tomlworks for both v1 tasksets and legacy v0 environments. Legacy and hosted-generated commands remain compatible with the existing hosted runner, while local v1 results upload through the Prime Evals flow. Once verifiers ships legacy env auto-detection (PrimeIntellect-ai/verifiers#1857),prime eval <v0-env>needs no flags at all.Note
High Risk
Large changes to the default eval and hosted submission paths, breaking Python 3.10 support and removing
prime env build, with new API payload shapes for v1 hosted environments.Overview
Local eval now shells out to Verifiers v1 (
verifiers.v1.cli.*) instead of the old plugin andverifiers.cli.commands.*modules.prime eval <environment>is the primary entry (bareprime evalshows run help);prime eval runstays hidden for hosted sandboxes. The bridge handles hub/local install, v0 argv and transitional TOML conversion, resume/@config paths, default output dirs and upload guards, and post-run hub upload viaread_upload_data.Hosted eval maps CLI overrides through
EvalConfig/ pydantic-config (v1 flags like--num-tasks), loads v1-native TOML ([taskset]) into anenvironmentsAPI payload, and drops the old argparse-based verifiers parser.prime eval validateis added;prime env buildis removed with a clear error directing users to prebuilt OpenEnv images.Dependencies and platform:
requires-pythonis>=3.11,<3.14;verifiersandprime-pydantic-config[toml]are pinned to dev PyPI versions (Git source removed). CI/release matrices test 3.11–3.13 only. Lab UI, push/discovery, and eval viewer convert native v1results.jsonltraces for display and upload.Reviewed by Cursor Bugbot for commit f30da53. Bugbot is set up for automated code reviews on this repo. Configure here.