EQL v3 benchmark suite with v2 regression comparison

[coderdan]

What

Parallel benchmark suite for the upcoming EQL v3 release (domain-specific types — eql_v3.text_search, eql_v3.integer_ord, eql_v3.integer_ord_ope, … — replacing the single eql_v2_encrypted composite). The v2 benches and committed results are untouched: they are the regression baseline. v3 results land in results/{query,ingest}/v3/, joined to v2 by identical criterion scenario ids.

Full comparison: report/V3_COMPARISON.md (regression tables, index-engagement audit, ingest deltas, index build times) with PNG+SVG charts in report/v3/ for docs/marketing.

Headlines (1M rows, M1 Max, PG 17)

• OPE is the fast path: encrypted range + ordered queries at 0.12 ms — ~1.2× plaintext Postgres and ~6× faster than ORE; the OPE index builds in 1 s vs 44 s for ORE at 1M.
• ORE LIMIT 100 range queries 4× faster than v2 (functional index engages where v2 seq-scanned).
• The bloom-match GIN engages via native array_ops (v3 ships no GIN opclass — this was the biggest open risk) and is worth 100–400× vs no index.
• Exact equality 0.12 ms (~1.3× plaintext); GROUP BY at parity or better vs v2.

Findings for the EQL team (pre-release)

1. String ingest −87%: text_search (the only eq+match text domain) requires the ORE term, capping string ingest at ORE-generation speed (~1.3k rec/s vs 9.6k on v2's unique+match). An hm+bf-only text domain would restore it.
2. ORE ordered scans +43% vs v2 — consistent with the plpgsql ore_block_256 opclass comparator; OPE is the escape hatch.
3. Diffuse +12–44% on JSON extractor scenarios (contains, field_order, field_eq) — per-row v3 extractor cost.
4. The +17,658% JSON field_eq/extractor outlier is a diagnosed bench artifact (every-row needle × sqlx generic-plan flip → full 1M-row GIN bitmap before LIMIT), documented in the report — not a v3 defect.

Methodology / caveats

• v3 payloads via eql-bindings::from_v2 over the pinned cipherstash-client 0.34.1-alpha.9 (the supported migration path); conversion is inside the measured ingest path (int ingest: ±2% vs v2, so conversion+CHECK cost ≈ 0).
• No v3 scalar query wire shape exists — query params are stored-shape payloads; server-side timings unaffected.
• OPE op terms are synthetic (fixed-width order-preserving hex; order-parity asserted at bench startup) until a client emits CLLW-OPE (CIP-3348) — plan shapes faithful, ciphertext size approximate.
• Tiers 10k/100k/1M (10M deferred). mise run setup-db-v3 builds the bundle from a local eql_v3 checkout until eql-3.x is released; the Cargo git dep should move to the release tag then.

Running

mise run setup-db-v3
mise run bench:v3:query:all 1000000
mise run bench:v3:ingest
mise run report:v3-compare

https://claude.ai/code/session_01StQnoycoFXMDdSpQ6zKDav

[freshtonic]

Cross-checked this suite against the current state of the EQL v3 stack (post eql-3.0.0-alpha.2 + protect-ffi 0.27 + the stack #547 re-baseline). This PR is where the entire v3 benchmark surface lives, so gaps here = gaps in v3 benchmarking overall. What's outstanding, split by when it can be addressed:

Actionable now (this PR or immediately after merge):

1. Released-bundle provenance — the methodology notes setup-db-v3 builds from a local eql_v3 checkout "until eql-3.x is released". eql-3.0.0-alpha.2 is now released (2026-07-04), so setup can move to the release artifact (the stack repo made the same swap in feat(stack,cli): EQL v3 Supabase adapter (encryptedSupabaseV3) + v3 install path stack#547).
2. eql_v3_internal split verification — alpha.2 moved the raw-jsonb SEM extractors (hmac_256/bloom_filter/ore_block_256/ope_cllw) into eql_v3_internal; the per-domain eq_term/ord_term/match_term/ord_ope_term wrappers stayed in eql_v3. The EXPLAIN audits here appear to use the wrappers (which survive unchanged), but worth one pass to confirm no index script or query shape calls a moved extractor — in the stack repo this broke 10 call sites silently until the live suite ran.
3. 10M tier — deferred here at 10k/100k/1M, but the v2 baseline (bench: refresh full query + ingest suite (10k–10M) against EQL 2.3 #14) ran to 10M, so the regression comparison is missing its top tier.

Post-merge follow-ups worth tracking (upstream-gated):

4. text_search restructure re-run — the −87% string-ingest finding (finding 1) is the empirical case for the in-flight upstream change making text_search equality+match-only (Linear CIP-3401). When it lands, the string-ingest tiers need re-running to confirm recovery — the tightest feedback loop between this suite and EQL design.
5. v3 scalar query wire shape — query params here are stored-shape payloads (correctly noted in the caveats; no scalar query envelope exists — same interim as the stack's Supabase adapter, protect-ffi throws EQL_V3_QUERY_UNSUPPORTED). When the term-only scalar query envelope ships (Linear CIP-3402), param-size/parse costs change and the query benches warrant a re-run.
6. Nice-to-have: a "Supabase profile" variant — no operator classes, functional-index-only, mirroring what managed Postgres actually permits. The OPE headline (natively bytea-sortable, 1s index build) already implies the answer there, but it's currently inferred rather than measured.

None of these block this PR — items 1–2 could ride along if convenient, 3–6 are follow-ups. The suite itself is the missing piece: everything above hangs off it landing.

[coderdan]

Thanks for the thorough cross-check, James — pushed as 80b5972. #1 and #2 done; #3–6 handled/tracked as below.

#1 — released bundle. setup-db-v3 now installs the pinned release (EQL_V3_VERSION, default eql-3.0.0-alpha.3) instead of building from a local checkout. Important heads-up: alpha.3 (2026-07-08) moved the per-domain types eql_v3.* → public.* and dropped the jsonb_query type, so this was a full migration of the schema/index/bench/binary SQL to public.<type> — not just a download swap. The eql_v3.* functions are unchanged.

#2 — eql_v3_internal split. Confirmed clean against a fresh alpha.3 install: nothing here calls a moved extractor (hmac_256/bloom_filter/ore_block_256/ope_cllw) directly — all through the public eql_v3.* wrappers. (Nice bonus: alpha.3 inlined the jsonb_entry CHECK, citing cipherstash/benches#23 — that's the fix for the +19% field_eq regression we reported.)

JSON bench. Two changes beyond the migration:

• Rewrote it to query through the named EQL JSON functions — eql_v3.jsonb_contains (GIN over eql_v3.jsonb_array(value)), eql_v3.jsonb_path_query_first(value, sel) feeding eql_v3.eq_term/eql_v3.ore_cllw — instead of casting to raw jsonb and using native @>/->.
• Added json/field_gt/functional, the encrypted equivalent of x -> 'y' > 10: WHERE jsonb_path_query_first(value, sel) > $1::jsonb::public.jsonb_entry — the > on public.jsonb_entry inlines to an ORE-CLLW compare, so it reuses the ore_cllw functional index (Index Scan at all tiers incl. 10M).

#3 — 10M tier. The JSON tiers are now re-ingested + re-run against alpha.3 at 10k/100k/1M/10M and both reports (V3_COMPARISON.md, BENCHMARK_REPORT.md) are regenerated. Notably contains/functional @10m improved 3.36ms → 1.05ms under the jsonb_array recipe. The non-JSON v3 scenarios are still the alpha.2-equivalent baseline — a full alpha.3 re-baseline of those is the tracked follow-up, flagged by a "Re-baseline status" banner in the report so the absolute numbers aren't misread.

#4–6 — ack'd as upstream-gated. FYI the local checkout already has the scalar query-operand terms in flight (CIP-3432), which is the #5 (EQL_V3_QUERY_UNSUPPORTED / scalar query wire shape) trigger; we'll re-run the query benches when that ships.