feat: additive EQL v3 benchmark scenarios + from_v2 conversion-overhead benchmark#21
Closed
freshtonic wants to merge 9 commits into
Closed
feat: additive EQL v3 benchmark scenarios + from_v2 conversion-overhead benchmark#21freshtonic wants to merge 9 commits into
freshtonic wants to merge 9 commits into
Conversation
Adds a v3 module to the bench library, built on the eql-bindings from_v2 converter (path dependency off the EQL repo's feat/eql-bindings-from-v2 branch; TODO switch to crates.io 0.2.0 once published): - v2_store_to_v3 / ciphertext_to_v3: convert cipherstash-client 0.38 storage payloads into eql_v3 domain payloads (terms the target does not require are dropped, bf reinterpreted as signed smallint[], k removed). - v3_scalar_to_v2_envelope / v3_scalar_to_ciphertext: rebuild the v2 ct envelope from a v3 scalar row for client-side decryption (decryption reads only c + i, never the index terms). - sample_plaintext_string_v3 + EncryptedQueryBuilderV3 / EncryptedQueryV3: v3 twins of the query-bench machinery. Scalar QUERY conversion is unsupported upstream, so probes are encrypted as STORAGE payloads and converted; results decode as plain jsonb (v3 domains are jsonb-backed, no composite wrapper). - IngestOptions.convert_to_v3(target): v3 ingest path converting each storage payload before INSERT. - ScenarioMetadata.version (2|3): the version axis for the metadata sidecars. Reporters treat an absent field as 2, existing v2 filenames and payloads are unchanged.
sql/schema-v3.sql defines the _v3 twins of the benchmarked tables,
typed as the per-capability eql_v3 jsonb domains:
string_encrypted_v3 value eql_v3.text_search (hm+ob+bf) — the only
single-column v3 domain serving both the EXACT
and MATCH families that share the v2 string
table; the extra ob term vs v2 is documented.
integer_encrypted_v3 value eql_v3.int4_ord_ore (v2 uses
ColumnType::Int / i32 → int4, not int8)
category_encrypted_v3 value eql_v3.text_eq
combo_encrypted_v3 name text_match / age int4_ord_ore /
category text_eq
json_ste_vec_*_v3 value eql_v3.json
Plaintext baseline tables are shared with the v2 schema (not
duplicated). A TODO scaffold documents the _ord_ope path blocked on
cipherstash-client emitting the op term (CIP-3280, unreleased).
sql/indexes/v3/ mirrors the v2 per-tier index intents through the v3
extractors: hash(eq_term), btree(ord_term), GIN(match_term), and the
canonical GIN((to_ste_vec_query(value)::jsonb) jsonb_path_ops) for
containment. All files verified to apply and roll back cleanly
against a scratch Postgres 16 with the built v3 installer.
v3 twins of the ingest bins encrypt through the unchanged
cipherstash-client (v2 wire) pipeline, convert each storage payload
with from_v2, and insert into the _v3 tables:
encrypt_string_v3 → text_search (adds an ORE index to the
column config — text_search requires ob —
so throughput is not directly comparable
to v2's encrypt_string; documented)
encrypt_int_v3 → int4_ord_ore (identical encrypt workload)
encrypt_category_v3 → text_eq (identical encrypt workload)
encrypt_combo_v3 → per-column text_match / int4_ord_ore /
text_eq
encrypt_ste_vec_small_v3 → json (SteVec Standard mode)
convert_overhead is the dedicated conversion-cost scenario: the same
encrypt workload with CONVERT_MODE=encrypt_only vs encrypt_convert,
no database writes, reported as its own ingest family — the delta
between the two hyperfine families is pure from_v2 cost.
Six criterion benches (EXACT_V3, MATCH_V3, ORE_V3, GROUP_BY_V3, COMBO_V3, JSON_V3) mirroring the v2 scenario families against the _v3 tables, with sidecar version: 3 and _v3-prefixed result/metadata filenames. All probe payloads are storage-payload conversions (no v3 scalar query wire shape exists); every query shape was validated against a live eql_v3 install on a scratch Postgres 16, including functional-index engagement for the ordered-range plan. Scenario notes: - exact: eql_cast (bare inlinable =) + eql_hash (eq_term extractor). - match: bloom containment only — the two v2 LIKE scenarios are dropped (v3 removes LIKE/ILIKE; documented in the bench header). Adds eql_bloom_bare to price the typed @> operator inlining. - ore: four range baselines + extractor-ordered range; selective scenarios stay disabled (EQL #230 applies unchanged); commented OPE scenario stub documents the CIP-3280 blocker. - group_by: encrypted scenarios only — plaintext baselines are version-independent and stay in the v2 family. - combo: LIKE filter replaced by the match_term @> recipe. - json: canonical to_ste_vec_query GIN containment recipe; field_eq bare now index-capable (v3 -> is inlinable, unlike v2's plpgsql); per-selector functional indexes built at bench startup.
…rhead tasks
Additive twins of the v2 tasks — no existing task is modified:
- setup-db-v3 installs the locally-built eql_v3 SQL (EQL_V3_SQL env
override, defaulting to the sibling EQL checkout's release/
artifact — there is no released v3 installer yet) plus
sql/schema-v3.sql.
- prepare:v3:_table mirrors prepare:_table with index scripts from
sql/indexes/v3/, plus per-table prepare:v3:* wrappers.
- bench:query:{exact,match,ore,group_by,combo,json}:v3 (via a shared
bench:query:_run_v3 helper) write results/query/<family>_v3_rows_N
files; bench:query:all:v3 sweeps the tiers.
- bench:ingest:{encrypt_int,encrypt_string,json-ste-vec-small}:v3
reuse the generic bench:ingest:_run.
- bench:ingest:convert-overhead runs the encrypt_only vs
encrypt_convert hyperfine pair (no DB writes) whose delta is pure
from_v2 conversion cost.
report_benchmarks.py: - discover ingest families by globbing results/ingest/*_combined.json (picks up encrypt_*_v3 and convert_overhead_* alongside the hardcoded v2 list, which is retired) - parse sidecar version (absent = 2) into ScenarioMetadata; derive the query-family version from the _V3 type suffix - overview table gains an EQL column; families sort so each _V3 twin sits next to its v2 counterpart for side-by-side comparison - static SQL/description maps and table/index-path resolution extended for the _V3 families (index scripts under sql/indexes/v3/) report_ingest.py: adds an eql version column (bench names ending _v3, plus convert_overhead_encrypt_convert, are v3; everything else including pre-existing files is v2). find_slow_queries.py needs no changes (bench-id keyed). Verified: both reporters render the existing v2-only results unchanged, and a mixed results dir (real v2 + synthetic v3 fixtures) renders both versions side by side.
- v3_scalar_to_v2_envelope now fails closed on non-v3 payloads (a v2 ct payload also carries c + i and would silently re-wrap) — guard plus a failing-first test. - group_by drain logic (both v2 and v3 twins, kept in sync) discriminates scenarios by column count instead of row count: a top-N result containing exactly one group would misdecode the group-key column as i64 under the row-count heuristic. - report_benchmarks.py overview prefers the sidecar-recorded version (source of truth) when a family's sidecars agree, falling back to the name-derived QueryResult version — both parsed fields are now read. - README project-structure tree covers the v3 files; Cargo.toml gets its trailing newline. Not changed, with reasons: the eql-bindings path dependency stays (the plan mandates consuming the unreleased crate via path dep; the TODO(crates.io 0.2.0) marker already documents the swap); encrypt_ste_vec_small_gin needs no ingest description (its task exports *_gin_hyperfine.json, never *_combined.json, so the glob cannot surface it); the six-bench scaffolding duplication mirrors the established v2 idiom deliberately — a shared-scaffold refactor spans both generations and is follow-up work.
The rebased eql-bindings (upstream #336 final form) added the k:"sv" form discriminator to v3 SteVec/jsonb documents (from_v2 emits it; the eql_v3.json domain CHECK requires it). Scalar v3 payloads still carry no k. Nothing broke here — documents are produced via from_v2, so they gained k automatically — but: - correct the v2_store_to_v3 doc and the text_eq test comment, which overgeneralised "k is removed" to all v3 payloads (scalars only); - add k:"sv" to the synthetic v3 SteVec fixture in v3_scalar_to_v2_envelope_rejects_ste_vec_documents so it mirrors the real wire shape; - add k:"sv" to the eql_v3.json document shape in the encrypt_ste_vec_small_v3 module docs.
coderdan
added a commit
that referenced
this pull request
Jul 4, 2026
…s approach) Reviewing James's PR #22 (stacked on #21/#20, a parallel v3 bench implementation) surfaced two adoptable facts: cipherstash-client 0.38.1 emits the scalar OPE-CLLW `op` term via Index::new_ope() (CIP-3280), and eql-bindings is published on crates.io. Adopt both into this branch rather than merging the parallel machinery (which duplicates ours under different table/task names and has no results): - Bump cipherstash-client + stack-profile to =0.38.1 (v2 wire format unchanged); switch eql-bindings from the git branch dep to crates.io =0.4.2. - encrypt_int_ope_v3 + benches/ope_v3.rs now use Index::new_ope() for REAL op terms end-to-end; the synthetic-op machinery is removed from src/v3.rs (to_v3_stored_with_synth_ope, ingest_v3_synth_ope, i32_order_key, SYNTH_OPE_HEX_WIDTH). The startup order-parity gate now verifies real crypto ordering. - OPE tiers re-ingested with real ciphertexts (33-byte op terms, ~2x the synthetic width) and all OPE query benches re-run: medians unchanged (0.118-0.124 ms flat at every tier) — the headline OPE numbers hold with real crypto. - All v3 ingest benches re-run under 0.38.1: ORE int and string at parity with the alpha.9 numbers (client bump doesn't shift existing comparisons), and a NEW headline: real OPE ingest runs 10,933 rec/s at 10k — 5.3x ORE int ingest (2,076 rec/s), since OPE term generation is far cheaper client-side than ORE block generation. - Synthetic-op caveats removed across README, reports, task descriptions. Claude-Session: https://claude.ai/code/session_01StQnoycoFXMDdSpQ6zKDav
coderdan
added a commit
that referenced
this pull request
Jul 4, 2026
…ipeline Replace the parallel IngestOptions::ingest_v3(f, "<target>") method with James's versioning-as-configuration shape from PR #21: the builder gains .convert_to_v3(TargetDomain) and the single ingest() branches on the stored eql_target. One batch loop instead of a duplicated one, and the target is a typed TargetDomain resolved at the builder call site (bad domain names fail at build time, not mid-ingest). TargetDomain is re-exported from dbbenches::v3 so binaries don't depend on eql-bindings directly; to_v3_stored_target() is the pre-parsed hot-loop conversion. The V3_CONVERT_ONLY decomposition flag moves into the Some(target) branch unchanged. The custom multi-column binaries (encrypt_combo_v3, encrypt_scalar_smoke_v3) keep calling to_v3_stored directly - they never used IngestOptions. Smoked both branches at 100 records each: v2 (None), integer_ord, integer_ord_ope (real op, 33-byte terms), text_search - all insert cleanly. This supersedes the ingest side of PRs #21/#22, which can now be closed (#20's 0.38.1 bump is merged and adopted here). Claude-Session: https://claude.ai/code/session_01StQnoycoFXMDdSpQ6zKDav
Contributor
|
Closing in favour of #23, which carries the same v3 bench scope with results attached (agreed with Dan). The two branches were parallel implementations of the same machinery (schema/tiered tables, index DDL, from_v2-based ingest conversion, ore_v3 scenarios, mise orchestration) under different naming. #23 became the canonical branch because it additionally has: three tiers of measured results (10k/100k/1M) for every family, match/json/combo/scalar-family/plaintext-baseline coverage, the v2-vs-v3 regression comparison report + docs/marketing charts, and a root-cause analysis of the flagged regressions ( Two things from this branch live on in #23, with credit in the commit messages:
Nothing else here is lost — every scenario shape this branch defined has a running, measured equivalent on #23. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an EQL v3 scenario suite alongside the existing v2 benchmarks (fully additive — v2 scenarios, filenames, and mise tasks untouched), enabling direct v2-vs-v3 performance comparison.
ScenarioMetadata.version(2|3, absent=2), v3 result filenames*_v3_rows_N.json; both Python reporters render v2/v3 side by sideeql_v3.text_search, integer→int4_ord_ore, category→text_eq, combo→text_match/int4_ord_ore/text_eq, json→eql_v3.json; 40 functional-index scripts undersql/indexes/v3/eql_bindings::from_v2→ insert into the v3-domain-typed twin; dedicated conversion-overhead scenario (convert_overheadbin: encrypt-only vs encrypt+convert) quantifies from_v2's pure costeql_v3.eq_term/match_term/ord_term/to_ste_vec_queryextractors; probes are stored-payload conversions (v3 scalar query conversion is unsupported upstream by design). LIKE scenarios dropped for v3 (no pattern matching in v3). OPE scenario stubbed — blocked on the client release with CIP-3280 (opemission)setup-db-v3,prepare:v3:*,bench:*:v3,bench:ingest:convert-overheadeql-bindings must be published to crates.io (0.2.0) and the path dependency in
Cargo.tomlswapped (TODO comment marks the spot). CI cannot resolve the local path dep until then. Tracked in Linear CIP-3346. Also stacked on #20 (retarget tomainafter it merges).Verification
cargo test/clippy/fmt clean · real v3 installer + schema + all 40 index scripts applied against live Postgres · every query shape executed with domain-valid payloads · EXPLAIN confirms
ord_termfunctional-index engagement · reporters render mixed v2+v3 result dirs. Outstanding (needs ZeroKMS creds + the benches postgresql@17 cluster): smallest-tier live runs (setup-db-v3,bench:ingest:encrypt_int:v3,bench:query:exact:v3 10000) and the carried v2 0.38 runtime check from #20. Risk noted:Index::new_ore()on a Text column (text_search needsob) unexercised live — fallback is a text_eq+text_match split.Linear: CIP-3345 (parent CIP-3299).