From 60ba7e984931bf93df296206fa5840ab4e791d5c Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 4 Jul 2026 13:16:19 +1000 Subject: [PATCH 1/2] fix(v3): inline the jsonb_entry CHECK; make the jsonb_query validator plpgsql MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Domain constraints cannot inline SQL functions, so eql_v3.jsonb_entry's CHECK calling eql_v3_internal.is_valid_ste_vec_entry_payload paid the per-call SQL-function executor (~18us) on every cast — the needle cast in every field_eq query, and the ENTIRE measured +19% v2->v3 regression on that scenario (cipherstash/benches#23). The CHECK now mirrors the validator body inline; the leading `VALUE IS NULL OR` preserves the validator's STRICT NULL-passes semantics (a bare COALESCE(..., false) would reject the NULL that `->` returns for a missing selector). Validated A->B->A on a live 100k bench database: 0.0287 -> 0.0137 -> 0.0285 ms/query in-DB, with validator calls dropping to zero. eql_v3.jsonb_query's CHECK CANNOT be inlined — validating the sv elements needs a subquery, which CHECK constraints forbid — so its validator is converted to plpgsql instead (cached plan vs the per-call SQL-function executor; the #353 finding), since the needle cast sits on the per-query hot path of every containment scenario. The eql_v3.json document CHECK is unchanged: it is part of the documented privilege contract (docs/reference/permissions.md, gated by v3_privilege_tests). permissions.md now notes the jsonb_entry cast no longer requires the internal grant; jsonb_query's still does. New family/jsonb_check tests: entry inline-CHECK <-> validator equivalence over a payload-shape corpus (accept/reject + NULL), a hardcoded accept/reject characterization for jsonb_query, and a language guard so a revert of the query validator to LANGUAGE sql fails CI. Closes #354 Claude-Session: https://claude.ai/code/session_01StQnoycoFXMDdSpQ6zKDav --- CHANGELOG.md | 2 + docs/reference/permissions.md | 7 +- src/v3/jsonb/types.sql | 48 +++++- .../encrypted_domain/family/jsonb_check.rs | 160 ++++++++++++++++++ .../sqlx/tests/encrypted_domain/family/mod.rs | 1 + 5 files changed, 211 insertions(+), 7 deletions(-) create mode 100644 tests/sqlx/tests/encrypted_domain/family/jsonb_check.rs diff --git a/CHANGELOG.md b/CHANGELOG.md index b439aa6e2..0dccba2f2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -72,6 +72,8 @@ Each entry that ships in a published release links to the PR that introduced it. - **The `eql_v2` schema and its entire surface are removed — EQL now ships only the self-contained `eql_v3` encrypted-domain surface.** Dropped with no `eql_v3` replacement: the `eql_v2_encrypted` composite column type and its operators (`=`, `<>`, `~~`/`~~*` `LIKE`/`ILIKE`, containment `@>`/`<@`, ORE comparisons), database-side configuration management (`eql_v2_configuration`, `add_search_config`, `add_column`, `migrate_config`, `diff_config`, `create_encrypted_columns`), the `encryptindex` migration machinery, boolean operators on the encrypted column type, operator-class-on-column indexing, and `GROUP BY` / `grouped_value` on the encrypted column type. Searchable-encryption capabilities (equality, ordered range, `MIN`/`MAX`, encrypted-JSONB document containment and path access) are all provided by the `eql_v3` encrypted-domain families and document surface. Why: `eql_v3` is now fully self-contained — it owns its own SEM index-term types (`eql_v3.hmac_256`, `eql_v3.ore_block_256`, `eql_v3.ore_cllw`, `eql_v3.bloom_filter`) and installs into a database with no `eql_v2` present — and the encryption client (CipherStash Proxy / ProtectJS) now owns the configuration model the database-side `eql_v2` functions previously provided. This is a major (3.0.0) public-API break; callers using the `eql_v2` schema migrate to the `eql_v3` encrypted-domain types. The user-facing reference documentation and tutorials have been migrated to teach only the `eql_v3` surface — the `eql_v2`-based examples, the database-side configuration guide, and the operator-class-on-column indexing recipe are removed. No per-capability migration guide is provided for the dropped capabilities (they have no `eql_v3` equivalent). ### Fixed +- **`eql_v3.jsonb_entry` CHECK inlined; `jsonb_query` validator converted to plpgsql — removes SQL-function-executor overhead from the per-query needle casts.** Domain constraints cannot inline SQL functions, so `jsonb_entry`'s function-call CHECK paid ~18 µs on every cast — the needle cast in every `field_eq` query was the ENTIRE +19% v2→v3 regression on that scenario (in-DB 0.011 → 0.029 ms/query with identical `eq_term` costs; cipherstash/benches#23). The `jsonb_entry` CHECK now mirrors the validator body inline, with a leading `VALUE IS NULL OR` preserving STRICT NULL-passes semantics (equivalence pinned over a payload corpus by `jsonb_check::jsonb_entry_check_matches_validator`). `jsonb_query`'s CHECK cannot be inlined — validating sv elements needs a subquery, which CHECK constraints forbid — so `is_valid_ste_vec_query_payload` is plpgsql instead (cached plan vs per-call SQL-function executor; the #353 finding), guarded by `jsonb_check::jsonb_query_validator_is_plpgsql`. The `eql_v3.json` document CHECK — part of the documented privilege contract — is unchanged; `docs/reference/permissions.md` now notes the `jsonb_entry` cast requires no internal grant. ([#354](https://github.com/cipherstash/encrypt-query-language/issues/354)) + - **`=` / `<>` on `eql_v2.ore_block_u64_8_256` now declare a `COMMUTATOR`, so equality joins over the ORE term no longer raise.** Both operators set `COMMUTATOR` to themselves (equality and inequality are symmetric, so each is its own commutator). Why: without it the planner raised `could not find commutator for operator` the first time an `ore_block_u64_8_256` equality was used as a join / mergejoin qualifier (e.g. via the inlined `eql_v3.integer_ord_ore` equality wrappers, since the operators carry `MERGES`). This only enables previously-erroring join plans — it cannot change which rows match or their ordering. ([#239](https://github.com/cipherstash/encrypt-query-language/pull/239)) - **The `eql_v3` ORE block comparator now orders ciphertexts of any block count, not just 8.** `eql_v3.compare_ore_block_256_term` derives the block count `N` from the term length (`octet_length = 49·N + 16`) instead of hardcoding 8, so encrypted types whose native ORE width exceeds 8 blocks — `numeric` (14) and `timestamp` (12) — order, range-query, `ORDER BY`, and `MIN`/`MAX` correctly instead of silently mis-ordering. Malformed terms (length not `49·N + 16` for `N ≥ 1`) now raise instead of returning a bogus comparison. The self-contained `eql_v3` SEM type was renamed `eql_v3.ore_block_u64_8_256 → eql_v3.ore_block_256` to reflect that it is width-agnostic (the `eql_v2` type is unchanged). No effect on existing 8-block types (a no-op for `N = 8`). ([#241](https://github.com/cipherstash/encrypt-query-language/issues/241), [#276](https://github.com/cipherstash/encrypt-query-language/pull/276)) diff --git a/docs/reference/permissions.md b/docs/reference/permissions.md index 10f12bfa0..5aa0f3307 100644 --- a/docs/reference/permissions.md +++ b/docs/reference/permissions.md @@ -73,9 +73,10 @@ Why the internal grant is needed even though you only call public objects: - The **ORE comparison** behind ordering and `MIN`/`MAX` calls pgcrypto `encrypt()`, which the installer places in the `extensions` schema — hence the `USAGE` there. -- **Casting raw jsonb to `eql_v3.json`** fires its domain `CHECK`, - `eql_v3_internal.is_valid_ste_vec_document_payload`. (Scalar domain CHECKs are - pure structural jsonb tests, so casting to a scalar domain needs no internal +- **Casting raw jsonb to `eql_v3.json` or `eql_v3.jsonb_query`** fires a + domain `CHECK` that calls an `eql_v3_internal.is_valid_*` validator. (Scalar + domain CHECKs — and, since issue #354, the `eql_v3.jsonb_entry` CHECK — are + pure structural jsonb tests, so casting to those domains needs no internal grant.) The hand-written jsonb containment **read** path (`eql_v3.ste_vec_contains` and diff --git a/src/v3/jsonb/types.sql b/src/v3/jsonb/types.sql index b4b919e9c..5023b5cc8 100644 --- a/src/v3/jsonb/types.sql +++ b/src/v3/jsonb/types.sql @@ -38,11 +38,18 @@ $$; --! @return boolean True when `val` is `{"sv":[...]}` and every element carries --! string `s`, no ciphertext, and exactly one string term (`hm` XOR --! `oc`). +--! @note plpgsql, not LANGUAGE sql (issues #353/#354): the only caller is the +--! eql_v3.jsonb_query domain CHECK, where a SQL function can never be +--! inlined (and the CHECK itself cannot absorb this body — it needs a +--! subquery over the sv elements, which CHECK constraints forbid). plpgsql +--! caches its plan across calls instead of paying the per-call SQL-function +--! executor on every needle cast. CREATE FUNCTION eql_v3_internal.is_valid_ste_vec_query_payload(val jsonb) RETURNS boolean - LANGUAGE sql IMMUTABLE STRICT PARALLEL SAFE + LANGUAGE plpgsql IMMUTABLE STRICT PARALLEL SAFE AS $$ - SELECT COALESCE( +BEGIN + RETURN COALESCE( jsonb_typeof(val) = 'object' AND jsonb_typeof(val -> 'sv') = 'array' AND NOT EXISTS ( @@ -62,7 +69,8 @@ AS $$ ), false) ), false - ) + ); +END; $$; --! @brief Validate a root SteVec document payload. @@ -119,9 +127,32 @@ CREATE DOMAIN eql_v3.json AS jsonb --! `i`/`v` merged in by `->`) are allowed. --! --! @see src/v3/jsonb/operators.sql +--! +--! @note The CHECK is an INLINE expression, not a call to +--! `eql_v3_internal.is_valid_ste_vec_entry_payload` (issue #354): domain +--! constraints cannot inline SQL functions, so the function-call form paid +--! the per-call SQL-function executor (~18 µs) on EVERY cast — the needle +--! cast in every field_eq query (+19% end-to-end vs v2, the entire measured +--! regression on that scenario; see cipherstash/benches#23). The expression +--! mirrors the validator body; the leading `VALUE IS NULL OR` preserves the +--! validator's STRICT NULL-passes semantics (a bare COALESCE(..., false) +--! would reject NULL, which `->` returns for a missing selector). Keep the +--! two in sync — `jsonb_entry_check_matches_validator` in +--! tests/sqlx pins the equivalence. CREATE DOMAIN eql_v3.jsonb_entry AS jsonb CHECK ( - eql_v3_internal.is_valid_ste_vec_entry_payload(VALUE) + VALUE IS NULL + OR COALESCE( + jsonb_typeof(VALUE) = 'object' + AND jsonb_typeof(VALUE -> 's') = 'string' + AND jsonb_typeof(VALUE -> 'c') = 'string' + AND ( + (jsonb_typeof(VALUE -> 'hm') = 'string' AND NOT (VALUE ? 'oc')) + OR + (jsonb_typeof(VALUE -> 'oc') = 'string' AND NOT (VALUE ? 'hm')) + ), + false + ) ); --! @brief Domain type for an STE-vec containment needle. @@ -135,6 +166,15 @@ CREATE DOMAIN eql_v3.jsonb_entry AS jsonb --! @note Construct from inline JSON via the DOMAIN cast: --! `'{"sv":[{"s":"","hm":""}]}'::eql_v3.jsonb_query`. --! @see eql_v3.to_ste_vec_query +--! +--! @note This CHECK CANNOT be inlined like eql_v3.jsonb_entry's (issue #354): +--! validating the sv elements requires a subquery +--! (`NOT EXISTS (SELECT ... FROM jsonb_array_elements(...))`), and CHECK +--! constraints forbid subqueries. The validator is plpgsql instead (cached +--! plan; substantially cheaper per call than a non-inlined LANGUAGE sql +--! function — the same finding as issue #353), since this cast sits on the +--! per-query hot path of every containment scenario +--! (`$1::jsonb::eql_v3.jsonb_query`). CREATE DOMAIN eql_v3.jsonb_query AS jsonb CHECK ( eql_v3_internal.is_valid_ste_vec_query_payload(VALUE) diff --git a/tests/sqlx/tests/encrypted_domain/family/jsonb_check.rs b/tests/sqlx/tests/encrypted_domain/family/jsonb_check.rs new file mode 100644 index 000000000..0c18c6275 --- /dev/null +++ b/tests/sqlx/tests/encrypted_domain/family/jsonb_check.rs @@ -0,0 +1,160 @@ +//! Equivalence guards for the inline SteVec domain CHECK expressions +//! (issue #354). +//! +//! `eql_v3.jsonb_entry` carries an INLINE CHECK expression rather than +//! calling `eql_v3_internal.is_valid_ste_vec_entry_payload`: domain +//! constraints cannot inline SQL functions, so the function-call form paid +//! the per-call SQL-function executor on every cast — the needle cast in +//! every field_eq query, the ENTIRE measured +19% v2→v3 regression on that +//! scenario (cipherstash/benches#23). `jsonb_entry_check_matches_validator` +//! pins the inline expression to the validator (still the source of truth +//! for direct callers) over a corpus of payload shapes; the one intentional +//! divergence is SQL NULL, which both forms accept (the validator via +//! STRICT, the inline expression via a leading `VALUE IS NULL OR`). +//! +//! `eql_v3.jsonb_query`'s CHECK CANNOT be inlined — validating sv elements +//! needs a subquery, which CHECK constraints forbid — so its validator is +//! plpgsql instead (cached plan vs the per-call SQL-function executor; the +//! issue #353 finding). `jsonb_query_check_behaviour` characterises the +//! accept/reject matrix, and `jsonb_query_validator_is_plpgsql` guards the +//! language so a revert to LANGUAGE sql fails here. + +use anyhow::Result; +use sqlx::PgPool; + +/// Try the domain cast for `payload`; Ok(true) = accepted, Ok(false) = CHECK +/// rejection. Any non-CHECK error propagates. +async fn cast_accepts(pool: &PgPool, domain: &str, payload: Option<&str>) -> Result { + let sql = format!("SELECT ($1::jsonb)::{domain} IS NOT DISTINCT FROM $1::jsonb"); + match sqlx::query_scalar::<_, bool>(&sql) + .bind(payload) + .fetch_one(pool) + .await + { + Ok(_) => Ok(true), + Err(e) if e.to_string().contains("check constraint") => Ok(false), + Err(e) => Err(e.into()), + } +} + +/// The validator's verdict for `payload`, with the STRICT NULL-passes rule +/// applied (SQL NULL is accepted by the domain even though the validator +/// returns NULL for it). +async fn validator_accepts(pool: &PgPool, validator: &str, payload: Option<&str>) -> Result { + if payload.is_none() { + return Ok(true); + } + let sql = format!("SELECT eql_v3_internal.{validator}($1::jsonb)"); + Ok(sqlx::query_scalar::<_, bool>(&sql) + .bind(payload) + .fetch_one(pool) + .await?) +} + +async fn assert_equivalent( + pool: &PgPool, + domain: &str, + validator: &str, + candidates: &[Option<&str>], +) -> Result<()> { + for payload in candidates { + let cast = cast_accepts(pool, domain, *payload).await?; + let valid = validator_accepts(pool, validator, *payload).await?; + anyhow::ensure!( + cast == valid, + "{domain} inline CHECK diverges from {validator} for payload {payload:?}: \ + cast accepted = {cast}, validator = {valid}" + ); + } + Ok(()) +} + +#[sqlx::test] +async fn jsonb_entry_check_matches_validator(pool: PgPool) -> Result<()> { + let candidates: &[Option<&str>] = &[ + // SQL NULL — accepted by both forms (STRICT / VALUE IS NULL OR). + None, + // Valid: hm entry, oc entry, extra fields allowed. + Some(r#"{"s":"sel","c":"ct","hm":"h"}"#), + Some(r#"{"s":"sel","c":"ct","oc":"o"}"#), + Some(r#"{"s":"sel","c":"ct","hm":"h","a":true,"i":{},"v":3}"#), + // Invalid: missing s / missing c / both terms / neither term. + Some(r#"{"c":"ct","hm":"h"}"#), + Some(r#"{"s":"sel","hm":"h"}"#), + Some(r#"{"s":"sel","c":"ct","hm":"h","oc":"o"}"#), + Some(r#"{"s":"sel","c":"ct"}"#), + // Invalid: non-string term / non-string s / wrong jsonb types. + Some(r#"{"s":"sel","c":"ct","hm":1}"#), + Some(r#"{"s":1,"c":"ct","hm":"h"}"#), + Some(r#""scalar""#), + Some("5"), + Some("null"), + Some("[]"), + Some("{}"), + ]; + assert_equivalent( + &pool, + "eql_v3.jsonb_entry", + "is_valid_ste_vec_entry_payload", + candidates, + ) + .await +} + +#[sqlx::test] +async fn jsonb_query_check_behaviour(pool: PgPool) -> Result<()> { + // (payload, expected accept) — hardcoded verdicts: the CHECK calls the + // validator, so a validator-equivalence assertion would be tautological. + let candidates: &[(Option<&str>, bool)] = &[ + (None, true), + // Valid: single- and multi-entry needles; empty sv is valid. + (Some(r#"{"sv":[{"s":"sel","hm":"h"}]}"#), true), + ( + Some(r#"{"sv":[{"s":"a","hm":"h"},{"s":"b","oc":"o"}]}"#), + true, + ), + (Some(r#"{"sv":[]}"#), true), + // Invalid: element carries a ciphertext / both terms / neither term / + // missing s. + (Some(r#"{"sv":[{"s":"sel","hm":"h","c":"ct"}]}"#), false), + (Some(r#"{"sv":[{"s":"sel","hm":"h","oc":"o"}]}"#), false), + (Some(r#"{"sv":[{"s":"sel"}]}"#), false), + (Some(r#"{"sv":[{"hm":"h"}]}"#), false), + // Invalid: sv not an array / missing sv / non-object roots. + (Some(r#"{"sv":{"s":"sel","hm":"h"}}"#), false), + (Some("{}"), false), + (Some(r#""scalar""#), false), + (Some("null"), false), + (Some("[]"), false), + ]; + for (payload, expected) in candidates { + let cast = cast_accepts(&pool, "eql_v3.jsonb_query", *payload).await?; + anyhow::ensure!( + cast == *expected, + "eql_v3.jsonb_query cast verdict changed for {payload:?}: \ + accepted = {cast}, expected = {expected}" + ); + } + Ok(()) +} + +/// The jsonb_query validator must stay plpgsql: its only caller is the domain +/// CHECK (a context that can never inline a SQL function), so LANGUAGE sql +/// pays the per-call SQL-function executor on every containment-needle cast +/// (issues #353/#354). A revert fails here. +#[sqlx::test] +async fn jsonb_query_validator_is_plpgsql(pool: PgPool) -> Result<()> { + let lang: String = sqlx::query_scalar( + "SELECT l.lanname FROM pg_proc p \ + JOIN pg_language l ON l.oid = p.prolang \ + WHERE p.proname = 'is_valid_ste_vec_query_payload' \ + AND p.pronamespace = 'eql_v3_internal'::regnamespace", + ) + .fetch_one(&pool) + .await?; + anyhow::ensure!( + lang == "plpgsql", + "is_valid_ste_vec_query_payload must be plpgsql (got {lang})" + ); + Ok(()) +} diff --git a/tests/sqlx/tests/encrypted_domain/family/mod.rs b/tests/sqlx/tests/encrypted_domain/family/mod.rs index e74049aed..9d2570e4b 100644 --- a/tests/sqlx/tests/encrypted_domain/family/mod.rs +++ b/tests/sqlx/tests/encrypted_domain/family/mod.rs @@ -2,6 +2,7 @@ //! the encrypted-domain family (not integer-specific). pub mod inlinability; +pub mod jsonb_check; pub mod jsonb_operator_surface; pub mod mutations; pub mod sem; From ad93e29776ed1b0f881c4acdefadd87542476f3a Mon Sep 17 00:00:00 2001 From: Dan Draper Date: Sat, 4 Jul 2026 13:21:30 +1000 Subject: [PATCH 2/2] docs: mark the CHECK implementation notes @internal MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The jsonb_entry / jsonb_query domain doc blocks are public API docs (Doxygen renders src/ with INTERNAL_DOCS at its default NO), and the issue-#354 performance rationale added there is an internal concern — wrap it in @internal/@endinternal so the published docs keep only the user-facing contract. The #357-style notes on the internal validator functions were already excluded via their blocks' @internal tags. Claude-Session: https://claude.ai/code/session_01StQnoycoFXMDdSpQ6zKDav --- src/v3/jsonb/types.sql | 42 +++++++++++++++++++++++------------------- 1 file changed, 23 insertions(+), 19 deletions(-) diff --git a/src/v3/jsonb/types.sql b/src/v3/jsonb/types.sql index 5023b5cc8..12774ffb6 100644 --- a/src/v3/jsonb/types.sql +++ b/src/v3/jsonb/types.sql @@ -128,17 +128,19 @@ CREATE DOMAIN eql_v3.json AS jsonb --! --! @see src/v3/jsonb/operators.sql --! ---! @note The CHECK is an INLINE expression, not a call to ---! `eql_v3_internal.is_valid_ste_vec_entry_payload` (issue #354): domain ---! constraints cannot inline SQL functions, so the function-call form paid ---! the per-call SQL-function executor (~18 µs) on EVERY cast — the needle ---! cast in every field_eq query (+19% end-to-end vs v2, the entire measured ---! regression on that scenario; see cipherstash/benches#23). The expression ---! mirrors the validator body; the leading `VALUE IS NULL OR` preserves the ---! validator's STRICT NULL-passes semantics (a bare COALESCE(..., false) ---! would reject NULL, which `->` returns for a missing selector). Keep the ---! two in sync — `jsonb_entry_check_matches_validator` in ---! tests/sqlx pins the equivalence. +--! @internal +--! Implementation note (issue #354): the CHECK is an INLINE expression, not a +--! call to `eql_v3_internal.is_valid_ste_vec_entry_payload` — domain +--! constraints cannot inline SQL functions, so the function-call form paid +--! the per-call SQL-function executor (~18 µs) on EVERY cast: the needle +--! cast in every field_eq query (+19% end-to-end vs v2, the entire measured +--! regression on that scenario; see cipherstash/benches#23). The expression +--! mirrors the validator body; the leading `VALUE IS NULL OR` preserves the +--! validator's STRICT NULL-passes semantics (a bare COALESCE(..., false) +--! would reject NULL, which `->` returns for a missing selector). Keep the +--! two in sync — `jsonb_entry_check_matches_validator` in tests/sqlx pins +--! the equivalence. +--! @endinternal CREATE DOMAIN eql_v3.jsonb_entry AS jsonb CHECK ( VALUE IS NULL @@ -167,14 +169,16 @@ CREATE DOMAIN eql_v3.jsonb_entry AS jsonb --! `'{"sv":[{"s":"","hm":""}]}'::eql_v3.jsonb_query`. --! @see eql_v3.to_ste_vec_query --! ---! @note This CHECK CANNOT be inlined like eql_v3.jsonb_entry's (issue #354): ---! validating the sv elements requires a subquery ---! (`NOT EXISTS (SELECT ... FROM jsonb_array_elements(...))`), and CHECK ---! constraints forbid subqueries. The validator is plpgsql instead (cached ---! plan; substantially cheaper per call than a non-inlined LANGUAGE sql ---! function — the same finding as issue #353), since this cast sits on the ---! per-query hot path of every containment scenario ---! (`$1::jsonb::eql_v3.jsonb_query`). +--! @internal +--! Implementation note (issue #354): this CHECK CANNOT be inlined like +--! eql_v3.jsonb_entry's — validating the sv elements requires a subquery +--! (`NOT EXISTS (SELECT ... FROM jsonb_array_elements(...))`), and CHECK +--! constraints forbid subqueries. The validator is plpgsql instead (cached +--! plan; substantially cheaper per call than a non-inlined LANGUAGE sql +--! function — the same finding as issue #353), since this cast sits on the +--! per-query hot path of every containment scenario +--! (`$1::jsonb::eql_v3.jsonb_query`). +--! @endinternal CREATE DOMAIN eql_v3.jsonb_query AS jsonb CHECK ( eql_v3_internal.is_valid_ste_vec_query_payload(VALUE)