Skip to content

perf(gfql/cypher): single-alias predicate pushdown in the row pipeline (L4)#1702

Open
lmeyerov wants to merge 5 commits into
feat/gfql-index-degreesfrom
perf/gfql-row-pushdown
Open

perf(gfql/cypher): single-alias predicate pushdown in the row pipeline (L4)#1702
lmeyerov wants to merge 5 commits into
feat/gfql-index-degreesfrom
perf/gfql-row-pushdown

Conversation

@lmeyerov

@lmeyerov lmeyerov commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

What

A disjunctive / searchAny WHERE collapses the whole MATCH onto the row pipeline, which materializes the full (a)-[e]->(b) binding table (O(E) joins) before filtering. This adds a lowering pass that attaches conjuncts referencing exactly one alias (and standalone searchAny ops) to rows(alias_prefilters={alias:[spec]}) as an advisory hint; the binding builder pre-filters each alias frame before the join.

Redundant-filter safety (why it's correct on every engine)

The hint is additive — the post-join where_rows / search_any ops are left in place:

  • Engines whose row builder honors the hint (pandas / cuDF via RowPipelineMixin) pre-filter each alias frame before the binding join and win.
  • Engines that ignore it (polars' native rows_binding_ops_polars, the shortestPath scalar builder) run the unchanged post-filter and stay correct.

A pushed conjunct is a literal sub-conjunct of the retained AND, so the pre-filter can only remove rows the post-filter also removes — never a result change, on any engine (present or future). Application reuses the SAME evaluator (_gfql_eval_string_expr + _gfql_bool_mask) and the SAME search_any_mask kernel, so semantics (incl. 3-valued NULL) are byte-identical.

Guards (skip pushing a conjunct/op)

OPTIONAL / reentry / row-guard / procedure / seed-row / multi-graph programs; shortestPath scalar bindings; multihop edge aliases; and any conjunct naming an internal __gfql_/__cypher_ column (EXISTS / searchAny markers + reentry carries read falsely single-alias).

Results (pandas, seed=7; base = pass-disabled)

scenario @100k @1m
search_edges 2.9× 4.62× (7.1s→1.5s)
panel_filters 1.32× 1.47× (24.9s→17.0s)
combined 1.97×
search_nodes 1.1×
exists_prune no-op (EXISTS not pushable)

Byte-identical result multisets. search_edges keeps its full win (post-filter search on the small post-join table is cheap); panel_filters gives back part of the win because the retained where_rows re-evaluates string predicates on the post-join table (a documented follow-on reclaims it via an executor handshake).

Scope

polars/polars-gpu already NIE the multi-node rows op for these disjunctive-WHERE row queries (pre-existing), so this accelerates the pandas + cuDF row-pipeline lanes; polars correctness is preserved by the retained post-filter.

Tests

  • New test_row_pushdown.py: 17 tests — parity pre/post pass (pandas + cuDF via TEST_CUDF) on edge/node search, panel, undirected, combined; guard no-push on EXISTS-marker / multi-alias / shortestPath; pass internals.
  • cuDF lane validated on dgx (RAPIDS 26.02): 17/17 passed.
  • Full cypher+row+conformance suite: 1942 passed; the 4 conformance reds (regex_unicode_trick[ceil/floor/round-filter], categorical_searchany) are pre-existing — verified failing on the clean base commit and with the pass disabled.

🤖 Generated with Claude Code

@lmeyerov lmeyerov force-pushed the feat/gfql-index-degrees branch from cd6f8ae to 3b6b855 Compare July 6, 2026 19:51
@lmeyerov lmeyerov force-pushed the perf/gfql-row-pushdown branch from f670c9e to a8ab8f2 Compare July 6, 2026 19:51
@lmeyerov lmeyerov force-pushed the perf/gfql-row-pushdown branch 3 times, most recently from 9b04eab to 043c2e1 Compare July 8, 2026 00:57
@lmeyerov lmeyerov force-pushed the feat/gfql-index-degrees branch from 63432b9 to ae66194 Compare July 8, 2026 03:11
@lmeyerov lmeyerov force-pushed the perf/gfql-row-pushdown branch from 90db1e3 to ed6e12c Compare July 8, 2026 03:14
@lmeyerov lmeyerov force-pushed the perf/gfql-row-pushdown branch from ed6e12c to 0a2e00a Compare July 8, 2026 08:55
@lmeyerov lmeyerov force-pushed the feat/gfql-index-degrees branch 2 times, most recently from 04bb41a to 0732157 Compare July 9, 2026 01:01
@lmeyerov lmeyerov force-pushed the perf/gfql-row-pushdown branch from e2d693e to d23a5f4 Compare July 9, 2026 01:03
@lmeyerov lmeyerov force-pushed the feat/gfql-index-degrees branch from 0732157 to 4cdec5a Compare July 9, 2026 01:47
@lmeyerov lmeyerov force-pushed the perf/gfql-row-pushdown branch 2 times, most recently from 75193ba to c91b86d Compare July 9, 2026 02:08
lmeyerov and others added 5 commits July 8, 2026 19:41
…e (L4)

A disjunctive / searchAny WHERE collapses the whole MATCH onto the row pipeline,
which materializes the full (a)-[e]->(b) binding table (O(E) joins) BEFORE
filtering. Add a lowering pass that attaches conjuncts referencing exactly ONE
alias (and standalone searchAny ops) to rows(alias_prefilters={alias:[spec]}) as
an ADVISORY HINT; the binding builder pre-filters each alias FRAME before the join.

Redundant-filter safety: the hint is additive — the post-join where_rows /
search_any ops are LEFT IN PLACE. Engines whose row builder honors the hint
(pandas / cuDF via RowPipelineMixin) pre-filter and win; engines that ignore it
(polars' native rows_binding_ops_polars, the shortestPath scalar builder) run the
unchanged post-filter and stay correct. A pushed conjunct is a literal
sub-conjunct of the retained AND, so the pre-filter can only remove rows the
post-filter also removes — never a result change, on any engine. Application
reuses the SAME evaluator (_gfql_eval_string_expr + _gfql_bool_mask) and the SAME
search_any_mask kernel.

Guards (skip pushing a conjunct/op): OPTIONAL / reentry / row-guard / procedure /
seed-row / multi-graph programs; shortestPath scalar bindings; multihop edge
aliases; and any conjunct naming an internal __gfql_/__cypher_ column (EXISTS /
searchAny markers + reentry carries read falsely single-alias).

Results (pandas, seed=7, base = pass-disabled):
  search_edges   @100k 2.9x   @1m 4.62x (7.1s->1.5s)
  panel_filters  @100k 1.32x  @1m 1.47x (24.9s->17.0s)
  combined 1.97x ; search_nodes 1.1x ; exists_prune no-op
Byte-identical result multisets. cuDF parity validated (dgx RAPIDS 26.02, 17/17).
Full cypher+row+conformance suite green (1942 passed; the 4 conformance reds are
pre-existing on the clean base, verified).

Scope: polars/polars-gpu already NIE the multi-node `rows` op for these
disjunctive-WHERE row queries (pre-existing), so this accelerates the pandas +
cuDF row-pipeline lanes; polars correctness is preserved via the retained
post-filter.

New: graphistry/compute/gfql/cypher/row_pushdown.py + test_row_pushdown.py (17
tests: parity + guards + pass internals; cuDF lane via TEST_CUDF).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012rRBmezaBEi2CJgDGkMNgR
@lmeyerov lmeyerov force-pushed the perf/gfql-row-pushdown branch from c91b86d to 5c2afb9 Compare July 9, 2026 02:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant