GFQL/polars: native rows(binding_ops) — traversal queries NIE (blocks polars for edge/multi-hop Cypher)

[lmeyerov]

Summary (native-polars feature; effort L, multi-day)

On polars, any Cypher query that crosses an edge (a traversal) NIEs: polars row pipeline does not yet support op 'rows' when rows carries binding_ops. This blocks graph-benchmark q1, q2, q8, q9 (and every multi-hop/edge query) on polars/polars-gpu. Single-MATCH-node queries already run natively on polars.

Enforcement sites

• graphistry/compute/gfql/lazy/engine/polars/chain.py:301-305 (_run_calls_polars) and 253-257 (_call_native_on_polars force-declines rows when it carries binding_ops/alias_endpoints).
• graphistry/compute/gfql/row/pipeline.py:4712 whitelist _POLARS_NATIVE_ROW_PIPELINE_CALLS = {"rows","skip","limit","distinct","drop_cols"}; force-decline at 4735-4744.

Why it's pandas/cuDF-bound

The traversal itself already runs on polars (each ASTNode/ASTEdge.execute(engine=POLARS) uses the polars hop engine). What is pandas/cuDF-API-bound is the connected-binding state-threading + materialization (the joins/merges):

• _gfql_connected_bindings_state (pipeline.py:3347), _gfql_connected_bindings_row_frame_from_state (3576)
• _gfql_cartesian_node_bindings_row_table (3788), _gfql_multihop_binding_rows (3251, ~93 lines)
• _gfql_disambiguate_has_edge_destination_nodes (3232) + _gfql_has_edge_destination_label_col (3221)
• _gfql_shortest_path_scalar_bindings_row_table (3721) + _gfql_shortest_path_scalar_native (3630)
• _gfql_bindings_row_table alias_endpoints variant (3849); EdgeSemantics.orient_edges (same_path/edge_semantics.py:36)

Idiom counts in the 3180–3860 range: .merge×7, .iloc×8, .drop(columns=)×11, .rename(columns=)×7, .drop_duplicates×5, .isin×3, .assign×4, .duplicated×2, .groupby().min().reset_index(), null-handling .astype(object).where(...).

Effort: L (~6–10 methods, ~400–600 lines), multi-day

Idiom mapping is mostly mechanical (.merge→.join, .isin→.is_in, .rename(columns=dict)→.rename(dict), .drop(columns=)→.drop, .drop_duplicates(subset=)→.unique(subset=), .groupby().min().reset_index()→.group_by().agg(), .iloc[0:0]→.clear()). The finicky parts:

1. .astype(object).where(~null_mask, None) null-normalization has no polars analog (polars nulls are typed); touches the shortest-path scalar path.
2. maintain_order / first-occurrence semantics for parity with pandas.
3. orient_edges must be polars-aware.

MUST land with a parity gate: compare polars results to pandas across many random graphs + the q1/q2/q8/q9 shapes before trusting any number. The repo forbids a polars→pandas→polars arrow bridge (chain.py:269,297,304, plans/gfql-polars-engine NO-CHEATING rule) — do NOT take that shortcut.

Extends #1665 (polars NIE surfaces). Depends on nothing, but pairs with the count(*) broadcast bug (separate P0 issue) — both are needed for correct polars analytical numbers.

[lmeyerov]

Confirmed still open + punted from the 2026-07-06 "small/easy" pass — this is genuine multi-day pandas→polars binding-materialization work, out of scope for the quick fixes. With #1707 + #1708 landed, polars runs graph-bench q3-q7 correctly; the polars/polars-gpu traversal queries (q1/q2/q8/q9) still NIE honestly (verified: clean `NotImplementedError` pointing at engine=pandas, not a silent wrong answer). This issue is the only thing blocking a full 4-engine graph-bench matrix.

[lmeyerov]

MVP landed on `dev/gfql-cypher-std-conformance` commit c9b178ad (2026-07-06): native polars `rows(binding_ops)` for fixed-length connected patterns — no pandas bridge.

Now native on polars (differential parity vs pandas oracle, +20 tests):

• Multi-alias property projections: `MATCH (a)-[e]->(b) RETURN a.x, e.w, b.y`
• graph-benchmark q1/q2 shape: `RETURN b.id, count(a) ... ORDER BY c DESC LIMIT k` (with GFQL/cypher: count(<non-active pattern alias>) NIEs at lowering with misleading 'one MATCH' error #1708's routing)
• q8/q9 shapes: fixed multi-hop counts `MATCH (a)-->(b)-->(c) [WHERE ...] RETURN count(*)`
• forward/reverse/undirected (both-orient concat, path multiplicity preserved), node/edge filter_dicts, edge-type filters, edge-alias payloads, grouped aggregates, DISTINCT, empty results
• plus native `with_(extend=True)` (bindings aggregate lowering needs it) and an honest NIE for `group_by(key_prefixes=...)` (was a latent silent-wrong-key trap once rows() stopped NIE-ing)

Perf @500k nodes / 2M edges (CPU, median-of-5):

query	pandas	polars	speedup
q1 top-k in-degree	2559ms	150ms	17×
seeded 1-hop projection	311ms	23ms	13×
seeded 2-hop count	252ms	35ms	7.3×

Still honestly deferred (NIE): var-length `-[*1..N]->` bindings (the remaining piece for q3-style typed multi-hop), cartesian `MATCH (a),(b)` mode, shortestPath scalar bindings, node/edge `query=`/endpoint-match params, seeded re-entry contexts, `alias_endpoints` variant, whole-row bindings grouping, cross-alias same-path WHERE (separate subsystem).

Benchmark impact: polars/polars-gpu can now run graph-benchmark q1/q2/q8/q9 as real GFQL — combined with #1707/#1708 this unblocks the full 4-engine matrix except q3 (var-length) on polars. polars-gpu shares this code path; on-GPU verification queued for the dgx box. Suites: gfql 3036 + compute 4323 green.

[lmeyerov]

Extended in commit 36857e5c: bounded directed var-length binding rows (`-[i..k]->`, typed `-[:TYPEi..k]->`, exactly-k) via iterative pair joins with Cypher path multiplicity (parallel edges multiply per hop, zero-hop rows for min 0 — parity with pandas `_gfql_multihop_binding_rows`).

This was the last engine-gap for the benchmark matrix: graph-bench q3's shape (`MATCH (a:Person)-[:FOLLOWS*1..2]->(b:Person) ... RETURN avg(b.age)`) now runs natively on polars with verified parity.

Coverage now: all q1-q9 shapes run as real GFQL on pandas, cuDF, AND polars (with #1707/#1708). Remaining honest NIEs on polars: unbounded `[*]`, undirected var-length, aliased var-length edges, cartesian mode, cross-alias same-path WHERE, non-aggregate var-length projections (traversal-chain Phase-1 limit, separate subsystem). polars-gpu shares the code path — on-GPU verification queued for the dgx full-data re-run (#1710 publish path). gfql suite 3042 passed.

Leaving this issue open for the named remainders; the benchmark-blocking scope is done.