feat(gfql): physical adjacency indexes for O(degree) seeded traversal#1658
Merged
Conversation
lmeyerov
added a commit
that referenced
this pull request
Jun 28, 2026
…(takeover) Took over the seeded-traversal CSR index (#1658). A 2-agent adversarial review (architecture + correctness) reproduced multiple wrong-answer divergences from the un-indexed scan oracle that the original parity scenarios missed (the cost gate masks them unless >=3 distinct seed keys hit the index path). All fixed; the index now matches scan or falls back — never a wrong answer. +6 engine-parametrized differential regression tests. BLOCKERs: - B1 max_hops silently ignored: the coverage gate omitted max_hops, so the index ran 'hops' (default 1) while the scan ran 'max_hops or hops'. Now resolve eff_hops = max_hops or hops in maybe_index_hop. - B2 duplicate node ids corrupted the node-id index: build treated a per-row (length-E) row_positions array as per-key (length-U). Now collapse to the first row per key + refuse (None) on non-unique ids; gfql_index_all skips node_id gracefully; traverse falls back when the output would carry duplicate id rows (the scan dedups output nodes by id). - B3 endpoints absent from the node table were dropped: the scan synthesizes a row per edge endpoint; the index only materialized existing rows. Now detect missing ids and fall back to scan. IMPORTANTs: - I4 node-row order: node-id-index materialization returned sorted-by-id order; now sort row positions ascending to preserve the .nodes table order. - I5 id(df) fingerprint staleness: an object id can be recycled after GC, so a new same-shape frame could pass validation -> stale index -> wrong answer. Now hold a strong ref + validate by object identity (recycle-proof). - I6 frontier dtype narrowing: an int64 id seed cast to int32 keys wraps and false- matches. Now promote both sides to a common dtype (never narrow). Latent integration bugs surfaced under force policy: - the polars/cudf index hooks passed un-converted pandas seeds to col_to_array (which assumes engine-native frames) -> AttributeError. Now normalize seeds via df_to_engine at the index entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
force-pushed
the
dev/gfql-seeded-traversal-index
branch
from
446d419 to
1e4cde8
Compare
lmeyerov
force-pushed
the
dev/gfql-lazy-gpu
branch
from
194671a to
e5321e4
Compare
lmeyerov
added a commit
that referenced
this pull request
Jun 28, 2026
…(takeover) Took over the seeded-traversal CSR index (#1658). A 2-agent adversarial review (architecture + correctness) reproduced multiple wrong-answer divergences from the un-indexed scan oracle that the original parity scenarios missed (the cost gate masks them unless >=3 distinct seed keys hit the index path). All fixed; the index now matches scan or falls back — never a wrong answer. +6 engine-parametrized differential regression tests. BLOCKERs: - B1 max_hops silently ignored: the coverage gate omitted max_hops, so the index ran 'hops' (default 1) while the scan ran 'max_hops or hops'. Now resolve eff_hops = max_hops or hops in maybe_index_hop. - B2 duplicate node ids corrupted the node-id index: build treated a per-row (length-E) row_positions array as per-key (length-U). Now collapse to the first row per key + refuse (None) on non-unique ids; gfql_index_all skips node_id gracefully; traverse falls back when the output would carry duplicate id rows (the scan dedups output nodes by id). - B3 endpoints absent from the node table were dropped: the scan synthesizes a row per edge endpoint; the index only materialized existing rows. Now detect missing ids and fall back to scan. IMPORTANTs: - I4 node-row order: node-id-index materialization returned sorted-by-id order; now sort row positions ascending to preserve the .nodes table order. - I5 id(df) fingerprint staleness: an object id can be recycled after GC, so a new same-shape frame could pass validation -> stale index -> wrong answer. Now hold a strong ref + validate by object identity (recycle-proof). - I6 frontier dtype narrowing: an int64 id seed cast to int32 keys wraps and false- matches. Now promote both sides to a common dtype (never narrow). Latent integration bugs surfaced under force policy: - the polars/cudf index hooks passed un-converted pandas seeds to col_to_array (which assumes engine-native frames) -> AttributeError. Now normalize seeds via df_to_engine at the index entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
force-pushed
the
dev/gfql-seeded-traversal-index
branch
from
1e4cde8 to
dec68fc
Compare
lmeyerov
force-pushed
the
dev/gfql-lazy-gpu
branch
from
e5321e4 to
5c60a46
Compare
lmeyerov
added a commit
that referenced
this pull request
Jun 28, 2026
…(takeover) Took over the seeded-traversal CSR index (#1658). A 2-agent adversarial review (architecture + correctness) reproduced multiple wrong-answer divergences from the un-indexed scan oracle that the original parity scenarios missed (the cost gate masks them unless >=3 distinct seed keys hit the index path). All fixed; the index now matches scan or falls back — never a wrong answer. +6 engine-parametrized differential regression tests. BLOCKERs: - B1 max_hops silently ignored: the coverage gate omitted max_hops, so the index ran 'hops' (default 1) while the scan ran 'max_hops or hops'. Now resolve eff_hops = max_hops or hops in maybe_index_hop. - B2 duplicate node ids corrupted the node-id index: build treated a per-row (length-E) row_positions array as per-key (length-U). Now collapse to the first row per key + refuse (None) on non-unique ids; gfql_index_all skips node_id gracefully; traverse falls back when the output would carry duplicate id rows (the scan dedups output nodes by id). - B3 endpoints absent from the node table were dropped: the scan synthesizes a row per edge endpoint; the index only materialized existing rows. Now detect missing ids and fall back to scan. IMPORTANTs: - I4 node-row order: node-id-index materialization returned sorted-by-id order; now sort row positions ascending to preserve the .nodes table order. - I5 id(df) fingerprint staleness: an object id can be recycled after GC, so a new same-shape frame could pass validation -> stale index -> wrong answer. Now hold a strong ref + validate by object identity (recycle-proof). - I6 frontier dtype narrowing: an int64 id seed cast to int32 keys wraps and false- matches. Now promote both sides to a common dtype (never narrow). Latent integration bugs surfaced under force policy: - the polars/cudf index hooks passed un-converted pandas seeds to col_to_array (which assumes engine-native frames) -> AttributeError. Now normalize seeds via df_to_engine at the index entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
force-pushed
the
dev/gfql-seeded-traversal-index
branch
from
a5e70f7 to
9cbe4d7
Compare
lmeyerov
added a commit
that referenced
this pull request
Jun 28, 2026
…(takeover) Took over the seeded-traversal CSR index (#1658). A 2-agent adversarial review (architecture + correctness) reproduced multiple wrong-answer divergences from the un-indexed scan oracle that the original parity scenarios missed (the cost gate masks them unless >=3 distinct seed keys hit the index path). All fixed; the index now matches scan or falls back — never a wrong answer. +6 engine-parametrized differential regression tests. BLOCKERs: - B1 max_hops silently ignored: the coverage gate omitted max_hops, so the index ran 'hops' (default 1) while the scan ran 'max_hops or hops'. Now resolve eff_hops = max_hops or hops in maybe_index_hop. - B2 duplicate node ids corrupted the node-id index: build treated a per-row (length-E) row_positions array as per-key (length-U). Now collapse to the first row per key + refuse (None) on non-unique ids; gfql_index_all skips node_id gracefully; traverse falls back when the output would carry duplicate id rows (the scan dedups output nodes by id). - B3 endpoints absent from the node table were dropped: the scan synthesizes a row per edge endpoint; the index only materialized existing rows. Now detect missing ids and fall back to scan. IMPORTANTs: - I4 node-row order: node-id-index materialization returned sorted-by-id order; now sort row positions ascending to preserve the .nodes table order. - I5 id(df) fingerprint staleness: an object id can be recycled after GC, so a new same-shape frame could pass validation -> stale index -> wrong answer. Now hold a strong ref + validate by object identity (recycle-proof). - I6 frontier dtype narrowing: an int64 id seed cast to int32 keys wraps and false- matches. Now promote both sides to a common dtype (never narrow). Latent integration bugs surfaced under force policy: - the polars/cudf index hooks passed un-converted pandas seeds to col_to_array (which assumes engine-native frames) -> AttributeError. Now normalize seeds via df_to_engine at the index entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
force-pushed
the
dev/gfql-seeded-traversal-index
branch
from
9cbe4d7 to
073eb80
Compare
lmeyerov
force-pushed
the
dev/gfql-lazy-gpu
branch
from
36cf1fd to
7207dc8
Compare
lmeyerov
added a commit
that referenced
this pull request
Jun 28, 2026
…(takeover) Took over the seeded-traversal CSR index (#1658). A 2-agent adversarial review (architecture + correctness) reproduced multiple wrong-answer divergences from the un-indexed scan oracle that the original parity scenarios missed (the cost gate masks them unless >=3 distinct seed keys hit the index path). All fixed; the index now matches scan or falls back — never a wrong answer. +6 engine-parametrized differential regression tests. BLOCKERs: - B1 max_hops silently ignored: the coverage gate omitted max_hops, so the index ran 'hops' (default 1) while the scan ran 'max_hops or hops'. Now resolve eff_hops = max_hops or hops in maybe_index_hop. - B2 duplicate node ids corrupted the node-id index: build treated a per-row (length-E) row_positions array as per-key (length-U). Now collapse to the first row per key + refuse (None) on non-unique ids; gfql_index_all skips node_id gracefully; traverse falls back when the output would carry duplicate id rows (the scan dedups output nodes by id). - B3 endpoints absent from the node table were dropped: the scan synthesizes a row per edge endpoint; the index only materialized existing rows. Now detect missing ids and fall back to scan. IMPORTANTs: - I4 node-row order: node-id-index materialization returned sorted-by-id order; now sort row positions ascending to preserve the .nodes table order. - I5 id(df) fingerprint staleness: an object id can be recycled after GC, so a new same-shape frame could pass validation -> stale index -> wrong answer. Now hold a strong ref + validate by object identity (recycle-proof). - I6 frontier dtype narrowing: an int64 id seed cast to int32 keys wraps and false- matches. Now promote both sides to a common dtype (never narrow). Latent integration bugs surfaced under force policy: - the polars/cudf index hooks passed un-converted pandas seeds to col_to_array (which assumes engine-native frames) -> AttributeError. Now normalize seeds via df_to_engine at the index entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
force-pushed
the
dev/gfql-seeded-traversal-index
branch
from
073eb80 to
bb4d7e3
Compare
lmeyerov
force-pushed
the
dev/gfql-lazy-gpu
branch
from
7207dc8 to
8591416
Compare
lmeyerov
added a commit
that referenced
this pull request
Jun 28, 2026
…(takeover) Took over the seeded-traversal CSR index (#1658). A 2-agent adversarial review (architecture + correctness) reproduced multiple wrong-answer divergences from the un-indexed scan oracle that the original parity scenarios missed (the cost gate masks them unless >=3 distinct seed keys hit the index path). All fixed; the index now matches scan or falls back — never a wrong answer. +6 engine-parametrized differential regression tests. BLOCKERs: - B1 max_hops silently ignored: the coverage gate omitted max_hops, so the index ran 'hops' (default 1) while the scan ran 'max_hops or hops'. Now resolve eff_hops = max_hops or hops in maybe_index_hop. - B2 duplicate node ids corrupted the node-id index: build treated a per-row (length-E) row_positions array as per-key (length-U). Now collapse to the first row per key + refuse (None) on non-unique ids; gfql_index_all skips node_id gracefully; traverse falls back when the output would carry duplicate id rows (the scan dedups output nodes by id). - B3 endpoints absent from the node table were dropped: the scan synthesizes a row per edge endpoint; the index only materialized existing rows. Now detect missing ids and fall back to scan. IMPORTANTs: - I4 node-row order: node-id-index materialization returned sorted-by-id order; now sort row positions ascending to preserve the .nodes table order. - I5 id(df) fingerprint staleness: an object id can be recycled after GC, so a new same-shape frame could pass validation -> stale index -> wrong answer. Now hold a strong ref + validate by object identity (recycle-proof). - I6 frontier dtype narrowing: an int64 id seed cast to int32 keys wraps and false- matches. Now promote both sides to a common dtype (never narrow). Latent integration bugs surfaced under force policy: - the polars/cudf index hooks passed un-converted pandas seeds to col_to_array (which assumes engine-native frames) -> AttributeError. Now normalize seeds via df_to_engine at the index entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
force-pushed
the
dev/gfql-seeded-traversal-index
branch
from
bb4d7e3 to
648f10d
Compare
lmeyerov
force-pushed
the
dev/gfql-lazy-gpu
branch
from
8591416 to
3d47a07
Compare
lmeyerov
added a commit
that referenced
this pull request
Jun 28, 2026
…(takeover) Took over the seeded-traversal CSR index (#1658). A 2-agent adversarial review (architecture + correctness) reproduced multiple wrong-answer divergences from the un-indexed scan oracle that the original parity scenarios missed (the cost gate masks them unless >=3 distinct seed keys hit the index path). All fixed; the index now matches scan or falls back — never a wrong answer. +6 engine-parametrized differential regression tests. BLOCKERs: - B1 max_hops silently ignored: the coverage gate omitted max_hops, so the index ran 'hops' (default 1) while the scan ran 'max_hops or hops'. Now resolve eff_hops = max_hops or hops in maybe_index_hop. - B2 duplicate node ids corrupted the node-id index: build treated a per-row (length-E) row_positions array as per-key (length-U). Now collapse to the first row per key + refuse (None) on non-unique ids; gfql_index_all skips node_id gracefully; traverse falls back when the output would carry duplicate id rows (the scan dedups output nodes by id). - B3 endpoints absent from the node table were dropped: the scan synthesizes a row per edge endpoint; the index only materialized existing rows. Now detect missing ids and fall back to scan. IMPORTANTs: - I4 node-row order: node-id-index materialization returned sorted-by-id order; now sort row positions ascending to preserve the .nodes table order. - I5 id(df) fingerprint staleness: an object id can be recycled after GC, so a new same-shape frame could pass validation -> stale index -> wrong answer. Now hold a strong ref + validate by object identity (recycle-proof). - I6 frontier dtype narrowing: an int64 id seed cast to int32 keys wraps and false- matches. Now promote both sides to a common dtype (never narrow). Latent integration bugs surfaced under force policy: - the polars/cudf index hooks passed un-converted pandas seeds to col_to_array (which assumes engine-native frames) -> AttributeError. Now normalize seeds via df_to_engine at the index entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
force-pushed
the
dev/gfql-seeded-traversal-index
branch
from
648f10d to
8759fd4
Compare
lmeyerov
added a commit
that referenced
this pull request
Jun 29, 2026
…(takeover) Took over the seeded-traversal CSR index (#1658). A 2-agent adversarial review (architecture + correctness) reproduced multiple wrong-answer divergences from the un-indexed scan oracle that the original parity scenarios missed (the cost gate masks them unless >=3 distinct seed keys hit the index path). All fixed; the index now matches scan or falls back — never a wrong answer. +6 engine-parametrized differential regression tests. BLOCKERs: - B1 max_hops silently ignored: the coverage gate omitted max_hops, so the index ran 'hops' (default 1) while the scan ran 'max_hops or hops'. Now resolve eff_hops = max_hops or hops in maybe_index_hop. - B2 duplicate node ids corrupted the node-id index: build treated a per-row (length-E) row_positions array as per-key (length-U). Now collapse to the first row per key + refuse (None) on non-unique ids; gfql_index_all skips node_id gracefully; traverse falls back when the output would carry duplicate id rows (the scan dedups output nodes by id). - B3 endpoints absent from the node table were dropped: the scan synthesizes a row per edge endpoint; the index only materialized existing rows. Now detect missing ids and fall back to scan. IMPORTANTs: - I4 node-row order: node-id-index materialization returned sorted-by-id order; now sort row positions ascending to preserve the .nodes table order. - I5 id(df) fingerprint staleness: an object id can be recycled after GC, so a new same-shape frame could pass validation -> stale index -> wrong answer. Now hold a strong ref + validate by object identity (recycle-proof). - I6 frontier dtype narrowing: an int64 id seed cast to int32 keys wraps and false- matches. Now promote both sides to a common dtype (never narrow). Latent integration bugs surfaced under force policy: - the polars/cudf index hooks passed un-converted pandas seeds to col_to_array (which assumes engine-native frames) -> AttributeError. Now normalize seeds via df_to_engine at the index entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
force-pushed
the
dev/gfql-seeded-traversal-index
branch
from
8759fd4 to
9014160
Compare
lmeyerov
force-pushed
the
dev/gfql-lazy-gpu
branch
from
16d5beb to
eea6c96
Compare
lmeyerov
added a commit
that referenced
this pull request
Jun 29, 2026
…(takeover) Took over the seeded-traversal CSR index (#1658). A 2-agent adversarial review (architecture + correctness) reproduced multiple wrong-answer divergences from the un-indexed scan oracle that the original parity scenarios missed (the cost gate masks them unless >=3 distinct seed keys hit the index path). All fixed; the index now matches scan or falls back — never a wrong answer. +6 engine-parametrized differential regression tests. BLOCKERs: - B1 max_hops silently ignored: the coverage gate omitted max_hops, so the index ran 'hops' (default 1) while the scan ran 'max_hops or hops'. Now resolve eff_hops = max_hops or hops in maybe_index_hop. - B2 duplicate node ids corrupted the node-id index: build treated a per-row (length-E) row_positions array as per-key (length-U). Now collapse to the first row per key + refuse (None) on non-unique ids; gfql_index_all skips node_id gracefully; traverse falls back when the output would carry duplicate id rows (the scan dedups output nodes by id). - B3 endpoints absent from the node table were dropped: the scan synthesizes a row per edge endpoint; the index only materialized existing rows. Now detect missing ids and fall back to scan. IMPORTANTs: - I4 node-row order: node-id-index materialization returned sorted-by-id order; now sort row positions ascending to preserve the .nodes table order. - I5 id(df) fingerprint staleness: an object id can be recycled after GC, so a new same-shape frame could pass validation -> stale index -> wrong answer. Now hold a strong ref + validate by object identity (recycle-proof). - I6 frontier dtype narrowing: an int64 id seed cast to int32 keys wraps and false- matches. Now promote both sides to a common dtype (never narrow). Latent integration bugs surfaced under force policy: - the polars/cudf index hooks passed un-converted pandas seeds to col_to_array (which assumes engine-native frames) -> AttributeError. Now normalize seeds via df_to_engine at the index entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
force-pushed
the
dev/gfql-seeded-traversal-index
branch
from
9014160 to
ca5dfab
Compare
lmeyerov
force-pushed
the
dev/gfql-lazy-gpu
branch
from
eea6c96 to
38a08e7
Compare
lmeyerov
added a commit
that referenced
this pull request
Jun 29, 2026
…(takeover) Took over the seeded-traversal CSR index (#1658). A 2-agent adversarial review (architecture + correctness) reproduced multiple wrong-answer divergences from the un-indexed scan oracle that the original parity scenarios missed (the cost gate masks them unless >=3 distinct seed keys hit the index path). All fixed; the index now matches scan or falls back — never a wrong answer. +6 engine-parametrized differential regression tests. BLOCKERs: - B1 max_hops silently ignored: the coverage gate omitted max_hops, so the index ran 'hops' (default 1) while the scan ran 'max_hops or hops'. Now resolve eff_hops = max_hops or hops in maybe_index_hop. - B2 duplicate node ids corrupted the node-id index: build treated a per-row (length-E) row_positions array as per-key (length-U). Now collapse to the first row per key + refuse (None) on non-unique ids; gfql_index_all skips node_id gracefully; traverse falls back when the output would carry duplicate id rows (the scan dedups output nodes by id). - B3 endpoints absent from the node table were dropped: the scan synthesizes a row per edge endpoint; the index only materialized existing rows. Now detect missing ids and fall back to scan. IMPORTANTs: - I4 node-row order: node-id-index materialization returned sorted-by-id order; now sort row positions ascending to preserve the .nodes table order. - I5 id(df) fingerprint staleness: an object id can be recycled after GC, so a new same-shape frame could pass validation -> stale index -> wrong answer. Now hold a strong ref + validate by object identity (recycle-proof). - I6 frontier dtype narrowing: an int64 id seed cast to int32 keys wraps and false- matches. Now promote both sides to a common dtype (never narrow). Latent integration bugs surfaced under force policy: - the polars/cudf index hooks passed un-converted pandas seeds to col_to_array (which assumes engine-native frames) -> AttributeError. Now normalize seeds via df_to_engine at the index entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
force-pushed
the
dev/gfql-seeded-traversal-index
branch
from
ca5dfab to
b65ca7f
Compare
lmeyerov
force-pushed
the
dev/gfql-lazy-gpu
branch
from
38a08e7 to
c5bc2fd
Compare
lmeyerov
added a commit
that referenced
this pull request
Jul 1, 2026
…(takeover) Took over the seeded-traversal CSR index (#1658). A 2-agent adversarial review (architecture + correctness) reproduced multiple wrong-answer divergences from the un-indexed scan oracle that the original parity scenarios missed (the cost gate masks them unless >=3 distinct seed keys hit the index path). All fixed; the index now matches scan or falls back — never a wrong answer. +6 engine-parametrized differential regression tests. BLOCKERs: - B1 max_hops silently ignored: the coverage gate omitted max_hops, so the index ran 'hops' (default 1) while the scan ran 'max_hops or hops'. Now resolve eff_hops = max_hops or hops in maybe_index_hop. - B2 duplicate node ids corrupted the node-id index: build treated a per-row (length-E) row_positions array as per-key (length-U). Now collapse to the first row per key + refuse (None) on non-unique ids; gfql_index_all skips node_id gracefully; traverse falls back when the output would carry duplicate id rows (the scan dedups output nodes by id). - B3 endpoints absent from the node table were dropped: the scan synthesizes a row per edge endpoint; the index only materialized existing rows. Now detect missing ids and fall back to scan. IMPORTANTs: - I4 node-row order: node-id-index materialization returned sorted-by-id order; now sort row positions ascending to preserve the .nodes table order. - I5 id(df) fingerprint staleness: an object id can be recycled after GC, so a new same-shape frame could pass validation -> stale index -> wrong answer. Now hold a strong ref + validate by object identity (recycle-proof). - I6 frontier dtype narrowing: an int64 id seed cast to int32 keys wraps and false- matches. Now promote both sides to a common dtype (never narrow). Latent integration bugs surfaced under force policy: - the polars/cudf index hooks passed un-converted pandas seeds to col_to_array (which assumes engine-native frames) -> AttributeError. Now normalize seeds via df_to_engine at the index entry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
lmeyerov
commented
Jul 8, 2026
| keys_sorted: Any # distinct key ids, ascending (len U) [array] | ||
| group_offsets: Any # CSR offsets into row_positions (len U+1) [array] | ||
| row_positions: Any # edge row indices grouped by key (len E) [array] | ||
| other_values: Any # neighbor id per edge row, ORIGINAL order (len E) [array] |
lmeyerov
commented
Jul 8, 2026
| ALL_KINDS = (EDGE_OUT_ADJ, EDGE_IN_ADJ, NODE_ID) | ||
|
|
||
|
|
||
| def frame_fingerprint(df: Any, cols: Tuple[str, ...], engine: Engine) -> Tuple: |
lmeyerov
commented
Jul 8, 2026
| @@ -0,0 +1,164 @@ | |||
| """Index-driven seeded traversal — the O(degree) fast path. | |||
lmeyerov
commented
Jul 8, 2026
| # ---- planner entry --------------------------------------------------------- | ||
|
|
||
| # Coverage: features the index fast path does NOT yet handle -> caller scans. | ||
| def _hop_is_index_coverable( |
lmeyerov
commented
Jul 8, 2026
| direction: str, | ||
| engine: Engine, | ||
| policy: IndexPolicy, | ||
| nodes: Any, |
lmeyerov
commented
Jul 8, 2026
|
|
||
| def maybe_index_hop( | ||
| g: Plottable, engine: Engine, *, nodes, hops, direction, return_as_wave_front, | ||
| to_fixed_point: bool = False, policy: str = "use", **rest: Any, |
lmeyerov
commented
Jul 8, 2026
| # base record + a `_bail` helper that logs *why* we fell back to scan. All of this | ||
| # is skipped entirely when not tracing, so the hot path pays nothing. | ||
| trace = _trace_active() | ||
| diag: Dict[str, Any] = {} |
lmeyerov
commented
Jul 8, 2026
| from .registry import AdjacencyIndex, NodeIdIndex, frame_fingerprint | ||
|
|
||
|
|
||
| class FrameLike(Protocol): |
Contributor
Author
There was a problem hiding this comment.
why new type vs reuse?
lmeyerov
commented
Jul 8, 2026
|
|
||
|
|
||
| ArrayLike = Any | ||
| ArrayNamespace = Any |
Contributor
Author
There was a problem hiding this comment.
this is malicious compliance, do it right, see DataFrameT etc for an example of doing the work
lmeyerov
commented
Jul 8, 2026
| # polars ~0.02. GPU values provisional (dgx-gated) and conservatively grouped with | ||
| # polars. | ||
| _COST_GATE_FRAC = {Engine.PANDAS: 0.5} | ||
| _COST_GATE_FRAC_DEFAULT = 0.02 |
lmeyerov
commented
Jul 8, 2026
| ) | ||
| if _idxed0 is not None: | ||
| return _idxed0 | ||
|
|
lmeyerov
commented
Jul 8, 2026
| return False | ||
|
|
||
|
|
||
| def _record(decision: Dict[str, Any]) -> None: |
lmeyerov
commented
Jul 8, 2026
| def _trace_active() -> bool: | ||
| """True only inside an ``index_trace()`` / ``gfql_explain`` context. Diagnostic | ||
| enrichment (LP1) is computed only when this is True → zero hot-path cost.""" | ||
| return getattr(_TRACE, "steps", None) is not None |
lmeyerov
commented
Jul 8, 2026
|
|
||
| def _attach(g: Plottable, registry: GfqlIndexRegistry) -> Plottable: | ||
| res = copy.copy(g) | ||
| setattr(res, REGISTRY_ATTR, registry) |
lmeyerov
commented
Jul 8, 2026
| return _attach(g, registry.without(kind)) | ||
|
|
||
|
|
||
| def show_indexes(g: Plottable) -> Any: |
lmeyerov
commented
Jul 8, 2026
| g: Any, | ||
| query: object, | ||
| *, | ||
| index_policy: str = "use", |
lmeyerov
commented
Jul 8, 2026
| FrameLike = DataFrameT | ||
|
|
||
|
|
||
| class ArrayLike(Protocol): |
lmeyerov
commented
Jul 8, 2026
lmeyerov
commented
Jul 8, 2026
| # Type variable for return type preservation in predicates | ||
| T = TypeVar('T') | ||
|
|
||
| class ArrayLike(Protocol): |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack position
Stacks on #1667 (polars-engine followups). Full spine:
master → #1666 (3VL) → #1660 (polars engine + GPU target; #1655 folded in) → #1667 (followups) → **this PR**.Restacked 2026-07-02: the followups PR proved independent of this index layer, so the most
experimental PR (this one) moved to the top of the spine — reviewable against the final
engine surface its hooks land on. Additive — no change to default (un-indexed) behavior
on any engine.
What this does
Opt-in, pay-as-you-go CSR adjacency / node-id indexes → seeded traversal becomes
O(degree), flat in graph size. Sidecar over edge row positions (never reorders
.edges/.nodes), fingerprint-validated by object IDENTITY (a.edges()rebind safelyinvalidates → scan fallback, never a wrong answer). Three uniform surfaces (Python /
Cypher DDL
CREATE/DROP/SHOW GFQL INDEX/ JSON wire) +gfql(..., index_policy=...).Engine-polymorphic: numpy host arrays for pandas/polars, cupy on-device for cudf.
Hooked at every seeded scan site; falls back to scan for uncovered features
(edge/source/dest match,
target_wave_front,min_hops>1, labeling, missing/duplicatenode ids, missing endpoints).
A 2-agent adversarial review reproduced wrong-answer divergences from the un-indexed
scan oracle that the original parity scenarios missed (the cost gate masks them unless
≥3 distinct seed keys hit the index path). All fixed — the index now matches scan or
falls back, never a wrong answer (+6 engine-parametrized differential regression tests):
max_hopswas silently ignored (hop(max_hops=3)ran 1 hop) → now honored.id(df)fingerprint recycling, int32/int64 seed narrowing, and apolars/cudf seed-conversion crash under
force→ all fixed.is_indeprecation → order-preserving semi-join;DROP GFQL INDEX <custom-name>silent no-op → resolve-by-name + raise.Results (dgx-spark, deg-8, warm median) — OUR reproduced numbers
Every benchmarked cell is GUARDED: the timing is discarded unless (a) the index path
was actually taken (
index_trace) AND (b) the index result == the scan result. No cellfailed either guard.
Seeded warm latency is FLAT in N (O(degree)) while the scan is O(E):
(10× more edges → pandas index latency 0.124 → 0.122 ms = flat, while the scan grows 10× = O(E). That is the O(degree) proof.)
cuDF / polars-GPU are flat but floored ~3 ms (cudf) / sub-ms (polars-gpu) by GPU kernel
launch — selective traversal is an indexing problem, not a compute one (CPU wins it).
vs graph DBs (0.8M, identical graph/seed, matched answer counts):
(Adversarial note: the original neo4j SEL2 query was exactly-2-hop — 70 rows vs the
1–2-hop 78 — and undercounted; re-run with
*1..2for a fair 78-row comparison.)Tests
graphistry/tests/compute/gfql/index/test_index.py: 75 passed on dgx (pandas/cudf/polars/polars-gpu), incl. 6 new adversarial regression tests (max_hops, dup ids, missing
endpoint, node order, mixed dtype, stale rebind) + drop-by-custom-name.
🤖 Generated with Claude Code
Review notes
dc4f9da3):DROP GFQL INDEX [IF EXISTS]semantics wired end-to-end (wiremissing_okwas serialized-but-ignored; plain DROP of a missing index now raises, IF EXISTS is a no-op, +test); dead utility code removed; internal review-wave codes scrubbed from production comments; real docstrings on theComputeMixinindex API incl. the previously-undocumentedgfql(index_policy=...); benchmark catalog added tobenchmarks/gfql/README.md(which harness pair backs the published vs-Kuzu claims); bulk-OLAP polars-gpu cell fixed to GPU-or-error.