perf(gfql/shortest-path): cache native graph state across pairwise calls#1705
Open
lmeyerov wants to merge 1 commit into
Open
perf(gfql/shortest-path): cache native graph state across pairwise calls#1705lmeyerov wants to merge 1 commit into
lmeyerov wants to merge 1 commit into
Conversation
Add opt-in graph-state caching to the igraph/cugraph shortest-path backends so multi-pair native calls reuse the built graph instead of rebuilding it per call. Validated on dgx-spark: cuDF selected-pair direct-native drops 104.4ms -> 57.8ms (~1.8x) on the full 100k-person / 2.42M-FOLLOWS graph-benchmark data, parity-clean on small and full datasets for both engines. The cache is threaded via optional cache/cache_key params (default None = identical prior behavior). Also fixes a min_hops==0 self-pair edge case: src==tgt with min_hops==0 now correctly returns hops=0 even when the node has no outgoing edges, instead of null. Note: this accelerates the native helper only. The integrated GFQL shortestPath query path is still dominated by per-query lowering overhead (~465ms/query on full data); making that fast needs seed-pushed lowering (tracked as P1), not this cache. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> (cherry picked from commit a885ffe7c36269feae37ade0799c25cf614e5aa5)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Opt-in graph-state caching for the igraph/cugraph shortestPath path: the native backend rebuilt graph state on every pairwise call, so multi-pair shortestPath queries paid repeated construction. Caches the constructed graph state across calls within a query.
Split out from the (misplaced) benchmark branch as a standalone same-path perf change — independent of the #1658 index stack. Touches
graphistry/compute/gfql/same_path/native_shortest_path.py(+ test). Cherry-picks cleanly onto master.🤖 Generated with Claude Code