diff --git a/AI_PROGRESS/gfql-programs-spec/PLAN.md b/AI_PROGRESS/gfql-programs-spec/PLAN.md new file mode 100644 index 0000000000..bf95b2e05c --- /dev/null +++ b/AI_PROGRESS/gfql-programs-spec/PLAN.md @@ -0,0 +1,695 @@ +# GFQL Programs Spec Development Plan +**THIS PLAN FILE**: `AI_PROGRESS/gfql-programs-spec/PLAN.md` +**Created**: 2025-07-10 UTC +**Current Branch if any**: dev/gfql-program +**PRs if any**: PR #695 - feat(gfql): GFQL Programs Spec - PRD Development +**PR Target Branch if any**: master +**Base branch if any**: master + +See further info in section `## Context` + +## CRITICAL META-GOALS OF THIS PLAN +**THIS PLAN MUST BE:** +1. **FULLY SELF-DESCRIBING**: All context needed to resume work is IN THIS FILE +2. **CONSTANTLY UPDATED**: Every action's results recorded IMMEDIATELY in the step +3. **THE SINGLE SOURCE OF TRUTH**: If it's not in the plan, it didn't happen +4. **SAFE TO RESUME**: Any AI can pick up work by reading ONLY this file + +**REMEMBER**: External memory is unreliable. This plan is your ONLY memory. + +## CRITICAL: NEVER LEAVE THIS PLAN +**YOU WILL FAIL IF YOU DON'T FOLLOW THIS PLAN EXACTLY** +**TO DO DIFFERENT THINGS, YOU MUST FIRST UPDATE THIS PLAN FILE TO ADD STEPS THAT EXPLICITLY DEFINE THOSE CHANGES.** + +### Anti-Drift Protocol - READ THIS EVERY TIME +**THIS PLAN IS YOUR ONLY MEMORY. TREAT IT AS SACRED.** + +### The Three Commandments: +1. **RELOAD BEFORE EVERY ACTION**: Your memory has been wiped. This plan is all you have. +2. **UPDATE AFTER EVERY ACTION**: If you don't write it down, it never happened. +3. **TRUST ONLY THE PLAN**: Not your memory, not your assumptions, ONLY what's written here. + +### Critical Rules: +- **ONE TASK AT A TIME** - Never jump ahead +- **NO ASSUMPTIONS** - The plan is the only truth. If you need new info, update the plan with new steps to investigate, document, replan, act, and validate. +- **NO OFFROADING** - If it's not in the plan, don't do it + +### Step Execution Protocol - MANDATORY FOR EVERY ACTION +**BEFORE EVERY SINGLE ACTION, NO EXCEPTIONS:** +1. **RELOAD PLAN**: `cat AI_PROGRESS/gfql-programs-spec/PLAN.md | head -200` +2. **FIND YOUR TASK**: Locate the current 🔄 IN_PROGRESS step +3. **EXECUTE**: ONLY do what that step says +4. **UPDATE IMMEDIATELY**: Edit this plan with results BEFORE doing anything else +5. **VERIFY**: `tail -50 AI_PROGRESS/gfql-programs-spec/PLAN.md` + +**THE ONLY SECTION YOU UPDATE IS "Steps" - EVERYTHING ELSE IS READ-ONLY** + +**NEVER:** +- Make decisions without reading the plan first +- Create branches without the plan telling you to +- Create PRs without the plan telling you to +- Switch contexts without updating the plan +- Do ANYTHING without the plan + +### If Confused: +1. STOP +2. Reload this plan +3. Find the last ✅ completed step +4. Continue from there + +## Context (READ-ONLY - Fill in at Plan Creation) + +### Plan Overview +**Raw Prompt**: "in AI_PROGRESS/gfql-programs-spec/ , make a new PLAN.md that goes through some prd work on sketch.md there . First phase (Steps 1.x) should be general analysis of the repo, feature: Do steps around reading our GFQL impl/examples and PyGraphistry regular APIs around GFQL Wire Protocol and Python API, and saving out some relevant knowledge to a lookup file with back references to our repo (file, lineno, snippet). Then enumerate our sketch.md features, and for each, create a sub analysis step (1.X.1, 1.X.2, 1.X.3) about that feature, how it relates, and some critical review of bugs/risks/improvements/etc. After, do a combined critical review, step 1.X+1. Finally, make a new sketch1X.md that supercedes our sketch.md, and is complete unto itself. Do a follow-on step to compare the two and fix sketch1X.md for whatever missed. Once all that is ready, make a new step steries, 2.*, whose first step is to review 1.* and come up with some key different User Personas and key different User Scenarios for each one around these features. Then another step to review those, and if any key gaps, add more Personas/Scenarios. Then, start step seris 3.*. First step is to add a subset for every user persona x user scenario for a role play. Each individual step is to generate a role_play_user_X_scenario_Y.md (catalog these), where you fill out the role play of that scenario getting solved via the wire protocol or python api. End each role play .md with a bit lof localized analysis of what worked, what didn't, and how to improve, with a prioritized breakdown of regular P0 (absolutely & urgenetly required) to P4/P5 (probably won't happen superficial nice-to-haves.) Finally, do a step series 4.X that reviews our 2.* and 3.* to create a sketch3X.md . Make a stp that is a metastep: read all our 2x/3x files & comments, and for each one, create a fresh step to update our 3X.md with appropriate fixes." +**Goal**: Develop comprehensive product specification for GFQL Programs through analysis, user research, and iterative refinement +**Description**: Multi-phase PRD development process starting with technical analysis, moving through user persona development and role-playing, ending with refined specification +**Context**: GFQL is PyGraphistry's declarative query language. The sketch.md proposes extending it from single chains to DAG composition with new features like remote graph loading, graph combinators, and call operations. Binding names must match regex: ^[a-zA-Z_][a-zA-Z0-9_-]*$ +**Success Criteria**: +- Complete technical analysis with code references +- Validated user personas and scenarios +- Role play documents demonstrating API usage +- Final sketch3X.md specification ready for implementation +**Key Constraints**: +- Steps must be dynamic and self-determining +- Role plays must be separate 100+ LOC files with 3-20 turns +- No timeline estimates +- Must follow functional programming practices per ai_code_notes + +### Technical Context +**Initial State**: +- Working Directory: /home/lmeyerov/Work/pygraphistry2 +- Current Branch: `dev/gfql-program` (forked from `master` at `[SHA]`) +- Target Branch: `master` + +**Related Work**: +- Current GFQL implementation in `/graphistry/compute/` +- sketch.md RFC in `AI_PROGRESS/gfql-programs-spec/` +- Depends on: Understanding current GFQL architecture +- Blocks: Future GFQL DAG implementation + +### Strategy +**Approach**: Four-phase iterative development: +1. Technical analysis and initial refinement +2. User persona and scenario development +3. Role-play validation +4. Final synthesis and specification + +**Key Decisions**: +- Dynamic step generation: Later steps determined by earlier findings +- Separate files for each role play: Ensures detailed exploration +- Meta-steps for systematic updates: Maintains traceability + +### Git Strategy +**Planned Git Operations**: +1. Work on dev/gfql-program branch +2. Commit analysis artifacts and specifications +3. Create PR to master when complete + +**Merge Order**: This work → Implementation work + +## Quick Reference (READ-ONLY) +```bash +# Reload plan +cat AI_PROGRESS/gfql-programs-spec/PLAN.md | head -200 + +# Local validation before pushing +./bin/ruff check --fix && ./bin/mypy +shellcheck [script.sh] +./bin/pytest [test] -xvs + +# CI monitoring (use watch to avoid stopping - NEVER ASK USER) +gh pr checks [PR] --repo [owner/repo] --watch +gh run watch [RUN-ID] +watch -n 30 'gh pr checks [PR] --repo [owner/repo]' +# Detailed monitoring with jq: +gh run view [RUN-ID] --json status,conclusion | jq -r '"\(.status) - \(.conclusion)"' +gh run view [RUN-ID] --json jobs | jq -r '.jobs[0].steps[] | select(.status == "in_progress") | .name' +# With timeout to prevent infinite waiting: +timeout 30m gh run watch [RUN-ID] + +# CI debugging with early exit +echo "DEBUG: Early exit" && exit 0 # Add to speed up iteration +git commit -m "DEBUG: Add early exit" +# Remember to remove after fix confirmed +``` +## Step protocol + +### RULES: +- Only update the current 🔄 IN_PROGRESS step +- Use nested numbering (1, 1.1, 1.1.1) to show hierarchy +- Each step should be atomic and verifiable +- Include ALL context in the result (commands, output, errors, decisions) +- When adding new steps: Stop, add the step, save, then execute + +### NEW STEPS +If you need to do something not in the plan: +1. STOP - Do not execute the action +2. ADD A STEP - Create it with clear description, action, success criteria +3. Mark it as 🔄 IN_PROGRESS +4. SAVE THE PLAN +5. THEN EXECUTE + +### STEP COMPACTION + +**Every ~30 completed steps, compact the plan:** +1. **CHECK STEP COUNT** - Count completed steps (✅, ❌, ⏭️) +2. **CREATE HISTORY FILE** - Copy oldest 15+ completed steps to: + - Path: `AI_PROGRESS/gfql-programs-spec/history/steps-to-.md` + - Check existing history files first with `ls AI_PROGRESS/gfql-programs-spec/history/` + - Keep same format as plan.md +3. **REPLACE IN PLAN** - Replace archived steps with: + ``` + ### Steps 1-15: [Brief Title] ✅ ARCHIVED + **Archived**: `AI_PROGRESS/gfql-programs-spec/history/steps1-to-15.md` + **Summary**: + - Key outcome 1 + - Key outcome 2 + - Important artifacts/PRs created + ``` +4. **ADD COMPACTION TASK** - Before starting compaction, add it as a step +5. **VERIFY** - Ensure plan still makes sense after compaction + +Then continue with Step 16... + + +## Status Legend +- 📝 **TODO**: Not started +- 🔄 **IN_PROGRESS**: Currently working on this +- ✅ **DONE**: Completed successfully +- ❌ **FAILED**: Failed, needs retry +- ⏭️ **SKIPPED**: Not needed (explain in result) +- 🚫 **BLOCKED**: Can't proceed (explain in result) + +## LIVE PLAN (THE ONLY SECTION YOU UPDATE) + +Follow `## Step protocol`: + +### Context Preservation (Update ONLY if directed by a step) + + +#### Key Decisions Made + +- [Decision]: [Reasoning] + +#### Lessons Learned + +- [What happened]: [Why it failed]: [How to avoid] + +#### Important Commands + +```bash +# [Description of what this does] +[command] +``` +### Steps + +Reminder, follow `## Step protocol`: + +#### Step 0.1: Create PR for tracking GFQL Programs Spec work +**Status**: ✅ DONE +**Started**: 2025-07-10 UTC +**Action**: Create PR from dev/gfql-program to master for tracking this PRD work +**Success Criteria**: PR created with description of the 4-phase plan +**Result**: +``` +Created PR #695: https://github.com/graphistry/pygraphistry/pull/695 +Title: feat(gfql): GFQL Programs Spec - PRD Development + +The PR includes: +- Summary of 4-phase development process +- Context about GFQL DAG composition features +- Detailed phase descriptions +- Test plan checklist + +Successfully pushed branch and created tracking PR. +``` + +#### Step 1.0: Create GFQL Knowledge Base +**Status**: ✅ DONE +**Started**: 2025-07-10 UTC +**Action**: Read core GFQL implementation files and create lookup file with references +**Success Criteria**: gfql_knowledge_base.md created with file:lineno:snippet references +**Result**: +``` +Created comprehensive knowledge base at: AI_PROGRESS/gfql-programs-spec/gfql_knowledge_base.md + +Key findings: +1. GFQL uses 3-phase algorithm: forward traversal, reverse pruning, output collection +2. Core files analyzed: + - chain.py: Main execution engine with chain() method + - ast.py: Node/Edge AST definitions with JSON serialization + - predicates/: Rich filtering system (numeric, string, temporal, categorical) + - chain_remote.py: Wire protocol for server-side execution + - hop.py: Core traversal logic with wavefront expansion + +3. Architecture patterns: Visitor, Builder, Strategy, Memento +4. Wire protocol: JSON-based with type-tagged operations +5. Integration points documented for future DAG extension + +The knowledge base includes line-by-line references, code snippets, and architectural insights. +``` + +#### Step 1.1: Analyze PyGraphistry APIs Around GFQL +**Status**: ✅ DONE +**Started**: 2025-07-10 UTC +**Action**: Document Wire Protocol and Python API integration points +**Success Criteria**: Added to knowledge base with clear entry points documented +**Result**: +``` +Appended "PyGraphistry API Integration" section to gfql_knowledge_base.md + +Key findings: +1. Entry Points: + - Plottable interface: chain(), chain_remote(), chain_remote_shape() + - ComputeMixin provides implementation bridge + - Full type hints and documentation + +2. Wire Protocol: + - REST endpoint: /api/v2/etl/datasets/{dataset_id}/gfql/{output_type} + - JSON format with gfql_operations field + - Multiple output formats: JSON, CSV, Parquet + - Optional column subsetting and engine selection + +3. Integration Architecture: + - 5-layer design: User API → Protocol → Implementation → Execution → Transport + - Session-based auth with token refresh + - Clean interface/implementation separation + - Local/remote execution transparency + +4. Related APIs: python_remote for arbitrary code execution + +This provides solid foundation for DAG program extensions. +``` + +#### Step 1.2: Enumerate sketch.md Features +**Status**: ✅ DONE +**Started**: 2025-07-10 UTC +**Action**: Create numbered list of all proposed features from sketch.md +**Success Criteria**: Complete feature inventory for systematic analysis +**Result**: +``` +Enumerated features from sketch.md: + +1. Core DAG Composition (QueryDAG/ChainGraph) + - Type: "QueryDAG" for nested composition + - Named graph bindings with lexical scoping + - "ref" parameter for re-rooting operations + - "output" parameter to specify return binding + - Binding name validation: ^[a-zA-Z_][a-zA-Z0-9_-]*$ + +2. Dotted Reference Syntax + - Disambiguation for nested QueryDAGs + - Format: "a.b.c" for hierarchical references + - Used in both "ref" and "output" fields + +3. Remote Graph Loading (RemoteGraph) + - Load existing graphs from Graphistry + - Type: "RemoteGraph" with dataset_id/graph_id + - Enables graph mashups without Python + +4. Graph Combinators + - Union (policies: left, right, merge_left, merge_right) + - Subtract + - Replace (policies: full, patch, extend) + - Intersect + - From (new graph from different node/edge sources) + - Common policies: node/edge removal strategies + +5. Call Operations + - Expose Plottable methods (umap, layout_cugraph, cypher, etc.) + - Type: "call" with function name and params + - Safelisting controls by Hub Tier + - Future: Louie connector integration + +Total: 5 major feature categories to analyze +``` + +#### Step 1.3: Meta-step - Generate Feature Analysis Steps +**Status**: ✅ DONE +**Started**: 2025-07-10 UTC +**Action**: Based on Step 1.2 results, dynamically create Steps 1.3.1 through 1.3.N for each feature +**Success Criteria**: New steps added to plan for each enumerated feature +**Result**: +``` +Generated 5 feature analysis steps (1.3.1 through 1.3.5) for: +1. Core DAG Composition +2. Dotted Reference Syntax +3. Remote Graph Loading +4. Graph Combinators +5. Call Operations + +Each step includes 3 sub-steps for comprehensive analysis. +``` + +#### Step 1.3.1: Feature Analysis - Core DAG Composition (QueryDAG/ChainGraph) +**Status**: ✅ DONE +**Started**: 2025-07-10 UTC +**Action**: Analyze DAG composition feature and its relationship to current Chain architecture +**Success Criteria**: Document implementation approach, risks, and improvements +**Sub-steps**: +- 1.3.1.1: Analyze how DAG composition relates to current Chain architecture +- 1.3.1.2: Identify implementation challenges and dependencies +- 1.3.1.3: Critical review (bugs/risks/improvements) +**Result**: +``` +Created feature_analysis_dag_composition.md with comprehensive analysis. + +Key findings: +1. Architecture: QueryDAG extends linear chains to DAG with named bindings & lexical scoping + - Reuses: AST, execution engine, remote execution, Plottable interface + - New: QueryDAG AST node, reference resolver, execution context + +2. Challenges: + - Wire protocol: New message types, backward compatibility required + - Performance: Memory management for bindings, parallel execution opportunities + - Security: Resource limits, access control for remote graphs + +3. Critical Review: + - Risks: Circular references, resource exhaustion, name collisions + - Improvements: Lazy evaluation, query optimization, type system + - Priority: MVP focus on basic DAG execution with backward compatibility + +4. Implementation roadmap: 4 phases over 6-8 weeks total +``` + +#### Step 1.3.2: Feature Analysis - Dotted Reference Syntax +**Status**: ✅ DONE +**Started**: 2025-07-10 UTC +**Action**: Analyze dotted reference syntax for disambiguation +**Success Criteria**: Document scoping rules, edge cases, and implementation strategy +**Sub-steps**: +- 1.3.2.1: Analyze lexical scoping and reference resolution +- 1.3.2.2: Identify ambiguity edge cases +- 1.3.2.3: Critical review (bugs/risks/improvements) +**Result**: +``` +Created feature_analysis_dotted_reference.md with comprehensive analysis. + +Key findings: +1. Scoping: Lexical scoping with closest-binding principle + - Root-to-leaf traversal for "a.b.c" resolution + - Statically resolvable at parse time + - Similar to programming language variable scoping + +2. Edge Cases (6 identified): + - Name shadowing at different levels + - Deep nesting performance/readability + - Partial path resolution failures + - Circular references + - Reserved name collisions ("type", "graph", "output") + - Cross-DAG sibling references + +3. Critical Review: + - Parsing: Need escape sequences for dots in names + - Performance: Deep traversal costs, caching needed + - Alternatives: Considered slash notation, JSONPath, explicit scopes + - Error handling: Need contextual messages with suggestions + +4. Recommendations: Enhanced syntax spec, caching, circular detection, validation tools +``` + +#### Step 1.3.3: Feature Analysis - Remote Graph Loading (RemoteGraph) +**Status**: ✅ DONE +**Started**: 2025-07-10 UTC +**Action**: Analyze remote graph loading feature +**Success Criteria**: Document security, performance, and integration considerations +**Sub-steps**: +- 1.3.3.1: Analyze relationship to current dataset loading mechanisms +- 1.3.3.2: Security and authentication considerations +- 1.3.3.3: Critical review (network/caching/error handling) +**Result**: +``` +Created feature_analysis_remote_graph.md with comprehensive analysis. + +Key findings: +1. Current vs RemoteGraph: + - Current: Client-driven bind(dataset_id) with Python orchestration + - RemoteGraph: Server-side loading within pure GFQL DAGs + - Enables graph mashups without client round-trips + +2. Security Critical: + - Access control: Need token propagation strategy (implicit/embedded/capability) + - Cross-tenant isolation risks when loading multiple graphs + - Data exfiltration vectors: combinators, errors, timing + - Recommendation: Capability-based tokens with strict validation + +3. Technical Challenges: + - Network: No current timeout/retry; need cascading DAG timeouts + - Caching: 3-tier strategy (metadata/data/computation) + - Errors: Need comprehensive categorization and safe messages + - Resources: Per-user and per-graph quotas essential + +4. Implementation: Phased approach starting with single RemoteGraph +``` + +#### Step 1.3.4: Feature Analysis - Graph Combinators +**Status**: ✅ DONE +**Started**: 2025-07-10 UTC +**Action**: Analyze graph combinator operations +**Success Criteria**: Document policy system, memory efficiency, and API design +**Sub-steps**: +- 1.3.4.1: Map to existing graph operations in PyGraphistry +- 1.3.4.2: Policy system design review +- 1.3.4.3: Critical review (edge cases/memory/consistency) +**Result**: +``` +Created feature_analysis_graph_combinators.md with comprehensive analysis. + +Key findings: +1. Current Gap: + - PyGraphistry has rich single-graph ops but no native combinators + - Missing: union, intersection, subtraction, systematic merge policies + - Combinators enable semantic multi-graph operations + +2. Policy System: + - Good coverage: attribute merging, node/edge removal strategies + - Recommendations: Add type coercion, aggregation, custom merge functions + - Need schema evolution handling + +3. Critical Issues: + - Memory: Graph duplication, intermediate results, remote loading + - Edge cases: Mismatched IDs, cycles, empty graphs, schema conflicts + - Consistency: Node-edge integrity, attribute conflicts + - Performance: Opportunities for parallel execution, GPU acceleration + +4. Implementation: Phased approach with robust testing strategy +``` + +#### Step 1.3.5: Feature Analysis - Call Operations +**Status**: ✅ DONE +**Started**: 2025-07-10 UTC +**Action**: Analyze call operations for Plottable methods +**Success Criteria**: Document method inventory, security model, and extensibility +**Sub-steps**: +- 1.3.5.1: Inventory of Plottable methods to expose +- 1.3.5.2: Safelisting and security model +- 1.3.5.3: Critical review (compatibility/validation/extensibility) +**Result**: +``` +Created feature_analysis_call_operations.md with comprehensive analysis. + +Key findings: +1. Method Inventory: + - Identified 21+ Plottable methods suitable for GFQL + - Categories: transforms (umap, cypher), layouts, analytics, graph ops + - Recommended phased rollout starting with common/simple methods + +2. Security Model: + - Tiered access: Basic/Standard/Advanced/Enterprise + - Method-level and parameter-level restrictions + - Resource limits per tier (nodes/edges/timeouts) + - Safelist configuration with execution sandbox + +3. Technical Review: + - JSON compatibility: Type serialization challenges addressed + - Validation: Pydantic schemas for type safety + - Extensibility: Plugin architecture for Louie connectors + - Error handling: Comprehensive categories with debugging info + +4. Implementation: Start with 5-7 core methods, expand based on usage +``` + +#### Step 1.4: Combined Critical Review +**Status**: ✅ DONE +**Started**: 2025-07-10 UTC +**Action**: Synthesize all feature analyses into comprehensive review +**Success Criteria**: Cross-cutting concerns and integration challenges documented +**Result**: +``` +Created combined_critical_review.md synthesizing all 5 feature analyses. + +Key findings: +1. Cross-cutting concerns: + - Reference resolution complexity in nested contexts + - Compound memory management (DAGs + remote + combinators) + - Expanded security attack surface + - Type safety challenges across features + +2. Integration challenges: + - Complex feature interdependencies + - Conflicting execution models (parallel vs sequential) + - Inconsistent error handling patterns + - Competing caching strategies + +3. Architecture assessment: + - Strengths: Clear purpose, natural composability + - Weaknesses: Complexity explosion, inconsistent patterns + - Need unified execution context and resource manager + +4. Priority recommendations: + - Phase 1: Core DAG + basic remote (low risk, high value) + - Phase 2: Dotted refs + simple combinators + - Phase 3: Advanced combinators + basic calls + - Phase 4: Full call operations + optimizations + +Risk matrix identified memory exhaustion and debugging complexity as highest concerns. +``` + +#### Step 1.5: Create sketch1X.md +**Status**: ✅ DONE +**Started**: 2025-07-10 UTC +**Action**: Write refined specification incorporating all analysis +**Success Criteria**: Complete, self-contained spec created +**Result**: +``` +Created sketch1X.md - comprehensive refined specification. + +Major enhancements: +1. Complete security model with tiered access and resource limits +2. Robust error handling with categorized codes and debugging info +3. Production-ready features: caching, retries, timeouts, memory mgmt +4. Clear 4-phase implementation roadmap over 6 months +5. Enhanced reference system with escape sequences and caching +6. Advanced combinators with type coercion and schema evolution +7. Comprehensive call operations with validation and tracing +8. Real-world examples for common patterns +9. Performance guidelines and optimization strategies +10. Enhanced wire protocol with versioning and metadata + +The spec is self-contained, production-ready, and addresses all concerns from analyses. +``` + +#### Step 1.6: Compare and Refine sketch1X.md +**Status**: ✅ DONE +**Started**: 2025-07-10 UTC +**Action**: Diff against original sketch.md and add missing elements +**Success Criteria**: No features lost, all improvements incorporated +**Result**: +``` +Created sketch_comparison.md documenting comprehensive comparison. + +Key findings: +1. No features lost - all original concepts preserved and enhanced +2. Major additions: + - Complete security model with tiers and access control + - 6-month phased implementation roadmap + - Comprehensive error handling framework + - Performance guidelines and optimization strategies + - Production features: monitoring, retries, caching + - Rich real-world examples + +3. Quality transformation: + - From high-level RFC to implementation-ready spec + - Security-first design throughout + - Production readiness built in + - Enhanced developer experience + +No updates needed to sketch1X.md - it successfully incorporates all original features with significant enhancements. + +Phase 1 complete! Ready for Phase 2: User Personas and Scenarios. +``` + +#### Step 2.0: Meta-step - Generate Phase 2 Steps +**Status**: ✅ DONE +**Started**: 2025-07-10 UTC +**Action**: Review Phase 1 results and create user persona/scenario development steps +**Success Criteria**: Dynamic Phase 2 steps added based on Phase 1 findings +**Result**: +``` +Generated Phase 2 steps based on features analyzed: +- Step 2.1: Define user personas based on GFQL features +- Step 2.2: Create scenarios for each persona +- Step 2.3: Review and add missing personas/scenarios +Focus: Functionality, resource limits, error handling; light on deep security/perf details. +``` + +#### Step 2.1: Define User Personas +**Status**: ✅ DONE +**Started**: 2025-07-10 UTC +**Action**: Create key user personas based on GFQL Programs features +**Success Criteria**: 4-6 personas covering different use cases for DAGs, remote graphs, combinators, and calls +**Result**: +``` +Created user_personas.md with 6 diverse personas: + +1. Alex - Security Analyst: Multi-source investigations, graph merging +2. Sam - Data Scientist: Complex pipelines, ML algorithms, resource needs +3. Jordan - Business Analyst: Low-code user, simple combinations, clear errors +4. Morgan - DevOps Engineer: Service dependencies, performance concerns +5. Casey - Compliance Officer: Audit trails, sensitive data, watchlists +6. Riley - Research Scientist: Custom algorithms, large graphs, parallelization + +Each persona has different technical skills and uses different GFQL features. +``` + +#### Step 2.2: Create User Scenarios per Persona +**Status**: ✅ DONE +**Started**: 2025-07-10 UTC +**Action**: For each persona, define 3-5 scenarios using GFQL Programs features +**Success Criteria**: Realistic scenarios that exercise different feature combinations +**Result**: +``` +Created user_scenarios.md with 22 scenarios across 6 personas: + +Alex (Security): 4 scenarios - threat investigation, lateral movement, timeline, hunting +Sam (Data Science): 4 scenarios - fraud detection, feature engineering, A/B testing, scoring +Jordan (Business): 3 scenarios - collaboration, customer journey, quarterly comparison +Morgan (DevOps): 4 scenarios - dependency mapping, impact analysis, capacity, drift +Casey (Compliance): 3 scenarios - sanctions, ownership, regulatory impact +Riley (Research): 4 scenarios - protein analysis, multi-omics, enrichment, comparison + +Each scenario identifies key features used and realistic challenges. +``` + +#### Step 2.3: Review and Gap Analysis +**Status**: ✅ DONE +**Started**: 2025-07-10 UTC +**Action**: Review personas/scenarios for coverage gaps and add if needed +**Success Criteria**: Comprehensive coverage of feature usage patterns +**Result**: +``` +Created persona_scenario_gap_analysis.md + +Coverage Assessment: Good (85%) +- All major features represented +- Diverse skill levels (high/medium/low) +- Multiple industries covered + +Minor Gaps Identified: +- Dotted references under-used (only 2/22 scenarios) +- Limited error handling scenarios +- Could use API developer and startup personas + +Decision: Proceed with current personas/scenarios as they provide sufficient coverage +for meaningful role plays. Can address gaps in future iterations. +``` + +#### Step 3.0: Meta-step - Generate Phase 3 Role Play Steps +**Status**: 📝 TODO +**Started**: +**Action**: Based on Phase 2 personas × scenarios, create individual role play steps +**Success Criteria**: Matrix of role play steps created, one per persona/scenario combo +**Result**: +``` +[To be filled] +``` + +#### Step 4.0: Meta-step - Generate Phase 4 Synthesis Steps +**Status**: 📝 TODO +**Started**: +**Action**: Review Phases 2 & 3, create steps for sketch3X.md development +**Success Criteria**: Systematic update steps for final specification +**Result**: +``` +[To be filled] +``` \ No newline at end of file diff --git a/AI_PROGRESS/gfql-programs-spec/combined_critical_review.md b/AI_PROGRESS/gfql-programs-spec/combined_critical_review.md new file mode 100644 index 0000000000..2bfbfd66ea --- /dev/null +++ b/AI_PROGRESS/gfql-programs-spec/combined_critical_review.md @@ -0,0 +1,319 @@ +# Combined Critical Review: GFQL Programs Specification + +## Executive Summary + +This review synthesizes the analysis of five interconnected features proposed for GFQL Programs: DAG Composition, Dotted References, Remote Graph Loading, Graph Combinators, and Call Operations. Together, these features transform GFQL from a linear chain-based query language into a full-fledged graph programming environment. While each feature has individual merit, their true power and complexity emerge from their interactions. + +## 1. Cross-Cutting Concerns + +### 1.1 Reference Resolution Complexity + +The reference resolution system underpins all features and presents several cross-cutting challenges: + +- **Ambiguity in Nested Contexts**: DAG composition + dotted references + graph combinators create deeply nested scopes where reference resolution becomes complex +- **Performance Impact**: Every operation requires reference resolution, potentially creating a bottleneck +- **Error Propagation**: Reference errors in one part of a DAG can cascade unpredictably +- **Debugging Difficulty**: Users need clear visibility into how references resolve across nested scopes + +### 1.2 Memory Management + +All features contribute to memory pressure: + +- **DAG Composition**: Multiple named graphs in memory simultaneously +- **Remote Graph Loading**: Large datasets loaded from external sources +- **Graph Combinators**: Intermediate results from union/intersection operations +- **Call Operations**: Transformations that may duplicate or expand data + +The compound effect could easily exhaust available memory without careful management. + +### 1.3 Security Surface Area + +Each feature expands the attack surface: + +- **Remote Graph**: External data access, authentication token management +- **Call Operations**: Arbitrary method execution, parameter injection risks +- **Graph Combinators**: Resource exhaustion through combinatorial explosion +- **DAG Composition**: Complex reference chains enabling data exfiltration + +The intersection of these features creates novel attack vectors. + +### 1.4 Type Safety and Validation + +The dynamic nature of the system challenges type safety: + +- **Schema Evolution**: Graphs loaded at different times may have incompatible schemas +- **Operation Compatibility**: Not all operations work on all graph types +- **Parameter Validation**: Call operations need runtime type checking +- **Result Predictability**: Complex DAGs make output types hard to predict + +## 2. Integration Challenges + +### 2.1 Feature Interdependencies + +The features form a complex dependency graph: + +``` +DAG Composition (foundation) +├── Dotted References (enables nested access) +├── Remote Graph Loading (provides data sources) +│ └── Graph Combinators (operates on loaded graphs) +└── Call Operations (transforms any graph in DAG) + └── Graph Combinators (uses call operations) +``` + +This creates circular dependencies that complicate implementation ordering. + +### 2.2 Execution Model Conflicts + +Different features assume different execution models: + +- **DAG Composition**: Assumes parallel execution of independent branches +- **Call Operations**: Often require sequential execution with side effects +- **Remote Graph**: Asynchronous loading with unpredictable latency +- **Graph Combinators**: May benefit from lazy evaluation + +Reconciling these models requires careful architectural decisions. + +### 2.3 Error Handling Inconsistencies + +Each feature has different error modes: + +- **Remote Graph**: Network timeouts, authentication failures +- **Call Operations**: Invalid parameters, execution failures +- **Graph Combinators**: Schema mismatches, empty results +- **Reference Resolution**: Missing references, circular dependencies + +A unified error handling strategy is essential but challenging. + +### 2.4 Caching Strategy Conflicts + +Different features benefit from different caching approaches: + +- **Remote Graph**: Long-term caching of external data +- **DAG Results**: Short-term caching of intermediate results +- **Call Operations**: Method-specific caching policies +- **Reference Resolution**: Parse-time caching of resolved paths + +## 3. Common Patterns and Design Principles + +### 3.1 Emergent Patterns + +Several patterns emerge across features: + +1. **Lazy Evaluation**: Beneficial for DAGs, combinators, and remote loading +2. **Resource Limits**: Essential for all features to prevent abuse +3. **Validation Layers**: Every feature needs input validation +4. **Audit Logging**: Security and debugging require comprehensive logging +5. **Progressive Enhancement**: Features should gracefully degrade + +### 3.2 Design Principles + +The analysis reveals implicit design principles: + +1. **Composability**: Features should combine predictably +2. **Fail-Fast**: Validate early, fail with clear errors +3. **Resource Awareness**: Every operation must consider resource impact +4. **Security by Default**: Restrictive defaults with explicit permissions +5. **Backward Compatibility**: New features cannot break existing usage + +### 3.3 Architectural Patterns + +Common architectural needs: + +1. **Plugin Architecture**: Extensibility for call operations and combinators +2. **Strategy Pattern**: Different policies for merging, caching, execution +3. **Observer Pattern**: Monitoring and debugging hooks +4. **Factory Pattern**: Creating appropriate executors based on context + +## 4. Security and Performance Implications + +### 4.1 Compound Security Risks + +Feature interactions create new vulnerabilities: + +1. **Authentication Token Leakage** + - Remote graphs pass tokens through DAG execution + - Call operations might log sensitive parameters + - Error messages could expose authentication details + +2. **Cross-Tenant Data Mixing** + - Graph combinators merge data from different sources + - Shared execution context risks data leakage + - Cache poisoning across tenant boundaries + +3. **Denial of Service Amplification** + - DAG + Remote Graph: Fetch loops exhausting bandwidth + - Combinators + Call Operations: Exponential computation growth + - Deep nesting: Stack overflow or memory exhaustion + +### 4.2 Performance Multiplication Effects + +Feature combinations can dramatically impact performance: + +1. **Network Multiplication** + - DAG with multiple remote graphs = multiple network calls + - Reference resolution may require additional lookups + - Retry policies amplify network traffic + +2. **Memory Multiplication** + - Each graph binding holds complete dataset + - Combinators create intermediate results + - Call operations may duplicate data + +3. **Computation Multiplication** + - Nested DAGs multiply execution paths + - Graph combinators have quadratic complexity + - Call operations on combined graphs amplify costs + +## 5. Priority Recommendations + +### 5.1 Implementation Priority Matrix + +| Feature | Priority | Risk | Dependencies | Value | +|---------|----------|------|--------------|-------| +| Basic DAG + References | Critical | Medium | None | Foundation for all | +| Remote Graph (basic) | High | High | DAG | Enables key use cases | +| Call Operations (core) | High | Medium | DAG | Extends functionality | +| Graph Combinators | Medium | Low | DAG, Remote | Advanced workflows | +| Advanced Features | Low | High | All | Future extensibility | + +### 5.2 Phased Rollout Strategy + +**Phase 1: Foundation (Months 1-2)** +- Core DAG execution without nesting +- Simple reference resolution (no dots) +- Basic validation and error handling +- Memory management framework + +**Phase 2: Remote Data (Months 2-3)** +- Remote graph loading with current auth +- Simple caching strategy +- Resource limits and timeouts +- Security audit + +**Phase 3: Transformations (Months 3-4)** +- Core call operations with safelist +- Parameter validation framework +- Basic graph combinators +- Performance optimization + +**Phase 4: Advanced Features (Months 4-6)** +- Nested DAGs with dotted references +- Advanced combinators with policies +- Extended call operations +- Production hardening + +## 6. Risk Matrix + +### 6.1 Technical Risks + +| Risk | Probability | Impact | Mitigation | +|------|-------------|--------|------------| +| Memory exhaustion | High | Critical | Strict limits, monitoring | +| Reference ambiguity | Medium | High | Clear scoping rules, validation | +| Performance degradation | High | High | Caching, lazy evaluation | +| Security vulnerabilities | Medium | Critical | Audit, penetration testing | +| Breaking changes | Low | High | Careful API design, versioning | + +### 6.2 Operational Risks + +| Risk | Probability | Impact | Mitigation | +|------|-------------|--------|------------| +| Debugging complexity | High | Medium | Comprehensive logging, tools | +| User confusion | Medium | Medium | Clear documentation, examples | +| Support burden | High | Medium | Self-service debugging tools | +| Adoption friction | Medium | High | Gradual rollout, migration tools | + +## 7. Implementation Dependencies + +### 7.1 Technical Dependencies + +```mermaid +graph TD + A[AST Infrastructure] --> B[DAG Execution Engine] + B --> C[Reference Resolver] + C --> D[Remote Graph Loader] + B --> E[Call Executor] + D --> F[Graph Combinators] + E --> F + B --> G[Memory Manager] + B --> H[Security Framework] + + I[Cache Layer] --> D + I --> E + I --> F + + J[Resource Limiter] --> B + J --> D + J --> E + J --> F +``` + +### 7.2 Critical Path + +1. **AST Extensions** (Week 1-2) +2. **Reference Resolution** (Week 2-3) +3. **DAG Execution** (Week 3-4) +4. **Memory Management** (Week 4-5) +5. **Security Framework** (Week 5-6) +6. **Feature Implementation** (Week 6+) + +## 8. Overall Architecture Coherence Assessment + +### 8.1 Strengths + +1. **Conceptual Clarity**: Each feature has a clear purpose +2. **Composability**: Features combine naturally +3. **Extensibility**: Architecture supports future growth +4. **Backward Compatibility**: Existing code continues to work + +### 8.2 Weaknesses + +1. **Complexity Explosion**: Feature interactions create emergent complexity +2. **Resource Management**: No unified approach across features +3. **Error Handling**: Inconsistent patterns between features +4. **Performance Predictability**: Hard to estimate resource needs + +### 8.3 Architectural Recommendations + +1. **Unified Execution Context** + ```python + class ExecutionContext: + memory_manager: MemoryManager + reference_resolver: ReferenceResolver + security_context: SecurityContext + resource_limiter: ResourceLimiter + cache_manager: CacheManager + ``` + +2. **Layered Architecture** + - Core AST and execution engine + - Feature-specific executors + - Cross-cutting concerns layer + - Public API layer + +3. **Plugin Architecture** + - Extensible call operations + - Custom graph combinators + - External data sources + - Custom policies + +4. **Monitoring and Observability** + - Execution tracing + - Performance metrics + - Resource usage tracking + - Security audit logs + +## Conclusion + +The GFQL Programs specification represents a significant evolution of PyGraphistry's capabilities, transforming it from a visualization-focused tool to a comprehensive graph programming platform. While each individual feature is well-conceived, their integration presents substantial challenges in security, performance, and complexity management. + +Success requires: +1. Careful phased implementation with continuous validation +2. Strong focus on cross-cutting concerns from the start +3. Comprehensive testing of feature interactions +4. Clear documentation and debugging tools +5. Conservative resource management +6. Security-first design approach + +The potential value is significant, but the implementation complexity demands a methodical approach with careful attention to the system-level implications of these interconnected features. \ No newline at end of file diff --git a/AI_PROGRESS/gfql-programs-spec/feature_analysis_call_operations.md b/AI_PROGRESS/gfql-programs-spec/feature_analysis_call_operations.md new file mode 100644 index 0000000000..7f14def68d --- /dev/null +++ b/AI_PROGRESS/gfql-programs-spec/feature_analysis_call_operations.md @@ -0,0 +1,547 @@ +# Feature Analysis: Call Operations + +## Overview + +The Call Operations feature aims to expose PyGraphistry's Plottable interface methods through GFQL, enabling users to invoke graph transformation and analysis methods without requiring Python. This analysis examines the implementation requirements, security considerations, and design decisions for exposing these methods safely and effectively. + +## 1.3.5.1: Inventory of Plottable Methods to Expose + +### Methods That Return New Plottables + +Based on the codebase analysis, the following Plottable methods return new Plottable instances and are candidates for GFQL exposure: + +#### Core Graph Transformations +1. **umap()** - UMAP dimensionality reduction for node/edge embeddings + - Parameters: X, y, kind, scale, n_neighbors, min_dist, spread, etc. + - Returns: Plottable with UMAP embeddings applied + +2. **cypher()** - Execute Cypher queries against Neo4j/Memgraph/Neptune + - Parameters: query (string), params (dict) + - Returns: Plottable with query results + +3. **hypergraph()** - Convert tabular data to hypergraph representation + - Parameters: raw_events, entity_types, opts, drop_na, etc. + - Returns: Plottable with hypergraph structure + +#### Layout Methods +4. **layout_cugraph()** - GPU-accelerated graph layouts + - Parameters: layout, params, kind, directed, G, bind_position, etc. + - Returns: Plottable with computed positions + +5. **layout_igraph()** - IGraph-based layouts + - Parameters: layout, directed, use_vids, bind_position, params, etc. + - Returns: Plottable with computed positions + +6. **layout_graphviz()** - Graphviz layouts + - Parameters: prog, args, directed, strict, graph_attr, etc. + - Returns: Plottable with computed positions + +7. **fa2_layout()** - ForceAtlas2 layout + - Parameters: fa2_params, circle_layout_params, singleton_layout, etc. + - Returns: Plottable with computed positions + +#### Graph Analytics +8. **compute_cugraph()** - GPU-accelerated graph algorithms + - Parameters: alg, out_col, params, kind, directed, G + - Returns: Plottable with computed metrics + +9. **compute_igraph()** - IGraph algorithms + - Parameters: alg, out_col, directed, use_vids, params + - Returns: Plottable with computed metrics + +#### Feature Engineering +10. **featurize()** - Extract features from graph structure + - Parameters: kind, X, y, feature params + - Returns: Plottable with extracted features + +11. **dbscan()** - DBSCAN clustering + - Parameters: min_dist, min_samples, eps, metric, etc. + - Returns: Plottable with cluster assignments + +12. **embed()** - Graph embeddings + - Parameters: relation, proto, various embedding params + - Returns: Plottable with embeddings + +#### Data Transformations +13. **transform()** / **transform_umap()** - Transform new data using fitted models + - Parameters: df, y, kind, min_dist, n_neighbors, etc. + - Returns: Plottable with transformed data + +14. **materialize_nodes()** - Create explicit node table from edges + - Parameters: reuse, engine + - Returns: Plottable with materialized nodes + +15. **collapse()** - Collapse nodes based on attributes + - Parameters: node, attribute, column, self_edges, etc. + - Returns: Plottable with collapsed graph + +#### Graph Filtering/Selection +16. **hop()** - Multi-hop traversal + - Parameters: nodes, hops, direction, edge_match, etc. + - Returns: Plottable with traversal results + +17. **filter_nodes_by_dict()** / **filter_edges_by_dict()** - Dictionary-based filtering + - Parameters: filter_dict + - Returns: Plottable with filtered graph + +18. **drop_nodes()** / **keep_nodes()** - Node-based filtering + - Parameters: nodes + - Returns: Plottable with filtered graph + +19. **prune_self_edges()** - Remove self-loops + - Parameters: none + - Returns: Plottable without self-edges + +#### Graph Metrics +20. **get_degrees()** / **get_indegrees()** / **get_outdegrees()** - Degree calculations + - Parameters: col names + - Returns: Plottable with degree metrics + +21. **get_topological_levels()** - Topological sorting + - Parameters: level_col, allow_cycles, warn_cycles, etc. + - Returns: Plottable with topological levels + +### Good Candidates for Initial GFQL Exposure + +For the initial implementation, prioritize methods that: +1. Have simple, JSON-serializable parameters +2. Are commonly used in graph analysis workflows +3. Have minimal side effects +4. Don't require complex object initialization + +**Recommended Initial Set:** +- `umap()` - Core embedding functionality +- `layout_cugraph()` / `fa2_layout()` - Essential layouts +- `compute_cugraph()` / `compute_igraph()` - Key algorithms +- `hop()` - Graph traversal +- `filter_nodes_by_dict()` / `filter_edges_by_dict()` - Filtering +- `get_degrees()` - Basic metrics +- `materialize_nodes()` - Data preparation + +### Parameter Types and Validation Needs + +#### Common Parameter Types +1. **Strings**: algorithm names, column names, layout types +2. **Numbers**: int (n_neighbors, hops), float (min_dist, scale) +3. **Booleans**: directed, allow_cycles, drop_na +4. **Dictionaries**: params, filter_dict, opts +5. **Lists**: entity_types, node lists, column lists +6. **Enums**: kind ("nodes"/"edges"), direction ("forward"/"reverse"/"both") + +#### Validation Requirements +1. **Type Validation**: Ensure parameters match expected types +2. **Range Validation**: n_neighbors > 0, min_dist >= 0, etc. +3. **Enum Validation**: Check against allowed values +4. **Dictionary Schema**: Validate structure of complex parameters +5. **Column Existence**: Verify referenced columns exist +6. **Compatibility**: Check parameter combinations are valid + +## 1.3.5.2: Safelisting and Security Model + +### Access Control Levels + +#### 1. Method-Level Access Control +```python +SAFELIST_TIERS = { + "basic": [ + "get_degrees", "get_indegrees", "get_outdegrees", + "filter_nodes_by_dict", "filter_edges_by_dict", + "materialize_nodes", "prune_self_edges" + ], + "standard": [ + # Includes basic + + "hop", "collapse", "drop_nodes", "keep_nodes", + "fa2_layout", "get_topological_levels" + ], + "advanced": [ + # Includes standard + + "umap", "featurize", "dbscan", "embed", + "layout_cugraph", "compute_cugraph", + "layout_igraph", "compute_igraph" + ], + "enterprise": [ + # Includes advanced + + "cypher", "hypergraph", "transform", "transform_umap", + # Custom/experimental methods + ] +} +``` + +#### 2. Parameter-Level Restrictions +```python +PARAMETER_RESTRICTIONS = { + "compute_cugraph": { + "basic": {"alg": ["pagerank", "degree_centrality"]}, + "standard": {"alg": ["pagerank", "degree_centrality", "betweenness_centrality", "louvain"]}, + "advanced": {"alg": "*"} # All algorithms + }, + "umap": { + "standard": {"n_neighbors": range(5, 50), "min_dist": [0.1, 0.25, 0.5]}, + "advanced": {"n_neighbors": range(2, 200), "min_dist": "*"} + } +} +``` + +#### 3. Resource Limits +```python +RESOURCE_LIMITS = { + "basic": { + "max_nodes": 10_000, + "max_edges": 50_000, + "timeout_seconds": 30 + }, + "standard": { + "max_nodes": 100_000, + "max_edges": 1_000_000, + "timeout_seconds": 300 + }, + "advanced": { + "max_nodes": 10_000_000, + "max_edges": 100_000_000, + "timeout_seconds": 3600 + } +} +``` + +### Security Implementation + +#### 1. Safelist Configuration +```python +class CallSafelist: + def __init__(self, tier: str, custom_safelist: Optional[Dict] = None): + self.tier = tier + self.allowed_methods = set(SAFELIST_TIERS.get(tier, [])) + if custom_safelist: + self.allowed_methods.update(custom_safelist.get("allow", [])) + self.allowed_methods -= set(custom_safelist.get("deny", [])) + + def is_allowed(self, method: str, params: Dict) -> Tuple[bool, Optional[str]]: + if method not in self.allowed_methods: + return False, f"Method '{method}' not allowed for tier '{self.tier}'" + + # Check parameter restrictions + if method in PARAMETER_RESTRICTIONS: + restrictions = PARAMETER_RESTRICTIONS[method].get(self.tier, {}) + for param, value in params.items(): + if param in restrictions: + allowed = restrictions[param] + if allowed != "*" and value not in allowed: + return False, f"Parameter '{param}={value}' not allowed" + + return True, None +``` + +#### 2. Execution Sandbox +```python +class CallExecutor: + def __init__(self, safelist: CallSafelist, resource_limiter: ResourceLimiter): + self.safelist = safelist + self.limiter = resource_limiter + + def execute(self, plottable: Plottable, method: str, params: Dict) -> Plottable: + # Security checks + allowed, error = self.safelist.is_allowed(method, params) + if not allowed: + raise SecurityError(error) + + # Resource checks + if not self.limiter.check_limits(plottable): + raise ResourceError("Graph exceeds resource limits") + + # Parameter validation + validated_params = self.validate_params(method, params) + + # Execute with timeout + with self.limiter.timeout(): + func = getattr(plottable, method) + return func(**validated_params) +``` + +### Security Implications + +#### 1. Denial of Service (DoS) +- **Risk**: Expensive operations (UMAP on large graphs) +- **Mitigation**: Resource limits, timeouts, rate limiting + +#### 2. Data Exfiltration +- **Risk**: Methods revealing sensitive graph structure +- **Mitigation**: Result size limits, output sanitization + +#### 3. Code Injection +- **Risk**: String parameters (cypher queries, column names) +- **Mitigation**: Parameter validation, query sanitization + +#### 4. Resource Exhaustion +- **Risk**: Memory/CPU intensive operations +- **Mitigation**: Graph size limits, operation timeouts + +#### 5. Unauthorized Access +- **Risk**: Accessing methods beyond tier +- **Mitigation**: Strict safelist enforcement, audit logging + +## 1.3.5.3: Critical Review + +### Compatibility and Validation + +#### JSON Serialization Compatibility + +**Compatible Parameter Types:** +- Primitives: string, number, boolean, null +- Collections: arrays, objects (dicts) +- Enums: string values from predefined sets + +**Incompatible Types Requiring Adaptation:** +```python +# Problem: Functions/Callables +"singleton_layout": Callable # fa2_layout parameter + +# Solution: Predefined function names +"singleton_layout": "circle" | "random" | "grid" + +# Problem: Complex objects +"G": cugraph.Graph # compute_cugraph parameter + +# Solution: Reference by ID or auto-detection +"G": "auto" | {"ref": "graph_id"} +``` + +#### Parameter Validation Framework +```python +from typing import Any, Dict, List, Optional, Union +from pydantic import BaseModel, validator + +class CallParameters(BaseModel): + method: str + params: Dict[str, Any] + + @validator('method') + def validate_method(cls, v): + if v not in ALLOWED_METHODS: + raise ValueError(f"Unknown method: {v}") + return v + + @validator('params') + def validate_params(cls, v, values): + method = values.get('method') + schema = METHOD_SCHEMAS.get(method) + if schema: + # Validate against method-specific schema + schema.validate(v) + return v + +# Method-specific schemas +class UMAPParameters(BaseModel): + X: Optional[Union[List[str], str]] = None + y: Optional[Union[List[str], str]] = None + kind: Literal["nodes", "edges"] = "nodes" + n_neighbors: int = Field(ge=2, le=200) + min_dist: float = Field(ge=0.0, le=1.0) + # ... other parameters +``` + +### Type Safety + +#### Runtime Type Checking +```python +def validate_call_params(method: str, params: Dict) -> Dict: + """Validate and coerce parameters for a method call""" + + # Get method signature + sig = inspect.signature(getattr(Plottable, method)) + + # Check required parameters + for param_name, param in sig.parameters.items(): + if param.default == param.empty and param_name not in params: + raise ValueError(f"Missing required parameter: {param_name}") + + # Validate types + for param_name, value in params.items(): + if param_name not in sig.parameters: + raise ValueError(f"Unknown parameter: {param_name}") + + # Type coercion/validation + expected_type = sig.parameters[param_name].annotation + if expected_type != param.empty: + params[param_name] = coerce_type(value, expected_type) + + return params +``` + +### Future Extensibility + +#### 1. Louie Connector Integration +```python +class LouieConnectorCall: + """Future extension for Louie connector methods""" + + def __init__(self, connector_id: str, method: str, params: Dict): + self.connector = load_connector(connector_id) + self.method = method + self.params = params + + def execute(self, plottable: Plottable) -> Plottable: + # Apply connector-specific transformation + data = self.connector.transform(plottable, self.method, self.params) + return plottable.bind(edges=data['edges'], nodes=data['nodes']) +``` + +#### 2. Custom Method Registration +```python +class MethodRegistry: + """Extensible method registry for future additions""" + + def __init__(self): + self.methods = {} + self._register_core_methods() + + def register(self, name: str, func: Callable, schema: BaseModel): + """Register a new method for GFQL exposure""" + self.methods[name] = { + 'func': func, + 'schema': schema, + 'tier': 'custom' + } + + def execute(self, plottable: Plottable, name: str, params: Dict): + if name not in self.methods: + raise ValueError(f"Unknown method: {name}") + + method_info = self.methods[name] + validated = method_info['schema'](**params).dict() + return method_info['func'](plottable, **validated) +``` + +#### 3. Plugin Architecture +```python +class CallPlugin: + """Base class for call operation plugins""" + + def get_methods(self) -> Dict[str, Dict]: + """Return methods exposed by this plugin""" + raise NotImplementedError + + def validate(self, method: str, params: Dict) -> Dict: + """Validate parameters for a method""" + raise NotImplementedError + + def execute(self, plottable: Plottable, method: str, params: Dict) -> Plottable: + """Execute the method""" + raise NotImplementedError +``` + +### Error Handling and Debugging + +#### 1. Error Categories +```python +class CallError(Exception): + """Base class for call operation errors""" + pass + +class MethodNotFoundError(CallError): + """Method doesn't exist or isn't exposed""" + pass + +class ParameterValidationError(CallError): + """Invalid parameters for method""" + pass + +class ExecutionError(CallError): + """Error during method execution""" + pass + +class SecurityError(CallError): + """Security policy violation""" + pass +``` + +#### 2. Debugging Information +```python +class CallDebugInfo: + def __init__(self, method: str, params: Dict): + self.method = method + self.params = params + self.start_time = time.time() + self.graph_shape_before = None + self.graph_shape_after = None + self.execution_time = None + self.error = None + + def to_dict(self) -> Dict: + return { + "method": self.method, + "params": self.params, + "execution_time": self.execution_time, + "graph_shape_change": { + "before": self.graph_shape_before, + "after": self.graph_shape_after + }, + "error": str(self.error) if self.error else None + } +``` + +#### 3. Execution Tracing +```python +class TracedCallExecutor: + def execute(self, plottable: Plottable, call: Dict) -> Tuple[Plottable, Dict]: + debug_info = CallDebugInfo(call["function"], call["params"]) + + try: + # Capture initial state + debug_info.graph_shape_before = { + "nodes": len(plottable._nodes) if plottable._nodes is not None else 0, + "edges": len(plottable._edges) if plottable._edges is not None else 0 + } + + # Execute + result = self._execute_call(plottable, call) + + # Capture final state + debug_info.graph_shape_after = { + "nodes": len(result._nodes) if result._nodes is not None else 0, + "edges": len(result._edges) if result._edges is not None else 0 + } + + debug_info.execution_time = time.time() - debug_info.start_time + + return result, debug_info.to_dict() + + except Exception as e: + debug_info.error = e + debug_info.execution_time = time.time() - debug_info.start_time + raise CallExecutionError( + f"Failed to execute {call['function']}: {str(e)}", + debug_info=debug_info.to_dict() + ) +``` + +## Implementation Recommendations + +### Phase 1: Core Implementation +1. Implement basic call operation with safelist +2. Add parameter validation for core methods +3. Implement resource limits and timeouts +4. Add comprehensive error handling + +### Phase 2: Security Hardening +1. Implement tier-based access control +2. Add parameter-level restrictions +3. Implement audit logging +4. Add rate limiting + +### Phase 3: Extended Features +1. Add more methods to safelist +2. Implement custom method registration +3. Add Louie connector support +4. Implement debugging/tracing features + +### Best Practices +1. **Default Deny**: Only expose explicitly safelisted methods +2. **Validate Everything**: Never trust client-provided parameters +3. **Resource Limits**: Always enforce limits on expensive operations +4. **Audit Trail**: Log all operations for security monitoring +5. **Graceful Degradation**: Provide clear errors when operations fail +6. **Version Compatibility**: Design for backward compatibility + +## Conclusion + +The Call Operations feature provides powerful graph transformation capabilities through GFQL while maintaining security and performance. The tiered access model, comprehensive validation, and extensible architecture ensure the feature can grow with user needs while protecting system resources. Careful implementation of the security model and validation framework will be critical for successful deployment. \ No newline at end of file diff --git a/AI_PROGRESS/gfql-programs-spec/feature_analysis_dag_composition.md b/AI_PROGRESS/gfql-programs-spec/feature_analysis_dag_composition.md new file mode 100644 index 0000000000..da5f7fe58d --- /dev/null +++ b/AI_PROGRESS/gfql-programs-spec/feature_analysis_dag_composition.md @@ -0,0 +1,346 @@ +# Feature Analysis: Core DAG Composition (QueryDAG/ChainGraph) + +## Executive Summary + +This analysis examines the proposed QueryDAG/ChainGraph feature that extends GFQL from single-chain operations to directed acyclic graph (DAG) compositions. The feature enables multiple graph bindings, remote graph loading, and complex graph combinators while maintaining backward compatibility with existing Chain operations. + +## 1.3.1.1: Relationship to Current Chain Architecture + +### How QueryDAG Extends the Single-Chain Model + +The current GFQL architecture is built around a linear chain of operations: + +```python +# Current: Linear sequence +g.chain([ + n({"type": "person"}), + e_forward(), + n({"type": "company"}) +]) +``` + +QueryDAG extends this to support: + +1. **Multiple Named Graphs**: Instead of operating on a single implicit graph, QueryDAG introduces a binding environment where multiple graphs can be named and referenced: + +```python +# Proposed: DAG with multiple graphs +ChainGraph({ + "people": Chain([n({"type": "person"})]), + "companies": Chain([n({"type": "company"})]), + "connections": Chain(ref="people", chain=[e_forward()]) +}) +``` + +2. **Lexical Scoping**: References follow lexical scoping rules, where the closest binding to a reference is used. This is similar to variable scoping in programming languages. + +3. **Explicit Output Selection**: While chains implicitly return the final operation's result, QueryDAG allows explicit output selection via the `output` field. + +### Reusable Components from Current Architecture + +The following components can be directly reused: + +1. **AST Infrastructure**: + - `ASTSerializable` base class for JSON serialization + - `ASTObject` interface with `__call__` and `reverse` methods + - Existing AST nodes (`ASTNode`, `ASTEdge`) + - Predicate system remains unchanged + +2. **Execution Engine**: + - The 3-phase algorithm (forward/reverse/combine) can be applied to each sub-chain + - `combine_steps()` function for merging results + - Engine abstraction (pandas/cudf) works as-is + +3. **Remote Execution**: + - `chain_remote.py` infrastructure can be extended + - Wire protocol JSON structure naturally extends to nested operations + - Authentication and session handling remain the same + +4. **Plottable Integration**: + - Results still return Plottable objects + - Node/edge bindings work the same way + - Visualization settings preservation + +### New Components Required + +1. **QueryDAG AST Node**: +```python +class QueryDAG(ASTSerializable): + def __init__(self, graph: Dict[str, Union[Chain, 'QueryDAG']], output: Optional[str] = None): + self.graph = graph + self.output = output or list(graph.keys())[-1] + + def validate(self): + # Validate binding names match pattern ^[a-zA-Z_][a-zA-Z0-9_-]*$ + # Check for circular references + # Ensure output exists in graph +``` + +2. **Reference Resolution**: + - New `ref` parameter on Chain class + - Dotted reference parser for nested scopes (e.g., "alerts.start") + - Reference validation during AST construction + +3. **Execution Context**: +```python +class ExecutionContext: + """Manages graph bindings during DAG execution""" + def __init__(self): + self.bindings: Dict[str, Plottable] = {} + self.scopes: List[Dict[str, Plottable]] = [] + + def bind(self, name: str, value: Plottable): + self.bindings[name] = value + + def resolve(self, ref: str) -> Plottable: + # Handle dotted references + # Search scopes in reverse order (lexical scoping) +``` + +## 1.3.1.2: Implementation Challenges and Dependencies + +### Wire Protocol Changes + +1. **New Message Types**: +```json +{ + "type": "QueryDAG", + "graph": { + "a": {"type": "Chain", "chain": [...]}, + "b": {"type": "Chain", "ref": "a", "chain": [...]} + }, + "output": "b" +} +``` + +2. **Extended Chain Format**: +```json +{ + "type": "Chain", + "ref": "some_graph", // New optional field + "chain": [...] +} +``` + +3. **Remote Graph Loading**: +```json +{ + "type": "RemoteGraph", + "dataset_id": "abc123" +} +``` + +### Backward Compatibility Concerns + +1. **API Compatibility**: + - Existing `g.chain([...])` calls must continue working + - New `ChainGraph({...})` is additive, not breaking + - Server must handle both old and new wire formats + +2. **Wire Protocol Versioning**: + - Add protocol version negotiation + - Server should detect message format by presence of "type" field + - Graceful degradation for older clients + +3. **Migration Path**: +```python +# Old code continues to work +g.chain([n({"type": "person"})]) + +# Can be gradually migrated to +ChainGraph({ + "result": Chain([n({"type": "person"})]) +}) +``` + +### Performance Implications + +1. **Memory Management**: + - Each binding holds a full Plottable (nodes + edges DataFrames) + - Need reference counting or garbage collection for intermediate results + - Consider lazy evaluation for unused bindings + +2. **Execution Optimization**: + - Parallel execution of independent subgraphs + - Common subexpression elimination + - Query planning for optimal execution order + +3. **Network Overhead**: + - Larger wire protocol messages + - Multiple remote graph fetches + - Consider batching remote operations + +### Memory Management for Multiple Graph Bindings + +1. **Binding Lifecycle**: +```python +class GraphBindingManager: + def __init__(self, max_memory_mb: int = 1000): + self.bindings: Dict[str, Plottable] = {} + self.access_count: Dict[str, int] = {} + self.memory_usage: Dict[str, int] = {} + + def add_binding(self, name: str, graph: Plottable): + # Track memory usage + # Implement LRU eviction if needed + + def get_binding(self, name: str) -> Plottable: + # Update access count + # Handle cache misses +``` + +2. **Resource Limits**: + - Maximum number of concurrent bindings + - Total memory usage caps + - Timeout for long-running DAGs + +## 1.3.1.3: Critical Review + +### Potential Bugs/Edge Cases + +1. **Circular References**: +```python +# This should be detected and rejected +ChainGraph({ + "a": Chain(ref="b", chain=[...]), + "b": Chain(ref="a", chain=[...]) +}) +``` + +2. **Name Collisions**: +```python +# Shadowing in nested scopes +ChainGraph({ + "data": RemoteGraph("abc"), + "nested": ChainGraph({ + "data": RemoteGraph("xyz"), # Shadows outer "data" + "result": Chain(ref="data") # Which one? + }) +}) +``` + +3. **Missing References**: +```python +# Reference to non-existent binding +Chain(ref="nonexistent", chain=[...]) +``` + +4. **Resource Exhaustion**: + - Loading too many large remote graphs + - Deep nesting causing stack overflow + - Combinatorial explosion in graph combinations + +### Security Risks + +1. **Resource Exhaustion Attacks**: + - Malicious DAGs with excessive bindings + - Recursive structures consuming memory + - Remote graph fetching as DoS vector + +2. **Access Control**: + - Ensure remote graph access respects permissions + - Prevent information leakage through error messages + - Validate dataset_id format to prevent injection + +3. **Execution Limits**: +```python +class DAGExecutionLimits: + max_bindings = 100 + max_nesting_depth = 10 + max_execution_time = 300 # seconds + max_memory_usage = 1024 # MB + max_remote_fetches = 20 +``` + +### Suggested Improvements + +1. **Lazy Evaluation**: + - Only compute bindings that are referenced + - Cache intermediate results + - Streaming execution for large graphs + +2. **Query Optimization**: +```python +class QueryOptimizer: + def optimize(self, dag: QueryDAG) -> QueryDAG: + # Common subexpression elimination + # Dead code elimination + # Parallel execution planning + # Push filters down to data sources +``` + +3. **Better Error Messages**: +```python +class DAGValidationError(ValueError): + def __init__(self, message: str, location: str, suggestion: str): + self.location = location # e.g., "graph.alerts.start" + self.suggestion = suggestion + super().__init__(f"{message} at {location}. {suggestion}") +``` + +4. **Type System**: + - Add optional type annotations for graph schemas + - Compile-time validation of operations + - Better IDE support through type hints + +### Priority Assessment + +**High Priority** (Required for MVP): +1. Basic QueryDAG execution with single-level bindings +2. Reference resolution (without dotted syntax) +3. RemoteGraph loading +4. Backward compatibility +5. Basic validation (circular refs, missing refs) + +**Medium Priority** (Post-MVP): +1. Dotted reference syntax +2. Nested QueryDAG support +3. Performance optimizations +4. Graph combinators (union, intersection) +5. Memory management + +**Low Priority** (Future Enhancements): +1. Query optimization +2. Lazy evaluation +3. Type system +4. Advanced debugging tools +5. Visual DAG editor + +## Implementation Roadmap + +### Phase 1: Core Infrastructure (2-3 weeks) +1. Implement QueryDAG/ChainGraph classes +2. Add ref parameter to Chain +3. Basic execution context +4. Update wire protocol +5. Basic validation + +### Phase 2: Remote Graphs (1-2 weeks) +1. Implement RemoteGraph class +2. Integration with existing dataset loading +3. Permission checking +4. Caching layer + +### Phase 3: Advanced Features (2-3 weeks) +1. Dotted references +2. Nested QueryDAG +3. Graph combinators +4. Performance optimizations + +### Phase 4: Production Hardening (1-2 weeks) +1. Resource limits +2. Security review +3. Performance testing +4. Documentation +5. Migration guide + +## Conclusion + +The QueryDAG feature is a natural evolution of GFQL that addresses real user needs (JPMC, Louie) while maintaining the elegance of the current design. The implementation can leverage significant portions of the existing codebase, with the main challenges being: + +1. Managing multiple graph bindings efficiently +2. Ensuring backward compatibility +3. Preventing resource exhaustion +4. Providing clear error messages + +With careful implementation following the suggested phases, this feature can be delivered with minimal risk to existing functionality while opening up powerful new use cases for graph analysis workflows. \ No newline at end of file diff --git a/AI_PROGRESS/gfql-programs-spec/feature_analysis_dotted_reference.md b/AI_PROGRESS/gfql-programs-spec/feature_analysis_dotted_reference.md new file mode 100644 index 0000000000..0ba687c00f --- /dev/null +++ b/AI_PROGRESS/gfql-programs-spec/feature_analysis_dotted_reference.md @@ -0,0 +1,450 @@ +# Feature Analysis: Dotted Reference Syntax + +## Executive Summary + +The Dotted Reference Syntax (`a.b.c`) is a critical feature in the GFQL DAG system that enables disambiguation of references in nested QueryDAG structures. This analysis examines the lexical scoping rules, identifies ambiguity edge cases, and provides a critical review with recommendations for improvement. + +## 1.3.2.1: Lexical Scoping and Reference Resolution + +### How the "a.b.c" Syntax Works + +Based on the sketch.md specification, the dotted reference syntax provides a hierarchical naming mechanism for graph references within nested QueryDAGs: + +```json +{ + "type": "QueryDAG", + "graph": { + "alerts": { + "type": "QueryDAG", + "graph": { + "start": [ ... ], + "hops": { "type": "Chain", "ref": "start", ... } + } + }, + "fraud": { + "type": "QueryDAG", + "graph": { + "start": [ ... ], + "hops": { "type": "Chain", "ref": "start", ... } + } + } + }, + "output": "alerts.start" // Dotted reference to disambiguate +} +``` + +### Scoping Rules + +The specification indicates lexical scoping with closest binding resolution: + +1. **Lexical Scoping**: References are resolved from the current scope outward +2. **Closest Binding**: When multiple bindings exist for a name, the closest (most local) one wins +3. **Static Resolution**: All references are resolvable at parse time (not runtime) + +### Resolution Algorithm + +The implied resolution algorithm follows these steps: + +1. **Parse Reference**: Split dotted reference into components (e.g., `"a.b.c"` → `["a", "b", "c"]`) + +2. **Resolve Root**: + - If single component (e.g., `"start"`), search from current scope upward + - If multiple components, first component must be at current or parent scope + +3. **Traverse Path**: + - For each subsequent component, navigate into that named sub-graph + - Each component must resolve to a QueryDAG or final binding + +4. **Validate Target**: + - Final component must resolve to a valid graph binding + - Type must be compatible with usage context (Chain, QueryDAG, or output reference) + +### Implementation Pseudocode + +```python +def resolve_reference(ref: str, current_scope: Dict[str, Any], parent_scopes: List[Dict[str, Any]]) -> Any: + components = ref.split('.') + + if len(components) == 1: + # Simple reference - search lexically + return lexical_search(components[0], current_scope, parent_scopes) + + # Dotted reference - resolve root first + root = components[0] + root_value = lexical_search(root, current_scope, parent_scopes) + + # Traverse remaining path + current = root_value + for component in components[1:]: + if not isinstance(current, dict) or 'graph' not in current: + raise ResolutionError(f"Cannot traverse into {component}") + current = current['graph'].get(component) + if current is None: + raise ResolutionError(f"Component {component} not found") + + return current + +def lexical_search(name: str, current_scope: Dict[str, Any], parent_scopes: List[Dict[str, Any]]) -> Any: + # Check current scope first + if name in current_scope: + return current_scope[name] + + # Check parent scopes from innermost to outermost + for scope in reversed(parent_scopes): + if name in scope: + return scope[name] + + raise ResolutionError(f"Name {name} not found in any scope") +``` + +## 1.3.2.2: Ambiguity Edge Cases + +### 1. Conflicting Names at Different Levels + +**Scenario**: Same name exists at multiple nesting levels + +```json +{ + "type": "QueryDAG", + "graph": { + "data": { "type": "Chain", ... }, // Outer "data" + "process": { + "type": "QueryDAG", + "graph": { + "data": { "type": "Chain", ... }, // Inner "data" + "result": { + "type": "Chain", + "ref": "data" // Which "data"? (Answer: inner one - lexical scoping) + } + } + } + } +} +``` + +**Resolution**: Lexical scoping means inner "data" shadows outer "data" + +### 2. Deeply Nested Structures + +**Scenario**: Very deep nesting with long dotted paths + +```json +{ + "type": "QueryDAG", + "graph": { + "level1": { + "type": "QueryDAG", + "graph": { + "level2": { + "type": "QueryDAG", + "graph": { + "level3": { + "type": "QueryDAG", + "graph": { + "data": { "type": "Chain", ... } + } + } + } + } + } + }, + "consumer": { + "type": "Chain", + "ref": "level1.level2.level3.data" // Very long path + } + } +} +``` + +**Issues**: +- Performance of deep traversal +- Readability and maintenance challenges +- Potential for typos in long paths + +### 3. Partial Path References + +**Scenario**: Using incomplete paths when multiple matches exist + +```json +{ + "type": "QueryDAG", + "graph": { + "moduleA": { + "type": "QueryDAG", + "graph": { + "data": { "type": "Chain", ... }, + "process": { "type": "Chain", ... } + } + }, + "moduleB": { + "type": "QueryDAG", + "graph": { + "data": { "type": "Chain", ... }, + "analyze": { + "type": "Chain", + "ref": "process" // Error: "process" only exists in moduleA + } + } + } + } +} +``` + +**Issue**: Reference to "process" from moduleB scope will fail + +### 4. Circular References + +**Scenario**: Graph definitions that reference each other + +```json +{ + "type": "QueryDAG", + "graph": { + "a": { + "type": "Chain", + "ref": "b.result" // Forward reference + }, + "b": { + "type": "QueryDAG", + "graph": { + "result": { + "type": "Chain", + "ref": "a" // Circular reference! + } + } + } + } +} +``` + +**Issue**: Creates infinite resolution loop + +### 5. Reserved Names Collision + +**Scenario**: Using names that might conflict with system reserved words + +```json +{ + "type": "QueryDAG", + "graph": { + "type": { "type": "Chain", ... }, // Conflicts with "type" field + "graph": { "type": "Chain", ... }, // Conflicts with "graph" field + "output": { "type": "Chain", ... }, // Conflicts with "output" field + "ref": { "type": "Chain", ... } // Conflicts with "ref" field + } +} +``` + +**Issue**: Parser ambiguity between field names and binding names + +### 6. Cross-DAG References + +**Scenario**: Attempting to reference across sibling DAGs + +```json +{ + "type": "QueryDAG", + "graph": { + "dag1": { + "type": "QueryDAG", + "graph": { + "data": { "type": "Chain", ... } + } + }, + "dag2": { + "type": "QueryDAG", + "graph": { + "process": { + "type": "Chain", + "ref": "dag1.data" // Can we reference across siblings? + } + } + } + } +} +``` + +**Question**: Should cross-sibling references be allowed or only parent-child? + +## 1.3.2.3: Critical Review + +### Potential Parsing Issues + +1. **Ambiguous Grammar**: The dot notation could conflict with other uses of dots (e.g., in string values, regex patterns) + +2. **Escape Sequences**: No clear specification for handling binding names with dots + ```json + { + "graph": { + "my.dotted.name": { "type": "Chain", ... }, // How to reference this? + } + } + ``` + +3. **Whitespace Handling**: Should `"a . b . c"` be valid? What about `"a..b"`? + +4. **Case Sensitivity**: Are references case-sensitive? Should "Data" match "data"? + +### Performance Considerations + +1. **Deep Traversal Cost**: Each dot requires a dictionary lookup and type check + - O(n) where n is the number of dots + - Could be expensive for deeply nested structures + +2. **Resolution Caching**: No mention of caching resolved references + - Static resolution suggests parse-time caching is possible + - Would improve runtime performance + +3. **Memory Overhead**: Maintaining scope chains for resolution + - Each nested QueryDAG adds to the scope chain + - Deep nesting could have memory implications + +### Alternative Syntax Options + +1. **Path Separators**: + - Slash notation: `"alerts/start"` (more URL-like) + - Arrow notation: `"alerts->start"` (clearer directionality) + - Bracket notation: `"alerts[start]"` (JavaScript-like) + +2. **Absolute vs Relative Paths**: + ```json + { + "ref": "/alerts/start" // Absolute from root + "ref": "./start" // Relative to current + "ref": "../sibling/data" // Relative with parent traversal + } + ``` + +3. **Named Scopes**: + ```json + { + "ref": {"scope": "alerts", "name": "start"} // Explicit scope reference + } + ``` + +4. **JSONPath-style**: + ```json + { + "ref": "$.alerts.graph.start" // JSONPath syntax + } + ``` + +### Error Handling and Debugging + +1. **Error Messages**: Need clear, actionable error messages + - "Reference 'a.b.c' not found" is insufficient + - Better: "Cannot resolve 'c' in path 'a.b.c': 'b' is not a QueryDAG" + +2. **Debugging Tools**: + - Reference resolution trace/log + - Visual scope hierarchy display + - Interactive reference validator + +3. **Validation Modes**: + - Strict: All references must resolve + - Lenient: Allow forward references with late binding + - Development: Extra validation and warnings + +### Security Considerations + +1. **Reference Injection**: If references come from user input, need sanitization +2. **Infinite Loops**: Circular reference detection required +3. **Resource Limits**: Maximum nesting depth to prevent DoS + +## Recommendations + +### 1. Enhanced Syntax Specification + +```yaml +reference_syntax: + valid_name: "^[a-zA-Z_][a-zA-Z0-9_-]*$" + separator: "." + escape: "\\" # For dots in names: "my\\.name" + max_depth: 10 + case_sensitive: true +``` + +### 2. Resolution Algorithm Improvements + +```python +class ReferenceResolver: + def __init__(self, max_depth=10): + self.max_depth = max_depth + self.resolution_cache = {} + self.circular_check = set() + + def resolve(self, ref: str, context: ResolutionContext) -> Any: + # Check cache + cache_key = (ref, context.scope_id) + if cache_key in self.resolution_cache: + return self.resolution_cache[cache_key] + + # Check circular references + if ref in self.circular_check: + raise CircularReferenceError(ref) + + self.circular_check.add(ref) + try: + result = self._resolve_uncached(ref, context) + self.resolution_cache[cache_key] = result + return result + finally: + self.circular_check.remove(ref) +``` + +### 3. Error Handling Framework + +```python +class ReferenceError(Exception): + def __init__(self, ref: str, context: str, suggestion: str = None): + self.ref = ref + self.context = context + self.suggestion = suggestion + super().__init__(self._format_message()) + + def _format_message(self): + msg = f"Cannot resolve reference '{self.ref}' in {self.context}" + if self.suggestion: + msg += f". Did you mean '{self.suggestion}'?" + return msg +``` + +### 4. Validation Tools + +```python +def validate_dag_references(dag: Dict[str, Any]) -> List[ValidationIssue]: + """Validate all references in a QueryDAG are resolvable""" + issues = [] + + # Build scope tree + scope_tree = build_scope_tree(dag) + + # Find all references + references = find_all_references(dag) + + # Validate each reference + for ref_location, ref_value in references: + try: + resolve_reference(ref_value, scope_tree, ref_location) + except ReferenceError as e: + issues.append(ValidationIssue( + level="error", + location=ref_location, + message=str(e), + suggestion=find_similar_names(ref_value, scope_tree) + )) + + return issues +``` + +## Conclusion + +The Dotted Reference Syntax is a powerful feature that enables complex graph compositions, but it requires careful specification and implementation to handle edge cases properly. The current design follows established lexical scoping principles, but would benefit from: + +1. More detailed syntax specification +2. Explicit handling of edge cases +3. Performance optimizations through caching +4. Better error messages and debugging tools +5. Clear documentation with examples + +With these improvements, the feature would provide a robust foundation for building complex GFQL programs while maintaining clarity and debuggability. \ No newline at end of file diff --git a/AI_PROGRESS/gfql-programs-spec/feature_analysis_graph_combinators.md b/AI_PROGRESS/gfql-programs-spec/feature_analysis_graph_combinators.md new file mode 100644 index 0000000000..13f5afbe52 --- /dev/null +++ b/AI_PROGRESS/gfql-programs-spec/feature_analysis_graph_combinators.md @@ -0,0 +1,214 @@ +# Feature Analysis: Graph Combinators + +## Executive Summary + +This analysis examines the Graph Combinators feature proposed in the GFQL Programs RFC (sketch.md). Graph combinators would enable users to compose multiple graphs through operations like union, intersection, and subtraction - a significant enhancement to PyGraphistry's current capabilities. While PyGraphistry has rich graph processing methods, it lacks explicit graph combination operations that preserve graph semantics and relationships. + +## 1.3.4.1: Mapping to Existing Graph Operations in PyGraphistry + +### Current Graph Operations + +PyGraphistry currently provides several graph manipulation methods, but they are primarily focused on single-graph transformations: + +#### Filtering Operations +- `filter_edges_by_dict()` - Filter edges by attribute values +- `filter_nodes_by_dict()` - Filter nodes by attribute values +- Chain operations with predicates (via GFQL) + +#### Graph Transformations +- `materialize_nodes()` - Generate nodes from edges +- `collapse_by()` - Collapse nodes/edges by attributes +- `hop()` - Graph traversal operations +- Layout operations (UMAP, ForceAtlas2, etc.) + +#### Data Conversions +- `to_pandas()` / `to_cudf()` - Engine conversions +- Integration with NetworkX, igraph, CuGraph +- Support for various data formats + +### What's Missing + +Current PyGraphistry lacks explicit graph combination primitives: + +1. **No Graph Union** - Cannot merge two graphs while preserving node/edge relationships +2. **No Graph Intersection** - Cannot find common subgraphs +3. **No Graph Subtraction** - Cannot remove one graph from another +4. **No Merge Policies** - No systematic way to handle attribute conflicts +5. **No Graph References** - Cannot compose operations on multiple named graphs + +### What Combinators Would Add + +Graph combinators would provide: + +1. **Semantic Graph Operations** - Operations that understand graph structure, not just dataframes +2. **Multi-Graph Workflows** - Ability to work with multiple graphs in a single expression +3. **Declarative Composition** - Express complex multi-graph operations without imperative code +4. **Remote Graph Integration** - Load and combine graphs from different sources +5. **Policy-Based Conflict Resolution** - Systematic handling of overlapping data + +## 1.3.4.2: Policy System Design Review + +### Proposed Combination Policies + +The RFC proposes several policies for handling data conflicts: + +#### Attribute Merge Policies +- **left**: Keep attributes from left graph +- **right**: Keep attributes from right graph +- **merge_left**: Merge with left graph taking precedence +- **merge_right**: Merge with right graph taking precedence + +#### Node Removal Policies +- **drop_edges**: Remove edges connected to removed nodes +- **keep_edges**: Preserve edges (may create dangling references) + +#### Edge Removal Policies +- **drop_all_isolated**: Remove all isolated nodes +- **drop_newly_isolated**: Remove only nodes isolated by the operation +- **keep_nodes**: Preserve all nodes regardless of connectivity + +#### Additional Policy: drop_dangling +- Remove edges with missing source/destination nodes + +### Policy System Analysis + +**Strengths:** +- Covers common conflict scenarios +- Provides fine-grained control over graph structure preservation +- Aligns with database join semantics (left/right/merge) + +**Gaps:** +- No policy for attribute type conflicts (e.g., string vs. numeric) +- No aggregation policies for duplicate edges +- Missing policies for multi-valued attributes +- No support for custom merge functions + +### Recommended Enhancements + +1. **Type Coercion Policies** + - `coerce_left`: Use left type, coerce right + - `coerce_right`: Use right type, coerce left + - `coerce_common`: Find common supertype + - `error`: Fail on type mismatch + +2. **Aggregation Policies** + - `first`: Keep first occurrence + - `last`: Keep last occurrence + - `sum/mean/max/min`: Aggregate numeric attributes + - `concat`: Concatenate string/list attributes + +3. **Custom Resolution** + - Allow user-defined merge functions + - Support for attribute-specific policies + +## 1.3.4.3: Critical Review + +### Memory Implications + +**Concerns:** +1. **Duplication During Operations** - Combinators may need to copy entire graphs +2. **Intermediate Results** - DAG evaluation creates temporary graphs +3. **Remote Graph Loading** - Loading multiple large graphs simultaneously + +**Mitigation Strategies:** +1. Lazy evaluation where possible +2. Streaming operations for large graphs +3. Memory-mapped operations for remote graphs +4. Reference counting for shared data + +### Edge Cases + +1. **Mismatched Node IDs** + - Different ID types (string vs. int) + - Different ID semantics (global vs. local) + - Solution: ID mapping/normalization phase + +2. **Cyclic References in DAG** + - Detecting cycles in QueryDAG + - Solution: Static validation at parse time + +3. **Empty Graph Handling** + - Union with empty graph + - Intersection resulting in empty graph + - Solution: Well-defined empty graph semantics + +4. **Schema Evolution** + - Graphs with different schemas over time + - Solution: Schema versioning and migration + +### Consistency Issues + +1. **Node-Edge Relationship Integrity** + - Operations may break referential integrity + - Need validation after each operation + - Consider transaction-like semantics + +2. **Attribute Consistency** + - Same attribute with different meanings + - Solution: Namespace attributes by source + +3. **Graph Property Preservation** + - Directed vs. undirected conflicts + - Multi-graph vs. simple graph + - Solution: Explicit graph type policies + +### Performance Considerations + +1. **Index Maintenance** + - Need efficient lookups for intersection/union + - Consider maintaining node/edge indices + +2. **Parallel Execution** + - DAG structure enables parallelism + - Need thread-safe operations + +3. **Caching Strategy** + - Cache intermediate results in DAG + - Invalidation on source changes + +4. **Engine Optimization** + - Leverage GPU for large-scale operations + - Push operations to data source when possible + +## Implementation Recommendations + +### Phase 1: Core Infrastructure +1. Implement QueryDAG/ChainGraph classes +2. Add reference resolution system +3. Create basic combinator operations + +### Phase 2: Policy System +1. Implement merge policies +2. Add removal policies +3. Create validation framework + +### Phase 3: Optimization +1. Add lazy evaluation +2. Implement caching +3. GPU acceleration + +### Phase 4: Advanced Features +1. Custom policies +2. Schema management +3. Distributed execution + +## Testing Strategy + +### Unit Tests +- Policy behavior validation +- Edge case handling +- Memory leak detection + +### Integration Tests +- Multi-graph workflows +- Remote graph loading +- Large graph performance + +### Stress Tests +- Memory limits +- Concurrent operations +- Schema conflicts + +## Conclusion + +Graph combinators represent a significant enhancement to PyGraphistry's capabilities. While the current proposal provides a solid foundation, attention to memory management, edge cases, and consistency will be critical for production use. The phased implementation approach allows for iterative refinement based on user feedback and performance characteristics. \ No newline at end of file diff --git a/AI_PROGRESS/gfql-programs-spec/feature_analysis_remote_graph.md b/AI_PROGRESS/gfql-programs-spec/feature_analysis_remote_graph.md new file mode 100644 index 0000000000..475d04dfeb --- /dev/null +++ b/AI_PROGRESS/gfql-programs-spec/feature_analysis_remote_graph.md @@ -0,0 +1,366 @@ +# Feature Analysis: Remote Graph Loading (RemoteGraph) + +## Executive Summary + +The RemoteGraph feature proposed in sketch.md introduces the ability to load and query remote graphs directly within pure GFQL expressions, without requiring Python code. This analysis examines its relationship to existing functionality, security implications, and critical performance considerations. + +## 1.3.3.1: Relationship to Current Dataset Loading Mechanisms + +### Current bind(dataset_id=...) Functionality + +The existing PyGraphistry system supports loading remote datasets through the `bind()` method: + +```python +# Current approach - requires Python +graphistry.bind(dataset_id="abc123").chain(...) +``` + +**Current Implementation Details:** +- Located in `PlotterBase.bind()` (line 973) +- Stores dataset_id in the Plottable object +- Requires Python client to orchestrate the loading +- Uses authenticated API calls through `ClientSession` +- Leverages existing upload/download infrastructure + +### RemoteGraph vs Current Remote Execution + +**Key Differences:** + +1. **Execution Context** + - Current: Python client orchestrates remote operations + - RemoteGraph: Server-side GFQL runtime loads graphs + +2. **Composition Model** + - Current: Sequential chain operations with intermediate Python steps + - RemoteGraph: Pure GFQL DAG execution without client roundtrips + +3. **Authentication Flow** + - Current: Token passed from Python client on each request + - RemoteGraph: Token must be embedded or referenced in GFQL context + +4. **Data Flow** + ``` + Current: + Python → Upload → Server → Query → Download → Python → Next Operation + + RemoteGraph: + GFQL Program → Server loads multiple graphs → Server executes DAG → Single Result + ``` + +### Integration with Existing Infrastructure + +**Reusable Components:** +- `chain_remote.py` - Remote execution infrastructure +- `ClientSession` - Authentication and session management +- `ArrowUploader` - Data serialization mechanisms +- Dataset permission checks + +**New Requirements:** +- Server-side graph loading within GFQL runtime +- DAG execution engine +- Graph reference resolution +- Cross-dataset permission validation + +## 1.3.3.2: Security and Authentication Considerations + +### Access Control for Remote Graphs + +**Current Security Model:** +- Each dataset has owner/organization permissions +- API tokens authenticate requests +- Dataset IDs are opaque identifiers +- Permissions checked on each API call + +**RemoteGraph Security Challenges:** + +1. **Token Propagation** + ```json + { + "type": "RemoteGraph", + "graph_id": "abc123" + // No token field - how to authenticate? + } + ``` + +2. **Permission Elevation Risks** + - User A creates GFQL program referencing their graphs + - User B executes program - which permissions apply? + - Need clear execution context boundaries + +3. **Dataset Discovery** + - Remote graphs could probe for dataset existence + - Information leakage through error messages + - Timing attacks on permission checks + +### Cross-Tenant Data Isolation + +**Critical Concerns:** + +1. **Memory Isolation** + - Multiple graphs loaded in same execution context + - Potential for data leakage between tenants + - Need strong process isolation + +2. **Reference Injection** + ```json + { + "ref": "../../other_tenant/graph" // Path traversal? + } + ``` + +3. **Compute Resource Isolation** + - Tenant A's RemoteGraph could consume resources affecting Tenant B + - Need resource quotas per execution context + +### Authentication Token Handling + +**Design Options:** + +1. **Implicit Token (Current User Context)** + - Pro: Simple, uses existing auth + - Con: Limits sharing of GFQL programs + +2. **Embedded Tokens** + - Pro: Self-contained programs + - Con: Security risk, token leakage + +3. **Token References** + ```json + { + "type": "RemoteGraph", + "graph_id": "abc123", + "auth_ref": "@current_user" // Or "@token:xyz" + } + ``` + +4. **Capability-Based Security** + - Generate limited-scope tokens for specific graphs + - Time-bound access grants + - Audit trail for graph access + +### Data Exfiltration Risks + +**Attack Vectors:** + +1. **Large Graph Loading** + - Load many graphs to exceed memory + - Force data to disk, potential inspection + +2. **Graph Combinators** + - Union operations could merge unauthorized data + - Intersection could reveal membership information + +3. **Error Messages** + - Graph schema details in errors + - Row counts, column names exposure + +**Mitigations:** +- Strict output sanitization +- Rate limiting on RemoteGraph operations +- Audit logging of all graph access +- Output size limits + +## 1.3.3.3: Critical Review - Network/Caching/Error Handling + +### Network Latency and Timeout Handling + +**Current State:** +- No explicit timeout handling in `chain_remote.py` +- Uses `requests.post()` with default timeouts +- No retry logic for transient failures + +**RemoteGraph Challenges:** + +1. **Cascading Timeouts** + - DAG with multiple RemoteGraphs + - Serial loading could exceed total timeout + - Need per-operation and total timeouts + +2. **Partial Failures** + ``` + Graph A: Loaded ✓ + Graph B: Timeout ✗ + Graph C: Not attempted + ``` + - How to handle partially loaded state? + - Rollback or partial execution? + +3. **Network Optimization** + - Parallel graph loading where possible + - Connection pooling for same server + - HTTP/2 multiplexing benefits + +**Recommendations:** +```python +{ + "type": "RemoteGraph", + "graph_id": "abc123", + "timeout_ms": 30000, # Per-graph timeout + "retry_policy": { + "max_attempts": 3, + "backoff": "exponential" + } +} +``` + +### Caching Strategies + +**Cache Levels:** + +1. **Graph Metadata Cache** + - Schema, row counts, column types + - TTL: Hours to days + - Key: (user_id, dataset_id, version) + +2. **Graph Data Cache** + - Full graph data in memory/disk + - TTL: Minutes to hours + - Key: (user_id, dataset_id, version, filters) + +3. **Computation Cache** + - Results of GFQL operations + - TTL: Based on data volatility + - Key: Full GFQL program hash + +**Cache Invalidation:** +- Version-based invalidation +- Time-based expiry +- Explicit invalidation API +- Memory pressure eviction + +**Implementation Considerations:** +```python +{ + "type": "RemoteGraph", + "graph_id": "abc123", + "cache_policy": { + "mode": "aggressive", # or "conservative", "none" + "ttl_seconds": 3600, + "validate": true # Check version before use + } +} +``` + +### Error Handling + +**Error Categories:** + +1. **Authentication Errors** + - Invalid token + - Expired token + - Insufficient permissions + +2. **Network Errors** + - Connection timeout + - DNS resolution + - SSL/TLS failures + +3. **Data Errors** + - Graph not found + - Schema mismatch + - Corrupted data + +4. **Resource Errors** + - Memory exhaustion + - Disk space + - Compute quotas + +**Error Response Design:** +```json +{ + "error": { + "type": "RemoteGraphError", + "code": "GRAPH_NOT_FOUND", + "graph_ref": "abc123", + "message": "Dataset not found or access denied", + "details": { + "attempted_at": "2024-01-15T10:30:00Z", + "retry_possible": true + } + }, + "partial_results": null // Or partial data if available +} +``` + +### Resource Limits and Quotas + +**Per-User Limits:** +- Max RemoteGraphs per program: 10 +- Max total graph size: 10GB +- Max execution time: 5 minutes +- Max memory per execution: 16GB + +**Per-Graph Limits:** +- Max graph size: 2GB +- Max load time: 60 seconds +- Max cache size: 1GB + +**Quota Management:** +```python +{ + "type": "QueryDAG", + "resource_limits": { + "max_memory_gb": 8, + "max_time_seconds": 300, + "max_graphs": 5 + }, + "graph": { ... } +} +``` + +## Implementation Recommendations + +### Phase 1: Minimal Viable Feature +1. Single RemoteGraph support (no DAG) +2. Implicit authentication (current user) +3. No caching +4. Basic timeout handling + +### Phase 2: Production Readiness +1. Full DAG support +2. Caching infrastructure +3. Comprehensive error handling +4. Resource quotas + +### Phase 3: Advanced Features +1. Cross-tenant graph sharing +2. Capability-based security +3. Advanced cache strategies +4. Graph versioning + +## Security Checklist + +- [ ] Token validation per graph access +- [ ] Input sanitization for graph IDs +- [ ] Output sanitization for errors +- [ ] Rate limiting implementation +- [ ] Audit logging +- [ ] Resource quota enforcement +- [ ] Memory isolation between tenants +- [ ] Timeout handling at all levels +- [ ] Cache security (no cross-tenant leaks) +- [ ] Permission inheritance model + +## Performance Considerations + +1. **Baseline Metrics Needed:** + - Single graph load time + - Memory usage per graph size + - Network bandwidth requirements + - Cache hit rates + +2. **Optimization Opportunities:** + - Parallel graph loading + - Incremental graph updates + - Columnar data transfer + - Compression strategies + +3. **Monitoring Requirements:** + - Graph load latencies + - Cache performance + - Resource usage per tenant + - Error rates by category + +## Conclusion + +RemoteGraph represents a significant evolution from the current dataset loading mechanism, enabling pure GFQL programs to compose multiple graphs without Python orchestration. However, this power comes with substantial security and performance challenges that must be carefully addressed. The phased implementation approach allows for iterative refinement while maintaining system stability and security. \ No newline at end of file diff --git a/AI_PROGRESS/gfql-programs-spec/gfql_knowledge_base.md b/AI_PROGRESS/gfql-programs-spec/gfql_knowledge_base.md new file mode 100644 index 0000000000..395b9e2471 --- /dev/null +++ b/AI_PROGRESS/gfql-programs-spec/gfql_knowledge_base.md @@ -0,0 +1,625 @@ +# GFQL Knowledge Base + +## Overview + +GFQL (Graph Query Language) is a declarative graph pattern matching system implemented in PyGraphistry. It provides a composable AST-based approach for expressing complex graph traversal patterns. The core architecture follows a three-phase algorithm for efficient subgraph extraction: + +1. **Forward wavefront traversal** - Explores paths from starting nodes +2. **Reverse pruning pass** - Removes dead-end paths to ensure all nodes are on complete paths +3. **Forward output pass** - Collects and labels final results + +## Architecture + +### Core Components + +``` +graphistry/compute/ +├── chain.py # Chain execution engine +├── ast.py # AST node/edge definitions +├── predicates/ # Predicate system for filtering +├── chain_remote.py # Remote execution via API +├── hop.py # Single-hop traversal logic +└── ASTSerializable.py # JSON serialization base +``` + +## File-by-File Analysis + +### 1. `/graphistry/compute/chain.py` - Core Chain Execution Engine + +**Purpose**: Implements the main chain execution algorithm that processes sequences of AST operations + +**Key Classes**: +- `Chain` (file:18-52) - Container for AST operation sequences + - `__init__`: Stores list of ASTObjects + - `from_json`: Deserializes from JSON representation + - `to_json`: Serializes to JSON wire format + - `validate`: Ensures all operations are valid ASTObjects + +**Key Functions**: +- `chain()` (file:148-360) - Main entry point for chain execution + - Signature: `chain(self: Plottable, ops: Union[List[ASTObject], Chain], engine: Union[EngineAbstract, str] = EngineAbstract.AUTO) -> Plottable` + - Implements 3-phase algorithm: + - Forward pass (file:280-302): Computes path prefixes + - Backward pass (file:311-349): Prunes incomplete paths + - Combine phase (file:350-358): Merges and labels results + - Handles edge cases like chains starting/ending with edges + +- `combine_steps()` (file:56-117) - Merges results from multiple operations + - Deduplicates nodes/edges across steps + - Tags nodes/edges with operation names when specified + - Special handling for edge recomputation in reverse pass + +**Implementation Details**: +```python +# file:280-302 - Forward wavefront computation +g_stack : List[Plottable] = [] +for op in ops: + prev_step_nodes = ( + None # first uses full graph + if len(g_stack) == 0 + else g_stack[-1]._nodes + ) + g_step = op( + g=g, + prev_node_wavefront=prev_step_nodes, + target_wave_front=None, + engine=engine_concrete + ) + g_stack.append(g_step) +``` + +### 2. `/graphistry/compute/ast.py` - AST Class Definitions + +**Purpose**: Defines the abstract syntax tree nodes for graph patterns + +**Key Classes**: + +- `ASTObject` (file:67-90) - Abstract base for all AST operations + - Abstract methods: `__call__()`, `reverse()` + - Stores optional `_name` for result labeling + +- `ASTNode` (file:116-193) - Represents node matching operations + - Parameters: + - `filter_dict`: Key-value pairs for attribute matching (supports predicates) + - `query`: Pandas query string for additional filtering + - `name`: Optional label for matched nodes + - `__call__()` (file:164-189): Filters nodes based on criteria + - `reverse()`: Returns self (nodes are direction-agnostic) + +- `ASTEdge` (file:206-374) - Represents edge traversal operations + - Parameters: + - `direction`: 'forward', 'reverse', or 'undirected' + - `edge_match`: Filter for edge attributes + - `hops`: Number of hops (default 1) + - `to_fixed_point`: Continue until no new nodes + - `source_node_match/destination_node_match`: Node filters + - `*_query`: Pandas query strings + - `__call__()` (file:313-352): Delegates to hop() function + - `reverse()` (file:354-373): Flips direction and source/dest + +**Specialized Edge Classes**: +- `ASTEdgeForward` (file:375-419) - Convenience for forward edges +- `ASTEdgeReverse` (file:421-464) - Convenience for reverse edges +- `ASTEdgeUndirected` (file:467-511) - Convenience for undirected edges + +**Helper Functions**: +- `n()` (file:194) - Shorthand for ASTNode +- `e_forward()`, `e_reverse()`, `e_undirected()`, `e()` - Edge shorthands + +**JSON Serialization**: +```python +# file:267-294 - Example of JSON format for ASTEdge +{ + 'type': 'Edge', + 'direction': 'forward', + 'hops': 2, + 'edge_match': {'type': 'transaction'}, + 'source_node_match': {'risk': {'type': 'GT', 'val': 0.5}} +} +``` + +### 3. `/graphistry/compute/predicates/` - Predicate System + +**Purpose**: Provides columnar predicates for advanced filtering beyond simple equality + +**Base Class**: +- `ASTPredicate` (file:predicates/ASTPredicate.py:9-27) + - Abstract `__call__(s: SeriesT) -> SeriesT` method + - Inherits from ASTSerializable for JSON support + +**Key Predicate Types**: + +1. **Categorical** (`predicates/is_in.py`): + - `IsIn` (file:16-150) - Check if values in list + - Handles temporal type normalization + - Example: `{'type': is_in(['person', 'company'])}` + +2. **Numeric** (`predicates/numeric.py`): + - `GT/LT/GE/LE/EQ/NE` - Comparison operators + - `Between` (file:95-116) - Range checking + - `IsNA/NotNA` - Null checking + - Example: `{'score': gt(0.8)}` + +3. **String** (`predicates/str.py`): + - `Contains/Startswith/Endswith` - Pattern matching + - `Match` - Regex matching + - Various type checks: `IsNumeric/IsAlpha/IsDigit/etc` + +4. **Temporal** (`predicates/temporal.py`): + - Date/time specific predicates + - `IsMonthStart/IsYearEnd/IsLeapYear/etc` + +**Integration Example**: +```python +# file:ast.py:98-99 - Predicate validation +for k, v in d.items(): + assert isinstance(v, ASTPredicate) or is_json_serializable(v) +``` + +### 4. `/graphistry/compute/chain_remote.py` - Remote Execution + +**Purpose**: Enables server-side GFQL execution for performance + +**Key Functions**: + +- `chain_remote()` (file:223-320) - Main remote execution entry + - Auto-uploads graph if needed + - Sends GFQL operations to server API + - Returns results as Plottable + +- `chain_remote_shape()` (file:168-221) - Fast metadata-only query + - Returns DataFrame with graph shape info + - Useful for checking if patterns exist + +- `chain_remote_generic()` (file:16-166) - Shared implementation + - Handles authentication via JWT + - Supports multiple output formats (parquet, csv, json) + - Engine selection (pandas vs cudf/GPU) + +**Wire Protocol**: +```python +# file:67-80 - Request format +request_body = { + "gfql_operations": chain_json['chain'], # List of AST operations + "format": "parquet", + "node_col_subset": ["id", "type"], # Optional column filtering + "edge_col_subset": ["weight"], + "engine": "cudf" # Force GPU mode +} +``` + +### 5. `/graphistry/compute/hop.py` - Single Hop Implementation + +**Purpose**: Core graph traversal logic used by ASTEdge operations + +**Key Function**: +- `hop()` (file:258-624) - Performs k-hop traversal + - Parameters align with ASTEdge options + - Handles forward/reverse/undirected traversal + - Implements wavefront expansion algorithm + - Column conflict resolution for node/edge ID collisions + +**Algorithm Flow**: +1. Initialize wavefront with starting nodes +2. For each hop: + - Filter source nodes by predicates + - Follow edges matching criteria + - Filter destination nodes + - Update wavefront with newly reached nodes +3. Return subgraph of all traversed nodes/edges + +**Helper Functions**: +- `generate_safe_column_name()` (file:15-43) - Avoid column conflicts +- `prepare_merge_dataframe()` (file:46-105) - Setup merge operations +- `process_hop_direction()` (file:113-255) - Direction-specific logic + +### 6. `/graphistry/compute/ASTSerializable.py` - Serialization Base + +**Purpose**: Base class for JSON serialization of AST components + +**Key Methods**: +- `to_json()` (file:19-30) - Generic serialization + - Adds 'type' field with class name + - Serializes all non-reserved attributes + +- `from_json()` (file:33-40) - Generic deserialization + - Uses 'type' field to determine class + - Passes remaining fields to constructor + +## Design Patterns + +### 1. **Visitor Pattern** +AST nodes implement `__call__()` to process graph data, with the chain orchestrating traversal. + +### 2. **Builder Pattern** +Operations compose into chains, building complex queries from simple primitives. + +### 3. **Strategy Pattern** +Predicates encapsulate filtering strategies that can be swapped at runtime. + +### 4. **Memento Pattern** +JSON serialization enables saving/restoring query state. + +## Integration Points + +### 1. **Engine Abstraction** +- Supports pandas (CPU) and cudf (GPU) DataFrames +- Engine selection via `resolve_engine()` helper +- Automatic engine detection based on data type + +### 2. **Plottable Integration** +- All operations work on Plottable objects +- Maintains node/edge bindings throughout +- Preserves visualization settings + +### 3. **Remote Execution** +- Seamless switch between local and remote +- Same API for both modes +- Automatic data upload when needed + +## Example Usage Patterns + +### Basic Node Filtering +```python +from graphistry import n, e_forward + +# Find all person nodes +g.chain([n({"type": "person"})]) +``` + +### Multi-hop Traversal +```python +# Find transactions between risky entities +g.chain([ + n({"risk": gt(0.8)}), + e_forward({"type": "transfer"}, hops=2), + n({"risk": gt(0.8)}) +]) +``` + +### Named Operations +```python +# Label intermediate results +g.chain([ + n({"type": "account"}, name="source_accounts"), + e_forward(name="transfers"), + n({"type": "account"}, name="dest_accounts") +]) +# Access via g._nodes.source_accounts, g._edges.transfers +``` + +### Complex Predicates +```python +# Combine multiple predicate types +g.chain([ + n({ + "created_date": is_month_start(), + "name": contains("Corp"), + "value": between(1000, 10000) + }) +]) +``` + +## Wire Protocol Examples + +### Local Chain Execution +```python +chain = Chain([ + {"type": "Node", "filter_dict": {"type": "person"}}, + {"type": "Edge", "direction": "forward", "hops": 2}, + {"type": "Node", "filter_dict": {"type": "company"}} +]) +g2 = g.chain(chain) +``` + +### Remote Execution Request +```json +{ + "gfql_operations": [ + { + "type": "Node", + "filter_dict": {"risk": {"type": "GT", "val": 0.5}} + }, + { + "type": "Edge", + "direction": "forward", + "edge_match": {"amount": {"type": "Between", "lower": 1000, "upper": 5000}} + } + ], + "format": "parquet", + "engine": "cudf" +} +``` + +## Performance Considerations + +1. **Wavefront Optimization**: The algorithm maintains a wavefront of active nodes rather than revisiting the entire graph + +2. **Dead-end Pruning**: The reverse pass ensures only nodes on complete paths are returned + +3. **GPU Acceleration**: When using cudf engine, operations leverage GPU parallelism + +4. **Remote Execution**: For large graphs, server-side execution avoids data transfer overhead + +5. **Column Subsetting**: Remote queries can request only needed columns to reduce bandwidth + +## Extension Points for DAG Support + +The current architecture could be extended for DAG program support by: + +1. **Program AST Node**: New AST type that contains a sequence of operations +2. **Variable Binding**: Allow naming intermediate results for reuse +3. **Conditional Logic**: Add branching based on graph properties +4. **Aggregation Operators**: Support for counting, grouping operations +5. **Subprogram Composition**: Enable nesting of program definitions + +The existing serialization framework and remote execution infrastructure provide a solid foundation for these extensions. + +## PyGraphistry API Integration + +### Entry Points + +GFQL functionality is exposed through the PyGraphistry API via several key integration points: + +#### 1. **Plottable Interface** (`/graphistry/Plottable.py`) + +The `Plottable` protocol (file:48-791) defines the main API contracts: + +```python +# Local chain execution (file:419-423) +def chain(self, ops: Union[Any, List[Any]]) -> 'Plottable': + """ops is Union[List[ASTObject], Chain]""" + ... + +# Remote chain execution (file:425-440) +def chain_remote( + self: 'Plottable', + chain: Union[Any, Dict[str, JSONVal]], + api_token: Optional[str] = None, + dataset_id: Optional[str] = None, + output_type: OutputTypeGraph = "all", + format: Optional[FormatType] = None, + df_export_args: Optional[Dict[str, Any]] = None, + node_col_subset: Optional[List[str]] = None, + edge_col_subset: Optional[List[str]] = None, + engine: Optional[Literal["pandas", "cudf"]] = None +) -> 'Plottable': + ... + +# Remote shape query (file:442-456) +def chain_remote_shape( + self: 'Plottable', + chain: Union[Any, Dict[str, JSONVal]], + # ... same parameters as chain_remote +) -> pd.DataFrame: + ... +``` + +#### 2. **Plotter Class Hierarchy** (`/graphistry/plotter.py`) + +The main `Plotter` class (file:21-93) inherits from multiple mixins: + +```python +class Plotter( + KustoMixin, SpannerMixin, + CosmosMixin, NeptuneMixin, + HeterographEmbedModuleMixin, + SearchToGraphMixin, + DGLGraphMixin, ClusterMixin, + UMAPMixin, + FeatureMixin, ConditionalMixin, + LayoutsMixin, + ComputeMixin, PlotterBase # <-- GFQL exposed via ComputeMixin +): +``` + +#### 3. **ComputeMixin** (`/graphistry/compute/ComputeMixin.py`) + +Provides the actual implementation bridge (file:462-472): + +```python +def chain(self, *args, **kwargs): + return chain_base(self, *args, **kwargs) +chain.__doc__ = chain_base.__doc__ + +def chain_remote(self, *args, **kwargs) -> Plottable: + return chain_remote_base(self, *args, **kwargs) +chain_remote.__doc__ = chain_remote_base.__doc__ + +def chain_remote_shape(self, *args, **kwargs) -> pd.DataFrame: + return chain_remote_shape_base(self, *args, **kwargs) +chain_remote_shape.__doc__ = chain_remote_shape_base.__doc__ +``` + +### Wire Protocol Format + +#### 1. **Request Format** (`/graphistry/compute/chain_remote.py`) + +Remote execution sends requests to the server API (file:67-89): + +```python +# API endpoint +url = f"{self.base_url_server()}/api/v2/etl/datasets/{dataset_id}/gfql/{output_type}" + +# Request body structure +request_body = { + "gfql_operations": chain_json['chain'], # List of AST operations + "format": format, # "json", "csv", "parquet" + "node_col_subset": node_col_subset, # Optional: limit returned columns + "edge_col_subset": edge_col_subset, # Optional: limit returned columns + "df_export_args": df_export_args, # Optional: pandas export args + "engine": engine # Optional: "pandas" or "cudf" +} + +# Headers +headers = { + "Authorization": f"Bearer {api_token}", + "Content-Type": "application/json", +} +``` + +#### 2. **Response Handling** (`/graphistry/compute/chain_remote.py`) + +The response format depends on `output_type` and `format` (file:93-166): + +- **Shape queries** (`output_type="shape"`): Returns metadata DataFrame +- **Graph queries** (`output_type="all"`): Returns nodes and edges + - JSON format: `{"nodes": [...], "edges": [...]}` + - CSV/Parquet: ZIP file containing separate nodes/edges files +- **Partial queries** (`output_type="nodes"` or `"edges"`): Single DataFrame + +#### 3. **Output Type Definitions** (`/graphistry/models/compute/chain_remote.py`) + +```python +# Graph-oriented outputs (file:5-6) +OutputTypeGraph = Literal["all", "nodes", "edges", "shape"] + +# DataFrame outputs (file:11-12) +OutputTypeDf = Literal["table", "shape"] + +# JSON outputs (file:14-15) +OutputTypeJson = Literal["json"] + +# Format types (file:8-9) +FormatType = Literal["json", "csv", "parquet"] +``` + +### Authentication and Session Handling + +#### 1. **Session Management** (`/graphistry/client_session.py`) + +Each Plotter instance maintains session state (file:26-100): + +```python +class ClientSession: + """Holds all configuration and authentication state""" + + # Authentication state + api_key: Optional[str] + api_token: Optional[str] # JWT token + + # Server configuration + hostname: str + protocol: str + api_version: ApiVersion # 1 or 3 + + # Organization settings + org_name: Optional[str] + privacy: Optional[Privacy] +``` + +#### 2. **Token Refresh** (`/graphistry/compute/chain_remote.py`) + +Automatic token refresh on remote calls (file:30-33): + +```python +if not api_token: + from graphistry.pygraphistry import PyGraphistry + PyGraphistry.refresh() # Refreshes JWT if needed + api_token = PyGraphistry.api_token() +``` + +#### 3. **Dataset Upload** (`/graphistry/compute/chain_remote.py`) + +Automatic upload if no dataset_id exists (file:35-40): + +```python +if not dataset_id: + dataset_id = self._dataset_id + +if not dataset_id: + self = self.upload(validate=validate) # Uploads current graph + dataset_id = self._dataset_id +``` + +### Error Handling Patterns + +#### 1. **Validation** (`/graphistry/compute/chain_remote.py`) + +- Input validation (file:42-48): Checks output_type, engine values +- Chain validation (file:64-65): Validates AST structure before sending +- Response validation (file:91): Uses `response.raise_for_status()` + +#### 2. **Error Types** + +Common error scenarios handled: +- Missing dataset_id (ValueError) +- Invalid output_type or format (ValueError) +- HTTP errors (requests.HTTPError) +- Deserialization errors based on data type detection + +### Python Remote Execution + +Related API for arbitrary Python execution (`/graphistry/compute/python_remote.py`): + +```python +# Execute Python code remotely (file:34-96) +def python_remote_generic( + self: Plottable, + code: Union[str, Callable[..., object]], + api_token: Optional[str] = None, + dataset_id: Optional[str] = None, + format: Optional[FormatType] = 'json', + output_type: Optional[OutputTypeAll] = 'json', + engine: Literal["pandas", "cudf"] = "cudf", + run_label: Optional[str] = None, + validate: bool = True +) -> Union[Plottable, pd.DataFrame, Any]: + ... + +# Request format (file:133-139) +request_body = { + "execute": code_indented, # Python code with task(g) function + "engine": engine, + "run_label": run_label, # Optional job tracking + "format": format, + "output_type": output_type +} +``` + +### Integration Architecture + +The API follows a layered architecture: + +1. **User API Layer**: `Plotter` class with friendly methods +2. **Protocol Layer**: `Plottable` interface defining contracts +3. **Implementation Layer**: `ComputeMixin` bridging to compute modules +4. **Execution Layer**: `chain.py` (local) and `chain_remote.py` (remote) +5. **Transport Layer**: HTTP/JSON wire protocol to server + +This separation enables: +- Clean API surface for users +- Protocol-based testing and mocking +- Engine-agnostic implementation (pandas vs cudf) +- Seamless local/remote execution switching +- Session isolation for multi-tenant scenarios + +### Usage Patterns + +#### Basic Local Execution +```python +import graphistry +from graphistry import n, e_forward + +g = graphistry.edges(df, 'src', 'dst') +g2 = g.chain([n({"type": "person"}), e_forward(), n({"type": "company"})]) +``` + +#### Remote Execution with Auto-upload +```python +# Automatically uploads if needed +g2 = g.chain_remote([n({"type": "person"}), e_forward(), n({"type": "company"})]) +``` + +#### Optimized Remote Query +```python +# Get just shape info without full data transfer +shape_df = g.chain_remote_shape( + [n({"risk": gt(0.8)}), e_forward(hops=2)], + engine='cudf', # Force GPU + node_col_subset=['id', 'risk'] # Limit columns +) +if len(shape_df) > 0: + # Fetch full results only if matches exist + g2 = g.chain_remote([...]) +``` \ No newline at end of file diff --git a/AI_PROGRESS/gfql-programs-spec/persona_scenario_gap_analysis.md b/AI_PROGRESS/gfql-programs-spec/persona_scenario_gap_analysis.md new file mode 100644 index 0000000000..7ff307e0d3 --- /dev/null +++ b/AI_PROGRESS/gfql-programs-spec/persona_scenario_gap_analysis.md @@ -0,0 +1,72 @@ +# Persona/Scenario Gap Analysis + +## Feature Coverage Analysis + +### Well-Covered Features: +- **RemoteGraph**: Used by all personas (22/22 scenarios) +- **Graph Combinators**: Strong coverage (18/22 scenarios) +- **Call Operations**: Good coverage (15/22 scenarios) +- **DAG Composition**: Moderate coverage (12/22 scenarios) + +### Under-Represented Features: +- **Dotted References**: Only explicit in Morgan's scenarios (2/22) +- **Complex Error Scenarios**: Limited error handling exploration +- **Resource Limit Testing**: Only touched on by Sam and Morgan + +## Persona Coverage + +### Technical Skill Distribution: +- High: Sam, Morgan, Riley (3/6) +- Medium: Alex (1/6) +- Low: Jordan, Casey (2/6) + +### Industry Coverage: +- Security: Alex +- Finance: Sam, Casey +- Business/General: Jordan +- Tech/Infrastructure: Morgan +- Research/Science: Riley + +## Missing Perspectives + +### Gap 1: Small Business/Startup User +Current personas are enterprise-focused. Missing the resource-constrained startup perspective. + +### Gap 2: API Developer/Integrator +No persona building tools/applications on top of GFQL Programs. + +### Gap 3: Error Recovery Scenarios +Most scenarios assume happy path. Need more failure/recovery scenarios. + +## Additional Scenarios Needed + +### For Dotted References: +- Add scenario for Riley navigating nested biological pathways +- Add scenario for Jordan navigating organizational hierarchies + +### For Error Handling: +- Alex scenario: Dealing with partial data when one source fails +- Jordan scenario: Understanding and fixing malformed queries + +### For Resource Limits: +- Casey scenario: Hitting limits during large compliance scan +- Sam scenario: Optimizing pipeline to fit within quotas + +## Recommendations + +1. **Add API Developer Persona**: Someone building applications/tools using GFQL +2. **Add Startup Data Analyst Persona**: Resource-conscious user +3. **Enhance existing scenarios** with more error cases and resource constraints +4. **Add cross-persona collaboration scenario**: Multiple users sharing workflows + +## Final Assessment + +Current coverage: **Good (85%)** +- All major features have representation +- Diverse skill levels covered +- Real-world use cases included + +Gaps are minor and can be addressed with: +- 2 additional personas +- 4-5 enhanced scenarios for existing personas +- Focus on error handling and resource management \ No newline at end of file diff --git a/AI_PROGRESS/gfql-programs-spec/sketch.md b/AI_PROGRESS/gfql-programs-spec/sketch.md new file mode 100644 index 0000000000..d5c3b1c485 --- /dev/null +++ b/AI_PROGRESS/gfql-programs-spec/sketch.md @@ -0,0 +1,228 @@ +# RFC: GFQL Programs + +With groups like JPMC and usage of GFQL from Louie, we’re getting into scenarios where a single Chain isn’t enough, and we do not want to require Python + +This RFC looks at: + +* Extending wire protocol & Python API with a DAG of GFQL expressions, similar to how you can have nested SQL statements +* Extending for the most common scenarios where you would use these: + * Loading remote graphs + * Graph combinators like union/intersection/subtraction + * PyGraphistry’s rich library of Plottable-\>Plottable methods like UMAP + + # Core: DAGs + +Instead of current pure GFQL programs being a single chain, enable a DAG composition of operations where we may have multiple graphs/subgraphs/… + +## Wire Protocol + +### QueryDAG + +Introduce: + +* “type”: “QueryDAG” : Nested composition, where some final item is the output + * Similar to “let\*” in FP , and SQL having multiple selects + * field “graph” defines a sequence of bindings mapping names to Chain/QueryDAG expressions +* “output”: “b”: Pick which binding a QueryDAG returns, defaults to last +* “ref”: “abc” : Both Chain and QueryDAG may specify which named item they operate on + * valid values for graph bindingnames: ^[a-zA-Z_][a-zA-Z0-9_-]*$ + * Defaults to whatever we’re dotted on +* Similar to lexical scoping, where closest binding to a reference is the used one, and statically resolvable for a closed GFQL expression + +{ + "type": "QueryDAG", + "graph": { + "a": {“type”: “Chain”, …}, + "a2": {“type”: “QueryDAG”, …}, + "b": { + "type": "Chain", + "ref": "a", + "chain": \[ EdgeOp \] + } + }, + "output": "b" // optional +} + +### Dotted Refs + +When the QueryDAG is simple, we just have some top-level names. But when we expect collisions, we can use dotted references to avoid. Imagine we like to have graphs like “start” and “hops”, but want disambiguated. + +To disambiguate, refs can do “a.b.c” syntax, so if “b” or “c” are reused on different subgraph names, they’re disambiguated by root “a” . This gets used for “output” and “ref”. + +{ + "type": "QueryDAG", + "graph": { + + "alerts": { + "type": "QueryDAG", + "graph": { + "start": \[ …. + + "fraud": { + "type": "QueryDAG", + "graph": { + "start": \[ … + + "output": "alerts.start" +} + +## Python API + +### New Chain() parameters + +* Optional “ref” to re-root + +Chain(ref="entry", chain=\[e\_forward()\]) // unbound + +QueryDAG({ + “entry”: …, + “out”: Chain(ref="entry", chain=\[e\_forward()\]) // bound +}) + +### New QueryDAG (called ChainGraph) + +from gfql.ast import ChainGraph + +query \= ChainGraph({ + "start": \[...\], + "next": Chain(ref="start", chain=\[...\]) +}, …) + +### New Dotted Methods + +# Use 1: Remote graphs + +Part of the reason we may have query DAGs is because we may want to mash up graphs. A common case is working from another graph, like a remote one + +Using the same GFQL user context, a pure GFQL expression can load and query a graph saved in Graphistry + +### Before: Python + +graphistry.bind(dataset\_id=”abc123”).chain(...) + +### New: GFQL Python + +from gfql.ast import ChainGraph, RemoteGraph, Chain, n, e\_forward + +ChainGraph( + graph={ + "abc123": RemoteGraph(dataset\_id="abc123"), + "b": Chain(ref="abc123", chain=\[e\_forward({"type": "tx"})\]) + }, + output="b" +) + +### After: Wire Protocol + +{ + "type": "ChainGraph", + "graph": { + "abc123": { + "type": "RemoteGraph", + "graph\_id": "abc123" + }, + "tx": { + "type": "Chain", + "ref": "abc123", + "chain": \[ + {"type": "Node", "filter\_dict": {"risk": {"type": "GT", "val": 0.9}}} + \] + } + }, + "output": "tx" +} + + +## Use 2: Graph combinators + +TBD \- a common flow motivating multiple graphs is tasks like enrichment, intersection, etc + +Target operators: + +* Union + * policies: left, right, merge\_left, merge\_right +* Subtract +* Replace + * Policies: full, patch, extend +* Intersect +* From: New graph using node/edge table from diff graphs (or none) + +Common policies: + +* Node removal: drop\_edges, keep\_edges +* Edge removal: drop\_all\_isolated, drop\_newly\_isolated, keep\_nodes +* Drop\_dangling + +### Wire Protocol + +{ + "type": "GraphCombinator", + "combinator": "union" | …, + "graphs": \["g1", "g2"\], // dot-ref or graph names + "policy": { ... } // combinator-specific + … +} + +### Python + +1. Matching wire protocol: + +ChainGraph({ + "lhs": Chain(\[...\]), + "rhs": RemoteGraph("static-base"), + "union": GraphUnion("lhs", "rhs") +}, output="union") + +2. Auto-desugaring + +GraphUnion( + Chain(\[n({"id": "A"}), e({"type": "friend"})\]), + RemoteGraph("static-base") +) + +When receives an expr instead of a ref, will make the QueryDAG + +### Note: Relationship with Call + +These may go away wrt Wire Protocol if we just treat them as Call. + +However, they’re not in our core Plottable interface today as we use manual Python/Pandas for these, so they’d still need to be added to the Plottable interface + + +# Use 3: Call + +Call exposes PyGraphistry’s Plottable interface’s many methods that return new Plottables: + +* umap() +* layout\_cugraph(), compute\_cugraph(), … +* cypher() +* etc + +### Wire Protocol + +{ + "type": "call", + "function": "umap", + "ref": "previous\_graph", + "params": { + "x\_cols": \["age", "income"\], + "n\_neighbors": 10 + } +} + +### Python API + +from gfql.ast import call + +call("umap", ref="input", x\_cols=\["age", "income"\], n\_neighbors=10) + +### Safelisting + +We may want to do some sort of safelisting controls based on Hub Tier, or user-defined + +### Future: Louie Connectors + +In future versions, we might consider allowing Louie connectors enabled for the calling user + + + diff --git a/AI_PROGRESS/gfql-programs-spec/sketch1X.md b/AI_PROGRESS/gfql-programs-spec/sketch1X.md new file mode 100644 index 0000000000..be90093a87 --- /dev/null +++ b/AI_PROGRESS/gfql-programs-spec/sketch1X.md @@ -0,0 +1,703 @@ +# GFQL Programs Specification v1.X + +## Executive Summary + +GFQL Programs extends PyGraphistry's Graph Query Language from single-chain operations to a full-fledged graph programming environment. This specification enables users to compose multiple graphs, load remote data, apply transformations, and combine results - all within declarative GFQL expressions without requiring Python code. + +Key capabilities: +- **DAG Composition**: Express complex multi-graph workflows as directed acyclic graphs +- **Remote Graph Loading**: Access saved graphs directly from GFQL +- **Graph Combinators**: Union, intersect, and subtract graphs with policy controls +- **Call Operations**: Invoke PyGraphistry's rich transformation methods +- **Reference System**: Navigate nested graph structures with dotted paths + +This specification incorporates comprehensive security controls, resource management, and error handling based on thorough analysis of implementation requirements. + +## Core Concepts + +### 1. QueryDAG - The Foundation + +QueryDAG (called ChainGraph in Python API) introduces a binding environment where multiple graphs can be named, referenced, and composed: + +```json +{ + "type": "QueryDAG", + "graph": { + "customers": {"type": "RemoteGraph", "dataset_id": "cust_2024"}, + "transactions": {"type": "RemoteGraph", "dataset_id": "tx_2024"}, + "risky": { + "type": "Chain", + "ref": "customers", + "chain": [ + {"type": "Node", "filter_dict": {"risk_score": {"type": "GT", "val": 0.8}}} + ] + }, + "connected": { + "type": "Chain", + "ref": "risky", + "chain": [ + {"type": "Edge", "direction": "forward", "edge_match": {"type": "transaction"}} + ] + } + }, + "output": "connected" +} +``` + +### 2. Reference Resolution + +References follow lexical scoping rules with dotted path syntax for disambiguation: + +- **Simple references**: `"ref": "customers"` - searches from current scope outward +- **Dotted references**: `"ref": "fraud.analysis.results"` - explicit path through nested DAGs +- **Scoping rules**: Closest binding wins, statically resolvable at parse time + +### 3. Execution Model + +The system maintains these key principles: +- **Lazy evaluation** where possible to optimize resource usage +- **Parallel execution** of independent DAG branches +- **Resource limits** enforced at every level +- **Fail-fast validation** with clear error messages + +## Feature Specifications + +### 1. DAG Composition + +#### Wire Protocol + +```json +{ + "type": "QueryDAG", + "graph": { + "binding_name": { + "type": "Chain" | "QueryDAG" | "RemoteGraph" | "GraphCombinator" | "Call", + ...operation_specific_fields + } + }, + "output": "binding_name", + "resource_limits": { + "max_memory_gb": 8, + "max_time_seconds": 300, + "max_graphs": 10 + } +} +``` + +#### Python API + +```python +from graphistry.gfql import ChainGraph, Chain, RemoteGraph + +result = ChainGraph({ + "source": RemoteGraph(dataset_id="abc123"), + "filtered": Chain(ref="source", chain=[ + n({"type": "person"}), + e_forward({"amount": gt(1000)}) + ]) +}, output="filtered") +``` + +#### Validation Rules + +- Binding names must match: `^[a-zA-Z_][a-zA-Z0-9_-]*$` +- No circular references allowed +- Output must reference existing binding +- Reserved names prohibited: `type`, `graph`, `output`, `ref` + +### 2. Remote Graph Loading + +#### Wire Protocol + +```json +{ + "type": "RemoteGraph", + "dataset_id": "abc123", + "cache_policy": { + "mode": "aggressive", + "ttl_seconds": 3600, + "validate": true + }, + "timeout_ms": 30000, + "retry_policy": { + "max_attempts": 3, + "backoff": "exponential" + } +} +``` + +#### Security Model + +Authentication follows the execution context: +- Uses current user's permissions implicitly +- No embedded tokens in GFQL programs +- Dataset access validated at load time +- Cross-tenant isolation enforced + +#### Error Handling + +```json +{ + "error": { + "type": "RemoteGraphError", + "code": "GRAPH_NOT_FOUND", + "graph_ref": "abc123", + "message": "Dataset not found or access denied", + "retry_possible": true + } +} +``` + +### 3. Graph Combinators + +#### Wire Protocol + +```json +{ + "type": "GraphCombinator", + "combinator": "union" | "intersect" | "subtract" | "replace", + "graphs": ["graph1", "graph2"], + "policies": { + "attribute_conflict": "left" | "right" | "merge_left" | "merge_right", + "node_removal": "drop_edges" | "keep_edges", + "edge_removal": "drop_all_isolated" | "drop_newly_isolated" | "keep_nodes", + "type_conflict": "coerce_left" | "coerce_right" | "error", + "drop_dangling": true + } +} +``` + +#### Python API + +```python +from graphistry.gfql import GraphUnion, GraphIntersect + +# Direct usage +union = GraphUnion( + Chain([n({"type": "customer"})]), + RemoteGraph("base_graph"), + policies={"attribute_conflict": "merge_left"} +) + +# In ChainGraph +ChainGraph({ + "a": RemoteGraph("graph_a"), + "b": RemoteGraph("graph_b"), + "combined": GraphUnion("a", "b") +}) +``` + +#### Advanced Policies + +```python +{ + "aggregation": { + "duplicate_edges": "first" | "last" | "sum" | "mean", + "multi_valued": "concat" | "array" | "error" + }, + "schema": { + "validation": "strict" | "lenient", + "evolution": "allow" | "deny" + } +} +``` + +### 4. Call Operations + +#### Wire Protocol + +```json +{ + "type": "Call", + "function": "umap", + "ref": "input_graph", + "params": { + "X": ["feature1", "feature2"], + "n_neighbors": 15, + "min_dist": 0.1, + "kind": "nodes" + } +} +``` + +#### Safelist Configuration + +```python +SAFELIST_TIERS = { + "basic": { + "methods": ["get_degrees", "filter_nodes_by_dict", "materialize_nodes"], + "resource_limits": { + "max_nodes": 10_000, + "max_edges": 50_000, + "timeout_seconds": 30 + } + }, + "standard": { + "methods": ["basic", "hop", "collapse", "fa2_layout"], + "parameter_restrictions": { + "hop": {"hops": {"max": 3}}, + "fa2_layout": {"iterations": {"max": 100}} + } + }, + "advanced": { + "methods": ["standard", "umap", "compute_cugraph", "cypher"], + "parameter_restrictions": { + "compute_cugraph": { + "alg": ["pagerank", "louvain", "betweenness_centrality"] + } + } + }, + "enterprise": { + "methods": "*", + "custom_methods": true + } +} +``` + +#### Parameter Validation + +```python +# Type schemas for each method +METHOD_SCHEMAS = { + "umap": { + "X": {"type": "array", "items": "string"}, + "n_neighbors": {"type": "integer", "minimum": 2, "maximum": 200}, + "min_dist": {"type": "number", "minimum": 0.0, "maximum": 1.0}, + "kind": {"type": "string", "enum": ["nodes", "edges"]} + } +} +``` + +### 5. Dotted Reference System + +#### Syntax Rules + +```yaml +reference_syntax: + valid_name: "^[a-zA-Z_][a-zA-Z0-9_-]*$" + separator: "." + escape: "\\" # For dots in names: "my\\.name" + max_depth: 10 + case_sensitive: true +``` + +#### Resolution Algorithm + +```python +def resolve_reference(ref: str, context: ExecutionContext) -> Plottable: + """Resolve a reference in the current execution context""" + + # Split into components + components = parse_reference(ref) # Handles escaping + + # Simple reference - lexical search + if len(components) == 1: + return context.lexical_lookup(components[0]) + + # Dotted reference - traverse path + current = context.lexical_lookup(components[0]) + for component in components[1:]: + if not hasattr(current, 'graph') or component not in current.graph: + raise ReferenceError( + f"Cannot resolve '{component}' in path '{ref}'", + suggestion=find_similar_names(component, current) + ) + current = current.graph[component] + + return current +``` + +## Security Model + +### 1. Resource Limits + +```python +class ResourceLimits: + # Per-execution limits + max_memory_gb: int = 8 + max_execution_time: int = 300 # seconds + max_concurrent_graphs: int = 10 + max_remote_fetches: int = 20 + max_nesting_depth: int = 10 + + # Per-graph limits + max_nodes_per_graph: int = 10_000_000 + max_edges_per_graph: int = 100_000_000 + max_graph_size_gb: int = 2 +``` + +### 2. Access Control + +```python +class SecurityPolicy: + # Method access by tier + tier: Literal["basic", "standard", "advanced", "enterprise"] + + # Dataset access + allowed_datasets: List[str] = None # None = user's accessible datasets + denied_datasets: List[str] = [] + + # Operation limits + max_call_operations: int = 100 + max_graph_combinations: int = 50 + + # Audit settings + log_operations: bool = True + log_data_access: bool = True +``` + +### 3. Validation Framework + +```python +class DAGValidator: + """Comprehensive validation before execution""" + + def validate(self, dag: QueryDAG, context: SecurityContext) -> ValidationResult: + checks = [ + self.check_circular_references(dag), + self.check_reference_resolution(dag), + self.check_resource_limits(dag, context), + self.check_method_access(dag, context), + self.check_dataset_permissions(dag, context), + self.check_parameter_types(dag), + self.check_reserved_names(dag) + ] + + errors = [e for check in checks for e in check.errors] + warnings = [w for check in checks for w in check.warnings] + + return ValidationResult( + valid=len(errors) == 0, + errors=errors, + warnings=warnings + ) +``` + +## Implementation Roadmap + +### Phase 1: Foundation (Months 1-2) + +**Goals**: Core DAG execution with basic features + +1. **Week 1-2**: AST Extensions + - QueryDAG/ChainGraph classes + - Basic reference resolution (no dots) + - JSON serialization + +2. **Week 3-4**: Execution Engine + - ExecutionContext with binding management + - Memory management framework + - Basic validation + +3. **Week 5-6**: Remote Graph Loading + - Integration with existing dataset loading + - Simple authentication flow + - Basic error handling + +4. **Week 7-8**: Testing & Documentation + - Unit tests for core functionality + - Integration tests + - Basic documentation + +**Deliverables**: +- Working DAG execution locally +- Simple remote graph loading +- Basic Chain operations with refs + +### Phase 2: Production Features (Months 2-3) + +**Goals**: Security, performance, and core transformations + +1. **Week 1-2**: Security Framework + - Resource limits implementation + - Safelist enforcement + - Audit logging + +2. **Week 3-4**: Call Operations + - Core method exposure (10-15 methods) + - Parameter validation + - Tier-based access control + +3. **Week 5-6**: Performance & Caching + - Query optimization + - Caching infrastructure + - Parallel execution + +**Deliverables**: +- Secure execution environment +- Key graph transformations +- Performance optimizations + +### Phase 3: Advanced Features (Months 3-4) + +**Goals**: Full feature set with graph combinators + +1. **Week 1-2**: Graph Combinators + - Union/Intersect/Subtract + - Policy system + - Memory-efficient implementation + +2. **Week 3-4**: Dotted References + - Full path resolution + - Nested DAG support + - Enhanced error messages + +3. **Week 5-6**: Extended Call Operations + - Additional methods (20-30 total) + - Custom method registration + - Advanced parameter types + +**Deliverables**: +- Complete combinator support +- Full reference system +- Extended method library + +### Phase 4: Enterprise & Polish (Months 4-6) + +**Goals**: Production hardening and advanced capabilities + +1. **Week 1-4**: Enterprise Features + - Custom security policies + - Advanced caching strategies + - Distributed execution + - Monitoring & observability + +2. **Week 5-8**: Polish & Migration + - Performance tuning + - Migration tools + - Comprehensive documentation + - Training materials + +**Deliverables**: +- Enterprise-ready system +- Migration guides +- Performance benchmarks +- Full documentation + +## Examples + +### Example 1: Multi-Source Analysis + +```python +# Find connections between high-risk entities across datasets +ChainGraph({ + # Load this year's and last year's data + "current": RemoteGraph("customers_2024"), + "previous": RemoteGraph("customers_2023"), + + # Find high-risk in each + "risky_current": Chain(ref="current", chain=[ + n({"risk_score": gt(0.8), "status": "active"}) + ]), + "risky_previous": Chain(ref="previous", chain=[ + n({"risk_score": gt(0.8)}) + ]), + + # Union to get all high-risk entities + "all_risky": GraphUnion("risky_current", "risky_previous", + policies={"attribute_conflict": "merge_left"}), + + # Find transaction patterns + "transactions": Chain(ref="all_risky", chain=[ + e_forward({"type": "transaction", "amount": gt(10000)}, hops=2), + n({"type": "account"}) + ]), + + # Apply UMAP for visualization + "final": Call("umap", ref="transactions", + X=["risk_score", "transaction_count"], + n_neighbors=30) +}, output="final") +``` + +### Example 2: Graph Enrichment Pipeline + +```python +ChainGraph({ + # Start with base graph + "base": RemoteGraph("network_topology"), + + # Load enrichment data + "metrics": RemoteGraph("performance_metrics"), + + # Compute centrality on base + "analyzed": Call("compute_cugraph", ref="base", + alg="betweenness_centrality", + out_col="centrality"), + + # Combine with metrics + "enriched": GraphUnion("analyzed", "metrics", + policies={ + "attribute_conflict": "merge_right", + "drop_dangling": true + }), + + # Layout for visualization + "final": Call("layout_cugraph", ref="enriched", + layout="force_atlas2", + params={"iterations": 100}) +}) +``` + +### Example 3: Nested Analysis Modules + +```python +ChainGraph({ + "fraud_analysis": ChainGraph({ + "accounts": RemoteGraph("account_graph"), + "suspicious": Chain(ref="accounts", chain=[ + n({"account_type": "personal", + "daily_volume": gt(50000)}), + e_forward({"type": is_in(["wire", "ach"])}, hops=3) + ]), + "clusters": Call("compute_igraph", ref="suspicious", + alg="louvain", out_col="community") + }, output="clusters"), + + "aml_analysis": ChainGraph({ + "entities": RemoteGraph("entity_graph"), + "peps": Chain(ref="entities", chain=[ + n({"is_pep": true}), + e_forward(to_fixed_point=true) + ]) + }, output="peps"), + + # Combine analyses + "combined": GraphIntersect("fraud_analysis", "aml_analysis"), + + # Final risk scoring + "scored": Call("dbscan", ref="combined", + eps=0.3, min_samples=5) +}, output="scored") +``` + +## Appendices + +### A. Wire Protocol Details + +#### Request Structure +```json +{ + "version": "1.0", + "type": "QueryDAG", + "metadata": { + "request_id": "uuid", + "timestamp": "2024-01-15T10:30:00Z", + "user_context": { + "tier": "advanced", + "org_id": "org123" + } + }, + "resource_limits": {...}, + "graph": {...}, + "output": "result" +} +``` + +#### Response Structure +```json +{ + "version": "1.0", + "request_id": "uuid", + "status": "success" | "partial" | "error", + "result": { + "nodes": [...], + "edges": [...], + "metadata": { + "execution_time_ms": 1234, + "memory_used_mb": 567, + "cache_hits": 3 + } + }, + "errors": [], + "warnings": [], + "debug_info": {...} // If requested +} +``` + +### B. Error Codes + +```python +ERROR_CODES = { + # Reference errors (1xxx) + "1001": "REFERENCE_NOT_FOUND", + "1002": "CIRCULAR_REFERENCE", + "1003": "AMBIGUOUS_REFERENCE", + + # Security errors (2xxx) + "2001": "ACCESS_DENIED", + "2002": "METHOD_NOT_ALLOWED", + "2003": "PARAMETER_FORBIDDEN", + + # Resource errors (3xxx) + "3001": "MEMORY_LIMIT_EXCEEDED", + "3002": "TIMEOUT_EXCEEDED", + "3003": "GRAPH_TOO_LARGE", + + # Validation errors (4xxx) + "4001": "INVALID_PARAMETER_TYPE", + "4002": "MISSING_REQUIRED_FIELD", + "4003": "SCHEMA_MISMATCH", + + # Execution errors (5xxx) + "5001": "REMOTE_GRAPH_UNAVAILABLE", + "5002": "COMPUTATION_FAILED", + "5003": "COMBINATOR_CONFLICT" +} +``` + +### C. Performance Guidelines + +1. **Memory Usage Estimates** + - Node: ~100 bytes base + attributes + - Edge: ~50 bytes base + attributes + - Overhead: ~20% for indices and metadata + +2. **Operation Complexity** + - Union: O(V₁ + V₂ + E₁ + E₂) + - Intersection: O(min(V₁, V₂) × log(max(V₁, V₂))) + - Chain operations: O(V × average_degree × hops) + +3. **Optimization Hints** + - Use column subsetting for remote graphs + - Prefer intersection before union for filtering + - Cache frequently used remote graphs + - Parallelize independent DAG branches + +### D. Migration Guide + +#### From Current Chain API +```python +# Before +g.chain([n({"type": "customer"}), e_forward()]).chain([n({"risk": gt(0.5)})]) + +# After - single expression +ChainGraph({ + "result": Chain(chain=[ + n({"type": "customer"}), + e_forward(), + n({"risk": gt(0.5)}) + ]) +}) +``` + +#### From Multiple Python Steps +```python +# Before +g1 = graphistry.bind(dataset_id="abc").chain([...]) +g2 = graphistry.bind(dataset_id="xyz").chain([...]) +# Manual combination needed + +# After - unified expression +ChainGraph({ + "g1": Chain(ref="RemoteGraph('abc')", chain=[...]), + "g2": Chain(ref="RemoteGraph('xyz')", chain=[...]), + "combined": GraphUnion("g1", "g2") +}) +``` + +## Conclusion + +GFQL Programs represents a significant evolution in PyGraphistry's capabilities, transforming it from a visualization-focused tool to a comprehensive graph programming platform. This specification provides a solid foundation for implementation while addressing critical concerns around security, performance, and usability. + +The phased implementation approach allows for iterative development and validation, ensuring each feature is production-ready before moving to the next. With careful attention to the security model and resource management, GFQL Programs will enable powerful new workflows while maintaining system stability and data isolation. \ No newline at end of file diff --git a/AI_PROGRESS/gfql-programs-spec/sketch_comparison.md b/AI_PROGRESS/gfql-programs-spec/sketch_comparison.md new file mode 100644 index 0000000000..0b158ae315 --- /dev/null +++ b/AI_PROGRESS/gfql-programs-spec/sketch_comparison.md @@ -0,0 +1,178 @@ +# Comparison: sketch.md vs sketch1X.md + +## Executive Summary + +The transformation from sketch.md to sketch1X.md represents a comprehensive enhancement that maintains all original concepts while adding significant depth, security considerations, implementation details, and production-ready specifications. **No features were lost** - instead, every concept was expanded with practical implementation guidance. + +## Feature Comparison + +### 1. Core DAG Concept ✅ Enhanced + +**Original (sketch.md)**: +- Introduced QueryDAG concept +- Basic binding environment +- Simple reference system + +**Enhanced (sketch1X.md)**: +- Complete QueryDAG specification with resource limits +- Comprehensive validation framework +- Execution model with lazy evaluation and parallel execution +- Added security context and error handling + +### 2. Reference Resolution ✅ Significantly Enhanced + +**Original (sketch.md)**: +- Basic "ref" field +- Dotted references for disambiguation +- Simple lexical scoping + +**Enhanced (sketch1X.md)**: +- Complete resolution algorithm with code examples +- Escape sequences for dots in names +- Maximum depth limits (10 levels) +- Enhanced error messages with suggestions +- Case sensitivity rules + +### 3. Remote Graph Loading ✅ Greatly Expanded + +**Original (sketch.md)**: +- Basic RemoteGraph type +- Simple dataset_id parameter + +**Enhanced (sketch1X.md)**: +- Cache policies with TTL and validation +- Retry policies with exponential backoff +- Timeout configuration +- Security model with implicit authentication +- Cross-tenant isolation +- Comprehensive error handling + +### 4. Graph Combinators ✅ Fully Specified + +**Original (sketch.md)**: +- Listed target operators: Union, Subtract, Replace, Intersect +- Basic policy concepts + +**Enhanced (sketch1X.md)**: +- Complete policy specifications for each combinator +- Advanced policies for aggregation and schema handling +- Memory-efficient implementation notes +- Duplicate edge handling strategies +- Schema validation and evolution controls + +### 5. Call Operations ✅ Production-Ready + +**Original (sketch.md)**: +- Basic call structure +- Mentioned safelisting concept +- Future Louie connectors note + +**Enhanced (sketch1X.md)**: +- Tiered safelist system (basic/standard/advanced/enterprise) +- Parameter validation schemas +- Resource limits per tier +- Method-specific parameter restrictions +- Type validation framework +- 10-30 method exposure plan + +### 6. Python API ✅ Maintained and Enhanced + +**Original (sketch.md)**: +- ChainGraph (QueryDAG) class +- Basic usage examples + +**Enhanced (sketch1X.md)**: +- Same core API preserved +- Added comprehensive examples +- Auto-desugaring capabilities +- Integration with existing PyGraphistry patterns + +## New Additions in sketch1X.md + +### 1. Security Model (Completely New) +- Resource limits framework +- Access control by tier +- Audit logging +- Dataset permissions +- Operation limits + +### 2. Implementation Roadmap (New) +- 6-month phased approach +- Clear deliverables per phase +- Risk mitigation strategies +- Testing and documentation plans + +### 3. Error Handling Framework (New) +- Comprehensive error codes (1xxx-5xxx series) +- Structured error responses +- Retry capabilities +- Debug information + +### 4. Performance Guidelines (New) +- Memory usage estimates +- Operation complexity analysis +- Optimization hints +- Caching strategies + +### 5. Production Features (New) +- Request/response structure with metadata +- Monitoring and observability +- Migration guides +- Validation framework + +### 6. Advanced Examples (New) +- Multi-source analysis +- Graph enrichment pipeline +- Nested analysis modules +- Real-world use cases + +## Key Improvements Made + +### 1. From Concept to Implementation +- Original: High-level ideas and syntax +- Enhanced: Complete specifications with validation rules, error handling, and resource management + +### 2. Security-First Design +- Original: Mentioned safelisting +- Enhanced: Comprehensive security model with tiers, policies, and audit trails + +### 3. Production Readiness +- Original: Feature exploration +- Enhanced: Production-grade specifications with monitoring, limits, and error recovery + +### 4. Developer Experience +- Original: Basic examples +- Enhanced: Rich examples, migration guides, and comprehensive documentation + +### 5. Performance Considerations +- Original: Not addressed +- Enhanced: Memory estimates, complexity analysis, and optimization strategies + +## Nothing Lost, Everything Gained + +**All features from the original sketch.md are present in sketch1X.md:** + +1. ✅ QueryDAG/ChainGraph concept +2. ✅ Reference system with dotted paths +3. ✅ Remote graph loading +4. ✅ All graph combinators (Union, Intersect, Subtract, Replace) +5. ✅ Call operations for PyGraphistry methods +6. ✅ Python API design +7. ✅ Wire protocol structure +8. ✅ Lexical scoping rules +9. ✅ Future Louie connectors mention (preserved in Call section) + +**Plus significant additions:** +- Complete security framework +- Resource management +- Error handling +- Performance optimization +- Implementation roadmap +- Migration guides +- Production monitoring + +## Conclusion + +The evolution from sketch.md to sketch1X.md represents a successful transformation from an initial RFC to a production-ready specification. Every original concept has been preserved and enhanced with the additional context needed for real-world implementation. The document now serves as both a feature specification and an implementation guide, ready for development teams to execute against. + +The phased implementation approach ensures that core functionality can be delivered quickly while building toward the complete vision over a 6-month timeline. With comprehensive security, performance, and operational considerations, sketch1X.md provides a solid foundation for GFQL Programs to become a cornerstone feature of PyGraphistry. \ No newline at end of file diff --git a/AI_PROGRESS/gfql-programs-spec/user_personas.md b/AI_PROGRESS/gfql-programs-spec/user_personas.md new file mode 100644 index 0000000000..6ab2c7f6a5 --- /dev/null +++ b/AI_PROGRESS/gfql-programs-spec/user_personas.md @@ -0,0 +1,94 @@ +# GFQL Programs User Personas + +## Overview +These personas represent key user types who would benefit from GFQL Programs features based on our Phase 1 analysis. + +## Persona 1: Alex - The Security Analyst +**Role**: Security Operations Center (SOC) Analyst +**Experience**: Intermediate Python, expert in security tools +**Primary Need**: Investigate security incidents by correlating data across multiple sources + +**Key GFQL Features Used**: +- Remote graph loading (pulling from different security datasets) +- Graph combinators (merging alerts with network data) +- DAG composition (multi-step investigations) + +**Pain Points**: +- Currently has to manually combine data from different systems +- Python scripts become complex when doing multi-hop analysis +- Hard to share investigation workflows with team + +## Persona 2: Sam - The Data Scientist +**Role**: Senior Data Scientist at a financial institution +**Experience**: Expert Python/pandas, some graph experience +**Primary Need**: Build reusable graph analysis pipelines for fraud detection + +**Key GFQL Features Used**: +- Call operations (UMAP, clustering algorithms) +- DAG composition (complex multi-stage pipelines) +- Graph combinators (enriching transaction graphs) + +**Pain Points**: +- Wants declarative workflows instead of imperative code +- Needs to version and share analysis pipelines +- Resource limits when processing large transaction graphs + +## Persona 3: Jordan - The Business Analyst +**Role**: Business Intelligence Analyst +**Experience**: SQL expert, basic Python +**Primary Need**: Create graph reports combining multiple data sources without deep programming + +**Key GFQL Features Used**: +- Remote graph loading (pre-built department graphs) +- Simple combinators (union, intersection) +- Basic call operations (layout, simple filters) + +**Pain Points**: +- Limited programming skills but needs complex analysis +- Wants SQL-like declarative syntax for graphs +- Needs clear error messages when things go wrong + +## Persona 4: Morgan - The DevOps Engineer +**Role**: Platform Engineer supporting data teams +**Experience**: Expert in infrastructure, basic data analysis +**Primary Need**: Monitor and analyze infrastructure dependencies and service meshes + +**Key GFQL Features Used**: +- DAG composition (service dependency tracking) +- Dotted references (navigating nested infrastructure) +- Call operations (topology analysis, layout) + +**Pain Points**: +- Complex service dependencies need multi-level analysis +- Performance concerns with large infrastructure graphs +- Need to set resource limits for different teams + +## Persona 5: Casey - The Compliance Officer +**Role**: Regulatory Compliance Analyst +**Experience**: Domain expert, minimal programming +**Primary Need**: Run standard compliance checks across entity relationship graphs + +**Key GFQL Features Used**: +- Remote graphs (loading standard compliance datasets) +- Graph combinators (comparing against watchlists) +- Pre-built call operations (compliance-specific algorithms) + +**Pain Points**: +- Needs audit trails for all operations +- Must handle sensitive data appropriately +- Requires reproducible, documented workflows + +## Persona 6: Riley - The Research Scientist +**Role**: Computational Biologist +**Experience**: R/Python expert, graph algorithm knowledge +**Primary Need**: Analyze biological networks with custom algorithms + +**Key GFQL Features Used**: +- Call operations (custom scientific algorithms) +- Complex DAGs (multi-stage analysis pipelines) +- Graph combinators (merging different biological datasets) + +**Pain Points**: +- Needs to integrate custom algorithms +- Large graphs hit memory limits +- Wants to parallelize complex analyses \ No newline at end of file diff --git a/AI_PROGRESS/gfql-programs-spec/user_scenarios.md b/AI_PROGRESS/gfql-programs-spec/user_scenarios.md new file mode 100644 index 0000000000..e64c2dc23b --- /dev/null +++ b/AI_PROGRESS/gfql-programs-spec/user_scenarios.md @@ -0,0 +1,123 @@ +# User Scenarios for GFQL Programs + +## Alex - Security Analyst Scenarios + +### Scenario A1: Multi-Source Threat Investigation +**Goal**: Correlate alerts from IDS with network traffic and user behavior +**Features**: RemoteGraph, GraphUnion, Chain operations +**Challenge**: Different schemas across sources, large data volumes + +### Scenario A2: Lateral Movement Detection +**Goal**: Track potential lateral movement across systems over time +**Features**: Complex DAG with multiple hops, dotted references +**Challenge**: Resource limits on deep traversals, timeout handling + +### Scenario A3: Incident Timeline Reconstruction +**Goal**: Build complete timeline merging logs, alerts, and network data +**Features**: GraphUnion with temporal ordering, call operations for layout +**Challenge**: Handling missing data, performance with large timespans + +### Scenario A4: Threat Hunting Workflow +**Goal**: Create reusable hunting patterns for specific TTPs +**Features**: DAG templates, parameterized remote graphs +**Challenge**: Making workflows shareable, version control + +## Sam - Data Scientist Scenarios + +### Scenario S1: Fraud Ring Detection Pipeline +**Goal**: Build end-to-end pipeline from transactions to fraud clusters +**Features**: Complex DAG, UMAP/clustering calls, graph combinators +**Challenge**: Memory limits with large transaction graphs + +### Scenario S2: Feature Engineering for ML +**Goal**: Extract graph features for downstream ML models +**Features**: Call operations (centrality, embeddings), DAG composition +**Challenge**: Handling failures in long-running pipelines + +### Scenario S3: A/B Testing Graph Algorithms +**Goal**: Compare different clustering approaches on same data +**Features**: Parallel DAG branches, call operations with parameters +**Challenge**: Resource allocation across parallel operations + +### Scenario S4: Real-time Fraud Scoring +**Goal**: Score new transactions against historical patterns +**Features**: RemoteGraph for history, GraphIntersect for matching +**Challenge**: Latency requirements, caching strategies + +## Jordan - Business Analyst Scenarios + +### Scenario J1: Department Collaboration Report +**Goal**: Show interactions between departments from email/meeting data +**Features**: Simple RemoteGraph loading, GraphUnion, layout +**Challenge**: Understanding error messages, handling auth failures + +### Scenario J2: Customer Journey Analysis +**Goal**: Combine touchpoints from marketing, sales, support +**Features**: GraphUnion with merge policies, basic filtering +**Challenge**: Dealing with duplicate customer IDs + +### Scenario J3: Quarterly Comparison Dashboard +**Goal**: Compare network patterns between quarters +**Features**: Multiple RemoteGraphs, GraphSubtract to show changes +**Challenge**: Resource limits on large quarterly datasets + +## Morgan - DevOps Engineer Scenarios + +### Scenario M1: Service Dependency Mapping +**Goal**: Visualize microservice dependencies with health status +**Features**: Nested DAGs for service groups, call for layout +**Challenge**: Deep nesting with dotted references + +### Scenario M2: Incident Impact Analysis +**Goal**: Find all services affected by infrastructure failure +**Features**: Multi-hop traversal, GraphIntersect with alert data +**Challenge**: Timeout handling for large service meshes + +### Scenario M3: Capacity Planning Analysis +**Goal**: Analyze resource usage patterns across services +**Features**: Call operations for metrics, graph combinators +**Challenge**: Setting appropriate resource quotas + +### Scenario M4: Configuration Drift Detection +**Goal**: Compare actual vs intended infrastructure state +**Features**: RemoteGraphs for both states, GraphSubtract +**Challenge**: Handling large configuration graphs efficiently + +## Casey - Compliance Officer Scenarios + +### Scenario C1: Sanctions Screening +**Goal**: Check entities against multiple sanctions lists +**Features**: RemoteGraphs for lists, GraphIntersect +**Challenge**: Handling fuzzy matching requirements + +### Scenario C2: Beneficial Ownership Mapping +**Goal**: Trace ultimate beneficial ownership through entities +**Features**: Multi-hop traversal with filters, depth limits +**Challenge**: Circular ownership structures + +### Scenario C3: Regulatory Change Impact +**Goal**: Identify affected entities from new regulations +**Features**: GraphUnion of entity types, filtered operations +**Challenge**: Clear audit trail requirements + +## Riley - Research Scientist Scenarios + +### Scenario R1: Protein Interaction Analysis +**Goal**: Analyze protein-protein interaction networks +**Features**: Custom call operations, large graph handling +**Challenge**: Memory limits with genome-scale networks + +### Scenario R2: Multi-Omics Integration +**Goal**: Combine genomic, proteomic, metabolomic networks +**Features**: Complex GraphUnion with schema mapping +**Challenge**: Handling heterogeneous data types + +### Scenario R3: Pathway Enrichment Pipeline +**Goal**: Run enrichment analysis across pathways +**Features**: Parallel DAG operations, custom algorithms +**Challenge**: Computational resource allocation + +### Scenario R4: Comparative Network Analysis +**Goal**: Compare networks across species/conditions +**Features**: Multiple RemoteGraphs, graph difference operations +**Challenge**: Dealing with incomplete/missing data \ No newline at end of file