earley to lalr parsing for cypher

CHANGELOG.md

@@ -13,6 +13,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm
 ### Changed
 - **GFQL Cypher parse memoization (perf)**: `parse_cypher` now memoizes its result (LRU over the deterministic lark parse+transform → immutable frozen AST). Repeated identical Cypher queries skip the ~15 ms parse — the dominant per-call cost of small queries (~50% of a Cypher call at 100k rows) — making end-to-end query latency ~1.3–1.7× faster at small/interactive sizes across pandas/polars/cuDF. Safe to share the cached AST: every Cypher AST node is `@dataclass(frozen=True)` and `compile_cypher_query` does not mutate the parsed tree; validation errors still raise and are not cached.
 - **GFQL structured whole-entity returns (#1650)**: Terminal Cypher `RETURN a` (whole node/edge) now emits **structured flattened columns** (`a.id`, `a.val`, `a.kind`, ...) instead of a single Cypher display string (`({id: 51, val: 51, kind: 'a'})`). The per-field columns already exist before projection, so this is "stop collapsing" rather than "rebuild": measured ~2–6.4× faster on pandas and ~2.7–4.3× on cuDF for whole-entity returns (the win grows with row count, since the old text render is O(rows) and the flat form is ~free), and the result is directly usable without re-parsing a string and survives JSON/CSV/Parquet/Arrow serialization and `plot()`. The human-readable Cypher display string remains available on demand via the `render_entity_text(result, alias)` presentation helper. OPTIONAL-MATCH / `WITH`-reentry / grouping paths that synthesize null/absent entities or still consume a single-column entity value are unchanged. Behavior change: callers that previously read the rendered display string from a terminal `RETURN a` column now receive flattened `a.*` columns. Edge case: a whole entity with NO fields to flatten — an entity with no id binding, no properties, and no type/label (in practice only an edge whose graph has no edge-id binding) — has no `{alias}.{field}` columns to emit, so it falls back to the single Cypher-display-text column under the bare alias (value is correct, e.g. `[]`); nodes always carry their id field and always flatten.
+- **GFQL Cypher parser switched to LALR (#1031)**: Unified the WHERE grammar so every clause parses as a single generic boolean `expr`, and replaced the Earley parser with one LALR(1) parser for all supported queries (~80× faster parse; OR/XOR/NOT-in-WHERE and post-WITH WHERE still parse). `generic_where_clause` now lifts a flat `where_predicate ("AND" where_predicate)*` chain of simple predicates (cmp / IS NULL / CONTAINS / STARTS/ENDS WITH / has-labels) back out to structured `filter_dict` predicates post-parse, reproducing exactly what the former `where_predicates` grammar rule matched; anything with parentheses, OR / XOR / NOT, or arithmetic stays on `where_rows`. No query-syntax or result changes.
 
 ### Performance
 - **GFQL temporal-detection dtype gate (#1650)**: `order_detect_temporal_mode` now short-circuits for numeric/bool/complex columns, which can never hold temporal *text*, instead of running an `astype(str)` + multi-regex `fullmatch` scan on every comparison. Eliminates spurious row-wise stringification in `where_rows`/comparison paths whose output never contains entity-text. Byte-identical results; measured `where_rows` speedups ~3.1× (pandas) and ~4.4–13.3× (cuDF, scaling with row count). Does not address whole-entity `RETURN a` text rendering, which is tracked separately.

graphistry/compute/gfql/cypher/ast.py

@@ -192,11 +192,12 @@ class WhereClause:
     populated by the parser; consumers MUST handle all three:
 
     - **Structured path**: ``predicates`` populated, ``expr_tree is None``.
-      ``predicates`` carries either ``WherePredicate`` entries (pure AND of
-      comparable / has-labels predicates routed via the ``where_predicates``
-      grammar rule, or AND-joined bare label predicates lifted by
-      ``generic_where_clause`` via label narrowing) or a single
-      ``WherePatternPredicate`` (pattern-only WHERE: ``WHERE (n)-[]->(m)``).
+      ``predicates`` carries either ``WherePredicate`` entries (a flat top-level
+      AND of simple predicates -- cmp / IS NULL / CONTAINS / STARTS/ENDS WITH /
+      has-labels -- lifted post-parse from the generic ``expr_tree`` by
+      ``generic_where_clause`` via ``_lift_label_only_and_spine`` /
+      ``_lift_and_spine_predicates``) or a single ``WherePatternPredicate``
+      (pattern-only WHERE: ``WHERE (n)-[]->(m)``).
     - **Tree path**: ``predicates == ()``, ``expr_tree`` populated.  Fires
       when ``generic_where_clause`` cannot lift to structured predicates
       (OR / XOR / NOT / parenthesized boolean / non-label atoms); consumers

graphistry/compute/gfql/cypher/parser.py

@@ -120,16 +120,22 @@
 properties: "{" [property_entry ("," property_entry)*] "}"
 property_entry: NAME ":" expr
 
-where_clause: "WHERE"i where_predicates
-            | "WHERE"i expr                -> generic_where_clause
-where_predicates: where_predicate ("AND"i where_predicate)*
+// Unified: every WHERE parses as a generic boolean ``expr`` (so LALR(1) accepts
+// OR/XOR/NOT/parenthesized clauses, no Earley). ``generic_where_clause`` lifts the
+// structured (filter_dict) predicates back out of the parsed tree. ``where_predicate``
+// is retained as the building block for the ``where_predicate_chain`` lift parser.
+where_clause: "WHERE"i expr                 -> generic_where_clause
 where_predicate: property_ref COMP_OP where_rhs -> cmp_where
                | property_ref "IS"i "NULL"i -> is_null_where
                | property_ref "IS"i "NOT"i "NULL"i -> is_not_null_where
                | property_ref "CONTAINS"i where_rhs -> contains_where
                | property_ref "STARTS"i "WITH"i where_rhs -> starts_with_where
                | property_ref "ENDS"i "WITH"i where_rhs -> ends_with_where
                | variable labels -> has_labels_where
+// Flat AND-chain of bare predicates; the start symbol for the lift parser. Parens
+// / OR / XOR / NOT are not ``where_predicate``s, so such clauses fail this parse and
+// stay on the row-filter path (which treats a missing property as null).
+where_predicate_chain: where_predicate ("AND"i where_predicate)*
 where_rhs: property_ref
          | value
 
@@ -533,15 +539,15 @@ class _BoundPattern:
 
 
 @lru_cache(maxsize=1)
-def _parser() -> _ParserLike:
+def _parser_lalr() -> _ParserLike:
     Lark, _, _, _ = _lark_imports()
-    # Earley required: LALR(1) cannot resolve `comparable AND WHERE_PATTERN`
-    # where WHERE_PATTERN is a leaf in `?primary`.  Ambiguity defaults to
-    # Lark's "resolve" mode (highest-priority derivation).
+    # Sole whole-query parser. The unified WHERE grammar is LALR(1)-parseable, so
+    # this accepts every supported query (~80x faster than the former Earley
+    # parser); generic_where_clause lifts the structured predicates back out.
     parser = Lark(
         _GRAMMAR,
         start="start",
-        parser="earley",
+        parser="lalr",
         maybe_placeholders=False,
         propagate_positions=True,
     )
@@ -551,16 +557,88 @@ def _parser() -> _ParserLike:
 @lru_cache(maxsize=1)
 def _pattern_parser() -> _ParserLike:
     Lark, _, _, _ = _lark_imports()
+    # Parses a single pattern fragment (e.g. ``(n)-[:R]->()``) for WHERE pattern
+    # predicates. LALR(1) accepts the pattern grammar, so no Earley is needed here.
     parser = Lark(
         _GRAMMAR,
         start="pattern",
-        parser="earley",
+        parser="lalr",
         maybe_placeholders=False,
         propagate_positions=True,
     )
     return cast(_ParserLike, parser)
 
 
+@lru_cache(maxsize=1)
+def _where_predicate_chain_parser() -> _ParserLike:
+    Lark, _, _, _ = _lark_imports()
+    # Parses a flat ``where_predicate ("AND" where_predicate)*`` chain to lift a WHERE
+    # body into structured (filter_dict) form. Parens / OR / XOR / NOT / arithmetic make
+    # the body NOT a flat chain, so this parse fails and the clause stays on the
+    # ``where_rows`` row-filter path (see _lift_and_spine_predicates for why).
+    parser = Lark(
+        _GRAMMAR,
+        start="where_predicate_chain",
+        parser="lalr",
+        maybe_placeholders=False,
+        propagate_positions=True,
+    )
+    return cast(_ParserLike, parser)
+
+
+def _retarget_span(s: SourceSpan, base: SourceSpan) -> SourceSpan:
+    """Shift an atom-relative span (from re-parsing an isolated atom substring) into
+    absolute query coordinates, given the atom's absolute ``base`` span. Exact for
+    single-line atoms (the normal case); best-effort across embedded newlines."""
+    return SourceSpan(
+        line=base.line + (s.line - 1),
+        column=(base.column + s.column - 1) if s.line == 1 else s.column,
+        end_line=base.line + (s.end_line - 1),
+        end_column=(base.column + s.end_column - 1) if s.end_line == 1 else s.end_column,
+        start_pos=base.start_pos + s.start_pos,
+        end_pos=base.start_pos + s.end_pos,
+    )
+
+
+def _retarget_predicate_spans(pred: "WherePredicate", base: SourceSpan) -> "WherePredicate":
+    """Rebuild a lifted predicate with every span shifted from atom-relative to
+    absolute (see ``_retarget_span``), so downstream errors (e.g. E108 in lowering)
+    point at the predicate's real position in the query instead of column 1."""
+    left = replace(pred.left, span=_retarget_span(pred.left.span, base))
+    right = pred.right
+    if isinstance(right, (PropertyRef, ParameterRef)):
+        right = replace(right, span=_retarget_span(right.span, base))
+    return replace(pred, left=left, right=right, span=_retarget_span(pred.span, base))
+
+
+def _lift_and_spine_predicates(
+    expr_text: str, base_span: Optional[SourceSpan] = None
+) -> Optional[List["WherePredicate"]]:
+    """Lift the WHERE body to structured ``filter_dict`` predicates iff it parses as a
+    flat ``where_predicate ("AND" where_predicate)*`` chain (cmp / IS NULL / CONTAINS /
+    STARTS|ENDS WITH / label); else ``None``, leaving it on ``expr_tree`` -> ``where_rows``.
+
+    Only flat chains lift; parentheses, OR / XOR / NOT and arithmetic stay on
+    ``where_rows``. This is a correctness boundary, not just routing: a parenthesized
+    predicate over a *missing* property (e.g. ``a IS NULL AND (b = 1)`` where ``b`` is
+    absent) must stay on ``where_rows``, which treats an absent property as null
+    (Cypher semantics); ``filter_dict`` requires the column to exist and would raise.
+
+    ``base_span`` is the WHERE body's absolute position; the body is parsed in isolation
+    (spans relative to it), so predicate spans are shifted back to absolute."""
+    _, _, LarkError, _ = _lark_imports()
+    try:
+        tree = _where_predicate_chain_parser().parse(expr_text)
+        preds = _build_transformer(expr_text).transform(tree)
+    except (LarkError, GFQLSyntaxError, GFQLValidationError):
+        return None
+    if not (isinstance(preds, tuple) and preds and all(isinstance(p, WherePredicate) for p in preds)):
+        return None
+    if base_span is None:
+        return list(preds)
+    return [_retarget_predicate_spans(p, base_span) for p in preds]
+
+
 def _to_syntax_error(message: str, *, line: Optional[int] = None, column: Optional[int] = None, **extra: Any) -> GFQLSyntaxError:
     return GFQLSyntaxError(
         ErrorCode.E107,
@@ -876,6 +954,10 @@ def where_rhs(self, _meta: Any, items: Sequence[Any]) -> object:
                 raise _to_syntax_error("Invalid WHERE right-hand side")
             return _unwrap_primitive_literal(items[0])
 
+        def where_predicate_chain(self, _meta: Any, items: Sequence[Any]) -> Tuple[WherePredicate, ...]:
+            # Flat AND-chain of bare predicates (the lift parser's start symbol).
+            return tuple(cast(WherePredicate, p) for p in items)
+
         def cmp_where(self, meta: Any, items: Sequence[Any]) -> WherePredicate:
             if len(items) != 3:
                 raise _to_syntax_error("Invalid WHERE comparison", line=meta.line, column=meta.column)
@@ -938,9 +1020,6 @@ def has_labels_where(self, meta: Any, items: Sequence[Any]) -> WherePredicate:
                 span=_span_from_meta(meta),
             )
 
-        def where_predicates(self, _meta: Any, items: Sequence[Any]) -> Tuple[WherePredicate, ...]:
-            return tuple(items)
-
         def _parse_single_where_pattern_predicate_text(self, pattern_item_text: str, span: SourceSpan) -> WherePatternPredicate:
             """Parse one bare-pattern-item text (no AND-chain) into a WherePatternPredicate.
 
@@ -1101,36 +1180,37 @@ def not_op(self, meta: Any, items: Sequence[Any]) -> BooleanExpr:
                 left=self._wrap_as_boolean_atom(items[0], meta),
             )
 
-        def where_clause(self, meta: Any, items: Sequence[Any]) -> WhereClause:
-            if len(items) != 1:
-                raise _to_syntax_error("WHERE clause cannot be empty", line=meta.line, column=meta.column)
-            predicates = cast(Tuple[WherePredicate, ...], items[0])
-            if len(predicates) == 0:
-                raise _to_syntax_error("WHERE clause cannot be empty", line=meta.line, column=meta.column)
-            return WhereClause(predicates=cast(Any, predicates), span=_span_from_meta(meta))
-
         def generic_where_clause(self, meta: Any, items: Sequence[Any]) -> WhereClause:
             span = _span_from_meta(meta)
             expr_text = self._slice(span)[len("WHERE"):].strip()
+            # Absolute span of the WHERE body (drops "WHERE" + whitespace): aligns the
+            # synth single-atom node and is the lift's span base (the lift parses
+            # ``expr_text`` in isolation, so its spans need shifting to absolute).
+            _body = self._slice(span)[len("WHERE"):]
+            _start_off = len("WHERE") + (len(_body) - len(_body.lstrip()))
+            body_span = replace(
+                span,
+                column=span.column + _start_off,
+                start_pos=span.start_pos + _start_off,
+                end_column=span.column + _start_off + len(expr_text),
+                end_pos=span.start_pos + _start_off + len(expr_text),
+            )
             # Capture Lark's structural expression tree.  Only ``and_op`` /
             # ``or_op`` / ``xor_op`` / ``not_op`` produce ``BooleanExpr``;
             # atomic WHERE expressions (single predicate) route here with a
             # non-BooleanExpr operand and are wrapped as a single-atom tree
             # so the invariant ``(expr is None) == (expr_tree is None)``
             # holds (#1213 sub-PR A / #1214).
-            expr_tree: BooleanExpr = (
-                cast(BooleanExpr, items[0])
-                if items and isinstance(items[0], BooleanExpr)
-                else BooleanExpr(op="atom", span=span, atom_text=expr_text, atom_span=span)
-            )
-            # Lark's ambiguity resolution prefers the generic `expr` path over
-            # `where_predicates` (#1194), so bare label predicates and AND
-            # chains of them land here as a ``BooleanExpr``.  Walk the parsed
-            # tree (single source of truth) to lift them to structured
-            # predicates instead of re-splitting the WHERE body text on
-            # top-level ``AND``.  Any non-AND boolean op, non-atom node, or
-            # non-bare-label atom causes a conservative fallback to raw expr
-            # (preserved on ``expr_tree`` for downstream consumers).
+            if items and isinstance(items[0], BooleanExpr):
+                expr_tree: BooleanExpr = cast(BooleanExpr, items[0])
+            else:
+                expr_tree = BooleanExpr(op="atom", span=body_span, atom_text=expr_text, atom_span=body_span)
+            # The unified grammar parses every WHERE as a generic `expr`, so bare
+            # label predicates and AND chains of them arrive here as a ``BooleanExpr``.
+            # Walk the parsed tree (single source of truth) to lift them to structured
+            # predicates instead of re-splitting the WHERE body text on top-level
+            # ``AND``.  Any non-AND boolean op, non-atom node, or non-bare-label atom
+            # causes a conservative fallback to raw expr (preserved on ``expr_tree``).
             lifted = _lift_label_only_and_spine(expr_tree)
             if lifted:
                 predicates = tuple(
@@ -1161,6 +1241,12 @@ def generic_where_clause(self, meta: Any, items: Sequence[Any]) -> WhereClause:
                     expr_text=expr_text,
                     span=span,
                 )
+            # Lift a flat AND chain of bare predicates to structured ``filter_dict``
+            # predicates; parens / OR / XOR / NOT keep the clause on ``expr_tree`` ->
+            # ``where_rows`` (see _lift_and_spine_predicates for why parens matter).
+            lifted_preds = _lift_and_spine_predicates(expr_text, body_span)
+            if lifted_preds is not None:
+                return WhereClause(predicates=tuple(lifted_preds), span=span)
             return WhereClause(
                 predicates=(),
                 expr_tree=expr_tree,
@@ -1873,10 +1959,9 @@ def _parse_cypher_cached(query: str) -> Union[CypherQuery, CypherUnionQuery, Cyp
     # Pre-parse detection of known-but-unsupported Cypher forms
     _check_unsupported_syntax_patterns(query)
 
-    parser = _parser()
     transformer = _build_transformer(query)
     try:
-        tree = parser.parse(query)
+        tree = _parser_lalr().parse(query)
         node = transformer.transform(tree)
     except Exception as exc:
         _, _, LarkError, _ = _lark_imports()

graphistry/tests/compute/gfql/cypher/test_lowering.py

@@ -16849,3 +16849,18 @@ def test_order_by_multi_column_no_crash() -> None:
             assert rows[i]["name"] <= rows[i + 1]["name"], (
                 f"Rows with equal score not sorted by name: {rows}"
             )
+
+
+def test_string_cypher_parenthesized_and_preserves_missing_property_as_null() -> None:
+    # A parenthesized AND must stay on the row-filter path, not the filter_dict path:
+    # the row engine treats an absent property as null (Cypher semantics), whereas
+    # filter_dict requires the column to exist and would raise.
+    g = _mk_graph(
+        pd.DataFrame({"id": ["a", "b", "c", "d"], "x": [1, 2, 1, 1]}),  # no 'missing' col
+        pd.DataFrame({"s": ["a", "b", "c"], "d": ["b", "c", "d"]}),
+    )
+    got = g.gfql("MATCH (n) WHERE n.missing IS NULL AND (n.x = 1) RETURN n.id AS id")
+    assert sorted(got._nodes["id"].tolist()) == ["a", "c", "d"]
+    # IS NOT NULL on the same absent property yields no rows (absent -> null)
+    none = g.gfql("MATCH (n) WHERE n.missing IS NOT NULL AND (n.x = 1) RETURN n.id AS id")
+    assert none._nodes["id"].tolist() == [] if "id" in none._nodes.columns else len(none._nodes) == 0