feat(extract): add Perl support (.pl/.pm, tree-sitter-perl)#1788
Open
ad-astra-bot wants to merge 21 commits into
Open
feat(extract): add Perl support (.pl/.pm, tree-sitter-perl)#1788ad-astra-bot wants to merge 21 commits into
ad-astra-bot wants to merge 21 commits into
Conversation
Add tests/fixtures/sample.pl + sample_module.pm and 17 Perl tests in tests/test_languages.py covering packages (incl. mid-file switch), subs, use/require imports (pragma exclusion), @ISA/use parent/use base inheritance, static + bare-sub calls (INFERRED, second-pass) and the untyped method-call negative case. Extractor (S3) and .pl/.pm dispatch (S4) not implemented yet, so all 17 fail (RED); existing suite stays green.
Bespoke tree-sitter-perl extractor under extractors/perl.py, re-exported via the extract.py facade and registered in LANGUAGE_EXTRACTORS. Emits package + sub nodes (mid-file package switches tracked), imports edges for use/require (pragmas and use parent/base excluded), and inherits edges for our \@isa / use parent / use base. Calls are handed to the shared second pass as raw_calls (INFERRED name-matching); method calls are flagged is_member_call so that pass drops them (untyped, unresolvable). tree-sitter-perl>=1.2.0,<2.0 added as a regular dependency.
…tin filter, edge dedup Review findings on the Perl extractor (S3b): - F1/F3: raw_calls now carry callee_package (qualifier) + caller_package (enclosing package). The shared second pass package-aware-filters candidates: a qualified Pkg::sub() binds only to a sub in that package; a bare call prefers the caller's own package, else requires unique import evidence. No unique match drops the edge (zero-edge, not a wrong-package guess). Additive + guarded: only Perl raw_calls carry the fields, so every other language's resolution is unchanged. - F4: external inheritance parents (RTL / other module) are emitted as stubs with empty source_file instead of falsely claiming the child file. - F5: builtin filter extended to the perlfunc core (open/close/system/exec/ index/substr/time/sleep/mkdir/unlink/...). - F6: edges deduped on (source, target, relation, context). Second-pass call tests remain RED until S4 wires .pl/.pm dispatch.
Register Perl in the four extension/dispatch tables so .pl/.pm files and extensionless #!/usr/bin/perl scripts reach extract_perl: - _DISPATCH: .pl/.pm -> extract_perl - _SHEBANG_DISPATCH: perl -> extract_perl (detect already routes perl shebang) - suffix->language map: .pl/.pm -> perl (language stats) - detect.CODE_EXTENSIONS: .pl/.pm (watch._WATCHED_EXTENSIONS inherits) .t deliberately excluded (ambiguous across ecosystems). Moves the perl example in test_extract from the unsupported-shebang test (now extract_perl) to the supported-dispatch test; tcsh takes its place as a known-but-unmapped interpreter.
The package-aware bare-call fallback only accepted direct symbol/module
import evidence, but `use P::A;` emits an imports edge to the module
label id (_make_id('P::A')), never to the sub id. A bare emit() call to a
sub defined in a use-imported foreign package therefore dropped even
though the import unambiguously names its package.
Bridge it with _has_package_import_evidence: the candidate sub's
enclosing package, re-idized the same way the use target is, must be
among the caller file's imports. Strictly scoped to the Perl/package
branch; ambiguity (two imported packages, same sub name) still drops.
Constraint: zero-edge over a guess preserved for the ambiguous case.
`use Foo::Bar;` / `require Foo::Bar;` emit an imports edge to a bare
module-label id (`_make_id('Foo::Bar')`) that matches no node when Foo::Bar
is a package defined elsewhere in the corpus — its package node's id is
`_make_id(stem, 'Foo::Bar')`. Add `_resolve_perl_imports`, run in the
top-level `extract()` AFTER the shared cross-file call pass, to re-point those
edges onto the real package node keyed by package LABEL (multi-package files
have unrelated stems). External modules keep their bare, dangling target,
matching how other languages leave unresolved external imports.
Ordering is load-bearing: `_has_package_import_evidence` reads imports targets
as bare module-label ids to bind a bare call to an imported package's sub, so
the re-point must run after call resolution, not before.
On the Foswiki core corpus (378 files): dangling imports 671 -> 174 (497
in-corpus edges re-pointed; the 174 remaining are genuine externals).
`my $cgi = new CGI();` (indirect-object constructor syntax, == `CGI->new`) parses as ambiguous_function_call_expression(function 'new', function_call_expression 'CGI()'); the inner `CGI()` was recorded as a bare function call, so a same-named sub in the corpus got a spurious `calls` edge. Detect the leading `new` on the parent node and mark the inner call `is_member_call` (edge-less), like any other untyped `$obj->meth()` dispatch. On the Foswiki core corpus this removes exactly the one wrong edge (prepareBody -> CGI, EXTRACTED) and changes nothing else.
A re-opened Perl package declared under the same fully-qualified label in two files (Foswiki-real: package Assert; in AssertOn.pm AND AssertOff.pm) is ambiguous: a bare `use P::A;` cannot say which file it means. The old setdefault map let the first package node in node order win, producing a guessed cross-file import edge — worse than dangling. Collect label_id -> list[node_ids] and re-point only when exactly one package node carries the label; >1 candidate stays dangling on the bare module-label id (zero-edge over a guess). Finding F1 (P1) from Review-3 of the S5b import re-pointer.
The re-pointer scoped package nodes and imports edges by .pl/.pm suffix, so extensionless #!/usr/bin/perl scripts — dispatched to extract_perl by shebang since S4 — had their imports never re-pointed (underreporting). extract() now passes the set of Perl source paths (where _get_extractor is extract_perl) into _resolve_perl_imports, which scopes on membership instead of suffix. Keyed on str(path) to match the nodes' source_file before relativization. Falls back to suffix matching when the set is omitted (direct callers). Finding F2 (P2) from Review-3 of the S5b import re-pointer.
…tate
`package Foo { ... }` created the Foo node but never walked the block, so
subs inside it were lost, and the current package leaked to following
top-level subs (mis-attributed to Foo). Distinguish block-form from
statement-form: push package state, walk the block, restore state.
…ckage
`sub Bar::baz {...}` was stored as label `Bar::baz()` under the current
package, so a call `Bar::baz()` (callee baz, package Bar) could never
resolve and the body's caller package was wrong. Split the qualified name:
the container is the qualified package (created if it has no `package`
statement of its own), the node label is `baz()`, and the sub body carries
caller package Bar.
- drop `.t` from the extract_perl docstring (only .pl/.pm are registered) - dedupe sprintf/wantarray in _PERL_BUILTINS - reword in-code development notes (TDD-stage and issue-shorthand markers, internal corpus references) as stable invariants - harden the method-call negative test to also reject a bare `render()` label, not only `render`/`::render`/`.render`
…walk & labels Model Perl's implicit `main` package explicitly (R1): a package-less sub now lives in a lazily-created `main` node instead of hanging off the file node, so a qualified `main::helper()` binds and a bare call from a package-less file is scoped to `main` — it can no longer be mis-bound to a same-named sub in an unrelated package without import evidence. Match the import re-pointer's provenance on resolved paths (R2): a cached fragment's `source_file` returns absolutized against the resolved root, so the exact-string membership test against the raw (relative) provenance paths missed and the re-pointer silently regressed to dangling on the second (cached) run. Compare on the resolved on-disk identity so fresh and cached runs re-point alike. Walk the AST iteratively (S1): `_string_parents`, `walk_statements` and `walk_calls` were recursive, so a crafted deep nest raised RecursionError and `_safe_extract` dropped the WHOLE file. Explicit stacks plus a coarse traversal budget keep the file node and partial graph instead of losing everything. Validate inheritance targets (S2): `@ISA` / `use parent` / `use base` content is raw string data; a value that is not a well-formed package name (control chars, markdown, over-long) is discarded rather than emitted as a node label into graph.json / the Obsidian export. Stamp `lang="perl"` on raw_calls (A1, extractor half) so the shared second pass can claim exactly Perl's calls by the extractor stamp, matching the cpp/csharp/java/objc consumers. Constraint: zero-edge over a guess — unresolvable/foreign bindings are dropped.
…egistry Move the Perl `imports` re-pointer off a private-underscore import called inline in `extract()`'s body onto the documented extension point (A2): it is now a registered `LanguageResolver`, so a language plugs in without editing `extract()`. It stays the last registered pass — after the shared call pass (which reads the bare module-label `use` targets via `_has_package_import_evidence`) and the member-call resolvers — the same position as before. Extend the registry with two optional hooks the re-pointer needs (both default off, so the existing 3-arg resolvers are untouched): - `activate`: a predicate over `paths` that overrides the suffix gate, so an extensionless `#!/usr/bin/perl` script (no `.pl`/`.pm` suffix, dispatched by shebang) still activates the pass by extractor PROVENANCE; - `wants_paths`: passes `paths` to `resolve` so the provenance-scoped pass can recompute which files it owns. Constraint: no behavior change to the re-pointer itself — same activation, same ordering, same fault isolation (log-and-skip).
…-evidence forms Gate the package-aware second-pass branch on the extractor-stamped `lang == "perl"` (A1, resolver half) instead of sniffing for a `*_package` field, matching the cpp/csharp/java/objc raw-call consumers — so the branch claims exactly Perl's calls and a future language that set a package field could not fall into it. Make `_has_package_import_evidence` accept both the bare module-label id and the real package-node id as import evidence (A3). The re-pointer rewrites `use` targets from the bare id onto the package node id; matching either shape means this check no longer depends on whether the re-pointer has run, so the pass ordering stops being semantically load-bearing. The current order (re-pointer after this pass) is kept regardless. Constraint: additive + zero-edge — no other language sets `lang="perl"`, so their candidate sets are untouched.
Python `\w` is Unicode-default, so accented (`Basé`), fullwidth (`Ｂase`) and digit-start (`Acme::1x`) names slipped the package-name validator and flowed into node labels, graph.json and the Obsidian export. Spell the classes out as explicit ASCII ranges, anchor every `::` component to a letter/underscore start, and switch to re.fullmatch so a trailing newline can no longer pass on a bare `$` anchor. 256-char cap unchanged. Constraint: fixes must not change real-corpus numbers (Foswiki core/lib: base and branch both 2776 nodes / 4611 edges / 211 inherits — Foswiki is ASCII, so no legitimate name is newly rejected).
The traversal budget was charged once per stack frame in walk_statements, then an unbounded number of sibling children were drained for free — a broad, flat file bypassed the guard entirely. _string_parents (the inheritance-string walk) was not budgeted at all. Charge _spend() per visited sibling in walk_statements (stop cleanly on exhaustion, keeping the partial graph) and per popped node in _string_parents, so all three walkers share one bounded budget as intended. walk_calls already charged per popped node — left as is.
The AST cache namespaced only on package version (cache/ast/v{version}/), so an
extractor OUTPUT-SHAPE change that ships within a release (no version bump) kept
serving pre-change entries on a cache hit — e.g. Perl fragments written before
the per-call `lang` stamp and lazy `main` package node. Add an integer
_AST_SCHEMA_VERSION riding alongside the version in the dir name
(v{version}-s{schema}/); bump it when the emitted shape changes without a
release. Follows the existing integer schema-version pattern (reflect.py,
diagnostics.py); the existing sibling-sweep cleanup removes the old namespace.
Verified on Foswiki core/lib: a poisoned base-commit cache under v0.9.12/ is not
reused after the bump — fresh extract into v0.9.12-s1/, identical numbers
(2776 nodes / 4611 edges), zero sentinel nodes in the output.
The shared registry API contract named the Perl import re-pointer and a `#!/usr/bin/perl` shebang as the motivating case for the activate/wants_paths hooks. This module deliberately knows nothing about any specific language; describe the hooks generically (provenance-scoped passes gated by which extractor produced a node, not by extension). The Perl-specific rationale already lives at its registration site in extract.py.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Perl as a new language:
.pl/.pmfiles and#!/usr/bin/perlshebang scripts are extracted via tree-sitter-perl (1.2.1, prebuilt wheels), following the documented extension path (ARCHITECTURE.md §Adding a new language extractor) and thegraphify/extractors/per-language layout.Extracted:
packagedeclarations (statement form, block formpackage Foo { }, and mid-file switches; package-less code is modeled as Perl's default packagemain),subdefinitions (including qualified declarationssub Bar::baz { })imports(EXTRACTED):use/require, with pragmas (strict,warnings,constant, …) excluded; in-corpus import targets are re-pointed to their real package nodes via a registeredLanguageResolver(unique-label guard: re-opened packages across files stay dangling rather than guessing)inherits(EXTRACTED):our @ISA,use parent(incl.-norequire),use base; unresolved parents become stub nodes with emptysource_filecalls(INFERRED): via the shared second pass, package-aware — qualified calls bind only within the matching package, bare calls prefer the caller's package, then uniqueuse-import evidence, otherwise the edge is dropped (zero-edge over guessing). Method calls ($obj->method), indirect-objectnew Foo(), and AUTOLOAD/symbolic refs intentionally produce no edges — statically undecidable in Perl.Hardening that came out of review: raw calls are
lang: "perl"-stamped for the shared pass; iterative tree walks with a traversal budget (no RecursionError file drops on crafted nesting); ASCII validation for inherited package labels; AST cache schema bump so pre-change cache entries invalidate; provenance-scoped resolver activation covers extensionless shebang files (new opt-inactivate/wants_pathshooks onLanguageResolver, defaults preserve existing resolvers).Real-world verification
Ran against the Foswiki 2.x open-source codebase (
core/lib, 378 Perl files, heavy classic bless-OO +@ISA): 2776 nodes / 4611 edges — 927 imports (713 re-pointed in-corpus, 214 genuine externals), 211 inherits, 1088 INFERRED calls, 0 parse failures, ~2.4s. Spot-checked edges against source by hand (inheritance chains,Foswiki::Funccross-file calls, ambiguousASSERTcorrectly dropped). Cache round-trip produces identical numbers.Tests
tests/test_languages.py(fixtures cover block/mid-file packages, qualified subs, pragma exclusion, ambiguity negatives, budget exhaustion, label validation, cache round-trip) + 4 resolver-hook tests--all-extras: 3201 passed, 0 failedHappy to adjust anything — naming, scope, or the resolver-hook design.