Fix codebase indexing for plain text files

[WebMad]

Summary

• add .txt to the extensions scanned by codebase indexing
• route plain text files through the existing line-based fallback chunking mechanism
• add regression coverage for .txt parsing and case-insensitive extension matching

Fixes #931

Testing

• cd src && npx vitest run services/code-index/processors/__tests__/parser.txt.spec.ts
• relevant parser/scanner suite: 55 tests passed
• repository lint passed via the pre-commit hook
• repository type-check passed via the pre-push hook
• manually verified semantic search with a workspace containing both README.md and manual.txt; content unique to both files was returned after full reindexing
• manually reproduced the original failure by temporarily removing .txt support and confirming that only the Markdown content was indexed

Summary by CodeRabbit

• New Features

  • Added support for plain-text (.txt) files in code indexing.
  • Plain-text files now use fallback chunking and preserve file metadata and content.
  • Uppercase .TXT extensions are recognized consistently.
  • Plain-text files no longer undergo structural code-definition parsing.

• Tests

  • Added coverage for plain-text indexing, fallback chunk generation, metadata, and extension handling.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 5b227996-ac79-48f5-ae28-95445df5932f

📥 Commits

Reviewing files that changed from the base of the PR and between 5f582a5 and f34dd54.

📒 Files selected for processing (4)

• src/services/code-index/processors/__tests__/parser.txt.spec.ts
• src/services/code-index/shared/supported-extensions.ts
• src/services/tree-sitter/__tests__/plainTextIntegration.spec.ts
• src/services/tree-sitter/index.ts

---

📝 Walkthrough

Walkthrough

Plain-text files are added to fallback chunking, producing indexed line-based blocks. Tree-sitter definition extraction now recognizes .txt files and exits without parsing them. Tests cover chunk metadata, case-insensitive extensions, parser bypassing, and mocked dependencies.

Changes

Plain-text indexing

Layer / File(s)	Summary
Plain-text fallback chunking src/services/code-index/shared/supported-extensions.ts, src/services/code-index/processors/__tests__/parser.txt.spec.ts	.txt is included in fallback extensions, with tests verifying fallback chunks, metadata, and uppercase extension handling.
Tree-sitter plain-text bypass src/services/tree-sitter/index.ts, src/services/tree-sitter/__tests__/plainTextIntegration.spec.ts	.txt files return no structural definitions without loading language parsers or reading file contents.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested reviewers: edelauna

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: adding plain-text file indexing support.
Description check	✅ Passed	The description covers the change summary and testing, though it is lighter than the full template.
Linked Issues check	✅ Passed	The PR implements the .txt indexing behavior requested in #931, including fallback chunking and regression tests.
Out of Scope Changes check	✅ Passed	The code changes stay focused on plain-text indexing support and related tests, with no obvious unrelated additions.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[WebMad]

test-zoocode-contribution.tar.gz

[WebMad]

Plain text files can also reach parseSourceCodeDefinitionsForFile() through the folded file context flow, which is separate from codebase indexing. The fallbackExtensions list only affects CodeParser, so it does not prevent this function from trying to load a Tree-sitter parser for .txt. Since no such parser exists, we return undefined early to avoid an Unsupported language: txt error