[pull] master from kevoreilly:master#500
Merged
Merged
Conversation
… YARA init guard - clamav.py: Add prefetch_clamav() using ThreadPoolExecutor to fan out ALLMATCHSCAN calls in parallel before the sequential per-file loop. Reduces ClamAV wall-clock from sum(scans) to slowest single scan. Adds module-level _CLAMAV_CACHE + _CACHE_LOCK for per-task caching. - CAPE.py: Call prefetch_clamav() with all target + dropped + procdump paths before the per-file processing loop; clear cache on exit. - file_extra_info.py: Share a module-level pebble.ProcessPool singleton (get_extractor_pool()) instead of creating/destroying one per file. Also re-run PortableExecutable() when pe.digital_signers is empty so previously-analysed files get the cert data filled in on reprocess. - parse_pe.py: get_actual_checksum() returns None when the PE reports checksum=0, skipping the expensive O(file_size) recompute. Fix backend.load_der_pkcs7_certificates → pkcs7.load_der_pkcs7_certificates for cryptography ≥ 40.x (old method removed from Backend). - objects.py: Add idempotency guard to File.init_yara() so repeated calls in multi-worker environments don't reload rules unnecessarily. - process.py: Eagerly call File.init_yara() in init_worker() so YARA rules are loaded once at worker startup rather than on first task.
…traction - process.py: Replace h.close()/removeHandler()/addHandler() calls with direct log.handlers[:] = [] and log.handlers.append() to avoid acquiring logging locks in child processes after fork() (deadlock risk if parent held the lock at fork time). - file_extra_info.py: Only re-run PortableExecutable cert extraction when DigiSig.json confirms the file is signed (aux_signers non-empty). Without this guard, every unsigned PE would trigger a redundant re-parse on reprocess.
Setting it inside the HAVE_YARA try block marked init complete after the first successful category (binaries), so any concurrent or subsequent init_yara() call would short-circuit before CAPE was reached — KeyError: CAPE in get_yara. Also fixes YARA_X path which never set yara_initialized at all.
…est-state leak
The new init_yara() idempotency guard exposed a latent cross-test pollution bug that fails
tests/test_yara.py::test_get_yaras in the full suite (passes in isolation):
- tests/test_objects.py::test_get_yara replaces the class-level File.yara_rules with a single
{"hello": ...} category and never restores it. Before the guard, a later File.init_yara()
recompiled every time so the leak was harmless; with the guard, test_get_yaras's
File.init_yara() is a no-op, so get_yara("CAPE") looks up the polluted dict, finds no CAPE
rules, and returns [] -> AssertionError.
Two fixes:
- objects.py get_yara(): if the requested category isn't present, force ONE full recompile
before giving up. A live category is never silently skipped just because yara_rules was
replaced/limited elsewhere or a prior init was partial. Production-robust; preserves the
guard's perf win for the normal (category-present) path.
- test_objects.py test_get_yara(): save/restore File.yara_rules so the single-category
injection no longer leaks into later tests.
perf: ClamAV parallel prefetch, shared pebble pool, PE checksum skip, YARA init guard
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )