[BUGFIX] Flush runtime cache periodically in LinkAnalyzer

[mikelwohlschlegel]

What this PR does

Bounds the memory footprint of brofix:checklinks on long CLI runs by
flushing TYPO3's runtime cache every 1000 processed records inside the
inner loop of LinkAnalyzer::generateBrokenLinkRecords(), plus a final
flush at the end of each chunk.

CacheManager is added as a nullable constructor argument with a
GeneralUtility::makeInstance() fallback to keep the public API
backwards compatible.

Why

Running brofix:checklinks over a large site tree from CLI causes a
PHP fatal "Allowed memory size exhausted" before the run finishes:

PHP Fatal error: Allowed memory size of 268435456 bytes exhausted
(tried to allocate 20480 bytes) in
.../typo3/cms-core/Classes/Cache/Backend/RedisBackend.php on line 338

The OOM site varies depending on cache backend configuration; it's just
the allocation that crosses the limit. The actual buildup is in TYPO3's
runtime cache (cache_runtime, TransientMemoryBackend).

Where the memory comes from

Inside LinkAnalyzer::generateBrokenLinkRecords(), the inner
while ($row = $result->fetchAssociative()) loop calls
isRecordsOnPageShouldBeChecked() and the link-parser pipeline, which
transitively invoke:

• BackendUtility::getRecord()
• BackendUtility::BEgetRootLine()
• BackendUtility::getPagesTSconfig()

Each populates entries in TYPO3's runtime cache:

• backendUtilityBeGetRootLine — rootlines for every visited page
• backendUtilityPageForRootLine — full page records for rootline resolution
• pageTsConfig-hash-to-object-* — parsed PageTSconfig objects
• backendUtilityTscPidCached — real-PID cache entries

The runtime cache uses TransientMemoryBackend (a plain PHP array) and
is never freed inside the process. In a web request that's fine; in a
CLI run that walks the whole site tree, it grows without bound.

Why per-chunk cleanup isn't enough

array_chunk($this->pids, $max) with $max = getMaxBindParameters() / 2 - 4
yields ≈ 32'763 on MariaDB/MySQL — so all page IDs of a typical site
fit into a single chunk. Any "between chunks" cleanup effectively runs
only at the very end of each table, by which point hundreds of MB of
runtime cache entries have already accumulated.

How

$result = $queryBuilder->executeQuery();
$processed = 0;
while ($row = $result->fetchAssociative()) {
    // ... existing record checks and link parsing ...
    $this->checkLinks($results, $linkTypes);

    if (++$processed % 1000 === 0) {
        $this->runtimeCache->flush();
    }
}
$this->runtimeCache->flush(); // remainder

The runtime cache is request-scoped and rebuilt on demand, so flushing
has no functional impact — the next getRecord() / getRootLine()
call just re-queries (or rebuilds from the regular page caches).

Memory diagnostics

Tested against an 18'320 pages / 90'110 tt_content install with
TYPO3 v13.4, PHP 8.3, MariaDB 10.11.

Variant	memory_limit	Result	Peak RSS
Stock v6.5.4	256M	OOM after ~27s	~256 MB
Stock v6.5.4	1024M	exit 0	~1066 MB
This PR (N=1000)	450M	exit 0	~382 MB

The unpatched run's RSS grows unbounded; the patched run plateaus
quickly and stays flat through the rest of the workload. RSS profile
(N=1000, 450M cap):

ETIME  RSS_MB  Phase
00:04  237     startup
01:05  377     ramp-up — runtime cache filling
02:05  382     ← peak Phase 1 (pages table)
06:09  382     plateau, flush loop bounds growth
09:43  290     ← table switch, accumulated cache fully freed
end    ~290    stable through Phase 2 (tt_content)

Why N=1000 (counterintuitive empirical finding)

We benchmarked several values for N against the same install at a
450M memory_limit, all with a warm brofix link target cache so HTTP
work was constant:

N	Peak Phase 1	Steady Phase 2
100	~464 MB	~356 MB
200	~461 MB	~339 MB
400	~468 MB	~355 MB
1000	~382 MB	~290 MB
2000	~439 MB	~386 MB

The intuitive expectation is "smaller N = lower peak", but the data
shows otherwise. Smaller intervals (≤400) all plateau at the same
~465 MB level, while N=1000 plateaus ~80 MB lower. The likely cause is
heap fragmentation: very frequent flush() calls churn through
allocate/free cycles on the runtime cache's internal arrays, leaving
fragments PHP cannot return to the OS. Larger N produces fewer such
cycles and lets PHP keep memory more compact. Going further to
N=2000 lets the cache itself grow back between flushes, raising the
peak again. N=1000 is the empirical sweet spot.

Trade-offs and alternatives considered

• Smaller N (50, 100, 200, 400): counterintuitively no improvement
  on memory and slightly worse fragmentation; see table above.
• Larger N (≥2000): cache grows back between flushes, peak rises.
• LIMIT/OFFSET pagination of the inner query instead of
  streaming + flush: adds query overhead (deep OFFSETs are slow on
  MariaDB) and requires QueryBuilder cloning. The streaming + counter
  approach is simpler and has no query-cost regression.
• Subprocess wrapper per site: works if the install has multiple
  configured TYPO3 sites; does nothing on single-site installs because
  the one subprocess still has to walk the whole page tree.

Compatibility

• Public constructor signature unchanged for existing callers
  (CacheManager is the new nullable trailing argument with a
  makeInstance() fallback).
• No behavior change in the backend module — the flush only affects
  what's held in PHP memory, and the runtime cache rebuilds on demand.
• Tested against TYPO3 v13.4. Earlier TYPO3 versions also expose
  cache_runtime via the same API.

[sypets]

Thanks @mikelwohlschlegel for your work. That looks pretty awesome.

I am looking at it. Looks pretty straightforward. I also created an issue to track memory optimizations #485

I resolved the conflicts. (You might have based your PR on an older version, please be sure to update your fork).

[sypets]

@mikelwohlschlegel Sorry about the late reply. I wasn't sure how to proceed. I tried to reproduce it but could not see huge memory buildups. Additionally, the allocated memory did not seem to be significantly reduced on every flush. I added logging output using:

$memUsageInBytes = memory_get_usage();
$realMemUsageInBytes = memory_get_usage(true);
$this->debug(sprintf('Memory usage: table=%s %d) real mem (in bytes)=%d, mem used (in bytes) = %d, real mem (in MB)=%d, mem used (in MB)=%d',
$table, $processed, $realMemUsageInBytes, $memUsageInBytes,
(int)($realMemUsageInBytes / 1024 / 1024),
(int)($memUsageInBytes / 1024 /1024)));

I am not sure if your detected "sweet spot" is something general or will depend on the system.

Also, the question is why there is a memory leak. Ideally, that should be fixed.

In any case, I could optimize the function LinkAnalyzer::isRecordsOnPageShouldBeChecked in #491

It is still not clear to me why that would cause a memory leak, but some DB queries were unnecessary.

Hopefully, that has already taken care of the memory situation.

It it is still necessary to flush memory, I would like to make it configurable. It would be nice if you could recheck if still reprocable with latest "main" branch.

If you set loglevel to "DEBUG" for LinkAnalyzer, you will get the memory usage output.

[mikelwohlschlegel]

@sypets Thanks for your reply.
I would say, that this is only helpful under special circumstances and upscaled environments. We are using that extension in multiple environments. In most cases, the command runs without failures. But in some installations with huge databases combined with our memory limitations (256MB) this OOM occurs and stops the whole command. In this installations, the extension can't be used, as it brings the kubernetes pods to crash and makes the results incomplete. We fixed that with a patch, which is the same as this MR. It runs in about 300 installations and fixes that OOM. But like I said, it is only for the combination "large db + limited server resources". In normal installations, this is not neccessary. I think, this error is hard to reproduce, as you need a large db and low limits for PHP.

You're decision. We can keep is as a patch. But of course it would be better for us to maintain, if this is fixed in the original code.

[sypets]

@mikelwohlschlegel Thanks for the reply. I was hoping the optimzations in LinkAnalyzer::isRecordsOnPageShouldBeChecked in #491 might have solved the problems, since the rootline checking is no longer done for every page and the other checks are based on the already performed join with pages. However, I cannot guarantee there won't be memory consumption on link checking with large DBs.

I can still merge the PR and add a configuration option so it won't be active by default. I will keep it open for now. Should make a decision in the next couple of days.