zlib: validate flush kind for brotli streams

[Ic3b3rg]

Summary

BrotliCompress.flush(kind) and BrotliDecompress.flush(kind) accepted numeric kind values that are not valid Brotli operations and forwarded them to the native Brotli encoder/decoder.

Passing zlib flush constants such as Z_FINISH (4) or Z_BLOCK (5) could make the process spin at 100% CPU indefinitely, because Brotli only supports operation values in the range 0..3.

This PR validates kind for Brotli streams before queuing the fake flush chunk. Numeric values outside the Brotli operation range now throw ERR_OUT_OF_RANGE, matching the existing options.flush and options.finishFlush validation path.

Scope

The validation is intentionally limited to Brotli streams. zlib and zstd stream flush() behavior is unchanged.

Valid Brotli operations continue to work:

• BROTLI_OPERATION_PROCESS (0)
• BROTLI_OPERATION_FLUSH (1)
• BROTLI_OPERATION_FINISH (2)
• BROTLI_OPERATION_EMIT_METADATA (3)

Observable behavior changes

Call	Before	After
flush(4) / Z_FINISH	Hangs / spins at 100% CPU	Throws ERR_OUT_OF_RANGE
flush(5) / Z_BLOCK	Hangs / spins at 100% CPU	Throws ERR_OUT_OF_RANGE
flush(6)	Throws incidentally from the fake-chunk path	Throws ERR_OUT_OF_RANGE during validation
flush(0..3)	Works	Works
flush() / flush(cb)	Works, uses default Brotli flush operation	Works, uses default Brotli flush operation

Error type

ERR_OUT_OF_RANGE is intentional for invalid numeric values: kind has the correct JavaScript type, but the value is outside the valid Brotli operation range. Non-number values should continue to use the existing type-validation behavior.

Cross-runtime note

Bun is fixing the same bug on their side (oven-sh/bun#31505) but plans to throw ERR_INVALID_ARG_TYPE to match the current incidental behaviour of flush(6) on Node. This PR chooses ERR_OUT_OF_RANGE for internal consistency with the checkRangesOrGetDefault validation already used elsewhere in lib/zlib.js.

Fixes: #63701

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/73919/

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/73925/

[alii]

WTBU that ERR_INVALID_ARG_TYPE is currently thrown for flush(6) but this PR uses ERR_OUT_OF_RANGE. Is that the desired behaviour? oven-sh/bun#31505 currently implemented ERR_INVALID_ARG_TYPE, but of course happy to switch before either of these merge.

[addaleax]

@Ic3b3rg Any thoughts on the most recent comment here?

[Ic3b3rg]

@alii The previous ERR_INVALID_ARG_TYPE for flush(6) was incidental: 6 fell through to the fake flush chunk path, where kFlushBuffers[6] was undefined, and that later became write(undefined). With explicit validation, kind is a number but outside the valid operation range, so ERR_OUT_OF_RANGE matches the existing checkRangesOrGetDefault() behavior used for options.flush and options.finishFlush.

Also, after @addaleax’s earlier suggestion, this PR now validates flush(kind) for all zlib-style streams, not only Brotli. I’ll update the PR description to reflect that.