Skip to content

feat(fast-inbox): flip consensus to streaming inbox (A-1384)#24789

Open
spalladino wants to merge 10 commits into
spl/a-1383-validator-streamingfrom
spl/a-1384-flip-streaming-inbox
Open

feat(fast-inbox): flip consensus to streaming inbox (A-1384)#24789
spalladino wants to merge 10 commits into
spl/a-1383-validator-streamingfrom
spl/a-1384-flip-streaming-inbox

Conversation

@spalladino

@spalladino spalladino commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Coordinated cutover of the Fast Inbox (AZIP-22) project (issue A-1384, FI-14). Flips the authoritative path across L1, circuits, and the node from the legacy inHash/frontier-tree consumption to the streaming rolling-hash Inbox.

Stacked on the node phase: #24784 (A-1379) -> #24785 (A-1380) -> #24786 (A-1381) -> #24787 (A-1382) -> #24788 (A-1383). Base is spl/a-1383-validator-streaming.

Merge gate

  • A-1431 (domain-separation decision) must resolve before this merges. It is deliberately deferred and nothing is adopted here. If it adopts a separator, the rolling-hash link-encoding change must be inserted below this PR in the stack.

L1 (l1-contracts)

  • ProposeLib.propose calls validateInboxConsumption against the parent checkpoint's consumed position (read from the parent temp-log record) instead of inbox.consume() + the Rollup__InvalidInHash check; the returned consumed total is stored in the new record.
  • ProposeArgs gains an unsigned bucketHint calldata field (out of the attested payload digest).
  • TempCheckpointLog / CompressedTempCheckpointLog carry {inboxRollingHash, inboxMsgTotal, inboxConsumedBucket}; genesis is {0,0,0}.
  • EpochProofLib.getEpochProofPublicInputs anchors the rolling-hash chain start to the record of checkpoint start - 1 (Rollup__InvalidPreviousInboxRollingHash).
  • Inbox.sendL2Message returns and emits the compact cumulative message index.

Circuits (noir-projects)

  • MAX_L1_TO_L2_MSGS_PER_BLOCK 1024 -> 256.
  • L1ToL2MessageBundle drops num_real_msgs; a single num_msgs drives the compact tree append and the message-sponge absorb.
  • SpongeBlob::absorb_block_end_data absorbs the L1-to-L2 tree root for every block.

Node (yarn-project)

  • Streaming is the only path: the streamingInbox flag is removed across foundation/stdlib config + the sequencer/validator plumbing.
  • Sequencer: sources the parent bucket from the fork's L1-to-L2 leaf count + getInboxBucketByTotalMsgCount (cross-checkpoint); feeds inHash zero; relaxes waitForMinTxs for message-only blocks; threads the consumed bucket seq as the propose bucketHint. Automine mirrors the selection.
  • Validator: per-block acceptance + checkpoint re-execution always run; the checkpoint rebuild derives the consumed message list from the buckets between the parent checkpoint's position and the last block.
  • World state: appendL1ToL2MessagesToTree / handleL2BlockAndMessages append real leaves at compact indices; the synchronizer derives each block's bundle from its leaf-index range.
  • Blob / constants / bundle: per-block blob root, MAX_L1_TO_L2_MSGS_PER_BLOCK = 256, L1ToL2MessageBundle drops numRealMsgs; provable per-checkpoint ceiling 2457 -> 2234.
  • Archiver / TXE: the dead inHash cross-check is removed; TXE appends unpadded compact leaves.
  • L1 ABI: @aztec/l1-artifacts regenerated for the new ProposeArgs (gitignored generated output; the ethereum package builds against it).

Verification

  • L1 forge test: full propose/inbox suite green (ProposeInboxConsumption, Inbox, InboxBuckets, Rollup incl. a new anchoring negative, RollupFieldRange, FeeRollup, ValidatorSelection, escape-hatch, tmnt207/419).
  • Circuits nargo test: parity, block_root structure, msgs_only green on touched crates.
  • Node jest (runnable locally): validator proposal_handler (34) + streaming checks (14), sequencer inbox selector (9), world-state native (56), blob-lib + stdlib. tsc for the broken-build-zone packages (prover-client, sequencer-client, validator-client, etc.) is CI's job of record.
  • Codex (gpt-5.6-terra) reviewed both the L1/circuits layer and the node wiring: no consensus-critical issue; bucket cursor/range (fromExclusive, toInclusive] and pipelining-parent sourcing verified correct.

VK regen and Prover.toml sample-input regen ride CI (local bb write_vk OOMs on rollup_root).

Node prover + archiver keying (completed in this PR)

  • Prover per-block message split: the checkpoint's messages are sliced per block from the blocks' L1-to-L2 leaf-count ranges; each block root appends its own real slice at compact indices with per-block start/end snapshots and a full-height frontier hint. startCheckpoint no longer inserts messages up front.
  • Checkpoint keying from records: getL1ToL2MessageCheckpoint binary-searches the block records for the first block whose L1-to-L2 leaf count exceeds the message index (portal claim helpers depend on this); the prover-node derives the checkpoint's consumed messages from the Inbox buckets.

Codex (gpt-5.6-terra) reviewed the prover/keying rework: no consensus defect; slice partitioning, bucket-range agreement, fork ordering, frontier height, and the checkpoint boundary rule all verified.

Left for FI-18 cleanup

The message_store InboxLeaf index-formula methods (smallestIndexForCheckpoint etc.) have no correctness-critical live callers post-flip; the public-simulator's next-checkpoint fetch degrades safely without them (simulates without the not-yet-consumed messages). Deleted in FI-18.

Accepted residual risk

A-1390 (unconsumed-bucket overwrite protection) stays post-flip: a consumption backlog older than the Inbox ring can overwrite unconsumed buckets and halt proposals until an upgrade. Acceptable for non-production lines.

Review follow-up (phase-2 final review)

A final review pass appended fix(fast-inbox): thread per-block message sponges and wire the msgs-only block root in the prover:

  • The prover supplied the checkpoint-wide message sponge as every non-first block root's start sponge, which the block-merge/checkpoint-root continuity asserts reject whenever a non-first block carries messages. The sponge is now threaded per block.
  • A zero-tx non-first block (the message-only shape this PR lets the sequencer produce) was rejected by BlockProvingState and the lightweight builder, and the orchestrator never selected the msgs-only block root. Both now accept it and route it through BlockRootMsgsOnlyRollupPrivateInputs.
  • The required bucketHint parameter added to enqueueProposeCheckpoint here is now passed by the publisher unit/integration tests and the e2e synching test (the integration suite's consumption model is reworked for streaming semantics in chore(fast-inbox): delete legacy node message paths (A-1388) #24793).

spalladino and others added 4 commits July 18, 2026 10:02
…384)

Switch Rollup.propose from the legacy inbox.consume()/inHash frontier flow to
the streaming-inbox validation (AZIP-22 Fast Inbox):

- ProposeLib.propose calls validateInboxConsumption against the parent
  checkpoint's consumed position, read from the parent temp-log record, and
  stores the returned consumed total in the new record. The Rollup__InvalidInHash
  check and the inbox.consume() call are removed.
- ProposeArgs gains an unsigned bucketHint calldata field (kept out of the
  attested payload digest).
- TempCheckpointLog / CompressedTempCheckpointLog carry the consumption record
  {inboxRollingHash, inboxMsgTotal, inboxConsumedBucket}; genesis is {0,0,0}.
- EpochProofLib.getEpochProofPublicInputs anchors the rolling-hash chain start to
  the record of checkpoint start-1, mirroring previousArchive
  (Rollup__InvalidPreviousInboxRollingHash).
- Inbox.sendL2Message returns and emits the compact cumulative message index.

Frontier trees keep running unread (deleted in cleanup). Propose-path test
harnesses reference the live buckets; inbox index and rolling-hash suites updated
for compact indexing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…uits (A-1384)

Flip the block-root circuits to the streaming inbox (AZIP-22 Fast Inbox):

- MAX_L1_TO_L2_MSGS_PER_BLOCK 1024 -> 256 (per-checkpoint cap and the InboxParity
  size ladder are unchanged).
- L1ToL2MessageBundle drops num_real_msgs; a single num_msgs count drives both the
  compact (unpadded) tree append and the message-sponge absorb.
- BlockRollupPublicInputsComposer::with_message_bundle appends and absorbs the real
  num_msgs leaves and drops the dual-count assert.
- SpongeBlob::absorb_block_end_data absorbs the L1-to-L2 tree root for every block
  (the root is now per-block), dropping the is_first_block_in_checkpoint gating.

InboxParity already absorbs the real count (A-1427), so the checkpoint-root sponge
equality holds. VK and Prover.toml sample-input regen ride CI.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…he node (A-1384)

Mirror the circuit blob-format flip in the node (AZIP-22 Fast Inbox):

- Regenerate MAX_L1_TO_L2_MSGS_PER_BLOCK to 256 in @aztec/constants.
- Every block's blob carries the L1-to-L2 message tree root; drop the first-block
  conditional in encode/decode, the producers (L2Block, prover orchestrator), and
  the NUM_FIRST_BLOCK_END_BLOB_FIELDS distinction.
- getNumBlockEndBlobFields loses its parameter; blob capacity accounting spends the
  block-end overhead (now 7 fields) for every block, dropping the provable
  per-checkpoint ceiling from 2457 to 2234.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-1384)

Post-flip the L1-to-L2 message tree grows by real message counts at compact
(unaligned) indices, so StateReference.validate no longer requires its
next-available leaf index to be a multiple of NUMBER_OF_L1_L2_MESSAGES_PER_ROLLUP
(AZIP-22 Fast Inbox). The per-tx partial-state checks are unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
spalladino and others added 5 commits July 18, 2026 11:23
Remove the streamingInbox flag and wire streaming as the sole path (AZIP-22 Fast
Inbox):

- Config: drop streamingInbox from foundation env vars, stdlib sequencer/validator
  config + schemas, and the sequencer/validator config plumbing.
- Sequencer: source the parent checkpoint's consumed bucket from the fork's
  L1-to-L2 tree leaf count + getInboxBucketByTotalMsgCount (cross-checkpoint, not
  genesis-only); feed the header/block inHash zero; relax waitForMinTxs for
  message-only blocks; pass the consumed bucket seq as the propose bucketHint.
  Automine selects its single block's bundle the same way.
- Validator: per-block acceptance and checkpoint re-execution always run; the
  checkpoint rebuild derives the consumed message list from the buckets between the
  parent checkpoint's position and the last block.
- World state: appendL1ToL2MessagesToTree and handleL2BlockAndMessages append real
  leaves at compact indices; the synchronizer derives each block's bundle from its
  leaf-index range.
- Lightweight checkpoint builder feeds the checkpoint inHash zero.

Legacy bulk-fetch/insert paths go dead (deleted in FI-18). Verified via jest:
validator proposal_handler (34) + streaming checks (14), sequencer inbox selector
(9), world-state native (56). tsc rides CI.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…h (A-1384)

Align the node message-bundle representation and prover orchestrator with the
circuit flip (AZIP-22 Fast Inbox):

- stdlib L1ToL2MessageBundle drops numRealMsgs (single numMsgs real count);
  makeL1ToL2MessageBundle and the noir conversion follow.
- Prover orchestrator: compact (unpadded) L1-to-L2 tree insertion, checkpoint
  InboxParity in_hash hint fed zero, getPaddedL1ToL2Messages/getNumRealL1ToL2Messages
  retired.

Known follow-up (flagged in-code): the prover still assigns the whole checkpoint's
messages to its first block; post-flip a block carries at most
MAX_L1_TO_L2_MSGS_PER_BLOCK and a checkpoint drains across up to four blocks, so
the per-block message split (with per-block snapshots and compact frontier hints)
still needs the proving-path rework, verified by epoch-proving e2e on CI.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dding (A-1384)

- Archiver l1_synchronizer stops cross-checking the (now-zero) checkpoint inHash;
  the consensus Inbox rolling hash is verified on L1 at propose (AZIP-22 Fast Inbox).
- TXE appends real message leaves unpadded at compact indices (synchronizer,
  block_creation) and stops padding empty blocks to a full subtree.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…A-1384)

Prove streaming per-block message insertion (AZIP-22 Fast Inbox): each block root
now appends its own real message slice at compact indices instead of the whole
checkpoint being padded into the first block.

- checkpoint-prover: slices the checkpoint's messages per block from the blocks'
  L1-to-L2 leaf-count ranges and feeds each block its slice.
- CheckpointSubTreeOrchestrator.startNewBlock: inserts the block's slice, capturing
  the per-block start/end snapshots and a full-height frontier at the compact start
  index (new getFrontierSiblingPath helper); startCheckpoint no longer inserts
  messages up front.
- BlockProvingState/CheckpointProvingState carry per-block snapshots + the block's
  own slice; getPaddedL1ToL2Messages/getNumRealL1ToL2Messages retired.

Prover orchestrator unit suites verify on CI (bb proving is not local); test mocks
updated for compact per-block insertion.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… index math (A-1384)

Post-flip, 1024-per-checkpoint index arithmetic is wrong for compact indices, so
the node answers from stored records instead (AZIP-22 Fast Inbox):

- getL1ToL2MessageCheckpoint binary-searches the block records for the first block
  whose L1-to-L2 tree leaf count exceeds the message index, returning that block's
  checkpoint (portal claim helpers depend on this).
- prover-node collectRegisterData derives the checkpoint's consumed messages from
  the Inbox buckets between the parent checkpoint's position and the last block,
  replacing the legacy per-checkpoint getL1ToL2Messages fetch.

The message_store index-formula methods (smallestIndexForCheckpoint etc.) have no
correctness-critical live callers left; the public-simulator fetch degrades safely
without them. They are retired in FI-18.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nly block root in the prover (A-1384)

Three flip gaps surfaced by the phase-2 review:

- The prover supplied the checkpoint-wide message sponge as every non-first
  block root's start sponge. The block merge and checkpoint root circuits
  assert per-block continuity (right.start_msg_sponge == left.end_msg_sponge,
  first block empty, merged end == InboxParity sponge), so any checkpoint
  whose non-first block carried messages was unprovable. The sponge is now
  threaded per block: each block starts from the previous block's end sponge
  and absorbs its own slice.
- A zero-tx non-first block (the message-only block shape the sequencer can
  now produce) was rejected outright by BlockProvingState and by the
  lightweight builder, and the orchestrator never selected the msgs-only
  block root circuit. Both now accept a zero-tx non-first block carrying a
  non-empty bundle, and the orchestrator routes it through
  BlockRootMsgsOnlyRollupPrivateInputs (dispatch, prover method, VK
  allowlists, and job plumbing already existed).
- SequencerPublisher.enqueueProposeCheckpoint gained a required bucketHint
  parameter but the publisher unit/integration tests and the e2e synching
  test still passed three arguments and could not typecheck. The call sites
  now pass a hint (the integration suite's consumption model is reworked for
  streaming semantics separately, on the node-cleanup branch).

Also updates comments the flip left stale: the EpochProofLib note claiming
the epoch rolling hashes are unvalidated, the flag-gated wording in the
validator slashing table and proposal-handler tests, and the lightweight
builder's padded first-block bundle note (both branches append compactly
post-flip, so the branch is gone too).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant