fix(sequencer): diagnose why no committee exists instead of a scary log#24760
Open
spalladino wants to merge 4 commits into
Open
Conversation
Operators saw "Cannot propose at target slot N since the committee does not exist on L1" and panicked. Replace it with a diagnosis derived from the live attester count and whether the chain has ever produced a block: - enough validators staked -> committee just waiting for the sampling lag (info) - too few staked and no blocks yet -> bootstrap, waiting for validators (info) - too few staked but blocks exist -> validator set shrank on a live chain (warn) Dedupe the log per epoch instead of per slot, and make the diagnostic L1 reads best-effort so the propose path never throws.
…atSeconds The "no committee yet" diagnosis reported the epoch a committee should exist by as currentEpoch + lag + 1 — a moving target that re-added the lag on every check and never counted down. Replay L1's validator-set sampling rule instead: a committee for epoch E exists iff at least targetCommitteeSize attesters were staked at epochStart(E) - lag * epochDuration, so read the historical attester count (via GSE.getAttesterCountAtTime) at each candidate epoch's sample time and report the first epoch that actually qualifies. The estimate is now a fixed epoch whose ETA converges and counts down, tagged by provenance (historical / projected-future / fallback) so the log wording reflects its confidence. Also moves formatSeconds out of the sequencer into @aztec/foundation/string.
…ittee.ts Move logMissingCommittee, estimateFirstCommitteeEpoch, and formatEpochEta out of the Sequencer class into missing_committee.ts as free functions taking a MissingCommitteeContext deps object, so the sequencer only wires them in. Drop the FirstCommitteeEpochProvenance enum: findFirstEpochWithCommittee now returns EpochNumber | undefined, and the caller reports a bounded epoch when it is undefined. Behavior-focused tests move alongside the functions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Operators on a freshly-canonical rollup saw
Cannot propose at target slot N since the committee does not exist on L1and panicked. The message was cryptic and, worse, my first attempt reasoned about it wrong: I assumed validators join at genesis, but they actually join after the rollup is made canonical via governance. Verified against the mainnet incident on0x91ff…: attester count stayed 0 for ~2 weeks after genesis, then jumped to ~3462 the night the rollup went canonical — so "committee should have formed by epoch 2" was nonsense.Approach
Diagnose the cause from two cheap live signals the sequencer already has — the current attester count and whether the chain has ever produced a block — rather than any genesis- or governance-time anchor (canonical-registration time is an expensive log scan and predicts validator eligibility, not behavior):
count ≥ target→ committee is just waiting for the sampling window to advance; expected bycurrent + lag + 1(info, with ETA).count < targetand no blocks ever → bootstrap, waiting for validators to stake (info).count < targetbut blocks exist → the validator set shrank below the required size on a live chain, a genuine incident (warn).The chain-tip signal (blocks-ever-produced) is what separates expected bootstrap from a real validator-set regression. The classification is a pure, unit-tested function; the sequencer composes and formats the log. The diagnostic is deduped per epoch (was per slot, so it spammed every ~36–72s) and the extra L1 reads are best-effort so the propose path never throws.
Fixes A-1423