Problem
The health_* views (health_multicast_user, health_publisher_subscriber_path, health_multicast_user_rate) treat the absence of mroute / MSDP data on a device as "endpoint missing" → unhealthy. But state-collect only runs on jump devices today — 31 of 97 activated devices as of writing — so any user, publisher, or subscriber pinned to a non-jump device renders as unhealthy even when it is functioning normally.
Concrete examples:
- Users on
dz100a-lax1-tsw, dz103a-lon1-tsw, dz115a-tyo2-tsw show health_status='unhealthy' with reasons like "publisher Tunnel529 not seen as RPF interface for (S,G) on dz100a-lax1-tsw" — but no mroute data is collected from that device at all.
- The rate dimension (
health_multicast_user_rate.expected_bps_5m) falls through to NULL for every subscriber in edge-solana-shreds because some publishers in the group are on non-jump devices, tripping the gpt_missing_publisher_count > 0 guard for the whole group.
Proposed fix
Add a state_collected_devices CTE (or refreshable materialized view) seeded from devices with any mroute / MSDP observation in the last 24 h. Then in each health_* view:
- Endpoint observation booleans (
publisher_iif_observed, subscriber_oif_observed, endpoints_reconciled) stay as-is.
- Add a verdict override: if the relevant device is NOT in
state_collected_devices, set health_status='unknown' (not unhealthy) with a mismatch_reason of "endpoint device not state-collected" (or similar).
- For path rows: track
publisher_device_state_collected and subscriber_device_state_collected separately so the UI can render which side is uncollected.
- For
gpt_missing_publisher_count: ignore publishers on non-state-collected devices when computing the gating count, so subscribers in a group still get an expected_bps_5m derived from the publishers we actually observe.
Why not now
The fix is straightforward but it changes verdicts that operators are already starting to learn — needs a notice / a coordinated rollout. Banner is in place (lake#643) so users read current verdicts as work-in-progress.
Refs
- malbeclabs/infra#1501 (multicast health umbrella)
- See
~/.claude/.../memory/reference_lake_state_collect_scope.md for the jump-vs-non-jump enumeration query.
Problem
The
health_*views (health_multicast_user,health_publisher_subscriber_path,health_multicast_user_rate) treat the absence of mroute / MSDP data on a device as "endpoint missing" →unhealthy. But state-collect only runs on jump devices today — 31 of 97 activated devices as of writing — so any user, publisher, or subscriber pinned to a non-jump device renders asunhealthyeven when it is functioning normally.Concrete examples:
dz100a-lax1-tsw,dz103a-lon1-tsw,dz115a-tyo2-tswshowhealth_status='unhealthy'with reasons like "publisher Tunnel529 not seen as RPF interface for (S,G) on dz100a-lax1-tsw" — but no mroute data is collected from that device at all.health_multicast_user_rate.expected_bps_5m) falls through toNULLfor every subscriber inedge-solana-shredsbecause some publishers in the group are on non-jump devices, tripping thegpt_missing_publisher_count > 0guard for the whole group.Proposed fix
Add a
state_collected_devicesCTE (or refreshable materialized view) seeded from devices with any mroute / MSDP observation in the last 24 h. Then in eachhealth_*view:publisher_iif_observed,subscriber_oif_observed,endpoints_reconciled) stay as-is.state_collected_devices, sethealth_status='unknown'(notunhealthy) with amismatch_reasonof "endpoint device not state-collected" (or similar).publisher_device_state_collectedandsubscriber_device_state_collectedseparately so the UI can render which side is uncollected.gpt_missing_publisher_count: ignore publishers on non-state-collected devices when computing the gating count, so subscribers in a group still get anexpected_bps_5mderived from the publishers we actually observe.Why not now
The fix is straightforward but it changes verdicts that operators are already starting to learn — needs a notice / a coordinated rollout. Banner is in place (lake#643) so users read current verdicts as work-in-progress.
Refs
~/.claude/.../memory/reference_lake_state_collect_scope.mdfor the jump-vs-non-jump enumeration query.