Skip to content

Add conditional download caching to LoadRuleGroups#7612

Open
PaurushGarg wants to merge 1 commit into
cortexproject:masterfrom
PaurushGarg:optimize-load-rule-groups-conditional-download
Open

Add conditional download caching to LoadRuleGroups#7612
PaurushGarg wants to merge 1 commit into
cortexproject:masterfrom
PaurushGarg:optimize-load-rule-groups-conditional-download

Conversation

@PaurushGarg

Copy link
Copy Markdown
Contributor

What this PR does:
Adds conditional download caching to LoadRuleGroups() in the bucket rule store. Instead of downloading every rule group file from object storage on each sync cycle, it checks if the file has changed (via HEAD/Attributes) before doing a full GET.

Why:
The ruler syncs rule groups from object storage every poll interval (default 60s). For tenants with hundreds of rule groups, downloading all files every cycle is wasteful — most files don't change between syncs. The full GET + protobuf unmarshal for each file adds up, causing sync duration to approach or exceed the poll interval and trigger missed syncs.

Flow:

  • First sync (cold cache): normal GET for all files — no behavior change
  • Subsequent syncs: HEAD to check LastModified → if file unchanged, return cached content (no GET, no unmarshal)
  • Cache is pruned after each sync to remove entries for rule groups no longer owned by this pod
  • If HEAD fails, falls through to normal GET

Observability:
New metric cortex_ruler_rule_group_load_cache_operations_total{result="hit|miss"} tracks cache effectiveness.

Testing:

  • TestLoadRuleGroupsCache — verifies cache hit on unchanged files
  • TestLoadRuleGroupsCacheMissOnModification — verifies modified files are re-downloaded
  • TestLoadRuleGroupsCachePrune — verifies stale entries are cleaned up after ring rebalance

Which issue(s) this PR fixes:
Fixes #

Checklist

  • Tests updated
  • Documentation added
  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
  • docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags

@PaurushGarg PaurushGarg force-pushed the optimize-load-rule-groups-conditional-download branch 3 times, most recently from f7ab123 to bfe4129 Compare June 10, 2026 00:42
Signed-off-by: Paurush Garg <paurushg@amazon.com>
@PaurushGarg PaurushGarg force-pushed the optimize-load-rule-groups-conditional-download branch from bfe4129 to 2866fa5 Compare June 10, 2026 01:05
@PaurushGarg PaurushGarg marked this pull request as ready for review June 10, 2026 02:07
@dosubot dosubot Bot added component/rules Bits & bobs todo with rules and alerts: the ruler, config service etc. go Pull requests that update Go code type/performance labels Jun 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component/rules Bits & bobs todo with rules and alerts: the ruler, config service etc. go Pull requests that update Go code size/L type/performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant