KV quota fixes: edge caching, compare scraper gate, kv-cache hardening#18
Merged
Conversation
A scraper botnet crawling the 25k noindexed /compare pages tripled traffic and blew through the 100k/day free-plan KV read quota — every kvCached call burns a read op whether it hits or misses, and Worker responses never entered the CDN cache because Cache-Control headers alone don't do that. - Store cacheable GET responses (pages, sitemaps, server fns) in the Cloudflare edge cache via the Cache API; repeat hits skip SSR, KV, and Turso entirely. Reseed flow already runs purge_everything. - Refuse /compare HTML to anything that isn't a real browser navigation (Accept: text/html + Sec-Fetch-Mode: navigate) and add Disallow: /compare to robots.txt. - kv-cache: per-isolate in-memory hot cache so repeat lookups don't spend KV reads; keep cache writes alive past the response with waitUntil; log put failures instead of swallowing them; fall through to the database when KV reads throw (quota exhaustion) instead of 500ing the page.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
unclouded | edae8fd | Commit Preview URL Branch Preview URL |
Jul 13 2026, 01:59 AM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A residential-proxy scraper botnet crawling the ~25k noindexed
/comparepages tripled traffic on Jul 11–12 (~25–30k → 84–94k req/day) and exhausted the 100k/day free-plan KV read quota (~108–112k reads/day). EverykvCachedcall costs a read op hit or miss, and Worker-generated responses never entered Cloudflare's CDN cache — thes-maxageheaders were decorative without a Cache API put.Changes
src/routes/-worker-entry.ts): cacheable GET responses (pages, sitemaps, robots,/_serverFnGETs) are stored in the CDN cache viacaches.default, keyed on the full URL (pagination query params survive). Repeat hits are served withX-Edge-Cache: HITand skip SSR, KV, and Turso entirely./searchstaysno-store; responses withSet-Cookieare never cached. The reseed flow'spurge_everythingalready invalidates this cache./compare/*HTML is only served to genuine browser navigations (Accept: text/html+Sec-Fetch-Mode: navigate); everything else gets a cheap 403 before SSR runs. Real users are unaffected — in-app nav goes through/_serverFn. Also addsDisallow: /compareto robots.txt (pages are already noindexed).db/kv-cache.ts): per-isolate in-memory hot cache (5 min TTL, 256 entries) so repeat lookups of hot keys don't spend KV reads; cache writes kept alive past the response withwaitUntiland failures logged instead of swallowed;kv.getfailures (e.g. quota exhaustion) fall through to the database instead of 500ing the page.Verification (against production)
X-Edge-Cache: HIT/compare/...→ 403; browser-navigation headers → 200getAppBySlug:*key readable viawrangler kv key get --remoteseconds later