negative-cache 404s, materialize desktop-only flag#19
Merged
Conversation
Scrapers probing nonexistent slugs and junk paths were re-running full SSR + Turso queries on every hit, forever: null results are skipped by the KV cache (a stored null reads back as a miss, and the 1k/day free-plan write quota is too scarce to spend on probe traffic), and only 200s entered the edge cache. - Edge-cache rendered 404s for an hour on every route, not just routes with a cache rule, so repeat probes of the same URL stop at the colo. - Remember null results in the per-isolate memory cache so repeat lookups within an isolate skip Turso without burning KV writes.
The desktopOnlyAppIds subquery scanned the entire app_tags join (~30k rows at 5.8k apps) inside listApps, searchApps, and getRecentApps on every cache miss — the row-read multiplier behind the July Turso spike whenever KV misses piled up. The set only changes at reseed, so compute it once into a new apps.desktop_only column (db/seed/lib/desktop-only.ts, run during seed:import) and filter on the column in the hot queries. Deploy order: pnpm db:push (additive column, safe under the old worker), pnpm db:backfill-desktop-only, then deploy — the new queries reference the column, so it must exist and be populated first.
Deploying with
|
| Status | Name | Latest Commit | Preview URL | Updated (UTC) |
|---|---|---|---|---|
| ✅ Deployment successful! View logs |
unclouded | d710821 | Commit Preview URL Branch Preview URL |
Jul 14 2026, 01:00 AM |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the two remaining Turso read-amplification leaks behind the July rows-read spike: uncached 404s (scrapers re-render dead slugs forever) and the
desktopOnlyAppIdssubquery that re-scanned the wholeapp_tagsjoin on every list-query cache miss.Key Changes
apps.desktop_onlycolumn, computed once at import time (db/seed/lib/desktop-only.ts, wired intoseed:import).listApps,searchApps, andgetRecentAppsfilter on the column instead of scanning ~30kapp_tagsrows per cache miss.pnpm db:backfill-desktop-onlyscript for the existing database.Motivation
Rows read peaked at ~78M/day on Jul 12 (baseline ~10M). The scraper botnet + KV TTL-expiry wave exposed both leaks: every 404 probe cost a full SSR + Turso round trips, and every list-query miss paid the
app_tagsscan three ways.Testing
curl -sI https://unclouded.app/apps/no-such-slugtwice → second response should be404+X-Edge-Cache: HIT.Risks / deploy order
The new queries reference
apps.desktop_only, so the column must exist and be populated before the worker deploys:If
db:pushproposes anything beyond a singleALTER TABLE apps ADD COLUMN(e.g. table recreation from unrelated drift), stop and review first.