Skip to content

Add media usage index write-side foundation#1746

Merged
ascorbic merged 30 commits into
mainfrom
feature/media-usage-index-core
Jul 6, 2026
Merged

Add media usage index write-side foundation#1746
ascorbic merged 30 commits into
mainfrom
feature/media-usage-index-core

Conversation

@khoinguyenpham04

@khoinguyenpham04 khoinguyenpham04 commented Jul 3, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

Adds the write-side foundation for the Media Usage index. This includes the database tables/status metadata, content media reference extraction, content source snapshots for columns and draft_overlay, repository APIs, runtime refresh wiring for normal content lifecycle writes, stale marking for known bypass paths, and generation-guarded source replacement/deletion so refresh races preserve current usage and mark coverage stale instead of failing content writes.

This PR intentionally stops before repair/rebuild, the usage read API, admin Used In UI, safe delete, replace-in-place behavior, or media delete behavior changes.

Related discussion: #1503

Type of change

  • Bug fix
  • Feature (requires maintainer-approved Discussion)
  • Refactor (no behavior change)
  • Translation
  • Documentation
  • Performance improvement
  • Tests
  • Chore (dependencies, CI, tooling)

Checklist

AI-generated code disclosure

  • This PR includes AI-generated code — model/tool: OpenCode with GPT-5.5 (openai/gpt-5.5)

Screenshots / test output

Non-visual backend/indexing change.

Verified:

  • pnpm lint
  • pnpm typecheck
  • pnpm format
  • pnpm --filter emdash exec vitest run tests/integration/database/media-usage-status-migration.test.ts tests/integration/database/media-usage-content-fields.test.ts tests/integration/database/media-usage-repository.test.ts tests/integration/database/media-usage-migration.test.ts tests/integration/database/media-usage-content-snapshots.test.ts tests/integration/database/media-usage-content-refresh.test.ts tests/integration/database/media-usage-runtime-refresh.test.ts tests/integration/database/media-usage-stale-bypass.test.ts tests/unit/media/usage-extractor.test.ts tests/unit/media/usage-source-key.test.ts (10 files, 98 tests)
  • git diff --check

Try this PR

Open a fresh playground →

A full working EmDash site, deployed from this branch. Each visit gets its own session-scoped sandbox: no login needed and no shared state. Try the admin, edit content, hit the public site.

Tracks feature/media-usage-index-core. Updated automatically when the playground redeploys.

khoinguyenpham04 and others added 25 commits July 1, 2026 00:40
Copilot AI review requested due to automatic review settings July 3, 2026 14:55
@changeset-bot

changeset-bot Bot commented Jul 3, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 042c0e5

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
Name Type
emdash Patch
@emdash-cms/cloudflare Patch
@emdash-cms/sandbox-workerd Patch
@emdash-cms/fixture-perf-site Patch
@emdash-cms/perf-demo-site Patch
@emdash-cms/cache-demo-site Patch
@emdash-cms/do-demo-site Patch
@emdash-cms/do-solo-demo-site Patch
@emdash-cms/admin Patch
@emdash-cms/auth Patch
@emdash-cms/blocks Patch
@emdash-cms/gutenberg-to-portable-text Patch
@emdash-cms/x402 Patch
create-emdash Patch
@emdash-cms/auth-atproto Patch
@emdash-cms/plugin-embeds Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 3, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
emdash-demo-cache 042c0e5 Jul 03 2026, 05:38 PM

@github-actions github-actions Bot added the review/needs-review No maintainer or bot review yet label Jul 3, 2026
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 3, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
emdash-playground 042c0e5 Jul 03 2026, 05:38 PM

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 3, 2026

Copy link
Copy Markdown

Deploying with  Cloudflare Workers  Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status Name Latest Commit Updated (UTC)
✅ Deployment successful!
View logs
docs 02c220d Jul 03 2026, 03:11 PM

@pkg-pr-new

pkg-pr-new Bot commented Jul 3, 2026

Copy link
Copy Markdown

Open in StackBlitz

@emdash-cms/admin

npm i https://pkg.pr.new/@emdash-cms/admin@1746

@emdash-cms/auth

npm i https://pkg.pr.new/@emdash-cms/auth@1746

@emdash-cms/auth-atproto

npm i https://pkg.pr.new/@emdash-cms/auth-atproto@1746

@emdash-cms/blocks

npm i https://pkg.pr.new/@emdash-cms/blocks@1746

@emdash-cms/cloudflare

npm i https://pkg.pr.new/@emdash-cms/cloudflare@1746

@emdash-cms/contentful-to-portable-text

npm i https://pkg.pr.new/@emdash-cms/contentful-to-portable-text@1746

emdash

npm i https://pkg.pr.new/emdash@1746

create-emdash

npm i https://pkg.pr.new/create-emdash@1746

@emdash-cms/gutenberg-to-portable-text

npm i https://pkg.pr.new/@emdash-cms/gutenberg-to-portable-text@1746

@emdash-cms/plugin-cli

npm i https://pkg.pr.new/@emdash-cms/plugin-cli@1746

@emdash-cms/plugin-types

npm i https://pkg.pr.new/@emdash-cms/plugin-types@1746

@emdash-cms/registry-client

npm i https://pkg.pr.new/@emdash-cms/registry-client@1746

@emdash-cms/registry-lexicons

npm i https://pkg.pr.new/@emdash-cms/registry-lexicons@1746

@emdash-cms/sandbox-workerd

npm i https://pkg.pr.new/@emdash-cms/sandbox-workerd@1746

@emdash-cms/x402

npm i https://pkg.pr.new/@emdash-cms/x402@1746

@emdash-cms/plugin-ai-moderation

npm i https://pkg.pr.new/@emdash-cms/plugin-ai-moderation@1746

@emdash-cms/plugin-atproto

npm i https://pkg.pr.new/@emdash-cms/plugin-atproto@1746

@emdash-cms/plugin-audit-log

npm i https://pkg.pr.new/@emdash-cms/plugin-audit-log@1746

@emdash-cms/plugin-color

npm i https://pkg.pr.new/@emdash-cms/plugin-color@1746

@emdash-cms/plugin-embeds

npm i https://pkg.pr.new/@emdash-cms/plugin-embeds@1746

@emdash-cms/plugin-field-kit

npm i https://pkg.pr.new/@emdash-cms/plugin-field-kit@1746

@emdash-cms/plugin-forms

npm i https://pkg.pr.new/@emdash-cms/plugin-forms@1746

@emdash-cms/plugin-webhook-notifier

npm i https://pkg.pr.new/@emdash-cms/plugin-webhook-notifier@1746

commit: 042c0e5

@khoinguyenpham04 khoinguyenpham04 self-assigned this Jul 3, 2026
@khoinguyenpham04 khoinguyenpham04 requested a review from ascorbic July 3, 2026 15:14
@github-actions github-actions Bot added review/needs-rereview Author pushed changes since the last review and removed review/awaiting-author Reviewed; waiting on the author to respond labels Jul 3, 2026

@ascorbic ascorbic left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great stuff. Just a couple of notes. However this is a very large PR. For future ones, can you keep them smaller in scope so they're easier to review.

Comment thread .changeset/media-usage-index-hardening.md
Comment thread packages/core/src/media/usage/content-refresh.ts Outdated
Comment thread packages/core/src/database/repositories/media-usage.ts Outdated
@github-actions github-actions Bot added review/awaiting-author Reviewed; waiting on the author to respond and removed review/needs-rereview Author pushed changes since the last review labels Jul 3, 2026
@khoinguyenpham04 khoinguyenpham04 added this to the 1.0 milestone Jul 3, 2026
@github-actions github-actions Bot added review/needs-rereview Author pushed changes since the last review overlap and removed review/awaiting-author Reviewed; waiting on the author to respond labels Jul 3, 2026
@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Overlapping PRs

This PR modifies files that are also changed by other open PRs:

This may cause merge conflicts or duplicated work. A maintainer will coordinate.

masonjames added a commit to masonjames/emdash that referenced this pull request Jul 3, 2026
@masonjames

Copy link
Copy Markdown
Contributor

Tested this on a real production site (masonjames.com — Ghost→EmDash migration, Node + SQLite, downstream fork with local media patches). Results:

  • Merge + tests: merged cleanly, 4364/4364 core tests passing including the new media-usage suites.
  • Migration on live data: 050_media_usage_index_status applied cleanly at boot on the production SQLite DB — all three tables present, ledger correct, no boot noise.
  • Write-side, end-to-end: inserting a media-library image into a post and publishing produced exactly the expected _emdash_media_usage row; lifecycle writes create content:{collection}:{id}:columns source rows as designed. Alt/caption edits and unreferenced uploads correctly created no rows. All collections initialized as stale / CONTENT_USAGE_STALE — the intended "incomplete until rebuild" state. No route errors, no interference with our local media patches.
  • Field observation for later phases: pre-existing content yields zero usage rows here — migrated bodies embed media as absolute URLs on a custom media domain, so there are no managed references to extract. Expected given this PR's scope, but imported sites will depend on repair/rebuild to populate the index — and it may be worth attributing absolute URLs that match a configured media/public URL, since WordPress and Ghost migrations commonly end up in this shape.

Happy to re-test when the read API / rebuild phases land.

@khoinguyenpham04

Copy link
Copy Markdown
Collaborator Author

@masonjames Thank you so much for testing this out! This is actually a really helpful validation :)

Also, I agree URL attribution for configured media/public domains is worth considering in the rebuild/import follow-up.

@masonjames

Copy link
Copy Markdown
Contributor

@khoinguyenpham04 awesome! I'm really excited about the media components in emdash. Feel free to tag me on PRs whenever you'd like additional testing. Always happy to do so. 🙌

@ascorbic ascorbic left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! And thanks @masonjames for the review

@ascorbic ascorbic merged commit a9e9dde into main Jul 6, 2026
59 checks passed
@ascorbic ascorbic deleted the feature/media-usage-index-core branch July 6, 2026 09:27
@github-actions github-actions Bot added review/approved Approved; no new commits since and removed review/needs-rereview Author pushed changes since the last review labels Jul 6, 2026
@emdashbot emdashbot Bot mentioned this pull request Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants