ci(e2e): harden staging e2e against deterministic and flaky reds by jacekradko · Pull Request #8756 · clerk/javascript

jacekradko · 2026-06-05T02:44:46Z

Fixing the deterministic failures in staging e2e.

Two always-failing tests simply cannot pass on staging today, and no retry helps. whatsapp-phone-code waits on a WhatsApp button the staging instance never renders (the channel isn't enabled there), and custom-pages "survives a parent rerender" validates an unreleased @clerk/react fix (#8604) while the staging leg installs published @latest. Both now skip only where they can't pass (staging, and E2E_SDK_SOURCE=latest respectively); PR CI still runs them against branch builds, so coverage there is unchanged.

The change that most deserves a look is concurrency. It was keyed on ref (effectively always main) with cancel-in-progress, so every new staging deploy cancelled the in-flight run and no commit could report its own result. It now keys on the clerk_go commit SHA, falling back to run_id for manual dispatches. The rest is a timeout bump above the 25-minute test step, plus a JSON Playwright report upload for later failure classification.

Deliberately out of scope and coming next: making validate-staging-instances actually gate, a small @smoke gating leg with the full suite informational, and reducing FAPI load on the shared with-email-codes instance.

Summary by CodeRabbit

Chores
- Improved CI concurrency to avoid canceling distinct staging deploys and added a small changeset entry.
- Made integration test timeouts configurable with a longer default for CI.
Tests
- Enhanced CI test reporting by producing JSON reports for non-cancelled runs.
- Made E2E tests more resilient: added environment-aware skips to avoid deterministic failures on staging and silenced related lint warnings.

changeset-bot · 2026-06-05T02:44:51Z

🦋 Changeset detected

Latest commit: 7b59e11

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 0 packages

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

vercel · 2026-06-05T02:44:53Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
clerk-js-sandbox	Ready	Preview, Comment	Jun 5, 2026 11:41am

coderabbitai · 2026-06-05T02:45:01Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Repository UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 922e17d8-743e-415c-a5ef-e68ae29d3115

📥 Commits

Reviewing files that changed from the base of the PR and between fc18bdf and 7b59e11.

📒 Files selected for processing (5)

.changeset/staging-e2e-resilience-p0.md
.github/workflows/e2e-staging.yml
integration/playwright.config.ts
integration/tests/custom-pages.test.ts
integration/tests/whatsapp-phone-code.test.ts

✅ Files skipped from review due to trivial changes (1)

.changeset/staging-e2e-resilience-p0.md

🚧 Files skipped from review as they are similar to previous changes (4)

integration/tests/whatsapp-phone-code.test.ts
integration/tests/custom-pages.test.ts
integration/playwright.config.ts
.github/workflows/e2e-staging.yml

📝 Walkthrough

Walkthrough

Staging E2E reliability: change workflow concurrency key to commit SHA, increase integration test timeout, upload Playwright JSON reports on non-cancelled runs, add CI reporter, and add env-gated skips and lint disables in Playwright tests.

Changes

E2E Staging Test Resilience

Layer / File(s)	Summary
Workflow resilience: concurrency, timeout, and artifact reporting `.github/workflows/e2e-staging.yml`, `.changeset/staging-e2e-resilience-p0.md`	`concurrency.group` now keys on `clerk-go-commit-sha` (with `run_id` fallback). Integration-tests `timeout-minutes` default uses `vars.TIMEOUT_MINUTES_LONG` or falls back to `30`. Adds `Upload Playwright JSON report` step for `playwright-report/results.json` on non-cancelled runs. Changeset added.
Test reporting and environment-based skipping `integration/playwright.config.ts`, `integration/tests/custom-pages.test.ts`, `integration/tests/whatsapp-phone-code.test.ts`	Playwright config emits dual reporters (list + JSON) in CI writing `playwright-report/results.json`. ESLint `turbo/no-undeclared-env-vars` disables added to tests. `custom-pages` test skips when `E2E_SDK_SOURCE === 'latest'`. WhatsApp phone-code suite is skipped on staging when `E2E_STAGING === '1'`.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested labels

integration

Suggested reviewers

wobsoriano
nikosdouvlis
brkalow

Poem

🐰 I hopped through CI's gated door,
Where SHAs keep runs from cancel wars.
Time stretched long, reports now sing,
Skips in place where mocks don't bring.
Hooray — the staging meadow springs! 🥕✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically describes the main change: hardening staging e2e tests against both deterministic failures and flaky tests through skips, concurrency fixes, timeout increases, and report generation.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

pkg-pr-new · 2026-06-05T02:47:01Z

Open in StackBlitz

@clerk/astro

npm i https://pkg.pr.new/@clerk/astro@8756

@clerk/backend

npm i https://pkg.pr.new/@clerk/backend@8756

@clerk/chrome-extension

npm i https://pkg.pr.new/@clerk/chrome-extension@8756

@clerk/clerk-js

npm i https://pkg.pr.new/@clerk/clerk-js@8756

@clerk/expo

npm i https://pkg.pr.new/@clerk/expo@8756

@clerk/expo-passkeys

npm i https://pkg.pr.new/@clerk/expo-passkeys@8756

@clerk/express

npm i https://pkg.pr.new/@clerk/express@8756

@clerk/fastify

npm i https://pkg.pr.new/@clerk/fastify@8756

@clerk/hono

npm i https://pkg.pr.new/@clerk/hono@8756

@clerk/localizations

npm i https://pkg.pr.new/@clerk/localizations@8756

@clerk/nextjs

npm i https://pkg.pr.new/@clerk/nextjs@8756

@clerk/nuxt

npm i https://pkg.pr.new/@clerk/nuxt@8756

@clerk/react

npm i https://pkg.pr.new/@clerk/react@8756

@clerk/react-router

npm i https://pkg.pr.new/@clerk/react-router@8756

@clerk/shared

npm i https://pkg.pr.new/@clerk/shared@8756

@clerk/tanstack-react-start

npm i https://pkg.pr.new/@clerk/tanstack-react-start@8756

@clerk/testing

npm i https://pkg.pr.new/@clerk/testing@8756

@clerk/ui

npm i https://pkg.pr.new/@clerk/ui@8756

@clerk/upgrade

npm i https://pkg.pr.new/@clerk/upgrade@8756

@clerk/vue

npm i https://pkg.pr.new/@clerk/vue@8756

commit: 7b59e11

@latest

The staging e2e "generic" leg was red on ~100% of runs because a few independent failures sat behind an all-or-nothing gate: - whatsapp-phone-code: the WhatsApp channel is not enabled on the staging instance, so the button never renders and the suite times out every run. It also bypasses the isStagingReady graceful-skip, so skip it explicitly on staging until the channel is provisioned. - custom-pages "survives a parent rerender": validates an unreleased @clerk/react fix (#8604), but the staging leg installs published @latest, so it is deterministically red until release. Skip when E2E_SDK_SOURCE=latest; PR CI (ref builds) still covers it. - concurrency was keyed on ref (effectively always "main") with cancel-in-progress, so each new staging deploy cancelled the in-flight run and no commit could report a status. Key on the clerk_go commit instead. - raise the job timeout above the 25-minute test step so the job cap no longer kills runs mid-suite. - emit and upload a JSON Playwright report in CI so the report job can classify failures (flaky vs failed, infra vs regression) later.

wobsoriano

nice

github-actions Bot added actions integration labels Jun 5, 2026

vercel Bot deployed to Preview June 5, 2026 02:45 View deployment

This was referenced Jun 5, 2026

ci(e2e): gate staging e2e on critical staging-instance config drift #8757

Open

ci(e2e): classify staging e2e results into a structured digest #8760

Open

jacekradko force-pushed the jacek/staging-e2e-resilience-p0 branch from fc18bdf to 7b59e11 Compare June 5, 2026 11:40

vercel Bot deployed to Preview June 5, 2026 11:41 View deployment

wobsoriano approved these changes Jun 5, 2026

View reviewed changes

jacekradko merged commit 5bf43a2 into main Jun 5, 2026
48 checks passed

jacekradko deleted the jacek/staging-e2e-resilience-p0 branch June 5, 2026 16:15

jacekradko mentioned this pull request Jun 5, 2026

ci(e2e): upload staging Playwright JSON report from integration/ #8766

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci(e2e): harden staging e2e against deterministic and flaky reds#8756

ci(e2e): harden staging e2e against deterministic and flaky reds#8756
jacekradko merged 1 commit into
mainfrom
jacek/staging-e2e-resilience-p0

jacekradko commented Jun 5, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

changeset-bot Bot commented Jun 5, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Jun 5, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jun 5, 2026 •

edited

Loading

Walkthrough

Changes

Estimated code review effort

Suggested labels

Suggested reviewers

Poem

Uh oh!

pkg-pr-new Bot commented Jun 5, 2026 •

edited

Loading

Uh oh!

wobsoriano left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jacekradko commented Jun 5, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

changeset-bot Bot commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

vercel Bot commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Suggested labels

Suggested reviewers

Poem

Uh oh!

pkg-pr-new Bot commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

wobsoriano left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jacekradko commented Jun 5, 2026 •

edited by coderabbitai Bot

Loading

changeset-bot Bot commented Jun 5, 2026 •

edited

Loading

vercel Bot commented Jun 5, 2026 •

edited

Loading

coderabbitai Bot commented Jun 5, 2026 •

edited

Loading

pkg-pr-new Bot commented Jun 5, 2026 •

edited

Loading