[pull] master from apify:master#244
Merged
Merged
Conversation
…3851) ## What Backports the schema-based router to the stable v3 line as a fully **opt-in, non-breaking** feature, complementing the compile-time route-map typing from #3747. A router can now take a [Standard Schema](https://standardschema.dev) (Zod, Valibot, ArkType, …) per label, which drives **both** the inferred `request.userData` type **and** its runtime validation. ```ts import { createHttpRouter, HttpCrawler } from 'crawlee'; import { z } from 'zod'; const router = createHttpRouter({ LIST: z.object({ offset: z.number() }), POKEMON: z.object({ id: z.string() }), }); router.addHandler('POKEMON', async ({ request }) => { request.userData.id; // string — inferred from the schema, validated before the handler runs }); ``` ## How - **Schema router (`@crawlee/core`)** — a third `Router.create` / `createXRouter` overload accepts a `{ label: schema }` map. The route map (`Routes`) is inferred from the schemas via `RoutesFromSchemas`. At dispatch, `request.userData` is validated against the label's schema and replaced with the parsed (coerced) value before the handler runs. - **Validation on add** — the crawler validates `userData` against the label's schema on the add paths it owns: `crawler.addRequests` (and thus `crawler.run` / `context.addRequests`) and `context.enqueueLinks`. A bad shape fails fast at insertion instead of deep inside a handler. This lives at the crawler level — the storage layer (`RequestProvider`) stays decoupled from the router — so a direct `requestQueue.addRequest` that bypasses the crawler is validated at dispatch rather than on add. - A mismatch throws the new non-retryable `RequestValidationError` (re-running the request would fail identically). Everything is opt-in and backwards compatible: a router created without a schema map behaves exactly as before, with no runtime cost, and labels with no registered schema are left untouched. The only difference vs. the v4 variant (#3748) is that `crawler.run([...])` inputs are not type-checked against the route map here — that needs a crawler-level generic, which is a breaking change kept for v4. Because this turns the v3 router work into a runtime feature rather than a type-only patch, it ships as a **minor** (3.18) with its own docs snapshot — so this PR also reverts the `version-3.17` snapshot edits that #3747 made (a frozen snapshot shouldn't have been touched) and documents the schema feature in the live TypeScript guide instead. Relates to #3082
…#3865) `StagehandCrawler` intermittently fails with `Failed to resolve V3 Page from Playwright page` when a page's AI methods (`extract()`/`act()`/`observe()`) run right after the page is created — most visibly under concurrency, where it looks like a "poisoned" browser instance whose every request keeps failing. Stagehand learns about pages from CDP `Target.attachedToTarget` events on its own connection and maps them by main frame id. Because we create pages through a separate `connectOverCDP()` handle, `context.newPage()` can resolve before Stagehand has processed that event, leaving the page unresolvable for a short window. Stagehand's own `newPage()` polls for exactly this reason. The fix makes `_newPage()` wait until Stagehand has registered the page (resolving it by main frame id) before returning it, with a 10s timeout. If registration never happens — e.g. Stagehand couldn't install its helper script into the page's CDP session — the page is closed and an error is thrown rather than handed back in an unusable state.
…#3864) Fixes platform-specific path splitting in FS KVS error message. --------- Signed-off-by: Anas Khan <83116240+anxkhn@users.noreply.github.com> Co-authored-by: Jindřich Bär <jindrichbar@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )