[pull] master from apify:master - #266
Merged
Merged
Conversation
BREAKING CHANGE: The project is now native ESM without a CJS alternative. This is fine since all supported node versions allow `require(esm)`. Also all the dependencies are updated to the latest versions, including cheerio v1.
BREAKING CHANGE: The crawler following options are removed: - `handleRequestFunction` -> `requestHandler` - `handlePageFunction` -> `requestHandler` - `handleRequestTimeoutSecs` -> `requestHandlerTimeoutSecs` - `handleFailedRequestFunction` -> `failedRequestHandler`
BREAKING CHANGE: The crawling context no longer includes the `Error` object for failed requests. Use the second parameter of the `errorHandler` or `failedRequestHandler` callbacks to access the error. Previously, the crawling context extended a `Record` type, allowing to access any property. This was changed to a strict type, which means that you can only access properties that are defined in the context.
….retireOnBlockedStatusCodes` BREAKING CHANGE: `additionalBlockedStatusCodes` parameter of `Session.retireOnBlockedStatusCodes` method is removed. Use the `blockedStatusCodes` crawler option instead.
….retireOnBlockedStatusCodes` BREAKING CHANGE: `additionalBlockedStatusCodes` parameter of `Session.retireOnBlockedStatusCodes` method is removed. Use the `blockedStatusCodes` crawler option instead.
also tries to bump better-sqlite3 to latest version to have prebuilds for node 22
- closes #2479 - closes #3106 - closes #3107 - closes #3078 In my opinion, it makes a lot of sense to do the remaining changes in a separate PR. - [x] Introduce a `ContextPipeline` abstraction - [x] Update crawlers to use it - [x] Make sure that existing tests pass - [ ] Refine the `ContextPipeline.compose` signature and the semantics of `BasicCrawlerOptions.contextPipelineEnhancer` to maximize DX - [x] Write tests for the `contextPipelineEnhancer` - [x] Resolve added TODO comments (fix immediately or make issues) - [ ] Update documentation The `context-pipeline` branch introduces a fundamental architectural change to how Crawlee crawlers build and enhance the crawling context passed to request handlers. The core motivation is to fix the composition and extensibility nightmare in the current crawler hierarchy. 1. **Rigid inheritance hierarchy**: Crawlers were stuck in a brittle inheritance chain where each layer manipulated the context object while assuming that it already satisfied its final type. Multiple overrides of `BasicCrawler` lifecycle methods made the execution flow even harder to follow. 2. **Context enhancement via monkey-patching**: Manual property assignment (`crawlingContext.page = page`, `crawlingContext.$ = $`) scattered everywhere. It was a mess to follow and impossible to reason about. 3. **Cleanup coordination**: Resource cleanup was handled by separate `_cleanupContext` methods that were not co-located with the initialization. 4. **Extension mechanism was broken**: The `CrawlerExtension.use()` API tried to let you extend crawlers (the ones based on `HttpCrawler`) by overwriting properties - completely type-unsafe and fragile as hell. Introduces `ContextPipeline` - a **middleware-based composition pattern** where: - Each crawler layer defines how it enhances the context through explicit `action` functions - Cleanup logic is co-located with initialization via optional `cleanup` functions - Type safety is maintained through TypeScript generics that track context transformations - The pipeline executes middleware sequentially with proper error handling and guaranteed cleanup Declarative middleware composition with co-located cleanup: ```typescript contextPipeline.compose({ action: async (context) => ({ page, $ }), cleanup: async (context) => { await page.close(); } }) ``` The `ContextPipeline<TBase, TFinal>` tracks type transformations through the chain: ```typescript ContextPipeline<CrawlingContext, CrawlingContext> .compose<{ page: Page }>(...) // ContextPipeline<CrawlingContext, CrawlingContext & { page: Page }> .compose<{ $: CheerioAPI }>(...) // ContextPipeline<CrawlingContext, CrawlingContext & { page: Page, $: CheerioAPI }> ``` The `CrawlerExtension.use()` is gone. New approach via `contextPipelineEnhancer`: ```typescript new BasicCrawler({ contextPipelineEnhancer: (pipeline) => pipeline.compose({ action: async (context) => ({ myCustomProp: ... }) }) }) ``` The current way to express a context pipeline middleware has some shortcomings (`ContextPipeline.compose`, `BasicCrawlerOptions.contextPipelineEnhancer`). I suggest resolving this in another PR. For most legitimate use cases, this should be non-breaking. Those who extend the Crawler classes in non-trivial ways may need to adjust their code though - the non-public interface of `BasicCrawler` and `HttpCrawler` changed quite a bit. The pipeline uses `Object.defineProperties` for each middleware. Is this a serious performance consideration? --------- Co-authored-by: Martin Adámek <banan23@gmail.com>
Extracts `ProxyConfiguration` to `BasicCrawler` (related to discussion under #2917). Pass the `ProxyConfiguration` instance to the `SessionPool` for new `Session` object creation. Store and read the `ProxyInfo` from the `Session` instance instead of calling the `ProxyConfiguration` methods in the crawlers. closes #3198
Phasing out `got-scraping`-specific interfaces in favour of native `fetch` API. Related to #3071
Fixes build toolchain errors caused by the recent rebase onto the current `master` ([more details here](https://apify.slack.com/archives/C02JQSN79V4/p1764373034961859)). The largest thing is probably updating the dependency versions in `package.json` - if `turborepo` doesn't find the matching version in the local workspace, it will build against the package pulled from `npm` (which doesn't match the v4 API at this point).
Related to #3275 --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…0.0 in the camoufox template The rebase onto master carried over master's renovate bump of camoufox-js to ^0.12.0, but the template (and the repo root) still pin the Playwright version whose bundled Firefox matches camoufox-js 0.11.
…tion The rebase onto master replaced the RobotsTxtFile factory bodies with master's versions (which keep the URL for enqueue-strategy filtering), dropping the @ts-ignore comments v4 needs because robots-parser's CJS default export is not callable under nodenext module resolution.
The domain matching in the restored `filterUrl`/`matchesEnqueueStrategy` helpers (carried over from master) uses tldts, which had been dropped from the package manifest during the rebase.
- requestQueue → requestManager in enqueueLinks options - await the now-async createRequestQueueMock - sessionPoolOptions → sessionPool instance in the redirect-cookie test - config → configuration in purgeDefaultStorages options - SitemapRequestList → SitemapRequestLoader in loader tests - handleCloudflareChallenge lost its session parameter in v4 - drop duplicate imports
- markRequestHandled → markRequestAsHandled on SitemapRequestLoader - await the now-async RequestQueue.getTotalCount() - transformRequestFunction skips now report the dedicated 'transform' reason - robots.txt mock needs getCrawlDelay - statistics/session-pool single-persistence tests observe KeyValueStore.setValue instead of the persistState methods RecoverableState replaced - pass an explicit logger to Sitemap.load in the aggregated-warning test
Moves the `requestManager`-bound enqueueing logic into `BasicCrawlerContext.addRequests`, and each DOM-aware crawler now exposes its own `extractLinks()` plus an `enqueueLinks()` that composes `extractLinks` + `addRequests`. This aligns the JS implementation with what Python does, to some extent. Closes #3081
The transplanted enqueueLinks split reverted a few master-carried behaviors in BasicCrawler; this restores them on top of the new design: - stop capturing statistics before teardown again, so the crawler state is saved before the final persistence event fires (prevents double persistence) - teardown() only emits an explicit PERSIST_STATE event for externally-managed event managers, and tears the owned session pool down with persistState matching event manager ownership (an unset flag previously fell back to the `persistState = true` default, double-persisting the pool) - the enqueue limit log distinguishes an explicit `limit` from the remaining maxRequestsPerCrawl budget again - adapt the master-carried tests to the addRequests() API; drop the explicit-undefined override tests for options that no longer exist on it
… tests Follow-up to the BasicCrawler.stats → statistics rename (#4028) for two master-carried tests it could not have known about.
…atalog dependency Two upstream v4 changes landed after the last E2E run and broke the suite: - the enqueueLinks split (#4010) changed the context helper's return value to the addRequestsBatched result, so the *-enqueue-links fixtures now assert on `addedRequests` being empty instead of deep-equality with the old shape - the zod validation unification (#3935) introduced a `catalog:` dependency, which npm cannot resolve when the platform builds the actor image; the E2E package-copy step now rewrites catalog deps to their pinned versions from pnpm-workspace.yaml, the same way it already rewrites `workspace:` deps
…oudflare fixture camoufox-js releases bundle a specific Firefox build that must match the one expected by the pinned playwright version (0.11 ↔ 1.60). The fixture pinned camoufox-js ^0.12.0 next to playwright 1.60.0, so the Cloudflare challenge kept failing on the platform even with the updated challenge markup handling (#4019) in place — master validates that fix with the 0.11/1.60 pairing.
The v3.18 release blog post and the 3.18 versioned upgrading guide linked to the current (unversioned) API reference. That resolved fine on master, where the current API was 3.18, but on v4 the docs build fails: StorageClient was renamed and RequestValidationError's page moved. Version-pinned API links are the established pattern in versioned content (see the 3.17 guide).
…ixture Cloudflare serves its challenge pages with a 403 status. On v3, handleCloudflareChallenge() received the session and removed 403 from the session pool's blocked status codes itself; v4 dropped that mechanism when the hook was redesigned, so challenged requests died in throwOnBlockedRequest() on every retry and the solver only ever got a single attempt. Solving the challenge is probabilistic, which is why the fixture passes on master (where retries reach the solver) and kept failing here. blockedStatusCodes is a public crawler option in v4, so the fixture opts out of 403 explicitly. Whether handleCloudflareChallengeHook() should handle this automatically again is a follow-up design question.
impit and fs-storage-native ship platform binaries as optionalDependencies, so --omit=optional (common in v3 Docker templates) breaks the install.
…on a read-after-write race The statistics record is persisted during crawler teardown and the platform key-value store is eventually consistent, so reading it immediately after the run can miss it. That crashed the whole test with a TypeError on stats.requestsFinished (seen in cheerio-curl-impersonate-ts) even though the actor run itself succeeded. The lookup now retries for up to ~30 seconds and falls back to an empty object, so a genuinely missing record fails the assertions cleanly.
Co-authored-by: Martin Adámek <banan23@gmail.com>
…ersion The 4.0 snapshot only existed so the v4 branch site build had a default version. On master, the v4 docs are the current (next) version, labeled "4.0 (RC)", and the real 4.0 snapshot will be generated by the release workflow when 4.0.0 ships. The default docs version stays 3.18 until then.
Lost in the v4 rebase; matches the createHttpRouter/createCheerioRouter overload set. Also removes the rebase reconciliation checklist, which is fully resolved by this commit.
Canaries publish from master under the v4 dist-tag (next once 4.0.0 is stable), the version-docs snapshot job is guarded to master, and RELEASE.md now documents the branch/dist-tag matrix including the 3.x maintenance branch.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )