-
-
+
+
-
+
A web scraping and browser automation library
-
-
+
+
-
-
-
-
-
+
+
+
+
+
Crawlee covers your crawling and scraping end-to-end and **helps you build reliable scrapers. Fast.**
diff --git a/RELEASE.md b/RELEASE.md
index b221ca3f2978..31551ce20e8d 100644
--- a/RELEASE.md
+++ b/RELEASE.md
@@ -1,54 +1,48 @@
-# How to release new versions of Apify SDK
-Release of new versions is managed by GitHub Actions. On pushes to the `master` branch, prerelease versions
-are automatically produced. Latest releases are triggered manually through the GitHub release tool.
-After creating a release there, Actions will automatically produce a latest version of the package.
-
-## TLDR;
-- To **NOT** release anything on a push to `master`, add `[skip ci]` to your commit message.
-- To release `beta`, just push to `master`. If it breaks with a `Version already exists error` increment version
- in `package.json` and push again.
-- To release `latest`, go to releases on GitHub, draft and publish a release. If you don't know how, read below.
-
-## Prerelease (beta) versions
-On each push to the `master` branch, a new prerelease version is automatically built and published
-by GitHub Actions. To skip the process, add `[skip ci]` to your commit message.
-
-### Release process
-1. Actions build is triggered by a push to `master` (typically a merge of a PR).
-2. Actions lint the source code and run tests in Node.js 10, 12 and 14.
-3. If all is well, a new prerelease version is published to NPM (`${VERSION}-beta.${COUNTER}`),
- where `VERSION` is the version in package.json and `COUNTER` is a zero based index of existing prereleases.
- Example: `0.15.1-beta.3`.
-4. The package is tagged with the `beta` NPM tag and a Git tag is associated with the triggering commit.
-5. A build of Apify docker images is triggered that updates the `beta` packages to use the newly published package.
-6. All done and ready to use.
-
-### Updating a release version
-When releasing breaking changes, new features or for any other reason that requires a version bump,
-manually increment the version in the `package.json` file. Such as from `0.14.15` to `0.15.0`.
-This will automatically trigger a prerelease build with the `0.15.0-beta.0` version.
-
-### Existing versions
-Actions will not allow you to publish a prerelease of a version that's already published. For example,
-if version `0.14.15` already exists on NPM, you can no longer release a `0.14.15-beta.0` version.
-
-## Latest release
-To trigger a latest release, go to the GitHub release tool (select `releases` under `<> Code`).
-There, draft a new release, fill the form (see below) and hit `Publish release`.
-Actions will automatically release the latest version of the package.
-
-### How to fill the form
-- The version tag should be in the format `v${VERSION}` where `VERSION` is the version from `package.json`.
- Such as `v0.15.0` or `v0.16.17`.
-- The target will typically be `master`, but you can also release any previous commit by selecting it or searching
- for it by ID. This is useful when there have been some changes in master from the latest prerelease and you'd
- like to release an older prerelease version as latest. You can find the commit ID easily by searching for the
- prerelease tag.
-- The title should be the same as the version tag.
-- Typically just adding changelog to the release would be fine, but feel free to add extra information.
-
-### Release process
-Similarly to the prerelease, the latest release process:
-1. Triggers build, lints, runs tests.
-2. Publishes new package to NPM with the `latest` tag and the version from package.json. Such as `0.15.1`.
-3. A build and deploy of Apify docker images is triggered with the `latest` tag.
+# How Crawlee releases work
+
+Releases are managed by GitHub Actions and lerna. There are two release lines:
+
+| line | branch | stable dist-tag | canary dist-tag |
+|---|---|---|---|
+| v4 (current development) | `master` | `latest` (from 4.0.0 on) | `v4` now, `next` once 4.0.0 is stable |
+| v3 (maintenance) | `3.x` | `latest` until 4.0.0 ships, then `latest-v3` | `next` now, `next-v3` once 4.0.0 is stable |
+
+Release candidates for the next major are published from `master` under the `rc` dist-tag.
+
+## Canary releases
+
+Every push to `master` or `3.x` (that is not a docs change and does not say `[skip ci]`)
+triggers the `release_next` job in `test-ci.yml`. It builds the packages, derives a canary
+version from `lerna.json` (`scripts/copy.ts --canary`) and invokes `publish-to-npm.yml`,
+which publishes all packages under the branch's canary dist-tag (see the table above; the
+tag is hardcoded in the branch's `publish:next` script in `package.json`). No commit or git
+tag is created for canaries.
+
+## Stable releases
+
+Trigger the `Release @latest` workflow (`release.yml`) manually via `workflow_dispatch`
+**from the branch you want to release**:
+
+- `master` + `bump: major` is how `4.0.0` goes out (lerna bumps `3.18.x` → `4.0.0`,
+ publishes to `latest`).
+- `3.x` + `bump: patch`/`minor` ships v3 maintenance releases.
+
+The workflow runs the tests, bumps versions via `lerna version` (conventional-commits
+changelog, GitHub release, git tag), pins internal dependency versions, publishes through
+`publish-to-npm.yml` with `dist-tag: prod`, and triggers Apify Docker image builds.
+
+On minor/major releases from `master`, the `version-docs` job snapshots the current docs
+into `website/versioned_docs` (Docusaurus `docs:version` + `api:version`). The job never
+runs for maintenance branches — 3.x docs live in the `version-3.18` snapshot on `master`.
+
+## RC releases
+
+Dispatch `publish-to-npm.yml` from `master` with `dist-tag: rc`. This publishes
+`4.0.0-rc.N` under the `rc` dist-tag and pushes a `v4.0.0-rc.N` git tag pointing at the
+released commit; no version-bump commit lands on the branch.
+
+## v4.0.0 release day
+
+See the tracking issue for the ordered checklist (publish 4.0.0 to `latest`, switch
+master canaries `v4` → `next` and `--canary=major` → `--canary=patch`, switch 3.x to
+`latest-v3`/`next-v3`, drop the RC docs label).
diff --git a/biome.json b/biome.json
deleted file mode 100644
index 8c23acdb00cb..000000000000
--- a/biome.json
+++ /dev/null
@@ -1,28 +0,0 @@
-{
- "formatter": {
- "includes": [
- "**",
- "!**/website/**",
- "!**/packages/**/*/dist/**",
- "!**/package.json",
- "!**/lerna.json",
- "!**/scripts/actions/docker-images/state.json"
- ],
- "formatWithErrors": true
- },
- "javascript": {
- "formatter": {
- "quoteStyle": "single",
- "semicolons": "always",
- "trailingCommas": "all",
- "lineWidth": 120,
- "indentStyle": "space",
- "indentWidth": 4,
- "quoteProperties": "preserve",
- "lineEnding": "lf"
- }
- },
- "linter": {
- "enabled": false
- }
-}
diff --git a/docs/deployment/apify_platform_init_exit.ts b/docs/deployment/apify_platform_init_exit.ts
index 49a10f100f23..8058d81c457f 100644
--- a/docs/deployment/apify_platform_init_exit.ts
+++ b/docs/deployment/apify_platform_init_exit.ts
@@ -13,7 +13,7 @@ const crawler = new CheerioCrawler({
// Add URLs that match the provided pattern.
await enqueueLinks({
- globs: ['https://www.iana.org/*'],
+ include: ['https://www.iana.org/*'],
});
// Save extracted data to dataset.
diff --git a/docs/deployment/apify_platform_main.ts b/docs/deployment/apify_platform_main.ts
index a338047e86ea..507c7fe1c6e2 100644
--- a/docs/deployment/apify_platform_main.ts
+++ b/docs/deployment/apify_platform_main.ts
@@ -12,7 +12,7 @@ await Actor.main(async () => {
// Add URLs that match the provided pattern.
await enqueueLinks({
- globs: ['https://www.iana.org/*'],
+ include: ['https://www.iana.org/*'],
});
// Save extracted data to dataset.
diff --git a/docs/examples/cheerio_crawler.ts b/docs/examples/cheerio_crawler.ts
index 7a308a6f0ba5..d32dc858c69e 100644
--- a/docs/examples/cheerio_crawler.ts
+++ b/docs/examples/cheerio_crawler.ts
@@ -9,7 +9,7 @@ log.setLevel(LogLevel.DEBUG);
// that automatically loads the URLs and parses their HTML using the cheerio library.
const crawler = new CheerioCrawler({
// The crawler downloads and processes the web pages in parallel, with a concurrency
- // automatically managed based on the available system memory and CPU (see AutoscaledPool class).
+ // automatically managed based on the available system memory and CPU (see ConcurrencySystem class).
// Here we define some hard limits for the concurrency.
minConcurrency: 10,
maxConcurrency: 50,
diff --git a/docs/examples/crawl_all_links.mdx b/docs/examples/crawl_all_links.mdx
index 371b834ca7c9..efd2786af85a 100644
--- a/docs/examples/crawl_all_links.mdx
+++ b/docs/examples/crawl_all_links.mdx
@@ -17,7 +17,7 @@ as the crawler navigates from page to page. This example can also be used to fin
:::tip
-If no options are given, by default the method will only add links that are under the same subdomain. This behavior can be controlled with the `strategy`
+If no options are given, by default the method will only add links that are under the same subdomain. This behavior can be controlled with the `strategy`
option. You can find more info about this option in the [`Crawl relative links`](./crawl-relative-links) examples.
:::
diff --git a/docs/examples/crawl_some_links.mdx b/docs/examples/crawl_some_links.mdx
index fb9cde71600e..11c6566790b3 100644
--- a/docs/examples/crawl_some_links.mdx
+++ b/docs/examples/crawl_some_links.mdx
@@ -7,7 +7,7 @@ import RunnableCodeBlock from '@site/src/components/RunnableCodeBlock';
import ApiLink from '@site/src/components/ApiLink';
import CrawlSource from '!!raw-loader!roa-loader!./crawl_some_links.ts';
-This `CheerioCrawler` example uses the `globs` property in the `enqueueLinks()` method to only add links to the `RequestQueue` queue if they match the specified pattern.
+This `CheerioCrawler` example uses the `include` property in the `enqueueLinks()` method to only add links to the `RequestQueue` queue if they match the specified pattern.
{CrawlSource}
diff --git a/docs/examples/crawl_some_links.ts b/docs/examples/crawl_some_links.ts
index 51912bb10f67..111f89165c0f 100644
--- a/docs/examples/crawl_some_links.ts
+++ b/docs/examples/crawl_some_links.ts
@@ -9,7 +9,7 @@ const crawler = new CheerioCrawler({
log.info(request.url);
// Add some links from page to the crawler's RequestQueue
await enqueueLinks({
- globs: ['http?(s)://crawlee.dev/*/*'],
+ include: ['http?(s)://crawlee.dev/*/*'],
});
},
});
diff --git a/docs/examples/crawler-plugins/playwright-extra.ts b/docs/examples/crawler-plugins/playwright-extra.ts
index 8cb6f75386cc..2121fc3df2eb 100644
--- a/docs/examples/crawler-plugins/playwright-extra.ts
+++ b/docs/examples/crawler-plugins/playwright-extra.ts
@@ -57,7 +57,7 @@ const crawler = new PlaywrightCrawler({
selector: '.morelink',
});
- if (infos.processedRequests.length === 0) log.info(`${request.url} is the last page!`);
+ if (infos.addedRequests.length === 0) log.info(`${request.url} is the last page!`);
},
// This function is called if the page processing failed more than maxRequestRetries+1 times.
diff --git a/docs/examples/crawler-plugins/puppeteer-extra.ts b/docs/examples/crawler-plugins/puppeteer-extra.ts
index a0a197f3890f..058119719acb 100644
--- a/docs/examples/crawler-plugins/puppeteer-extra.ts
+++ b/docs/examples/crawler-plugins/puppeteer-extra.ts
@@ -55,7 +55,7 @@ const crawler = new PuppeteerCrawler({
selector: '.morelink',
});
- if (infos.processedRequests.length === 0) log.info(`${request.url} is the last page!`);
+ if (infos.addedRequests.length === 0) log.info(`${request.url} is the last page!`);
},
// This function is called if the page processing failed more than maxRequestRetries+1 times.
diff --git a/docs/examples/file_download.ts b/docs/examples/file_download.ts
index a6b42555e9ba..4ec682ea7002 100644
--- a/docs/examples/file_download.ts
+++ b/docs/examples/file_download.ts
@@ -2,11 +2,11 @@ import { FileDownload } from 'crawlee';
// Create a FileDownload - a custom crawler instance that will download files from URLs.
const crawler = new FileDownload({
- async requestHandler({ body, request, contentType, getKeyValueStore }) {
+ async requestHandler({ request, response, contentType, getKeyValueStore }) {
const url = new URL(request.url);
const kvs = await getKeyValueStore();
- await kvs.setValue(url.pathname.replace(/\//g, '_'), body, { contentType: contentType.type });
+ await kvs.setValue(url.pathname.replace(/\//g, '_'), response.body, { contentType: contentType.type });
},
});
diff --git a/docs/examples/file_download_stream.ts b/docs/examples/file_download_stream.ts
index a7f39a70f59a..9517531b5bd2 100644
--- a/docs/examples/file_download_stream.ts
+++ b/docs/examples/file_download_stream.ts
@@ -1,9 +1,9 @@
-import { pipeline, Transform } from 'stream';
+import { pipeline, Transform } from 'node:stream';
-import { FileDownload, type Log } from 'crawlee';
+import { FileDownload, type CrawleeLogger } from 'crawlee';
// A sample Transform stream logging the download progress.
-function createProgressTracker({ url, log, totalBytes }: { url: URL; log: Log; totalBytes: number }) {
+function createProgressTracker({ url, log, totalBytes }: { url: URL; log: CrawleeLogger; totalBytes: number }) {
let downloadedBytes = 0;
return new Transform({
@@ -23,32 +23,27 @@ function createProgressTracker({ url, log, totalBytes }: { url: URL; log: Log; t
// Create a FileDownload - a custom crawler instance that will download files from URLs.
const crawler = new FileDownload({
- async streamHandler({ stream, request, log, getKeyValueStore }) {
+ async requestHandler({ response, request, log, getKeyValueStore }) {
const url = new URL(request.url);
log.info(`Downloading ${url} to ${url.pathname.replace(/\//g, '_')}...`);
- await new Promise((resolve, reject) => {
- // With the 'response' event, we have received the headers of the response.
- stream.on('response', async (response) => {
- const kvs = await getKeyValueStore();
- await kvs.setValue(
- url.pathname.replace(/\//g, '_'),
- pipeline(
- stream,
- createProgressTracker({ url, log, totalBytes: Number(response.headers['content-length']) }),
- (error) => {
- if (error) reject(error);
- },
- ),
- { contentType: response.headers['content-type'] },
- );
-
- log.info(`Downloaded ${url} to ${url.pathname.replace(/\//g, '_')}.`);
-
- resolve();
- });
- });
+ if (!response.body) return;
+
+ const kvs = await getKeyValueStore();
+ await kvs.setValue(
+ url.pathname.replace(/\//g, '_'),
+ pipeline(
+ response.body,
+ createProgressTracker({ url, log, totalBytes: Number(response.headers.get('content-length')) }),
+ (error) => {
+ if (error) log.error(`Failed to download ${url}: ${error.message}`);
+ },
+ ),
+ response.headers.get('content-type') ? { contentType: response.headers.get('content-type')! } : {},
+ );
+
+ log.info(`Downloaded ${url} to ${url.pathname.replace(/\//g, '_')}.`);
},
});
diff --git a/docs/examples/http_crawler.ts b/docs/examples/http_crawler.ts
index add990c59076..7f661e116dad 100644
--- a/docs/examples/http_crawler.ts
+++ b/docs/examples/http_crawler.ts
@@ -9,7 +9,7 @@ log.setLevel(LogLevel.DEBUG);
// that automatically loads the URLs and saves their HTML.
const crawler = new HttpCrawler({
// The crawler downloads and processes the web pages in parallel, with a concurrency
- // automatically managed based on the available system memory and CPU (see AutoscaledPool class).
+ // automatically managed based on the available system memory and CPU (see ConcurrencySystem class).
// Here we define some hard limits for the concurrency.
minConcurrency: 10,
maxConcurrency: 50,
diff --git a/docs/examples/jsdom_crawler.ts b/docs/examples/jsdom_crawler.ts
index 6db02c4ef62b..ed5b8a8cfd4f 100644
--- a/docs/examples/jsdom_crawler.ts
+++ b/docs/examples/jsdom_crawler.ts
@@ -9,7 +9,7 @@ log.setLevel(LogLevel.DEBUG);
// that automatically loads the URLs and parses their HTML using the jsdom library.
const crawler = new JSDOMCrawler({
// The crawler downloads and processes the web pages in parallel, with a concurrency
- // automatically managed based on the available system memory and CPU (see AutoscaledPool class).
+ // automatically managed based on the available system memory and CPU (see ConcurrencySystem class).
// Here we define some hard limits for the concurrency.
minConcurrency: 10,
maxConcurrency: 50,
diff --git a/docs/examples/playwright_crawler.ts b/docs/examples/playwright_crawler.ts
index 9595a021b31a..e8861322c646 100644
--- a/docs/examples/playwright_crawler.ts
+++ b/docs/examples/playwright_crawler.ts
@@ -47,7 +47,7 @@ const crawler = new PlaywrightCrawler({
selector: '.morelink',
});
- if (infos.processedRequests.length === 0) log.info(`${request.url} is the last page!`);
+ if (infos.addedRequests.length === 0) log.info(`${request.url} is the last page!`);
},
// This function is called if the page processing failed more than maxRequestRetries+1 times.
diff --git a/docs/examples/puppeteer_crawler.ts b/docs/examples/puppeteer_crawler.ts
index 6172ea0f272e..1f365f5c743f 100644
--- a/docs/examples/puppeteer_crawler.ts
+++ b/docs/examples/puppeteer_crawler.ts
@@ -47,7 +47,7 @@ const crawler = new PuppeteerCrawler({
selector: '.morelink',
});
- if (infos.processedRequests.length === 0) log.info(`${request.url} is the last page!`);
+ if (infos.addedRequests.length === 0) log.info(`${request.url} is the last page!`);
},
// This function is called if the page processing failed more than maxRequestRetries+1 times.
diff --git a/docs/examples/puppeteer_recursive_crawl.ts b/docs/examples/puppeteer_recursive_crawl.ts
index ad48b324796b..dde92685ea5f 100644
--- a/docs/examples/puppeteer_recursive_crawl.ts
+++ b/docs/examples/puppeteer_recursive_crawl.ts
@@ -6,7 +6,7 @@ const crawler = new PuppeteerCrawler({
log.info(`Title of ${request.url}: ${title}`);
await enqueueLinks({
- globs: ['http?(s)://www.iana.org/**'],
+ include: ['http?(s)://www.iana.org/**'],
});
},
maxRequestsPerCrawl: 10,
diff --git a/docs/examples/skip-navigation.ts b/docs/examples/skip-navigation.ts
index 0bbde53c1375..867fb473271e 100644
--- a/docs/examples/skip-navigation.ts
+++ b/docs/examples/skip-navigation.ts
@@ -1,17 +1,22 @@
import { PlaywrightCrawler, KeyValueStore } from 'crawlee';
// Create a key value store for all images we find
-const imageStore = await KeyValueStore.open('images');
+const imageStore = await KeyValueStore.open({ name: 'images' });
const crawler = new PlaywrightCrawler({
async requestHandler({ request, page, sendRequest }) {
// The request should have the navigation skipped
if (request.skipNavigation) {
// Request the image and get its buffer back
- const imageResponse = await sendRequest({ responseType: 'buffer' });
-
- // Save the image in the key-value store
- await imageStore.setValue(`${request.userData.key}.png`, imageResponse.body);
+ const imageResponse = await sendRequest();
+
+ // Saves the image in the key-value store.
+ //
+ // Note: For large-scale file downloads, consider using FileDownload crawler:
+ // https://crawlee.dev/js/api/http-crawler/class/FileDownload
+ await imageStore.setValue(`${request.userData.key}.svg`, await imageResponse.bytes(), {
+ contentType: 'image/svg+xml',
+ });
// Prevent executing the rest of the code as we do not need it
return;
diff --git a/docs/experiments/request_locking.mdx b/docs/experiments/request_locking.mdx
deleted file mode 100644
index 46ad112f48c1..000000000000
--- a/docs/experiments/request_locking.mdx
+++ /dev/null
@@ -1,146 +0,0 @@
----
-id: experiments-request-locking
-title: Request Locking
-description: Parallelize crawlers with ease using request locking
----
-
-import ApiLink from '@site/src/components/ApiLink';
-
-:::tip Release announcement
-
-As of **May 2024** (`crawlee` version `3.10.0`), this experiment is now enabled by default! With that said, if you encounter issues you can:
-
-- set `requestLocking` to `false` in the `experiments` object of your crawler options
-- update all imports of `RequestQueue` to `RequestQueueV1`
-- open an issue on our [GitHub repository](https://github.com/apify/crawlee)
-
-The content below is kept for documentation purposes.
-If you're interested in the changes, you can read the [blog post about the new Request Queue storage system on the Apify blog](https://blog.apify.com/new-apify-request-queue/).
-
-:::
-
----
-
-:::caution
-
-This is an experimental feature. While we welcome testers, keep in mind that it is currently not recommended to use this in production.
-
-The API is subject to change, and we might introduce breaking changes in the future.
-
-Should you be using this, feel free to open issues on our [GitHub repository](https://github.com/apify/crawlee), and we'll take a look.
-
-:::
-
-Starting with `crawlee` version `3.5.5`, we have introduced a new crawler option that lets you enable using a new request locking
-API. With this API, you will be able to pass a `RequestQueue` to multiple crawlers to parallelize the crawling process.
-
-:::info Keep in mind
-
-The request queue that supports request locking is currently exported via the `RequestQueueV2` class. Once the experiment is over, this class will replace
-the current `RequestQueue` class
-
-:::
-
-## How to enable the experiment
-
-### In crawlers
-
-:::note
-
-This example shows how to enable the experiment in the `CheerioCrawler`,
-but you can apply this to any crawler type.
-
-:::
-
-```ts
-import { CheerioCrawler } from 'crawlee';
-
-const crawler = new CheerioCrawler({
- // highlight-next-line
- experiments: {
- // highlight-next-line
- requestLocking: true,
- // highlight-next-line
- },
- async requestHandler({ $, request }) {
- const title = $('title').text();
- console.log(`The title of "${request.url}" is: ${title}.`);
- },
-});
-
-await crawler.run(['https://crawlee.dev']);
-```
-
-### Outside crawlers (to setup your own request queue that supports locking)
-
-Previously, you would import `RequestQueue` from `crawlee`. To switch to the queue that supports locking, you need to import `RequestQueueV2` instead.
-
-```ts
-// highlight-next-line
-import { RequestQueueV2 } from 'crawlee';
-
-const queue = await RequestQueueV2.open('my-locking-queue');
-await queue.addRequests([
- { url: 'https://crawlee.dev' },
- { url: 'https://crawlee.dev/js/docs' },
- { url: 'https://crawlee.dev/js/api' },
-]);
-```
-
-### Using the new request queue in crawlers
-
-If you make your own request queue that supports locking, you will also need to enable the experiment in your crawlers.
-
-:::danger
-
-If you do not enable the experiment, you will receive a runtime error and the crawler will not start.
-
-:::
-
-```ts
-import { CheerioCrawler, RequestQueueV2 } from 'crawlee';
-
-// highlight-next-line
-const queue = await RequestQueueV2.open('my-locking-queue');
-
-const crawler = new CheerioCrawler({
- // highlight-next-line
- experiments: {
- // highlight-next-line
- requestLocking: true,
- // highlight-next-line
- },
- // highlight-next-line
- requestQueue: queue,
- async requestHandler({ $, request }) {
- const title = $('title').text();
- console.log(`The title of "${request.url}" is: ${title}.`);
- },
-});
-
-await crawler.run();
-```
-
-## Other changes
-
-:::info
-
-This section is only useful if you're a tinkerer and want to see what's going on under the hood.
-
-:::
-
-In order to facilitate the new request locking API, as well as keep both the current request queue logic and the new, locking based request queue
-logic, we have implemented a common starting point called `RequestProvider`.
-
-This class implements almost all functions by default, but expects you, the developer, to implement the following methods:
-`fetchNextRequest` and `ensureHeadIsNotEmpty`. These methods are responsible for loading and returning requests to process,
-and tell crawlers if there are more requests to process.
-
-You can use this base class to implement your own request providers if you need to fetch requests from a different source.
-
-:::tip
-
-We recommend you use TypeScript when implementing your own request provider, as it comes with suggestions for the abstract methods, as well as
-giving you the exact types you need to return.
-
-:::
diff --git a/docs/experiments/systemInfoV2.mdx b/docs/experiments/systemInfoV2.mdx
deleted file mode 100644
index fb9a481c45f2..000000000000
--- a/docs/experiments/systemInfoV2.mdx
+++ /dev/null
@@ -1,95 +0,0 @@
----
-id: experiments-system-information-v2
-title: System Information V2
-description: Improved autoscaling through cgroup aware metric collection.
----
-
-import ApiLink from '@site/src/components/ApiLink';
-
-:::caution
-
-This is an experimental feature. While we welcome testers, keep in mind that it is currently not recommended to use this in production.
-
-The API is subject to change, and we might introduce breaking changes in the future.
-
-Should you be using this, feel free to open issues on our [GitHub repository](https://github.com/apify/crawlee), and we'll take a look.
-
-:::
-
-Starting with the newest `crawlee` beta, we have introduced a new crawler option that enables an improved metric collection system.
-This new system should collect cpu and memory metrics more accurately in containerised environments by checking for cgroup enforce limits.
-
-## How to enable the experiment
-
-:::note
-
-This example shows how to enable the experiment in the `CheerioCrawler`,
-but you can apply this to any crawler type.
-
-:::
-
-```ts
-import { CheerioCrawler, Configuration } from 'crawlee';
-
-Configuration.set('systemInfoV2', true);
-
-const crawler = new CheerioCrawler({
- async requestHandler({ $, request }) {
- const title = $('title').text();
- console.log(`The title of "${request.url}" is: ${title}.`);
- },
-});
-
-await crawler.run(['https://crawlee.dev']);
-```
-
-## Other changes
-
-:::info
-
-This section is only useful if you're a tinkerer and want to see what's going on under the hood.
-
-:::
-
-The existing solution checked the bare metal metrics for how much cpu and memory was being used and how much headroom was available.
-This is an intuitive solution but unfortunately doesnt account for when there is an external limit on the amount of resources a process can consume.
-This is often the case in containerized environments where each container will have a quota for its cpu and memory usage.
-
-This experiment attempts to address this issue by introducing a new `isContainerized()` utility function and changing the way resources are collected
-when a container is detected.
-
-:::note
-
-This `isContainerized()` function is very similar to the existing `isDocker()` function however for now they both work side by side.
-If this experiment is successful, `isDocker()` may eventually be deprecated in favour of `isContainerized()`.
-
-:::
-
-### Cgroup detection
-
-On linux, to detect if cgroup is available, we check if there is a directory at `/sys/fs/cgroup`.
-If the directory exists, a version of cgroup is installed.
-Next we check the version of cgroup installed by checking for a directory at `/sys/fs/cgroup/memory/`.
-If it exists, cgroup V1 is installed. If it is missing, it is assumed cgroup V2 is installed.
-
-### CPU metric collection
-
-The existing solution worked by checking the fraction of cpu idle ticks to the total number of cpu ticks since the last profile.
-If 100000 ticks elapse and 5000 were idle, the cpu is at 95% utilisation.
-
-In this experiment, the method of cpu load calculation depends on the result of `isContainerized()` or if set, the `CRAWLEE_CONTAINERIZED` environment variable.
-If `isContainerized()` returns true, the new cgroup aware metric collection will be used over the "bare metal" numbers.
-This works by inspecting the `/sys/fs/cgroup/cpuacct/cpuacct.usage`, `/sys/fs/cgroup/cpu/cpu.cfs_quota_us` and `/sys/fs/cgroup/cpu/cpu.cfs_period_us`
-files for cgroup V1 and the `/sys/fs/cgroup/cpu.stat` and `/sys/fs/cgroup/cpu.max` files for cgroup V2.
-The actual cpu usage figure is calculated in the same manner as the "bare metal" figure by comparing the total number of ticks elapsed to the number
-of idle ticks between profiles but by using the figures from the cgroup files.
-If no cgroup quota is enforced, the "bare metal" numbers will be used.
-
-### Memory metric collection
-
-The existing solution was already cgroup aware however an improvement has been made to memory metric collection when running on windows.
-The existing solution used an external package `apify/ps-tree` to find the amount of memory crawlee and any child processes were using.
-On Windows, this package used the depreciated "WMIC" command line utility to determine memory usage.
-
-In this experiment, `apify/ps-tree` has been removed and replaced by the `packages/utils/src/internals/ps-tree.ts` file. This works in much the
-same manner however, instead of using "WMIC", it uses "powershell" to collect the same data.
\ No newline at end of file
diff --git a/docs/guides/avoid_blocking.mdx b/docs/guides/avoid_blocking.mdx
index ed10846f51e6..f2acff31c25f 100644
--- a/docs/guides/avoid_blocking.mdx
+++ b/docs/guides/avoid_blocking.mdx
@@ -4,6 +4,8 @@ title: Avoid getting blocked
description: How to avoid getting blocked when scraping
---
+import ApiLink from '@site/src/components/ApiLink';
+
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
import CodeBlock from '@theme/CodeBlock';
@@ -18,6 +20,8 @@ A scraper might get blocked for numerous reasons. Let's narrow it down to the tw
Browser fingerprint is a collection of browser attributes and significant features that can show if our browser is a bot or a real user. Moreover, most browsers have these unique features that allow the website to track the browser even within different IP addresses. This is the main reason why scrapers should change browser fingerprints while doing browser-based scraping. In return, it should significantly reduce the blocking.
+The two are not handled separately. In Crawlee a `Session` ties an IP, a cookie jar, and a fingerprint together into one consistent identity, and the `SessionPool` rotates those identities as a unit — so a fresh fingerprint always arrives with a fresh IP. This guide covers the fingerprint half; see the [session management guide](./session-management) for how to control the rotation, and the [proxy management guide](./proxy-management) for the IP half.
+
## Using browser fingerprints
Changing browser fingerprints can be a tedious job. Luckily, Crawlee provides this feature with zero configuration necessary - the usage of fingerprints is enabled by default and available in `PlaywrightCrawler` and `PuppeteerCrawler`. So whenever we build a scraper that is using one of these crawlers - the fingerprints are going to be generated for the default browser and the operating system out of the box.
@@ -41,7 +45,7 @@ In certain cases we want to narrow down the fingerprints used - e.g. specify a c
## Disabling browser fingerprints
-On the contrary, sometimes we want to entirely disable the usage of browser fingerprints. This is easy to do with Crawlee too. All we have to do is set the `useFingerprints` option of the `browserPoolOptions` to `false`:
+On the contrary, sometimes we want to entirely disable the usage of browser fingerprints. This is easy to do with Crawlee too. All we have to do is build the crawler's browser pool with `useFingerprints` set to `false`:
@@ -56,9 +60,38 @@ On the contrary, sometimes we want to entirely disable the usage of browser fing
+## Fingerprints for HTTP crawlers
+
+Every session carries a lightweight fingerprint hint — a `browser`, `platform`, and `device` triple — that the request's HTTP client receives and applies on a best-effort basis.
+By default each session is given a realistic, randomized fingerprint (the host operating system as `platform`, with a plausible `browser`/`device` for it), and it rotates with the session just like the IP and cookies do.
+
+How much of the hint is used depends on the client. The [`impit`](impit-http-client) HTTP client maps the session's `browser` hint to a matching TLS and HTTP impersonation profile,
+so the connection's low-level signature lines up with the headers being sent.
+
+The *same* hint also drives browser crawlers, where it seeds the generated browser fingerprint. The hint only fixes the broad strokes — the browser family, operating system, and device — so a session presents a coherent profile, but it does not make the two backends produce byte-identical fingerprints: `impit` and a real browser will still differ in the finer details (a slightly different user-agent string, for example).
+
+You can pin the fingerprint explicitly through `sessionOptions` when you need a specific profile:
+
+```js
+import { CheerioCrawler, SessionPool } from 'crawlee';
+import { ImpitHttpClient } from '@crawlee/impit-client';
+
+const crawler = new CheerioCrawler({
+ httpClient: new ImpitHttpClient(),
+ sessionPool: new SessionPool({
+ sessionOptions: {
+ fingerprint: { browser: 'firefox', platform: 'windows', device: 'desktop' },
+ },
+ }),
+ requestHandler: async ({ $ }) => {
+ // requests impersonate desktop Firefox on Windows
+ },
+});
+```
+
## Camoufox
-For some protections, using our integrated solutions is not enough, one example could be the Cloudflare challenge. For such pages, you can try [Camoufox](https://camoufox.com/), a custom stealthy build of Firefox for web scraping. It might not get you through the challenge automatically, but with our `handleCloudflareChallenge` helper, it should be able to successfully mimic the required user action and get you through it.
+For some protections, using our integrated solutions is not enough, one example could be the Cloudflare challenge. For such pages, you can try [Camoufox](https://camoufox.com/), a custom stealthy build of Firefox for web scraping. It might not get you through the challenge automatically, but with our `handleCloudflareChallengeHook` post-navigation hook, it should be able to successfully mimic the required user action and get you through it. The hook also reloads the page after the challenge clears and propagates the fresh response back into the crawling context.
{PlaywrightCamoufox}
diff --git a/docs/guides/avoid_blocking_camoufox.ts b/docs/guides/avoid_blocking_camoufox.ts
index 131234578e62..65e3ffb6f0ca 100644
--- a/docs/guides/avoid_blocking_camoufox.ts
+++ b/docs/guides/avoid_blocking_camoufox.ts
@@ -1,22 +1,18 @@
-import { PlaywrightCrawler } from 'crawlee';
+import { PlaywrightCrawler, handleCloudflareChallengeHook, playwrightBrowserPool } from 'crawlee';
import { launchOptions } from 'camoufox-js';
import { firefox } from 'playwright';
const crawler = new PlaywrightCrawler({
- postNavigationHooks: [
- async ({ handleCloudflareChallenge }) => {
- await handleCloudflareChallenge();
- },
- ],
- browserPoolOptions: {
+ postNavigationHooks: [handleCloudflareChallengeHook()],
+ browserPool: playwrightBrowserPool({
// Disable the default fingerprint spoofing to avoid conflicts with Camoufox.
useFingerprints: false,
- },
- launchContext: {
- launcher: firefox,
- launchOptions: await launchOptions({
- headless: true,
- }),
- },
+ launchContext: {
+ launcher: firefox,
+ launchOptions: await launchOptions({
+ headless: true,
+ }),
+ },
+ }),
// ...
});
diff --git a/docs/guides/avoid_blocking_playwright.ts b/docs/guides/avoid_blocking_playwright.ts
index 1e5e23aeae42..ec88f284ce4c 100644
--- a/docs/guides/avoid_blocking_playwright.ts
+++ b/docs/guides/avoid_blocking_playwright.ts
@@ -1,8 +1,8 @@
-import { PlaywrightCrawler } from 'crawlee';
+import { PlaywrightCrawler, playwrightBrowserPool } from 'crawlee';
import { BrowserName, DeviceCategory, OperatingSystemsName } from '@crawlee/browser-pool';
const crawler = new PlaywrightCrawler({
- browserPoolOptions: {
+ browserPool: playwrightBrowserPool({
useFingerprints: true, // this is the default
fingerprintOptions: {
fingerprintGeneratorOptions: {
@@ -16,6 +16,6 @@ const crawler = new PlaywrightCrawler({
operatingSystems: [OperatingSystemsName.windows],
},
},
- },
+ }),
// ...
});
diff --git a/docs/guides/avoid_blocking_playwright_fingerprints_off.ts b/docs/guides/avoid_blocking_playwright_fingerprints_off.ts
index c36e52fc3aba..18a4ee900de8 100644
--- a/docs/guides/avoid_blocking_playwright_fingerprints_off.ts
+++ b/docs/guides/avoid_blocking_playwright_fingerprints_off.ts
@@ -1,8 +1,8 @@
-import { PlaywrightCrawler } from 'crawlee';
+import { PlaywrightCrawler, playwrightBrowserPool } from 'crawlee';
const crawler = new PlaywrightCrawler({
- browserPoolOptions: {
+ browserPool: playwrightBrowserPool({
useFingerprints: false,
- },
+ }),
// ...
});
diff --git a/docs/guides/avoid_blocking_puppeteer.ts b/docs/guides/avoid_blocking_puppeteer.ts
index 5af3ca7fa3b5..5ca21c82c42a 100644
--- a/docs/guides/avoid_blocking_puppeteer.ts
+++ b/docs/guides/avoid_blocking_puppeteer.ts
@@ -1,8 +1,8 @@
-import { PuppeteerCrawler } from 'crawlee';
+import { PuppeteerCrawler, puppeteerBrowserPool } from 'crawlee';
import { BrowserName, DeviceCategory } from '@crawlee/browser-pool';
const crawler = new PuppeteerCrawler({
- browserPoolOptions: {
+ browserPool: puppeteerBrowserPool({
useFingerprints: true, // this is the default
fingerprintOptions: {
fingerprintGeneratorOptions: {
@@ -11,6 +11,6 @@ const crawler = new PuppeteerCrawler({
locales: ['en-US'],
},
},
- },
+ }),
// ...
});
diff --git a/docs/guides/avoid_blocking_puppeteer_fingerprints_off.ts b/docs/guides/avoid_blocking_puppeteer_fingerprints_off.ts
index 425b4d3d5ba6..cf97daaab2a9 100644
--- a/docs/guides/avoid_blocking_puppeteer_fingerprints_off.ts
+++ b/docs/guides/avoid_blocking_puppeteer_fingerprints_off.ts
@@ -1,8 +1,8 @@
-import { PuppeteerCrawler } from 'crawlee';
+import { PuppeteerCrawler, puppeteerBrowserPool } from 'crawlee';
const crawler = new PuppeteerCrawler({
- browserPoolOptions: {
+ browserPool: puppeteerBrowserPool({
useFingerprints: false,
- },
+ }),
// ...
});
diff --git a/docs/guides/configuration.mdx b/docs/guides/configuration.mdx
index 597c3dcc2fa4..5fd8ea33dce6 100644
--- a/docs/guides/configuration.mdx
+++ b/docs/guides/configuration.mdx
@@ -15,13 +15,13 @@ There are three ways of changing the configuration parameters:
- using the `Configuration` class
You could also combine all the above, but you should keep in mind, that the precedence for these 3 options is the following:
-***`crawlee.json`*** < ***constructor options*** < ***environment variables***.
+***constructor options*** > ***environment variables*** > ***`crawlee.json`***.
-`crawlee.json` is a baseline. The options provided in the `Configuration` constructor will override the options provided in the JSON. Environment variables will override both.
+Constructor options have the highest priority. Environment variables override `crawlee.json`. The JSON file serves as a baseline.
## `crawlee.json`
-The first option you could use for configuring Crawlee is `crawlee.json` file. The only thing you need to do is specify the `ConfigurationOptions` in the file, place the file in the root of your project, and Crawlee will use provided options as global configuration.
+The first option you could use for configuring Crawlee is `crawlee.json` file. The only thing you need to do is specify the configuration options in the file, place the file in the root of your project, and Crawlee will use provided options as global configuration. See the `Configuration` class for the full list of supported options.
```json title="crawlee.json"
{
@@ -57,7 +57,7 @@ crawler.router.addDefaultHandler(async ({ request }) => {
await crawler.run(['https://www.example.com/1']);
```
-If you run this example (assuming you placed the `crawlee.json` file with `persistStateIntervalMillis` and `logLevel` specified there in the root of your project), you will find the `SDK_CRAWLER_STATISTICS` file in default Key-Value store,
+If you run this example (assuming you placed the `crawlee.json` file with `persistStateIntervalMillis` and `logLevel` specified there in the root of your project), you will find the `CRAWLEE_CRAWLER_STATISTICS` file in default Key-Value store,
which would show, that there's 1 finished request and crawler runtime was ~10 seconds.
This confirms that the state was persisted after 10 seconds, as it was set in `crawlee.json`.
Besides, you should see `DEBUG` logs in addition to `INFO` ones in your terminal, as `logLevel` was set to `DEBUG` in the `crawlee.json`, meaning Crawlee picked both provided options correctly.
@@ -90,11 +90,10 @@ The default request queue has ID `default`. Setting this environment variable ov
#### `CRAWLEE_PURGE_ON_START`
-Storage directories are purged by default. If set to `false` - local storage directories would not be purged automatically at the start of the crawler run or before opening of some storage explicitly (e.g. via `Dataset.open()`). Useful if we're trying e.g. to add more items to dataset with each next run (and keep the previously saved/scraped items).
+Run-scoped storage directories (the default storage and any opened with an `alias`) are purged by default; named storages are never purged. If set to `false` - local storage directories would not be purged automatically at the start of the crawler run or before opening of some storage explicitly (e.g. via `Dataset.open()`). Useful if we're trying e.g. to add more items to dataset with each next run (and keep the previously saved/scraped items).
#### `CRAWLEE_CONTAINERIZED`
-This variable is only effective when the systemInfoV2 experiment is enabled.
Changes how crawlee measures its CPU and Memory usage and limits. If unset, crawlee will determine if it is containerised using common features of containerized environments using the `isContainerized` utility function.
- A file at `/.dockerenv`.
- A file at `/proc/self/cgroup` containing `docker`.
@@ -127,31 +126,35 @@ Enables verbose logging if set to `true`. If not explicitly set to `true` - for
#### `CRAWLEE_MEMORY_MBYTES`
-Sets the amount of system memory in megabytes to be used by the `AutoscaledPool`.
-It is used to limit the number of concurrently running tasks. By default, the max amount of memory
+Sets the amount of system memory in megabytes to be used by the `ConcurrencySystem`.
+It is used to limit the number of concurrently running requests. By default, the max amount of memory
to be used is set to one quarter of total system memory, i.e. on a system with 8192 MB of memory,
the autoscaling feature will only use up to 2048 MB of memory.
## Configuration class
-The last option to adjust Crawlee configuration is to use the `Configuration` class in the code.
+The last option to adjust Crawlee configuration is to use the `Configuration` class in the code. Configuration is immutable — values are set via the constructor and cannot be changed afterwards.
### Global Configuration
-By default, there is a global singleton instance of `Configuration` class, it is used by the crawlers and some other classes that depend on a configurable behavior. In most cases you don't need to adjust any options there, but if needed - you can get access to it via `Configuration.getGlobalConfig()` function. Now you can easily `get` and `set` the `ConfigurationOptions`.
+By default, there is a global singleton instance of `Configuration` class, it is used by the crawlers and some other classes that depend on a configurable behavior. In most cases you don't need to adjust any options there, but if needed - you can access it via `Configuration.getGlobalConfiguration()`, which delegates to the global `serviceLocator` — the single source of truth for Crawlee's shared services (for example the configuration, event manager, storage backend, and logger). You can also reach the same instance directly via `serviceLocator.getConfiguration()` or swap services globally with `serviceLocator.setConfiguration(...)` before any crawler is created. Configuration values are accessible directly as properties on the instance.
```js
import { CheerioCrawler, Configuration, sleep } from 'crawlee';
// Get the global configuration
-const config = Configuration.getGlobalConfig();
-// Set the 'persistStateIntervalMillis' option
-// of global configuration to 10 seconds
-config.set('persistStateIntervalMillis', 10_000);
+const config = Configuration.getGlobalConfiguration();
+// Access configuration values directly as properties
+console.log(config.persistStateIntervalMillis);
-// Note, that we are not passing the configuration to the crawler
-// as it's using the global configuration
-const crawler = new CheerioCrawler();
+// To use custom configuration values, create a new Configuration instance
+const configuration = new Configuration({
+ // Set the 'persistStateIntervalMillis' option to 10 seconds
+ persistStateIntervalMillis: 10_000,
+});
+
+// Pass the configuration to the crawler
+const crawler = new CheerioCrawler({ configuration });
crawler.router.addDefaultHandler(async ({ request }) => {
// For the first request we wait for 5 seconds,
@@ -171,16 +174,14 @@ crawler.router.addDefaultHandler(async ({ request }) => {
await crawler.run(['https://www.example.com/1']);
```
-This is pretty much the same example we used for showing `crawlee.json` usage,
-but now we're using the global configuration, which is the only difference.
-If you run this example - you will find the `SDK_CRAWLER_STATISTICS` file in default Key-Value store as before,
-which would show the same number of finishes requests (one) and the same crawler runtime (~10 seconds).
-This confirms that provided parameters worked: the state was persisted after 10 seconds, as it was set in the global configuration.
+If you run this example - you will find the `CRAWLEE_CRAWLER_STATISTICS` file in default Key-Value store,
+which would show the same number of finished requests (one) and the same crawler runtime (~10 seconds).
+This confirms that provided parameters worked: the state was persisted after 10 seconds, as it was set in the configuration.
:::note
-After running the same example with commented two lines of code related to `Configuration` there will be
-no `SDK_CRAWLER_STATISTICS` file stored in the default Key-Value store:
+After running the same example without the custom configuration, there will be
+no `CRAWLEE_CRAWLER_STATISTICS` file stored in the default Key-Value store:
as we did not change the `persistStateIntervalMillis`, Crawlee used the default value of 60 seconds,
and the crawler was forcefully aborted after ~15 seconds of run time before it persisted the state for the first time.
@@ -188,19 +189,19 @@ and the crawler was forcefully aborted after ~15 seconds of run time before it p
### Custom configuration
-Alternatively, you can create a custom configuration. In this case you need to pass it to the class that is going to use it, e.g. to the crawler. Let's adjust the previous example:
+You can create a custom configuration and pass it to the crawler via the `configuration` option:
```js
import { CheerioCrawler, Configuration, sleep } from 'crawlee';
// Create new configuration
-const config = new Configuration({
+const configuration = new Configuration({
// Set the 'persistStateIntervalMillis' option to 10 seconds
persistStateIntervalMillis: 10_000,
});
-// Now we need to pass the configuration to the crawler
-const crawler = new CheerioCrawler({}, config);
+// Pass the configuration to the crawler
+const crawler = new CheerioCrawler({ configuration });
crawler.router.addDefaultHandler(async ({ request }) => {
// for the first request we wait for 5 seconds,
@@ -221,13 +222,13 @@ await crawler.run(['https://www.example.com/1']);
```
If you run this example - it would work exactly the same as before,
-with the same `SDK_CRAWLER_STATISTICS` file in default Key-Value store after the run,
+with the same `CRAWLEE_CRAWLER_STATISTICS` file in default Key-Value store after the run,
showing the same number of finished requests and the same crawler run time.
:::note
If you would not pass the configuration to the crawler, there again will be
-no `SDK_CRAWLER_STATISTICS` file stored in the default Key-Value store, this time for a different reason though.
+no `CRAWLEE_CRAWLER_STATISTICS` file stored in the default Key-Value store, this time for a different reason though.
Since we did not pass the configuration to the crawler,
the crawler will use the global configuration, which is using the default `persistStateIntervalMillis`.
So again, the run was aborted before the state was persisted for the first time.
diff --git a/docs/guides/custom-http-client/custom-http-client.mdx b/docs/guides/custom-http-client/custom-http-client.mdx
index c593ec3ba239..4e1b9f04c010 100644
--- a/docs/guides/custom-http-client/custom-http-client.mdx
+++ b/docs/guides/custom-http-client/custom-http-client.mdx
@@ -10,14 +10,34 @@ import CodeBlock from '@theme/CodeBlock';
import ImplementationSource from '!!raw-loader!./implementation.ts';
import UsageSource from '!!raw-loader!./usage.ts';
-The `BasicCrawler` class allows you to configure the HTTP client implementation using the `httpClient` constructor option. This might be useful for testing or if you need to swap out the default implementation based on `got-scraping` for something else, such as `curl-impersonate` or `axios`.
+The `BasicCrawler` class allows you to configure the HTTP client implementation using the `httpClient` constructor option. This might be useful for testing or if you need to swap out the default implementation based on `got-scraping` for something else, such as `curl-impersonate`.
-The HTTP client implementation needs to conform to the `BaseHttpClient` interface. For a rough idea on how it might look, see a skeleton implementation that uses the standard `fetch` interface:
+## Built-in HTTP clients
+
+Crawlee provides several HTTP client implementations out of the box:
+
+- **`ImpitHttpClient`** (default) - Uses the `impit` library for making requests that closely mimic browser behavior.
+- **`GotScrapingHttpClient`** - Uses the `got-scraping` library for browser-like requests with support for custom headers, browser fingerprints, and proxies. This was the default HTTP client in Crawlee v3.
+- **`FetchHttpClient`** - Simple implementation using the native `fetch` API (does not support proxies).
+
+## Implementing a custom HTTP client
+
+To create a custom HTTP client, extend the `BaseHttpClient` abstract class from `@crawlee/http-client`. The base class handles common functionality like cookie management, redirect following, session integration, proxy support, and timeout handling.
+
+Your custom implementation only needs to override the `fetch` method to perform the actual network request:
{ImplementationSource}
+By extending `BaseHttpClient`, your implementation automatically gets:
+- Cookie jar management (applying cookies before requests, saving cookies from responses)
+- Automatic redirect following (up to 10 redirects)
+- Session integration (proxy URL and cookies from session)
+- Timeout handling via AbortSignal
+- Proxy URL support
+
You may then instantiate it and pass to a crawler constructor:
{UsageSource}
-Please note that the interface is experimental and it will likely change with Crawlee version 4.
+Alternatively, you can implement the `BaseHttpClient` interface directly if you need full control over all aspects of the HTTP request handling, including cookies, redirects, and sessions. However, this approach requires implementing significantly more logic yourself.
+
diff --git a/docs/guides/custom-http-client/implementation.ts b/docs/guides/custom-http-client/implementation.ts
index 504f0b532f98..aac71784ff7e 100644
--- a/docs/guides/custom-http-client/implementation.ts
+++ b/docs/guides/custom-http-client/implementation.ts
@@ -1,122 +1,14 @@
-import type {
- BaseHttpClient,
- HttpRequest,
- HttpResponse,
- RedirectHandler,
- ResponseTypes,
- StreamingHttpResponse,
-} from '@crawlee/core';
-import { Readable } from 'node:stream';
-
-export class CustomHttpClient implements BaseHttpClient {
- async sendRequest(
- request: HttpRequest,
- ): Promise> {
- const requestHeaders = new Headers();
- for (let [headerName, headerValues] of Object.entries(request.headers ?? {})) {
- if (headerValues === undefined) {
- continue;
- }
-
- if (!Array.isArray(headerValues)) {
- headerValues = [headerValues];
- }
-
- for (const value of headerValues) {
- requestHeaders.append(headerName, value);
- }
- }
-
- const response = await fetch(request.url, {
- method: request.method,
- headers: requestHeaders,
- body: request.body as string, // TODO implement stream/generator handling
- signal: request.signal,
- // TODO implement the rest of request parameters (e.g., timeout, proxyUrl, cookieJar, ...)
- });
-
- const headers: Record = {};
-
- response.headers.forEach((value, headerName) => {
- headers[headerName] = value;
- });
-
- return {
- complete: true,
- request,
- url: response.url,
- statusCode: response.status,
- redirectUrls: [], // TODO you need to handle redirects manually to track them
- headers,
- trailers: {}, // TODO not supported by fetch
- ip: undefined,
- body:
- request.responseType === 'text'
- ? await response.text()
- : request.responseType === 'json'
- ? await response.json()
- : Buffer.from(await response.text()),
- };
- }
-
- async stream(request: HttpRequest, _onRedirect?: RedirectHandler): Promise {
- const fetchResponse = await fetch(request.url, {
- method: request.method,
- headers: new Headers(),
- body: request.body as string, // TODO implement stream/generator handling
- signal: request.signal,
- // TODO implement the rest of request parameters (e.g., timeout, proxyUrl, cookieJar, ...)
- });
-
- const headers: Record = {}; // TODO same as in sendRequest()
-
- async function* read() {
- const reader = fetchResponse.body?.getReader();
-
- const stream = new ReadableStream({
- start(controller) {
- if (!reader) {
- return null;
- }
- return pump();
- function pump(): Promise {
- return reader!.read().then(({ done, value }) => {
- // When no more data needs to be consumed, close the stream
- if (done) {
- controller.close();
- return;
- }
- // Enqueue the next data chunk into our target stream
- controller.enqueue(value);
- return pump();
- });
- }
- },
- });
-
- for await (const chunk of stream) {
- yield chunk;
- }
- }
-
- const response = {
- complete: false,
- request,
- url: fetchResponse.url,
- statusCode: fetchResponse.status,
- redirectUrls: [], // TODO you need to handle redirects manually to track them
- headers,
- trailers: {}, // TODO not supported by fetch
- ip: undefined,
- stream: Readable.from(read()),
- get downloadProgress() {
- return { percent: 0, transferred: 0 }; // TODO track this
- },
- get uploadProgress() {
- return { percent: 0, transferred: 0 }; // TODO track this
- },
- };
-
- return response;
+import { BaseHttpClient, type CustomFetchOptions } from '@crawlee/http-client';
+
+/**
+ * A simple HTTP client implementation using the native `fetch` API.
+ *
+ * Custom implementations only need to override the `fetch` method.
+ */
+export class CustomFetchClient extends BaseHttpClient {
+ protected override async fetch(request: Request, options?: RequestInit & CustomFetchOptions): Promise {
+ // The base class handles cookies, redirects, sessions, and timeouts.
+ // We only need to perform the actual network request here.
+ return fetch(request, options);
}
}
diff --git a/docs/guides/custom-http-client/usage.ts b/docs/guides/custom-http-client/usage.ts
index ebe52c236d3b..28fa63c5802a 100644
--- a/docs/guides/custom-http-client/usage.ts
+++ b/docs/guides/custom-http-client/usage.ts
@@ -1,8 +1,8 @@
import { HttpCrawler } from 'crawlee';
-import { CustomHttpClient } from './implementation.js';
+import { CustomFetchClient } from './implementation.js';
const crawler = new HttpCrawler({
- httpClient: new CustomHttpClient(),
+ httpClient: new CustomFetchClient(),
async requestHandler() {
/* ... */
},
diff --git a/docs/guides/custom-logger/custom-logger.mdx b/docs/guides/custom-logger/custom-logger.mdx
new file mode 100644
index 000000000000..8d024b6cbf64
--- /dev/null
+++ b/docs/guides/custom-logger/custom-logger.mdx
@@ -0,0 +1,88 @@
+---
+id: custom-logger
+title: Custom logger
+description: Use your own logging library (Winston, Pino, etc.) with Crawlee
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+import WinstonSource from '!!raw-loader!./winston.ts';
+import PinoSource from '!!raw-loader!./pino.ts';
+
+Crawlee uses `@apify/log` as its default logging library, but you can replace it with any logger you prefer, such as Winston or Pino. This is done by implementing a small adapter and passing it to the crawler.
+
+## Creating an adapter
+
+All Crawlee logging goes through the `CrawleeLogger` interface. To plug in your own logger, extend the `BaseCrawleeLogger` abstract class and implement two methods:
+
+- **`logWithLevel(level, message, data)`** — dispatches a log message to your logging library. The `level` parameter uses `LogLevel` constants (`ERROR = 1`, `SOFT_FAIL = 2`, `WARNING = 3`, `INFO = 4`, `DEBUG = 5`, `PERF = 6`). Map these to your logger's native levels. The `message` is a human-readable `string`, and `data` is an optional `Record` with structured context (e.g. `{ url, statusCode }`) — pass it to your logger as metadata or structured fields.
+- **`createChild(options)`** — returns a new child logger instance scoped to a specific component. Crawlee calls this internally to give each subsystem (e.g. `CheerioCrawler`, `AutoscaledPool`, `SessionPool`) its own identifiable logger. The `options` parameter is a `CrawleeLoggerOptions` object with a single field: `prefix` — a string label prepended to each log line from that component.
+
+All other methods (`error`, `warning`, `info`, `debug`, `exception`, `perf`, etc.) are derived automatically from `logWithLevel` — you don't need to implement them.
+
+:::info Level filtering
+
+`logWithLevel()` is called for **every** log message, regardless of the configured level. Level filtering is the responsibility of the underlying logging library (e.g. Winston's `level` option or Pino's `level` setting). This means your adapter doesn't need to check log levels — just forward everything and let the library decide what to output.
+
+:::
+
+## Injecting the logger
+
+There are two ways to inject a custom logger: per-crawler and globally.
+
+### Per-crawler logger
+
+Pass your adapter via the `logger` option in the crawler constructor. When a `logger` is provided, the crawler creates its own isolated `ServiceLocator` instance, so the custom logger is used by all internal components of that crawler (autoscaling, session pool, statistics, etc.):
+
+```ts
+import { CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ logger: new WinstonAdapter(winstonLogger),
+ async requestHandler({ log }) {
+ // `log` is a child of your custom logger, with prefix set to the crawler class name
+ log.info('Hello from my custom logger!');
+ },
+});
+```
+
+The same logger is available as `crawler.log` outside of the request handler, for example when setting up routes.
+
+### Global logger via service locator
+
+Instead of passing the logger to each crawler individually, you can set it globally via the `serviceLocator`. This is useful when you run multiple crawlers and want them all to use the same logging backend:
+
+```ts
+import { serviceLocator, CheerioCrawler, PlaywrightCrawler } from 'crawlee';
+
+// Set the logger globally — must be done before creating any crawlers
+serviceLocator.setLogger(new WinstonAdapter(winstonLogger));
+
+// Both crawlers will use the Winston logger
+const cheerioCrawler = new CheerioCrawler({ /* ... */ });
+const playwrightCrawler = new PlaywrightCrawler({ /* ... */ });
+```
+
+:::warning
+
+`serviceLocator.setLogger()` must be called **before** any crawler is created. Once a logger has been retrieved from the service locator (which happens during crawler construction), it cannot be replaced — an error will be thrown.
+
+:::
+
+## Full examples
+
+
+
+
+{WinstonSource}
+
+
+
+
+{PinoSource}
+
+
+
diff --git a/docs/guides/custom-logger/pino.ts b/docs/guides/custom-logger/pino.ts
new file mode 100644
index 000000000000..2cf60813aa74
--- /dev/null
+++ b/docs/guides/custom-logger/pino.ts
@@ -0,0 +1,48 @@
+import { CheerioCrawler, BaseCrawleeLogger, LogLevel } from 'crawlee';
+import type { CrawleeLogger, CrawleeLoggerOptions } from 'crawlee';
+import pino from 'pino';
+
+// Map Crawlee log levels to Pino levels
+const CRAWLEE_TO_PINO: Record = {
+ [LogLevel.ERROR]: 'error',
+ [LogLevel.SOFT_FAIL]: 'warn',
+ [LogLevel.WARNING]: 'warn',
+ [LogLevel.INFO]: 'info',
+ [LogLevel.DEBUG]: 'debug',
+ [LogLevel.PERF]: 'trace',
+};
+
+class PinoAdapter extends BaseCrawleeLogger {
+ constructor(
+ private logger: pino.Logger,
+ options?: Partial,
+ ) {
+ super(options);
+ }
+
+ logWithLevel(level: number, message: string, data?: Record): void {
+ const pinoLevel = CRAWLEE_TO_PINO[level] ?? 'info';
+ this.logger[pinoLevel as pino.Level](data ?? {}, message);
+ }
+
+ protected createChild(options: Partial): CrawleeLogger {
+ return new PinoAdapter(this.logger.child({ prefix: options.prefix }), { ...this.getOptions(), ...options });
+ }
+}
+
+// Create a Pino logger with your preferred configuration
+const pinoLogger = pino({
+ level: 'debug',
+});
+
+// Pass the adapter to the crawler via the `logger` option
+const crawler = new CheerioCrawler({
+ logger: new PinoAdapter(pinoLogger),
+ async requestHandler({ request, $, log }) {
+ log.info(`Processing ${request.url}`);
+ const title = $('title').text();
+ log.debug('Page title extracted', { title });
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/guides/custom-logger/winston.ts b/docs/guides/custom-logger/winston.ts
new file mode 100644
index 000000000000..9a967988b193
--- /dev/null
+++ b/docs/guides/custom-logger/winston.ts
@@ -0,0 +1,57 @@
+import { CheerioCrawler, BaseCrawleeLogger, LogLevel } from 'crawlee';
+import type { CrawleeLogger, CrawleeLoggerOptions } from 'crawlee';
+import winston from 'winston';
+
+// Map Crawlee log levels to Winston levels
+const CRAWLEE_TO_WINSTON: Record = {
+ [LogLevel.ERROR]: 'error',
+ [LogLevel.SOFT_FAIL]: 'warn',
+ [LogLevel.WARNING]: 'warn',
+ [LogLevel.INFO]: 'info',
+ [LogLevel.DEBUG]: 'debug',
+ [LogLevel.PERF]: 'debug',
+};
+
+class WinstonAdapter extends BaseCrawleeLogger {
+ constructor(
+ private logger: winston.Logger,
+ options?: Partial,
+ ) {
+ super(options);
+ }
+
+ logWithLevel(level: number, message: string, data?: Record): void {
+ const winstonLevel = CRAWLEE_TO_WINSTON[level] ?? 'info';
+ this.logger.log(winstonLevel, message, data);
+ }
+
+ protected createChild(options: Partial): CrawleeLogger {
+ return new WinstonAdapter(this.logger.child({ prefix: options.prefix }), { ...this.getOptions(), ...options });
+ }
+}
+
+// Create a Winston logger with your preferred configuration
+const winstonLogger = winston.createLogger({
+ level: 'debug',
+ format: winston.format.combine(
+ winston.format.colorize(),
+ winston.format.timestamp(),
+ winston.format.printf(({ level, message, timestamp, prefix }) => {
+ const tag = prefix ? `[${prefix}] ` : '';
+ return `${timestamp} ${level}: ${tag}${message}`;
+ }),
+ ),
+ transports: [new winston.transports.Console()],
+});
+
+// Pass the adapter to the crawler via the `logger` option
+const crawler = new CheerioCrawler({
+ logger: new WinstonAdapter(winstonLogger),
+ async requestHandler({ request, $, log }) {
+ log.info(`Processing ${request.url}`);
+ const title = $('title').text();
+ log.debug('Page title extracted', { title });
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/guides/custom_statistics.mdx b/docs/guides/custom_statistics.mdx
new file mode 100644
index 000000000000..4e00dbcfd639
--- /dev/null
+++ b/docs/guides/custom_statistics.mdx
@@ -0,0 +1,52 @@
+---
+id: custom-statistics
+title: Custom statistics fields
+description: Track your own counters alongside the built-in crawler statistics
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+import CodeBlock from '@theme/CodeBlock';
+
+import BasicSource from '!!raw-loader!./custom_statistics_basic.ts';
+import ValidatedSource from '!!raw-loader!./custom_statistics_validated.ts';
+import AdaptiveSource from '!!raw-loader!./custom_statistics_adaptive.ts';
+
+Every crawler collects run statistics — requests finished, failures, retries, durations — into a `Statistics` instance exposed as `crawler.statistics`. You can track your own counters in the same place, and get the persistence, logging and lifecycle handling that the built-in fields already have.
+
+Keeping a counter in a plain variable works right up until the run is interrupted. Statistics state is written to the key-value store and restored on migration or resurrect, so a counter that lives in `statistics.state` survives a run that a plain variable would not.
+
+## Declaring custom fields
+
+Pass `stateExtension` when constructing a `Statistics` instance, and inject it into the crawler via the `statistics` option. The fields become part of `state`, typed exactly as you declared them:
+
+{BasicSource}
+
+The field types flow through to `crawler.statistics.state`, so they are available anywhere you hold the crawler, not just where you hold the statistics instance. Reading an undeclared field is a compile-time error.
+
+`defaultState` is also what `reset()` restores, and it determines which keys are read back from a persisted record. A field you add later simply keeps its default when an older record is restored, so growing the state between runs does not need a migration.
+
+## Validating the fields on the way back
+
+Declaring `defaultState` alone means a restored record is taken at its word: whatever `productsFound` holds in the key-value store becomes the state, and an increment on a string that says it is a number poisons every later one. Pass a `deserialize` conversion to check the record before trusting it — either a plain function or, as here, a [Standard Schema](https://standardschema.dev):
+
+{ValidatedSource}
+
+A schema that gives every field a default doubles as the declaration of the fields, so `defaultState` can be omitted — one place to add a field, and the defaults cannot drift from the conversion that has to accept them back. If the record fails to validate, the custom fields start from their defaults with a warning; the built-in statistics are unaffected.
+
+The fields have to be JSON-serializable, because the state round-trips through the key-value store. A `Date` or a `Set` comes back from that trip as a string or a plain object, so either keep to numbers, strings, booleans, `null`, and plain arrays and objects of those, or pair `deserialize` with a `serialize` that converts them, as `lastSeenAt` does above.
+
+:::note Injected instances are not reset
+
+A crawler resets the statistics it built itself at the start of every `run()`, but never one you passed in — a supplied instance keeps whatever state it was handed. If you reuse a crawler for several runs and want the counters to start from zero each time, call `statistics.reset()` yourself.
+
+:::
+
+## Crawlers that track fields of their own
+
+`AdaptivePlaywrightCrawler` tracks a few extra fields itself (how many requests were handled with and without a browser, and how many rendering type mispredictions occurred). It builds a suitable `Statistics` instance by default, so you only need to do anything if you want to add fields of your own — in which case your declaration has to include the crawler's fields as well:
+
+{AdaptiveSource}
+
+## Custom statistics implementations
+
+If you need something more than extra fields — reporting to an external metrics backend, for example — the `statistics` option accepts any object implementing `IStatistics`, not just the built-in class. The crawler drives it purely through that interface.
diff --git a/docs/guides/custom_statistics_adaptive.ts b/docs/guides/custom_statistics_adaptive.ts
new file mode 100644
index 000000000000..6596e3f0826e
--- /dev/null
+++ b/docs/guides/custom_statistics_adaptive.ts
@@ -0,0 +1,22 @@
+import { AdaptivePlaywrightCrawler, adaptivePlaywrightCrawlerStatisticState, Statistics } from 'crawlee';
+import { z } from 'zod';
+
+const statistics = new Statistics({
+ stateExtension: {
+ // The adaptive crawler tracks fields of its own, so an injected instance has to carry them too.
+ deserialize: adaptivePlaywrightCrawlerStatisticState.deserialize.extend({
+ productsFound: z.number().default(0),
+ }),
+ },
+});
+
+const crawler = new AdaptivePlaywrightCrawler({
+ statistics,
+ async requestHandler({ querySelectorAll }) {
+ statistics.state.productsFound += (await querySelectorAll('.product')).length;
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
+
+console.log(`Handled ${crawler.statistics.state.httpOnlyRequestHandlerRuns} requests without a browser`);
diff --git a/docs/guides/custom_statistics_basic.ts b/docs/guides/custom_statistics_basic.ts
new file mode 100644
index 000000000000..2342c794bdf5
--- /dev/null
+++ b/docs/guides/custom_statistics_basic.ts
@@ -0,0 +1,20 @@
+import { CheerioCrawler, Statistics } from 'crawlee';
+
+// Declare the extra fields and their initial values. Their types flow into `statistics.state`.
+const statistics = new Statistics({
+ stateExtension: { defaultState: { productsFound: 0 } },
+});
+
+const crawler = new CheerioCrawler({
+ statistics,
+ async requestHandler({ $, enqueueLinks }) {
+ statistics.state.productsFound += $('.product').length;
+
+ await enqueueLinks();
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
+
+// The custom fields are typed on `crawler.statistics` too.
+console.log(`Found ${crawler.statistics.state.productsFound} products`);
diff --git a/docs/guides/custom_statistics_validated.ts b/docs/guides/custom_statistics_validated.ts
new file mode 100644
index 000000000000..c05a77eff76a
--- /dev/null
+++ b/docs/guides/custom_statistics_validated.ts
@@ -0,0 +1,27 @@
+import { CheerioCrawler, Statistics } from 'crawlee';
+import { z } from 'zod';
+
+const statistics = new Statistics({
+ stateExtension: {
+ // A default per field means the schema is the only place the fields are declared.
+ deserialize: z.object({
+ productsFound: z.number().default(0),
+ lastSeenAt: z.coerce.date().default(() => new Date()),
+ }),
+ // `Date` is not JSON - this is how it gets into the record for `z.coerce.date()` to read back.
+ serialize: ({ productsFound, lastSeenAt }) => ({
+ productsFound,
+ lastSeenAt: lastSeenAt.toISOString(),
+ }),
+ },
+});
+
+const crawler = new CheerioCrawler({
+ statistics,
+ async requestHandler({ $ }) {
+ statistics.state.productsFound += $('.product').length;
+ statistics.state.lastSeenAt = new Date();
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/guides/http-clients.mdx b/docs/guides/http-clients.mdx
index 9956cb3077a3..ea751e7e9a95 100644
--- a/docs/guides/http-clients.mdx
+++ b/docs/guides/http-clients.mdx
@@ -49,7 +49,7 @@ BaseHttpClient --|> GotScrapingHttpClient
## Switching between HTTP clients
-Crawlee currently provides two main HTTP clients: `GotScrapingHttpClient`, which uses the `got-scraping` library, and `ImpitHttpClient`, which uses the `impit` library. You can switch between them by setting the `BasehttpClient` parameter when initializing a crawler class. The default HTTP client is `GotScrapingHttpClient`. For more details on anti-blocking features, see our [avoid getting blocked guide](./avoid-blocking).
+Crawlee currently provides two main HTTP clients: `GotScrapingHttpClient`, which uses the `got-scraping` library, and `ImpitHttpClient`, which uses the `impit` library. You can switch between them by setting the `BasehttpClient` parameter when initializing a crawler class. The default HTTP client is `GotScrapingHttpClient`. For more details on anti-blocking features, see our [avoid getting blocked guide](./avoid-blocking).
Below are examples of how to configure the HTTP client for the `CheerioCrawler`:
@@ -68,7 +68,7 @@ Below are examples of how to configure the HTTP client for the `GotScrapingHttpClient` is the default HTTP client, it's included with the base Crawlee installation and requires no additional packages.
+Since `GotScrapingHttpClient` is the default HTTP client, it's included with the base Crawlee installation and requires no additional packages.
For `ImpitHttpClient`, you need to install a separate `@crawlee/impit-client` package:
@@ -78,7 +78,7 @@ npm i @crawlee/impit-client
## Creating custom HTTP clients
-Crawlee provides an interface, `BaseHttpClient`, which defines the interface that all HTTP clients must implement. This allows you to create custom HTTP clients tailored to your specific requirements.
+Crawlee provides an interface, `BaseHttpClient`, which defines the interface that all HTTP clients must implement. This allows you to create custom HTTP clients tailored to your specific requirements.
HTTP clients are responsible for several key operations:
@@ -88,10 +88,10 @@ HTTP clients are responsible for several key operations:
- managing proxy configurations,
- connection pooling with timeout management.
-To create a custom HTTP client, you need to implement the `BaseHttpClient` interface. Your implementation must be async-compatible and include proper cleanup and resource management to work seamlessly with Crawlee's concurrent processing model.
+To create a custom HTTP client, you need to implement the `BaseHttpClient` interface. Your implementation must be async-compatible and include proper cleanup and resource management to work seamlessly with Crawlee's concurrent processing model.
## Conclusion
-This guide introduced you to the HTTP clients available in Crawlee and demonstrated how to switch between them, including their installation requirements and usage examples. You also learned about the responsibilities of HTTP clients and how to implement your own custom HTTP client by inheriting from the `BaseHttpClient` base class.
+This guide introduced you to the HTTP clients available in Crawlee and demonstrated how to switch between them, including their installation requirements and usage examples. You also learned about the responsibilities of HTTP clients and how to implement your own custom HTTP client by inheriting from the `BaseHttpClient` base class.
If you have questions or need assistance, feel free to reach out on our [GitHub](https://github.com/apify/crawlee) or join our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping!
diff --git a/docs/guides/http-clients/cheerio-got-scraping-example.ts b/docs/guides/http-clients/cheerio-got-scraping-example.ts
index a2cab0af3807..7a6b8e6ad24e 100644
--- a/docs/guides/http-clients/cheerio-got-scraping-example.ts
+++ b/docs/guides/http-clients/cheerio-got-scraping-example.ts
@@ -1,4 +1,5 @@
-import { CheerioCrawler, GotScrapingHttpClient } from 'crawlee';
+import { CheerioCrawler } from 'crawlee';
+import { GotScrapingHttpClient } from '@crawlee/got-scraping-client';
const crawler = new CheerioCrawler({
httpClient: new GotScrapingHttpClient(),
diff --git a/docs/guides/impit-http-client/basic-usage.ts b/docs/guides/impit-http-client/basic-usage.ts
index 1a8754c9fa11..51b414913bdc 100644
--- a/docs/guides/impit-http-client/basic-usage.ts
+++ b/docs/guides/impit-http-client/basic-usage.ts
@@ -7,7 +7,7 @@ const crawler = new BasicCrawler({
}),
async requestHandler({ sendRequest, log }) {
const response = await sendRequest();
- log.info('Received response', { statusCode: response.statusCode });
+ log.info('Received response', { status: response.status });
},
});
diff --git a/docs/guides/impit-http-client/impit-http-client.mdx b/docs/guides/impit-http-client/impit-http-client.mdx
index 89c71e82fa5d..5bfca4bf2d09 100644
--- a/docs/guides/impit-http-client/impit-http-client.mdx
+++ b/docs/guides/impit-http-client/impit-http-client.mdx
@@ -11,8 +11,6 @@ import CheerioCrawlerSource from '!!raw-loader!./cheerio-crawler.ts';
import HttpCrawlerSource from '!!raw-loader!./http-crawler.ts';
import AdvancedConfigSource from '!!raw-loader!./advanced-config.ts';
-## Introduction
-
The `ImpitHttpClient` is an HTTP client implementation based on the [Impit](https://github.com/apify/impit) library. It enables browser impersonation for HTTP requests, helping you bypass bot detection systems without running an actual browser.
:::info Successor to got-scraping
diff --git a/docs/guides/parallel-scraping/parallel-scraper.mjs b/docs/guides/parallel-scraping/parallel-scraper.mjs
index 6bee4f4ff13a..3b2fe80a1b13 100644
--- a/docs/guides/parallel-scraping/parallel-scraper.mjs
+++ b/docs/guides/parallel-scraping/parallel-scraper.mjs
@@ -1,5 +1,6 @@
import { fork } from 'node:child_process';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
import { Configuration, Dataset, PlaywrightCrawler, log } from 'crawlee';
import { router } from './routes.mjs';
@@ -73,18 +74,21 @@ if (!process.env.IN_WORKER_THREAD) {
// or a configuration option. This is just for show 😈
workerLogger.setLevel(log.LEVELS.DEBUG);
- // Disable the automatic purge on start
- // This is needed when running locally, as otherwise multiple processes will try to clear the default storage (and that will cause clashes)
- Configuration.set('purgeOnStart', false);
-
// Get the request queue
const requestQueue = await getOrInitQueue(false);
- // Configure crawlee to store the worker-specific data in a separate directory (needs to be done AFTER the queue is initialized when running locally)
+ // Disable the automatic purge on start, so we don't lose the queue we prepared
const config = new Configuration({
- storageClientOptions: {
- localDataDirectory: `./storage/worker-${process.env.WORKER_INDEX}`,
- },
+ purgeOnStart: false,
+ });
+
+ // Store the worker's own internal state (its default dataset, key-value store, etc.) in a separate
+ // directory so the workers don't collide with each other. This directory is private to a single
+ // worker, so we set `requestQueueAccess: 'single'` — the concurrency-safe locking only matters for
+ // the shared `shop-urls` queue, which gets its own storage backend in `requestQueue.mjs`.
+ const storageBackend = new FileSystemStorageBackend({
+ localDataDirectory: `./storage/worker-${process.env.WORKER_INDEX}`,
+ requestQueueAccess: 'single',
});
workerLogger.debug('Setting up crawler.');
@@ -94,16 +98,16 @@ if (!process.env.IN_WORKER_THREAD) {
// Instead of the long requestHandler with
// if clauses we provide a router instance.
requestHandler: router,
- // Enable the request locking experiment so that we can actually use the queue.
- // highlight-start
- experiments: {
- requestLocking: true,
- },
// Provide the request queue we've pre-filled in previous steps
+ // highlight-start
requestQueue,
// highlight-end
// Let's also limit the crawler's concurrency, we don't want to overload a single process 🐌
maxConcurrency: 5,
+ // Use the worker-specific, concurrency-safe storage backend we created above
+ // highlight-start
+ storageBackend,
+ // highlight-end
},
config,
);
diff --git a/docs/guides/parallel-scraping/parallel-scraping.mdx b/docs/guides/parallel-scraping/parallel-scraping.mdx
index 5e05532c859b..82c0b3be9231 100644
--- a/docs/guides/parallel-scraping/parallel-scraping.mdx
+++ b/docs/guides/parallel-scraping/parallel-scraping.mdx
@@ -12,12 +12,6 @@ import AdaptedRoutesSource from '!!raw-loader!./adapted-routes.mjs';
import ParallelScraperSource from '!!raw-loader!./parallel-scraper.mjs';
import ModifiedDetailRouteSource from '!!raw-loader!./modified-detail-route.mjs';
-:::warning Experimental features ahead
-
-At the time of writing this guide (December 2023), request locking is still an experimental feature. You can read more about the experiment by visiting the [request locking experiment](../experiments/experiments-request-locking) page.
-
-:::
-
In this guide, we will walk you through how you can turn your single scraper into a scraper that can be parallelized and run in multiple instances. This guide assumes you've read and walked through our [introduction guide](../introduction/setting-up) (or have a fully-fledged scraper already built), but if you haven't done so yet, take a break, go read through all that, and come back. We'll be waiting...
*Oh, you're back already! Let's proceed in making that scraper parallel!*
@@ -66,6 +60,16 @@ The first step in our conversion process will be creating a common file (let's c
The exported function, `getOrInitQueue`, might seem like it does a lot. In essence, it just ensures the request queue is initialized, and if requested, ensures it starts off with an empty state.
+:::caution Make the shared queue concurrency-safe with `requestQueueAccess: 'shared'`
+
+Because every worker process opens this same `shop-urls` queue at the same time, it **must** use the concurrency-safe locking behavior of `FileSystemStorageBackend`. That's why `getOrInitQueue` opens the queue with a storage backend constructed with `requestQueueAccess: 'shared'`.
+
+By default, `FileSystemStorageBackend` assumes it is the *sole* consumer of a queue (`requestQueueAccess: 'single'`). On open it immediately reclaims any requests left *in progress* — great for a single-process crawl recovering after a crash, but disastrous when workers run side by side: each worker would happily grab requests another worker is still processing, so the same URL gets scraped multiple times.
+
+Setting `requestQueueAccess: 'shared'` tells the client to treat an in-progress request as a potential live peer's lock and only reclaim it once the lock expires on the wall clock, so two workers never process the same request at once.
+
+:::
+
### Adapting our previous scraper to enqueue the product URLs to the new queue
In the `src/routes.mjs` file of the scraper we previously built, we have a handler for the `CATEGORY` label. Let's adapt that handler to enqueue the product URLs to the new queue we created.
@@ -128,37 +132,44 @@ This will check how the script is executed as. If this value has _any_ value, it
We use this to ensure the parent process stays alive until all the worker processes exit. Otherwise, the worker processes would just get spawned, and lose the ability to communicate with the parent. You might not need this depending on your use case (maybe you just need to spawn workers and let them process).
-#### What's with all those `Configuration` calls?
+#### What's with all the `Configuration` and storage backend setup?
-There are three steps we want to do for the worker processes:
+There are two things we want to do for the worker processes:
-- ensure the default storages do **not** get purged on start, as otherwise we'd lose the queue we prepared
-- get the queue that supports locking from the same location as the parent process
-- initialize a special storage for worker processes so they do not collide with each other
+- get the shared queue from the same location as the parent process (it already comes with the concurrency-safe storage backend we set up in `requestQueue.mjs`)
+- ensure the default storages do **not** get purged on start, as otherwise we'd lose the queue we prepared, and give each worker its own private storage directory for its internal state so the workers don't collide with each other
In order, that's what these lines do:
```javascript title="src/parallel-scraper.mjs"
-// Disable the automatic purge on start (step 1)
-// This is needed when running locally, as otherwise multiple processes will try to clear the default storage (and that will cause clashes)
-Configuration.set('purgeOnStart', false);
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
-// Get the request queue from the parent process (step 2)
+// Get the shared request queue from the parent process (step 1)
const requestQueue = await getOrInitQueue(false);
-// Configure crawlee to store the worker-specific data in a separate directory (needs to be done AFTER the queue is initialized when running locally) (step 3)
-const config = new Configuration({
- storageClientOptions: {
- localDataDirectory: `./storage/worker-${process.env.WORKER_INDEX}`,
- },
+// Disable the automatic purge on start, so we don't lose the queue we prepared (step 2)
+const config = new Configuration({ purgeOnStart: false });
+
+// Store the worker's own internal state in a separate directory so workers don't collide (step 2,
+// cont.). This directory is private to a single worker, so we explicitly set
+// `requestQueueAccess: 'single'`.
+const storageBackend = new FileSystemStorageBackend({
+ localDataDirectory: `./storage/worker-${process.env.WORKER_INDEX}`,
+ requestQueueAccess: 'single',
});
```
-#### Enabling the request locking experiment, and telling the crawler to use the worker configuration
+:::note Why no `requestQueueAccess: 'shared'` here?
+
+Each worker's `./storage/worker-N` directory is private to that single worker — nothing else opens it — so the default `requestQueueAccess: 'single'` is exactly right. The concurrency-safe locking only matters for storage that is genuinely shared across processes, which is the `shop-urls` queue in `requestQueue.mjs`, not this per-worker internal state.
+
+:::
+
+#### Telling the crawler to use the worker configuration
-You might have noticed several lines highlighted in the code above. Those show how you can enable the request locking experiment, as well as how you provide the request queue to the crawler. You can read more about the experiment by visiting the [request locking experiment](../experiments/experiments-request-locking) page.
+You might have noticed several lines highlighted in the code above. Those show how you provide the shared request queue to the crawler.
-You might have also noticed we passed in a second parameter to the constructor of the crawler, the `config` variable we created earlier. This is needed to ensure the crawler uses the worker-specific storages for internal states, and that they do not collide with each other.
+You might have also noticed we passed in the `config` and `storageBackend` we created earlier to the crawler. These ensure the crawler uses the worker-specific storages for its own internal state (so the workers do not collide with each other), while still consuming the shared, concurrency-safe `shop-urls` queue we provided explicitly.
#### Why do we use `process.send` instead of `context.pushData`?
diff --git a/docs/guides/parallel-scraping/shared.mjs b/docs/guides/parallel-scraping/shared.mjs
index ff627fdee401..bef086bfb6e1 100644
--- a/docs/guides/parallel-scraping/shared.mjs
+++ b/docs/guides/parallel-scraping/shared.mjs
@@ -1,8 +1,19 @@
-import { RequestQueueV2 } from 'crawlee';
+import { FileSystemStorageBackend } from '@crawlee/fs-storage';
+import { RequestQueue } from 'crawlee';
-// Create the request queue that also supports parallelization
+// The request queue shared by all the parallel workers
let queue;
+// The `shop-urls` queue is opened concurrently by every worker process, so it must use the
+// concurrency-safe locking behavior. With `requestQueueAccess: 'shared'`, a request another worker
+// is still processing is treated as a live peer's lock and is not handed out again until that lock
+// expires — so two workers never scrape the same URL at once. (We point at the `./storage`
+// location, which is where this shared queue lives.)
+const sharedStorageBackend = new FileSystemStorageBackend({
+ localDataDirectory: './storage',
+ requestQueueAccess: 'shared',
+});
+
/**
* @param {boolean} makeFresh Whether the queue should be cleared before returning it
* @returns The queue
@@ -12,11 +23,11 @@ export async function getOrInitQueue(makeFresh = false) {
return queue;
}
- queue = await RequestQueueV2.open('shop-urls');
+ queue = await RequestQueue.open('shop-urls', { storageBackend: sharedStorageBackend });
if (makeFresh) {
await queue.drop();
- queue = await RequestQueueV2.open('shop-urls');
+ queue = await RequestQueue.open('shop-urls', { storageBackend: sharedStorageBackend });
}
return queue;
diff --git a/docs/guides/proxy_management.mdx b/docs/guides/proxy_management.mdx
index 8bf385f1c5b5..bc7253aa6ec9 100644
--- a/docs/guides/proxy_management.mdx
+++ b/docs/guides/proxy_management.mdx
@@ -31,7 +31,7 @@ import InspectionPuppeteerSource from '!!raw-loader!./proxy_management_inspectio
and most effective ways of preventing access to a website. It is therefore paramount for
a good web scraping library to provide easy to use but powerful tools which can work around
IP blocking. The most powerful weapon in our anti IP blocking arsenal is a
-[proxy server](https://en.wikipedia.org/wiki/Proxy_server).
+[proxy server](https://en.wikipedia.org/wiki/Proxy_server).
With Crawlee we can use our own proxy servers or proxy servers acquired from
third-party providers.
@@ -83,7 +83,7 @@ The `ProxyConfiguration` class allows you to provide a custom function to pick a
```javascript
const proxyConfiguration = new ProxyConfiguration({
- newUrlFunction: (sessionId, { request }) => {
+ newUrlFunction: ({ request } = {}) => {
if (request?.url.includes('crawlee.dev')) {
return null; // for crawlee.dev, we don't use a proxy
}
@@ -93,39 +93,10 @@ const proxyConfiguration = new ProxyConfiguration({
});
```
-The `newUrlFunction` receives two parameters - `sessionId` and `options` - and returns a string containing the proxy URL.
-
-The `sessionId` parameter is always provided and allows us to differentiate between different sessions - e.g. when Crawlee recognizes your crawlers are being blocked, it will automatically create a new session with a different id.
+The `newUrlFunction` receives a single optional `options` parameter and returns a string with the proxy URL (or `null` to skip the proxy for the current request).
The `options` parameter is an object containing a `Request`, which is the request that will be made. Note that this object is not always available, for example when we are using the `newUrl` function directly. Your custom function should therefore not rely on the `request` object being present and provide a default behavior when it is not.
-### Tiered proxies
-
-You can also provide a list of proxy tiers to the `ProxyConfiguration` class. This is useful when you want to switch between different proxies automatically based on the blocking behavior of the website.
-
-:::warning
-
-Note that the `tieredProxyUrls` option requires `ProxyConfiguration` to be used from a crawler instance ([see below](#crawler-integration)).
-
-Using this configuration through the `newUrl` calls will not yield the expected results.
-
-:::
-
-```javascript
-const proxyConfiguration = new ProxyConfiguration({
- tieredProxyUrls: [
- [null], // At first, we try to connect without a proxy
- ['http://okay-proxy.com'],
- ['http://slightly-better-proxy.com', 'http://slightly-better-proxy-2.com'],
- ['http://very-good-and-expensive-proxy.com'],
- ]
-});
-```
-
-This configuration will start with no proxy, then switch to `http://okay-proxy.com` if Crawlee recognizes we're getting blocked by the target website. If that proxy is also blocked, we will switch to one of the `slightly-better-proxy` URLs. If those are blocked, we will switch to the `very-good-and-expensive-proxy.com` URL.
-
-Crawlee also periodically probes lower tier proxies to see if they are unblocked, and if they are, it will switch back to them.
-
## Crawler integration
`ProxyConfiguration` integrates seamlessly into `HttpCrawler`, `CheerioCrawler`, `JSDOMCrawler`, `PlaywrightCrawler` and `PuppeteerCrawler`.
@@ -162,9 +133,7 @@ Our crawlers will now use the selected proxies for all connections.
## IP Rotation and session management
-`proxyConfiguration.newUrl()` allows us to pass a `sessionId` parameter. It will then be used to create a `sessionId`-`proxyUrl` pair, and subsequent `newUrl()` calls with the same `sessionId` will always return the same `proxyUrl`. This is extremely useful in scraping, because we want to create the impression of a real user. See the [session management guide](../guides/session-management) and `SessionPool` class for more information on how keeping a real session helps us avoid blocking.
-
-When no `sessionId` is provided, our proxy URLs are rotated round-robin.
+Each call to `proxyConfiguration.newUrl()` generates a new proxy URL. Crawler instances pair these URLs with `Session` instances and rotate those together with browser fingerprints, impersonated headers, and more. This is extremely useful in scraping, because we want to create the impression of a real user. See the [session management guide](../guides/session-management) and `SessionPool` class for more information on how keeping a real session helps us avoid blocking.
@@ -202,7 +171,7 @@ When no `sessionId` is provided, our proxy URLs are rotated round-robin.
## Inspecting current proxy in Crawlers
`HttpCrawler`, `CheerioCrawler`, `JSDOMCrawler`, `PlaywrightCrawler` and `PuppeteerCrawler` grant access to information about the currently used proxy
-in their `requestHandler` using a `proxyInfo` object.
+in their `requestHandler` using a `proxyInfo` object.
With the `proxyInfo` object, we can easily access the proxy URL.
diff --git a/docs/guides/proxy_management_session_cheerio.ts b/docs/guides/proxy_management_session_cheerio.ts
index bb19a5b88d35..1e23ec5d5b86 100644
--- a/docs/guides/proxy_management_session_cheerio.ts
+++ b/docs/guides/proxy_management_session_cheerio.ts
@@ -5,8 +5,7 @@ const proxyConfiguration = new ProxyConfiguration({
});
const crawler = new CheerioCrawler({
- useSessionPool: true,
- persistCookiesPerSession: true,
+ saveResponseCookies: true,
proxyConfiguration,
// ...
});
diff --git a/docs/guides/proxy_management_session_http.ts b/docs/guides/proxy_management_session_http.ts
index c8c289de4877..4677cb946273 100644
--- a/docs/guides/proxy_management_session_http.ts
+++ b/docs/guides/proxy_management_session_http.ts
@@ -5,8 +5,7 @@ const proxyConfiguration = new ProxyConfiguration({
});
const crawler = new HttpCrawler({
- useSessionPool: true,
- persistCookiesPerSession: true,
+ saveResponseCookies: true,
proxyConfiguration,
// ...
});
diff --git a/docs/guides/proxy_management_session_jsdom.ts b/docs/guides/proxy_management_session_jsdom.ts
index 98e71d904070..8162643bd1b3 100644
--- a/docs/guides/proxy_management_session_jsdom.ts
+++ b/docs/guides/proxy_management_session_jsdom.ts
@@ -5,8 +5,7 @@ const proxyConfiguration = new ProxyConfiguration({
});
const crawler = new JSDOMCrawler({
- useSessionPool: true,
- persistCookiesPerSession: true,
+ saveResponseCookies: true,
proxyConfiguration,
// ...
});
diff --git a/docs/guides/proxy_management_session_playwright.ts b/docs/guides/proxy_management_session_playwright.ts
index 70edcb79a033..c137f0191877 100644
--- a/docs/guides/proxy_management_session_playwright.ts
+++ b/docs/guides/proxy_management_session_playwright.ts
@@ -5,8 +5,7 @@ const proxyConfiguration = new ProxyConfiguration({
});
const crawler = new PlaywrightCrawler({
- useSessionPool: true,
- persistCookiesPerSession: true,
+ saveResponseCookies: true,
proxyConfiguration,
// ...
});
diff --git a/docs/guides/proxy_management_session_puppeteer.ts b/docs/guides/proxy_management_session_puppeteer.ts
index fcd1e14427f2..4e21121051a3 100644
--- a/docs/guides/proxy_management_session_puppeteer.ts
+++ b/docs/guides/proxy_management_session_puppeteer.ts
@@ -5,8 +5,7 @@ const proxyConfiguration = new ProxyConfiguration({
});
const crawler = new PuppeteerCrawler({
- useSessionPool: true,
- persistCookiesPerSession: true,
+ saveResponseCookies: true,
proxyConfiguration,
// ...
});
diff --git a/docs/guides/proxy_management_session_standalone.ts b/docs/guides/proxy_management_session_standalone.ts
index bc2010f79b18..dec095d03408 100644
--- a/docs/guides/proxy_management_session_standalone.ts
+++ b/docs/guides/proxy_management_session_standalone.ts
@@ -4,10 +4,4 @@ const proxyConfiguration = new ProxyConfiguration({
/* opts */
});
-const sessionPool = await SessionPool.open({
- /* opts */
-});
-
-const session = await sessionPool.getSession();
-
-const proxyUrl = await proxyConfiguration.newUrl(session.id);
+const proxyUrl = await proxyConfiguration.newUrl();
diff --git a/docs/guides/remote_browser.mdx b/docs/guides/remote_browser.mdx
new file mode 100644
index 000000000000..61e7a64b25f3
--- /dev/null
+++ b/docs/guides/remote_browser.mdx
@@ -0,0 +1,93 @@
+---
+id: remote-browser
+title: "Remote browser services"
+sidebar_label: "Remote browsers"
+description: Connect Crawlee crawlers to remote browser services like Browserbase, Browserless, or Steel.
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+import CodeBlock from '@theme/CodeBlock';
+
+import RemoteBrowserConfigSource from '!!raw-loader!./remote_browser_config.ts';
+import RemoteBrowserProviderSource from '!!raw-loader!./remote_browser_provider.ts';
+import RemoteBrowserPuppeteerSource from '!!raw-loader!./remote_browser_puppeteer.ts';
+
+Instead of launching a local browser, Crawlee can connect to a remote browser service like [Browserbase](https://browserbase.com/), [Browserless](https://browserless.io/), [Steel](https://steel.dev/), or any service that exposes a WebSocket/CDP endpoint. The crawler manages session rotation and the request lifecycle the same way it does locally — only the browser itself runs elsewhere.
+
+Use this when you need IPs in specific regions, want to offload CPU/memory from your runner, or need stealth features the service provides.
+
+## How it works
+
+Set the crawler's `remoteBrowser` option with the connection details. The crawler builds a `RemoteBrowserPool` around its own browser plugin, so the connection is always for the matching browser — there's no plugin to construct and no way to mismatch the pool with the crawler. The pool (an `IBrowserPool` wrapping the regular `BrowserPool`) owns everything remote: resolving the endpoint, releasing sessions when browsers close, and capping how many remote browsers run at once.
+
+## Basic usage
+
+The simplest form is a static connection URL. Use this when the service exposes a single endpoint and doesn't need per-session setup.
+
+{RemoteBrowserConfigSource}
+
+`endpoint` can also be a function returning `{ url, context }`, called once per browser launch. Pair it with a `release` callback (it receives the `context`) to clean up sessions on the service side when the browser closes, crashes, or the pool is destroyed.
+
+`maxOpenBrowsers` caps the number of concurrent remote browsers — set it to the service's concurrent-session limit to avoid 429 errors. The pool enforces it inside `newPage()`, which waits for a free slot rather than overshooting.
+
+### Self-hosted
+
+Some services ship a Docker image you can run locally or on your own infrastructure. For example, [Browserless](https://www.browserless.io/) has an open-source Chromium image:
+
+```bash
+docker run -p 3000:3000 -e CONCURRENT=4 ghcr.io/browserless/chromium
+```
+
+Point the pool at the local endpoint with `endpoint: 'ws://localhost:3000'`.
+
+## Custom provider
+
+For services with a session-create / session-release lifecycle, extend `RemoteBrowserProvider` and pass the instance as the pool's `endpoint`. `connect()` runs once per browser launch and returns the connection URL plus an optional `context` object passed back to `release()`. `maxOpenBrowsers` set on the provider is adopted by the pool.
+
+{RemoteBrowserProviderSource}
+
+## Puppeteer
+
+`PuppeteerCrawler` works the same way — build the pool with a `PuppeteerPlugin`. Puppeteer connects over CDP:
+
+{RemoteBrowserPuppeteerSource}
+
+For Playwright you can choose the protocol via the `remoteBrowser.connection.protocol` option: `'cdp'` (default, `connectOverCDP()`) or `'playwright'` (`connect()`, Playwright's own WebSocket protocol).
+
+## Sharing a pool across crawlers
+
+`remoteBrowser` builds a pool the crawler owns and tears down. To share one remote pool across multiple crawlers, construct a `RemoteBrowserPool` yourself and pass it as the `browserPool` option instead — a pool supplied that way is never destroyed by the crawler, so you control its lifecycle. Use `remoteBrowser` *or* `browserPool`, not both.
+
+Declare it with `await using` and the browsers (and their remote sessions) are released once you are done with the pool:
+
+```ts
+import { PlaywrightPlugin, RemoteBrowserPool } from '@crawlee/browser-pool';
+import { PlaywrightCrawler } from 'crawlee';
+import playwright from 'playwright';
+
+await using browserPool = new RemoteBrowserPool({
+ browserPlugins: [new PlaywrightPlugin(playwright.chromium)],
+ endpoint: 'wss://production-sfo.browserless.io?token=xxx',
+ maxOpenBrowsers: 2,
+});
+
+await new PlaywrightCrawler({ browserPool, requestHandler: async () => { /* ... */ } }).run(['https://crawlee.dev']);
+await new PlaywrightCrawler({ browserPool, requestHandler: async () => { /* ... */ } }).run(['https://apify.com']);
+```
+
+:::note Requires Node.js 24
+
+The `await using` syntax needs Node.js 24 or later. On Node.js 22 call `destroy()` yourself instead — it is what the disposal hook calls anyway.
+
+:::
+
+## Limitations
+
+- **`headless` and `launchOptions` don't apply.** The remote service controls headless mode and browser flags; configure them on the service side.
+- **`useIncognitoPages` is forced to `true`** for Playwright remote connections — `connect()` / `connectOverCDP()` don't accept persistent contexts. For state shared across requests, use the `SessionPool`.
+- **`userDataDir` has no effect** — there's no local profile when the browser runs remotely. Use the service's persistence API (e.g. Browserbase Contexts, Steel Profiles).
+
+## Further reading
+
+- `RemoteBrowserPool` API reference
+- `RemoteBrowserProvider` API reference
diff --git a/docs/guides/remote_browser_config.ts b/docs/guides/remote_browser_config.ts
new file mode 100644
index 000000000000..41f4e0542fe8
--- /dev/null
+++ b/docs/guides/remote_browser_config.ts
@@ -0,0 +1,19 @@
+import { PlaywrightCrawler } from 'crawlee';
+
+const token = process.env.BROWSERLESS_TOKEN!;
+
+const crawler = new PlaywrightCrawler({
+ // Connect to a remote browser instead of launching locally. The crawler builds the right
+ // pool for its browser — you only supply the connection details.
+ remoteBrowser: {
+ endpoint: `wss://production-sfo.browserless.io?token=${token}`,
+ // Optional — respect the service's concurrent session limit.
+ maxOpenBrowsers: 5,
+ },
+ async requestHandler({ page, request, log }) {
+ const title = await page.title();
+ log.info(`${request.loadedUrl} — "${title}"`);
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/guides/remote_browser_provider.ts b/docs/guides/remote_browser_provider.ts
new file mode 100644
index 000000000000..45594d0fe4f4
--- /dev/null
+++ b/docs/guides/remote_browser_provider.ts
@@ -0,0 +1,46 @@
+import { RemoteBrowserProvider } from '@crawlee/browser-pool';
+import { PlaywrightCrawler } from 'crawlee';
+
+const apiKey = process.env.BROWSERBASE_API_KEY!;
+const projectId = process.env.BROWSERBASE_PROJECT_ID!;
+
+class BrowserbaseProvider extends RemoteBrowserProvider<{ id: string }> {
+ // Respect the service's concurrent session limit to avoid 429s.
+ override maxOpenBrowsers = 5;
+
+ async connect() {
+ const response = await fetch('https://api.browserbase.com/v1/sessions', {
+ method: 'POST',
+ headers: { 'x-bb-api-key': apiKey, 'Content-Type': 'application/json' },
+ body: JSON.stringify({ projectId }),
+ });
+
+ if (!response.ok) {
+ throw new Error(`Failed to create session: ${response.status} ${response.statusText}`);
+ }
+
+ const session = (await response.json()) as { id: string; connectUrl: string };
+ return { url: session.connectUrl, context: { id: session.id } };
+ }
+
+ override async release({ id }: { id: string }) {
+ await fetch(`https://api.browserbase.com/v1/sessions/${id}`, {
+ method: 'POST',
+ headers: { 'x-bb-api-key': apiKey, 'Content-Type': 'application/json' },
+ body: JSON.stringify({ status: 'REQUEST_RELEASE' }),
+ });
+ }
+}
+
+const crawler = new PlaywrightCrawler({
+ // Pass the provider as the `endpoint`; the crawler's pool calls connect()/release() per browser.
+ remoteBrowser: {
+ endpoint: new BrowserbaseProvider(),
+ },
+ async requestHandler({ page, request, log }) {
+ const title = await page.title();
+ log.info(`${request.loadedUrl} — "${title}"`);
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/guides/remote_browser_puppeteer.ts b/docs/guides/remote_browser_puppeteer.ts
new file mode 100644
index 000000000000..2bfc14be3d65
--- /dev/null
+++ b/docs/guides/remote_browser_puppeteer.ts
@@ -0,0 +1,16 @@
+import { PuppeteerCrawler } from 'crawlee';
+
+const token = process.env.BROWSERLESS_TOKEN!;
+
+const crawler = new PuppeteerCrawler({
+ // PuppeteerCrawler connects over CDP. Same `remoteBrowser` option, matching browser guaranteed.
+ remoteBrowser: {
+ endpoint: `wss://production-sfo.browserless.io?token=${token}`,
+ },
+ async requestHandler({ page, request, log }) {
+ const title = await page.title();
+ log.info(`${request.loadedUrl} — "${title}"`);
+ },
+});
+
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/guides/request_loaders.mdx b/docs/guides/request_loaders.mdx
new file mode 100644
index 000000000000..57a9794dc729
--- /dev/null
+++ b/docs/guides/request_loaders.mdx
@@ -0,0 +1,242 @@
+---
+id: request-loaders
+title: Request loaders
+description: How to manage the requests your crawler will go through.
+---
+
+import ApiLink from '@site/src/components/ApiLink';
+
+import Tabs from '@theme/Tabs';
+import TabItem from '@theme/TabItem';
+import CodeBlock from '@theme/CodeBlock';
+
+import RlBasicSource from '!!raw-loader!./request_loaders_rl_basic.ts';
+import SitemapBasicSource from '!!raw-loader!./request_loaders_sitemap_basic.ts';
+import RlTandemExplicitSource from '!!raw-loader!./request_loaders_rl_tandem_explicit.ts';
+import RlTandemHelperSource from '!!raw-loader!./request_loaders_rl_tandem_helper.ts';
+import SitemapTandemExplicitSource from '!!raw-loader!./request_loaders_sitemap_tandem_explicit.ts';
+import SitemapTandemHelperSource from '!!raw-loader!./request_loaders_sitemap_tandem_helper.ts';
+
+Request loaders extend the functionality of the `RequestQueue`, providing additional tools for managing URLs and requests. If you are new to Crawlee and unfamiliar with the `RequestQueue`, consider starting with the [Request storage](./request-storage) guide first. Request loaders define how requests are fetched and stored, enabling various use cases such as reading URLs from a static list, a sitemap, an external API, or combining multiple sources together.
+
+## Overview
+
+The request loader abstractions are built around two interfaces and a couple of helpers:
+
+- `IRequestLoader`: The base interface for reading requests in a crawl.
+- `IRequestManager`: Extends `IRequestLoader` with write capabilities (adding and reclaiming requests).
+- `RequestManagerTandem`: Combines a read-only `IRequestLoader` with a writable `IRequestManager`.
+- `ThrottlingRequestManager`: Wraps a writable `IRequestManager` and paces requests per domain.
+
+And the concrete request loader implementations:
+
+- `RequestList`: A lightweight implementation for managing a static list of URLs.
+- `SitemapRequestLoader`: A specialized loader that reads URLs from XML and plain-text sitemaps following the [Sitemaps protocol](https://www.sitemaps.org/protocol.html), with filtering capabilities.
+
+Below is a class diagram that illustrates the relationships between these components and the `RequestQueue`:
+
+```mermaid
+---
+config:
+ class:
+ hideEmptyMembersBox: true
+---
+
+classDiagram
+
+%% ========================
+%% Abstract interfaces
+%% ========================
+
+class IRequestLoader {
+ <>
+ + getTotalCount()
+ + getPendingCount()
+ + getHandledCount()
+ + fetchNextRequest()
+ + markRequestAsHandled()
+ + isEmpty()
+ + isFinished()
+ + toTandem()
+}
+
+class IRequestManager {
+ <>
+ + addRequest()
+ + addRequestsBatched()
+ + reclaimRequest()
+ + purge()
+}
+
+%% ========================
+%% Concrete classes
+%% ========================
+
+class RequestQueue
+
+class RequestList
+
+class SitemapRequestLoader
+
+class RequestManagerTandem
+
+class ThrottlingRequestManager
+
+%% ========================
+%% Inheritance arrows
+%% ========================
+
+IRequestLoader <|-- IRequestManager
+IRequestLoader <|.. RequestList
+IRequestLoader <|.. SitemapRequestLoader
+IRequestManager <|.. RequestQueue
+IRequestManager <|.. RequestManagerTandem
+IRequestManager <|.. ThrottlingRequestManager
+```
+
+:::info Crawler usage
+
+A crawler reads its requests from a single `IRequestManager`, passed via the `requestManager` option. A `RequestQueue` is itself a request manager, so it can be passed directly. A read-only loader (such as `RequestList`) cannot — combine it with a queue into a tandem first, see the [Request manager tandem](#request-manager-tandem) section below.
+
+:::
+
+## Request loaders
+
+The `IRequestLoader` interface defines the foundation for fetching requests during a crawl. It provides methods for basic operations like retrieving the next request, marking requests as handled, and checking whether the loader is empty or finished. It is intentionally **read-only** — it does not allow adding new requests. Concrete implementations such as `RequestList` build on this interface to handle specific scenarios. You can create your own custom loader that reads from an external file, web endpoint, database, or any other data source.
+
+### Request list
+
+The `RequestList` manages a static list of URLs to crawl. The list is created for a single crawler run and, unlike a queue, cannot have requests added to or removed from it after initialization. It can hold a large number of URLs (even millions) with significantly lower overhead than enqueueing them one by one.
+
+Here is a basic example of working with the `RequestList`:
+
+
+ {RlBasicSource}
+
+
+### Sitemap request loader
+
+The `SitemapRequestLoader` is a specialized request loader that reads URLs from sitemaps following the [Sitemaps protocol](https://www.sitemaps.org/protocol.html). It supports both XML and plain-text sitemap formats and is particularly useful when you want to crawl a website systematically by following its sitemap structure. Loading happens in the background, so crawling can start before the sitemap is fully parsed.
+
+:::note
+
+The `SitemapRequestLoader` is designed specifically for sitemaps that follow the standard Sitemaps protocol. HTML pages containing links are not supported by this loader — those should be handled by regular crawlers using the `enqueueLinks` functionality.
+
+:::
+
+The loader supports filtering URLs using glob patterns and regular expressions, allowing you to include or exclude specific types of URLs.
+
+
+ {SitemapBasicSource}
+
+
+## Request managers
+
+The `IRequestManager` interface extends `IRequestLoader` with **write** capabilities. In addition to reading requests, a request manager can add new requests and reclaim failed ones. This is essential for dynamic crawling, where new URLs emerge during the crawl, or when requests fail and need to be retried. The `RequestQueue` is the primary built-in request manager — see the [Request storage](./request-storage) guide for details.
+
+## Per-domain throttling
+
+Some sites answer bursts of traffic with HTTP 429 (Too Many Requests) rather than an outright block. By default a 429 is treated as a blocked session: the session is retired and the request is retried straight away on a fresh one, which churns through proxies without actually slowing down.
+
+The `ThrottlingRequestManager` handles it at the scheduling layer instead. Wrap your request manager in it and list the domains you want paced:
+
+```ts
+import { CheerioCrawler, RequestQueue, ThrottlingRequestManager } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ requestManager: new ThrottlingRequestManager({
+ inner: await RequestQueue.open(),
+ domains: ['api.example.com'],
+ // optional, these are the defaults
+ baseDelaySecs: 2,
+ maxDelaySecs: 60,
+ maxDomainStallSecs: 900,
+ }),
+ requestHandler: async ({ request }) => {
+ // ...
+ },
+});
+```
+
+Requests for a listed domain are routed into their own queue as they are added. When one of those domains answers with a 429, the crawler honours its `Retry-After` header — or backs off exponentially from `baseDelaySecs` up to `maxDelaySecs` if there is none — and holds that domain's requests back for the duration. Requests for every other domain keep flowing at full speed, the throttled request is retried later without counting against `maxRequestRetries`, and its session is left alone, because a rate limit says nothing about the session.
+
+Because a throttled request costs no retries, a domain that never stops rate-limiting would otherwise keep the crawl alive forever. If one goes `maxDomainStallSecs` without letting a single request through, the crawl shuts down with a `PersistentRateLimitError` — at that point the concurrency is too high for that domain, or it has blocked you outright, and waiting longer will not help. Its requests are left in their queue on purpose, so re-running the crawl with `purgeOnStart` disabled resumes them if the rate limit lifts. A crawler running with `keepAlive` is exempt, since staying up regardless is what it was asked to do.
+
+Matching is exact and case-insensitive, with no wildcard support, so list each subdomain you care about — or set `throttleBy: 'registrableDomain'`, which groups a site and all of its subdomains under a single set of clocks.
+
+### The two clocks
+
+Every throttled domain runs two of them, and is dispatched to once both have run out:
+
+- **Backoff** — reactive and temporary, set by HTTP 429 responses and decaying once the domain stops turning you away.
+- **Crawl delay** — proactive and constant, a minimum interval between two dispatches, armed after each one. It is whatever the domain's robots.txt asks for (with `respectRobotsTxtFile` enabled), floored by `minCrawlDelaySecs`. A domain asking for longer gets longer.
+
+### Throttling every domain
+
+`domains: 'all'` gives a clock to each domain the crawl encounters instead of a fixed list, and a queue of its own the first time it is seen — so a domain that is waiting out a delay is skipped rather than repeatedly fetched and put back:
+
+```ts
+const crawler = new CheerioCrawler({
+ requestManager: new ThrottlingRequestManager({
+ inner: await RequestQueue.open(),
+ domains: 'all',
+ minCrawlDelaySecs: 1,
+ throttleBy: 'registrableDomain',
+ }),
+ requestHandler: async ({ request }) => {
+ // ...
+ },
+});
+```
+
+This is what the crawler's own `sameDomainDelaySecs` option is built on — it wraps the crawler's request manager in a `ThrottlingRequestManager` configured exactly like the above. Dropping `minCrawlDelaySecs` is just as useful: 429 backoff and robots.txt `Crawl-delay` then apply to every domain, with no pacing of your own on top.
+
+A queue per domain is not free, so a run may only throttle `maxThrottledDomains` of them (100 by default) before it throws. If you are crawling more domains than that, pace the crawl with `maxRequestsPerMinute` instead. The list of domains discovered so far is kept in the default key-value store, under `persistStateKey`, so that a restart reopens their queues rather than leaving whatever they still hold uncrawled.
+
+## Request manager tandem
+
+The `RequestManagerTandem` class combines the read-only capabilities of an `IRequestLoader` (like `RequestList`) with the read-write capabilities of an `IRequestManager` (like `RequestQueue`). This is useful when you need to load initial requests from a static source (such as a file, sitemap, or database) and also dynamically add or retry requests during the crawl.
+
+Under the hood, the tandem checks whether the read-only loader still has pending requests. If so, each request from the loader is transferred to the manager (the queue) before being processed. Any newly added or reclaimed requests go directly to the manager side. Because every request passes through the queue, deduplication and retries are handled consistently and a single URL is not crawled multiple times.
+
+The easiest way to build a tandem is the `toTandem()` helper available on the loaders. Called without arguments, it pairs the loader with the default `RequestQueue`; you can also pass a specific request manager to use instead.
+
+### Request list with request queue
+
+This setup is useful when you have a static list of URLs to crawl, but also need to handle dynamic requests discovered during the crawl. Requests from the `RequestList` are processed first by being enqueued into the `RequestQueue`, which handles persistence and retries.
+
+
+
+
+ {RlTandemHelperSource}
+
+
+
+
+ {RlTandemExplicitSource}
+
+
+
+
+### Sitemap request loader with request queue
+
+Similarly, you can combine a `SitemapRequestLoader` with a `RequestQueue`. This is particularly useful when you want to crawl URLs from a sitemap while also handling dynamic requests discovered during the crawl. URLs from the sitemap are processed first by being enqueued into the queue, which handles persistence and retries.
+
+
+
+
+ {SitemapTandemHelperSource}
+
+
+
+
+ {SitemapTandemExplicitSource}
+
+
+
+
+## Conclusion
+
+This guide introduced the request loader abstractions: the read-only `IRequestLoader`, the writable `IRequestManager`, and the `RequestManagerTandem` that combines them, along with the `RequestList` and `SitemapRequestLoader` implementations. You also saw how to pair a loader with a queue using the `toTandem()` helper to handle both static and dynamically discovered requests, and how `ThrottlingRequestManager` paces requests to individual domains.
+
+If you have questions or need assistance, feel free to reach out on our [GitHub](https://github.com/apify/crawlee) or join our [Discord community](https://discord.com/invite/jyEM2PRvMU). Happy scraping!
diff --git a/docs/guides/request_loaders_rl_basic.ts b/docs/guides/request_loaders_rl_basic.ts
new file mode 100644
index 000000000000..bea941f734db
--- /dev/null
+++ b/docs/guides/request_loaders_rl_basic.ts
@@ -0,0 +1,15 @@
+import { RequestList } from 'crawlee';
+
+// Open a request list with a static set of URLs.
+// The name is used to persist the list's state in the default key-value store.
+const requestList = await RequestList.open('my-list', [
+ 'https://crawlee.dev/',
+ 'https://crawlee.dev/docs',
+ 'https://crawlee.dev/api',
+]);
+
+// Iterate over the requests manually (a crawler does this for you under the hood).
+for await (const request of requestList) {
+ console.log(request.url);
+ await requestList.markRequestAsHandled(request);
+}
diff --git a/docs/guides/request_loaders_rl_tandem_explicit.ts b/docs/guides/request_loaders_rl_tandem_explicit.ts
new file mode 100644
index 000000000000..8014ddc79337
--- /dev/null
+++ b/docs/guides/request_loaders_rl_tandem_explicit.ts
@@ -0,0 +1,21 @@
+import { CheerioCrawler, RequestList, RequestManagerTandem, RequestQueue } from 'crawlee';
+
+// A static list of URLs to start from (can hold millions of URLs).
+const requestList = await RequestList.open('my-list', ['https://crawlee.dev/', 'https://crawlee.dev/docs']);
+
+// A writable queue that holds requests discovered during the crawl.
+const requestQueue = await RequestQueue.open();
+
+// Combine them: the tandem reads from the list first, transferring each request
+// into the queue, and lets you enqueue new requests during the crawl.
+const requestManager = new RequestManagerTandem(requestList, requestQueue);
+
+const crawler = new CheerioCrawler({
+ requestManager,
+ async requestHandler({ enqueueLinks }) {
+ // Newly discovered links go to the queue side of the tandem.
+ await enqueueLinks();
+ },
+});
+
+await crawler.run();
diff --git a/docs/guides/request_loaders_rl_tandem_helper.ts b/docs/guides/request_loaders_rl_tandem_helper.ts
new file mode 100644
index 000000000000..8637cb510e9a
--- /dev/null
+++ b/docs/guides/request_loaders_rl_tandem_helper.ts
@@ -0,0 +1,17 @@
+import { CheerioCrawler, RequestList } from 'crawlee';
+
+// A static list of URLs to start from.
+const requestList = await RequestList.open('my-list', ['https://crawlee.dev/', 'https://crawlee.dev/docs']);
+
+// `toTandem()` is a shortcut that pairs the loader with a request queue.
+// Without arguments it opens the default `RequestQueue`.
+const requestManager = await requestList.toTandem();
+
+const crawler = new CheerioCrawler({
+ requestManager,
+ async requestHandler({ enqueueLinks }) {
+ await enqueueLinks();
+ },
+});
+
+await crawler.run();
diff --git a/docs/guides/request_loaders_sitemap_basic.ts b/docs/guides/request_loaders_sitemap_basic.ts
new file mode 100644
index 000000000000..29bc87f46334
--- /dev/null
+++ b/docs/guides/request_loaders_sitemap_basic.ts
@@ -0,0 +1,14 @@
+import { SitemapRequestLoader } from 'crawlee';
+
+// Open a sitemap request list. The sitemap is fetched and parsed in the background,
+// so crawling can start before the whole sitemap is loaded.
+const sitemapRequestLoader = await SitemapRequestLoader.open({
+ sitemapUrls: ['https://crawlee.dev/sitemap.xml'],
+ // Optionally filter the URLs read from the sitemap:
+ // include: ['https://crawlee.dev/docs/**'],
+});
+
+for await (const request of sitemapRequestLoader) {
+ console.log(request.url);
+ await sitemapRequestLoader.markRequestAsHandled(request);
+}
diff --git a/docs/guides/request_loaders_sitemap_tandem_explicit.ts b/docs/guides/request_loaders_sitemap_tandem_explicit.ts
new file mode 100644
index 000000000000..48d2f936e9cd
--- /dev/null
+++ b/docs/guides/request_loaders_sitemap_tandem_explicit.ts
@@ -0,0 +1,20 @@
+import { CheerioCrawler, RequestManagerTandem, RequestQueue, SitemapRequestLoader } from 'crawlee';
+
+// Read the initial URLs from a sitemap.
+const sitemapRequestLoader = await SitemapRequestLoader.open({
+ sitemapUrls: ['https://crawlee.dev/sitemap.xml'],
+});
+
+// A writable queue for requests discovered during the crawl.
+const requestQueue = await RequestQueue.open();
+
+const requestManager = new RequestManagerTandem(sitemapRequestLoader, requestQueue);
+
+const crawler = new CheerioCrawler({
+ requestManager,
+ async requestHandler({ enqueueLinks }) {
+ await enqueueLinks();
+ },
+});
+
+await crawler.run();
diff --git a/docs/guides/request_loaders_sitemap_tandem_helper.ts b/docs/guides/request_loaders_sitemap_tandem_helper.ts
new file mode 100644
index 000000000000..bcf1c2ea0715
--- /dev/null
+++ b/docs/guides/request_loaders_sitemap_tandem_helper.ts
@@ -0,0 +1,18 @@
+import { CheerioCrawler, SitemapRequestLoader } from 'crawlee';
+
+// Read the initial URLs from a sitemap.
+const sitemapRequestLoader = await SitemapRequestLoader.open({
+ sitemapUrls: ['https://crawlee.dev/sitemap.xml'],
+});
+
+// Pair the loader with the default `RequestQueue` via the `toTandem()` shortcut.
+const requestManager = await sitemapRequestLoader.toTandem();
+
+const crawler = new CheerioCrawler({
+ requestManager,
+ async requestHandler({ enqueueLinks }) {
+ await enqueueLinks();
+ },
+});
+
+await crawler.run();
diff --git a/docs/guides/request_storage.mdx b/docs/guides/request_storage.mdx
index 8da5489b5faf..01f1808efd2c 100644
--- a/docs/guides/request_storage.mdx
+++ b/docs/guides/request_storage.mdx
@@ -14,7 +14,6 @@ import BasicOperationsSource from '!!raw-loader!./request_storage_queue_basic.ts
import CrawlerExplicitSource from '!!raw-loader!./request_storage_queue_crawler_explicit.ts';
import CrawlerSource from '!!raw-loader!./request_storage_queue_crawler.ts';
-import RequestQueueListSource from '!!raw-loader!./request_storage_queue_list.ts';
import RequestQueueAddRequestsSource from '!!raw-loader!./request_storage_queue_only.ts';
Crawlee has several request storage types that are useful for specific tasks. The requests are stored on local disk to a directory defined by the `CRAWLEE_STORAGE_DIR` environment variable. If this variable is not defined, by default Crawlee sets `CRAWLEE_STORAGE_DIR` to `./storage` in the current working directory.
@@ -27,7 +26,7 @@ Each Crawlee project run is associated with a **default request queue**. Typical
In Crawlee, the request queue is represented by the `RequestQueue` class.
-The request queue is managed by `MemoryStorage` class and its data is stored in memory, while also being off-loaded to the local directory specified by the `CRAWLEE_STORAGE_DIR` environment variable as follows:
+By default, the request queue is managed by the `FileSystemStorageBackend` class and its data is stored in the local directory specified by the `CRAWLEE_STORAGE_DIR` environment variable as follows:
```text
{CRAWLEE_STORAGE_DIR}/request_queues/{QUEUE_ID}/entries.json
@@ -67,75 +66,21 @@ The following code demonstrates the usage of the request queue:
To see more detailed example of how to use the request queue with a crawler, see the [Puppeteer Crawler](/js/docs/examples/puppeteer-crawler) example.
-## Request list
+The request queue is not optimized for adding numerous URLs in a single batch — historically, requests were added one by one. To enqueue a large set of initial URLs efficiently, use the `addRequests()` method (or simply pass the URLs to `crawler.run()`), which adds requests in batches:
-The request list is not a storage per se - it represents the list of URLs to crawl that is stored in a crawler run memory (or optionally in default [Key-Value Store](../guides/result-storage#key-value-store) associated with the run, if specified). The list is used for the crawling of a large number of URLs, when we know all the URLs which should be visited by the crawler and no URLs would be added during the run. The URLs can be provided either in code or parsed from a text file hosted on the web.
+
+ {RequestQueueAddRequestsSource}
+
-Request list is created exclusively for the crawler run and only if its usage is explicitly specified in the code. Its usage is optional.
+## Reading requests from other sources
-In Crawlee, the request list is represented by the `RequestList` class.
+Sometimes you don't want to start from a dynamic queue, but from a static list of URLs (for example, parsed from a file) or from a website's sitemap. Crawlee provides **request loaders** for these read-only sources — `RequestList` and `SitemapRequestLoader` — which can be combined with a request queue when you also need to enqueue requests discovered during the crawl.
-The following code demonstrates basic operations of the request list:
-
-```javascript
-import { RequestList, PuppeteerCrawler } from 'crawlee';
-
-// Prepare the sources array with URLs to visit
-const sources = [
- { url: 'http://www.example.com/page-1' },
- { url: 'http://www.example.com/page-2' },
- { url: 'http://www.example.com/page-3' },
-];
-
-// Open the request list.
-// List name is used to persist the sources and the list state in the key-value store
-const requestList = await RequestList.open('my-list', sources);
-
-// The crawler will automatically process requests from the list
-// It's used the same way for Cheerio /Playwright crawlers.
-const crawler = new PuppeteerCrawler({
- requestList,
- async requestHandler({ page, request }) {
- // Process the page (extract data, take page screenshot, etc).
- // No more requests could be added to the request list here
- },
-});
-```
-
-## Which one to choose?
-
-When using Request queue - we would normally have several start URLs (e.g. category pages on e-commerce website) and then recursively add more (e.g. individual item pages) programmatically to the queue, it supports dynamic adding and removing of requests. No more URLs can be added to Request list after its initialization as it is immutable, URLs cannot be removed from the list either.
-
-On the other hand, the Request queue is not optimized for adding or removing numerous URLs in a batch. This is technically possible, but requests are added one by one to the queue, and thus it would take significant time with a larger number of requests. Request list however can contain even millions of URLs, and it would take significantly less time to add them to the list, compared to the queue.
-
-Note that Request queue and Request list can be used together by the same crawler. In such cases, each request from the Request list is enqueued into the Request queue first (to the foremost position in the queue, even if Request queue is not empty) and then consumed from the latter. This is necessary to avoid the same URL being processed more than once (from the list first and then possibly from the queue). In practical terms, such a combination can be useful when there are numerous initial URLs, but more URLs would be added dynamically by the crawler.
-
-:::tip
-
-In Crawlee, there is not much need to combine the request queue together with the request list (although it's technically possible).
-
-Previously there was no way to add the initial requests to the queue in batches (to add an array of requests), i.e. we could have only added the requests one by one to the queue with the help of `addRequest()` function.
-
-However, now we could use the `addRequests()` function, which adds requests in batches. Thus, instead of combining the request queue and the request list, we can use only the request queue for such use-cases now. See the examples below.
-
-:::
-
-
-
-
- {RequestQueueAddRequestsSource}
-
-
-
-
- {RequestQueueListSource}
-
-
-
+See the dedicated [Request loaders](./request-loaders) guide for details on loaders, request managers, and how to combine them with a queue into a `RequestManagerTandem`.
## Cleaning up the storages
-Default storages are purged before the crawler starts if not specified otherwise. This happens as early as when we try to open some storage (e.g. via `RequestQueue.open()`) or when we try to work with a default storage via one of the helper methods (e.g. `crawler.addRequests()` that under the hood calls `RequestQueue.open()`). If we don't work with storages explicitly in our code, the purging will eventually happen when the `run` method of our crawler is executed. In case we need to purge the storages sooner, we can use the `purgeDefaultStorages()` helper explicitly:
+Run-scoped storages - the default one and any opened with an `alias` - are purged before the crawler starts if not specified otherwise. Storages opened with a `name` persist across runs and are never purged. This happens as early as when we try to open some storage (e.g. via `RequestQueue.open()`) or when we try to work with a default storage via one of the helper methods (e.g. `crawler.addRequests()` that under the hood calls `RequestQueue.open()`). If we don't work with storages explicitly in our code, the purging will eventually happen when the `run` method of our crawler is executed. In case we need to purge the storages sooner, we can use the `purgeDefaultStorages()` helper explicitly:
```javascript
import { purgeDefaultStorages } from 'crawlee';
@@ -143,4 +88,4 @@ import { purgeDefaultStorages } from 'crawlee';
await purgeDefaultStorages();
```
-Calling this function will clean up the default request storage directory (and also the request list stored in default key-value store). This is a shortcut for running (optional) `purge` method on the `StorageClient` interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. You can make sure the storage is purged only once for a given execution context if you set `onlyPurgeOnce` to `true` in the `options` object.
+Calling this function will clean up the run-scoped request storage directories - the default queue and any alias-keyed one - along with the request list stored in the default key-value store. This is a shortcut for running (optional) `purge` method on the `StorageBackend` interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. You can make sure the storage is purged only once for a given execution context if you set `onlyPurgeOnce` to `true` in the `options` object.
diff --git a/docs/guides/request_storage_queue_basic.ts b/docs/guides/request_storage_queue_basic.ts
index 66d3d337212d..1555e5bbd595 100644
--- a/docs/guides/request_storage_queue_basic.ts
+++ b/docs/guides/request_storage_queue_basic.ts
@@ -11,7 +11,7 @@ await requestQueue.addRequests([
]);
// Open the named request queue
-const namedRequestQueue = await RequestQueue.open('named-queue');
+const namedRequestQueue = await RequestQueue.open({ name: 'named-queue' });
// Remove the named request queue
await namedRequestQueue.drop();
diff --git a/docs/guides/request_storage_queue_crawler.ts b/docs/guides/request_storage_queue_crawler.ts
index 07af11ffa712..d9c37f57f7de 100644
--- a/docs/guides/request_storage_queue_crawler.ts
+++ b/docs/guides/request_storage_queue_crawler.ts
@@ -4,7 +4,7 @@ import { CheerioCrawler } from 'crawlee';
// It's used the same way for Puppeteer/Playwright crawlers.
const crawler = new CheerioCrawler({
// Note that we're not specifying the requestQueue here
- async requestHandler({ crawler, enqueueLinks }) {
+ async requestHandler({ enqueueLinks }) {
// Add new request to the queue
await crawler.addRequests([{ url: 'https://example.com/new-page' }]);
// Add links found on page to the queue
diff --git a/docs/guides/request_storage_queue_list.ts b/docs/guides/request_storage_queue_list.ts
deleted file mode 100644
index 456caf1c015c..000000000000
--- a/docs/guides/request_storage_queue_list.ts
+++ /dev/null
@@ -1,42 +0,0 @@
-// This is technically correct, but
-// we need to explicitly open/use both the request queue and the request list.
-// We suggest using the request queue and batch add the requests instead.
-
-import { RequestList, RequestQueue, PuppeteerCrawler } from 'crawlee';
-
-// Prepare the sources array with URLs to visit (it can contain millions of URLs)
-const sources = [
- { url: 'http://www.example.com/page-1' },
- { url: 'http://www.example.com/page-2' },
- { url: 'http://www.example.com/page-3' },
- // ...
-];
-
-// Open the request list with the initial sources array
-const requestList = await RequestList.open('my-list', sources);
-
-// Open the default request queue. It's not necessary to add any requests to the queue
-const requestQueue = await RequestQueue.open();
-
-// The crawler will automatically process requests from the list and the queue.
-// It's used the same way for Cheerio/Playwright crawlers
-const crawler = new PuppeteerCrawler({
- requestList,
- requestQueue,
- // Each request from the request list is enqueued to the request queue one by one.
- // At this point request with the same URL would exist in the list and the queue
- async requestHandler({ crawler, enqueueLinks }) {
- // Add new request to the queue
- await crawler.addRequests(['http://www.example.com/new-page']);
-
- // Add links found on page to the queue
- await enqueueLinks();
-
- // The requests above would be added to the queue (but not to the list)
- // and would be processed after the request list is empty.
- // No more requests could be added to the list here
- },
-});
-
-// Run the crawler
-await crawler.run();
diff --git a/docs/guides/request_storage_queue_only.ts b/docs/guides/request_storage_queue_only.ts
index 5d9a31379597..3054135504f3 100644
--- a/docs/guides/request_storage_queue_only.ts
+++ b/docs/guides/request_storage_queue_only.ts
@@ -15,7 +15,7 @@ const sources = [
// The crawler will automatically process requests from the queue.
// It's used the same way for Cheerio/Playwright crawlers
const crawler = new PuppeteerCrawler({
- async requestHandler({ crawler, enqueueLinks }) {
+ async requestHandler({ enqueueLinks }) {
// Add new request to the queue
await crawler.addRequests(['http://www.example.com/new-page']);
diff --git a/docs/guides/result_storage.mdx b/docs/guides/result_storage.mdx
index eff313aef81f..af17763e0fbc 100644
--- a/docs/guides/result_storage.mdx
+++ b/docs/guides/result_storage.mdx
@@ -8,7 +8,7 @@ import ApiLink from '@site/src/components/ApiLink';
Crawlee has several result storage types that are useful for specific tasks. The data is stored on a local disk to the directory defined by the `CRAWLEE_STORAGE_DIR` environment variable. If this variable is not defined, by default Crawlee sets `CRAWLEE_STORAGE_DIR` to `./storage` in the current working directory.
-Crawlee storage is managed by `MemoryStorage` class. During the crawler run all information is stored in memory, while also being off-loaded to the local files in respective storage type folders.
+By default, Crawlee storage is managed by the `FileSystemStorageBackend` class, which stores all information as local files in the respective storage type folders.
## Key-value store
@@ -102,7 +102,7 @@ To see how to use the dataset to store crawler results, see the [Cheerio Crawler
## Cleaning up the storages
-Default storages are purged before the crawler starts if not specified otherwise. This happens as early as when we try to open some storage (e.g. via `Dataset.open()`) or when we try to work with a default storage via one of the helper methods (e.g. `Dataset.pushData()` that under the hood calls `Dataset.open()`). If we don't work with storages explicitly in our code, the purging will eventually happen when the `run` method of our crawler is executed. In case we need to purge the storages sooner, we can use the `purgeDefaultStorages()` helper explicitly:
+Run-scoped storages - the default one and any opened with an `alias` - are purged before the crawler starts if not specified otherwise. Storages opened with a `name` persist across runs and are never purged. This happens as early as when we try to open some storage (e.g. via `Dataset.open()`) or when we try to work with a default storage via one of the helper methods (e.g. `Dataset.pushData()` that under the hood calls `Dataset.open()`). If we don't work with storages explicitly in our code, the purging will eventually happen when the `run` method of our crawler is executed. In case we need to purge the storages sooner, we can use the `purgeDefaultStorages()` helper explicitly:
```javascript
import { purgeDefaultStorages } from 'crawlee';
@@ -110,4 +110,84 @@ import { purgeDefaultStorages } from 'crawlee';
await purgeDefaultStorages();
```
-Calling this function will clean up the default results storage directories except the `INPUT` key in default key-value store directory. This is a shortcut for running (optional) `purge` method on the `StorageClient` interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. In addition, this method will make sure the storage is purged only once for a given execution context, so it is safe to call it multiple times.
+Calling this function will clean up the run-scoped results storage directories - the default ones and any alias-keyed one - except the `INPUT` key in the default key-value store directory. This is a shortcut for running (optional) `purge` method on the `StorageBackend` interface, in other words it will call the `purge` method of the underlying storage implementation we are currently using. In addition, this method will make sure the storage is purged only once for a given execution context, so it is safe to call it multiple times.
+
+## Transactional storage
+
+A request handler either finishes, or it doesn't. Without further care, a handler that pushes a few dataset items, updates a key-value record and *then* throws would leave those writes behind — and when the request is retried, they would happen again, duplicating your data.
+
+Crawlee prevents that by wrapping every request in a **storage transaction**. Writes made through the storage classes (`Dataset`, `KeyValueStore`, `RequestQueue`) and the context helpers (`pushData`, `enqueueLinks`, ...) while a request is being handled are recorded rather than applied. When the request handler succeeds, they are replayed into real storage all together; when it throws, they are dropped. A retry therefore never double-writes, and a failed request leaves no partial results behind.
+
+This is **on by default** for every crawler and covers the whole request lifecycle — `preNavigationHooks`, `postNavigationHooks`, `extendContext` and the request handler alike.
+
+### What you can rely on
+
+- **Atomicity per request.** A request's storage writes become visible all together on success, or not at all on failure.
+- **Read-your-own-writes.** A read that follows a write in the same handler sees the written value — `getData()`, `getValue()`, key listings, dataset iteration and `getInfo()` all merge the handler's buffered writes with the real storage contents.
+- **Isolation of uncommitted writes.** One handler's uncommitted writes are invisible to concurrently running handlers — to pass data between handlers, use `useState()` instead of the key-value store.
+- **Fidelity.** Values are captured at write time (via `structuredClone`), so mutating an object after passing it to `pushData()` or `setValue()` affects neither what the handler reads back nor what gets committed.
+
+### What the transaction deliberately does *not* cover
+
+- **Request queue additions (by default).** Adding requests is idempotent (the queue deduplicates by `uniqueKey`), and buffering them would starve the crawl of new work until the handler finishes. New requests are therefore applied immediately (the `writeThrough` policy) and are *not* rolled back when the handler fails. If you need strict all-or-nothing enqueues, opt into buffering:
+
+ ```typescript
+ const crawler = new CheerioCrawler({
+ transactionalStorage: { requestQueue: 'deferred' },
+ // ...
+ });
+ ```
+
+ Under the `deferred` policy, the `requestId` returned by an `addRequests()` call inside a handler is provisional — the real id is assigned by the storage backend when the transaction commits.
+
+- **Shared state.** The object returned by `useState()` / `KeyValueStore.getAutoSavedValue()` is the sanctioned live channel shared by all handlers, and it stays live: mutations of it are **not** rolled back when a handler fails.
+
+- **Cross-storage atomicity.** The commit spans multiple storages and multiple calls; delivery is *at-least-once*. If a commit fails partway, the request fails and is retried, and the retry may re-apply what already landed.
+
+- **Error handlers.** `errorHandler` and `failedRequestHandler` run after the failed request's transaction has been rolled back, so their writes go straight to real storage — that is what error handlers are for.
+
+- **Deferred cleanups.** Callbacks registered with `registerDeferredCleanup()` run after the transaction is closed, so their writes land immediately and are not rolled back. In `AdaptivePlaywrightCrawler` they run once per request handler attempt, so a write there can land twice for one request — push your results from the request handler instead.
+
+### Escape hatches
+
+The feature is escapable at three granularities:
+
+- **Whole feature:** `transactionalStorage: false` on the crawler options disables the mechanism entirely; every storage call behaves exactly as if no transactions existed. (Not supported by `AdaptivePlaywrightCrawler`, which needs per-attempt buffering to work at all.)
+- **Per storage type:** the `transactionalStorage: { requestQueue: 'writeThrough' | 'deferred' }` policy object, described above.
+- **Per call site:** `withDirectStorageAccess()` runs a callback outside the transaction, so its writes land immediately and are never rolled back. Use it for progress files, streaming output, and anything that genuinely must not wait for the handler to succeed:
+
+ ```typescript
+ import { withDirectStorageAccess } from 'crawlee';
+
+ async function requestHandler({ request }) {
+ await withDirectStorageAccess(async () => {
+ const store = await KeyValueStore.open();
+ await store.setValue(`progress-${request.id}`, { startedAt: new Date() });
+ });
+ // ... the rest of the handler is transactional as usual
+ }
+ ```
+
+### Operations that throw inside a transaction
+
+A few operations cannot be buffered, and silently letting them through would produce storage states that no rollback can undo. They throw inside a transaction, with an error pointing at `withDirectStorageAccess()`:
+
+- `Dataset.drop()`, `KeyValueStore.drop()`, `RequestQueue.drop()` and `RequestQueue.purge()`,
+- the request queue processing internals (`fetchNextRequest()`, `markRequestAsHandled()`, `reclaimRequest()`),
+- `KeyValueStore.setValue()` with a **stream** value — a stream can only be consumed once, so it cannot serve both a read within the handler and the commit replay. Write streams under `withDirectStorageAccess()`.
+
+### Programmatic use
+
+Outside a crawler, storage is non-transactional unless you ask for it explicitly:
+
+```typescript
+import { withStorageTransaction, Dataset } from 'crawlee';
+
+await withStorageTransaction(async () => {
+ const dataset = await Dataset.open();
+ await dataset.pushData({ some: 'data' }); // buffered
+ // committed when the callback returns, rolled back if it throws
+});
+```
+
+For full control (e.g. deciding whether to commit only after inspecting the results), use `createStorageTransaction()` and drive `run()` / `commit()` / `rollback()` / `dispose()` yourself. This is exactly what `AdaptivePlaywrightCrawler` does with its per-attempt transactions.
diff --git a/docs/guides/running-in-web-server/web-server.mjs b/docs/guides/running-in-web-server/web-server.mjs
index 7c677db912bd..29e6b27367bd 100644
--- a/docs/guides/running-in-web-server/web-server.mjs
+++ b/docs/guides/running-in-web-server/web-server.mjs
@@ -1,6 +1,6 @@
import { randomUUID } from 'node:crypto';
import { CheerioCrawler, log } from 'crawlee';
-import { createServer } from 'http';
+import { createServer } from 'node:http';
// We will bind an HTTP response that we want to send to the Request.uniqueKey
const requestsToResponses = new Map();
diff --git a/docs/guides/scaling_crawlers.mdx b/docs/guides/scaling_crawlers.mdx
index e296cc90975f..1ce2caba727b 100644
--- a/docs/guides/scaling_crawlers.mdx
+++ b/docs/guides/scaling_crawlers.mdx
@@ -10,7 +10,10 @@ import CodeBlock from '@theme/CodeBlock';
import MinMaxConcurrencySource from '!!raw-loader!./scaling_crawlers_minMaxConcurrency.ts';
import MaxRequestsPerMinuteSource from '!!raw-loader!./scaling_crawlers_maxRequestsPerMinute.ts';
-import AutoscaledPoolOptionsSource from '!!raw-loader!./scaling_crawlers_autoscaledPoolOptions.ts';
+import ConcurrencySystemSource from '!!raw-loader!./scaling_crawlers_concurrencySystem.ts';
+import LoadSignalsOffSource from '!!raw-loader!./scaling_crawlers_loadSignalsOff.ts';
+import CustomLoadSignalSource from '!!raw-loader!./scaling_crawlers_customLoadSignal.ts';
+import WrapLoadSignalSource from '!!raw-loader!./scaling_crawlers_wrapLoadSignal.ts';
As we build our crawler, we might want to control how many requests we do to the website at a time. Crawlee provides several options to fine tune how many parallel requests should be made at any time, how many requests should be done per minute, and how should scaling work based on the available system resources.
@@ -45,7 +48,7 @@ It's recommended to leave it at the default value that is provided and letting t
## Advanced options
-While the options above should be enough for most users, if we wanted to get super deep into the configuration of the autoscaling pool (the internal utility in Crawlee that helps us allow crawlers to scale up and down), we can do so through the `autoscaledPoolOptions` object available on crawler options.
+While the options above should be enough for most users, if we wanted to get super deep into the configuration of autoscaling (the internal machinery in Crawlee that helps us allow crawlers to scale up and down), we can do so by injecting a pre-configured `ConcurrencySystem` through the `concurrencySystem` crawler option. All the fine-grained scaling configuration lives on that instance (see `ConcurrencySystemOptions`).
:::danger Complex options up ahead!
@@ -53,47 +56,51 @@ This section is super advanced and, unless you test the changes extensively and
:::
-With that warning aside, if we're feeling adventurous, this is how we would pass these options when using a crawler:
+With that warning aside, this is how we pass those options. One thing to watch: the `minConcurrency`/`maxConcurrency`/`maxRequestsPerMinute` shortcuts cannot be combined with an injected system — they configure the default one it replaces, so set those limits on the instance instead.
- {AutoscaledPoolOptionsSource}
+ {ConcurrencySystemSource}
-### `desiredConcurrency`
+:::note Requires Node.js 24
-This option specifies the amount of requests that should be running in parallel at the start of the crawler, assuming there are so many available. It defaults to the same value as `minConcurrency`.
+The `await using` syntax needs Node.js 24 or later. On Node.js 22 call `stop()` yourself in a `finally` block instead — it is what the disposal hook calls anyway.
-### `desiredConcurrencyRatio`
+:::
-The minimum ratio of concurrency to reach before more scaling up is allowed (a number between `0` and `1`). By default, it is set to `0.95`.
+:::tip Capping the combined concurrency of several crawlers
-We can think of this as the point where the autoscaling pool can attempt to scale up (or down), monitor if there's any changes, and correct them if necessary.
+Injecting the *same* `ConcurrencySystem` instance into several crawlers makes them share a single concurrency budget, capping their combined parallelism instead of letting each crawler scale independently.
-### `scaleUpStepRatio` and `scaleDownStepRatio`
+The budget is shared, not divided: slots go to whoever asks first, so a crawler that fills it early keeps refilling it as its own requests finish, and one that starts later may get a much smaller share of it. If a crawler needs a guaranteed slice, give it its own instance — or implement `IConcurrencySystem`, whose allocation methods are told which crawler is asking, and allocate however we see fit.
-These values define the fractional amount of desired concurrency to be added or subtracted as the autoscaling pool scales up or down. Both of these values default to `0.05`.
+:::
-Every time the autoscaled pool attempts to scale up or down, this value will be added or subtracted from the current concurrency, and, based on the [`desiredConcurrencyRatio`](#desiredconcurrencyratio) and [`maxConcurrency`](#minconcurrency-and-maxconcurrency), determines how many requests can run concurrently.
+### `desiredConcurrency`
-### `maybeRunIntervalSecs`
+This option specifies the amount of requests that should be running in parallel at the start of the crawler, assuming there are so many available. It defaults to the same value as `minConcurrency`.
-Indicates how often the autoscaling pool should check if more requests can be started and, if that's true, starts a new request if there are any available. This value is represented in seconds, and defaults to `0.5`.
+### `desiredConcurrencyRatio`
-:::info
+The minimum ratio of concurrency to reach before more scaling up is allowed (a number between `0` and `1`). By default, it is set to `0.9`.
-Changing this has no effect for requests that are fired immediately after the previous ones are finished. However, it will influence how fast new requests will be started after the autoscaled pool scales up.
+We can think of this as the point where the concurrency system can attempt to scale up (or down), monitor if there's any changes, and correct them if necessary.
-:::
+### `scaleUpStepRatio` and `scaleDownStepRatio`
+
+These values define the fractional amount of desired concurrency to be added or subtracted as the concurrency system scales up or down. Both of these values default to `0.05`.
+
+Every time the concurrency system attempts to scale up or down, this value will be added or subtracted from the current concurrency, and, based on the [`desiredConcurrencyRatio`](#desiredconcurrencyratio) and [`maxConcurrency`](#minconcurrency-and-maxconcurrency), determines how many requests can run concurrently.
### `loggingIntervalSecs`
-This option lets us control how often the autoscaled pool should log its current state (the current concurrency ratio, desired ratios, if the system is overloaded and so on).
+This option lets us control how often the concurrency system should log its current state (the current concurrency ratio, desired ratios, if the system is overloaded and so on).
We can disable logging altogether by setting this to `null`. By default, it is set to `60` seconds.
### `autoscaleIntervalSecs`
-This option lets us control how often the autoscaling pool should check if it can and should scale up or down. This value is represented in seconds, and defaults to `10`.
+This option lets us control how often the concurrency system should check if it can and should scale up or down. This value is represented in seconds, and defaults to `10`.
:::tip
@@ -118,3 +125,35 @@ This controls how many total requests can be made per minute. It counts the amou
This option can be set by specifying [`maxRequestsPerMinute`](#maxrequestsperminute) in your crawler options too, as it is a shortcut for visibility and ease of access.
:::
+
+## Load signals
+
+Whether the machine counts as overloaded is decided by *load signals*. Four are built in — memory, event loop, CPU and the storage backend's rate-limit errors — and each is configured by its own bag under `loadSignals`, carrying both its limits and the `overloadedRatio` at which it fires. If any signal reports overload, the system is overloaded and concurrency is held down.
+
+A signal we don't want watched at all can be switched **off** with `false`, which stops it being collected as well as evaluated (its entry in the reported status then simply reads as not overloaded):
+
+
+ {LoadSignalsOffSource}
+
+
+We can also watch resources of our own by implementing `LoadSignal` — navigation timeouts or proxy health, say — and passing them in `loadSignals.custom`. The `SnapshotStore` helper does the time-windowed bookkeeping for us:
+
+
+ {CustomLoadSignalSource}
+
+
+Each built-in is a public class too — `MemoryLoadSignal`, `EventLoopLoadSignal`, `CpuLoadSignal`, `StorageBackendLoadSignal` — taking exactly the bag its `loadSignals` key accepts. Constructing one directly is how we *wrap* a built-in rather than reimplement it (to hold its overload verdict for a while after the resource recovers, say): switch the original off with `cpu: false`, and pass a signal that delegates to the instance we built in `custom`.
+
+
+ {WrapLoadSignalSource}
+
+
+:::info Signal names are the keys of the reported status
+
+A signal's `name` is the key its verdict appears under in the reported status, so two signals cannot share one — a duplicate throws at construction time. That is why taking over a built-in name (`memInfo`, `eventLoopInfo`, `cpuInfo`, `storageBackendInfo`) requires switching that built-in off.
+
+:::
+
+### `snapshotHistorySecs` and `currentHistorySecs`
+
+Signals are not read as a single instantaneous measurement but averaged over a window, and there are two: `currentHistorySecs` (default `5`) is the short window that gates whether one more request may start, while `snapshotHistorySecs` (default `30`) is the longer window autoscaling decisions are based on. Dispatch therefore reacts to spikes quickly while scaling stays stable. Both apply to every signal alike, built-in or custom, and signals size their snapshot retention to the wider of the two — so raising `snapshotHistorySecs` is what costs memory.
diff --git a/docs/guides/scaling_crawlers_autoscaledPoolOptions.ts b/docs/guides/scaling_crawlers_autoscaledPoolOptions.ts
deleted file mode 100644
index 5d8f53c4dcc4..000000000000
--- a/docs/guides/scaling_crawlers_autoscaledPoolOptions.ts
+++ /dev/null
@@ -1,8 +0,0 @@
-import { CheerioCrawler } from 'crawlee';
-
-const crawler = new CheerioCrawler({
- // Pass in advanced options by providing them in the autoscaledPoolOptions
- autoscaledPoolOptions: {
- // ...
- },
-});
diff --git a/docs/guides/scaling_crawlers_concurrencySystem.ts b/docs/guides/scaling_crawlers_concurrencySystem.ts
new file mode 100644
index 000000000000..ba3184420db2
--- /dev/null
+++ b/docs/guides/scaling_crawlers_concurrencySystem.ts
@@ -0,0 +1,16 @@
+import { CheerioCrawler, ConcurrencySystem } from 'crawlee';
+
+// Advanced scaling options live on a pre-configured ConcurrencySystem.
+// An injected system's lifecycle is owned by us, not the crawler - `await using` stops it for us
+// once we are done with it.
+await using concurrencySystem = new ConcurrencySystem({
+ // ...
+});
+
+const crawler = new CheerioCrawler({
+ concurrencySystem,
+ // ...
+});
+
+await concurrencySystem.start();
+await crawler.run(['https://crawlee.dev']);
diff --git a/docs/guides/scaling_crawlers_customLoadSignal.ts b/docs/guides/scaling_crawlers_customLoadSignal.ts
new file mode 100644
index 000000000000..ee5d093f031e
--- /dev/null
+++ b/docs/guides/scaling_crawlers_customLoadSignal.ts
@@ -0,0 +1,33 @@
+import type { LoadSignal } from 'crawlee';
+import { ConcurrencySystem, SnapshotStore } from 'crawlee';
+
+// The only part that is ours: anything we can poll and reduce to "is this resource in trouble?"
+async function areProxiesStruggling(): Promise {
+ const response = await fetch('https://proxy-monitor.example.com/health');
+
+ return !response.ok;
+}
+
+const store = new SnapshotStore();
+let interval: NodeJS.Timeout;
+
+const proxyHealth: LoadSignal = {
+ name: 'proxyHealth',
+ overloadedRatio: 0.3,
+ async start({ maxSampleWindowMillis }) {
+ // Retain exactly the window we will be sampled over, and drop anything measured before a restart.
+ store.useSampleWindow(maxSampleWindowMillis);
+ store.clear();
+
+ interval = setInterval(async () => {
+ const createdAt = new Date();
+ store.push({ createdAt, isOverloaded: await areProxiesStruggling() }, createdAt);
+ }, 1_000);
+ },
+ async stop() {
+ clearInterval(interval);
+ },
+ getSample: (sampleDurationMillis) => store.getSample(sampleDurationMillis),
+};
+
+const concurrencySystem = new ConcurrencySystem({ loadSignals: { custom: [proxyHealth] } });
diff --git a/docs/guides/scaling_crawlers_loadSignalsOff.ts b/docs/guides/scaling_crawlers_loadSignalsOff.ts
new file mode 100644
index 000000000000..aa982a0943da
--- /dev/null
+++ b/docs/guides/scaling_crawlers_loadSignalsOff.ts
@@ -0,0 +1,9 @@
+import { ConcurrencySystem } from 'crawlee';
+
+const concurrencySystem = new ConcurrencySystem({
+ loadSignals: {
+ // Our storage backend reports no rate-limit statistics, so stop polling it every second.
+ storageBackend: false,
+ eventLoop: { maxBlockedMillis: 100 },
+ },
+});
diff --git a/docs/guides/scaling_crawlers_wrapLoadSignal.ts b/docs/guides/scaling_crawlers_wrapLoadSignal.ts
new file mode 100644
index 000000000000..9d5b307c132d
--- /dev/null
+++ b/docs/guides/scaling_crawlers_wrapLoadSignal.ts
@@ -0,0 +1,35 @@
+import type { LoadSignal, LoadSignalStartContext, LoadSnapshot } from 'crawlee';
+import { ConcurrencySystem, CpuLoadSignal } from 'crawlee';
+
+const cooldownMillis = 10_000;
+
+// The built-in still does all the measuring; we only reinterpret what it measured.
+const cpu = new CpuLoadSignal();
+
+const stickyCpu: LoadSignal = {
+ // Taking the built-in's name over is allowed only because we switch the built-in off below.
+ name: cpu.name,
+ overloadedRatio: cpu.overloadedRatio,
+ start: (context: LoadSignalStartContext) => cpu.start(context),
+ stop: () => cpu.stop(),
+ getSample(sampleDurationMillis?: number): LoadSnapshot[] {
+ // Keep reporting overload for a while after the CPU recovers, so that scaling up does not immediately
+ // overload it again.
+ let overloadedUntil = 0;
+
+ return cpu.getSample(sampleDurationMillis).map((snapshot) => {
+ if (snapshot.isOverloaded) {
+ overloadedUntil = +snapshot.createdAt + cooldownMillis;
+ }
+
+ return { ...snapshot, isOverloaded: +snapshot.createdAt < overloadedUntil };
+ });
+ },
+};
+
+const concurrencySystem = new ConcurrencySystem({
+ loadSignals: {
+ cpu: false,
+ custom: [stickyCpu],
+ },
+});
diff --git a/docs/guides/session_management.mdx b/docs/guides/session_management.mdx
index ad0eca80e96e..f019f13d0617 100644
--- a/docs/guides/session_management.mdx
+++ b/docs/guides/session_management.mdx
@@ -18,18 +18,15 @@ import PlaywrightSource from '!!raw-loader!./session_management_playwright.ts';
import PuppeteerSource from '!!raw-loader!./session_management_puppeteer.ts';
import StandaloneSource from '!!raw-loader!./session_management_standalone.ts';
-`SessionPool` is a class that allows us to handle the rotation of proxy IP addresses along with cookies and other custom settings in Crawlee.
+`SessionPool` manages the rotation of proxy IP addresses, cookies, and browser fingerprints in Crawlee. A single `Session` bundles all the identifying state of one "virtual user" — its cookie jar, its proxy (and therefore its IP), and a fingerprint hint — so that everything that makes a series of requests look like it comes from one person rotates together. When a session gets blocked, the whole bundle is thrown away at once and a fresh identity takes over, rather than reusing a burnt IP with new cookies (or vice versa).
-The main benefit of using Session pool is that we can filter out blocked or non-working proxies,
-so our actor does not retry requests over known blocked/non-working proxies.
-Another benefit of using SessionPool is that we can store information tied tightly to an IP address,
-such as cookies, auth tokens, and particular headers. Having our cookies and other identifiers used only with a specific IP will reduce the chance of being blocked.
-The last but not least benefit is the even rotation of IP addresses - SessionPool picks the session randomly,
-which should prevent burning out a small pool of available IPs.
+The main benefits of the session pool are that it filters out blocked or non-working proxies so the crawler does not keep retrying over them, it keeps identity-bound state (cookies, auth tokens, headers) tied to the IP that obtained it, and it spreads requests across IPs to avoid burning a small pool. The selection strategy is configurable — see [Choosing a rotation strategy](#choosing-a-rotation-strategy) below.
-Check out the [avoid blocking guide](./avoid-blocking) for more information about blocking.
+All crawler instances now require a `SessionPool`. In most cases you do not create one yourself: you just read the `session` from the request handler and let the crawler mark it good or bad for you. You only construct a `SessionPool` explicitly when you want to override its defaults or share one instance across several crawlers.
-Now let's take a look at the examples of how to use Session pool:
+Check out the [avoid blocking guide](./avoid-blocking) for the bigger picture on why blocking happens and how fingerprints fit in.
+
+Now let's take a look at the examples of how to use the session pool:
- with `BasicCrawler`;
- with `HttpCrawler`;
- with `CheerioCrawler`;
@@ -76,6 +73,253 @@ Now let's take a look at the examples of how to use Session pool:
-These are the basics of configuring SessionPool.
-Please, bear in mind that a Session pool needs time to find working IPs and build up the pool,
-so we will probably see a lot of errors until it becomes stabilized.
+These are the basics of configuring the session pool. The rest of this guide covers how to control which session is used, what state it carries, and when it is thrown away.
+
+## How a session is retired
+
+A session stays in the pool and keeps being handed out as long as `isUsable()` returns `true`. It stops being usable — and is dropped from rotation — as soon as any of the following happens:
+
+- its **error score** reaches `maxErrorScore` (default `3`),
+- its **usage count** reaches `maxUsageCount` (default `50`),
+- it is older than `maxAgeSecs` (default `3000` seconds), or
+- it has been explicitly **retired**.
+
+You influence this with three methods on the session. `markGood()` records a successful use — it increments the usage count and heals the error score a little (by `errorScoreDecrement`, default `0.5`). `markBad()` records a failure that *might* be the session's fault and *might* just be bad luck — it raises the error score by one, so a session needs to fail repeatedly before it is dropped. `retire()` drops the session immediately and permanently; this is what you call when you are certain the identity itself is burnt (for example, a `403` response).
+
+The distinction between `markBad()` and `retire()` matters. Use `markBad()` for transient, external problems such as a timeout or a `5XX` response — the IP is probably fine and a couple of retries should not throw it away. Use `retire()` for problems that prove the session is blocked, where reusing it is pointless. Retirement is terminal: once a session is retired, a later `markGood()` will not bring it back.
+
+When using a crawler you rarely call `markGood()` yourself — the crawler calls it automatically after a successful request handler run. You only need to reach for `markBad()` / `retire()` (or let blocked status codes do it for you, see [below](#letting-blocked-responses-retire-sessions)) when you detect a problem the crawler cannot see, such as a "you are blocked" message inside an otherwise `200` response.
+
+## Managing cookies
+
+Every session owns a [`tough-cookie`](https://github.com/salesforce/tough-cookie) cookie jar, reachable as `session.cookieJar`. Cookies arriving in `Set-Cookie` response headers are stored in it automatically — this is controlled by the `saveResponseCookies` crawler option (default `true`) — so they are replayed on every later request that reuses the same session. Set `saveResponseCookies: false` to keep response cookies out of the session jar.
+
+You can also seed or read cookies yourself. `session.setCookie('name=value', url)` adds a single cookie, `session.getCookieString(url)` returns the `Cookie` header value the session would send for that URL, and `session.cookieJar` gives you the full jar for anything more involved.
+
+```js
+const crawler = new CheerioCrawler({
+ requestHandler: async ({ session, request }) => {
+ await session.setCookie('consent=yes', request.url);
+ },
+});
+```
+
+### Cookie precedence and overrides
+
+When an HTTP-based crawler (or a direct `sendRequest` call) builds the outgoing `Cookie` header, it starts from a **base jar** and then overlays any cookies set on the request:
+
+- The base jar is the explicit `cookieJar` passed to `sendRequest` if you provide one, otherwise the session's own cookie jar.
+- A `Cookie` header on the request (`request.headers.Cookie`) is merged on top of that base. A cookie set this way wins over a base-jar cookie of the same name, but it is *not* persisted back into the session.
+
+So a `Cookie` request header always beats the stored cookie of the same name regardless of which jar is the base, while passing an explicit `cookieJar` swaps out the whole base for that single call. To override a single cookie for one request, set it on the request header:
+
+```js
+import { HttpCrawler } from 'crawlee';
+import { CookieJar } from 'tough-cookie';
+
+const crawler = new HttpCrawler({
+ preNavigationHooks: [
+ async ({ request }) => {
+ // wins over any same-named cookie in the session jar, for this request only
+ request.headers = { ...request.headers, Cookie: 'token=override' };
+ },
+ ],
+ requestHandler: async ({ sendRequest }) => {
+ // ...or to fully replace the jar for a single call:
+ const jar = new CookieJar();
+ await jar.setCookie('token=override', 'https://example.com');
+ await sendRequest({ url: 'https://example.com' }, { cookieJar: jar });
+ },
+});
+```
+
+A `Cookie` header you set on a request is always honored — it is never silently overwritten by the session jar.
+
+## Choosing a rotation strategy
+
+The `sessionReuseStrategy` option decides *which* session `getSession()` hands out, and it is the main lever for matching the pool's behavior to a target site. Three strategies are available, each suited to a different use case.
+
+**Maximise IP and fingerprint diversity** — use `'random'` (the default). The pool creates a brand-new session for every request until it reaches `maxPoolSize`, then picks a usable session at random. This spreads traffic as widely as possible across IPs and fingerprints and is the right default for most large crawls.
+
+**Distribute load evenly across sessions** — use `'round-robin'`. Like `random`, the pool fills up to `maxPoolSize` first, but then cycles through sessions in order instead of picking randomly. This is useful when you want every session to do roughly the same amount of work — for example, combined with `maxUsageCount` so all sessions reach their limit and rotate out at about the same time.
+
+**Use a single IP until it breaks** — use `'use-until-failure'`. The pool returns the *same* session on every call and only moves to the next one once the current session is retired. This is the strategy for sites that reward consistency: where switching IP mid-flow looks suspicious, where you have logged in and want to stay logged in, or where you simply want to squeeze a working proxy for as long as it lasts before paying for another.
+
+
+
+
+```js
+import { SessionPool } from 'crawlee';
+
+const sessionPool = new SessionPool({
+ sessionReuseStrategy: 'random',
+});
+```
+
+
+
+
+```js
+import { SessionPool } from 'crawlee';
+
+const sessionPool = new SessionPool({
+ sessionReuseStrategy: 'round-robin',
+ // make every session retire after the same amount of work
+ sessionOptions: { maxUsageCount: 100 },
+});
+```
+
+
+
+
+```js
+import { SessionPool } from 'crawlee';
+
+const sessionPool = new SessionPool({
+ sessionReuseStrategy: 'use-until-failure',
+});
+```
+
+
+
+
+Whichever strategy you pick, you can cap how hard each session works through `sessionOptions`. Set `maxUsageCount` when you know a site starts blocking after roughly _N_ requests from one IP, `maxAgeSecs` when sessions should be cycled on a time basis, and `maxErrorScore` to control how forgiving the pool is about intermittent failures before dropping a session.
+
+```js
+const sessionPool = new SessionPool({
+ maxPoolSize: 25,
+ sessionOptions: {
+ maxAgeSecs: 600,
+ maxUsageCount: 150, // e.g. when you know the site blocks after ~150 requests
+ },
+});
+```
+
+## Letting blocked responses retire sessions
+
+You do not have to inspect every response by hand. Crawlers treat a configurable set of HTTP status codes as proof that a session is blocked and retire it automatically, retrying the request with a fresh session. This is controlled by the `blockedStatusCodes` crawler option (default `[401, 403, 429]`).
+
+```js
+import { CheerioCrawler } from 'crawlee';
+
+const crawler = new CheerioCrawler({
+ // a 403 or 429 will retire the current session and retry on a new one
+ blockedStatusCodes: [403, 429],
+ requestHandler: async ({ session, request }) => {
+ // session is already a working, non-blocked one
+ },
+});
+```
+
+For sites that respond with a `200` page that is actually a bot wall (Cloudflare challenges, Google's rate-limit page), set `retryOnBlocked: true` to have the crawler detect those by content and retry as well. For deeper anti-blocking measures see the [avoid blocking guide](./avoid-blocking).
+
+:::tip A 429 is a rate limit, not a block
+
+Retiring a session on HTTP 429 burns proxies without slowing anything down — the site is asking you to wait, not telling you the session is unwelcome. Wrap your request manager in a `ThrottlingRequestManager` to back off per domain instead of rotating; see [per-domain throttling](./request-loaders#per-domain-throttling). Sessions are left untouched for the domains it covers.
+
+Those domains are handled as rate limits before `blockedStatusCodes` is consulted, so leave 429 in the list — removing it only changes what happens for domains the manager does not cover.
+
+:::
+
+## Sharing a session pool between crawlers
+
+A `SessionPool` instance can be shared across multiple crawlers by passing the same object to each crawler's `sessionPool` option. This is useful in multi-stage scrapers — for example a fast `CheerioCrawler` that discovers links and a `PlaywrightCrawler` that renders detail pages — where you want both stages to reuse the same proven, non-blocked identities and their cookies instead of each warming up its own pool from scratch.
+
+```js
+import { CheerioCrawler, PlaywrightCrawler, SessionPool } from 'crawlee';
+
+const sessionPool = new SessionPool({ maxPoolSize: 100 });
+
+const listingCrawler = new CheerioCrawler({ sessionPool, requestHandler: async () => { /* ... */ } });
+const detailCrawler = new PlaywrightCrawler({ sessionPool, requestHandler: async () => { /* ... */ } });
+```
+
+A pool you construct yourself is owned by you, not the crawler — the crawler will never tear it down or reset it between runs. Dispose of it with `await using` when you are done, which persists its final state and stops it listening for persistence events:
+
+```js
+import { CheerioCrawler, SessionPool } from 'crawlee';
+
+await using sessionPool = new SessionPool({ maxPoolSize: 100 });
+
+const crawler = new CheerioCrawler({ sessionPool, requestHandler: async () => { /* ... */ } });
+await crawler.run(['https://crawlee.dev']);
+// sessionPool is torn down here, even if the crawl throws
+```
+
+:::note Requires Node.js 24
+
+The `await using` syntax needs Node.js 24 or later. On Node.js 22 call `teardown()` yourself instead — it is what the disposal hook calls anyway.
+
+:::
+
+## Custom session pools
+
+A crawler accepts any object implementing the `ISessionPool` interface as its `sessionPool` option, not just the built-in `SessionPool`. The contract is intentionally tiny — a single `getSession()` / `getSession(id)` method that hands out an `ISession` for a request. This lets you plug in a remote, shared, or database-backed session strategy without subclassing `SessionPool` or copying its internals.
+
+```ts
+import { BasicCrawler, Session, type ISessionPool } from 'crawlee';
+
+class MySessionPool implements ISessionPool {
+ private readonly sessions = new Map();
+
+ async getSession(sessionId?: string): Promise {
+ if (sessionId) {
+ const existing = this.sessions.get(sessionId);
+ return existing?.isUsable() ? existing : undefined;
+ }
+
+ const usable = [...this.sessions.values()].find((s) => s.isUsable());
+ if (usable) return usable;
+
+ const fresh = new Session();
+ this.sessions.set(fresh.id, fresh);
+ return fresh;
+ }
+}
+
+const crawler = new BasicCrawler({
+ sessionPool: new MySessionPool(),
+ requestHandler: async ({ session }) => {
+ // session is a Session instance, use it as usual
+ },
+});
+```
+
+The returned objects just need to implement `ISession` — the crawler only calls `markGood()`, `markBad()`, `retire()`, and reads `cookieJar`, `proxyInfo`, and `fingerprint`, all of which are part of that interface.
+
+## Pinning a request to a specific session
+
+By default the pool decides which session a request gets. Setting `request.sessionId` overrides that and forces the request — and all of its retries — onto the session with that id. You can create a custom named session with `addSession()`, giving each its own proxy, cookies, or fingerprint. Because a session bundles a proxy, this is how you bind specific requests to specific proxies.
+
+One important consequence: if a named session is retired — whether through accumulated `markBad()` calls, hitting `maxUsageCount`, or an explicit `retire()` — any subsequent `getSession(id)` call for that id returns `undefined`.
+The crawler treats that as a `MissingSessionError`, counts it as a regular request error, and retries the request with the same `sessionId`. If the session stays retired, retries keep failing and the request eventually exhausts `maxRequestRetries`.
+When a named session can be retired, handle this in your `errorHandler`: either recreate the session via `addSession()` with the same id, or clear `request.sessionId` to let the pool assign a fresh one.
+
+A common usage pattern is escalating between proxy "tiers": add a cheap session and a premium one, start requests on the cheap session, and reassign `request.sessionId` to the premium one in an `errorHandler` so the retry goes out over the better proxy.
+
+```ts
+import { BasicCrawler, SessionPool } from 'crawlee';
+
+const proxyInfoFromUrl = (proxyUrl: string) => {
+ const { username, password, hostname, port } = new URL(proxyUrl);
+ return { url: proxyUrl, username, password, hostname, port };
+};
+
+const sessionPool = new SessionPool();
+await sessionPool.addSession({ id: 'cheap', proxyInfo: proxyInfoFromUrl('http://cheap-proxy.com') });
+await sessionPool.addSession({ id: 'premium', proxyInfo: proxyInfoFromUrl('http://expensive-proxy.com') });
+
+const crawler = new BasicCrawler({
+ sessionPool,
+ retryOnBlocked: true,
+ requestHandler: async ({ sendRequest, request }) => {
+ await sendRequest({ url: request.url });
+ },
+ errorHandler: async ({ request }) => {
+ request.sessionId = 'premium'; // escalate the retry to the premium proxy
+ },
+});
+
+await crawler.run([{ url: 'https://example.com', sessionId: 'cheap' }]);
+```
+
diff --git a/docs/guides/session_management_basic.ts b/docs/guides/session_management_basic.ts
index c7b7ec37c361..4a9f46748b4d 100644
--- a/docs/guides/session_management_basic.ts
+++ b/docs/guides/session_management_basic.ts
@@ -1,33 +1,29 @@
-import { BasicCrawler, ProxyConfiguration } from 'crawlee';
-import { gotScraping } from 'got-scraping';
+import { BasicCrawler, ProxyConfiguration, SessionPool } from 'crawlee';
+import { Impit } from 'impit';
+import { Cookie } from 'tough-cookie';
const proxyConfiguration = new ProxyConfiguration({
/* opts */
});
const crawler = new BasicCrawler({
- // Activates the Session pool (default is true).
- useSessionPool: true,
// Overrides default Session pool configuration.
- sessionPoolOptions: { maxPoolSize: 100 },
+ sessionPool: new SessionPool({ maxPoolSize: 100 }),
async requestHandler({ request, session }) {
const { url } = request;
- const requestOptions = {
- url,
- // We use session id in order to have the same proxyUrl
- // for all the requests using the same session.
- proxyUrl: await proxyConfiguration.newUrl(session?.id),
- throwHttpErrors: false,
+ const client = new Impit({
+ proxyUrl: await proxyConfiguration.newUrl(),
+ ignoreTlsErrors: true,
headers: {
// If you want to use the cookieJar.
// This way you get the Cookie headers string from session.
- Cookie: session?.getCookieString(url),
+ Cookie: (await session?.cookieJar.getCookieString(url)) ?? '',
},
- };
+ });
let response;
try {
- response = await gotScraping(requestOptions);
+ response = await client.fetch(url);
} catch (e) {
if (e === 'SomeNetworkError') {
// If a network error happens, such as timeout, socket hangup, etc.
@@ -38,10 +34,7 @@ const crawler = new BasicCrawler({
throw e;
}
- // Automatically retires the session based on response HTTP status code.
- session?.retireOnBlockedStatusCodes(response.statusCode);
-
- if (response.body.includes('You are blocked!')) {
+ if ((await response.text()).includes('You are blocked!')) {
// You are sure it is blocked.
// This will throw away the session.
session?.retire();
@@ -51,6 +44,17 @@ const crawler = new BasicCrawler({
// No need to call session.markGood -> BasicCrawler calls it for you.
// If you want to use the CookieJar in session you need.
- session?.setCookiesFromResponse(response);
+ if (response.headers.has('set-cookie')) {
+ const newCookies = response.headers
+ .get('set-cookie')
+ ?.split(';')
+ .map((x) => Cookie.parse(x));
+
+ for (const cookie of newCookies ?? []) {
+ if (cookie) {
+ await session?.cookieJar?.setCookie(cookie, url);
+ }
+ }
+ }
},
});
diff --git a/docs/guides/session_management_cheerio.ts b/docs/guides/session_management_cheerio.ts
index 7f8b2f90a09a..bda80505992d 100644
--- a/docs/guides/session_management_cheerio.ts
+++ b/docs/guides/session_management_cheerio.ts
@@ -1,4 +1,4 @@
-import { CheerioCrawler, ProxyConfiguration } from 'crawlee';
+import { CheerioCrawler, ProxyConfiguration, SessionPool } from 'crawlee';
const proxyConfiguration = new ProxyConfiguration({
/* opts */
@@ -7,13 +7,11 @@ const proxyConfiguration = new ProxyConfiguration({
const crawler = new CheerioCrawler({
// To use the proxy IP session rotation logic, you must turn the proxy usage on.
proxyConfiguration,
- // Activates the Session pool (default is true).
- useSessionPool: true,
// Overrides default Session pool configuration.
- sessionPoolOptions: { maxPoolSize: 100 },
+ sessionPool: new SessionPool({ maxPoolSize: 100 }),
// Set to true if you want the crawler to save cookies per session,
// and set the cookie header to request automatically (default is true).
- persistCookiesPerSession: true,
+ saveResponseCookies: true,
async requestHandler({ session, $ }) {
const title = $('title').text();
diff --git a/docs/guides/session_management_http.ts b/docs/guides/session_management_http.ts
index 9c684bcb0566..bb55dc3e69da 100644
--- a/docs/guides/session_management_http.ts
+++ b/docs/guides/session_management_http.ts
@@ -1,4 +1,4 @@
-import { HttpCrawler, ProxyConfiguration } from 'crawlee';
+import { HttpCrawler, ProxyConfiguration, SessionPool } from 'crawlee';
const proxyConfiguration = new ProxyConfiguration({
/* opts */
@@ -7,15 +7,13 @@ const proxyConfiguration = new ProxyConfiguration({
const crawler = new HttpCrawler({
// To use the proxy IP session rotation logic, you must turn the proxy usage on.
proxyConfiguration,
- // Activates the Session pool (default is true).
- useSessionPool: true,
// Overrides default Session pool configuration.
- sessionPoolOptions: { maxPoolSize: 100 },
+ sessionPool: new SessionPool({ maxPoolSize: 100 }),
// Set to true if you want the crawler to save cookies per session,
// and set the cookie header to request automatically (default is true).
- persistCookiesPerSession: true,
+ saveResponseCookies: true,
async requestHandler({ session, body }) {
- const title = (body as string).match(/(.*?)<\/title>/)?.[1];
+ const title = /(.*?)<\/title>/.exec(body as string)?.[1];
if (title === 'Blocked') {
session?.retire();
diff --git a/docs/guides/session_management_jsdom.ts b/docs/guides/session_management_jsdom.ts
index ef55b6632640..ee8e7cfffda3 100644
--- a/docs/guides/session_management_jsdom.ts
+++ b/docs/guides/session_management_jsdom.ts
@@ -1,4 +1,4 @@
-import { JSDOMCrawler, ProxyConfiguration } from 'crawlee';
+import { JSDOMCrawler, ProxyConfiguration, SessionPool } from 'crawlee';
const proxyConfiguration = new ProxyConfiguration({
/* opts */
@@ -7,13 +7,11 @@ const proxyConfiguration = new ProxyConfiguration({
const crawler = new JSDOMCrawler({
// To use the proxy IP session rotation logic, you must turn the proxy usage on.
proxyConfiguration,
- // Activates the Session pool (default is true).
- useSessionPool: true,
// Overrides default Session pool configuration.
- sessionPoolOptions: { maxPoolSize: 100 },
+ sessionPool: new SessionPool({ maxPoolSize: 100 }),
// Set to true if you want the crawler to save cookies per session,
// and set the cookie header to request automatically (default is true).
- persistCookiesPerSession: true,
+ saveResponseCookies: true,
async requestHandler({ session, window }) {
const title = window.document.title;
diff --git a/docs/guides/session_management_playwright.ts b/docs/guides/session_management_playwright.ts
index f4f2f7c80f6f..01749fccddbb 100644
--- a/docs/guides/session_management_playwright.ts
+++ b/docs/guides/session_management_playwright.ts
@@ -1,4 +1,4 @@
-import { PlaywrightCrawler, ProxyConfiguration } from 'crawlee';
+import { PlaywrightCrawler, ProxyConfiguration, SessionPool } from 'crawlee';
const proxyConfiguration = new ProxyConfiguration({
/* opts */
@@ -7,13 +7,11 @@ const proxyConfiguration = new ProxyConfiguration({
const crawler = new PlaywrightCrawler({
// To use the proxy IP session rotation logic, you must turn the proxy usage on.
proxyConfiguration,
- // Activates the Session pool (default is true).
- useSessionPool: true,
// Overrides default Session pool configuration
- sessionPoolOptions: { maxPoolSize: 100 },
+ sessionPool: new SessionPool({ maxPoolSize: 100 }),
// Set to true if you want the crawler to save cookies per session,
// and set the cookies to page before navigation automatically (default is true).
- persistCookiesPerSession: true,
+ saveResponseCookies: true,
async requestHandler({ page, session }) {
const title = await page.title();
diff --git a/docs/guides/session_management_puppeteer.ts b/docs/guides/session_management_puppeteer.ts
index 63b342146397..76ad3fcc7ee5 100644
--- a/docs/guides/session_management_puppeteer.ts
+++ b/docs/guides/session_management_puppeteer.ts
@@ -1,4 +1,4 @@
-import { PuppeteerCrawler, ProxyConfiguration } from 'crawlee';
+import { PuppeteerCrawler, ProxyConfiguration, SessionPool } from 'crawlee';
const proxyConfiguration = new ProxyConfiguration({
/* opts */
@@ -7,13 +7,11 @@ const proxyConfiguration = new ProxyConfiguration({
const crawler = new PuppeteerCrawler({
// To use the proxy IP session rotation logic, you must turn the proxy usage on.
proxyConfiguration,
- // Activates the Session pool (default is true).
- useSessionPool: true,
// Overrides default Session pool configuration
- sessionPoolOptions: { maxPoolSize: 100 },
+ sessionPool: new SessionPool({ maxPoolSize: 100 }),
// Set to true if you want the crawler to save cookies per session,
// and set the cookies to page before navigation automatically (default is true).
- persistCookiesPerSession: true,
+ saveResponseCookies: true,
async requestHandler({ page, session }) {
const title = await page.title();
diff --git a/docs/guides/session_management_standalone.ts b/docs/guides/session_management_standalone.ts
index c6fa33d82170..8ac133d9501c 100644
--- a/docs/guides/session_management_standalone.ts
+++ b/docs/guides/session_management_standalone.ts
@@ -5,17 +5,16 @@ const sessionPoolOptions = {
maxPoolSize: 100,
};
-// Open Session Pool.
-const sessionPool = await SessionPool.open(sessionPoolOptions);
+const sessionPool = new SessionPool(sessionPoolOptions);
// Get session.
const session = await sessionPool.getSession();
// Increase the errorScore.
-session.markBad();
+session?.markBad();
// Throw away the session.
-session.retire();
+session?.retire();
// Lower the errorScore and mark the session good.
-session.markGood();
+session?.markGood();
diff --git a/docs/introduction/03-adding-urls.mdx b/docs/introduction/03-adding-urls.mdx
index 387b86fb9450..c987dc732cfb 100644
--- a/docs/introduction/03-adding-urls.mdx
+++ b/docs/introduction/03-adding-urls.mdx
@@ -43,7 +43,7 @@ The process is simple:
4. Visit the newly enqueued links.
5. Repeat the process.
-In the following paragraphs you will learn about the `enqueueLinks` function which simplifies crawling to a single function call. For comparison and learning purposes we will show an equivalent solution written without `enqueueLinks` in the second code tab.
+In the following paragraphs you will learn about the `enqueueLinks` function which simplifies crawling to a single function call. For comparison and learning purposes we will show an equivalent solution written without `enqueueLinks` in the second code tab.
:::tip `enqueueLinks` context awareness
@@ -130,7 +130,7 @@ await enqueueLinks({
### Filter URLs with patterns
-For even more control, you can use `globs`, `regexps` and `pseudoUrls` to filter the URLs. Each of those arguments is always an `Array`, but the contents can take on many forms. See the reference for more information about them as well as other options.
+For even more control, you can use `include` and `exclude` to filter the URLs. Each accepts an `Array` of glob pattern strings, `{ glob: string }` objects, `RegExp` instances, or `{ regexp: RegExp }` objects. See the reference for more information about them as well as other options.
:::caution Defaults override
@@ -140,17 +140,17 @@ If you provide one of those options, the default `same-hostname` strategy will *
```ts
await enqueueLinks({
- globs: ['http?(s)://apify.com/*/*'],
+ include: ['http?(s)://apify.com/*/*'],
});
```
### Transform requests
-To have absolute control, we have the `transformRequestFunction`. Just before a new `Request` is constructed and enqueued to the `RequestQueue`, this function can be used to skip it or modify its contents such as `userData`, `payload` or, most importantly, `uniqueKey`. This is useful when you need to enqueue multiple requests to the queue, and these requests share the same URL, but differ in methods or payloads. Another use case is to dynamically update or create the `userData`.
+To have absolute control, we have the `transformRequestFunction`. After request options are filtered by `include`/`exclude` patterns, this function can be used to skip them or modify their contents such as `userData`, `payload` or, most importantly, `uniqueKey`. This is useful when you need to enqueue multiple requests to the queue, and these requests share the same URL, but differ in methods or payloads. Another use case is to dynamically update or create the `userData`.
```ts
await enqueueLinks({
- globs: ['http?(s)://apify.com/*/*'],
+ include: ['http?(s)://apify.com/*/*'],
transformRequestFunction(req) {
// ignore all links ending with `.pdf`
if (req.url.endsWith('.pdf')) return false;
diff --git a/docs/introduction/08-refactoring.mdx b/docs/introduction/08-refactoring.mdx
index 450048013258..235fe88f0597 100644
--- a/docs/introduction/08-refactoring.mdx
+++ b/docs/introduction/08-refactoring.mdx
@@ -154,6 +154,30 @@ Initially, using a simple `if/else` statement for selecting different logic base
It's good practice in any programming language to split your logic into bite-sized chunks that are easy to read and reason about. Scrolling through a thousand line long `requestHandler()` where everything interacts with everything and variables can be used everywhere is not a beautiful thing to do and a pain to debug. That's why we prefer the separation of routes into their own files.
+### Giving a route its own timeout
+
+The crawler-wide `requestHandlerTimeoutSecs` applies to every request alike. Once the routes are separated, one of them can be given its own timeout — longer or shorter — without changing the limit for the rest, by passing a per-route timeout as the last argument of `addHandler`. This is handy when one page type does noticeably more work than the others, like a `CATEGORY` page that paginates through a long list:
+
+```js
+router.addHandler('CATEGORY', async ({ page, enqueueLinks }) => {
+ // ...
+}, { requestHandlerTimeoutSecs: 120 });
+
+router.addHandler('DETAIL', async ({ page }) => {
+ // ...
+}); // keeps the crawler's default
+```
+
+If how much longer a route needs only becomes clear once it is already running, call `extendTimeout()` from inside the handler to buy more time on the spot:
+
+```js
+router.addHandler('CATEGORY', async ({ page, enqueueLinks, extendTimeout }) => {
+ const pageCount = await countPages(page);
+ extendTimeout(pageCount * 10); // ask for 10 more seconds per page
+ // ...
+});
+```
+
## Next steps
In the next and final step, you'll see how to deploy your Crawlee project to the cloud. If you used the CLI to bootstrap your project, you already have a **Dockerfile** ready, and the next section will show you how to deploy it to the [Apify Platform](../deployment/apify-platform) with ease.
diff --git a/docs/package.json b/docs/package.json
index 51ae4027ed12..5e9c57042805 100644
--- a/docs/package.json
+++ b/docs/package.json
@@ -2,7 +2,6 @@
"name": "crawlee-docs",
"description": "Documentation and examples for Crawlee. This package is not published to npm, only used locally for TS build checks.",
"type": "module",
- "packageManager": "yarn@4.10.3",
"scripts": {
"typecheck": "tsc --noEmit"
},
@@ -10,8 +9,19 @@
"typescript": "^6.0.0"
},
"dependencies": {
+ "@crawlee/browser-pool": "workspace:*",
+ "@crawlee/core": "workspace:*",
+ "@crawlee/got-scraping-client": "workspace:*",
+ "@crawlee/http-client": "workspace:*",
+ "@crawlee/impit-client": "workspace:*",
+ "@crawlee/stagehand": "workspace:*",
+ "apify": "*",
+ "crawlee": "workspace:*",
+ "impit": "^0.14.2",
+ "pino": "^9.6.0",
"playwright-extra": "^4.3.6",
"puppeteer-extra": "^3.3.6",
- "puppeteer-extra-plugin-stealth": "^2.11.2"
+ "puppeteer-extra-plugin-stealth": "^2.11.2",
+ "winston": "^3.17.0"
}
}
diff --git a/docs/public-api/README.md b/docs/public-api/README.md
new file mode 100644
index 000000000000..2a92effa0579
--- /dev/null
+++ b/docs/public-api/README.md
@@ -0,0 +1,83 @@
+# Public API surface maps
+
+Each `*.api.md` file in this folder is a generated **map of the public, type-level
+interface** of one publishable `@crawlee/*` package — every exported class, method,
+property, function, and type, with full signatures. These reports define **where we
+promise backwards compatibility**.
+
+They are produced by [API Extractor](https://api-extractor.com/) from the built
+`dist/index.d.ts` of each package.
+
+## Workflow
+
+- After changing any package's public surface, regenerate the reports and commit them:
+
+ ```sh
+ pnpm build # the reports are generated from dist/
+ pnpm api:extract
+ ```
+
+- CI runs `pnpm api:check`, which fails if a committed report is out of date. A failing
+ check means you changed the public API: either that change is intentional (commit the
+ updated report — reviewers will see the surface diff) or it was accidental (fix it).
+
+- `api:check` also fails if a report ends up referencing a symbol it never declares, which
+ leaves the committed map describing a type nothing in it defines. Regenerating cannot fix
+ that; it has to be fixed in the source. In practice it means a `@public` symbol's signature
+ references an `@internal`/`@ignore`-d one, so the referenced type is trimmed out from under
+ it. Either drop the referenced type's tag (it is reachable from the public API, so users can
+ already depend on it) or keep it out of the public signature. An untagged symbol is
+ implicitly public, which is the convention here — the codebase does not use explicit
+ `@public` tags.
+
+ A symbol that is merely missing from the package's exports does **not** need fixing: see the
+ note on forgotten exports below.
+
+## Notes
+
+- The reports are generated as API Extractor's **`public`** variant, so symbols tagged
+ `@internal` (`@alpha`/`@beta` too) are excluded — only `@public` surface is tracked.
+ The legacy `@ignore` tag counts as `@internal` here; the generator rewrites it before
+ extraction, so an `@ignore`-d symbol is excluded too and cannot be referenced from a
+ `@public` signature.
+ The generator stages the variant as `.public.api.md` under `temp/` and promotes it
+ onto the committed `.api.md`, so the tracked filenames stay stable.
+- API Extractor builds the import list before it trims the non-`@public` declarations and
+ never revisits it, so a type reachable only from an `@internal` member would linger as a
+ bare import and read as public surface. There is no config option for this, so the
+ generator post-processes each report: it parses the fenced TypeScript and drops imports
+ whose binding is referenced by no declaration that survived the trim.
+- **Forgotten exports** — types the public API references but the entry point never exports —
+ are included in the report via `includeForgottenExports` and carry an explicit banner:
+
+ ```ts
+ // Not exported by the entry point; reachable only as a referenced type.
+ // @public (undocumented)
+ interface SitemapUrlData {
+ ```
+
+ Their *shape* is part of the surface we promise not to break, but their *name* is not
+ importable, so they are emitted without `export`. API Extractor labels them `@public
+ (undocumented)` like anything else, which is indistinguishable from a real export at a
+ glance, hence the added banner. The alternative was exporting every such type from its
+ package — ~38 new public exports, committing us to names we never meant to publish. If you
+ *want* one importable, export it deliberately and the report will show it with `export`.
+- Because API Extractor decides both of the above before the `@public` trim, it also offers
+ declarations for symbols reachable only from members that never reach the report. The
+ generator drops those the same way it drops dead imports, so the report carries nothing it
+ does not refer to. Only symbols flagged `ae-forgotten-export` are eligible, which is what
+ keeps genuinely reachable declarations (e.g. the `social` namespace in `@crawlee/utils`,
+ whose members are exposed through a `declare namespace` block) from being pruned.
+- `docs/public-api/temp/` holds intermediate reports (including the staged `.public.api.md`
+ files) and is git-ignored.
+- `@crawlee/cli` and `@crawlee/templates` are deliberately excluded — they are tooling
+ (a CLI binary and project scaffolding), not an importable API where we promise BC. The
+ exclude list lives in `scripts/api-extractor/run.ts`.
+- The generator lives in `scripts/api-extractor/`. It temporarily strips the build's
+ injected `// @ts-ignore` comment lines from the `.d.ts` files (restoring them
+ afterwards) because API Extractor's AST walker trips over some of them; a small number
+ of packages additionally need a sanitized-mirror fallback. See the comments in
+ `scripts/api-extractor/run.ts` for details.
+- These reports now cover only the `@public` surface. Further shrinking them — genuinely
+ hiding class internals (untagged `protected`/`_`-prefixed members) rather than merely
+ tagging them — is the goal tracked in issue #3109.
diff --git a/docs/public-api/crawlee-basic.api.md b/docs/public-api/crawlee-basic.api.md
new file mode 100644
index 000000000000..77f5c72605ec
--- /dev/null
+++ b/docs/public-api/crawlee-basic.api.md
@@ -0,0 +1,222 @@
+## Public API Report File for "@crawlee/basic"
+
+> Do not edit this file. It is a report generated by [API Extractor](https://api-extractor.com/).
+
+```ts
+
+import type { AddRequestsBatchedOptions } from '@crawlee/core';
+import type { AddRequestsBatchedResult } from '@crawlee/core';
+import type { Awaitable } from '@crawlee/types';
+import { BaseHttpClient } from '@crawlee/http-client';
+import { ConcurrencySystem } from '@crawlee/core';
+import { ConcurrencySystemOptions } from '@crawlee/core';
+import { Configuration } from '@crawlee/core';
+import { ContextPipeline } from '@crawlee/core';
+import type { CrawleeLogger } from '@crawlee/core';
+import { CrawlingContext } from '@crawlee/core';
+import { Dataset } from '@crawlee/core';
+import type { DatasetExportOptions } from '@crawlee/core';
+import type { Dictionary } from '@crawlee/types';
+import type { EnqueueUrlsOptions } from '@crawlee/core';
+import { EventManager } from '@crawlee/core';
+import type { FinalStatistics } from '@crawlee/core';
+import type { GetUserDataFromRequest } from '@crawlee/core';
+import { IConcurrencySystem } from '@crawlee/core';
+import { IProxyConfiguration } from '@crawlee/core';
+import { IRequestLoader } from '@crawlee/core';
+import { IRequestManager } from '@crawlee/core';
+import type { ISession } from '@crawlee/types';
+import type { ISessionPool } from '@crawlee/types';
+import { IStatistics } from '@crawlee/core';
+import type { ProxyInfo } from '@crawlee/types';
+import type { ReadonlyDeep } from 'type-fest';
+import { Request as Request_2 } from '@crawlee/core';
+import { RequestQueue } from '@crawlee/core';
+import { RobotsTxtFile } from '@crawlee/utils';
+import type { RouterHandler } from '@crawlee/core';
+import type { RouterRoutes } from '@crawlee/core';
+import type { SetStatusMessageOptions } from '@crawlee/types';
+import type { SkippedRequestCallback } from '@crawlee/core';
+import type { StatisticState } from '@crawlee/core';
+import type { StorageBackend } from '@crawlee/types';
+import type { StorageIdentifier } from '@crawlee/core';
+import { StorageWritePolicy } from '@crawlee/core';
+import type { TaskLoopPredicates } from '@crawlee/core';
+import { TimeoutError } from '@apify/timeout';
+import type { TypedRequestsLike } from '@crawlee/core';
+
+// @public (undocumented)
+export class BasicCrawler, ExtendedContext extends Context = Context & ContextExtension, Routes extends Record = Record>, StatisticStateExtension extends object = {}> {
+ constructor(options?: BasicCrawlerOptions & RequireContextPipeline);
+ // (undocumented)
+ protected readonly additionalHttpErrorStatusCodes: Set;
+ addRequests(requests: ReadonlyDeep>, options?: CrawlerAddRequestsOptions): Promise;
+ get basicContextPipeline(): ContextPipeline<{
+ request: Request_2;
+ }, CrawlingContext>;
+ // (undocumented)
+ protected blockedStatusCodes: Set;
+ protected buildContextPipeline(): ContextPipeline;
+ get concurrencySystem(): IConcurrencySystem | undefined;
+ // (undocumented)
+ get contextPipeline(): ContextPipeline;
+ protected createDefaultConcurrencySystem(options: ConcurrencySystemOptions): ConcurrencySystem;
+ exportData(path: string, format?: 'json' | 'csv', options?: DatasetExportOptions): Promise;
+ // (undocumented)
+ protected getCookieHeaderFromRequest(request: Request_2): string;
+ getData(...args: Parameters): ReturnType;
+ getDataset(identifier?: string | StorageIdentifier): Promise;
+ protected getMessageFromError(error: Error, forceStack?: boolean): string | TimeoutError | undefined;
+ protected getNavigationTimeoutMillis(): number;
+ getRequestManager(): Promise;
+ // @deprecated (undocumented)
+ getRequestQueue(): Promise;
+ // (undocumented)
+ protected getRobotsTxtFileForUrl(url: string): Promise;
+ // (undocumented)
+ hasFinishedBefore: boolean;
+ // (undocumented)
+ protected readonly httpClient: BaseHttpClient;
+ protected init(): Promise;
+ // (undocumented)
+ protected readonly internalTimeoutMillis: number;
+ protected isErrorStatusCode(status: number): boolean;
+ protected isProxyError(error: Error): boolean;
+ // (undocumented)
+ get log(): CrawleeLogger;
+ pause(timeoutSecs?: number): Promise;
+ readonly proxyConfiguration?: IProxyConfiguration;
+ pushData(data: Parameters[0], datasetIdentifier?: string | StorageIdentifier): Promise;
+ protected recordDomainRateLimit(url: string, retryAfterHeader?: string | null): boolean;
+ // (undocumented)
+ protected readonly requestHandler: RequestHandler;
+ protected requestManager?: IRequestManager;
+ resume(): void;
+ // (undocumented)
+ protected readonly retryOnBlocked: boolean;
+ readonly router: RouterHandler;
+ run(requests?: TypedRequestsLike, options?: CrawlerRunOptions): Promise;
+ // (undocumented)
+ running: boolean;
+ // (undocumented)
+ protected runRequestHandler(crawlingContext: ExtendedContext): Promise;
+ get sessionPool(): ISessionPool;
+ setStatusMessage(message: string, options?: SetStatusMessageOptions): void;
+ get statistics(): IStatistics;
+ stop(reason?: string): void;
+ teardown(): Promise;
+ protected throwOnBlockedRequest(statusCode: number): void;
+ // (undocumented)
+ useState(defaultValue?: State): Promise;
+}
+
+// @public (undocumented)
+export interface BasicCrawlerOptions, ExtendedContext extends Context = Context & ContextExtension, Routes extends Record = Record>, StatisticStateExtension extends object = {}> {
+ additionalHttpErrorStatusCodes?: number[];
+ blockedStatusCodes?: number[];
+ concurrencySystem?: IConcurrencySystem;
+ configuration?: Configuration;
+ contextPipelineBuilder?: () => ContextPipeline;
+ errorHandler?: ErrorHandler;
+ eventManager?: EventManager;
+ extendContext?: (context: CrawlingContext) => Awaitable;
+ failedRequestHandler?: ErrorHandler;
+ httpClient?: BaseHttpClient;
+ id?: string;
+ ignoreHttpErrorStatusCodes?: number[];
+ keepAlive?: boolean;
+ logger?: CrawleeLogger;
+ maxConcurrency?: number;
+ maxCrawlDepth?: number;
+ maxRequestRetries?: number;
+ maxRequestsPerCrawl?: number;
+ maxRequestsPerMinute?: number;
+ minConcurrency?: number;
+ onSkippedRequest?: SkippedRequestCallback;
+ proxyConfiguration?: IProxyConfiguration;
+ requestHandler?: RouterHandler | RequestHandler;
+ requestHandlerTimeoutSecs?: number;
+ // @deprecated
+ requestList?: IRequestLoader;
+ requestManager?: IRequestManager;
+ // @deprecated
+ requestQueue?: RequestQueue;
+ respectRobotsTxtFile?: boolean | {
+ userAgent?: string;
+ };
+ retryOnBlocked?: boolean;
+ sameDomainDelaySecs?: number;
+ sessionPool?: ISessionPool;
+ statistics?: IStatistics;
+ statusMessageCallback?: StatusMessageCallback;
+ statusMessageLoggingInterval?: number;
+ storageBackend?: StorageBackend;
+ taskLoopOptions?: TaskLoopPredicates;
+ transactionalStorage?: boolean | Partial;
+}
+
+// @public (undocumented)
+export interface BasicCrawlingContext extends CrawlingContext {
+}
+
+// @public (undocumented)
+export interface CrawlerAddRequestsOptions extends AddRequestsBatchedOptions, EnqueueUrlsOptions {
+}
+
+// @public (undocumented)
+export interface CrawlerAddRequestsResult extends AddRequestsBatchedResult {
+}
+
+// @public (undocumented)
+export interface CrawlerRunOptions extends CrawlerAddRequestsOptions {
+ purgeRequestQueue?: boolean;
+}
+
+// @public
+export function createBasicRouter = Record>>(routes?: RouterRoutes): RouterHandler;
+
+// @public (undocumented)
+export function createBasicRouter>(routes?: RouterRoutes>): RouterHandler>;
+
+// @public (undocumented)
+export interface CreateContextOptions {
+ // (undocumented)
+ proxyInfo?: ProxyInfo;
+ // (undocumented)
+ request: Request_2;
+ // (undocumented)
+ session: ISession;
+}
+
+// @public
+export type ErrorHandler = (inputs: BaseContext & Partial, error: Error) => Awaitable;
+
+// @public (undocumented)
+export type RequestHandler = (inputs: Context) => Awaitable;
+
+// @public (undocumented)
+export type RequireContextPipeline = DefaultContextType extends FinalContextType ? {} : {
+ contextPipelineBuilder: () => ContextPipeline;
+};
+
+// @public (undocumented)
+export type StatusMessageCallback = BasicCrawler> = (params: StatusMessageCallbackParams) => Awaitable;
+
+// @public (undocumented)
+export interface StatusMessageCallbackParams = BasicCrawler> {
+ // (undocumented)
+ crawler: Crawler;
+ // (undocumented)
+ message: string;
+ // (undocumented)
+ previousState: StatisticState;
+ // (undocumented)
+ state: StatisticState;
+}
+
+
+export * from "@crawlee/core";
+
+// (No @packageDocumentation comment for this package)
+
+```
diff --git a/docs/public-api/crawlee-browser-pool.api.md b/docs/public-api/crawlee-browser-pool.api.md
new file mode 100644
index 000000000000..3692e6313837
--- /dev/null
+++ b/docs/public-api/crawlee-browser-pool.api.md
@@ -0,0 +1,656 @@
+## Public API Report File for "@crawlee/browser-pool"
+
+> Do not edit this file. It is a report generated by [API Extractor](https://api-extractor.com/).
+
+```ts
+
+import type { Browser } from 'playwright';
+import type { BrowserContext } from 'playwright';
+import type { BrowserFingerprintWithHeaders } from 'fingerprint-generator';
+import type { BrowserType } from 'playwright';
+import type { Cookie } from '@crawlee/types';
+import { CrawleeLogger } from '@crawlee/core';
+import { CriticalError } from '@crawlee/core';
+import type { Dictionary } from '@crawlee/types';
+import { EventEmitter } from 'node:events';
+import { FingerprintGenerator as FingerprintGenerator_2 } from 'fingerprint-generator';
+import type { FingerprintGeneratorOptions as FingerprintGeneratorOptions_2 } from 'fingerprint-generator';
+import { FingerprintInjector } from 'fingerprint-injector';
+import { IBrowserPool } from '@crawlee/types';
+import { NewPageOptions } from '@crawlee/types';
+import type { Page } from 'playwright';
+import type { PageState } from '@crawlee/types';
+import type Puppeteer from 'puppeteer';
+import type * as PuppeteerTypes from 'puppeteer';
+import QuickLRU from 'quick-lru';
+import { TypedEmitter } from 'tiny-typed-emitter';
+
+// @public (undocumented)
+export interface AnonymizeProxySugarOptions {
+ // (undocumented)
+ ignoreProxyCertificate?: boolean;
+}
+
+// @public (undocumented)
+export enum BROWSER_CONTROLLER_EVENTS {
+ // (undocumented)
+ BROWSER_CLOSED = "browserClosed"
+}
+
+// @public (undocumented)
+export enum BROWSER_POOL_EVENTS {
+ // (undocumented)
+ BROWSER_CLOSED = "browserClosed",
+ // (undocumented)
+ BROWSER_LAUNCHED = "browserLaunched",
+ // (undocumented)
+ BROWSER_RETIRED = "browserRetired",
+ // (undocumented)
+ PAGE_CLOSED = "pageClosed",
+ // (undocumented)
+ PAGE_CREATED = "pageCreated"
+}
+
+// @public
+export abstract class BrowserController[0], LaunchResult extends CommonBrowser = UnwrapPromise>, NewPageOptions = Parameters[0], NewPageResult = UnwrapPromise>> extends TypedEmitter> implements IBrowserController {
+ constructor(browserPlugin: BrowserPlugin);
+ // (undocumented)
+ activePages: number;
+ browser: LaunchResult;
+ readonly browserPlugin: BrowserPlugin;
+ close(): Promise;
+ // (undocumented)
+ protected abstract _close(): Promise;
+ // (undocumented)
+ getCookies(page: NewPageResult): Promise;
+ // (undocumented)
+ protected abstract _getCookies(page: NewPageResult): Promise;
+ // (undocumented)
+ readonly id: string;
+ // (undocumented)
+ isActive: boolean;
+ kill(): Promise;
+ // (undocumented)
+ protected abstract _kill(): Promise;
+ // (undocumented)
+ lastPageOpenedAt: number;
+ launchContext: LaunchContext;
+ // (undocumented)
+ protected readonly log: CrawleeLogger;
+ // (undocumented)
+ protected abstract _newPage(pageOptions?: NewPageOptions): Promise;
+ // (undocumented)
+ abstract normalizeProxyOptions(proxyUrl: string | undefined, pageOptions: any): Record;
+ proxyUrl?: string;
+ // (undocumented)
+ setCookies(page: NewPageResult, cookies: Cookie[]): Promise;
+ // (undocumented)
+ protected abstract _setCookies(page: NewPageResult, cookies: Cookie[]): Promise;
+ // (undocumented)
+ totalPages: number;
+}
+
+// @public (undocumented)
+export interface BrowserControllerEvents[0], LaunchResult extends CommonBrowser = UnwrapPromise>, NewPageOptions = Parameters[0], NewPageResult = UnwrapPromise>> {
+ // (undocumented)
+ [BROWSER_CONTROLLER_EVENTS.BROWSER_CLOSED]: (controller: BrowserController) => void;
+}
+
+// @public (undocumented)
+export class BrowserLaunchError extends CriticalError {
+ constructor(...args: ConstructorParameters);
+}
+
+// @public (undocumented)
+export enum BrowserName {
+ // (undocumented)
+ chrome = "chrome",
+ // (undocumented)
+ edge = "edge",
+ // (undocumented)
+ firefox = "firefox",
+ // (undocumented)
+ safari = "safari"
+}
+
+// Not exported by the entry point; reachable only as a referenced type.
+// @public (undocumented)
+interface BrowserOptions {
+ // (undocumented)
+ browserContext: BrowserContext;
+ // (undocumented)
+ version: string;
+}
+
+// @public
+export abstract class BrowserPlugin[0], LaunchResult extends CommonBrowser = UnwrapPromise>, NewPageOptions = Parameters[0], NewPageResult = UnwrapPromise>> {
+ constructor(library: Library, options?: BrowserPluginOptions);
+ // (undocumented)
+ protected abstract addProxyToLaunchOptions(launchContext: LaunchContext): Promise;
+ // (undocumented)
+ readonly browserPerProxy?: boolean;
+ protected connectToRemoteBrowser(launchContext: LaunchContext, connect: (url: string) => Promise): Promise;
+ // (undocumented)
+ abstract createController(): BrowserController;
+ createLaunchContext(options?: CreateLaunchContextOptions): LaunchContext;
+ // (undocumented)
+ readonly ignoreProxyCertificate?: boolean;
+ // (undocumented)
+ protected abstract isChromiumBasedBrowser(launchContext: LaunchContext): boolean;
+ launch(launchContext?: LaunchContext): Promise;
+ // (undocumented)
+ protected abstract _launch(launchContext: LaunchContext): Promise;
+ // (undocumented)
+ readonly launchOptions: LibraryOptions;
+ // (undocumented)
+ readonly library: Library;
+ // (undocumented)
+ protected readonly log: CrawleeLogger;
+ // (undocumented)
+ readonly name: string;
+ // (undocumented)
+ readonly proxyUrl?: string;
+ // (undocumented)
+ protected throwAugmentedLaunchError(cause: unknown, executablePath: string | undefined, dockerImage: string, moduleInstallCommand: string): never;
+ // (undocumented)
+ useIncognitoPages: boolean;
+ // (undocumented)
+ readonly userDataDir?: string;
+}
+
+// @public (undocumented)
+export interface BrowserPluginOptions {
+ browserPerProxy?: boolean;
+ ignoreProxyCertificate?: boolean;
+ launchOptions?: LibraryOptions;
+ proxyUrl?: string;
+ useIncognitoPages?: boolean;
+ userDataDir?: string;
+}
+
+// @public
+export class BrowserPool, BrowserControllerReturn extends BrowserController = ReturnType, LaunchContextReturn extends LaunchContext = ReturnType, PageOptions = Parameters[0], PageReturn extends UnwrapPromise> = UnwrapPromise>> extends TypedEmitter> implements IBrowserPool {
+ // (undocumented)
+ [Symbol.asyncDispose](): Promise;
+ constructor(options: Options & BrowserPoolHooks);
+ // (undocumented)
+ activeBrowserControllers: Set;
+ // (undocumented)
+ browserPlugins: BrowserPlugins;
+ closeAllBrowsers(): Promise;
+ // (undocumented)
+ closeInactiveBrowserAfterMillis: number;
+ closePage(page: PageReturn, options?: {
+ error?: Error;
+ }): Promise;
+ destroy(): Promise;
+ extractPageState(page: PageReturn): Promise;
+ // (undocumented)
+ fingerprintCache?: QuickLRU;
+ // (undocumented)
+ fingerprintGenerator?: FingerprintGenerator_2;
+ // (undocumented)
+ fingerprintInjector?: FingerprintInjector;
+ // (undocumented)
+ fingerprintOptions: FingerprintOptions;
+ getBrowserControllerByPage(page: PageReturn): BrowserControllerReturn | undefined;
+ getPage(id: string): PageReturn | undefined;
+ getPageId(page: PageReturn): string | undefined;
+ hasActiveBrowserWithFreeCapacity(): boolean;
+ hasFreeBrowserSlot(): boolean;
+ injectPageState(page: PageReturn, state: PageState): Promise;
+ // (undocumented)
+ maxOpenBrowsers: number;
+ // (undocumented)
+ maxOpenPagesPerBrowser: number;
+ newPage(options?: BrowserPoolNewPageOptions): Promise;
+ newPageInNewBrowser(options?: BrowserPoolNewPageInNewBrowserOptions): Promise;
+ newPageWithEachPlugin(optionsList?: Omit, 'browserPlugin'>[]): Promise;
+ // (undocumented)
+ operationTimeoutMillis: number;
+ // (undocumented)
+ pageCounter: number;
+ // (undocumented)
+ pageIds: WeakMap;
+ // (undocumented)
+ pages: Map;
+ // (undocumented)
+ pageToBrowserController: WeakMap;
+ // (undocumented)
+ postLaunchHooks: PostLaunchHook[];
+ // (undocumented)
+ postPageCloseHooks: PostPageCloseHook[];
+ // (undocumented)
+ postPageCreateHooks: PostPageCreateHook[];
+ // (undocumented)
+ preLaunchHooks: PreLaunchHook[];
+ // (undocumented)
+ prePageCloseHooks: PrePageCloseHook[];
+ // (undocumented)
+ prePageCreateHooks: PrePageCreateHook[];
+ retireAllBrowsers(): void;
+ // (undocumented)
+ retireBrowserAfterPageCount: number;
+ retireBrowserByPage(page: PageReturn): void;
+ retireBrowserController(browserController: BrowserControllerReturn): void;
+ // (undocumented)
+ retiredBrowserControllers: Set;
+ // (undocumented)
+ startingBrowserControllers: Set;
+ // (undocumented)
+ useFingerprints?: boolean;
+}
+
+// @public (undocumented)
+export interface BrowserPoolEvents {
+ // (undocumented)
+ [BROWSER_POOL_EVENTS.BROWSER_LAUNCHED]: (browserController: BC) => void | Promise;
+ // (undocumented)
+ [BROWSER_POOL_EVENTS.BROWSER_RETIRED]: (browserController: BC) => void | Promise;
+ // (undocumented)
+ [BROWSER_POOL_EVENTS.PAGE_CLOSED]: (page: Page) => void | Promise;
+ // (undocumented)
+ [BROWSER_POOL_EVENTS.PAGE_CREATED]: (page: Page) => void | Promise;
+}
+
+// @public (undocumented)
+export interface BrowserPoolHooks> = UnwrapPromise>> {
+ postLaunchHooks?: PostLaunchHook[];
+ postPageCloseHooks?: PostPageCloseHook[];
+ postPageCreateHooks?: PostPageCreateHook[];
+ preLaunchHooks?: PreLaunchHook[];
+ prePageCloseHooks?: PrePageCloseHook[];
+ prePageCreateHooks?: PrePageCreateHook[];
+}
+
+// @public (undocumented)
+export interface BrowserPoolNewPageInNewBrowserOptions {
+ browserPlugin?: BP;
+ id?: string;
+ launchOptions?: BP['launchOptions'];
+ pageOptions?: PageOptions;
+}
+
+// @public (undocumented)
+export interface BrowserPoolNewPageOptions extends NewPageOptions {
+ browserPlugin?: BP;
+ ignoreTlsErrors?: boolean;
+ pageOptions?: PageOptions;
+ proxyUrl?: string;
+}
+
+// @public (undocumented)
+export interface BrowserPoolOptions {
+ browserPlugins: readonly Plugin[];
+ closeInactiveBrowserAfterSecs?: number;
+ // (undocumented)
+ fingerprintOptions?: FingerprintOptions;
+ maxOpenPagesPerBrowser?: number;
+ operationTimeoutSecs?: number;
+ retireBrowserAfterPageCount?: number;
+ retireInactiveBrowserAfterSecs?: number;
+ useFingerprints?: boolean;
+}
+
+// @public (undocumented)
+export interface BrowserSpecification {
+ httpVersion?: HttpVersion;
+ maxVersion?: number;
+ minVersion?: number;
+ name: BrowserName;
+}
+
+// Not exported by the entry point; reachable only as a referenced type.
+// @public (undocumented)
+interface CommonBrowser {
+ // (undocumented)
+ newPage(...args: unknown[]): Promise;
+}
+
+// @public
+export interface CommonLibrary {
+ // (undocumented)
+ launch(opts?: Dictionary): Promise;
+ // (undocumented)
+ name?: () => string;
+ // (undocumented)
+ product?: string;
+}
+
+// @public (undocumented)
+export interface CommonPage {
+ // (undocumented)
+ close(...args: unknown[]): Promise;
+ // (undocumented)
+ evaluate(pageFunction: ((...args: any[]) => unknown) | string, ...args: unknown[]): Promise;
+ // (undocumented)
+ url(): string | Promise;
+}
+
+// @public
+export type CrawlerRemoteBrowserOptions = Omit;
+
+// @public (undocumented)
+export interface CreateLaunchContextOptions[0], LaunchResult extends CommonBrowser = UnwrapPromise>, NewPageOptions = Parameters[0], NewPageResult = UnwrapPromise>> extends Partial, 'browserPlugin'>> {
+}
+
+// @public
+export const DEFAULT_USER_AGENT = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36";
+
+// @public (undocumented)
+export enum DeviceCategory {
+ desktop = "desktop",
+ mobile = "mobile"
+}
+
+// @public (undocumented)
+export interface FingerprintGenerator {
+ // (undocumented)
+ getFingerprint: (fingerprintGeneratorOptions?: FingerprintGeneratorOptions) => GetFingerprintReturn;
+}
+
+// @public (undocumented)
+export interface FingerprintGeneratorOptions extends Partial {
+}
+
+// @public
+export interface FingerprintOptions {
+ fingerprintCacheSize?: number;
+ fingerprintGeneratorOptions?: FingerprintGeneratorOptions;
+ useFingerprintCache?: boolean;
+}
+
+// @public (undocumented)
+export interface GetFingerprintReturn {
+ // (undocumented)
+ fingerprint: BrowserFingerprintWithHeaders;
+}
+
+// Not exported by the entry point; reachable only as a referenced type.
+// @public
+type HttpVersion = (typeof SUPPORTED_HTTP_VERSIONS)[number];
+
+// @public
+export interface IBrowserController {
+ readonly browser: unknown;
+ close(): Promise;
+ getCookies(page: Page): Promise;
+ readonly id: string;
+ readonly launchContext: IBrowserLaunchContext;
+ setCookies(page: Page, cookies: Cookie[]): Promise;
+}
+
+// @public
+export interface IBrowserLaunchContext {
+ fingerprint?: unknown;
+ launchOptions?: Dictionary | undefined;
+ proxyUrl?: string;
+ useIncognitoPages?: boolean;
+}
+
+export { IBrowserPool }
+
+// @public (undocumented)
+export type InferBrowserPluginArray = Input extends readonly [infer FirstValue, ...infer Rest] | [infer FirstValue, ...infer Rest] ? FirstValue extends PlaywrightPlugin ? InferBrowserPluginArray : FirstValue extends PuppeteerPlugin ? InferBrowserPluginArray : never : Input extends [] ? Result : Input extends readonly (infer U)[] ? [
+U
+] extends [PuppeteerPlugin | PlaywrightPlugin] ? U[] : never : Result;
+
+// @public (undocumented)
+export class LaunchContext[0], LaunchResult extends CommonBrowser = UnwrapPromise>, NewPageOptions = Parameters[0], NewPageResult = UnwrapPromise>> {
+ constructor(options: LaunchContextOptions);
+ // (undocumented)
+ [K: PropertyKey]: unknown;
+ // (undocumented)
+ browserPerProxy?: boolean;
+ // (undocumented)
+ browserPlugin: BrowserPlugin;
+ extend>(fields: T): void;
+ // (undocumented)
+ fingerprint?: BrowserFingerprintWithHeaders;
+ // (undocumented)
+ id?: string;
+ // (undocumented)
+ ignoreProxyCertificate?: boolean;
+ // (undocumented)
+ readonly isRemote: boolean;
+ // (undocumented)
+ launchOptions: LibraryOptions;
+ set proxyUrl(url: string | undefined);
+ get proxyUrl(): string | undefined;
+ // (undocumented)
+ useIncognitoPages: boolean;
+ // (undocumented)
+ userDataDir: string;
+}
+
+// @public
+export interface LaunchContextOptions[0], LaunchResult extends CommonBrowser = UnwrapPromise>, NewPageOptions = Parameters[0], NewPageResult = UnwrapPromise>> {
+ browserPerProxy?: boolean;
+ browserPlugin: BrowserPlugin;
+ id?: string;
+ ignoreProxyCertificate?: boolean;
+ isRemote?: boolean;
+ launchOptions: LibraryOptions;
+ // (undocumented)
+ proxyUrl?: string;
+ useIncognitoPages?: boolean;
+ userDataDir?: string;
+}
+
+export { NewPageOptions }
+
+// @public (undocumented)
+export enum OperatingSystemsName {
+ android = "android",
+ ios = "ios",
+ // (undocumented)
+ linux = "linux",
+ // (undocumented)
+ macos = "macos",
+ // (undocumented)
+ windows = "windows"
+}
+
+// @public
+export class PlaywrightBrowser extends EventEmitter {
+ // (undocumented)
+ [Symbol.asyncDispose](): Promise;
+ constructor(options: BrowserOptions);
+ // (undocumented)
+ browserType(): BrowserType;
+ // (undocumented)
+ close(): Promise;
+ // (undocumented)
+ contexts(): BrowserContext[];
+ // (undocumented)
+ isConnected(): boolean;
+ // (undocumented)
+ newBrowserCDPSession(): Promise;
+ // (undocumented)
+ newContext(): Promise;
+ // (undocumented)
+ newPage(...args: Parameters): ReturnType;
+ // (undocumented)
+ startTracing(): Promise;
+ // (undocumented)
+ stopTracing(): Promise;
+ // (undocumented)
+ version(): string;
+}
+
+// @public (undocumented)
+export class PlaywrightController extends BrowserController[0], Browser> {
+ // (undocumented)
+ protected _close(): Promise;
+ // (undocumented)
+ protected _getCookies(page: Page): Promise;
+ // (undocumented)
+ protected _kill(): Promise;
+ // (undocumented)
+ protected _newPage(contextOptions?: SafeParameters[0]): Promise;
+ // (undocumented)
+ normalizeProxyOptions(proxyUrl: string | undefined, pageOptions: any): Record;
+ // (undocumented)
+ protected _setCookies(page: Page, cookies: Cookie[]): Promise;
+}
+
+// @public (undocumented)
+export class PlaywrightPlugin extends BrowserPlugin[0], Browser> {
+ // (undocumented)
+ protected addProxyToLaunchOptions(launchContext: LaunchContext): Promise;
+ // (undocumented)
+ createController(): PlaywrightController;
+ // (undocumented)
+ protected isChromiumBasedBrowser(): boolean;
+ // (undocumented)
+ protected _launch(launchContext: LaunchContext): Promise;
+ useRemoteConnection(connection: RemoteConnection, parameters?: RemoteConnectionParameters): void;
+}
+
+// @public
+export type PostLaunchHook = (pageId: string, browserController: BC) => void | Promise;
+
+// @public
+export type PostPageCloseHook = (pageId: string, browserController: BC) => void | Promise;
+
+// @public
+export type PostPageCreateHook>> = (page: Page, browserController: BC) => void | Promise;
+
+// @public
+export type PreLaunchHook = (pageId: string, launchContext: LC) => void | Promise;
+
+// @public
+export type PrePageCloseHook>> = (page: Page, browserController: BC) => void | Promise;
+
+// @public
+export type PrePageCreateHook[0]> = (pageId: string, browserController: BC, pageOptions?: PO) => void | Promise;
+
+// @public (undocumented)
+export class PuppeteerController extends BrowserController {
+ // (undocumented)
+ protected _close(): Promise;
+ // (undocumented)
+ protected _getCookies(page: PuppeteerTypes.Page): Promise;
+ // (undocumented)
+ protected _kill(): Promise;
+ // (undocumented)
+ protected _newPage(contextOptions?: PuppeteerNewPageOptions): Promise;
+ // (undocumented)
+ normalizeProxyOptions(proxyUrl: string | undefined, pageOptions: any): Record;
+ // (undocumented)
+ protected _setCookies(page: PuppeteerTypes.Page, cookies: Cookie[]): Promise
+ 
- 
- 
+