Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
219 commits
Select commit Hold shift + click to select a range
304bb66
chore(deps): update patch/minor dependencies
renovate[bot] Aug 18, 2026
c7eb771
ci: test on node 22 and 24
B4nan May 17, 2025
cc01b18
refactor: convert to native ESM
B4nan May 19, 2025
4ddca29
refactor: remove deprecated crawler options
B4nan May 20, 2025
8ce049c
refactor: make crawling context strict and remove the error fallback
B4nan May 20, 2025
9e8ad43
refactor: remove `additionalBlockedStatusCodes` parameter of `Session…
B4nan May 20, 2025
577135d
refactor: remove `additionalBlockedStatusCodes` parameter of `Session…
B4nan May 20, 2025
cbbc0bc
chore: skip docker image builds for v4
B4nan May 20, 2025
954b320
chore: use `v4` dist tag
B4nan May 20, 2025
d284417
chore: run tests on v4 branch
B4nan May 20, 2025
dfe017a
chore: fix build
B4nan May 20, 2025
be74290
chore: fix v4 publishing
B4nan May 20, 2025
cf93415
chore: use node 24 in e2e tests and project templates
B4nan May 21, 2025
b312bf7
chore: improve types to get rid of some `as any`
B4nan Jun 4, 2025
6b317b7
chore: remove some deadcode
B4nan Jun 4, 2025
7d01334
chore: bump a few more dependencies
B4nan Jun 11, 2025
8952c78
fix CLI
B4nan Jun 11, 2025
e6b7dc0
fix CLI 2
B4nan Jun 11, 2025
db404a1
fix: remove old system info implementation
B4nan Jun 11, 2025
7b5ffbe
chore: replace `lodash.isequal` with `util.isDeepStrictEqual`
B4nan Jun 17, 2025
70aa8c9
refactor!: Introduce the `ContextPipeline` abstraction (#3119)
janbuchar Nov 24, 2025
14ed530
feat: store `ProxyInfo` inside `Session` instances (#3199)
barjin Nov 26, 2025
32caa0c
feat!: use native `Request` / `Response` interface (#3163)
barjin Nov 27, 2025
52fe3d1
chore: fix build errors from `master` rebase (#3285)
barjin Dec 1, 2025
6361d06
chore: fix broken types in docs examples (#3287)
barjin Dec 1, 2025
8c5bc0f
chore(docs): Add (temporary) 4.0 docs snapshot (#3292)
janbuchar Dec 2, 2025
28468a3
chore: Fix HttpCrawler context types (#3291)
janbuchar Dec 2, 2025
34d579b
fix: `KVS.getPublicUrl()` reads the public URL directly from storage …
barjin Dec 2, 2025
55b4983
feat: replace `got`-specific `HttpRequest` with native `Request` inte…
barjin Dec 5, 2025
f57ff3d
feat: drop `gotOptions` params from the public interface (#3300)
barjin Dec 16, 2025
57d1f43
fix: patch `GotScrapingHttpClient` header handling and proxies (#3308)
barjin Dec 17, 2025
b96749d
feat: custom `httpClient` implementation for `@crawlee/utils` and `Re…
barjin Dec 18, 2025
58eb9fa
fix: log warning on sharing `useState()`, add `id` option to `BasicCr…
barjin Dec 18, 2025
db31875
fix: `sendRequest` uses correct `Session` instance and allows for ove…
barjin Dec 18, 2025
e44f7f3
feat: `fetch` API for custom http clients (#3326)
barjin Jan 20, 2026
f886e46
chore: drop `got-scraping` as the direct `crawlee` dependency (#3358)
barjin Jan 21, 2026
c6ab237
chore: add `discoverValidSitemaps` utility (#3392)
barjin Feb 6, 2026
6b507b8
fix: Fix file download hang (#3411)
janbuchar Feb 16, 2026
b663940
fix: rectify transformRequestFunction precedence rules (#3372)
l2ysho Feb 17, 2026
bcde2b0
refactor!: Extract service management from Configuration into Service…
janbuchar Feb 23, 2026
4738869
fix: treat `CookieJar` write methods as potentially throwing (#3426)
Copilot Feb 23, 2026
98be258
chore(deps): update got-scraping package (#3452)
l2ysho Feb 27, 2026
b5bea97
feat: decouple log configuration from apifylog (#3399)
l2ysho Mar 3, 2026
d33ab5f
refactor: use `ContextPipeline` to initialize `BasicCrawler`'s contex…
barjin Mar 4, 2026
146e02e
chore: Lint
janbuchar Mar 5, 2026
b821af2
fix: Migrate StagehandCrawler to ContextPipeline
janbuchar Mar 6, 2026
51bba83
chore: Fix more build issues
janbuchar Mar 6, 2026
181ddac
chore: run linter
barjin Mar 9, 2026
6653fb4
docs: sync 4.0 version with the latest changes in `master`
barjin Mar 9, 2026
8d53131
chore: fix TypeScript errors in tests
barjin Mar 9, 2026
0d838da
docs: fix types in docs examples
barjin Mar 9, 2026
c9a1eaf
chore: fix tests (new error messages)
barjin Mar 10, 2026
9e49c22
chore: bump vitest to `beta`, fix failing ALS tests
barjin Mar 10, 2026
c3e5697
feat: add `AbortSignal` and rename `timeout` → `timeoutMillis` in `Se…
Copilot Mar 10, 2026
94603bb
chore: Fix rebase conflict resolution errors
janbuchar Mar 11, 2026
0b5b3f8
chore(memory-storage): sort dependencies and update yarn.lock (#3493)
l2ysho Mar 16, 2026
6456a64
fix: improve ServiceConflictError message readability (#3480)
l2ysho Mar 16, 2026
2e1e450
feat: drop `useSessionPool` option, make `SessionPool` required in al…
barjin Mar 17, 2026
46c473c
docs: add custom logger guide (#3473)
l2ysho Mar 17, 2026
9c0bd50
refactor: use `BasicCrawler` session in browser launch context (#3489)
barjin Mar 18, 2026
8bf2407
refactor: move blocked HTTP status code management to `BasicCrawler` …
barjin Mar 19, 2026
a23e79b
feat: lazy-initialize `SessionPool`, drop `SessionPool.open()` (#3513)
barjin Mar 24, 2026
1508ab9
refactor: replace service locator forwarding object with Proxy (#3515)
B4nan Mar 25, 2026
0eda96a
refactor: resolve last direct @apify/log calls
l2ysho Mar 25, 2026
d9b5fe0
fix: Fix `skipNavigation` behavior in `HttpCrawler` subclasses (#3381)
janbuchar Mar 26, 2026
1783146
fix: clean-up cookie handling precedence rules (#3517)
barjin Mar 31, 2026
0174b81
fix(puppeteer-crawler): migrate cookie handling to browser-context AP…
B4nan Apr 15, 2026
1640c40
chore: migrate from eslint/biome to oxlint/oxfmt (#3569)
B4nan Apr 17, 2026
cb1b984
feat: share `SessionPool` instance between crawlers (#3499)
barjin Apr 17, 2026
52bc273
feat: allow binding a Session instance to a Request (#3518)
barjin Apr 17, 2026
fa06fab
chore: migrate to pnpm and enable minimum release age (#3581)
B4nan Apr 20, 2026
b2e40e8
chore: migrate `patch-package` to `pnpm patch` (#3585)
barjin Apr 21, 2026
6cbd3ac
refactor!: Overhaul the StorageClient interface (#3570)
janbuchar Apr 22, 2026
9047dc2
feat(`session-pool`): add `sessionReuseStrategy` option (#3587)
barjin Apr 23, 2026
2fad6be
refactor: move `describe` helper to `@crawlee/utils` as `inspectValue…
l2ysho Apr 27, 2026
b8cea5a
chore: use `@apify/oxlint-config` as shared lint preset (#3607)
B4nan Apr 28, 2026
03e343a
chore(deps-dev): bump oxlint to 1.62 and oxfmt to 0.46 (#3608)
B4nan Apr 28, 2026
cfb6dde
chore: pass --no-error-on-unmatched-pattern to oxlint in lint-staged …
B4nan Apr 29, 2026
7b60610
feat: redesign `Configuration` class for v4 (#3484)
B4nan Apr 29, 2026
69157f5
chore(ci): pin lerna's minimatch to v3 and fail step on lerna error
B4nan Apr 29, 2026
1093e31
chore(ci): treat any non-prod dist-tag as canary in publish workflow
B4nan Apr 29, 2026
e4afb37
chore(ci): bake --yes into publish scripts
B4nan Apr 29, 2026
ac41130
chore(ci): restore Docker image dispatch, gated to master
B4nan Apr 29, 2026
99b751f
refactor!: remove `tieredProxyUrls` support (#3599)
barjin Apr 30, 2026
9ffa53c
fix(linkedom-crawler): declare cheerio as a dependency (#3620)
B4nan Apr 30, 2026
1cf1060
feat: reuse `Sessions` correctly in `BasicCrawler` (#3605)
barjin May 4, 2026
337b54a
feat: rename `persistCookiesPerSession` to `saveResponseCookies`, var…
barjin May 6, 2026
1a8de68
refactor!: treat session errors as regular request retries (#3631)
barjin May 11, 2026
b6e5dd6
refactor!: Rename StorageManager to StorageInstanceManager and use it…
janbuchar May 11, 2026
63ef023
refactor: add `Configuration.reset()` static method (#3649)
B4nan May 12, 2026
f673ca5
chore: migrate v4 to pnpm 11 (#3645)
B4nan May 12, 2026
a65778a
ci: disable pnpm 11's verifyDepsBeforeRun to unbreak parallel turbo c…
B4nan May 13, 2026
412ad09
refactor: revert Configuration.reset() static method (#3651)
B4nan May 13, 2026
37a0c57
refactor!: remove the `EventEmitter` interface from `SessionPool` (#3…
barjin May 15, 2026
dcf8d6b
feat: export `SessionPool` interface, minimize `ISession`, allow cust…
barjin May 25, 2026
49e417d
chore: bump puppeteer to 25 and playwright to 1.60 (#3703)
barjin Jun 1, 2026
59ec997
refactor!: Align `DatasetClient` and `KeyValueStoreClient` interfaces…
janbuchar Jun 1, 2026
7edc793
feat: Implement storage aliases (#3636)
janbuchar Jun 4, 2026
342dd44
feat: store browser / HTTP client fingerprint in Session (#3646)
barjin Jun 5, 2026
a54e308
feat: allow navigation hooks to override crawling context members (#3…
barjin Jun 8, 2026
4041322
refactor!: Finalize transition to IRequestManager and IRequestLoader …
janbuchar Jun 9, 2026
97701bd
refactor!: Introduce IBrowserPool interface and utilize it in Browser…
janbuchar Jun 10, 2026
32c0e74
refactor!: Align `RequestQueueClient` interface with its Python count…
janbuchar Jun 15, 2026
2f0a60d
refactor!: Split MemoryStorage into FileSystemStorageClient and Memor…
janbuchar Jun 17, 2026
a63239a
feat: make `impit` the default HTTP client (#3767)
barjin Jun 23, 2026
60e9e36
docs: reflect the recent `SessionPool` changes in a guide (#3724)
barjin Jun 24, 2026
c067c50
refactor!: Centralize KVS value semantics in @crawlee/core (#3774)
janbuchar Jun 24, 2026
8af187d
feat: rename KVS objects from `SDK_` to `CRAWLEE_` (#3783)
barjin Jun 26, 2026
1dc8812
fix: read content encoding hints from `<meta>` tags (#3780)
barjin Jun 30, 2026
4f0c4c4
fix: disable storage access on `requestHandler` timeouts (#3788)
barjin Jul 1, 2026
a72033d
docs: Autogenerated public API documentation (#3798)
janbuchar Jul 1, 2026
5e2c92d
refactor!: Dissolve @crawlee/memory-storage into @crawlee/core (#3802)
janbuchar Jul 1, 2026
6c2fec0
fix: `RequestManagerTandem` returns correct total count (#3804)
barjin Jul 1, 2026
50293fe
fix: Prevent browser-based crawling mode in AdaptivePlaywrightCrawler…
janbuchar Jul 1, 2026
08ac4fe
fix(e2e): repair E2E test infrastructure for pnpm + crawlee v4 (#3805)
janbuchar Jul 2, 2026
43effa1
fix: Fix repeated Storage.open() using the legacy idOrName parameter …
janbuchar Jul 2, 2026
a57768b
fix: Rectify inconsistencies in the StorageClient interface family (#…
janbuchar Jul 5, 2026
ef1654a
test: Eliminate MemoryStorageEmulator (#3806)
janbuchar Jul 5, 2026
2944b32
refactor!: Rewrite the FilesystemStorageClient to use apify/crawlee-s…
janbuchar Jul 5, 2026
d73291e
refactor!: Move status message handling from StorageClient to the eve…
janbuchar Jul 5, 2026
d0c5803
fix(memory-storage): store JSON snapshots in `DatasetClient.pushData(…
AmariahAK Jul 9, 2026
d7e44dc
refactor!: Rename `StorageClient` to `StorageBackend` (#3825)
janbuchar Jul 13, 2026
6c50d37
ci: exclude `@crawlee/fs-storage-native` from canary/pin version rewr…
B4nan Jul 13, 2026
01ab6bb
feat: connect to remote browser services (#3545)
l2ysho Jul 14, 2026
8eab134
feat: unify extended unique key format with Python, add `alwaysEnqueu…
barjin Jul 15, 2026
3090359
chore: fix `v4` CI build issues (#3856)
barjin Jul 15, 2026
fcca6fe
feat: make `cheerio` and `proxy-chain` imports dynamic (#3836)
barjin Jul 15, 2026
8f95bb2
chore: make `format:check` actually check the formatting (#3858)
B4nan Jul 15, 2026
895978c
test: run the browser crawler tests serially (#3859)
B4nan Jul 15, 2026
b6eb805
chore: fix master-v4 rebase issues
barjin Jul 20, 2026
90ebb89
chore: regenerate TypeScript API reports
barjin Jul 20, 2026
40631aa
chore: fix oxlint errors
barjin Jul 20, 2026
f745f87
chore(ci): fix failing e2e and docs actions after rebase (#3874)
barjin Jul 21, 2026
e614738
feat: move autoscaling helpers to `@crawlee/core` (#3835)
barjin Jul 21, 2026
ffd1dab
fix: remove noisy implicit configuration warning from service locator…
barjin Jul 21, 2026
de94603
fix: Invoke deferred cleanup after error handling (#3876)
janbuchar Jul 22, 2026
92a356f
test: fix storage pollution in browser crawler tests (#3861)
B4nan Jul 22, 2026
f83b6f0
refactor!: Remove unnecessary parameters to storage frontend construc…
janbuchar Jul 22, 2026
83003fb
chore: Exclude internal symbols from public API docs (#3882)
janbuchar Jul 23, 2026
5098248
fix: Fix ErrorHandler type (#3881)
janbuchar Jul 23, 2026
bf20a15
feat: use crawler id for `Statistics` and non-default `RequestQueue`s…
barjin Jul 23, 2026
219be89
refactor: Introduce IProxyConfiguration (#3892)
janbuchar Jul 24, 2026
a78527d
refactor!: Privatize unintentionally exposed internals (#3893)
janbuchar Jul 24, 2026
2b07f02
feat: Finalize AdaptivePlaywrightCrawler (#3907)
janbuchar Jul 24, 2026
aeee91d
fix: Run extendContext before navigation hooks (#3908)
janbuchar Jul 24, 2026
4dff929
docs: Fill in missing stuff in upgrading guide (#3914)
janbuchar Jul 27, 2026
ec5ff9b
refactor: Introduce OwnedOrInjected dependency tracking helper (#3891)
janbuchar Jul 27, 2026
b9e9d9d
chore: Hide ignored symbols in public API spec (#3915)
janbuchar Jul 27, 2026
39f626d
refactor: move debug helpers to `@crawlee/core` (#3916)
barjin Jul 29, 2026
2070c10
feat: infer router route map on the crawler & type request inputs (#3…
B4nan Jul 29, 2026
3471064
refactor!: unify `config` and `configuration` naming (#3926)
B4nan Jul 29, 2026
6be354a
refactor!: remove unused exports from `@crawlee/utils` (#3929)
barjin Jul 30, 2026
2e70010
refactor!: Split off ConcurrencySystem from AutoscaledPool (#3917)
janbuchar Jul 30, 2026
9bde1e6
refactor: Introduce IRenderingTypePredictor and unify the ownership p…
janbuchar Jul 31, 2026
fc2930f
refactor: drop re-exported types from `@crawlee/utils` (#3931)
barjin Jul 31, 2026
3ac7fc4
refactor: move buffer and stream helpers to `@crawlee/core` (#3928)
barjin Jul 31, 2026
83451f2
refactor!: Make AutoscaledPool internal (#3947)
janbuchar Aug 3, 2026
4631399
fix(core): log a warning on malformed cookie fragments in `mergeCooki…
barjin Aug 3, 2026
6bf2dc5
refactor: move async iterable helpers to `@crawlee/core` (#3930)
barjin Aug 3, 2026
9667ab4
fix(core): preserve subclass overrides when binding methods to a scop…
B4nan Aug 3, 2026
fb39222
fix(browser-crawler): persist session cookies after requestHandler (#…
Ayush7614 Aug 4, 2026
f01b620
feat: redesign request timeouts and allow per-route overrides (#3860)
B4nan Aug 5, 2026
f2f4397
refactor: Remove unnecessary retries when enqueueing new requests (#3…
janbuchar Aug 5, 2026
d53ab9b
chore: Improve public API snapshot generation (#3964)
janbuchar Aug 5, 2026
3adf3ae
feat: simplify `enqueueLinks` interface (#3533)
l2ysho Aug 5, 2026
7c90ca0
chore: Add missing upgrading guide entries (#3965)
janbuchar Aug 5, 2026
1e482e4
docs: Reorganize the v4 upgrading guide (#3967)
janbuchar Aug 5, 2026
7373844
refactor: drop `fs-extra` dependency (#3969)
barjin Aug 6, 2026
cdab1b1
perf: lazy-load `cheerio` in `playwright`/`puppeteer` `parseWithCheer…
barjin Aug 6, 2026
f2ef899
docs: document removal of gotoOptions argument from browser navigatio…
B4nan Aug 6, 2026
a99f894
docs: Update SessionPool section in upgrading guide (#3974)
janbuchar Aug 6, 2026
6a1ea15
test: fix flaky gotoOptions assertion in browser crawler test (#3973)
B4nan Aug 6, 2026
d915e53
refactor!: Accept a pre-configured Statistics instance in crawlers in…
janbuchar Aug 6, 2026
28e24b4
feat: add user data generics to browser navigation hooks (#3970)
B4nan Aug 6, 2026
bb6fc1d
feat!: Transactional storage access (#3953)
janbuchar Aug 6, 2026
80effc1
docs: Document LoadSignal customization for ConcurrencySystem (#3976)
janbuchar Aug 6, 2026
4d14072
refactor!: use native private class fields and remove underscore pref…
B4nan Aug 6, 2026
c596d23
fix: don't ship `idcac-playwright` as a hard dependency (#3977)
barjin Aug 7, 2026
dd7748d
fix: declare own `CookieJar` types in `@crawlee/types` (#3978)
barjin Aug 7, 2026
d95cf20
fix(browser-pool): make PlaywrightBrowser.browserType() return the br…
B4nan Aug 7, 2026
60c4b55
perf: lazy-load `file-type` and `csv-stringify` (#3989)
barjin Aug 7, 2026
6a94b86
feat(utils): split exports into public and /internal entry points (#3…
barjin Aug 7, 2026
7bca635
feat: implement per-domain request throttling (ThrottlingRequestManag…
harryautomazione Aug 10, 2026
e09444c
perf(utils): lazily import sax (#4001)
barjin Aug 10, 2026
ea5b8c2
perf(http-client): lazy-load tough-cookie (#4007)
barjin Aug 11, 2026
400fadf
chore: convert const enums to regular enums (#4008)
vladfrangu Aug 12, 2026
d48cd5b
chore: wire the `ignoreTlsErrors` crawler option (formerly `ignoreSsl…
B4nan Aug 12, 2026
bb2f267
chore: support RC releases in the publish workflow (#4009)
B4nan Aug 12, 2026
4dbce0a
fix: Fix purgeOnStart on aliased storages (#4013)
janbuchar Aug 12, 2026
0d88042
feat: Extend RecoverableState API to accommodate future adopters (#4006)
janbuchar Aug 12, 2026
f08e8d6
refactor: Use RecoverableState to implement Statistics persistence (#…
janbuchar Aug 12, 2026
ea9da4c
test: adapt E2E suite to v4 and fix regressions it caught (#4012)
B4nan Aug 12, 2026
dd6e3b9
chore!: unify argument validation to zod (#3935)
vladfrangu Aug 12, 2026
3ce18ea
chore(templates): keep camoufox-js ^0.11.0 paired with playwright 1.6…
B4nan Aug 12, 2026
bbd72d8
chore: restore @ts-ignore on robotsParser calls under nodenext resolu…
B4nan Aug 12, 2026
bd20a49
chore(utils): re-add tldts dependency for enqueue-strategy URL filtering
B4nan Aug 12, 2026
75412fc
chore(test): adapt master-carried tests to v4 APIs after rebase
B4nan Aug 12, 2026
375b08c
chore(test): finish adapting master-carried tests to v4 APIs
B4nan Aug 12, 2026
6bcd06e
feat: split `enqueueLinks` into `extractLinks` + `addRequests` (#4010)
barjin Aug 12, 2026
8e3194b
chore: Rename client load signal to storageBackend (#4023)
janbuchar Aug 12, 2026
d50177c
feat: Improve sameDomainDelaySecs implementation via ThrottlingReques…
janbuchar Aug 12, 2026
e00e8ce
feat: Accept custom extensions in the IStatistics interface (#3982)
janbuchar Aug 12, 2026
839a9b2
chore: re-apply master-side fixes over the enqueueLinks split (#4010)
B4nan Aug 12, 2026
39148e6
refactor!: replace `browserPoolOptions` with browser pool factories (…
janbuchar Aug 12, 2026
41a2584
refactor!: Rename BasicCrawler.stats to statistics (#4028)
janbuchar Aug 12, 2026
4bca1e6
chore(test): use crawler.statistics in the maxRequestsPerCrawl budget…
B4nan Aug 12, 2026
c324386
chore: fix formatting and regenerate public API reports after the rebase
B4nan Aug 12, 2026
5a7d0b9
chore(e2e): repair E2E suite for the enqueueLinks split and the zod c…
B4nan Aug 12, 2026
bcc97ff
chore(e2e): pair camoufox-js ^0.11.0 with playwright 1.60.0 in the cl…
B4nan Aug 12, 2026
af00b2a
docs: pin 3.18 API links in content carried over from master
B4nan Aug 12, 2026
7aea1f5
chore(e2e): keep 403 responses retryable in the camoufox-cloudflare f…
B4nan Aug 12, 2026
a7b2f1c
feat: Add Symbol.asyncDispose to disposable services for DX (#4032)
janbuchar Aug 14, 2026
ffd240d
docs: warn against --omit=optional in v4 upgrading guide (#4034)
barjin Aug 14, 2026
6188678
chore(e2e): retry the platform statistics lookup instead of crashing …
B4nan Aug 14, 2026
ceb936a
refactor!: Continue public API minimization (#4036)
janbuchar Aug 17, 2026
80d7010
chore: Add oxlint plugin to warn about TS-private class members (#4035)
janbuchar Aug 17, 2026
2766e87
docs: drop the 4.0 snapshot in favor of serving v4 docs as the next v…
B4nan Aug 18, 2026
a1512cf
chore: restore the typed schema-router overloads on createFileRouter
B4nan Aug 18, 2026
2137c70
chore(ci): point the release workflows at master for the v4 line
B4nan Aug 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion .github/workflows/check-pr-title.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ jobs:
name: 'Check PR title'
runs-on: ubuntu-22.04
steps:
- uses: apify/actions/pr-title-check@v1.4.0
- uses: apify/actions/pr-title-check@v1.4.1
13 changes: 2 additions & 11 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,21 +25,12 @@ jobs:
node-version: 24
package-manager-cache: false

- name: Enable corepack
run: |
corepack enable
corepack prepare yarn@stable --activate
- uses: apify/workflows/pnpm-install@main

- name: Build & deploy docs
run: |
# install project deps
yarn
# go to website dir
cd website
# install website deps
yarn
# build the docs
yarn build
pnpm build
env:
APIFY_SIGNING_TOKEN: ${{ secrets.APIFY_SIGNING_TOKEN }}
SEGMENT_TOKEN: ${{ secrets.SEGMENT_TOKEN }}
Expand Down
54 changes: 32 additions & 22 deletions .github/workflows/publish-to-npm.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ on:
required: true
type: string
dist-tag:
description: NPM dist-tag (prod or next)
description: NPM dist-tag (prod, next or rc)
required: false
type: string
default: 'prod'
Expand All @@ -19,7 +19,7 @@ on:
required: true
type: string
dist-tag:
description: NPM dist-tag (prod or next)
description: NPM dist-tag (prod, next or rc)
required: false
type: string
default: 'prod'
Expand Down Expand Up @@ -49,16 +49,6 @@ jobs:
node-version: 24
package-manager-cache: false

- name: Enable corepack
run: |
corepack enable
corepack prepare yarn@stable --activate

- name: Activate cache for Node.js 24
uses: actions/setup-node@v7
with:
cache: 'yarn'

- name: Turbo cache
id: turbo-cache
uses: actions/cache@v6
Expand All @@ -68,25 +58,24 @@ jobs:
restore-keys: |
turbo-${{ github.job }}-${{ github.ref_name }}-

- name: Install dependencies
run: yarn
- uses: apify/workflows/pnpm-install@main

- name: Build packages
run: yarn ci:build
run: pnpm ci:build

- name: Bump canary versions
if: inputs.dist-tag == 'next'
if: inputs.dist-tag != 'prod'
run: |
yarn turbo copy --force -- --canary --preid=beta
pnpm turbo copy --force -- --canary=major --preid=${{ inputs.dist-tag == 'rc' && 'rc' || 'beta' }}

- name: Commit changes
if: inputs.dist-tag == 'next'
if: inputs.dist-tag != 'prod'
uses: EndBug/add-and-commit@v11
id: commit
with:
author_name: Apify Release Bot
author_email: noreply@apify.com
message: 'chore: bump canary versions [skip ci]'
message: "chore: bump ${{ inputs.dist-tag == 'rc' && 'RC' || 'canary' }} versions [skip ci]"
push: false

- name: Publish to NPM (@latest)
Expand All @@ -96,17 +85,38 @@ jobs:
timeout_minutes: 10
max_attempts: 5
retry_wait_seconds: 30
command: git checkout . && yarn publish:prod --yes
command: git checkout . && pnpm publish:prod
env:
GH_TOKEN: ${{ secrets.APIFY_SERVICE_ACCOUNT_GITHUB_TOKEN }}

- name: Publish to NPM (@next)
if: inputs.dist-tag == 'next'
if: inputs.dist-tag != 'prod' && inputs.dist-tag != 'rc'
uses: nick-fields/retry@v4
with:
timeout_minutes: 10
max_attempts: 5
retry_wait_seconds: 30
command: git checkout . && yarn publish:next --yes
command: git checkout . && pnpm publish:next
env:
GH_TOKEN: ${{ secrets.APIFY_SERVICE_ACCOUNT_GITHUB_TOKEN }}

- name: Publish to NPM (@rc)
if: inputs.dist-tag == 'rc'
uses: nick-fields/retry@v4
with:
timeout_minutes: 10
max_attempts: 5
retry_wait_seconds: 30
command: git checkout . && pnpm publish:rc
env:
GH_TOKEN: ${{ secrets.APIFY_SERVICE_ACCOUNT_GITHUB_TOKEN }}

# RC releases produce no changelog and no commit on the branch, only a git tag
# pointing at the released commit (the version bump lives only on npm).
- name: Create git tag
if: inputs.dist-tag == 'rc'
run: |
VERSION=$(node -p "require('./packages/core/dist/package.json').version")
git reset --hard HEAD~1
git tag "v$VERSION"
git push origin "refs/tags/v$VERSION"
81 changes: 17 additions & 64 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
@@ -1,7 +1,6 @@
name: Release @latest

env:
YARN_IGNORE_NODE: 1
RETRY_TESTS: 1

on:
Expand Down Expand Up @@ -37,7 +36,7 @@ jobs:
matrix:
# We don't test on Windows as the tests are flaky
os: [ ubuntu-22.04 ]
node-version: [ 20, 22, 24 ]
node-version: [ 22, 24, 26 ]

runs-on: ${{ matrix.os }}

Expand All @@ -51,16 +50,6 @@ jobs:
node-version: ${{ matrix.node-version }}
package-manager-cache: false

- name: Enable corepack
run: |
corepack enable
corepack prepare yarn@stable --activate

- name: Activate cache for Node.js ${{ matrix.node-version }}
uses: actions/setup-node@v7
with:
cache: 'yarn'

- name: Turbo cache
id: turbo-cache
uses: actions/cache@v6
Expand All @@ -70,26 +59,16 @@ jobs:
restore-keys: |
turbo-${{ github.job }}-${{ github.ref_name }}-

- name: Install Dependencies
run: yarn

- name: Cache Playwright browsers
uses: actions/cache@v6
with:
path: ~/.cache/ms-playwright
key: playwright-${{ runner.os }}-${{ hashFiles('**/yarn.lock') }}
restore-keys: |
playwright-${{ runner.os }}-
- uses: apify/workflows/pnpm-install@main

- name: Install Playwright browsers
timeout-minutes: 10
run: yarn playwright install --with-deps
run: pnpm exec playwright install --with-deps

- name: Build
run: yarn ci:build
run: pnpm ci:build

- name: Tests
run: yarn test
run: pnpm test

release:
name: "Bump Crawlee: ${{ inputs.version }} version (${{ inputs.custom_version || 'n/a' }} custom version)"
Expand All @@ -110,16 +89,6 @@ jobs:
node-version: 24
package-manager-cache: false

- name: Enable corepack
run: |
corepack enable
corepack prepare yarn@stable --activate

- name: Activate cache for Node.js 24
uses: actions/setup-node@v7
with:
cache: 'yarn'

- name: Turbo cache
id: turbo-cache
uses: actions/cache@v6
Expand All @@ -129,15 +98,14 @@ jobs:
restore-keys: |
turbo-${{ github.job }}-${{ github.ref_name }}-

- name: Install Dependencies
run: yarn
- uses: apify/workflows/pnpm-install@main

- name: Build
run: yarn ci:build
run: pnpm ci:build

- name: Bump version to custom version
if: ${{ github.event.inputs.version == 'custom' && github.event.inputs.custom_version != '' }}
run: yarn lerna version ${{ github.event.inputs.custom_version }} --force-publish --yes
run: pnpm exec lerna version ${{ github.event.inputs.custom_version }} --force-publish --yes
env:
GH_TOKEN: ${{ secrets.APIFY_SERVICE_ACCOUNT_GITHUB_TOKEN }}
GIT_AUTHOR_NAME: Apify Release Bot
Expand All @@ -147,7 +115,7 @@ jobs:

- name: Bump version to ${{ github.event.inputs.version }} version
if: ${{ github.event.inputs.version != 'custom' }}
run: yarn lerna version ${{ github.event.inputs.version }} --force-publish --yes
run: pnpm exec lerna version ${{ github.event.inputs.version }} --force-publish --yes
env:
GH_TOKEN: ${{ secrets.APIFY_SERVICE_ACCOUNT_GITHUB_TOKEN }}
GIT_AUTHOR_NAME: Apify Release Bot
Expand All @@ -157,8 +125,8 @@ jobs:

- name: Pin versions in internal dependencies and update lockfile
run: |
yarn release:pin-versions
yarn install --no-immutable
pnpm release:pin-versions
pnpm install --no-frozen-lockfile

- name: Commit changes
id: commit
Expand All @@ -170,7 +138,7 @@ jobs:
pull: '--rebase --autostash'

- name: Publish packages
uses: apify/actions/execute-workflow@v1.4.0
uses: apify/actions/execute-workflow@v1.4.1
with:
workflow: publish-to-npm.yml
inputs: >
Expand All @@ -189,7 +157,8 @@ jobs:
version-docs:
needs: release
runs-on: ubuntu-22.04
if: (github.event.inputs.version == 'minor' || github.event.inputs.version == 'major')
# docs snapshots are only ever generated from master — maintenance branches must not trigger them
if: github.ref == 'refs/heads/master' && (github.event.inputs.version == 'minor' || github.event.inputs.version == 'major')

steps:
- name: Checkout repository
Expand All @@ -208,23 +177,7 @@ jobs:
- name: Install jq
run: sudo apt-get install jq

- name: Enable corepack
run: |
corepack enable
corepack prepare yarn@stable --activate

- name: Activate cache for Node.js 24
uses: actions/setup-node@v7
with:
cache: 'yarn'

- name: Install dependencies
run: |
# install project deps
yarn
# install website deps
cd website
yarn
- uses: apify/workflows/pnpm-install@main

- name: Snapshot the current version
run: |
Expand All @@ -236,8 +189,8 @@ jobs:
fi
echo "VERSION=$VERSION" >> "$GITHUB_ENV"
MAJOR_MINOR=$(echo $VERSION | cut -d. -f1,2)
yarn docusaurus docs:version $MAJOR_MINOR
yarn docusaurus api:version $MAJOR_MINOR
pnpm docusaurus docs:version $MAJOR_MINOR
pnpm docusaurus api:version $MAJOR_MINOR

- name: Commit and push the version snapshot
uses: EndBug/add-and-commit@v11
Expand Down
Loading
Loading