[pull] master from apify:master

docs/guides/request_loaders.mdx

@@ -24,7 +24,7 @@ Request loaders extend the functionality of the <ApiLink to="core/class/RequestQ
 The request loader abstractions are built around two interfaces and a couple of helpers:
 
 - <ApiLink to="core/interface/IRequestLoader">`IRequestLoader`</ApiLink>: The base interface for reading requests in a crawl.
-- <ApiLink to="core/interface/IRequestManager">`IRequestManager`</ApiLink>: Extends `IRequestLoader` with write capabilities (adding and reclaiming requests).
+- <ApiLink to="core/interface/IRequestManager">`IRequestManager`</ApiLink>: Extends `IRequestLoader` with write capabilities (adding and reclaiming requests), and with the pacing signals a crawler reports back.
 - <ApiLink to="core/class/RequestManagerTandem">`RequestManagerTandem`</ApiLink>: Combines a read-only `IRequestLoader` with a writable `IRequestManager`.
 - <ApiLink to="core/class/ThrottlingRequestManager">`ThrottlingRequestManager`</ApiLink>: Wraps a writable `IRequestManager` and paces requests per domain.
 
@@ -55,8 +55,7 @@ class IRequestLoader {
     + getHandledCount()
     + fetchNextRequest()
     + markRequestAsHandled()
-    + isEmpty()
-    + isFinished()
+    + checkReadiness()
     + toTandem()
 }
 
@@ -65,6 +64,7 @@ class IRequestManager {
     + addRequest()
     + addRequestsBatched()
     + reclaimRequest()
+    + recordPacingSignal()
     + purge()
 }
 
@@ -102,7 +102,7 @@ A crawler reads its requests from a single <ApiLink to="core/interface/IRequestM
 
 ## Request loaders
 
-The <ApiLink to="core/interface/IRequestLoader">`IRequestLoader`</ApiLink> interface defines the foundation for fetching requests during a crawl. It provides methods for basic operations like retrieving the next request, marking requests as handled, and checking whether the loader is empty or finished. It is intentionally **read-only** — it does not allow adding new requests. Concrete implementations such as <ApiLink to="core/class/RequestList">`RequestList`</ApiLink> build on this interface to handle specific scenarios. You can create your own custom loader that reads from an external file, web endpoint, database, or any other data source.
+The <ApiLink to="core/interface/IRequestLoader">`IRequestLoader`</ApiLink> interface defines the foundation for fetching requests during a crawl. It provides methods for basic operations like retrieving the next request, marking requests as handled, and reporting whether the loader has a request ready, is waiting on one, or is done. It is intentionally **read-only** — it does not allow adding new requests. Concrete implementations such as <ApiLink to="core/class/RequestList">`RequestList`</ApiLink> build on this interface to handle specific scenarios. You can create your own custom loader that reads from an external file, web endpoint, database, or any other data source.
 
 ### Request list
 
@@ -134,18 +134,27 @@ The loader supports filtering URLs using glob patterns and regular expressions,
 
 The <ApiLink to="core/interface/IRequestManager">`IRequestManager`</ApiLink> interface extends `IRequestLoader` with **write** capabilities. In addition to reading requests, a request manager can add new requests and reclaim failed ones. This is essential for dynamic crawling, where new URLs emerge during the crawl, or when requests fail and need to be retried. The <ApiLink to="core/class/RequestQueue">`RequestQueue`</ApiLink> is the primary built-in request manager — see the [Request storage](./request-storage) guide for details.
 
+### Pacing signals
+
+A manager decides *when* each request goes out, so it is also where a crawler reports back what a site said about the pace it wants to be crawled at. That arrives through one method, <ApiLink to="core/interface/IRequestManager#recordPacingSignal">`recordPacingSignal()`</ApiLink>: a source refused a request because we were going too fast (`reason: 'rateLimited'`, optionally carrying the wait it asked for), or a source declared a standing floor on how often it may be requested (`reason: 'minInterval'`, carrying that interval). Nothing in the payload names HTTP status codes, response headers or robots.txt — where a signal came from is the crawler's business, not the manager's.
+
+Every signal also carries a `scope`: how much of the URL space it covers. `'hostname'` and `'registrableDomain'` are what Crawlee's own reporters send, and the type suggests them, but it accepts any string — so a pacer keyed on something other than a host, an account or an API key say, can be reported to in its own vocabulary. A manager may apply a signal to a **wider** scope than it was given, since a floor that holds for one host still holds when a whole site is paced by it, but never to a narrower one, which would leave part of what the signal covers running unpaced.
+
+A third `reason`, `minIntervalEverywhere`, is a floor under the pace of **every** domain the manager dispatches to, declared by whoever owns the crawl rather than by a source — which is why it is the one variant carrying no `url`. It is how a crawler offers a manager its <ApiLink to="basic-crawler/interface/BasicCrawlerOptions#sameDomainDelaySecs">`sameDomainDelaySecs`</ApiLink>: whatever paces takes the floor, and only when nothing does the crawler add a pacer of its own.
+
+Two things follow for implementors. The method is required, so reporting is never a question of support: a manager that does not pace — a plain queue — returns `false` and the crawler warns that the signal was dropped, while one that **wraps** another forwards it, as <ApiLink to="core/class/RequestManagerTandem">`RequestManagerTandem`</ApiLink> does, or a nested pacer goes deaf. And forwarding needs no knowledge of the payload, which is why this is one method taking a value — and why what a signal applies to travels inside that value.
+
 ## Per-domain throttling
 
 Some sites answer bursts of traffic with HTTP 429 (Too Many Requests) rather than an outright block. By default a 429 is treated as a blocked session: the session is retired and the request is retried straight away on a fresh one, which churns through proxies without actually slowing down.
 
-The <ApiLink to="core/class/ThrottlingRequestManager">`ThrottlingRequestManager`</ApiLink> handles it at the scheduling layer instead. Wrap your request manager in it and list the domains you want paced:
+The <ApiLink to="core/class/ThrottlingRequestManager">`ThrottlingRequestManager`</ApiLink> handles it at the scheduling layer instead. List the domains you want paced:
 
 ```ts
-import { CheerioCrawler, RequestQueue, ThrottlingRequestManager } from 'crawlee';
+import { CheerioCrawler, ThrottlingRequestManager } from 'crawlee';
 
 const crawler = new CheerioCrawler({
     requestManager: new ThrottlingRequestManager({
-        inner: await RequestQueue.open(),
         domains: ['api.example.com'],
         // optional, these are the defaults
         baseDelaySecs: 2,
@@ -158,11 +167,15 @@ const crawler = new CheerioCrawler({
 });
 ```
 
+Requests for domains it does not pace go to the default request queue, opened on first use. Pass `inner` to wrap a manager of your own instead — a queue you opened, or a [tandem](#request-manager-tandem) over a `requestList`.
+
+The pacer works wherever you put it: pass it to the crawler directly as above, or nest it inside a [tandem](#request-manager-tandem) as the writable side of a loader — the tandem forwards the crawler's [pacing signals](#pacing-signals) to the manager it wraps, so listed domains are paced either way.
+
 Requests for a listed domain are routed into their own queue as they are added. When one of those domains answers with a 429, the crawler honours its `Retry-After` header — or backs off exponentially from `baseDelaySecs` up to `maxDelaySecs` if there is none — and holds that domain's requests back for the duration. Requests for every other domain keep flowing at full speed, the throttled request is retried later without counting against `maxRequestRetries`, and its session is left alone, because a rate limit says nothing about the session.
 
 Because a throttled request costs no retries, a domain that never stops rate-limiting would otherwise keep the crawl alive forever. If one goes `maxDomainStallSecs` without letting a single request through, the crawl shuts down with a `PersistentRateLimitError` — at that point the concurrency is too high for that domain, or it has blocked you outright, and waiting longer will not help. Its requests are left in their queue on purpose, so re-running the crawl with `purgeOnStart` disabled resumes them if the rate limit lifts. A crawler running with `keepAlive` is exempt, since staying up regardless is what it was asked to do.
 
-Matching is exact and case-insensitive, with no wildcard support, so list each subdomain you care about — or set `throttleBy: 'registrableDomain'`, which groups a site and all of its subdomains under a single set of clocks.
+Matching is exact and case-insensitive, with no wildcard support, so list each subdomain you care about — or set `throttleBy: 'registrableDomain'`, which groups a site and all of its subdomains under a single set of clocks. That grouping is also the finest granularity this manager can pace at, since holding a domain back means holding its queue back: a [pacing signal](#pacing-signals) scoped more narrowly is applied to the whole group, and one scoped more widely throws rather than being quietly under-applied.
 
 ### The two clocks
 
@@ -178,7 +191,6 @@ Every throttled domain runs two of them, and is dispatched to once both have run
 ```ts
 const crawler = new CheerioCrawler({
     requestManager: new ThrottlingRequestManager({
-        inner: await RequestQueue.open(),
         domains: 'all',
         minCrawlDelaySecs: 1,
         throttleBy: 'registrableDomain',
@@ -189,7 +201,7 @@ const crawler = new CheerioCrawler({
 });
 ```
 
-This is what the crawler's own <ApiLink to="basic-crawler/interface/BasicCrawlerOptions#sameDomainDelaySecs">`sameDomainDelaySecs`</ApiLink> option is built on — it wraps the crawler's request manager in a `ThrottlingRequestManager` configured exactly like the above. Dropping `minCrawlDelaySecs` is just as useful: 429 backoff and robots.txt `Crawl-delay` then apply to every domain, with no pacing of your own on top.
+This is what the crawler's own <ApiLink to="basic-crawler/interface/BasicCrawlerOptions#sameDomainDelaySecs">`sameDomainDelaySecs`</ApiLink> option is built on — with no manager of your own to report the floor to, it wraps the crawler's request manager in a `ThrottlingRequestManager` configured exactly like the above. Pass a manager like this one yourself and it takes the floor instead, so your configuration keeps the crawl to one clock per domain. Dropping `minCrawlDelaySecs` is just as useful: 429 backoff and robots.txt `Crawl-delay` then apply to every domain, with no pacing of your own on top.
 
 A queue per domain is not free, so a run may only throttle `maxThrottledDomains` of them (100 by default) before it throws. If you are crawling more domains than that, pace the crawl with `maxRequestsPerMinute` instead. The list of domains discovered so far is kept in the default key-value store, under `persistStateKey`, so that a restart reopens their queues rather than leaving whatever they still hold uncrawled.