Crawl page is very slow.

[github0null]

Hello !

I deployed this project by docker compose. this is my config:

services:
  crawl4ai:
    image: unclecode/crawl4ai:0.9.2
    container_name: crawl4ai
    ports:
      - "11235:11235"
    environment:
      CRAWL4AI_API_TOKEN: sk-xxxxxx
    shm_size: "2gb"
    user: "appuser"
    init: true
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    pids_limit: 512
    read_only: true
    tmpfs:
      - /tmp
      - /var/lib/redis:uid=999,gid=999,mode=0700
      - /var/lib/crawl4ai/outputs:uid=999,gid=999,mode=0700
      - /home/appuser/.crawl4ai:uid=999,gid=999,mode=0700
      - /home/appuser/.cache/url_seeder:uid=999,gid=999,mode=0700
      - /home/appuser/.gunicorn:uid=999,gid=999,mode=0700
    volumes:
      - ./override/config.yml:/app/config.yml:ro
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:11235/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

config.yml

# Application Configuration
app:
  title: "Crawl4AI API"
  version: "1.0.0"
  # Loopback by default. Exposing on 0.0.0.0 requires a credential
  # (CRAWL4AI_API_TOKEN); the startup auth guard refuses an open non-loopback
  # bind without one. Put a TLS-terminating reverse proxy in front for exposure.
  host: "127.0.0.1"
  port: 11235
  reload: False
  workers: 1
  timeout_keep_alive: 300

# Default LLM Configuration
llm:
  provider: "openai/gpt-4o-mini"
  # api_key: sk-...  # If you pass the API key directly (not recommended)

# Redis Configuration
# Set task_ttl_seconds to automatically expire task data in Redis.
# This prevents unbounded memory growth from accumulated task results.
# Default: 3600 (1 hour). Set to 0 to disable TTL (not recommended).
# Can be overridden with REDIS_TASK_TTL environment variable.
redis:
  host: "localhost"
  port: 6379
  db: 0
  password: ""
  task_ttl_seconds: 3600  # TTL for task data (1 hour default)
  ssl: False
  ssl_cert_reqs: None
  ssl_ca_certs: None
  ssl_certfile: None
  ssl_keyfile: None

# Resource governance (DoS protection). Any limit set to 0 means "unbounded"
# (the pre-hardening behavior), so you can relax or disable each one.
limits:
  max_body_bytes: 10485760  # 10 MiB request body cap (413 if exceeded); 0 = unbounded
  max_pages: 100            # deep-crawl page budget clamp (defense in depth)
  max_depth: 5              # deep-crawl depth clamp
  wall_clock_s: 0           # per-crawl deadline in seconds (504 on timeout); 0 = no deadline
  queue:                    # background job queue for /crawl/job and /llm/job
    maxsize: 1000           # max queued jobs (503 when full); 0 = unbounded
    workers: 4              # concurrent background workers
    per_principal: 0        # max concurrent jobs per caller (429 over cap); 0 = unlimited

# Rate Limiting Configuration
rate_limiting:
  enabled: false
  default_limit: "1000/minute"
  trusted_proxies: []
  storage_uri: "memory://"  # Use "redis://localhost:6379" for production

# Security Configuration
# WARNING: For production deployments, enable security and use proper SECRET_KEY:
#   - Set jwt_enabled: true for authentication
#   - Set SECRET_KEY environment variable to a secure random value
#   - Set CRAWL4AI_HOOKS_ENABLED=true only if you need hooks (RCE risk)
security:
  enabled: true
  jwt_enabled: false
  api_token: ""  # When set, /token endpoint requires this secret to issue JWTs
  https_redirect: false
  trusted_hosts: ["*"]
  cors_allow_origins: []  # deny-by-default; list explicit origins to allow CORS
  headers:
    x_content_type_options: "nosniff"
    x_frame_options: "DENY"
    content_security_policy: "default-src 'self'"
    strict_transport_security: "max-age=63072000; includeSubDomains"

# Crawler Configuration
crawler:
  base_config:
    simulate_user: true
  memory_threshold_percent: 95.0
  rate_limiter:
    enabled: false
    base_delay: [1.0, 2.0]
  timeouts:
    stream_init: 30.0  # Timeout for stream initialization
    batch_process: 300.0  # Timeout for batch processing
  pool:
    max_pages: 40                          # ← GLOBAL_SEM permits
    idle_ttl_sec: 86400                    # ← 30 min janitor cutoff
  browser:
    kwargs:
      headless: true
      text_mode: true
    extra_args:
      # - "--single-process"
      # SECURITY: --no-sandbox disables the Chromium renderer sandbox (a renderer
      # exploit can then reach the container). It is required only because the
      # container runs as non-root without a usable sandbox. To remove it, run
      # the container with an unprivileged user namespace
      # (host: unprivileged_userns_clone=1) OR a verified seccomp profile
      # (compose: security_opt: seccomp=...), then delete this flag and confirm
      # Chromium still starts. Tracked as a build-gated hardening item.
      - "--no-sandbox"
      - "--disable-dev-shm-usage"
      - "--disable-gpu"
      - "--disable-software-rasterizer"
      # --disable-web-security removed for security (enables cross-origin access)
      # --allow-insecure-localhost / --ignore-certificate-errors removed: they
      # disabled TLS verification for every crawl (MITM / SSRF-to-internal-TLS).
      # Set CRAWL4AI_ALLOW_INSECURE_TLS=true only for trusted internal testing.

# Logging Configuration
logging:
  level: "INFO"
  format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"

# Observability Configuration
observability:
  prometheus:
    enabled: True
    endpoint: "/metrics"
  health_check:
    endpoint: "/health"

# Webhook Configuration
webhooks:
  enabled: true
  default_url: null  # Optional: default webhook URL for all jobs
  data_in_payload: false  # Optional: default behavior for including data
  retry:
    max_attempts: 5
    initial_delay_ms: 1000  # 1s, 2s, 4s, 8s, 16s exponential backoff
    max_delay_ms: 32000
    timeout_ms: 30000  # 30s timeout per webhook call
  headers:  # Optional: default headers to include
    User-Agent: "Crawl4AI-Webhook/1.0"

I think my computer's performance is strong enough,

But later, I found that when crawling the same website, the speed varies; sometimes it's fast, and sometimes it's quite slow.

crawl4ai  | 2026-08-03 08:02:39,475 - crawler_pool - INFO - ♨️  Using hot pool browser (sig=5134d196, active=1)
crawl4ai  | [SCRAPE].. ◆ Failed to remove overlay elements: Page.evaluate: Execution context was destroyed, most likely because of a navigation. 
crawl4ai  | [FETCH]... ↓ https://raw.githubusercontent.com/apache/nuttx/master/sched/pthread/pthread_create.c                 | ✓ | ⏱: 30.82s 
crawl4ai  | [SCRAPE].. ◆ https://raw.githubusercontent.com/apache/nuttx/master/sched/pthread/pthread_create.c                 | ✓ | ⏱: 0.01s 
crawl4ai  | [COMPLETE] ● https://raw.githubusercontent.com/apache/nuttx/master/sched/pthread/pthread_create.c                 | ✓ | ⏱: 30.84s 
crawl4ai  | 2026-08-03 08:03:10,322 - api - INFO - Memory usage: Start: 260.1953125 MB, End: 260.1953125 MB, Delta: 0.0 MB, Peak: 260.1953125 MB
crawl4ai  | 2026-08-03 08:19:02,054 - crawler_pool - INFO - ♨️  Using hot pool browser (sig=5134d196, active=1)
crawl4ai  | [SCRAPE].. ◆ Failed to remove overlay elements: Page.evaluate: Execution context was destroyed, most likely because of a navigation. 
crawl4ai  | [FETCH]... ↓ https://raw.githubusercontent.com/apache/nuttx/master/include/malloc.h                               | ✓ | ⏱: 30.96s 
crawl4ai  | [SCRAPE].. ◆ https://raw.githubusercontent.com/apache/nuttx/master/include/malloc.h                               | ✓ | ⏱: 0.00s 
crawl4ai  | [COMPLETE] ● https://raw.githubusercontent.com/apache/nuttx/master/include/malloc.h                               | ✓ | ⏱: 30.97s 
crawl4ai  | 2026-08-03 08:19:33,022 - api - INFO - Memory usage: Start: 260.1953125 MB, End: 260.1953125 MB, Delta: 0.0 MB, Peak: 260.1953125 MB

When I used curl, it was completed almost immediately, so I don't think it's a network issue.

I don't know where I went wrong. This is my crawling configuration.

"urls": urls,
                        "crawler_config": {
                            "type": "CrawlerRunConfig",
                            "params": {
                                "excluded_tags": [
                                    "script",
                                    "style",
                                    "noscript",
                                    "nav",
                                    "header",
                                    "footer",
                                    "aside",
                                    "form",
                                    "iframe",
                                    "svg",
                                    "canvas"
                                ],
                                "excluded_selector": ".ad, .ads, .sidebar, .share, .comments, [id*=\"header\"], [id*=\"footer\"], [class*=\"header\"], [class*=\"footer\"]",
                                "remove_overlay_elements": True,
                                "remove_forms": True,
                                "exclude_social_media_links": True,
                                "exclude_external_links": True,
                                "exclude_external_images": True,
                                "process_iframes": False,
                                "only_text": True,
                                "word_count_threshold": 10,
                                "page_timeout": 10000
                            }
                        },
                        "browser_config": {
                            "type": "BrowserConfig",
                            "params": {
                                "enable_stealth": True
                            }
                        }

Any help would be greatly appreciated.

[SohamKukreti]

Hi @github0null thanks for the detailed report, your deployment is fine. Your compose file, hardware, and network are all fine. This is a bug in crawl4ai itself, and we were able to reproduce it exactly (same ~31s timing, same warning) outside Docker.

The culprit is this line in your crawler config:

"remove_overlay_elements": True

When that option is on, crawl4ai injects a small script into the page to close popups/cookie banners. That script finishes by waiting 50ms via setTimeout.

However, raw.githubusercontent.com serves files with this header:

Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'; sandbox

The CSP sandbox directive makes Chromium treat the page as fully sandboxed with scripting disabled — so setTimeout callbacks never fire. The injected script waits for a timer that never rings, Chromium tears down the JS context after ~30 seconds, and you get exactly the log line you're seeing:

[SCRAPE].. ◆ Failed to remove overlay elements: Page.evaluate: Execution context was destroyed, most likely because of a navigation.

Workaround (immediate)

Remove "remove_overlay_elements": True from your config. For raw source files it does nothing anyway (there are no popups on a plain-text page). We verified with your exact config, flipping only that flag reduced the crawling time to less than 1s with byte identical markdown.

On our side

This is a genuine bug: An innocuous, documented option shouldn't silently cost a flat 30s per page. We'll work on a fix for this soon.

Thanks again for the report!

[github0null]

Thank you for your answer.

[ntohidi]

Hi @github0null , quick update: the fix for this is now merged into the develop branch (PR #2139) and will ship in the next release.

Short version of what changed: the injected overlay/consent-removal scripts had setTimeout waits that never fired on pages with a CSP sandbox header. Those waits are now either moved to the Python side (where page CSP can't block them) or only run when a consent button was actually clicked. Crawl time on raw.githubusercontent.com went from ~31s to ~1.3s, with no change in output on normal pages.

Until the release is out, the workaround still stands: leave remove_overlay_elements off for plain-text pages. Or you can pull from develop if you want the fix now.

Thanks again for the detailed report, it made this easy to track down.