Surprisingly difficult Infinite scroll problem

[enoc2222]

Hi All,
I recently found crawl4ai while trying to do some simple scraping on a classified site(https://classifieds.ksl.com).
I am new to scraping and figured this would be a good site to learn/practice some basics on scraping data.
After trying selenium and Requests/Beautifulsoup approach with no luck (apparently they've implemented some decent scraping/bot blockers), I saw a suggestion on reddit to try crawl4ai.
I was able to get a basic script up and kind of running pretty quick and got some initial data but I am hitting a wall trying to get the complete data set from the search.
The classified search result page loads/function like a classic infinite scroll, adding more listings as you scroll down.
With my script, I can usually get the first 11-20 items, sometimes up to 34 (once got 80), but can't ever get the full search results(which as of writing is 254 for the url in the code below).
I've tried virtual scroll but could only ever get the first 11 using the selector ".grid". I've also tried scroll_delay with a range of 0.5 to 10 seconds with no noticeable difference.

I've been banging my head on this till the wee hours and figured wiser minds may have better insight if this is a simple fix or a deeper dive.

Here's my current best working code:

import json
import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, VirtualScrollConfig
from crawl4ai import JsonXPathExtractionStrategy
from crawl4ai.async_configs import CacheMode

async def extract_crypto_prices_xpath():

	schema = {
		"name": "Crypto Prices via XPath",
		#"baseSelector": "//a[contains(@role, 'listitem')]",#Alternate selector
		"baseSelector": "//a[contains(@class, 'grid-flow-col')]",
		"fields": [
			{
				"name": "title",
				"selector": ".//div[contains(@class, 'line-clamp-2')]",
				"type": "text"
			}
		]
	}
	config = CrawlerRunConfig(
		extraction_strategy=JsonXPathExtractionStrategy(schema, verbose=True),
		scan_full_page=True,
	)

	raw_url = "https://classifieds.ksl.com/v2/search/keyword/bin"

	async with AsyncWebCrawler(verbose=False) as crawler:
		result = await crawler.arun(
			url=raw_url,
			config=config,
			magic = True,
		)

		if not result.success:
			print("Crawl failed:", result.error_message)
			return

		data = json.loads(result.extracted_content)

		print(f"Extracted {len(data)} rows")
		for d in data:
			print(f'Title: {d}')

asyncio.run(extract_crypto_prices_xpath())

Let me know if you have any thoughts/suggestions
Thanks!

[hafezparast]

The issue is that scan_full_page=True scrolls the page, but infinite scroll sites like KSL Classifieds load new items via API calls triggered by scroll position — and the default scroll speed is often too fast for the content to load before the next scroll happens.

A few things to try:

1. Use js_code with manual scroll + wait loop

This gives you full control over the scroll behavior:

config = CrawlerRunConfig(
    extraction_strategy=JsonXPathExtractionStrategy(schema, verbose=True),
    js_code="""
        // Scroll to bottom repeatedly with pauses for content to load
        const delay = ms => new Promise(r => setTimeout(r, ms));
        let lastHeight = 0;
        let attempts = 0;
        while (attempts < 50) {
            window.scrollTo(0, document.body.scrollHeight);
            await delay(2000);  // Wait 2s for new items to load
            let newHeight = document.body.scrollHeight;
            if (newHeight === lastHeight) {
                attempts++;
                if (attempts >= 3) break;  // No new content after 3 tries
            } else {
                attempts = 0;
                lastHeight = newHeight;
            }
        }
    """,
    wait_for="css:.grid-flow-col:nth-child(100)",  # Wait until at least 100 items exist
    wait_for_timeout=120000,  # 2 min timeout
)

2. Use scan_full_page with a longer scroll_delay

The default scroll_delay is 0.2s — way too fast for API-loaded content. Try 2-3 seconds:

config = CrawlerRunConfig(
    extraction_strategy=JsonXPathExtractionStrategy(schema, verbose=True),
    scan_full_page=True,
    scroll_delay=3.0,  # Wait 3s between scrolls
    delay_before_return_html=5.0,  # Extra wait after all scrolling
)

3. Use VirtualScrollConfig with the right container

For virtualized scrolling (where the DOM recycles elements), you need to identify the scrollable container. On KSL it looks like the main page body:

from crawl4ai import VirtualScrollConfig

config = CrawlerRunConfig(
    extraction_strategy=JsonXPathExtractionStrategy(schema, verbose=True),
    virtual_scroll=VirtualScrollConfig(
        container_selector="body",
        scroll_count=50,
        scroll_by="page_height",
        wait_after_scroll=2.0,
    ),
)

Why results vary (11, 20, 34, 80...)

This is a timing issue. The site's API takes variable time to respond depending on server load. Sometimes the scroll outruns the API, sometimes it doesn't. The js_code approach with explicit height-change detection is the most reliable because it waits for actual content to appear rather than using fixed delays.

Let us know which approach works best for your case.

[nanridao2026]

I encountered the same problem.When I simulated the pull-down action using Crawl4AI version 0.9.1, I initially did not observe the pull-down behavior in the viewport. Even using the newer method, undetected_adapter = UndetectedAdapter(), did not work.I tried the three methods you mentioned, but none of them seemed to work.@hafezparast
The test website is here