[Bug]: Disallow: /*? in robots.txt blocks the whole website.

[Nalhin]

crawl4ai version

0.9.0

Expected Behavior

The rule in robots.txt disallows scraping urls with query params, the rest of the page works as expected.

Current Behavior

crawl4ai 0.9.x patches the stdlib urllib.robotparser to support wildcards, but it still matches rules against normalized paths (paths where the query string has already been stripped). Ecommerce sites commonly ship (this is advised by multiple SEO blogs online):

Disallow: /*?

The intent is to block only URLs carrying a query string, e.g. /?s=search. After normalization the ? is gone, so the rule collapses to Disallow: /* and the entire site reads as disallowed.

Is this reproducible?

Yes

Inputs Causing the Bug

- Disallow: /*? in robots.txt

When check_robots_txt=True

Steps to Reproduce

- Scrape any website (check_robots_txt=True) with Disallow: /*? in robots.txt (test webserver is sufficient in this case)

Code snippets

OS

macOS

Python version

3.9.10

Browser

Chrome

Browser version

No response

Error logs & Screenshots (if applicable)

No response