product-scraper#2
Conversation
…etplaces with language-specific headers
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (8)
WalkthroughDie PR erweitert den HTTPX-Fetch-Pfad um adapterbasierte Header und Warmup-URLs, ergänzt neue und angepasste Shop-Adapter, aktualisiert das Dashboard um Alert- und Target-Price-Ansichten und passt mehrere Repo-Hygiene-Dateien an. ChangesShop-Fetch und Adapter-Erweiterungen
Dashboard-Ansichten
API-Key-Routen
Workflow-, Seed- und Migrationsanpassungen
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant check_price
participant BaseAdapter
participant _fetch_httpx
participant httpx.AsyncClient
check_price->>BaseAdapter: fetch_headers(link_url)
BaseAdapter-->>check_price: extra_headers oder None
check_price->>BaseAdapter: warmup_url(link_url)
BaseAdapter-->>check_price: warmup_url oder None
check_price->>_fetch_httpx: URL, extra_headers, warmup_url
_fetch_httpx->>httpx.AsyncClient: optionaler Warmup-GET
_fetch_httpx->>httpx.AsyncClient: eigentlicher Fetch mit gemergten Headern
httpx.AsyncClient-->>_fetch_httpx: HTML
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 inconclusive)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (2)
app/shop_adapters/_amazon.py (2)
52-56: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winCAPTCHA-Erkennung ist eng gefasst.
Es wird nur nach
form[action*='validateCaptcha']gesucht. Andere Amazon-Blockseiten-Varianten (z. B. abweichendes Markup ohne dieses Formular) würden nicht alsblockederkannt, sondern fallen in den generischen "Price element not found"-Error-Pfad. Ein sicherer Fallback existiert bereits, daher nur eine Anregung zur Robustheit — z. B. zusätzlich auf typische Captcha-Texte oder den URL-Pfad/errors/validateCaptchaprüfen.♻️ Möglicher robusterer Check
- if tree.css_first("form[action*='validateCaptcha']"): + if tree.css_first("form[action*='validateCaptcha']") or "/errors/validateCaptcha" in url: return AdapterResult( status="blocked", error_message="Amazon returned a CAPTCHA challenge page.", )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@app/shop_adapters/_amazon.py` around lines 52 - 56, The CAPTCHA detection in the Amazon adapter is too narrow because `AmazonAdapter` only checks for `form[action*='validateCaptcha']`, so other block pages can slip through to the generic missing-price error path. Update the blocked-page check in the adapter’s parsing flow to also recognize common Amazon CAPTCHA indicators such as typical CAPTCHA text and the `/errors/validateCaptcha` URL pattern, while keeping the existing fallback to `blocked` when these signals are present.
49-95: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftFehlende Testabdeckung für
extract().Für die zentrale Extraktionslogik (success/error/blocked-Pfade, Preis-Parsing-Fehler, fehlende Elemente) existieren keine Tests. Da dies der kritische Pfad des neuen Adapters ist, wäre eine Unit-Test-Abdeckung sinnvoll.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@app/shop_adapters/_amazon.py` around lines 49 - 95, The Amazon adapter’s central extract() flow currently has no unit tests covering the blocked, error, and success paths. Add focused tests for _amazon.py’s extract method to verify the CAPTCHA branch, missing price element behavior, parse_price failure handling, and a successful extraction with title, price, and availability. Use the existing AdapterResult outputs and the extract/parse_price symbols to keep the tests targeted and resilient to HTML changes.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@app/shop_adapters/_amazon.py`:
- Around line 52-56: The CAPTCHA detection in the Amazon adapter is too narrow
because `AmazonAdapter` only checks for `form[action*='validateCaptcha']`, so
other block pages can slip through to the generic missing-price error path.
Update the blocked-page check in the adapter’s parsing flow to also recognize
common Amazon CAPTCHA indicators such as typical CAPTCHA text and the
`/errors/validateCaptcha` URL pattern, while keeping the existing fallback to
`blocked` when these signals are present.
- Around line 49-95: The Amazon adapter’s central extract() flow currently has
no unit tests covering the blocked, error, and success paths. Add focused tests
for _amazon.py’s extract method to verify the CAPTCHA branch, missing price
element behavior, parse_price failure handling, and a successful extraction with
title, price, and availability. Use the existing AdapterResult outputs and the
extract/parse_price symbols to keep the tests targeted and resilient to HTML
changes.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 2c416d2d-b700-4031-9899-059b65c86215
📒 Files selected for processing (4)
app/price_check_service.pyapp/shop_adapters/_amazon.pyapp/shop_adapters/base.pyapp/shop_adapters/registry.py
…ndling, and clean up price parsing logic
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
app/shop_adapters/_ebay.py (1)
15-51: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winDomain-Matching in
fetch_headers/warmup_urlignoriert das neue Suffix-Fallback ausregistry.py.
get_adapter()inregistry.pyliefert diesen Adapter jetzt auch für Subdomains eines registrierten eBay-Hosts (z. B.cgi.ebay.com,m.ebay.de) über die neue Suffix-Logik.fetch_headers/warmup_urlmachen aber einen exakten Lookup auf_ACCEPT_LANGUAGE.get(domain)mit dem vollen Hostnamen (nurwww.wird entfernt) — für eine solche Subdomain gäbe es dort keinen Treffer, wodurchAccept-Languageund Warmup-Request stillschweigend entfallen, obwohl der Adapter aktiv für diese URL ausgewählt wurde.Empfehlung: Domain-Auflösung analog zum Suffix-Matching in
registry.get_adapterdurchführen (z. B. gemeinsame Hilfsfunktion), statt eines reinen exakten Dict-Lookups.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@app/shop_adapters/_ebay.py` around lines 15 - 51, `EbayAdapter.fetch_headers` and `EbayAdapter.warmup_url` currently do an exact `_ACCEPT_LANGUAGE` lookup after only stripping `www.`, so URLs matched via the new suffix logic in `registry.get_adapter` (for example subdomains like `cgi.ebay.com` or `m.ebay.de`) lose both the `Accept-Language` header and warmup request. Update domain resolution in `EbayAdapter` to mirror `registry.get_adapter`’s suffix matching, ideally via a shared helper, so the adapter can derive the registered eBay host from subdomains before looking up `_ACCEPT_LANGUAGE` and building the homepage warmup URL.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@app/templates/dashboard/index.html`:
- Line 110: The hover state uses a non-standard Tailwind color token, so replace
the `hover:bg-gray-750` usage in the dashboard template with an existing
Tailwind gray shade that is already available. Update the class list where the
button/card styling is defined so the hover background applies correctly without
requiring a custom palette extension.
---
Nitpick comments:
In `@app/shop_adapters/_ebay.py`:
- Around line 15-51: `EbayAdapter.fetch_headers` and `EbayAdapter.warmup_url`
currently do an exact `_ACCEPT_LANGUAGE` lookup after only stripping `www.`, so
URLs matched via the new suffix logic in `registry.get_adapter` (for example
subdomains like `cgi.ebay.com` or `m.ebay.de`) lose both the `Accept-Language`
header and warmup request. Update domain resolution in `EbayAdapter` to mirror
`registry.get_adapter`’s suffix matching, ideally via a shared helper, so the
adapter can derive the registered eBay host from subdomains before looking up
`_ACCEPT_LANGUAGE` and building the homepage warmup URL.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 5997baba-4e70-468b-9dd4-ed014c5e15c3
📒 Files selected for processing (15)
CLAUDE.mdREADME.mdapp/price_check_service.pyapp/routes/shop_rules.pyapp/shop_adapters/_3djake.pyapp/shop_adapters/_amazon.pyapp/shop_adapters/_anycubic.pyapp/shop_adapters/_ebay.pyapp/shop_adapters/_elegoo.pyapp/shop_adapters/_esun.pyapp/shop_adapters/_prusa.pyapp/shop_adapters/base.pyapp/shop_adapters/registry.pyapp/templates/dashboard/index.htmlmigrations/versions/20260702_1108_033d388a81e3_rename_spool_status_enum_members_to_.py
💤 Files with no reviewable changes (3)
- app/shop_adapters/_esun.py
- app/shop_adapters/_amazon.py
- migrations/versions/20260702_1108_033d388a81e3_rename_spool_status_enum_members_to_.py
✅ Files skipped from review due to trivial changes (3)
- README.md
- app/routes/shop_rules.py
- CLAUDE.md
🚧 Files skipped from review as they are similar to previous changes (1)
- app/price_check_service.py
…h and pull request events
…es in Amazon scraping
1 participant
Summary by CodeRabbit