Repository navigation
Conversation
UNGM's notice search now requires an ASP.NET antiforgery token and the
list/notice markup changed, so the scraper got no results.
- Load the search page and POST in one requests.Session, sending the
__RequestVerificationToken back in the RequestVerificationToken header
- Add the new search payload fields sent by the site
- Use consistent, current Firefox headers for GET and POST
- Add timeouts to every UNGM request, longer backoff between retries
- Pause 2-5s between notice requests and skip notices that fail to load
- Parse list rows by tableCell instead of positional .contents indexes
- Limit notice parsing to #noticeDetail (the menu reuses status-tag)
- Tests and fixtures for the 2026 list and notice pages
- Search contract awards via /Public/ContractAward/PublicSearch with the
payload the site sends now (SupplierCountries, no UNSPSCs)
- Copy the awards payload instead of mutating the shared PAYLOAD dict
- Extract the contract id in parse_contract_id, return None on no results
- Fix save_award calling .save() on the update_or_create tuple
- Store a missing contract value as None instead of '' (FloatField)
- Tests and fixtures for the 2026 award search and award page- - Send UNSPSC ids as numbers and drop NoticeTASStatus, matching the site's search payload
- Only search awards of tenders that expired in the last 90 days, and
pause between requests, to keep the request volume to UNGM low
- Match search results on the exact tender reference (UNGM's filter
also matches partially)
- Fetch every matching award (one per lot) and merge them into one
Award like TED and IUCN: summed value, all vendors, latest date
- Normalize vendor names with transform_vendor_name
- Return dates from string_to_date so award dates can be compared
- Describe what the command does in its help text
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
UNGM changed its site and the UNGM scraper stopped importing tenders:
Contract awards (add_award) were broken too:
Changes
Tender search and parsing (app/server_requests.py, app/parsers/ungm.py)
Contract awards (app/management/commands/add_award.py)
the same old tenders were searched again on every run.
IUCN do: summed value, all vendors, latest date.