Skip to content

Fix UNGM scraper/parser - #339

Open
andreighh wants to merge 3 commits into
masterfrom
fix-ungm-parser
Open

andreighh wants to merge 3 commits into
masterfrom
fix-ungm-parser

Conversation

@andreighh

Copy link
Copy Markdown
Contributor

UNGM changed its site and the UNGM scraper stopped importing tenders:

  • the notice search now requires an ASP.NET anti-forgery token;
  • the markup of the result list and the notice pages changed.

Contract awards (add_award) were broken too:

  • the award search moved to a new endpoint;
  • save_award crashed on every award, so awards had never been saved.

Changes

Tender search and parsing (app/server_requests.py, app/parsers/ungm.py)

  • Load the search page and send the POST in one requests.Session, returning the __RequestVerificationToken in the RequestVerificationToken header.
  • The search payload now matches the site's: the new fields, UNSPSC IDs sent as numbers, NoticeTASStatus dropped. The UNSPSC filter still works the same way.
  • Use the same current Firefox headers for GET and POST, add timeouts to every UNGM request, and back off longer between retries.
  • Pause 2–5 s between notice requests, and skip a notice that fails to load instead of crashing the run.
  • Parse list rows by tableCell instead of positions in .contents, and limit notice parsing to #noticeDetail (the menu reuses status-tag).

Contract awards (app/management/commands/add_award.py)

  • Search through /Public/ContractAward/PublicSearch with the site's current payload.
  • Only consider UNGM tenders whose deadline passed in the last 90 days and that have no award yet, and pause between requests. Each tender costs at least 3 requests, and without the window
    the same old tenders were searched again on every run.
  • Keep only results whose reference exactly matches the tender's (UNGM's filter also matches partially). Fetch every matching award (one per lot) and merge them into one Award, as TED and
    IUCN do: summed value, all vendors, latest date.
  • Fix save_award: it called .save() on the update_or_create tuple. Store a missing contract value as None instead of '', and normalize vendor names with transform_vendor_name.
  • The help text now explains that the command only handles UNGM; TED and IUCN awards come in through their own scrapers.

  UNGM's notice search now requires an ASP.NET antiforgery token and the
  list/notice markup changed, so the scraper got no results.

  - Load the search page and POST in one requests.Session, sending the
    __RequestVerificationToken back in the RequestVerificationToken header
  - Add the new search payload fields sent by the site
  - Use consistent, current Firefox headers for GET and POST
  - Add timeouts to every UNGM request, longer backoff between retries
  - Pause 2-5s between notice requests and skip notices that fail to load
  - Parse list rows by tableCell instead of positional .contents indexes
  - Limit notice parsing to #noticeDetail (the menu reuses status-tag)
  - Tests and fixtures for the 2026 list and notice pages
  - Search contract awards via /Public/ContractAward/PublicSearch with the
    payload the site sends now (SupplierCountries, no UNSPSCs)
  - Copy the awards payload instead of mutating the shared PAYLOAD dict
  - Extract the contract id in parse_contract_id, return None on no results
  - Fix save_award calling .save() on the update_or_create tuple
  - Store a missing contract value as None instead of '' (FloatField)
  - Tests and fixtures for the 2026 award search and award page- - Send UNSPSC ids as numbers and drop NoticeTASStatus, matching the site's search payload
  - Only search awards of tenders that expired in the last 90 days, and
    pause between requests, to keep the request volume to UNGM low
  - Match search results on the exact tender reference (UNGM's filter
    also matches partially)
  - Fetch every matching award (one per lot) and merge them into one
    Award like TED and IUCN: summed value, all vendors, latest date
  - Normalize vendor names with transform_vendor_name
  - Return dates from string_to_date so award dates can be compared
  - Describe what the command does in its help text
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant