Repository navigation
fix(selection): drop group members that fail the checks - #324
Merged
MoonBoi9001 merged 4 commits intoOct 26, 2026
Merged
MoonBoi9001 merged 4 commits into
MoonBoi9001 merged 4 commits into
Conversation
The generator could put a whole number beyond the float range, such as 10**400, in a per-entity price column. pandas can't build that table, so tests failed while setting up a case the service could never load, on any run that didn't use CI's fixed seed.
Dipper re-asks for each request's group daily, but an indexer already in it stayed even when blocklisted or when its own advertised prices failed the checks. Both now leave the group; one with no price data stays, since a failed price fetch looks the same and leaving cancels its agreement.
When the price checks left nobody, the answer was empty and dipper cancelled every agreement on the deployment, as on a day the score job reaches no indexer's price endpoint. The members that may stay are now the answer.
…-lock' into mb9/drop-group-members-that-fail-the-checks
MoonBoi9001
marked this pull request as ready for review
October 26, 2026 21:21
MoonBoi9001
merged commit Oct 26, 2026
e5e94ed
into
mb9/build-service-image-from-the-lock
7 checks passed
MoonBoi9001
added a commit
that referenced
this pull request
Oct 26, 2026
* build(image): install the service's packages from uv.lock The service image installed whatever newest versions pyproject.toml allowed on the day it was built, so production could run versions CI never tested. It now installs uv.lock's exact versions, and CI's image test job runs on those under the image's Python 3.12. * ci(audit): check the service's locked versions for advisories The audit checked whatever versions pip picked from pyproject.toml on the day, which the service image no longer installs. It now checks the exact versions in uv.lock that the image ships. * build(image): keep uv and dev groups out of the service image The image kept a copy of uv it only needs while installing, and would have installed a dev dependency group if one were ever added. uv is now mounted for the install step alone. * docs(readme): say the service image runs the locked versions The README described the locked versions and the image's install as 2 different sets that CI tests, which they no longer are. * fix(selection): drop group members that fail the checks (#324) * test(properties): stop generating prices no score table can hold The generator could put a whole number beyond the float range, such as 10**400, in a per-entity price column. pandas can't build that table, so tests failed while setting up a case the service could never load, on any run that didn't use CI's fixed seed. * fix(selection): drop group members that are blocklisted or priced out Dipper re-asks for each request's group daily, but an indexer already in it stayed even when blocklisted or when its own advertised prices failed the checks. Both now leave the group; one with no price data stays, since a failed price fetch looks the same and leaving cancels its agreement. * fix(selection): keep group members when no candidate is left When the price checks left nobody, the answer was empty and dipper cancelled every agreement on the deployment, as on a day the score job reaches no indexer's price endpoint. The members that may stay are now the answer.
MoonBoi9001
added a commit
that referenced
this pull request
Oct 26, 2026
* build(deps): list the cronjob's packages only in its requirements The cronjob's packages were also copied into the service's dev extras, and Dependabot bumps the 2 lists in separate PRs, so they drifted apart. The copies go, and the type check installs the cronjob's requirements.txt instead. * ci(typecheck): keep the cronjob's shared packages at locked versions Installing the cronjob's packages for the type check could quietly replace a locked version when a cronjob bump needed a newer one. The lock now constrains that install, so it fails instead. * fix(image): run the service on the versions CI tested (#323) * build(image): install the service's packages from uv.lock The service image installed whatever newest versions pyproject.toml allowed on the day it was built, so production could run versions CI never tested. It now installs uv.lock's exact versions, and CI's image test job runs on those under the image's Python 3.12. * ci(audit): check the service's locked versions for advisories The audit checked whatever versions pip picked from pyproject.toml on the day, which the service image no longer installs. It now checks the exact versions in uv.lock that the image ships. * build(image): keep uv and dev groups out of the service image The image kept a copy of uv it only needs while installing, and would have installed a dev dependency group if one were ever added. uv is now mounted for the install step alone. * docs(readme): say the service image runs the locked versions The README described the locked versions and the image's install as 2 different sets that CI tests, which they no longer are. * fix(selection): drop group members that fail the checks (#324) * test(properties): stop generating prices no score table can hold The generator could put a whole number beyond the float range, such as 10**400, in a per-entity price column. pandas can't build that table, so tests failed while setting up a case the service could never load, on any run that didn't use CI's fixed seed. * fix(selection): drop group members that are blocklisted or priced out Dipper re-asks for each request's group daily, but an indexer already in it stayed even when blocklisted or when its own advertised prices failed the checks. Both now leave the group; one with no price data stays, since a failed price fetch looks the same and leaving cancels its agreement. * fix(selection): keep group members when no candidate is left When the price checks left nobody, the answer was empty and dipper cancelled every agreement on the deployment, as on a day the score job reaches no indexer's price endpoint. The members that may stay are now the answer.
MoonBoi9001
added a commit
that referenced
this pull request
Oct 26, 2026
* fix(ci): stop the weekly cronjob rebuild failing on an existing branch Reruns on an unchanged main commit pushed the same manifest update branch again and failed. Now a rerun leaves an open PR for the same image alone, or else pushes a run-specific branch, so a closed PR's branch is never force pushed over. The step moves to a script so it can be tested. * fix(ci): queue cronjob rebuilds instead of running them side by side A push to main and the weekly schedule can start runs on the same commit at the same moment. Both would find no manifest update branch and push one under the same name, so 1 would fail. Runs now take turns; GitHub keeps only the newest waiting run, which builds the latest main. * chore(deps): list the cronjob's packages in 1 place (#322) * build(deps): list the cronjob's packages only in its requirements The cronjob's packages were also copied into the service's dev extras, and Dependabot bumps the 2 lists in separate PRs, so they drifted apart. The copies go, and the type check installs the cronjob's requirements.txt instead. * ci(typecheck): keep the cronjob's shared packages at locked versions Installing the cronjob's packages for the type check could quietly replace a locked version when a cronjob bump needed a newer one. The lock now constrains that install, so it fails instead. * fix(image): run the service on the versions CI tested (#323) * build(image): install the service's packages from uv.lock The service image installed whatever newest versions pyproject.toml allowed on the day it was built, so production could run versions CI never tested. It now installs uv.lock's exact versions, and CI's image test job runs on those under the image's Python 3.12. * ci(audit): check the service's locked versions for advisories The audit checked whatever versions pip picked from pyproject.toml on the day, which the service image no longer installs. It now checks the exact versions in uv.lock that the image ships. * build(image): keep uv and dev groups out of the service image The image kept a copy of uv it only needs while installing, and would have installed a dev dependency group if one were ever added. uv is now mounted for the install step alone. * docs(readme): say the service image runs the locked versions The README described the locked versions and the image's install as 2 different sets that CI tests, which they no longer are. * fix(selection): drop group members that fail the checks (#324) * test(properties): stop generating prices no score table can hold The generator could put a whole number beyond the float range, such as 10**400, in a per-entity price column. pandas can't build that table, so tests failed while setting up a case the service could never load, on any run that didn't use CI's fixed seed. * fix(selection): drop group members that are blocklisted or priced out Dipper re-asks for each request's group daily, but an indexer already in it stayed even when blocklisted or when its own advertised prices failed the checks. Both now leave the group; one with no price data stays, since a failed price fetch looks the same and leaving cancels its agreement. * fix(selection): keep group members when no candidate is left When the price checks left nobody, the answer was empty and dipper cancelled every agreement on the deployment, as on a day the score job reaches no indexer's price endpoint. The members that may stay are now the answer.
MoonBoi9001
added a commit
that referenced
this pull request
Oct 26, 2026
* chore(deps): group every cronjob dependency update into 1 weekly PR The cronjob's requirements.txt has version ranges and no lock file, so Dependabot can't tell a major from a patch and its minor/patch filter sent each bump to its own PR (5 on 23 Oct). Dropping that filter puts every cronjob update, majors included, into 1 grouped weekly PR. * docs(deps): warn that majors capped in both files need a joint bump CI installs the cronjob's requirements.txt with pyproject.toml, and both cap pyarrow, numpy and 6 more packages. With majors now in the weekly group PR, any such major fails the whole PR until both caps are raised by hand. pyarrow 26.0.0 is already out against a <25 cap. * fix(ci): stop the weekly cronjob rebuild failing on reruns (#309) * fix(ci): stop the weekly cronjob rebuild failing on an existing branch Reruns on an unchanged main commit pushed the same manifest update branch again and failed. Now a rerun leaves an open PR for the same image alone, or else pushes a run-specific branch, so a closed PR's branch is never force pushed over. The step moves to a script so it can be tested. * fix(ci): queue cronjob rebuilds instead of running them side by side A push to main and the weekly schedule can start runs on the same commit at the same moment. Both would find no manifest update branch and push one under the same name, so 1 would fail. Runs now take turns; GitHub keeps only the newest waiting run, which builds the latest main. * chore(deps): list the cronjob's packages in 1 place (#322) * build(deps): list the cronjob's packages only in its requirements The cronjob's packages were also copied into the service's dev extras, and Dependabot bumps the 2 lists in separate PRs, so they drifted apart. The copies go, and the type check installs the cronjob's requirements.txt instead. * ci(typecheck): keep the cronjob's shared packages at locked versions Installing the cronjob's packages for the type check could quietly replace a locked version when a cronjob bump needed a newer one. The lock now constrains that install, so it fails instead. * fix(image): run the service on the versions CI tested (#323) * build(image): install the service's packages from uv.lock The service image installed whatever newest versions pyproject.toml allowed on the day it was built, so production could run versions CI never tested. It now installs uv.lock's exact versions, and CI's image test job runs on those under the image's Python 3.12. * ci(audit): check the service's locked versions for advisories The audit checked whatever versions pip picked from pyproject.toml on the day, which the service image no longer installs. It now checks the exact versions in uv.lock that the image ships. * build(image): keep uv and dev groups out of the service image The image kept a copy of uv it only needs while installing, and would have installed a dev dependency group if one were ever added. uv is now mounted for the install step alone. * docs(readme): say the service image runs the locked versions The README described the locked versions and the image's install as 2 different sets that CI tests, which they no longer are. * fix(selection): drop group members that fail the checks (#324) * test(properties): stop generating prices no score table can hold The generator could put a whole number beyond the float range, such as 10**400, in a per-entity price column. pandas can't build that table, so tests failed while setting up a case the service could never load, on any run that didn't use CI's fixed seed. * fix(selection): drop group members that are blocklisted or priced out Dipper re-asks for each request's group daily, but an indexer already in it stayed even when blocklisted or when its own advertised prices failed the checks. Both now leave the group; one with no price data stays, since a failed price fetch looks the same and leaving cancels its agreement. * fix(selection): keep group members when no candidate is left When the price checks left nobody, the answer was empty and dipper cancelled every agreement on the deployment, as on a day the score job reaches no indexer's price endpoint. The members that may stay are now the answer.
MoonBoi9001
added a commit
that referenced
this pull request
Oct 26, 2026
* test(layout): move the daily job's tests into a tests/cronjob folder The service and the daily score job ship different package versions, so CI needs their tests apart. A new layout test fails when a test that loads the job sits outside tests/cronjob. * ci(tests): test the service against the exact versions in uv.lock CI tested whatever pip resolved on the day, not the versions uv.lock pins for the OSV scan. Service jobs now install with uv sync --locked and fail on a stale lock. The daily job's tests run on a fresh install of its requirements.txt, like its image, and fail if test tools alter it. * test(layout): move the stray subgraph tests to where CI runs them The daily job's 9 subgraph paging and indexer discovery tests sat beside its code, where neither CI job looks, so they never ran and shipped in its image. A check now fails on any test file outside tests/. * ci(tests): fail when the cronjob imports a package its image lacks The cronjob's test install adds packages its image never gets, such as pydantic and pyyaml, so an import only those satisfy passed CI and crashed the job. A check now reads every import in the job's code against an install of just its requirements.txt. * ci(tests): also test the service on the versions its image installs The service image installs its dependencies with pip on Python 3.12 without reading uv.lock, so it ships newer versions than the lock-based jobs test. A new job tests the service on that. * chore(deps): group every cronjob dependency bump into 1 PR (#307) * chore(deps): group every cronjob dependency update into 1 weekly PR The cronjob's requirements.txt has version ranges and no lock file, so Dependabot can't tell a major from a patch and its minor/patch filter sent each bump to its own PR (5 on 23 Oct). Dropping that filter puts every cronjob update, majors included, into 1 grouped weekly PR. * docs(deps): warn that majors capped in both files need a joint bump CI installs the cronjob's requirements.txt with pyproject.toml, and both cap pyarrow, numpy and 6 more packages. With majors now in the weekly group PR, any such major fails the whole PR until both caps are raised by hand. pyarrow 26.0.0 is already out against a <25 cap. * fix(ci): stop the weekly cronjob rebuild failing on reruns (#309) * fix(ci): stop the weekly cronjob rebuild failing on an existing branch Reruns on an unchanged main commit pushed the same manifest update branch again and failed. Now a rerun leaves an open PR for the same image alone, or else pushes a run-specific branch, so a closed PR's branch is never force pushed over. The step moves to a script so it can be tested. * fix(ci): queue cronjob rebuilds instead of running them side by side A push to main and the weekly schedule can start runs on the same commit at the same moment. Both would find no manifest update branch and push one under the same name, so 1 would fail. Runs now take turns; GitHub keeps only the newest waiting run, which builds the latest main. * chore(deps): list the cronjob's packages in 1 place (#322) * build(deps): list the cronjob's packages only in its requirements The cronjob's packages were also copied into the service's dev extras, and Dependabot bumps the 2 lists in separate PRs, so they drifted apart. The copies go, and the type check installs the cronjob's requirements.txt instead. * ci(typecheck): keep the cronjob's shared packages at locked versions Installing the cronjob's packages for the type check could quietly replace a locked version when a cronjob bump needed a newer one. The lock now constrains that install, so it fails instead. * fix(image): run the service on the versions CI tested (#323) * build(image): install the service's packages from uv.lock The service image installed whatever newest versions pyproject.toml allowed on the day it was built, so production could run versions CI never tested. It now installs uv.lock's exact versions, and CI's image test job runs on those under the image's Python 3.12. * ci(audit): check the service's locked versions for advisories The audit checked whatever versions pip picked from pyproject.toml on the day, which the service image no longer installs. It now checks the exact versions in uv.lock that the image ships. * build(image): keep uv and dev groups out of the service image The image kept a copy of uv it only needs while installing, and would have installed a dev dependency group if one were ever added. uv is now mounted for the install step alone. * docs(readme): say the service image runs the locked versions The README described the locked versions and the image's install as 2 different sets that CI tests, which they no longer are. * fix(selection): drop group members that fail the checks (#324) * test(properties): stop generating prices no score table can hold The generator could put a whole number beyond the float range, such as 10**400, in a per-entity price column. pandas can't build that table, so tests failed while setting up a case the service could never load, on any run that didn't use CI's fixed seed. * fix(selection): drop group members that are blocklisted or priced out Dipper re-asks for each request's group daily, but an indexer already in it stayed even when blocklisted or when its own advertised prices failed the checks. Both now leave the group; one with no price data stays, since a failed price fetch looks the same and leaving cancels its agreement. * fix(selection): keep group members when no candidate is left When the price checks left nobody, the answer was empty and dipper cancelled every agreement on the deployment, as on a day the score job reaches no indexer's price endpoint. The members that may stay are now the answer.
MoonBoi9001
added a commit
that referenced
this pull request
Oct 26, 2026
* test(api): fuzz every endpoint from its OpenAPI schema Schemathesis sends generated valid and invalid requests to the app in process, with the push token set and scores seeded, and fails on any 5xx or any answer the schema does not list. The bugs it finds fail here until the fixes that follow; 3 cases needing a decision are skipped. * fix(sync): skip unreadable sync status entries instead of failing A pushed sync status maps each indexer to an object holding a list of deployment ids. An entry that was not an object, or whose ids were not a list of text, failed the whole push with a 500. Each such entry is now skipped with a warning, so the readable entries still load. * fix(api): answer a bearer token with non-ASCII letters with a 401 The push token check compares tokens in constant time, and that comparison refuses text with letters outside ASCII, so a header such as "Bearer café" failed the request with a 500. Both tokens are now compared as bytes, so any wrong token gets the usual 401. * fix(scores): answer a push of records with no fields with a 422 A scores push whose records held no fields at all made a table with rows but no columns, which pandas calls empty, so the check meant for a push with no rows failed it with a 500. Only a push with no rows counts as empty now, so this one gets the 422 for its missing columns. * fix(api): give a rejected scores push the documented 422 error list The API schema says every 422 carries a list of errors, each naming where it is and why. A scores push that was empty or lacked a required column got a 422 holding plain text instead. An empty push is now refused by the schema itself, and missing columns come back as 1 error. * docs(api): list the 400, 401 and 503 answers each route can give The API schema listed only 200 and 422, yet routes also answer 400 when a body cannot be read as text, 401 for a missing or wrong bearer token, and 503 before the service has scores. Each route now lists the ones it gives, with the one-field error body they all carry. * fix(scores): reject a scores push whose indexer is not text A record whose indexer was a number, null, list or object was accepted, then every selection with 2 or more candidates answered 500. The push now answers 422 naming each such record. * refactor(scores): list rejected indexer records before their errors The type checker could not tell that each error's location holds a record number, so the rejection message now takes the first one from the list of rejected records instead. * fix(scores): serve scores pushed without org or location columns Selection reads each indexer's org and location to spread a group, so a push without them made every selection of 2 or more answer 500. They now go in empty, with a warning naming them. * fix(selection): compare a pushed org or location as text A push may hold any JSON value as an indexer's org or location. A list or object there could not be compared, so every selection that checked how spread out a group was answered 500. * fix(scores): read a pushed uptime as a number before scaling it Uptime is scaled from 0 to 1 into a percentage when scores are pushed. An object there answered 500, and text was repeated 100 times. Either now reads as missing, scores 0 and is logged. * fix(scores): echo indexers that are all numbers as plain JSON in the 422 When every record's indexer was a number, pandas held them in its own number type, which the 422 rejecting them could not write as JSON, so the push answered 500 instead. * test(api): push unusual scores, then check every read still answers The schema fuzz seeds fresh scores before each request, so an accepted push that broke later reads went unseen. Each score column is now pushed absent or holding an unusual value, then read back. * test(api): cut the schema fuzz runtime to fit its 1 minute budget It added about 74s to the CI test step. State is now seeded again only after a push changes it, both versions of each request share 1 generated case, and the token test uses fewer examples. * chore(deps): pin schemathesis to 1 release for the API fuzz test CI installs the newest release in range, and each release can change what the fuzz test sends and checks, so it could start failing on unrelated changes. 4.30.0 is the newest over 2 weeks old. * test(api): send half the fuzz examples twice and skip repeated reads The CI test step still took 64s longer than without the fuzz module. Each generated case is sent raw and with seeded ids, so half the examples still sends the full count of requests. * chore(ci): test the service against the versions in uv.lock (#310) * test(layout): move the daily job's tests into a tests/cronjob folder The service and the daily score job ship different package versions, so CI needs their tests apart. A new layout test fails when a test that loads the job sits outside tests/cronjob. * ci(tests): test the service against the exact versions in uv.lock CI tested whatever pip resolved on the day, not the versions uv.lock pins for the OSV scan. Service jobs now install with uv sync --locked and fail on a stale lock. The daily job's tests run on a fresh install of its requirements.txt, like its image, and fail if test tools alter it. * test(layout): move the stray subgraph tests to where CI runs them The daily job's 9 subgraph paging and indexer discovery tests sat beside its code, where neither CI job looks, so they never ran and shipped in its image. A check now fails on any test file outside tests/. * ci(tests): fail when the cronjob imports a package its image lacks The cronjob's test install adds packages its image never gets, such as pydantic and pyyaml, so an import only those satisfy passed CI and crashed the job. A check now reads every import in the job's code against an install of just its requirements.txt. * ci(tests): also test the service on the versions its image installs The service image installs its dependencies with pip on Python 3.12 without reading uv.lock, so it ships newer versions than the lock-based jobs test. A new job tests the service on that. * chore(deps): group every cronjob dependency bump into 1 PR (#307) * chore(deps): group every cronjob dependency update into 1 weekly PR The cronjob's requirements.txt has version ranges and no lock file, so Dependabot can't tell a major from a patch and its minor/patch filter sent each bump to its own PR (5 on 23 Oct). Dropping that filter puts every cronjob update, majors included, into 1 grouped weekly PR. * docs(deps): warn that majors capped in both files need a joint bump CI installs the cronjob's requirements.txt with pyproject.toml, and both cap pyarrow, numpy and 6 more packages. With majors now in the weekly group PR, any such major fails the whole PR until both caps are raised by hand. pyarrow 26.0.0 is already out against a <25 cap. * fix(ci): stop the weekly cronjob rebuild failing on reruns (#309) * fix(ci): stop the weekly cronjob rebuild failing on an existing branch Reruns on an unchanged main commit pushed the same manifest update branch again and failed. Now a rerun leaves an open PR for the same image alone, or else pushes a run-specific branch, so a closed PR's branch is never force pushed over. The step moves to a script so it can be tested. * fix(ci): queue cronjob rebuilds instead of running them side by side A push to main and the weekly schedule can start runs on the same commit at the same moment. Both would find no manifest update branch and push one under the same name, so 1 would fail. Runs now take turns; GitHub keeps only the newest waiting run, which builds the latest main. * chore(deps): list the cronjob's packages in 1 place (#322) * build(deps): list the cronjob's packages only in its requirements The cronjob's packages were also copied into the service's dev extras, and Dependabot bumps the 2 lists in separate PRs, so they drifted apart. The copies go, and the type check installs the cronjob's requirements.txt instead. * ci(typecheck): keep the cronjob's shared packages at locked versions Installing the cronjob's packages for the type check could quietly replace a locked version when a cronjob bump needed a newer one. The lock now constrains that install, so it fails instead. * fix(image): run the service on the versions CI tested (#323) * build(image): install the service's packages from uv.lock The service image installed whatever newest versions pyproject.toml allowed on the day it was built, so production could run versions CI never tested. It now installs uv.lock's exact versions, and CI's image test job runs on those under the image's Python 3.12. * ci(audit): check the service's locked versions for advisories The audit checked whatever versions pip picked from pyproject.toml on the day, which the service image no longer installs. It now checks the exact versions in uv.lock that the image ships. * build(image): keep uv and dev groups out of the service image The image kept a copy of uv it only needs while installing, and would have installed a dev dependency group if one were ever added. uv is now mounted for the install step alone. * docs(readme): say the service image runs the locked versions The README described the locked versions and the image's install as 2 different sets that CI tests, which they no longer are. * fix(selection): drop group members that fail the checks (#324) * test(properties): stop generating prices no score table can hold The generator could put a whole number beyond the float range, such as 10**400, in a per-entity price column. pandas can't build that table, so tests failed while setting up a case the service could never load, on any run that didn't use CI's fixed seed. * fix(selection): drop group members that are blocklisted or priced out Dipper re-asks for each request's group daily, but an indexer already in it stayed even when blocklisted or when its own advertised prices failed the checks. Both now leave the group; one with no price data stays, since a failed price fetch looks the same and leaving cancels its agreement. * fix(selection): keep group members when no candidate is left When the price checks left nobody, the answer was empty and dipper cancelled every agreement on the deployment, as on a day the score job reaches no indexer's price endpoint. The members that may stay are now the answer. * fix(image): point every manifest that runs the image at a new build The image now also runs the sync-status fetcher, but the update script only moved the CronJob, so the fetcher would keep its old image. An open PR now counts only if it moves both. * docs(readme): say the service tests run on the Python the image runs CI no longer runs them on both 3.11 and 3.12: every job now takes its Python from .python-version, and CI fails if the image runs a different one.
MoonBoi9001
added a commit
that referenced
this pull request
Oct 27, 2026
* test(selection): add property tests for pricing and scoring (#314) * fix(selection): drop indexers priced above the per-entity price ceiling Dipper caps what it pays per billion entities, but selection only filtered on the monthly base price. Both selection and the DIPs-accepting list now take that cap and drop indexers above it; an indexer with no per-entity price still passes, since it is offered 0 per entity. * fix(selection): reject a negative or non-numeric per-entity price cap A ceiling of NaN, infinity or below 0 would quietly drop every indexer with a per-entity price. Both endpoints now refuse such a value with a 422, so a bad caller setting fails loudly. * docs(selection): restore what omitting chain or a ceiling does The dips-indexers endpoint description lost 2 still-true details when the per-entity ceiling was added: a missing chain is rejected, and an omitted ceiling is not applied. * fix(selection): log a base price ceiling of 0 instead of showing none The selection request log printed none for a 0 GRT ceiling even though selection still applied it, hiding why every priced indexer was dropped. * fix(selection): reject a negative or non-numeric base price cap A base price ceiling of NaN, infinity or below 0 would quietly drop every priced indexer. Both endpoints now refuse one with a 422, as they already do for the per-entity ceiling. * fix(selection): answer a NaN or infinite price cap with a 422, not a 500 The standard error reply repeats the rejected value, and NaN or infinity can't be written as JSON, so a bad price cap in a selection request crashed the reply. Such values are now sent as text. * fix(processing): ignore NaN or negative base prices from indexers Indexers advertise a base price per chain and can send any JSON. A NaN or below-0 price was scored as the cheapest and won selection. Such a price is now dropped and logged, so the indexer counts as unpriced on that chain and is left out there. * fix(processing): treat an invalid per-entity price as no DIPs info An indexer that sends no per-entity price is offered 0 for it, so dropping a NaN or negative value would reward the indexer. Its whole price response is now ignored and logged instead. * fix(selection): count a NaN or negative chain price as no price Selection read each indexer's base price for a chain unchecked, so NaN or a below-0 value was scored as the cheapest and won. Such a price, or an infinite one, now counts as missing, which leaves the indexer out on that chain. * fix(processing): skip an indexer whose /dips/info body is unreadable Any indexer can reply to /dips/info with an empty body, an array or broken JSON. That raised out of the fetch and failed it for every indexer, so no scores were pushed at all. Such a reply is now logged and the indexer counted as having no DIPs info, without retrying. * fix(processing): store advertised prices as parsed numbers, not raw text Prices were checked with Python's float(), which also reads text like "1_000", but stored as sent. The selection service reads per-entity prices with pandas, which finds no number there, so it scored the indexer as free while dipper offered the indexer's real, higher price. * fix(processing): escape and shorten indexer values in price warnings The warnings for a dropped price quote the chain name and value the indexer sent. A line break in the chain name started a fake log line, and a multi-megabyte value was logged in full on every run. Both are now quoted with escapes and cut to at most 200 characters. * fix(selection): read each per-entity price the same way in every step Each step read an indexer's per-entity price its own way: the ceiling failed unreadable text, scoring counted it as 0, the cheapest, and dipper was offered a negative price as sent. Now 1 parser reads it everywhere, and a value that is not a finite number >= 0 is never a price. * fix(selection): always drop indexers with an invalid per-entity price An indexer whose per-entity price is set but is not a finite number >= 0 could still be picked when the request sent no per-entity ceiling. The price filter now always drops it and logs a warning naming the indexer and the value it sent. * fix(selection): require a chain on every indexer selection request A selection request sent without a chain skipped every price filter, including both price caps. The chain is now required and must not be empty, so such a request gets a 422 instead. Helpers that only ever receive the request's chain no longer handle a missing one. * docs(selection): note price filters also need the DIPs score columns The comment on chain_id implied a chain is all the price filters need. Loaded scores without a DIPs column still skip that filter step, so the comment now says so. * docs(selection): stop claiming the request matches dipper's struct Dipper's Rust SelectionRequest still types chain_id as optional, while IISA now rejects a request without one, so the docstring says how the two differ instead of claiming a match. * fix(selection): reject negative or non-finite optimistic DIPs fees Dipper sends each indexer's expected DIPs fees so selection can spread new agreements around. A negative fee inflated that indexer's stake-to-fees score and a NaN made it NaN. Each fee must now be a finite number of at least 0, or the request is a 422. * docs(selection): correct what a NaN optimistic DIPs fee used to do The selector filled a NaN optimistic fee with 0, so it was ignored rather than making the score NaN, and an infinite fee sank the indexer's stake-to-fees score. The comments now say so. * test(selection): add property tests for price filters and scores Hypothesis feeds the price readers, filters, normalisers and selection NaN, infinite, negative, huge, unreadable and missing values. The bugs it finds fail here until the fixes that follow; 4 properties that need a decision are strict xfails. * fix(selection): read a price map that is not a JSON object as no price An indexer's per-chain price cell holding JSON such as null or a list raised an error, failing every /select-indexers and /dips-indexers request with a 500. It now drops only that indexer. * fix(selection): read any JSON the decoder rejects as an unreadable cell A price or supported-networks cell nested over 1,000 levels deep, or a networks cell holding an integer over 4,300 digits, raised past the readers and failed selection with a 500. * fix(selection): read each metric as a number before scoring it Text in a metric column, or a column holding only empty values, raised inside scoring, so every indexer was scored on only the metrics done before the error. Text now scores 0 and is logged. * fix(selection): score a pushed metric score outside 0 to 1 as 0 The service keeps the latency, uptime and success-rate scores the CronJob pushes already scaled 0 to 1. A pushed 2 or infinity outranked every indexer and text failed selection; both score 0. * fix(selection): leave unreadable metric values out of normalisation Text in a metric column was normalised as if missing before it scored 0, so stake to fees read it as a best value above the max and base price read it as free, which moved every other indexer's score. Now only readable values are normalised. * fix(selection): score a missing latency, uptime or success rate as 0 A missing reading in these metrics scored 1, the best, because the maths turned it into a blank that the cap to 1 then read as full marks. So an indexer with no data outranked one with a real reading. A missing reading now scores 0, as the scoring notes already said it should. * fix(selection): keep a weighted score with missing metrics at most 1 A weighted score divides the sum of weighted metric scores by the sum of the weights used. The two sums added their terms in different orders, so full marks with a metric missing could round to just above 1. Both now add up in the same order. * test(selection): name the error each known failing test expects Each test marked as a known failure now names the error it expects, so a new crash fails the run. Generated groups that are too big, blocklisted or badly priced had never run, so a passing test now runs them and checks every indexer added to a group follows the rules. * test(selection): point pushed-score tests at latency and stake_to_fees The selector works out uptime and success rate itself and ignores the copies the CronJob sends, so feeding bad copies of those in tested nothing. The examples now feed bad latency and stake_to_fees scores, the 2 the selector keeps as sent and checks fall between 0 and 1. * test(selection): cover unreadable values on a pickable-only scale An unreadable value scores 0 and stays off the scale, and only indexers a request can pick set that scale. When none of those reads as a number, the readable values set the scale instead. * test(api): fuzz the API from its schema and fix what it finds (#318) * test(api): fuzz every endpoint from its OpenAPI schema Schemathesis sends generated valid and invalid requests to the app in process, with the push token set and scores seeded, and fails on any 5xx or any answer the schema does not list. The bugs it finds fail here until the fixes that follow; 3 cases needing a decision are skipped. * fix(sync): skip unreadable sync status entries instead of failing A pushed sync status maps each indexer to an object holding a list of deployment ids. An entry that was not an object, or whose ids were not a list of text, failed the whole push with a 500. Each such entry is now skipped with a warning, so the readable entries still load. * fix(api): answer a bearer token with non-ASCII letters with a 401 The push token check compares tokens in constant time, and that comparison refuses text with letters outside ASCII, so a header such as "Bearer café" failed the request with a 500. Both tokens are now compared as bytes, so any wrong token gets the usual 401. * fix(scores): answer a push of records with no fields with a 422 A scores push whose records held no fields at all made a table with rows but no columns, which pandas calls empty, so the check meant for a push with no rows failed it with a 500. Only a push with no rows counts as empty now, so this one gets the 422 for its missing columns. * fix(api): give a rejected scores push the documented 422 error list The API schema says every 422 carries a list of errors, each naming where it is and why. A scores push that was empty or lacked a required column got a 422 holding plain text instead. An empty push is now refused by the schema itself, and missing columns come back as 1 error. * docs(api): list the 400, 401 and 503 answers each route can give The API schema listed only 200 and 422, yet routes also answer 400 when a body cannot be read as text, 401 for a missing or wrong bearer token, and 503 before the service has scores. Each route now lists the ones it gives, with the one-field error body they all carry. * fix(scores): reject a scores push whose indexer is not text A record whose indexer was a number, null, list or object was accepted, then every selection with 2 or more candidates answered 500. The push now answers 422 naming each such record. * refactor(scores): list rejected indexer records before their errors The type checker could not tell that each error's location holds a record number, so the rejection message now takes the first one from the list of rejected records instead. * fix(scores): serve scores pushed without org or location columns Selection reads each indexer's org and location to spread a group, so a push without them made every selection of 2 or more answer 500. They now go in empty, with a warning naming them. * fix(selection): compare a pushed org or location as text A push may hold any JSON value as an indexer's org or location. A list or object there could not be compared, so every selection that checked how spread out a group was answered 500. * fix(scores): read a pushed uptime as a number before scaling it Uptime is scaled from 0 to 1 into a percentage when scores are pushed. An object there answered 500, and text was repeated 100 times. Either now reads as missing, scores 0 and is logged. * fix(scores): echo indexers that are all numbers as plain JSON in the 422 When every record's indexer was a number, pandas held them in its own number type, which the 422 rejecting them could not write as JSON, so the push answered 500 instead. * test(api): push unusual scores, then check every read still answers The schema fuzz seeds fresh scores before each request, so an accepted push that broke later reads went unseen. Each score column is now pushed absent or holding an unusual value, then read back. * test(api): cut the schema fuzz runtime to fit its 1 minute budget It added about 74s to the CI test step. State is now seeded again only after a push changes it, both versions of each request share 1 generated case, and the token test uses fewer examples. * chore(deps): pin schemathesis to 1 release for the API fuzz test CI installs the newest release in range, and each release can change what the fuzz test sends and checks, so it could start failing on unrelated changes. 4.30.0 is the newest over 2 weeks old. * test(api): send half the fuzz examples twice and skip repeated reads The CI test step still took 64s longer than without the fuzz module. Each generated case is sent raw and with seeded ids, so half the examples still sends the full count of requests. * chore(ci): test the service against the versions in uv.lock (#310) * test(layout): move the daily job's tests into a tests/cronjob folder The service and the daily score job ship different package versions, so CI needs their tests apart. A new layout test fails when a test that loads the job sits outside tests/cronjob. * ci(tests): test the service against the exact versions in uv.lock CI tested whatever pip resolved on the day, not the versions uv.lock pins for the OSV scan. Service jobs now install with uv sync --locked and fail on a stale lock. The daily job's tests run on a fresh install of its requirements.txt, like its image, and fail if test tools alter it. * test(layout): move the stray subgraph tests to where CI runs them The daily job's 9 subgraph paging and indexer discovery tests sat beside its code, where neither CI job looks, so they never ran and shipped in its image. A check now fails on any test file outside tests/. * ci(tests): fail when the cronjob imports a package its image lacks The cronjob's test install adds packages its image never gets, such as pydantic and pyyaml, so an import only those satisfy passed CI and crashed the job. A check now reads every import in the job's code against an install of just its requirements.txt. * ci(tests): also test the service on the versions its image installs The service image installs its dependencies with pip on Python 3.12 without reading uv.lock, so it ships newer versions than the lock-based jobs test. A new job tests the service on that. * chore(deps): group every cronjob dependency bump into 1 PR (#307) * chore(deps): group every cronjob dependency update into 1 weekly PR The cronjob's requirements.txt has version ranges and no lock file, so Dependabot can't tell a major from a patch and its minor/patch filter sent each bump to its own PR (5 on 23 Oct). Dropping that filter puts every cronjob update, majors included, into 1 grouped weekly PR. * docs(deps): warn that majors capped in both files need a joint bump CI installs the cronjob's requirements.txt with pyproject.toml, and both cap pyarrow, numpy and 6 more packages. With majors now in the weekly group PR, any such major fails the whole PR until both caps are raised by hand. pyarrow 26.0.0 is already out against a <25 cap. * fix(ci): stop the weekly cronjob rebuild failing on reruns (#309) * fix(ci): stop the weekly cronjob rebuild failing on an existing branch Reruns on an unchanged main commit pushed the same manifest update branch again and failed. Now a rerun leaves an open PR for the same image alone, or else pushes a run-specific branch, so a closed PR's branch is never force pushed over. The step moves to a script so it can be tested. * fix(ci): queue cronjob rebuilds instead of running them side by side A push to main and the weekly schedule can start runs on the same commit at the same moment. Both would find no manifest update branch and push one under the same name, so 1 would fail. Runs now take turns; GitHub keeps only the newest waiting run, which builds the latest main. * chore(deps): list the cronjob's packages in 1 place (#322) * build(deps): list the cronjob's packages only in its requirements The cronjob's packages were also copied into the service's dev extras, and Dependabot bumps the 2 lists in separate PRs, so they drifted apart. The copies go, and the type check installs the cronjob's requirements.txt instead. * ci(typecheck): keep the cronjob's shared packages at locked versions Installing the cronjob's packages for the type check could quietly replace a locked version when a cronjob bump needed a newer one. The lock now constrains that install, so it fails instead. * fix(image): run the service on the versions CI tested (#323) * build(image): install the service's packages from uv.lock The service image installed whatever newest versions pyproject.toml allowed on the day it was built, so production could run versions CI never tested. It now installs uv.lock's exact versions, and CI's image test job runs on those under the image's Python 3.12. * ci(audit): check the service's locked versions for advisories The audit checked whatever versions pip picked from pyproject.toml on the day, which the service image no longer installs. It now checks the exact versions in uv.lock that the image ships. * build(image): keep uv and dev groups out of the service image The image kept a copy of uv it only needs while installing, and would have installed a dev dependency group if one were ever added. uv is now mounted for the install step alone. * docs(readme): say the service image runs the locked versions The README described the locked versions and the image's install as 2 different sets that CI tests, which they no longer are. * fix(selection): drop group members that fail the checks (#324) * test(properties): stop generating prices no score table can hold The generator could put a whole number beyond the float range, such as 10**400, in a per-entity price column. pandas can't build that table, so tests failed while setting up a case the service could never load, on any run that didn't use CI's fixed seed. * fix(selection): drop group members that are blocklisted or priced out Dipper re-asks for each request's group daily, but an indexer already in it stayed even when blocklisted or when its own advertised prices failed the checks. Both now leave the group; one with no price data stays, since a failed price fetch looks the same and leaving cancels its agreement. * fix(selection): keep group members when no candidate is left When the price checks left nobody, the answer was empty and dipper cancelled every agreement on the deployment, as on a day the score job reaches no indexer's price endpoint. The members that may stay are now the answer. * fix(image): point every manifest that runs the image at a new build The image now also runs the sync-status fetcher, but the update script only moved the CronJob, so the fetcher would keep its old image. An open PR now counts only if it moves both. * docs(readme): say the service tests run on the Python the image runs CI no longer runs them on both 3.11 and 3.12: every job now takes its Python from .python-version, and CI fails if the image runs a different one. * test(layout): skip the daily job's tests where its packages are missing The dev extra leaves out the job's packages, so a plain pytest failed to import them. Locally it now skips tests/cronjob and says why in its header; on CI a missing package still fails. * docs(ci): write the run count in the image workflow's comment as a numeral * chore: drop the CI job that repeats the service tests (#374) * ci(tests): drop the job that repeated the service tests It ran the service tests on the image's Python and locked versions, which the main test job now does too, since every job takes its Python from .python-version. * ci(image): check the service imports only what its image installs CI tests the service with the dev extra installed, which the image leaves out, so code importing a dev-only package passed and then crashed the image at startup. The daily job's check now runs on the service too, before the dev extra is installed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Selection now drops a group member that the request blocklists or whose own advertised prices fail the checks, so dipper cancels its agreement; a member with no price data stays, since a failed price fetch looks the same. When no candidate passes, it answers with the members that may stay, not with nobody, which made dipper cancel every agreement on the deployment.
Important
Deploy graphprotocol/dipper#764 first, or members that missed an offer on another deployment get cancelled.