Repository navigation
The three reds of the 2026-09-19 nightly - #108
Merged
Merged
Conversation
Companion to corNgsild's revert of 37a49f3. § 10.4.3.4 names a CLOSED list - type / attrs / q / GeoQuery / local scope - and says in prose that entity identifiers alone are not enough. ETSI 019_03_05 and 021_24 assert the 400 and both went red in the 2026-09-19 nightly, on both HTTP backends, twice. ⭐ THE SAME RULE WAS IMPLEMENTED THREE TIMES AND THE THREE DISAGREED: corNgsild ldParamsValidate.c GET (hook, runs first) +scopeQ +id +idPattern getEntities.c POST entityOperations +id getEntitiesTemporal.c GET temporal +id +idPattern All three now carry the clause's list and nothing else. The disagreement was invisible because the hook runs first for GET, so getEntities.c's copy was unreachable on the path anyone tested - and its comment described behaviour the broker did not have. query_entities_too_wide inverted back: 03 (id) and 04 (idPattern) are 400 again, and a step 07 added for scopeQ-only, which used to get a different message than the rule it broke. Two other tests asserted the old wording. Refs the ETSI conformance goal - this is a conformance fix, not a preference. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…eport Every valgrind shard of the 2026-09-19 nightly failed - both HTTP backends, all four ranges - on the same eight tests: FAIL: valgrind: report INCOMPLETE — broker killed before finish, cannot trust The .vg is 508 bytes: valgrind's six header lines and nothing after them. All eight PASS functionally; it is only the memory verdict that is missing. Cause: their --TEARDOWN-- never calls coraineStop, so the harness kills the broker instead of letting it exit, and valgrind never writes its summary. A normal run cannot notice - the broker is killed either way and the assertions have already passed - which is why they went in green and only the nightly found them. The eight are every test added on 09-17 and 09-18. A scan of all 666 finds exactly three others without coraineStop, all legitimate: ftclient_smoke and usage start no broker, and feature_builds starts its own outside the harness and kills them by pid file. Verified as the rule says: reverted the teardowns, ran one under -vt, watched it fail with the nightly's exact message, restored, ran all eight under -vt - green, no leaks. ⭐ Worth a harness guard: a test that calls coraineStart and never coraineStop is malformed the same way a missing TEARDOWN marker is, and corTest already refuses the latter. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…kwards
The performance job failed on 09-18 and 09-19 and on no nightly before that.
It is not a regression: the broker was 80-180% FASTER than the median on
every throughput metric in the run that went red.
perfCompare computed `change = (now - ref) / ref * 100` for every metric and
failed at -50%. Right for requests/second, exactly backwards for the `_p99us`
p99 latency metrics. A fast runner raises throughput AND lowers p99, so both
halves of each pair say "this runner was fast" and one of them is read as a
collapse:
query_c200 8103 vs 4496 +80.2%
query_c200_p99us 31170 vs 62665 -50.3% ❌
retrieve_c50 27037 vs 12150 +122.5%
retrieve_c50_p99us 3330 vs 7450 -55.3% ❌
⭐ The dangerous half is the other direction: with the sign unhandled, a p99
that TRIPLES scores +200% and the gate stays green. The gate was loud about
improvements and blind to the regression it exists to catch.
Thresholds are now applied to a direction-corrected delta; the table still
prints the true change, plus a "latency, lower is better" note on those rows.
Verified both ways against the recorded history: the run that went red now
exits 0 ("within noise"), and a synthetic run with p99 tripled and throughput
unchanged exits 1.
Why it started on 09-18: the _p99us metrics were added on 09-16 and WINDOW=5
needs history before a metric is compared at all. A shared runner moves ~2x
between runs, so this would have fired on roughly every other night.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
C8 the closed selector list of § 10.4.3.4 and the three disagreeing copies, C9 the eight tests that never stopped their broker, C10 the perf gate reading a latency improvement as a collapse. Plus the two upstream PRs that carry the tutorial half - CRUD-Operations #24 and Concise-Format #3 - and why attrs could not be restored (it cannot be combined with pick). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The first nightly after the tutorials sweep went red almost everywhere: both ETSI jobs, all eight valgrind shards, and
performance. Three independent causes, none of them the same kind of problem.1. ETSI —
idalone does not satisfy the too-wide-query ruleTwo TPs, both backends, both failed twice (the suite reruns failures once, so these are regressions, not flakes):
019_03_05and021_24, both named "Without Minimal Parameters", both asserting 400 and getting 200.Cause is corNgsild
37a49f3of 09-18. § 10.4.3.4 names a closed list — type / attrs / q / GeoQuery / local scope — and says in prose that "it is not possible to retrieve a set of entities by only specifying desired Entity identifiers". § 11.3.3.4 repeats it for the temporal query. The reasoning in37a49f3is sound about registration matching and beside the point about what the clause says.⭐ The revert exposed the real problem: the same rule was implemented three times, and the three disagreed.
corNgsild/ldParamsValidate.cgetEntities.c/entityOperations/querygetEntitiesTemporal.cThe disagreement was invisible because the hook runs first for GET, so
getEntities.c's copy was unreachable on the path anyone tested — and its comment described behaviour the broker did not have.query_entities_too_wideinverted back (03idand 04idPatternare 400 again), step 07 added forscopeQ-only, and two other tests updated for the wording.2. valgrind — eight tests never stopped their broker
Every shard, both backends, the same eight tests:
The
.vgis 508 bytes — valgrind's six header lines and nothing else. All eight pass functionally; only the memory verdict is missing. Their--TEARDOWN--never callscoraineStop, so the harness kills the broker instead of letting it exit and valgrind never writes its summary. A normal run cannot notice: the broker is killed either way and the assertions have already passed.The eight are exactly the tests added on 09-17 and 09-18. A scan of all 666 finds three others without
coraineStop, all legitimate —ftclient_smokeandusagestart no broker,feature_buildsstarts its own outside the harness and kills them by pid file.Verified the way the rule says: reverted the teardowns, ran one under
-vt, watched it fail with the nightly's exact message, restored, ran all eight — green, no leaks.⭐ Worth a follow-up: a test that calls
coraineStartand nevercoraineStopis malformed the same way a missing--TEARDOWN--marker is, and corTest already refuses the latter. Not done here.3. performance — the gate had the sign backwards
Not a regression. The broker was 80–180% faster than the median on every throughput metric in the run that went red.
perfCompare.pycomputed onechangefor every metric and failed at -50%. Correct for requests/second, exactly backwards for the_p99usp99 latency metrics. A fast runner raises throughput and lowers p99, so both halves of each pair say the same thing and one is read as a collapse:query_c200query_c200_p99usretrieve_c50retrieve_c50_p99us⭐ The dangerous half is the other direction: with the sign unhandled a p99 that triples scores +200% and the gate stays green. Loud about improvements, blind to the regression it exists to catch.
Thresholds now apply to a direction-corrected delta; the table still prints the true change plus a "latency, lower is better" note. Verified both ways against the recorded history — the run that went red exits 0 ("within noise"), and a synthetic run with p99 tripled and throughput flat exits 1.
Why it began on 09-18 and not before: the
_p99usmetrics were added on 09-16 andWINDOW = 5needs history before a metric is compared at all. A shared runner moves ~2x between runs, so this would have fired on roughly every other night.Tutorial half
The tutorial that prompted
37a49f3is fixed upstream rather than in the broker: FIWARE/tutorials.CRUD-Operations#24 and FIWARE/tutorials.Concise-Format#3, three requests in all, each addinglocal=true.attrscould not simply be restored — a broker answers?attrs= cannot be combined with ?pick= or ?omit=. Found by running the proposed fix before proposing it.doc/tutorial-doubts.mdgains a phase 7 with all three (C8, C9, C10).🤖 Generated with Claude Code