Skip to content

The three reds of the 2026-09-19 nightly - #108

Merged
kzangeli merged 4 commits into
mainfrom
fix/nightly-20260919-reds
Sep 19, 2026
Merged

kzangeli merged 4 commits into
mainfrom
fix/nightly-20260919-reds

Conversation

@kzangeli

Copy link
Copy Markdown
Collaborator

The first nightly after the tutorials sweep went red almost everywhere: both ETSI jobs, all eight valgrind shards, and performance. Three independent causes, none of them the same kind of problem.

⚠️ Merge SEAMWARE-cor/corNgsild#28 first. The first fix below is half in that repo — CI here is red until it lands.


1. ETSI — id alone does not satisfy the too-wide-query rule

Two TPs, both backends, both failed twice (the suite reruns failures once, so these are regressions, not flakes): 019_03_05 and 021_24, both named "Without Minimal Parameters", both asserting 400 and getting 200.

Cause is corNgsild 37a49f3 of 09-18. § 10.4.3.4 names a closed list — type / attrs / q / GeoQuery / local scope — and says in prose that "it is not possible to retrieve a set of entities by only specifying desired Entity identifiers". § 11.3.3.4 repeats it for the temporal query. The reasoning in 37a49f3 is sound about registration matching and beside the point about what the clause says.

⭐ The revert exposed the real problem: the same rule was implemented three times, and the three disagreed.

where reached by accepted, before after
corNgsild/ldParamsValidate.c GET (parse hook, runs first) type/attrs/q/geo/scopeQ/local/id/idPattern the clause's list
getEntities.c POST /entityOperations/query type/attrs/q/geo/local/id the clause's list
getEntitiesTemporal.c GET temporal id/idPattern/type/attrs/q/geo/local the clause's list

The disagreement was invisible because the hook runs first for GET, so getEntities.c's copy was unreachable on the path anyone tested — and its comment described behaviour the broker did not have.

query_entities_too_wide inverted back (03 id and 04 idPattern are 400 again), step 07 added for scopeQ-only, and two other tests updated for the wording.

2. valgrind — eight tests never stopped their broker

Every shard, both backends, the same eight tests:

FAIL: valgrind: report INCOMPLETE — broker killed before finish, cannot trust

The .vg is 508 bytes — valgrind's six header lines and nothing else. All eight pass functionally; only the memory verdict is missing. Their --TEARDOWN-- never calls coraineStop, so the harness kills the broker instead of letting it exit and valgrind never writes its summary. A normal run cannot notice: the broker is killed either way and the assertions have already passed.

The eight are exactly the tests added on 09-17 and 09-18. A scan of all 666 finds three others without coraineStop, all legitimate — ftclient_smoke and usage start no broker, feature_builds starts its own outside the harness and kills them by pid file.

Verified the way the rule says: reverted the teardowns, ran one under -vt, watched it fail with the nightly's exact message, restored, ran all eight — green, no leaks.

⭐ Worth a follow-up: a test that calls coraineStart and never coraineStop is malformed the same way a missing --TEARDOWN-- marker is, and corTest already refuses the latter. Not done here.

3. performance — the gate had the sign backwards

Not a regression. The broker was 80–180% faster than the median on every throughput metric in the run that went red.

perfCompare.py computed one change for every metric and failed at -50%. Correct for requests/second, exactly backwards for the _p99us p99 latency metrics. A fast runner raises throughput and lowers p99, so both halves of each pair say the same thing and one is read as a collapse:

metric now median change
query_c200 8103 4496 +80.2%
query_c200_p99us 31170 62665 -50.3% ❌
retrieve_c50 27037 12150 +122.5%
retrieve_c50_p99us 3330 7450 -55.3% ❌

⭐ The dangerous half is the other direction: with the sign unhandled a p99 that triples scores +200% and the gate stays green. Loud about improvements, blind to the regression it exists to catch.

Thresholds now apply to a direction-corrected delta; the table still prints the true change plus a "latency, lower is better" note. Verified both ways against the recorded history — the run that went red exits 0 ("within noise"), and a synthetic run with p99 tripled and throughput flat exits 1.

Why it began on 09-18 and not before: the _p99us metrics were added on 09-16 and WINDOW = 5 needs history before a metric is compared at all. A shared runner moves ~2x between runs, so this would have fired on roughly every other night.


Tutorial half

The tutorial that prompted 37a49f3 is fixed upstream rather than in the broker: FIWARE/tutorials.CRUD-Operations#24 and FIWARE/tutorials.Concise-Format#3, three requests in all, each adding local=true.

attrs could not simply be restored — a broker answers ?attrs= cannot be combined with ?pick= or ?omit=. Found by running the proposed fix before proposing it.

doc/tutorial-doubts.md gains a phase 7 with all three (C8, C9, C10).

🤖 Generated with Claude Code

kzangeli and others added 4 commits September 19, 2026 10:37
Companion to corNgsild's revert of 37a49f3. § 10.4.3.4 names a CLOSED list -
type / attrs / q / GeoQuery / local scope - and says in prose that entity
identifiers alone are not enough. ETSI 019_03_05 and 021_24 assert the 400 and
both went red in the 2026-09-19 nightly, on both HTTP backends, twice.

⭐ THE SAME RULE WAS IMPLEMENTED THREE TIMES AND THE THREE DISAGREED:

  corNgsild ldParamsValidate.c  GET (hook, runs first)  +scopeQ +id +idPattern
  getEntities.c                 POST entityOperations   +id
  getEntitiesTemporal.c         GET temporal            +id +idPattern

All three now carry the clause's list and nothing else. The disagreement was
invisible because the hook runs first for GET, so getEntities.c's copy was
unreachable on the path anyone tested - and its comment described behaviour
the broker did not have.

query_entities_too_wide inverted back: 03 (id) and 04 (idPattern) are 400
again, and a step 07 added for scopeQ-only, which used to get a different
message than the rule it broke. Two other tests asserted the old wording.

Refs the ETSI conformance goal - this is a conformance fix, not a preference.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…eport

Every valgrind shard of the 2026-09-19 nightly failed - both HTTP backends,
all four ranges - on the same eight tests:

  FAIL: valgrind: report INCOMPLETE — broker killed before finish, cannot trust

The .vg is 508 bytes: valgrind's six header lines and nothing after them. All
eight PASS functionally; it is only the memory verdict that is missing.

Cause: their --TEARDOWN-- never calls coraineStop, so the harness kills the
broker instead of letting it exit, and valgrind never writes its summary. A
normal run cannot notice - the broker is killed either way and the assertions
have already passed - which is why they went in green and only the nightly
found them.

The eight are every test added on 09-17 and 09-18. A scan of all 666 finds
exactly three others without coraineStop, all legitimate: ftclient_smoke and
usage start no broker, and feature_builds starts its own outside the harness
and kills them by pid file.

Verified as the rule says: reverted the teardowns, ran one under -vt, watched
it fail with the nightly's exact message, restored, ran all eight under -vt -
green, no leaks.

⭐ Worth a harness guard: a test that calls coraineStart and never coraineStop
is malformed the same way a missing TEARDOWN marker is, and corTest already
refuses the latter.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…kwards

The performance job failed on 09-18 and 09-19 and on no nightly before that.
It is not a regression: the broker was 80-180% FASTER than the median on
every throughput metric in the run that went red.

perfCompare computed `change = (now - ref) / ref * 100` for every metric and
failed at -50%. Right for requests/second, exactly backwards for the `_p99us`
p99 latency metrics. A fast runner raises throughput AND lowers p99, so both
halves of each pair say "this runner was fast" and one of them is read as a
collapse:

  query_c200        8103 vs  4496   +80.2%
  query_c200_p99us 31170 vs 62665   -50.3%  ❌
  retrieve_c50     27037 vs 12150  +122.5%
  retrieve_c50_p99us 3330 vs 7450   -55.3%  ❌

⭐ The dangerous half is the other direction: with the sign unhandled, a p99
that TRIPLES scores +200% and the gate stays green. The gate was loud about
improvements and blind to the regression it exists to catch.

Thresholds are now applied to a direction-corrected delta; the table still
prints the true change, plus a "latency, lower is better" note on those rows.
Verified both ways against the recorded history: the run that went red now
exits 0 ("within noise"), and a synthetic run with p99 tripled and throughput
unchanged exits 1.

Why it started on 09-18: the _p99us metrics were added on 09-16 and WINDOW=5
needs history before a metric is compared at all. A shared runner moves ~2x
between runs, so this would have fired on roughly every other night.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
C8 the closed selector list of § 10.4.3.4 and the three disagreeing copies,
C9 the eight tests that never stopped their broker, C10 the perf gate reading
a latency improvement as a collapse. Plus the two upstream PRs that carry the
tutorial half - CRUD-Operations #24 and Concise-Format #3 - and why attrs
could not be restored (it cannot be combined with pick).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kzangeli
kzangeli merged commit 809437f into main Sep 19, 2026
8 checks passed
@kzangeli
kzangeli deleted the fix/nightly-20260919-reds branch September 19, 2026 12:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant