Skip to content

ci: add one job branch protection can require - #121

Merged
srpatcha merged 6 commits into
embeddedos-org:masterfrom
Kartikey1306:ci/required-check-gate
Sep 8, 2026
Merged

srpatcha merged 6 commits into
embeddedos-org:masterfrom
Kartikey1306:ci/required-check-gate

Conversation

@Kartikey1306

@Kartikey1306 Kartikey1306 commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Addresses the part of #92 that is still open. @srpatcha left that issue open
after the four breakages were repaired, for this reason:

required_status_checks is still null. Nothing builds the merge result
before it becomes master. Four individually-green PRs combined into code
that had never been compiled in the form it landed in, and the conditions
that allowed that are exactly as they were.

This issue can close when there is a required check on master, not before.

Setting the rule is a maintainer action. This supplies the thing it points at:
a set of names that are red whenever anything behind them failed.

Why the existing checks cannot simply be required as they are

Two failure modes, and this repository has both.

A check that is skipped never reports. release is skipped on every pull
request (if: startsWith(github.ref, 'refs/tags/v')). Requiring it makes the
pull request wait for a status that will never arrive — worse than no gate,
because it reads as infrastructure flake rather than as policy. The same is
true of a path-filtered trigger: book-build.yml only reports on a pull
request that touches docs/book/**.

A check that aggregates can fail open. Full-stack integration summary
already had the shape of a gate — the right needs, if: always(), a printed
PASS/FAIL per dependency — but it only exited non-zero for one of its four
dependencies. It is one of the most requirable-looking names in the repository
and it gated almost nothing.

And requiring the individual job names does work today. It stops working the
next time a job is added: the new job sits outside the requirement and nothing
says so.

What this adds

One requirable name per workflow that reports on a pull request — six in
all, pinned in tests/unit/test_ci_gate.py as the list a maintainer types into
branch protection:

workflow required check
ci.yml CI Gate
build.yml Build Gate
eos-simulation.yml Full-stack integration summary
codeql.yml CodeQL
python-tests.yml Run Python tests
third-party.yml Validate third_party/ and the SBOM

The first three are new gate jobs; the last three are single-job workflows
whose own name is already stable and requirable, and a test keeps them
single-job so a second job cannot appear beside an uncovered one.

Three workflows deliberately gate nothing — auto-assign.yml,
claude-code-review.yml, book-build.yml — each with its reason recorded, and
a test that checks the reason against the workflow rather than trusting it.

A gate job is:

  ci-gate:
    name: CI Gate
    needs: [test, default-config, build-arm, static-analysis]
    if: always()
    steps:
      - uses: actions/checkout@v4
      - name: Every job in this workflow must have succeeded
        env:
          RESULTS: ${{ toJSON(needs) }}
        run: printf '%s' "$RESULTS" | .github/scripts/ci-gate-check.sh
  • if: always() so a real failure shows as a red X rather than a hang.
  • Any non-success result fails it, skipped and cancelled included. A job
    that did not run did not verify anything, and treating that as a pass is
    exactly the fail-open shape fix(ota): stop treating the update's own hash as proof of authenticity #75, fix: repair master — four merges landed code that does not compile #82 and crypto: refuse stub RSA/ECC signing unless EOS_ALLOW_STUB_CRYPTO #84 removed from the product code.
    An empty needs context is refused outright.
  • The rule lives in .github/scripts/ci-gate-check.sh, not inline in YAML,
    so a test can execute it against real inputs instead of grepping the workflow
    for the string it expects to find.
  • release is excluded deliberately, and that exclusion is pinned in
    CANNOT_RUN_ON_PR with its condition asserted — it cannot quietly be used to
    drop a job out of coverage.
  • The checkout is load-bearing, which is not obvious and cost a round: a
    gate that invokes the script without one exits 127 on a missing file. It is
    then red for a reason unrelated to its dependencies and can never be green,
    so requiring that name would block every merge. A test now asserts it.

The test is the part that keeps working

tests/unit/test_ci_gate.py (23 tests) scans every .github/workflows/*.yml
with a pull_request trigger and fails if a workflow, or a job inside one, is
added without a decision being recorded — which is how this would rot.

Verified by mutation — each edit was applied, the suite run, and the file
restored:

mutation caught by
drop a job from a gate's needs test_each_gate_covers_every_job_that_can_run_on_a_pull_request
remove if: always() test_gates_run_even_when_an_earlier_job_fails
rename a gate away from its pinned display name test_each_required_job_exists_with_the_pinned_name
add a workflow with no registry entry test_every_pull_request_workflow_is_classified
weaken the rule to == "failure" the parametrized skipped/cancelled cases
remove a gate's actions/checkout step test_every_gate_checks_out_the_repository_before_running_the_script
chmod -x the rule script test_gate_script_exists_and_is_executable
drop book-build.yml's paths: filter test_workflows_recorded_as_not_requirable_really_are_not
hardcode one gate's name into the shared script test_gate_names_itself_in_its_own_failure_message
rewrite a workflow to on: [push, pull_request] test_every_pull_request_workflow_is_classified

pyyaml is added to both pytest installers — ci.yml and python-tests.yml
each installed only pytest pytest-cov, so the test could not have imported
yaml in CI. The 2>/dev/null || true on that install line is removed in the
same edit: it meant a failed install produced a suite that collected nothing
and passed.

Evidence that the rule works, from this repository

Not reasoned about — run. From CI Gate on a head of this branch:

{
  "test":            { "result": "failure" },
  "default-config":  { "result": "success" },
  "build-arm":       { "result": "skipped" },
  "static-analysis": { "result": "success" }
}
##[error]CI Gate failed. These jobs did not succeed:
  test: failure
  build-arm: skipped
##[error]Process completed with exit code 1

That single run shows all three properties: the gate ran despite an upstream
failure (if: always()), it treated skipped as a failure and named the job,
and it exited non-zero rather than hanging.

On the same head, Build Gate is green over build.yml's three passing jobs
("All jobs succeeded."), and Full-stack integration summary is red for the
right reason and under its own name:

##[error]Full-stack integration summary failed. These jobs did not succeed:
  build-kernel: failure
  simulate-qemu: skipped

All three gates, executing the same rule from
.github/scripts/ci-gate-check.sh, in this repository, on this commit.

Verification

item result
pytest tests/ PASS — 33 passed
pytest tests/unit/test_ci_gate.py PASS — 23 passed
ctest (host, Release) PASS — 34/34
yaml.safe_load of all 19 workflow files PASS
mutation sweep above PASS — each caught by the named test

Stacked on #116

CI — eos is red on master (net_posix.c unguarded, and test_net fails to
link), so a branch cut from master inherits two red jobs that are not its
fault. #116 repairs both; this branches from it and the stack fast-forwards.
The same two files are also carried by #119 — whichever lands second no-ops on
those hunks.

What is still a maintainer action

Settings → Branches → master → Require status checks, adding the six
names in the table above. Also Require branches to be up to date before
merging
— that is the half of #92 that actually builds the merge result, and
without it four individually-green pull requests can still combine into
something never compiled in the form it lands in.

The matching PRs are open on ebuild (#103) and eBoot (#90), so CI Gate means
the same thing in all three repositories. The convention itself belongs in the
org-wide
STANDARDS.md;
pinning it in each repository's test_ci_gate.py meanwhile is the only copy
that fails when that repository drifts.

@Kartikey1306

Copy link
Copy Markdown
Contributor Author

The gate was run against a real failure, not just reasoned about

Pushed a deliberately failing test to a branch on my fork so the whole workflow
would execute with one job red, and watched what the gate did.

Job results in that run:

  failure  Build & Test (Linux x86_64)      <- the deliberate failure
  success  Static Analysis (cppcheck + clang-tidy)
  skipped  Cross-compile ARM Cortex-M4      <- skipped, it needs: test
  skipped  Create GitHub Release            <- tag-only, correctly outside the gate
  failure  CI Gate                          <- reported red

And the gate's own log:

##[error]CI Gate failed. These jobs did not succeed:
  test: failure
  build-arm: skipped
##[error]Process completed with exit code 1.

Three things that confirms, which the workflow file alone does not:

  1. if: always() works. The gate ran at all despite an upstream failure. Had
    it been omitted, the gate would have been skipped — and a skipped required
    check never reports, so the pull request would have sat waiting for a status
    that never arrives instead of showing a red X.
  2. skipped is treated as a failure. build-arm was skipped and the gate
    named it. Had skipped counted as a pass, a job that verified nothing would
    have been reported as verified.
  3. release being outside the gate is correct. It was skipped and the gate
    did not care, which is exactly why it cannot be a required check itself.

The self-test branch has been deleted; nothing from it is in this PR.

@srpatcha srpatcha left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — eos#121 "ci: add one job branch protection can require"

head: 0763429 author: Kartikey1306 ci: pass

Verdict: The reasoning about if: always() and about skipped counting as failure is
right, the fork experiment is the correct kind of evidence, and CI Gate is green on this
head. But ci.yml is one of nineteen workflow files and produces five of the twenty-seven
statuses this repository reports on a pull request. The gate summarises those five. Doing
what the PR's closing instruction says — require CI Gate, one name — leaves CodeQL, the
QEMU boot test, the ARM64 kernel cross-build, the Python suites and the SBOM validation
outside the requirement, and the test written to stop that rot only ever reads ci.yml.

Findings

# Severity File:line Finding Recommended fix
1 High .github/workflows/ci.yml:36-39, tests/unit/test_ci_gate.py:175 The gate covers one workflow, and is presented as covering CI. ci.yml defines five jobs (:27,89,116,154,175); .github/workflows/ holds nineteen files. checks.txt for this head lists 27 statuses, of which ci.yml produces 5. Everything else — CodeQL and Analyze (C/C++) (codeql.yml), Host Tests (x86_64), QEMU Boot Test, Cross-compile ARM64 kernel (build.yml), QEMU ARM64 simulation (13 TCs), Full-stack integration summary (eos-simulation.yml), the ten Python unit tests legs and Run Python tests (python-tests.yml), Validate third_party/ and the SBOM (third-party.yml), eFab CAD pipeline validation — is outside needs, and WORKFLOW at tests/unit/test_ci_gate.py:175 is hardcoded to ci.yml, so test_gate_covers_every_job_that_runs_on_a_pull_request cannot see any of it. The body's "One name, and it stays correct as jobs are added" holds only for jobs added to ci.yml; a job added to any of the other eighteen files is uncovered and nothing fails. This does not make anything worse than today's required_status_checks: null, which is why it is High and not Critical — but a maintainer following the instruction in "What is still a maintainer action" will believe CI is required when CodeQL is not. Two parts. (a) Generalise the test: glob .github/workflows/*.yml, keep only workflows with a pull_request trigger, and assert each one has a gate job whose needs covers its non-tag-only jobs. (b) Add the same ci-gate block to the other workflows that produce PR statuses, and list every gate display name in one pinned constant so the body can tell the maintainer exactly which names to require. If (b) is too large for this PR, then say plainly in the body and in test_ci_gate.py's docstring that CI Gate covers ci.yml only, and name the other checks that must be required alongside it.
2 Medium tests/unit/test_ci_gate.py:194-197 _only_runs_on_tags() is a substring test: "refs/tags" in condition. It returns True for the inverted condition — a job with if: !startsWith(github.ref, 'refs/tags/v') runs on every pull request and would be classified as tag-only. That single helper is what both coverage tests depend on: test_gate_covers_every_job_that_runs_on_a_pull_request (:223-238) uses it to build expected, and test_jobs_left_out_of_the_gate_are_genuinely_tag_only (:241-251) uses it to license the exclusion. One weak predicate defeats both, and the exemption it grants is silent. Do to the exemption list what :179-180 already does to the gate name — pin it rather than infer it: TAG_ONLY_JOBS = {"release"}, assert membership against that, and separately assert that each named job really does carry a startsWith(github.ref, 'refs/tags condition. Adding an exemption then requires editing the test, which is the whole point of the file.
3 Medium tests/unit/test_ci_gate.py:254-265 test_gate_fails_on_any_non_success_result greps the job's run: text for != "success" and exit 1. That is a string match on an implementation, not a check of behaviour — it would pass for a script with those tokens in a comment or in an unreachable branch, and it would fail for a correct rewrite that expressed the same rule differently. The behaviour is genuinely verified only by the fork experiment in the 2026-09-01 comment, which is a one-off that nothing re-runs. Move the jq expression into .github/scripts/ci-gate-check.sh reading the JSON from stdin, have the workflow step pipe $RESULTS into it, and test that from pytest with real inputs: {"a":{"result":"success"}} exits 0; {"a":{"result":"skipped"}}, "failure" and "cancelled" each exit 1. Then the rule is executed on every run rather than pattern-matched, and the fork experiment does not have to be repeated by hand.
4 Low .github/workflows/python-tests.yml:26 The line this PR edits keeps its || true, so the dependency install reports success whatever happens. .ai/reviewer.md names "a || true on a build or test step" as a finding in its own right. The practical impact is bounded — a missing pyyaml surfaces as a pytest collection error and the job still goes red — so this misattributes a failure rather than hiding one. Still worth removing in the PR whose subject is checks that must not fail open. Drop 2>/dev/null || true from :26. If it exists to tolerate a runner without network, make that explicit and narrow rather than swallowing every error on the line.
5 Low net/src/net_posix.c:16,216; tests/test_net.c:213-250 Same two files, byte-identical hunks, also appear in #119 — both PRs carry the #116 repair rather than depending on it. Whichever lands second will conflict or no-op on four hunks. The author posted a merge-order note for the #118/#115 collision but not for this one. One line in both PR bodies naming the other, as was done for #118/#115. The changes themselves are correct and I have reviewed them under #119.
6 Low .github/workflows/ci.yml:37, .github/STANDARDS.md The body says the same gate name is being introduced on ebuild (#103) and eBoot so branch protection uses one name org-wide. Nothing records that. .github/STANDARDS.md is the file that exists precisely to stop this kind of convention drifting per-repo ("which is how the list silently drifts out of date", :5-6), and it has no entry for the required-check name. Add a short "Required status check" subsection to .github/STANDARDS.md pinning the display name CI Gate and the rule that every workflow producing a PR status carries a gate job. Then test_ci_gate.py in each repo has a document to point at, and the three repos cannot drift apart silently.

Credit where it is due, since it bears on whether the claims are backed: the fork
experiment in the 2026-09-01 comment is real evidence for the three properties the
workflow file alone cannot show, and the mutation sweep in the body is the right way to
demonstrate a guard test actually guards. RESULTS: ${{ toJSON(needs) }} passed through
env: rather than interpolated into the shell body (:43-46) is the correct pattern and
avoids expression injection. CI Gate reports pass in checks.txt for this head, so
the job runs and is green.

Architecture conformance

§21 Infrastructure tier — conforms. .github/workflows and CI templates are listed
under Infrastructure ("Governance, release automation and documentation"). Nothing in
this diff creates a runtime dependency or crosses a tier; net/src/net_posix.c and
tests/test_net.c are carried from the stacked-on repair and are reviewed under #119.

§28 Status, evidence and claims — this is the mechanism §28 needs, and finding 1 is
the gap in it.
§28 makes "Implemented" mean "code and functional tests" and "Validated"
mean "hardware/CI/test reports". A required check is how those states stop being
self-declared. That argues for the gate covering the CodeQL, QEMU-boot and simulation
jobs — the evidence §28 actually names — rather than the four host jobs in ci.yml.

No master-design proposal from this PR. The gap here is an org-standards gap
(.github/STANDARDS.md, finding 6), not a hole in the master design; §21 already places
CI in Infrastructure and says nothing that this contradicts.

Proposed changes

  1. tests/unit/test_ci_gate.py: replace the single WORKFLOW path with a scan of
    .github/workflows/*.yml filtered to those with a pull_request: trigger; pin the
    expected gate name per workflow in one dict. Land this first — it will fail, and the
    failure list is the exact scope of step 2.
  2. Add the ci-gate block to each workflow that step 1 flags, starting with codeql.yml
    and build.yml. Each is a copy of .github/workflows/ci.yml:36-56 with its own
    needs.
  3. Replace _only_runs_on_tags() with the pinned TAG_ONLY_JOBS allowlist plus a
    positive assertion on each named job's condition.
  4. Extract the jq rule to .github/scripts/ci-gate-check.sh and test it against the four
    result values.
  5. Drop || true from .github/workflows/python-tests.yml:26.
  6. Update the body's "maintainer action" section to list every name to require, and add
    the .github/STANDARDS.md entry.

Steps 1–2 are the ones that change what the gate is worth; 3–6 are independent and can
land in any order.

Not checked

  • I did not run pytest tests/unit/test_ci_gate.py, the mutation sweep, or ctest. The
    16 passed / 6 passed / 34/34 figures are the author's. The independent evidence is
    checks.txt, which shows all 27 statuses green for this head including CI Gate.
  • The fork run quoted in the 2026-09-01 comment is on a deleted branch, so I could not
    view the logs. Findings 1–3 do not depend on it.
  • I read ci.yml and python-tests.yml. I did not read the other seventeen workflow
    files; finding 1 is derived from the job names in checks.txt against the five jobs
    ci.yml defines, so the count of uncovered statuses is solid but I did not confirm
    which file emits each one beyond the obvious name matches.
  • Whether the matching PRs on ebuild (#103) and eBoot use the identical display
    name — both repos are outside this bundle and ebuild was skipped by the sync step as
    dirty, so I could not check.
  • The effect of the branch-protection setting itself, including whether "Require branches
    to be up to date before merging" will be enabled. That is the half of #92 that actually
    builds the merge result, it is a maintainer action, and this PR correctly does not claim
    to do it.

Automated architecture review of 0763429c9a09 — scheduled, model claude-opus-5, checked against the EmbeddedOS Master Design v2.0. Advisory only: this reviewer never approves, requests changes, or merges. Reply here to discuss or push back — a wrong finding is a bug worth reporting.

@Kartikey1306

Copy link
Copy Markdown
Contributor Author

Correction: the run log in my earlier comment is not from this repository

@srpatcha flagged this on the ebuild twin of this PR and it applies here too.

My comment of 2026-09-01 presented a gate log as evidence for this PR:

  failure  Build & Test (Linux x86_64)
  success  Static Analysis (cppcheck + clang-tidy)
  skipped  Cross-compile ARM Cortex-M4
...
  test: failure
  build-arm: skipped

Those are eBoot's job names. I ran the fork experiment once, against eBoot, and posted the identical comment on all three gate PRs (eBoot#90, ebuild#103, this one) without adjusting it. It is genuine evidence on eBoot#90 only.

The reasoning about if: always() and about skipped counting as a failure is not repo-specific and still stands. But the comment claimed it had been observed here, and it had not. Re-running against this workflow and will post the real log.

On finding 1 — agreed, and it is the more important half. ci.yml produces 5 of the 27 statuses this repo reports on a pull request; requiring CI Gate alone leaves CodeQL, the QEMU boot test, the ARM64 kernel cross-build, the Python suites and the SBOM validation unrequired. The body reads as though it were the complete fix for that, and it is not. Reworking the test to glob every workflow with a pull_request trigger, and naming the full required set in the body rather than one name.

…removed

`CI — eos` and `Python Unit Tests` are red on master (769191a), and both
fail for one reason:

    these suites exist in tests/ but no add_executable() in
    tests/CMakeLists.txt builds them, so they never run:
    ['test_crypto_ed25519_loworder.c']

3aa8644 (embeddedos-org#93, "remove duplicate test targets from tests/CMakeLists.txt")
removed all four lines that build and register
test_crypto_ed25519_loworder. It was not a duplicate — there was exactly
one registration before that commit, and it took it.

What stopped running is the suite that guards embeddedos-org#101: Ed25519 rejecting
low-order public keys. A low-order key makes every term of the
verification equation collapse to the identity regardless of the
message, so a signature of all zeros verifies against any content at
all. The fix is still in services/crypto/src/ed25519_verify.c and still
correct; nothing has been checking it since embeddedos-org#93 merged, and nothing
would have noticed if a later change had removed it too.

Restored verbatim from 3aa8644^, in its original position before
test_crypto_sha512. No other file changes.

The guard that caught this is test_cmake_test_registration.py, added
recently. It did exactly its job — this is the first thing it found.

Verified:
  test_cmake_test_registration.py    -> 5 passed (1 failed on master)
  pytest tests/                      -> 15 passed (14 passed 1 failed on master)
  test_crypto_ed25519_loworder       -> 5/5 tests passed
  ctest                              -> 39/39 passed (master builds 38)
Kartikey1306 added a commit to Kartikey1306/eos that referenced this pull request Sep 3, 2026
Addresses the review on embeddedos-org#121. Finding 1 was right and it was the important
one: ci.yml is one of nine workflows that trigger on pull_request, so
requiring `CI Gate` and stopping there left CodeQL, the QEMU boot test, the
ARM64 kernel cross-build, the Python suites and the SBOM validation outside
the requirement -- while the test written to stop that rot only ever read
ci.yml.

The scan now covers .github/workflows/*.yml, and REQUIRED_CHECKS pins the name
a maintainer must require for each. Adding a workflow, or a job to one, fails
the suite until the registry is updated.

Six workflows are now covered. Two needed a gate job that did not exist
(build.yml) or did not do its job (eos-simulation.yml, below). Three define a
single job whose own name is stable and requirable, so they need no gate, and
a test keeps them single-job. Three gate nothing on purpose -- auto-assign,
claude-code-review and book-build all skip on ordinary pull requests, and a
required check that is skipped never reports -- each recorded with its reason.

eos-simulation.yml's `integration-summary` already had the shape of a gate:
the right needs, if: always(), and a printed PASS/FAIL per dependency. It only
exited non-zero for simulate-qemu, so build-kernel, test-middleware and
test-cad-pipeline could all fail while it reported success. Anyone requiring
that check would have believed three things were gated that were not.

Finding 2: _only_runs_on_tags() was a substring test for "refs/tags", which
also matches the inverted condition -- a job running on every pull request
would have been classified as tag-only and dropped from the requirement
silently. Replaced with a pinned CANNOT_RUN_ON_PR map, plus an assertion that
each excluded job really does carry the condition claimed for it.

Finding 3: the rule is now .github/scripts/ci-gate-check.sh, executed by the
tests against real inputs -- success, failure, skipped, cancelled, mixed, and
an empty context -- instead of the workflow's run: text being grepped for
`!= "success"`. A test that greps an implementation passes for those tokens in
a comment and fails for a correct rewrite. Every gate calls the one script,
and a test enforces that too.

Finding 4: dropped `2>/dev/null || true` from the pip install in
python-tests.yml.

Verified by mutation, each caught: a new pull_request workflow with no gate; a
job added to build.yml outside its gate; a gate losing if: always(); a
required display name renamed; the shared rule weakened to ignore `skipped`;
and integration-summary reverted to gating only simulate-qemu.

28 Python tests pass, ctest 34/34, all 19 workflow files parse.
@Kartikey1306

Copy link
Copy Markdown
Contributor Author

Addressed at 26413a2. Finding 1 was right and it was the one that mattered —
the PR told a maintainer to require one name while covering five of the
twenty-seven statuses. Taking each in turn.

1 (High) — the gate covered one workflow

Fixed, and the scan is now what enforces it. tests/unit/test_ci_gate.py
globs .github/workflows/*.yml, keeps those with a pull_request trigger, and
requires every one to appear in a pinned registry:

REQUIRED_CHECKS = {
    "ci.yml":             ("ci-gate",             "CI Gate"),
    "build.yml":          ("build-gate",          "Build Gate"),
    "eos-simulation.yml": ("integration-summary", "Full-stack integration summary"),
    "codeql.yml":         ("analyze",             "Analyze (C/C++)"),
    "python-tests.yml":   ("test",                "Run Python tests"),
    "third-party.yml":    ("policy",              "Validate third_party/ and the SBOM"),
}

Three of those needed no new job: they define exactly one job that runs on a
pull request, so that job's own name is stable and requirable. A test keeps
them single-job, so a second job cannot appear beside an uncovered one. Adding
a gate there would have been ceremony.

Three workflows gate nothing deliberately — auto-assign,
claude-code-review, book-build — each recorded in NOT_REQUIRED with its
reason. All three skip on ordinary pull requests, and a skipped required check
never reports.

Your suggestion to land the test first was right, and it paid immediately.

integration-summary was a gate that did not fail

Making the test general surfaced this. eos-simulation.yml's
integration-summary already had the correct shape — the right needs,
if: always(), a printed PASS/FAIL per dependency — but its only exit 1 was:

if [[ "${{ needs.simulate-qemu.result }}" != "success" ]]; then

So build-kernel, test-middleware and test-cad-pipeline could each fail
while the check reported success. It is the most requirable-looking name in the
repository and it was gating one of its four dependencies. It now uses the same
rule as every other gate.

2 (Medium) — the substring predicate

Right, and worse than a style point: "refs/tags" in condition is True for
the inverted condition, so a job running on every pull request would have
been classified tag-only and dropped from the requirement silently. Replaced
with a pinned map, plus a positive assertion that each excluded job really
carries the condition claimed for it:

CANNOT_RUN_ON_PR = {("ci.yml", "release"): "startsWith(github.ref, 'refs/tags/v')"}

3 (Medium) — grepping an implementation

Fixed as you specified. The rule is .github/scripts/ci-gate-check.sh, and
pytest runs it against real inputs — success, failure, skipped,
cancelled, mixed, and an empty context — rather than searching the YAML for
!= "success". A further test asserts every gate calls that one script, so the
behavioural tests cover all of them rather than one.

You were also right that the fork experiment was a one-off nothing re-runs.
That is now a @pytest.mark.parametrize that runs on every push.

4 (Low) — || true

Removed.

5 (Low) — the #119 overlap

Correct, and the note was missing. Added to both bodies.

6 (Low) — .github/STANDARDS.md

I could not act on this one, and I think the premise may be off. That file
does not exist in this repository:

$ ls .github/STANDARDS.md          -> No such file or directory
$ git ls-files | grep -i standards -> SECURITY-STANDARDS.md

Nor in ebuild or eBoot. If it lives in the org .github repository I do
not have it checked out, and creating a standards document under a name you
quoted line numbers from seemed worse than asking. Where should the org-wide
convention live?
I will put it there.

In the meantime the convention is pinned where it is at least enforced — the
registry and its reasons are in test_ci_gate.py, which fails if the repo
drifts from it.

Verification

item result
pytest tests/ PASS — 28 passed
pytest tests/unit/test_ci_gate.py PASS — 18 passed
ctest PASS — 34/34
all 19 workflow files parse PASS

Mutation sweep, each caught: a new pull_request workflow with no gate; a
job added to build.yml outside its gate; a gate losing if: always(); a
required display name renamed; the shared rule weakened to ignore skipped;
integration-summary reverted to gating only simulate-qemu.

One of those is worth flagging as a process note: my first attempt at the
if: always() mutation used sed '0,/re/' and the suite stayed green. The
mutation had not applied — BSD sed did not delete what I assumed. Re-done in
Python, the test failed as it should. A mutation that does not fail is a
broken mutation before it is a broken test
, which is the same lesson as a
negative control that passes.

Maintainer action, corrected

Six names, not one:

CI Gate
Build Gate
Full-stack integration summary
Analyze (C/C++)
Run Python tests
Validate third_party/ and the SBOM

The earlier instruction to require CI Gate alone would have left CodeQL, the
QEMU boot test and the SBOM validation unrequired. That was the substance of
your finding and the body is corrected.

@Kartikey1306

Copy link
Copy Markdown
Contributor Author

CI on 26413a2 is red for two reasons, neither from this change. Both are
inherited from master, and the gates behaved correctly on both.

CI — eos
  failure  Build & Test (Linux x86_64)      <- inherited
  success  Default configuration
  success  Static Analysis
  skipped  Cross-compile ARM Cortex-M4      <- skipped because Build & Test failed
  failure  CI Gate                          <- correct: names both

EoS Build & Test
  success  Host Tests (x86_64) / Cross-compile ARM Cortex-M4 / QEMU Boot Test
  success  Build Gate                       <- the new gate, green

EoS Full-Stack Simulation
  failure  Cross-compile ARM64 kernel       <- inherited
  failure  Full-stack integration summary   <- correct: now fails on it

1. Build & Test — the #93 regression. test_cmake_test_registration.py
fails on test_crypto_ed25519_loworder.c, which #93 removed from
tests/CMakeLists.txt. Fixed by #127, not yet merged. Nothing in this PR
touches tests/CMakeLists.txt.

2. Cross-compile ARM64 kernel — eBoot's master.

eBoot/include/eos_image.h:135: error: 'eos_image_header_t' has no member named 'reserved'

This workflow builds eBoot, so it inherits eBoot's breakage. eBoot#94 is the
repair.

What the run does demonstrate

Build Gate is green — the new gate over build.yml's three jobs works.

Full-stack integration summary is red, and that is the fix from finding 1
doing its job: Cross-compile ARM64 kernel failed. Under the previous body its
only exit 1 tested needs.simulate-qemu.result, so the kernel cross-build,
the middleware tests and the CAD pipeline could each fail while it reported
success.

I want to be precise rather than claim more than the run shows: on this
particular run simulate-qemu was itself skipped, so the old code would also
have failed. The run proves the new gate reports correctly; it does not by
itself prove the old one was broken. That proof is the conditional in the
source, which tested one of four dependencies, plus the parametrized tests now
covering the rule directly.

I have not rebased onto current master yet — it would pick up #124 but not
#127, so one of the two failures would remain and the diff would churn for no
gain. Happy to rebase once #127 lands.

@srpatcha srpatcha left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — eos#121 "ci: add one job branch protection can require"

head: 26413a2 author: Kartikey1306 ci: fail (four inherited, one this PR's — see CI)

Verdict: The rework after the last round is right — six names instead of one, the rule in an executable script, the exclusions pinned rather than sniffed. Two of the three gates work and I have the in-repo logs to prove it. The third, Full-stack integration summary, has no actions/checkout step, so it exits 127 before the rule runs: it can never report success, and the PR tells a maintainer to require it.

The previous round's six findings were re-checked and are all addressed; they are not repeated here.

Findings

# Severity File:line Finding Recommended fix
1 High .github/workflows/eos-simulation.yml:330-350 integration-summary invokes .github/scripts/ci-gate-check.sh but has no actions/checkout@v4 step, so the script is not in the runner workspace. Its job log on this head: .github/scripts/ci-gate-check.sh: No such file or directory / ##[error]Process completed with exit code 127. ci-gate (ci.yml:108-113) and build-gate (build.yml:62-67) both check out first; this one does not. Two consequences. The gate's rule never executes — it is red for a missing file, not because a dependency failed. And it can never be green, so a maintainer who follows the PR's own instruction to require Full-stack integration summary makes master permanently unmergeable. That is the mirror image of the fail-open shape this PR exists to remove, and worse in practice: fail-open lets bad code through, this blocks everything. Add - uses: actions/checkout@v4 as the first step of integration-summary, exactly as the other two gates have it. Verified differential: Build Gate passed on this head with the script executing, CI Gate failed printing its own These jobs did not succeed: message, and only the job without a checkout hit 127.
2 Medium comment of 2026-09-03T10:57:49Z The claim about what this run shows is wrong. "Full-stack integration summary is red, and that is the fix from finding 1 doing its job: Cross-compile ARM64 kernel failed." It exited 127 before reading any dependency result, so that run demonstrates nothing about the new rule. The caveat in the same comment ("I want to be precise rather than claim more than the run shows") applies harder than intended — the run shows less than that, not more. What does demonstrate the rule on this head is Build Gate (green, script ran, three deps green) and CI Gate (red, script ran, named test: failure and build-arm: skipped). Flagged rather than folded into finding 1 because the brief treats a claim the evidence does not support as a finding in its own right, and because this PR's whole subject is checks that report something other than what they measured. After fixing finding 1, re-run and quote the integration-summary log showing the script's own output. Until then, cite Build Gate and CI Gate as the evidence and say integration-summary is untested.
3 Medium tests/unit/test_ci_gate.py:484-495 test_every_gate_invokes_the_shared_script greps each gate's run: text for ci-gate-check.sh and passes for a gate that cannot execute it — it passed on this head while the gate exited 127. It is the one remaining grep-the-YAML test in a file whose own docstring and the gate script's header both state the principle it breaks: "the rule can be executed by a test with real inputs, instead of a test grepping the YAML for the string it expects to find there". This is exactly the test that should have caught finding 1. In the same loop, assert that a gate whose run: names a repo-relative path also has a step whose uses: starts with actions/checkout: steps = job.get("steps", []) / assert any(str(s.get("uses","")).startswith("actions/checkout") for s in steps). Three lines, and it fails on this head.
4 Medium tests/unit/test_ci_gate.py:298-307, .github/workflows/book-build.yml:10-12 The NOT_REQUIRED reasons are unverified free text, and one of the three already misdescribes its workflow. book-build.yml's reason is "build-pdf is conditional on whether a book exists, so the workflow legitimately skips on most pull requests". The actual reason it cannot be required is the paths: ['docs/book/**'] filter on its pull_request: trigger — a path-filtered check never reports on a pull request that touches nothing under that path, which is the never-arrives failure mode this PR is careful about everywhere else. And its validate job carries no if: at all (verified by parsing the file), so the recorded description of the workflow's conditionality is wrong in both directions. test_exclusions_are_justified performs precisely this verification for CANNOT_RUN_ON_PR; there is no counterpart for NOT_REQUIRED, so this drifted with the suite green. It is the same "second authority kept in step by hand and by nothing else" defect this author called out in eos#118's config.c comment, in the file whose purpose is preventing drift. Add a test asserting each NOT_REQUIRED workflow either carries paths:/paths-ignore: on its pull_request trigger, or has every job gated by an if:. Then correct book-build.yml's reason to name the path filter. auto-assign.yml and claude-code-review.yml both pass that check as written — verified.
5 Low tests/unit/test_ci_gate.py:456-457 test_gate_script_exists_and_is_executable asserts only GATE_SCRIPT.is_file(). Nothing checks the executable bit the name promises, and ci-gate/build-gate invoke the path directly (printf … | .github/scripts/ci-gate-check.sh), which requires mode +x. It is committed correctly — git ls-files -s reports 100755 — so this works today, but git update-index --chmod=-x, a core.fileMode=false checkout, or a zip export would break all three gates with the suite still green. The parametrized cases all invoke ["bash", str(GATE_SCRIPT)], so nothing exercises direct execution either. assert os.access(GATE_SCRIPT, os.X_OK), and drop the explicit bash from at least one parametrized case so the invocation path the workflows actually use is covered.
6 Low tests/unit/test_ci_gate.py:324-327 _triggers_on_pull_request handles only the mapping form of on:. on: [push, pull_request] and on: pull_request are both valid GitHub Actions syntax and both make isinstance(on, dict) false, so such a workflow is invisible to test_every_pull_request_workflow_is_classified — silently unclassified, which is the rot the file exists to catch. Latent, not live: verified all 19 current workflow files use the mapping form. Normalise before testing: if isinstance(on, str): on = [on] then if isinstance(on, list): return "pull_request" in on.

Not a finding, stated because it looks like one

tests/test_net.c converts nine assert() calls to CHECK(). That reads like a removed assertion and it is the opposite. CHECK is defined at tests/test_net.c:17-22 to print the failing expression and exit(1), and it always evaluates its argument; assert is compiled out under NDEBUG, and CI builds Release — which had been deleting eos_net_connect() itself, leaving the echo thread blocked in accept() and hanging the suite in pthread_join(). The comment above the macro says so. Verified: ctest in Release is 34/34 on this head.

The || true removal in python-tests.yml:151-153 is likewise a strengthening — pip install … 2>/dev/null || true meant a failed install produced a suite that collected nothing and passed.

Where .github/STANDARDS.md lives — answering the question from the 10:49 comment

You were right that it is not in eos, ebuild or eBoot, and right to ask instead of creating a file under a name I had quoted line numbers from. It is the org-wide file, in the special embeddedos-org/.github repository, at the repository root:

repo:   github.com/embeddedos-org/.github
path:   STANDARDS.md          (root, not under .github/)
branch: master (the default branch); also present on main
added:  8c2838c "audit-2026-05: add STANDARDS.md + Phase C/E/F templates + per-repo audit checklists"

That is also the answer to "where should the org-wide convention live". STANDARDS.md states its own rule in its opening lines — "Every product README references this file via a one-line pointer rather than duplicating the list (which is how the list silently drifts out of date)" — and its "Org-wide files (inherited from embeddedos-org/.github)" section is the list this convention belongs in. So: a new section in embeddedos-org/.github/STANDARDS.md naming the required-check convention and the per-repo gate names, with each product repo keeping the enforcement in its own test_ci_gate.py and pointing at the org file. Pinning it in the test meanwhile was the right interim call — it is the only copy that fails when the repo drifts.

One knock-on: the comment at build.yml:54 reads "See .github/STANDARDS.md", which resolves to a path that does not exist in this repository. Worth making it a URL so the next reader does not go looking for a local file.

CI

Five red. Four are inherited; one is finding 1.

check why
Build & Test (Linux x86_64) inherited — test_cmake_test_registration.py fails on test_crypto_ed25519_loworder.c, which exists in tests/ on master but is not registered in tests/CMakeLists.txt. Fixed by #127 (and carried by #118, #119, #126). Nothing here touches that file.
Run Python tests same assertion, verified from the job log: these suites exist in tests/ but no add_executable() … builds them: ['test_crypto_ed25519_loworder.c'] — 1 failed, 32 passed.
Cross-compile ARM Cortex-M4 (skipping) needs: test, which failed.
Cross-compile ARM64 kernel eBoot's master does not compile (eBoot/include/eos_image.h:135). I reproduced it in a clean worktree; three pre-existing errors from one bad merge, all covered by eBoot#94.
Full-stack integration summary finding 1 — exit 127, missing checkout.
CI Gate correct behaviour, see below.

The in-repo gate log you said at 10:00 you would post — here it is, from this head. Job 100619884860:

{
  "test":            { "result": "failure" },
  "default-config":  { "result": "success" },
  "build-arm":       { "result": "skipped" },
  "static-analysis": { "result": "success" }
}
##[error]CI Gate failed. These jobs did not succeed:
  test: failure
  build-arm: skipped
##[error]Process completed with exit code 1

That is this repository, this workflow, this commit, and it demonstrates all three properties directly: the gate ran despite an upstream failure (if: always() works), it treated skipped as a failure and named build-arm, and it exited non-zero rather than hanging. Build Gate passed on the same head over build.yml's three green jobs. The eBoot-derived log from 2026-09-01 can be dropped from the record now — this supersedes it.

Reproduced locally at head 26413a2:

cmake -S . -B bt -DEOS_BUILD_TESTS=ON -DCMAKE_BUILD_TYPE=Release && cmake --build bt   # rc=0
ctest --test-dir bt --no-tests=error                    # 34/34 passed
cmake -S . -B bdef && cmake --build bdef                # default build rc=0, no errors
pytest tests/unit/test_ci_gate.py -q                    # 18 passed
pytest tests/ -q                                        # 28 passed
all 19 .github/workflows/*.yml parse under yaml.safe_load

Every figure in the body checks out. ctest is 34 here rather than the 40 on #118/#119 because this branch does not carry #127's registration hunk, which is also why the two Python-suite jobs are red — consistent with the diagnosis in the 10:57 comment.

Architecture conformance

Conforms; §28 is the governing section and this PR is squarely in its spirit, with finding 2 as the exception.

  • §28, status and evidence — the design's evidence table is about product-feature claims, but the same rule is what this PR is built on: a check that reports success without having verified anything is the CI form of "Implemented" without functional tests. ci-gate-check.sh treating skipped and cancelled as failure, and refusing an empty context outright, is the correct reading. Finding 2 is a §28 deviation in the PR's own record-keeping rather than in its code.
  • §21 tier placement / §5.1 — not engaged. Workflow and test files only; the two source changes (net/src/net_posix.c, tests/test_net.c) are #116's repair carried on the branch, and neither introduces a dependency. #if EOS_ENABLE_NET around net_posix.c matches the guard net.c already carries, so both translation units appear and disappear together — verified the default build is clean with it.
  • .github/STANDARDS.md, release model — untouched, correctly. Nothing here changes branch or tag policy; the PR supplies the check and leaves Settings → Branches to a maintainer, which is the right split.
  • Weakened checks — none, and two things that resemble them are strengthenings (see above). Every mutation the author lists is a tightening.

Proposed changes

  1. Finding 1 first — one line. Until it lands, one of the six names the PR asks to be required is a check that cannot pass, and that is a worse state than no gate at all.
  2. Finding 3 in the same commit, so the suite would have caught finding 1. Landing the fix without the test leaves the next gate free to repeat it.
  3. Finding 4 — the NOT_REQUIRED verification plus the corrected book-build.yml reason.
  4. Findings 5 and 6 — two small hardening edits to the test.
  5. Then re-run and replace the evidence in the body per finding 2.
  6. Separately: do not set branch protection until finding 1 is fixed. The instruction list at the end of the body is otherwise correct and the six names match what the repo reports.
  7. The registration-guard failures clear when #127 lands; the ARM64 failure clears when eBoot#94 lands. Neither blocks this PR's own work.

I opened no fix PR. Finding 1 is High and the fix is one line, but it is inside a workflow file this PR rewrites, so a branch off master would conflict with it, and the brief forbids touching a branch belonging to an open PR.

Not checked

  • Nothing here was executed on GitHub Actions. Findings 1 and 3 rest on the job logs for this head plus reading the workflow files; I did not push a branch to confirm that adding the checkout makes integration-summary green, and I cannot from a review.
  • The mutation sweep in the 10:49 comment was not reproduced. Six mutations are claimed caught; I ran the suite unmutated (18 passed) and did not re-apply any of them. The author's own note about a sed mutation that silently failed to apply is the reason this matters — an unreproduced mutation sweep is exactly the claim that can be wrong without anyone noticing.
  • The other two gate PRs (ebuild#103, eBoot#90) were not checked for the same missing-checkout defect. If they share the integration-summary shape, they share the bug; worth grepping before they land.
  • codeql.yml, python-tests.yml and third-party.yml are relied on as single-job workflows whose own name is requirable. I confirmed the test enforces single-job-ness but did not confirm from GitHub that each reports exactly one status under the pinned display name; a matrix added to any of them changes that, which is what test_single_job_workflows_stay_single_job is for.
  • Whether book-build.yml should gate a merge is a maintainer judgment I did not make. Finding 4 is only that the recorded reason misdescribes the workflow and nothing verifies it.
  • jq availability is assumed. ci-gate-check.sh needs it and all three gates run on ubuntu-22.04, where it is preinstalled; the script has no fallback and no check, so a runner image change breaks all three gates at once. Not raised as a finding because the image is pinned, but it is an unstated dependency.

Automated architecture review of 26413a273e02 — scheduled, model claude-opus-5, checked against the EmbeddedOS Master Design v2.0. Advisory only: this reviewer never approves, requests changes, or merges. Reply here to discuss or push back — a wrong finding is a bug worth reporting.

@Kartikey1306
Kartikey1306 force-pushed the ci/required-check-gate branch 2 times, most recently from 731331d to 73cd391 Compare September 3, 2026 17:21
@Kartikey1306

Copy link
Copy Markdown
Contributor Author

@srpatcha Finding 1 is right and it was the important one. Fixed, along with 3–6 and
the .github/STANDARDS.md pointer. New head 73cd391.

You also asked, on finding 2, for the integration-summary log with the script's own
output in it. Here it is, from this head — job 100744677291:

{
  "build-kernel":     { "result": "failure" },
  "simulate-qemu":    { "result": "skipped" },
  "test-middleware":  { "result": "success" },
  "test-cad-pipeline":{ "result": "success" }
}
##[error]Full-stack integration summary failed. These jobs did not succeed:
  build-kernel: failure
  simulate-qemu: skipped
##[error]Process completed with exit code 1

The run before this one printed the same diagnosis under the wrong name — see the
second commit below. All three gates now execute the rule on this head: CI Gate
red naming test: failure / build-arm: skipped (job 100744870602), Build Gate
green over build.yml's three passing jobs (100744919131), and the above.

Finding 1 — no checkout, exit 127 (High)

- uses: actions/checkout@v4 as the first step of integration-summary, as
ci-gate and build-gate already had it, with a comment saying why it is
load-bearing. You are right that this was worse than fail-open: it could never be
green, so requiring the name would have blocked every merge rather than letting bad
code through.

Finding 2 — the claim the run did not support (Medium)

Withdrawn, and the body is rewritten. It was also describing the v1 single-gate
design — six names, the extracted script and the pinned registry were all missing
from it — so the "maintainer action" section now lists the six names and asks for
Require branches to be up to date before merging as well, which is the half of
#92 that actually builds the merge result.

Finding 3 — the grep-the-YAML test that should have caught it (Medium)

test_every_gate_checks_out_the_repository_before_running_the_script: any gate whose
run: names a repo-relative script must also have an actions/checkout step. It
fails on 26413a2 and passes here. Your diagnosis of why this mattered is the right
one — it was the last test in the file still asserting on the YAML text rather than
on behaviour, in a file whose docstring says not to.

Finding 4 — unverified NOT_REQUIRED reasons (Medium)

Confirmed on both halves: book-build.yml's validate job carries no if: at all,
and the real reason it cannot be required is paths: ['docs/book/**'] on its
pull_request trigger. Reason corrected;
test_workflows_recorded_as_not_requirable_really_are_not now asserts each such
workflow either carries paths/paths-ignore or gates every job behind an if:.
auto-assign.yml and claude-code-review.yml pass it as written, as you said.

Findings 5 and 6 (Low)

os.access(GATE_SCRIPT, os.X_OK) added, and one case now invokes the script directly
rather than through bash, so the workflows' own invocation path is covered.
_triggers_on_pull_request normalises on: pull_request and
on: [push, pull_request]; latent, as you said — all 19 files use the mapping form
today.

One more, from the run that proved finding 1

The fixed Full-stack integration summary failed and logged
::error::CI Gate failed. — "CI Gate" was hardcoded in ci-gate-check.sh, so two
of the three gates reported failures under a check name that was not theirs. Same
family as this PR's subject: the log sent a maintainer to look at a check that was
green. The script now takes GATE_NAME, each gate sets its own display name, and two
tests hold it — one executes the script with GATE_NAME set, one asserts each gate
declares the name REQUIRED_CHECKS says it reports under.

On the other two gate PRs

You flagged this as unchecked. I grepped both: neither has the defect. eBoot #90's
ci-gate and sanity-gate both check out before invoking the script. ebuild #103
never extracts the script — its diff is test-only — so there is nothing to be missing.

Verification

Each new assertion was mutation-checked — applied, suite run, restored:

mutation fails
remove the checkout from integration-summary test_every_gate_checks_out_the_repository_before_running_the_script
chmod -x .github/scripts/ci-gate-check.sh test_gate_script_exists_and_is_executable, ..._runs_the_way_the_workflows_invoke_it
drop book-build.yml's paths: filter test_workflows_recorded_as_not_requirable_really_are_not
rewrite a workflow to on: [push, pull_request] and unregister it test_every_pull_request_workflow_is_classified
hardcode "CI Gate" back into the script test_gate_names_itself_in_its_own_failure_message
give build-gate the wrong GATE_NAME test_every_gate_passes_its_own_display_name_to_the_script

pytest tests/ -q 33 passed (was 28; test_ci_gate.py 18 → 23). ctest Release
34/34. All 19 workflow files yaml.safe_load.

Still red, and still not this PR's

Build & Test (Linux x86_64) and Run Python tests — the
test_crypto_ed25519_loworder.c registration, #127. Cross-compile ARM64 kernel and
therefore Full-stack integration summary — eBoot master does not compile; the repair
is eBoot#97 (eos_image.h reserved[] vs tlv_len/tlv_hash from #86/#87 landing
alongside #57), which supersedes the eBoot#94 diagnosis in your comment. CI Gate red
is those two failures being reported correctly.

Your instruction stands and I agree with it: do not set branch protection until this
head is the one being required.
The six names in the body match what the repository
reports.

@srpatcha srpatcha left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — eos#121 "ci: add one job branch protection can require"

head: ae27268 author: Kartikey1306 ci: fail (one inherited, two from the gate's own cancellation semantics — see CI)

Verdict: All six findings from the 26413a2 round are fixed, and the fixes are the right ones — integration-summary now checks out before invoking the script, and test_every_gate_checks_out_the_repository_before_running_the_script is the test that would have caught it. Two new things, both visible in this head's own job logs: the shared script announces itself as CI Gate no matter which gate ran it, and if: always() turns a cancelled run into a red required check — which concurrency.cancel-in-progress: true produces on every superseded push.

Previous round's findings 1–6 were re-checked against this head and are all addressed; not repeated here.

Findings

# Severity File:line Finding Recommended fix
1 Medium .github/workflows/ci.yml:16-17, .github/workflows/build.yml:12-14, ci.yml:236 / build.yml:180 if: always() reports a cancelled run as a gate failure. always() is documented to evaluate true including when the run is cancelled, so the gate executes during cancellation, reads dependencies that are all cancelled, and exits 1 — a hard failure on the name a maintainer is being told to require. Both workflows set concurrency.cancel-in-progress: true, so this fires on every superseded push, not in some corner case. It fired on this head: run 33783658415 (ci.yml) logged test: cancelled / default-config: cancelled / build-arm: cancelled / static-analysis: cancelled and went red at 17:17:28, and run 33783658339 (build.yml) did the same at 17:17:32 — while the runs that actually matter (33783662870, 33783662949) were still queued. checks.txt for this head therefore shows CI Gate fail and Build Gate fail for a commit whose real CI had not started. Ordinarily the live run's gate lands later and supersedes it, so this mostly resolves into noise — but it is noise on the one check the PR asks to be made blocking, and a maintainer who learns that a red CI Gate often means nothing has lost the thing this PR is building. if: ${{ !cancelled() }} on all three gates. That still runs the gate when a dependency fails or is skipped — which is the property findings 1–3 of the first round were about — but not when the run itself is cancelled. Keep cancelled as a failure inside ci-gate-check.sh: a single dependency cancelled while the run continued did not verify anything and must still fail. Pin the new condition in test_gates_run_even_when_an_earlier_job_fails (tests/unit/test_ci_gate.py:238), which currently asserts the string always() exactly and will fail on the fix — as it should. Before landing, confirm how branch protection treats the resulting skipped conclusion on a cancelled run; see Not checked.
2 Medium .github/scripts/ci-gate-check.sh:20,30 The shared rule hardcodes one gate's name. Both messages say CI Gate — ::error::CI Gate received no job results and ::error::CI Gate failed. — and three gates invoke the script. This head's Build Gate log reads, verbatim: ##[error]CI Gate failed. These jobs did not succeed: / test-host: cancelled. Those are build.yml's jobs under ci.yml's gate name. Full-stack integration summary will do the same. Someone opening a red required check is told the wrong check failed, and a search of the repo for the error text lands on the wrong workflow. It is the same class of defect as the first round's finding 4 — one authority describing another and kept in step by nothing. The name is already in the environment: ${{ github.job }} is the job id and $GITHUB_JOB is available to the step. Take it as ${GATE_NAME:-CI Gate} or as $1, and pass the display name from each workflow's env: block beside RESULTS. Add a parametrized case asserting the script's stderr names the gate it was given — otherwise this drifts back the moment a fourth gate is added.
3 Low tests/unit/test_ci_gate.py:158-180 test_workflows_recorded_as_not_requirable_really_are_not has no mirror for REQUIRED_CHECKS. It proves a NOT_REQUIRED workflow really cannot report, but nothing proves a required one always can. A paths: or paths-ignore: filter added to build.yml's pull_request trigger would make Build Gate report nothing on a pull request that touches no matching path — the never-arrives hang this file's docstring is written against — and all eighteen tests stay green. Latent, not live: verified that none of ci.yml:7, build.yml:6, eos-simulation.yml:6, codeql.yml:6, python-tests.yml:6, third-party.yml:12 carries a path filter today; each restricts only branches:. Add the inverse assertion in the same loop shape: for every workflow in REQUIRED_CHECKS, assert "paths" not in trigger and "paths-ignore" not in trigger, with the message explaining that a path-filtered workflow cannot be a required check. Six lines, and it makes the two halves of the registry symmetric.
4 Low tests/unit/test_ci_gate.py:213 set(jobs[job_id].get("needs", [])) assumes needs is a list. needs: test — a bare scalar — is valid Actions syntax, and set("test") is {'t','e','s'}, so expected - declared would report every real dependency as missing, or in the mirror case at :249 a single-character job id would silently satisfy the check. The file already normalises exactly this shape for on: (:96-104, added after the last round) and for good reason; needs: was not given the same treatment. No workflow uses the scalar form today, so this is latent. Same normalisation: needs = job.get("needs", []) / if isinstance(needs, str): needs = [needs]. Apply it at :213, :249 and :287 where needs is read.

Architecture conformance

§21 Infrastructure — conforms. Workflow files, one shell script and one pytest module; .github and CI templates are Infrastructure tier. No runtime dependency is created and nothing crosses a tier. net/src/net_posix.c and tests/test_net.c are #116's repair carried on the branch and were reviewed under #119; they are unchanged from the last round.

§28 Status, evidence and claims — this remains the right mechanism, and finding 1 is a defect in it rather than in its premise. §28 makes "Implemented" mean code plus functional tests and "Validated" mean CI or test reports; a required check is how those stop being self-declared. ci-gate-check.sh:24-29 treating skipped as non-success, and :17-20 refusing an empty context outright, are the correct reading of that section. Finding 1 does not weaken the rule — it makes the check red when it should be silent, which is the safe direction, but it costs the check its credibility.

Weakened checks — none. The || true removal at python-tests.yml:26 and the simulate-qemu-only conditional replaced at eos-simulation.yml:349 are both strengthenings, as recorded last round.

The org-standards gap from the last round's finding 6 is still open, and correctly so — it is a change to embeddedos-org/.github, not to this repository. build.yml:171-172 now links the org file by URL rather than referring to a local path that does not exist, which was the actionable half.

Proposed changes

  1. Finding 1 first, since it is the one that touches what the gate is worth: if: ${{ !cancelled() }} on ci-gate, build-gate and integration-summary, with tests/unit/test_ci_gate.py:238 updated to pin the new condition and its docstring explaining both halves — run despite a failed dependency, stay silent on a cancelled run.
  2. Finding 2 in the same commit: GATE_NAME through env: from each workflow, defaulted in the script, plus a test on the message.
  3. Findings 3 and 4 — two small additions to the guard test, independent of each other and of the above.
  4. Re-run and quote a Build Gate log showing its own name, and a superseded push that leaves no red gate behind. That is the evidence for findings 1 and 2 and it cannot be produced from a review.
  5. Do not set branch protection until finding 1 lands. The six names are correct and the registry now enforces them, but requiring a check that goes red on every superseded push will be read as flake within a week.
  6. Unchanged from last round: Run Python tests clears when #127 lands. Nothing here touches tests/CMakeLists.txt.

CI

Three red on this head. One is inherited; two are finding 1.

check why
Run Python tests inherited — tests/unit/test_cmake_test_registration.py::test_every_c_suite_is_built_or_listed fails on test_crypto_ed25519_loworder.c, which exists in tests/ but no add_executable() builds. Job log: 1 failed, 35 passed in 0.85s. Fixed by #127. The 35 passing include all 18 of test_ci_gate.py — coverage on that file reports 94.97%, so the new suite runs and is green.
CI Gate finding 1 — run 33783658415 was superseded and cancelled; all four dependencies cancelled; the gate ran anyway and exited 1.
Build Gate finding 1 — same, run 33783658339, and the log carries finding 2's wrong name.
Cross-compile ARM Cortex-M4, Create GitHub Release same cancelled ci.yml run. Not independent failures.

Everything from the live runs (33783662870, 33783662949, 33783662891, 33783663071) was still pending when this bundle was cut, so this head has no result yet for Build & Test (Linux x86_64), Analyze (C/C++), QEMU Boot Test, Cross-compile ARM64 kernel, Host Tests (x86_64), Static Analysis or Full-stack integration summary — which is the one job whose fix this round is about.

One record-keeping item left from the last round: the 2026-09-03T10:57 comment's claim that Full-stack integration summary going red "is the fix from finding 1 doing its job" is still standing uncorrected — it exited 127 before reading any dependency. The code fix landed; the comment did not.

Not checked

  • Nothing was executed here. Findings 1 and 2 rest on the job logs for this head (gh run view --job 100743136485 / 100743137637 / 100743132302 --log-failed) plus reading the workflow files and the script. Findings 3 and 4 are from reading tests/unit/test_ci_gate.py against the six workflows' triggers; I confirmed the absence of path filters by reading each trigger, and did not construct the failing case.
  • Whether Full-stack integration summary is green on this head. It had not run. The checkout fix is present and correct in the diff, and the new test covers it, but this run does not demonstrate the job now reaches its script.
  • How branch protection treats a skipped gate, which is what finding 1's recommended fix produces on a cancelled run. GitHub's documented behaviour for a job skipped by if: differs from a workflow that never triggers, and the PR body treats them as the same case. Confirm this before landing the fix — if a skipped gate blocks rather than passes, the right answer is a script-side distinction instead, and I would rather flag the uncertainty than pick for you.
  • gh api repos/embeddedos-org/eos/branches/master/protection returned no required_status_checks to this token, so I could not read the current rule or confirm it is still null as #92 states.
  • The mutation sweep was not reproduced, same as last round. The six mutations in the 10:49 comment predate this head and none were re-applied here.
  • The ebuild (#103) and eBoot (#90) twins were not checked for findings 1 and 2. Both share ci-gate-check.sh and if: always() by the author's account, so both are likely to share them; worth grepping before any of the three lands.
  • jq availability. ci-gate-check.sh:26 depends on it with no fallback and no check. All three gates run on ubuntu-22.04, where it is preinstalled, so this is stated rather than raised — but a runner-image change breaks all three at once.

Automated architecture review of ae27268c94cb — scheduled, model claude-opus-5, checked against the EmbeddedOS Master Design v2.0. Advisory only: this reviewer never approves, requests changes, or merges. Reply here to discuss or push back — a wrong finding is a bug worth reporting.

embeddedos-org#92 stays open on this point: `required_status_checks` is null on master, so
nothing builds the merge result before it becomes master. Four individually
green PRs combined into code that had never been compiled in the form it
landed in, and the gap that allowed it is still open.

Marking a check required is a settings change, but it needs a name worth
pointing at. `release` is skipped on every pull request, and a required check
that is skipped never reports, so the pull request waits for a status that
never arrives. Requiring the four real jobs individually works only until the
next one is added and silently sits outside the requirement.

`ci-gate` is one job, one stable name, that succeeds only if every other job
that runs on a pull request succeeded. `if: always()` so a real failure shows
as a red X rather than a pull request that hangs. Any non-success result fails
it, `skipped` included -- a job that did not run did not verify anything, and
treating that as a pass is the fail-open shape embeddedos-org#75, embeddedos-org#82 and embeddedos-org#84 removed
elsewhere.

tests/unit/test_ci_gate.py fails if a job is added without being wired into
the gate. Verified by mutation: dropping a job from `needs`, removing
`if: always()`, renaming the gate, and adding an unwired job are each caught.

pyyaml added to both pytest installers, since the test parses the workflow and
neither installer had it.

Verified: 16 Python tests pass, ctest 34/34.
Addresses the review on embeddedos-org#121. Finding 1 was right and it was the important
one: ci.yml is one of nine workflows that trigger on pull_request, so
requiring `CI Gate` and stopping there left CodeQL, the QEMU boot test, the
ARM64 kernel cross-build, the Python suites and the SBOM validation outside
the requirement -- while the test written to stop that rot only ever read
ci.yml.

The scan now covers .github/workflows/*.yml, and REQUIRED_CHECKS pins the name
a maintainer must require for each. Adding a workflow, or a job to one, fails
the suite until the registry is updated.

Six workflows are now covered. Two needed a gate job that did not exist
(build.yml) or did not do its job (eos-simulation.yml, below). Three define a
single job whose own name is stable and requirable, so they need no gate, and
a test keeps them single-job. Three gate nothing on purpose -- auto-assign,
claude-code-review and book-build all skip on ordinary pull requests, and a
required check that is skipped never reports -- each recorded with its reason.

eos-simulation.yml's `integration-summary` already had the shape of a gate:
the right needs, if: always(), and a printed PASS/FAIL per dependency. It only
exited non-zero for simulate-qemu, so build-kernel, test-middleware and
test-cad-pipeline could all fail while it reported success. Anyone requiring
that check would have believed three things were gated that were not.

Finding 2: _only_runs_on_tags() was a substring test for "refs/tags", which
also matches the inverted condition -- a job running on every pull request
would have been classified as tag-only and dropped from the requirement
silently. Replaced with a pinned CANNOT_RUN_ON_PR map, plus an assertion that
each excluded job really does carry the condition claimed for it.

Finding 3: the rule is now .github/scripts/ci-gate-check.sh, executed by the
tests against real inputs -- success, failure, skipped, cancelled, mixed, and
an empty context -- instead of the workflow's run: text being grepped for
`!= "success"`. A test that greps an implementation passes for those tokens in
a comment and fails for a correct rewrite. Every gate calls the one script,
and a test enforces that too.

Finding 4: dropped `2>/dev/null || true` from the pip install in
python-tests.yml.

Verified by mutation, each caught: a new pull_request workflow with no gate; a
job added to build.yml outside its gate; a gate losing if: always(); a
required display name renamed; the shared rule weakened to ignore `skipped`;
and integration-summary reverted to gating only simulate-qemu.

28 Python tests pass, ctest 34/34, all 19 workflow files parse.
`Full-stack integration summary` invoked .github/scripts/ci-gate-check.sh
with no `actions/checkout` step, so on this head the step exited 127 on a
missing file. The gate's rule never ran: it was red because the script was
not in the workspace, not because a dependency failed, and it could never
be green -- so a maintainer following this PR's own instruction to require
that name would have made master permanently unmergeable. That is the
mirror image of the fail-open shape this PR exists to remove, and worse in
practice: fail-open lets bad code through, this blocks everything.

`test_every_gate_invokes_the_shared_script` passed on that commit, because
it greps the job's `run:` text for the script name and a job that cannot
execute the script still names it. That is the one remaining
grep-the-YAML test in a file whose own docstring says the rule should be
executed rather than pattern-matched, and it is exactly the test that
should have caught this. So the fix and the test that would have caught it
land together:

- eos-simulation.yml: `actions/checkout@v4` as the first step of
  integration-summary, as ci-gate and build-gate already have.
- test_every_gate_checks_out_the_repository_before_running_the_script:
  any gate whose `run:` names a repo-relative script must also have a
  checkout step. Fails without the fix.

Three further hardenings to the same file:

- The NOT_REQUIRED reasons were unverified free text and one had already
  drifted: book-build.yml was recorded as skipping because build-pdf is
  conditional, but its `validate` job carries no `if:` at all and the real
  reason it cannot be required is the `paths: ['docs/book/**']` filter on
  its pull_request trigger -- the never-arrives failure mode this PR is
  careful about everywhere else. The reason is corrected, and a test now
  asserts each NOT_REQUIRED workflow either carries paths/paths-ignore or
  gates every job behind an `if:`. auto-assign.yml and
  claude-code-review.yml pass it as written.
- test_gate_script_exists_and_is_executable checked only `is_file()`.
  The gates pipe into the path itself, which needs mode +x; it is
  committed 100755, but `git update-index --chmod=-x` or a
  core.fileMode=false checkout would break all three gates with the suite
  green. It now asserts os.access(X_OK), and one case invokes the script
  directly rather than via `bash`, so the invocation path the workflows
  use is covered.
- `_triggers_on_pull_request` handled only the mapping form of `on:`.
  `on: pull_request` and `on: [push, pull_request]` are both valid and
  both made a workflow invisible to the scan -- silently unclassified,
  which is the rot this file exists to catch. Latent, not live: all 19
  current workflow files use the mapping form.

Also: the comment at build.yml:174 pointed at `.github/STANDARDS.md`,
which does not exist in this repository. STANDARDS.md is the org-wide file
at embeddedos-org/.github; the comment now links it.

Each of the four new assertions was mutation-checked: removing the
checkout, chmod -x on the script, dropping book-build's paths filter, and
rewriting a workflow to the list form of `on:` each fail the suite and
each is caught by the intended test.

pytest tests/ -q: 31 passed (18 -> 21 in test_ci_gate.py).
ctest Release: 34/34.
The run that proved the checkout fix also showed the shared script naming
the wrong check. `Full-stack integration summary` failed and logged:

    ::error::CI Gate failed. These jobs did not succeed:
      build-kernel: failure
      simulate-qemu: skipped

The diagnosis is right and the check name is not: "CI Gate" was hardcoded
in ci-gate-check.sh, so two of the three gates report a failure under a
check name that is not theirs. In a pull request whose subject is checks
that report something other than what they measured, that is not a
cosmetic detail -- a maintainer reading the log of a red merge is told to
go look at a check that is green.

The script now takes `GATE_NAME` (defaulting to "Gate"), each gate step
sets it to its own display name, and two tests hold it in place: one runs
the script with GATE_NAME set and asserts the message uses it, one asserts
each gate declares the display name that REQUIRED_CHECKS says it reports
under. Both mutation-checked -- hardcoding the name in the script fails the
first, giving build-gate the wrong name fails the second.

pytest tests/ -q: 33 passed.
@Kartikey1306
Kartikey1306 force-pushed the ci/required-check-gate branch from 73cd391 to 86994dd Compare September 3, 2026 17:52

@srpatcha srpatcha left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — eos#121 "ci: add one job branch protection can require"

head: 86994dd author: Kartikey1306 ci: fail (2 of 26 — both cross-repo, gates behaved correctly)

Verdict: The design is right and the prior rounds closed real defects — the six-name registry, the extracted rule, the integration-summary fix that took it from gating one of four dependencies to gating all four, and the missing checkout that would have made a required check permanently red. I verified the structure independently: the gate needs: lists cover every non-release job in all three workflows, and all six required names report on this head. But the shared rule itself fails open on the one input GitHub actually produces for a dependency-less gate, and the two tests meant to prevent exactly that both miss it.

Findings

# Severity File:line Finding Recommended fix
1 High .github/scripts/ci-gate-check.sh:24-27 (with tests/unit/test_ci_gate.py:320-323 and :330) The gate passes on an empty job set — the fail-open shape the script's own header says it exists to remove. The guard rejects "" and "null", but a structurally empty context is neither. to_entries[] over {} yields no entries, bad is empty, and the script prints All jobs succeeded. and exits 0. Executed against this head with jq present:
{} → rc=0 All jobs succeeded.
[] → rc=0 All jobs succeeded.
{"a":{"result":"skipped"}} → rc=1 ✓
{"a":{}} → rc=1 (a: null) ✓
null / "" → rc=1 ✓
{} is precisely what ${{ toJSON(needs) }} renders when a job has no needs: — GitHub never renders "" or null there. So a gate that loses its last needs entry, or a new gate added without one, becomes a required check that is green forever. The script's comment says "a required check that passes on 'did not run' is the fail-open shape this repository has spent #75, #82 and #84 removing"; {} is that case. Both safety nets miss it: test_gate_script_refuses_an_empty_context is @pytest.mark.parametrize("payload", ["", "null"]) — its docstring, "No results is not the same as no failures", states the right principle for two of the three ways to express no results and omits the one that occurs; and test_every_gate_invokes_the_shared_script opens with if not job.get("needs"): continue (:330), which exempts a needs-less gate from even having to call the script. A gate with no dependencies is therefore unchecked by one test and passed by the other.
Two lines in the script, after the existing guard:
n=$(printf '%s' "$results" | jq 'if type=="object" then length else -1 end')
if [ "$n" -lt 1 ]; then echo "::error::${gate} was given no jobs to check; refusing to pass." >&2; exit 1; fi
(the -1 for a non-object also closes the [] case). Then extend the parametrize to ["", "null", "{}", "[]"], and replace the continue at :330 with an assertion — a registered gate with no needs is itself the defect, so assert job.get("needs"), f"{workflow}: {job_id!r} is a required check with no needs". Both changes are mutation-checkable the same way as the six already in the table.
2 Medium tests/CMakeLists.txt:93-97 An undisclosed change, and it is the one that makes this PR's own evidence green. The diff adds the test_crypto_ed25519_loworder registration block (5 lines). The PR body's file inventory, its verification table and its "Stacked on #116" section never mention tests/CMakeLists.txt, ed25519, or #127 — and an earlier comment on this PR states "Nothing in this PR touches tests/CMakeLists.txt", which was true at 26413a2 and is not true here. This matters for two reasons. First, it is why CI Gate reports pass on this head: test_cmake_test_registration.py no longer fails, so Build & Test (Linux x86_64) is green. The PR's headline evidence that the gate works is partly produced by an unmentioned hunk. Second, this is the four-way conflict already flagged elsewhere — #118, #119, #126 and #127 all add the identical block at the identical anchor (#126 adds a stray blank line there). This PR silently joins that set. Relatedly the body's ctest … 34/34 is stale: with the block registered I measure 39 tests, 39 passed at this head. Say so in the body — one sentence naming the hunk, why it is here, and that it collides with #118/#119/#126/#127. Then either drop it and depend on #127 landing first, or keep it and state that whichever of the five lands second must drop the hunk. Update the ctest figure to 39/39.
3 Medium (maintainer action) Requiring all six names today makes master unmergeable, and the body does not say so. Full-stack integration summary is red on this head, and not for any reason inside this repository: Cross-compile ARM64 kernel fails building eBoot (eBoot/include/eos_image.h:135:23: error: 'eos_image_header_t' has no member named 'reserved', also :142). Because the fix in this PR correctly makes that gate fail on any non-success dependency, requiring the name couples eos's ability to merge anything to eBoot's trunk compiling. The gate is behaving correctly — this is the right red — but the "What is still a maintainer action" section lists all six names to add without noting that one of them cannot go green until another repository is repaired. That is a footgun for whoever performs the maintainer action. Add the precondition to the body: require the five that are green now, and add Full-stack integration summary once eBoot's eos_image.h is fixed — or state plainly that all six should wait on it. The general problem is already recorded as a proposal ("§5.1 governs runtime dependencies and says nothing about build- or CI-time ones, so one repository's trunk decides whether another's pull requests can be green" in .ai/autoreview/proposals/2026-09.md), so no new proposal is needed; this PR just needs to not walk a maintainer into it.
4 Low (CI) 2 of 26 checks fail, neither from this diff, and the gates diagnosed both correctly — which is the PR working. Cross-compile ARM64 kernel → eBoot's header (finding 3); Full-stack integration summary → red because that dependency is red. All six required names report, verified against checks.txt: CI Gate pass, Build Gate pass, Full-stack integration summary fail, Analyze (C/C++) pass, Run Python tests pass, Validate third_party/ and the SBOM pass. Confirmed pre-existing: the same workflow fails on origin/master at feee2726, and eos#126, #129 and #134 each report the identical 2 failures. Nothing for this author.

Architecture conformance

Conforms. Master design §21 places CI templates and release automation under Infrastructure, and eos itself in Tier 1 — Foundation. This change is entirely within .github/ plus one test file and one CMake registration; it adds no runtime code, no #include, no link line and no manifest entry, so §5.1's dependency direction is untouched. §21.1 is not in play.

It serves §28 directly. §28's evidence table requires "Code and functional tests" for Implemented and "Hardware/CI/test reports" for Validated, and the existing §28.2 proposal in this file's proposals log ("The evidence policy is silent on checks that verify nothing") is about exactly this failure mode — a green check that ran nothing. Extracting the rule into .github/scripts/ci-gate-check.sh so a test can execute it with real inputs, rather than grepping the YAML for != "success", is the correct structural answer to that, and it is what let me find finding 1 by running the rule rather than reading it.

Two checks were strengthened, none weakened. || true is removed from the dependency install in both ci.yml:42 (pip3 install pytest pytest-cov pyyaml) and python-tests.yml:26 — previously pip install pytest pytest-cov 2>/dev/null || true, which meant a failed install produced a job that ran no tests and passed. And eos-simulation.yml's integration-summary goes from one exit 1 testing needs.simulate-qemu.result to the shared all-must-succeed rule, closing a gate that covered one of its four dependencies while carrying the most requirable-looking name in the repository.

I verified the coverage claim structurally rather than taking it. For each gated workflow, every job that is not release and not the gate itself appears in the gate's needs:

workflow jobs (excl. gate) gate needs covered
ci.yml test, default-config, build-arm, static-analysis, release test, default-config, build-arm, static-analysis all but release, which is refs/tags/v-only and documented
build.yml test-host, build-arm, test-qemu test-host, build-arm, test-qemu all
eos-simulation.yml build-kernel, simulate-qemu, test-middleware, test-cad-pipeline same four all

All three gates carry if: always(), actions/checkout@v4 as the first step, and their own GATE_NAME, so the earlier "wrong check name in the log" and "exit 127, never green" defects are genuinely closed at this head.

The author's correction of their own 2026-09-01 comment — that the run log posted as evidence here was eBoot's, not this repository's — is accurate and was the right thing to post. So is the observation that a mutation which does not fail is a broken mutation before it is a broken test.

Proposed changes

  1. Finding 1, the script guard and the two test changes. This is the only one I would hold the PR for:
# after the existing empty/null guard
n=$(printf '%s' "$results" | jq 'if type == "object" then length else -1 end')
if [ "$n" -lt 1 ]; then
    echo "::error::${gate} was given no jobs to check; refusing to pass." >&2
    exit 1
fi
@pytest.mark.parametrize("payload", ["", "null", "{}", "[]"])
def test_gate_script_refuses_an_empty_context(payload):
    """No results is not the same as no failures.

    `{}` is what `toJSON(needs)` renders for a job with no `needs:` --
    the only one of these GitHub actually produces.
    """
    assert _run_gate(payload) == 1

and at test_ci_gate.py:330, replace the skip with an assertion:

        assert job.get("needs"), (
            f"{workflow}: {job_id!r} is a required check with no needs, "
            f"so it verifies nothing and cannot fail."
        )
  1. Finding 2, disclose the tests/CMakeLists.txt hunk in the body, record the collision with #118/#119/#126/#127, correct 34/34 to 39/39.
  2. Finding 3, add the eBoot precondition to the maintainer-action section.

Not checked

  • tests/unit/test_ci_gate.py — I could not run it. It import yaml at :41 and pyyaml is not installed in my environment, so collection errors with ModuleNotFoundError: No module named 'yaml'. This is an environment failure on my side, not a PR failure — and note the PR is what adds pyyaml to both CI installs, so CI will have it. Consequence: the author's 23 passed for that file is unverified by me, and every assertion in it — the six-name registry, NOT_REQUIRED reasons, the checkout check, the GATE_NAME checks, single-job enforcement — I read but did not execute. Finding 1's claims about :320 and :330 come from reading the source, not from running it; the claims about the script's behaviour are executed.
  • The mutation sweep in the comments (six mutations, each named against a test) I did not reproduce. Given finding 1, the sweep's value is bounded by which inputs the tests cover, which is the point.
  • The gate has never run with {} in CI, because no gate currently lacks a needs. Finding 1 is a latent fail-open, not an active one — I reached it by invoking the script directly, not by observing a green gate that should have been red. Nothing is currently mis-reporting.
  • GitHub-side behaviour — that toJSON(needs) renders {} for an absent needs: is from GitHub's expression semantics, not something I observed on a runner. I did not push a branch to test it, and per the run instructions I did not push anything.
  • Branch protection was not inspected or changed. Whether required_status_checks is still null on master I did not check; setting it is a maintainer action and outside this run.
  • What I did run: git archive of the head into a temp directory; cmake -S . -B build2 -DEOS_BUILD_TESTS=ON && cmake --build build2 -j4 → exit 0, no errors; ctest --no-tests=error → 39/39 passed; pytest tests/ -q --ignore=tests/unit/test_ci_gate.py → 15 passed; and the eight-input probe of ci-gate-check.sh above. GCC 15.2.0, x86_64 Linux, jq present, single configuration. I built no cross target and ran no workflow.

Automated architecture review of 86994dddd2d9 — scheduled, model claude-opus-5, checked against the EmbeddedOS Master Design v2.0. Advisory only: this reviewer never approves, requests changes, or merges. Reply here to discuss or push back — a wrong finding is a bug worth reporting.

Kartikey1306 added a commit to Kartikey1306/eos that referenced this pull request Sep 4, 2026
…atch that stops posing as a check

Answers the overnight review on embeddedos-org#129.

Finding 3 (Low, and the structural one) -- the five-line install recipe was
copy-pasted at seven sites across two workflows, and the PR's own history
shows the cost: -e dropped "at all five sites, not two", runner.temp at all
of them, shell: bash at several, each a separate round because each site was
a separate edit. Now .github/actions/install-eosim owns it -- clone at the
pinned commit, install, stage platforms/, assert the dist version -- and all
seven sites are a four-line `uses:`. The platforms/ workaround and the
version assertion get one place to be deleted from when EoSim fixes its
packaging. The triplicated inline importlib.metadata assert is gone with it.

Finding 5 (Low), folded into the composite -- SITE_PACKAGES comes from
sysconfig.get_paths()["purelib"], which needs no import and cannot be
shadowed, matching eBoot's copy of the workaround instead of solving the same
problem a second way.

Finding 1 (Medium) -- the inline gate accepted nothing as everything: an
empty RESULTS printed "All EoSim sanity jobs succeeded." and exited 0, and
null produced a raw jq error, where eos#121's extracted ci-gate-check.sh
refuses both. Both gates now refuse empty/null input with a diagnosis before
jq runs, under set -euo pipefail, and carry a note to become a call to embeddedos-org#121's
script once it is on master so the two cannot drift.

  Reproduced the review's side-by-side on the hardened body:
      ""      rc=1  gate received no results to check   (was rc=0 PASS)
      null    rc=1  gate received no results to check   (was raw jq error)
      {}      rc=0  PASS   -- shared with embeddedos-org#121's script; reported there, and
      []      rc=0  PASS      unreachable here: needs has five entries
      skipped rc=1  FAIL: a: skipped
      success rc=0  PASS

Also applied to simulation-test.yml's Simulation Gate, which still had the
original fail-open shape -- needs: [simulate, cross-platform], a test of
`simulate` alone, then "All simulation tests passed". Same toJSON(needs)
body, same guards.

Finding 4 (Low) -- published-artifact-watch could not fail and sat in
sanity-gate's needs under a name that made a green tick assert a deficiency.
Now "EoSim artifact publication (informational)", schedule/dispatch only, out
of the gate: pull requests no longer carry a permanently-green status about
another repository's release process, and the debt still gets said out loud
every night.

Finding 6 (Low) -- simulation-test's EOSIM_COMMIT comment argued tag
selection for what is now a SHA; it now states the actual reason (EoSim's tag
names do not identify their contents) and gains the EOSIM_EXPECTED_DIST the
composite asserts.

Finding 2 (Medium) is a two-line change in eos#121's registry, not here --
both workflows need NOT_REQUIRED entries with a paths-filter reason once both
PRs land, or embeddedos-org#121's classification test fails. Posted on both threads with
the exact entries; not folded in, because that registry lives on embeddedos-org#121's
branch and this PR должен not carry another PR's hunks.

Verified locally before push:
  all three YAML files parse
  composite referenced at 7 sites; zero inline clone/install/copy remnants
  eosim-sanity gate needs == every job except itself and the informational
    watch; sim gate iterates toJSON(needs), no single-dependency branch
  the five-input table above, executed
  the workflows themselves: this push's pull_request run is the test, and I
  will report what it shows rather than predict it.

Refs embeddedos-org#129
Kartikey1306 added a commit to Kartikey1306/eos that referenced this pull request Sep 4, 2026
…atch that stops posing as a check

Answers the overnight review on embeddedos-org#129.

Finding 3 (Low, and the structural one) -- the five-line install recipe was
copy-pasted at seven sites across two workflows, and the PR's own history
shows the cost: -e dropped "at all five sites, not two", runner.temp at all
of them, shell: bash at several, each a separate round because each site was
a separate edit. Now .github/actions/install-eosim owns it -- clone at the
pinned commit, install, stage platforms/, assert the dist version -- and all
seven sites are a four-line `uses:`. The platforms/ workaround and the
version assertion get one place to be deleted from when EoSim fixes its
packaging. The triplicated inline importlib.metadata assert is gone with it.

Finding 5 (Low), folded into the composite -- SITE_PACKAGES comes from
sysconfig.get_paths()["purelib"], which needs no import and cannot be
shadowed, matching eBoot's copy of the workaround instead of solving the same
problem a second way.

Finding 1 (Medium) -- the inline gate accepted nothing as everything: an
empty RESULTS printed "All EoSim sanity jobs succeeded." and exited 0, and
null produced a raw jq error, where eos#121's extracted ci-gate-check.sh
refuses both. Both gates now refuse empty/null input with a diagnosis before
jq runs, under set -euo pipefail, and carry a note to become a call to embeddedos-org#121's
script once it is on master so the two cannot drift.

  Reproduced the review's side-by-side on the hardened body:
      ""      rc=1  gate received no results to check   (was rc=0 PASS)
      null    rc=1  gate received no results to check   (was raw jq error)
      {}      rc=0  PASS   -- shared with embeddedos-org#121's script; reported there, and
      []      rc=0  PASS      unreachable here: needs has five entries
      skipped rc=1  FAIL: a: skipped
      success rc=0  PASS

Also applied to simulation-test.yml's Simulation Gate, which still had the
original fail-open shape -- needs: [simulate, cross-platform], a test of
`simulate` alone, then "All simulation tests passed". Same toJSON(needs)
body, same guards.

Finding 4 (Low) -- published-artifact-watch could not fail and sat in
sanity-gate's needs under a name that made a green tick assert a deficiency.
Now "EoSim artifact publication (informational)", schedule/dispatch only, out
of the gate: pull requests no longer carry a permanently-green status about
another repository's release process, and the debt still gets said out loud
every night.

Finding 6 (Low) -- simulation-test's EOSIM_COMMIT comment argued tag
selection for what is now a SHA; it now states the actual reason (EoSim's tag
names do not identify their contents) and gains the EOSIM_EXPECTED_DIST the
composite asserts.

Finding 2 (Medium) is a two-line change in eos#121's registry, not here --
both workflows need NOT_REQUIRED entries with a paths-filter reason once both
PRs land, or embeddedos-org#121's classification test fails. Posted on both threads with
the exact entries; not folded in, because that registry lives on embeddedos-org#121's
branch and this PR must not carry another PR's hunks.

Verified locally before push:
  all three YAML files parse
  composite referenced at 7 sites; zero inline clone/install/copy remnants
  eosim-sanity gate needs == every job except itself and the informational
    watch; sim gate iterates toJSON(needs), no single-dependency branch
  the five-input table above, executed
  the workflows themselves: this push's pull_request run is the test, and I
  will report what it shows rather than predict it.

Refs embeddedos-org#129
@Kartikey1306

Copy link
Copy Markdown
Contributor Author

Coordination from #129, so the interaction the review found there does not land as a surprise: #129 gives eosim-sanity.yml and simulation-test.yml a pull_request trigger (paths-filtered to the two files themselves — it is what produced the first executions either workflow has ever had). Once both PRs merge, test_every_pull_request_workflow_is_classified here will fail unless those two are registered. The two entries, shaped like the existing book-build.yml one:

"eosim-sanity.yml":
    "pull_request trigger is filtered to paths: [its own file and "
    "simulation-test.yml], so it reports only on changes to itself; "
    "requiring it would block every other PR on a status that never arrives",
"simulation-test.yml":
    "same paths-filtered self-test trigger as eosim-sanity.yml",

Both carry paths:, so test_workflows_recorded_as_not_requirable_really_are_not is satisfied without further change. Whichever PR lands second picks this up — not folding it into #129, since this registry is this PR's file.

Kartikey1306 added a commit to Kartikey1306/eos that referenced this pull request Sep 4, 2026
The pin in the previous commit stops an eBoot or ebuild master that does not
compile from turning every eos pull request red. It also stops that break
being visible from here at all, which on its own is the worse half of the
trade: the next upstream breakage would be silent instead of loud, and the
pin would sit at a172a6d indefinitely because nothing would ever ask -- while
quietly dropping the six eBoot commits that follow it.

`.github/workflows/upstream-drift.yml` is the other half. It builds eBoot at
master and compares both pins against their master, so an upstream break
still surfaces, on a run that blocks no pull request and names the repository
responsible.

Design choices, each one avoiding a defect this repo has been filing against:

  Its own file, schedule + workflow_dispatch only. The first draft was a job
  inside eos-simulation.yml guarded by `if: github.event_name == 'schedule'`.
  That works, but the run showed it on the PR as a SKIPPED check -- and a
  required-check gate that treats any non-success as a failure, which is the
  design in embeddedos-org#121, would trip on exactly that. Moving it out removes the check
  from PR runs entirely rather than relying on every future gate to exempt it.

  Not `continue-on-error`. A job reporting success over a failure is the
  fail-open shape we keep finding in aggregating gates. This one may fail
  honestly precisely because it gates nothing.

  Failing is reserved for "upstream does not build". A pin merely behind a
  healthy master is a warning: actionable, not broken.

  The pins are read out of eos-simulation.yml, not copied. A second copy of a
  SHA is a second thing to forget. The read demands a 40-character hex value
  and fails closed if it cannot find one, so a renamed or reformatted key
  stops the job instead of silently letting it check a pin of "".

  The failure message is scoped to the build step's own outcome, so a
  checkout, apt or pin-read failure cannot report itself as evidence about
  eBoot's master. The summary says eBoot is built and ebuild is only
  compared, rather than implying both were checked.

Verified by execution, not by reading:

  eBoot @ a172a6d (the pin)   host build rc=0, 0 errors
  eBoot @ 22d8f8b (master)    host build rc=2, "no member named 'reserved'
                              in 'eos_image_header_t'" at eos_image.h:135
                              and :142 -- the same defect CI reports, so the
                              pin is demonstrably the difference

  pin extraction, run against the real file and two mutations of it:
    real workflow                -> both SHAs read correctly, and the eBoot
                                    one resolves to embeddedos-org#86's merge
    key renamed to EBOOT_SHA     -> fails closed, rc=1
    value unquoted and shortened -> fails closed, rc=1

  drift logic, run against the real repositories:
    pin 6 behind healthy master  -> "behind by 6" warning, no failure
    pin absent from master       -> "not an ancestor" warning, exit 0 under
                                    set -e (the first draft printed "pinned
                                    pin not found on master commit(s)")
    pin equal to master          -> no warning

  eos on this branch: cmake build rc=0, ctest 39/39, pytest 15/15 with the
  new workflow file present, test_crypto_ed25519_loworder registered and
  passing.

When eBoot embeddedos-org#94 lands, bump EBOOT_COMMIT to that merge; this job is what will
say whether it holds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
srpatcha pushed a commit that referenced this pull request Sep 8, 2026
…135)

* fix(build): restore the low-order Ed25519 suite that #93 removed

`CI — eos` and `Python Unit Tests` are red on master (769191a), and both
fail for one reason:

    these suites exist in tests/ but no add_executable() in
    tests/CMakeLists.txt builds them, so they never run:
    ['test_crypto_ed25519_loworder.c']

3aa8644 (#93, "remove duplicate test targets from tests/CMakeLists.txt")
removed all four lines that build and register
test_crypto_ed25519_loworder. It was not a duplicate — there was exactly
one registration before that commit, and it took it.

What stopped running is the suite that guards #101: Ed25519 rejecting
low-order public keys. A low-order key makes every term of the
verification equation collapse to the identity regardless of the
message, so a signature of all zeros verifies against any content at
all. The fix is still in services/crypto/src/ed25519_verify.c and still
correct; nothing has been checking it since #93 merged, and nothing
would have noticed if a later change had removed it too.

Restored verbatim from 3aa8644^, in its original position before
test_crypto_sha512. No other file changes.

The guard that caught this is test_cmake_test_registration.py, added
recently. It did exactly its job — this is the first thing it found.

Verified:
  test_cmake_test_registration.py    -> 5 passed (1 failed on master)
  pytest tests/                      -> 15 passed (14 passed 1 failed on master)
  test_crypto_ed25519_loworder       -> 5/5 tests passed
  ctest                              -> 39/39 passed (master builds 38)

* ci: pin the eBoot and ebuild checkouts instead of floating on master

Every open eos pull request is currently red on `EoS Full-Stack Simulation`,
and none of them caused it. The kernel job checks out embeddedos-org/eBoot at
`ref: master`, and eBoot master does not compile: include/eos_image.h:135 and
:142 static-assert offsetof(eos_image_header_t, reserved) for a member that
became tlv_len + tlv_hash, and core/ed25519_verify.c redefines
point_is_identity -- two pairs of PRs that each merged clean and broke the
build together. eBoot #94 repairs it and waits on review; until it lands, no
change to any eos PR can make this job pass, and after it lands the next
broken eBoot master does the same thing again.

A floating `ref: master` makes every eos PR's CI depend on the moment-to-
moment state of another repository's default branch. Pinning the two
compiled-against checkouts makes the job a function of this repository plus
two recorded SHAs -- red means the PR broke something, green means it did not,
and a cross-repo bump is a reviewable one-line diff with CI on it.

  EBOOT_COMMIT  a172a6d6e1e6  the newest eBoot master ancestor that compiles;
                found by walking master back and building each candidate.
                Its child 8a015b2 already carries the ed25519 redefinition.
  EBUILD_COMMIT e5d8052f3e2c  ebuild's current master, frozen as-is. Both of
                its checkouts feed steps that are commented out (eFab does
                not exist), so this pin only removes the float.

The middleware matrix (10 repos) still floats; it runs those repos' own
tests rather than compiling eos against them, and 10 more pins deserve their
own decision. When eBoot #94 merges, bump EBOOT_COMMIT to the merge SHA.

Verified before pushing, master vs pin under the job's own configuration
(-DEBLDR_BOARD=qemu_arm64 -DEBLDR_VERIFY_STAGE1=OFF -DCMAKE_BUILD_TYPE=Release,
cross-compile mode): eBoot@master fails at the eos_image.h asserts, the exact
error in the job log; eBoot@a172a6d builds every target, rc=0. Both pinned
SHAs verified reachable from their repos' master. Workflow parses; the three
`ref:` edits and the env block are the whole diff. This PR's own simulation
run is the live test: it uses this branch's workflow and should go green
while every master-based PR stays red.

* ci: make the pin visible when it goes stale, not silently permanent

The pin in the previous commit stops an eBoot or ebuild master that does not
compile from turning every eos pull request red. It also stops that break
being visible from here at all, which on its own is the worse half of the
trade: the next upstream breakage would be silent instead of loud, and the
pin would sit at a172a6d indefinitely because nothing would ever ask -- while
quietly dropping the six eBoot commits that follow it.

`.github/workflows/upstream-drift.yml` is the other half. It builds eBoot at
master and compares both pins against their master, so an upstream break
still surfaces, on a run that blocks no pull request and names the repository
responsible.

Design choices, each one avoiding a defect this repo has been filing against:

  Its own file, schedule + workflow_dispatch only. The first draft was a job
  inside eos-simulation.yml guarded by `if: github.event_name == 'schedule'`.
  That works, but the run showed it on the PR as a SKIPPED check -- and a
  required-check gate that treats any non-success as a failure, which is the
  design in #121, would trip on exactly that. Moving it out removes the check
  from PR runs entirely rather than relying on every future gate to exempt it.

  Not `continue-on-error`. A job reporting success over a failure is the
  fail-open shape we keep finding in aggregating gates. This one may fail
  honestly precisely because it gates nothing.

  Failing is reserved for "upstream does not build". A pin merely behind a
  healthy master is a warning: actionable, not broken.

  The pins are read out of eos-simulation.yml, not copied. A second copy of a
  SHA is a second thing to forget. The read demands a 40-character hex value
  and fails closed if it cannot find one, so a renamed or reformatted key
  stops the job instead of silently letting it check a pin of "".

  The failure message is scoped to the build step's own outcome, so a
  checkout, apt or pin-read failure cannot report itself as evidence about
  eBoot's master. The summary says eBoot is built and ebuild is only
  compared, rather than implying both were checked.

Verified by execution, not by reading:

  eBoot @ a172a6d (the pin)   host build rc=0, 0 errors
  eBoot @ 22d8f8b (master)    host build rc=2, "no member named 'reserved'
                              in 'eos_image_header_t'" at eos_image.h:135
                              and :142 -- the same defect CI reports, so the
                              pin is demonstrably the difference

  pin extraction, run against the real file and two mutations of it:
    real workflow                -> both SHAs read correctly, and the eBoot
                                    one resolves to #86's merge
    key renamed to EBOOT_SHA     -> fails closed, rc=1
    value unquoted and shortened -> fails closed, rc=1

  drift logic, run against the real repositories:
    pin 6 behind healthy master  -> "behind by 6" warning, no failure
    pin absent from master       -> "not an ancestor" warning, exit 0 under
                                    set -e (the first draft printed "pinned
                                    pin not found on master commit(s)")
    pin equal to master          -> no warning

  eos on this branch: cmake build rc=0, ctest 39/39, pytest 15/15 with the
  new workflow file present, test_crypto_ed25519_loworder registered and
  passing.

When eBoot #94 lands, bump EBOOT_COMMIT to that merge; this job is what will
say whether it holds.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@srpatcha
srpatcha merged commit 2049a86 into embeddedos-org:master Sep 8, 2026
16 of 22 checks passed
srpatcha pushed a commit that referenced this pull request Sep 8, 2026
…129)

* fix(build): restore the low-order Ed25519 suite that #93 removed

`CI — eos` and `Python Unit Tests` are red on master (769191a), and both
fail for one reason:

    these suites exist in tests/ but no add_executable() in
    tests/CMakeLists.txt builds them, so they never run:
    ['test_crypto_ed25519_loworder.c']

3aa8644 (#93, "remove duplicate test targets from tests/CMakeLists.txt")
removed all four lines that build and register
test_crypto_ed25519_loworder. It was not a duplicate — there was exactly
one registration before that commit, and it took it.

What stopped running is the suite that guards #101: Ed25519 rejecting
low-order public keys. A low-order key makes every term of the
verification equation collapse to the identity regardless of the
message, so a signature of all zeros verifies against any content at
all. The fix is still in services/crypto/src/ed25519_verify.c and still
correct; nothing has been checking it since #93 merged, and nothing
would have noticed if a later change had removed it too.

Restored verbatim from 3aa8644^, in its original position before
test_crypto_sha512. No other file changes.

The guard that caught this is test_cmake_test_registration.py, added
recently. It did exactly its job — this is the first thing it found.

Verified:
  test_cmake_test_registration.py    -> 5 passed (1 failed on master)
  pytest tests/                      -> 15 passed (14 passed 1 failed on master)
  test_crypto_ed25519_loworder       -> 5/5 tests passed
  ctest                              -> 39/39 passed (master builds 38)

* ci: finish the EoSim fix — a tag that exists, and no wheel anywhere

`Simulation Test` and `EoSim Sanity` are both red on master. #106 fixed
this class in eosim-sanity.yml but did not reach every site.

simulation-test.yml still pins EOSIM_VERSION: "0.1.0". EoSim has no
v0.1.0 tag — its releases run v1.0.0 to v3.0.1 — so every run failed at
the clone:

    fatal: Remote branch v0.1.0 not found in upstream origin

Bumped to 1.5.0, the release marked Latest and the one eosim-sanity.yml
already pins.

eosim-sanity.yml still installs a wheel on its Windows and macOS legs:

    ERROR: HTTP error 404 while getting
    .../releases/download/v1.5.0/eosim-1.5.0-py3-none-any.whl

No EoSim release ships a wheel, so that URL 404s at any version — the
version bump alone could never fix these two. The other three jobs in
the same workflow already install from a clone; these two were missed.
Converted to the same clone + `pip install -e`.

Verified, against the real remote rather than from the error message:

    wheel URL                       -> HTTP 404
    git ls-remote --tags | v0.1.0   -> 0 matches
    git ls-remote --tags | v1.5.0   -> 1 match
    clone v1.5.0 + pip install -e   -> rc 0
    eosim --version                 -> eosim, version 2.0.0
    eosim doctor                    -> rc 0

(The v1.5.0 tag reporting version 2.0.0 is EoSim's own inconsistency,
not a wrong pin — v1.5.0 is what its releases page marks Latest.)

Both files still parse as YAML. Stacked on the #127 registration fix so
this branch's CI is not red for an unrelated reason.

* ci(eosim): pin the commit, assert the version pip resolved, and let the workflow test itself

Answers the review on #129. The two diagnoses in the previous commit hold --
there is no v0.1.0 tag and no EoSim release ships a Python artifact -- and
this addresses what the replacement pin did not.

Finding 1 (High) -- `EOSIM_VERSION: "1.5.0"` is a label, not a version.
Re-verified against the remote rather than taken from the review:

  tag      commit    committed    commit subject             pyproject  __init__
  v1.5.0   7dec3460  2026-05-27   "production-ready v1.4.0"    3.0.1      3.0.1
  v3.0.1   b297ec26  2026-05-16   "v3.0.1 — unified ..."       3.0.1      2.0.0

v1.5.0 is the release GitHub marks Latest and the newest tag by commit date,
tags a commit whose own message claims v1.4.0, and installs a dist declaring
3.0.1. A reader of `EOSIM_VERSION: "1.5.0"` would believe eos is simulated
against EoSim 1.5.0. It is not, and nothing in the file would reveal that.

Both workflows now pin `EOSIM_COMMIT` to the full SHA that tag resolves to
today, with the table above in the env block and the date it was checked.
Not v3.0.1 instead: its `__init__.py` says 2.0.0, so it is only marginally
less confusing. A SHA cannot be moved under us; the comment says to go back to
a tag once EoSim's tags mean something.

Finding 3 (Medium) -- `eosim --version` cannot detect a wrong install.
EoSim hardcodes it (`eosim/cli/main.py:49`,
`@click.version_option(version="2.0.0")`), confirmed by reading that file at
v1.5.0, so it prints 2.0.0 from every tag -- which is exactly why the previous
commit could record the version oddity as a curiosity rather than as a failed
pin. All three sites now also assert `importlib.metadata.version("eosim")`
against `EOSIM_EXPECTED_DIST`, which is the only version the commit pin
controls.

Finding 4 (Medium) -- this PR could not be validated by CI and was not: both
workflows were schedule- and dispatch-only, so neither appeared in this head's
checks and all 25 green ones were other workflows. Added
`pull_request: paths:` on both files listing both files, so a change to either
now runs both. `workflow_dispatch` on the fork is not available to me --
GitHub resolves the workflow on the fork's default branch and returns 404 --
so this trigger is the evidence that is actually obtainable, and it applies to
every future change to these files rather than to this one only.

Finding 2 (High) -- after this, nothing in the org exercises a published EoSim
artifact, and none exists. Verified independently: 13 releases whose assets
are promo .mp4 files, a manifest.json and EoSim-guide.pdf; no wheel or sdist
at any tag; `https://pypi.org/pypi/eosim/json` returns 404. Converting to a
clone is right -- a permanent 404 is not a useful red -- but it would also
make the gap green and invisible, which is the "verification whose result is
discarded" shape.

So `published-artifact-watch` runs on every execution, checks the releases and
PyPI, and says out loud that the packaging debt is open. It cannot fail the
workflow, deliberately: it is a tracking signal, not a gate on someone else's
repo. It emits a `::notice::` the day an artifact appears, which is when it
should be deleted and the installs pointed at it. The debt is also stated in
the PR body rather than left implied, and raised against EoSim.

Finding 5 (Low) -- `/tmp/EoSim` was hardcoded in the two jobs that run on
windows-latest and macos-latest, where the Windows job sets no `shell:` and
gets pwsh. Now `${{ runner.temp }}/EoSim` everywhere.

Finding 6 (Low) -- dropped `-e`. Nothing needs an editable install, it is
further from what a developer does in a workflow meant to prove the install
works, and it made the dist-metadata assertion depend on the clone staying
put. Changed at all five sites, not two, so they are consistent.

Also, while in this file: `sanity-gate` had the fail-open shape --
`needs:` five jobs, tested `install-validate` alone, then printed "All EoSim
sanity checks passed". Replaced with the `toJSON(needs)` body, which cannot
fall out of step with `needs:`, and the new job is included in it.

Verified:
  ctest                                       39/39 PASS
  pytest tests/                               15 passed
  both workflows parse, and structurally:
    pull_request paths list this file           both
    EOSIM_VERSION gone from every code line     both
    EOSIM_COMMIT is a full 40-char sha          both
    no /tmp/EoSim literal, no `pip install -e`  both
    every `git clone --filter` has a matching `checkout --detach`
    three importlib.metadata assertions
    sanity-gate needs == every other job in the file
  the SHA resolves: refs/tags/v1.5.0 -> tag object 978f215f -> commit
    7dec3460cba76083b6645eb4a867e39aa2af97df

  NOT RUN: neither workflow. `workflow_dispatch` on the fork returns 404
  because GitHub resolves it on the fork's default branch. The
  `pull_request: paths:` trigger should make both run on this PR for the
  first time -- that will be the first execution either has ever had, and I
  will report what it shows rather than predicting it.

Finding 7 (Low): still stacked on #127; the tests/CMakeLists.txt hunk goes
when that lands.

Refs #129

* ci(eosim): copy platforms/ in every job that runs a simulation

The first real execution of these workflows -- which the pull_request trigger
added in the previous commit made possible -- failed, and showed the next
blocker behind the missing v0.1.0 tag:

    FileNotFoundError: [Errno 2] No such file or directory:
      '/opt/hostedtoolcache/Python/3.12.14/x64/lib/python3.12/site-packages/platforms'

on all three Guest OS Install legs and all seven Nested Simulation platforms.

platforms/ is data, not package data: eosim/cli/main.py sets PLATFORMS_DIR to
<site-packages>/platforms while the wheel ships only
eosim/platforms/__init__.py. None of the five jobs that run eosim copied it,
so none of them could ever have worked. A tag that exists was necessary and
not sufficient, exactly as the review said.

  Worth recording: this is also what the dropped `pip install -e` was
  accidentally papering over in the jobs that had a copy elsewhere. With an
  editable install eosim.__file__ points into the clone, so PLATFORMS_DIR
  landed beside the checked-out platforms/ by luck. A plain install is the
  right thing and it made the real dependency visible.

install-validate went green in that run and the tracking job reported the
packaging debt as intended; the failures were confined to the jobs that
actually boot a platform.

Verified:
  both workflows parse; all five eosim-running jobs now copy platforms/
  the run that produced the error: 33783625398 (EoSim Sanity), first
  execution this workflow has ever had

Refs #129

* ci(eosim): run the bash steps in bash, including on the Windows legs

Second real execution, after the platforms/ fix: ubuntu and macOS now pass
Install & Validate; all three windows-latest legs fail.

    Install & Validate (windows-latest, Python 3.12)
      #   FileNotFoundError: .../site-packages/platforms
      ##[error]Process completed with exit code 1

The give-away is the leading `#` in that line: pwsh echoed my comment as a
command. install-validate is a matrix over ubuntu/windows/macos and set no
`shell:`, so on the Windows legs it got the runner default, where
`VAR=$(...)`, `cp -r` and a leading `#` are all wrong. windows-sanity had
the same gap.

`shell: bash` on the six run steps in the two Windows-capable jobs. This is
the same defect the review raised as finding 5 for the hardcoded /tmp path --
${{ runner.temp }} fixed where the clone goes, and this fixes what interprets
the script.

Worth stating plainly: none of this was visible before the pull_request
trigger. Two runs have now found two blockers behind the missing v0.1.0 tag --
platforms/ never copied in any job, and bash steps running under pwsh -- in a
workflow that had never executed once. The trigger is doing more work than the
pin.

Verified:
  parses; no run step in a windows-capable job is left without a shell
  run 33784349071: ubuntu + macOS Install & Validate green, windows red
    with the error above

Refs #129

* ci(sim): run the bash steps in bash on the Windows leg too

Third run. EoSim Sanity is green -- 23/23 jobs, the first green run that
workflow has ever had. Simulation Test has one failure left,
`Cross-Platform (windows-latest)`, and it is the same defect I fixed in
eosim-sanity.yml one commit ago and did not apply here:

    Cross-Platform (windows-latest)  Install EoSim from source
      # fails with FileNotFoundError.
      ##[error]Process completed with exit code 1

pwsh echoing a `#` comment as a command. The cross-platform matrix includes
windows-latest and its two run steps set no `shell:`.

Refs #129

* ci(eosim): one install recipe, gates that refuse empty input, and a watch that stops posing as a check

Answers the overnight review on #129.

Finding 3 (Low, and the structural one) -- the five-line install recipe was
copy-pasted at seven sites across two workflows, and the PR's own history
shows the cost: -e dropped "at all five sites, not two", runner.temp at all
of them, shell: bash at several, each a separate round because each site was
a separate edit. Now .github/actions/install-eosim owns it -- clone at the
pinned commit, install, stage platforms/, assert the dist version -- and all
seven sites are a four-line `uses:`. The platforms/ workaround and the
version assertion get one place to be deleted from when EoSim fixes its
packaging. The triplicated inline importlib.metadata assert is gone with it.

Finding 5 (Low), folded into the composite -- SITE_PACKAGES comes from
sysconfig.get_paths()["purelib"], which needs no import and cannot be
shadowed, matching eBoot's copy of the workaround instead of solving the same
problem a second way.

Finding 1 (Medium) -- the inline gate accepted nothing as everything: an
empty RESULTS printed "All EoSim sanity jobs succeeded." and exited 0, and
null produced a raw jq error, where eos#121's extracted ci-gate-check.sh
refuses both. Both gates now refuse empty/null input with a diagnosis before
jq runs, under set -euo pipefail, and carry a note to become a call to #121's
script once it is on master so the two cannot drift.

  Reproduced the review's side-by-side on the hardened body:
      ""      rc=1  gate received no results to check   (was rc=0 PASS)
      null    rc=1  gate received no results to check   (was raw jq error)
      {}      rc=0  PASS   -- shared with #121's script; reported there, and
      []      rc=0  PASS      unreachable here: needs has five entries
      skipped rc=1  FAIL: a: skipped
      success rc=0  PASS

Also applied to simulation-test.yml's Simulation Gate, which still had the
original fail-open shape -- needs: [simulate, cross-platform], a test of
`simulate` alone, then "All simulation tests passed". Same toJSON(needs)
body, same guards.

Finding 4 (Low) -- published-artifact-watch could not fail and sat in
sanity-gate's needs under a name that made a green tick assert a deficiency.
Now "EoSim artifact publication (informational)", schedule/dispatch only, out
of the gate: pull requests no longer carry a permanently-green status about
another repository's release process, and the debt still gets said out loud
every night.

Finding 6 (Low) -- simulation-test's EOSIM_COMMIT comment argued tag
selection for what is now a SHA; it now states the actual reason (EoSim's tag
names do not identify their contents) and gains the EOSIM_EXPECTED_DIST the
composite asserts.

Finding 2 (Medium) is a two-line change in eos#121's registry, not here --
both workflows need NOT_REQUIRED entries with a paths-filter reason once both
PRs land, or #121's classification test fails. Posted on both threads with
the exact entries; not folded in, because that registry lives on #121's
branch and this PR must not carry another PR's hunks.

Verified locally before push:
  all three YAML files parse
  composite referenced at 7 sites; zero inline clone/install/copy remnants
  eosim-sanity gate needs == every job except itself and the informational
    watch; sim gate iterates toJSON(needs), no single-dependency branch
  the five-input table above, executed
  the workflows themselves: this push's pull_request run is the test, and I
  will report what it shows rather than predict it.

Refs #129

* ci: pin the eBoot and ebuild checkouts instead of floating on master

Every open eos pull request is currently red on `EoS Full-Stack Simulation`,
and none of them caused it. The kernel job checks out embeddedos-org/eBoot at
`ref: master`, and eBoot master does not compile: include/eos_image.h:135 and
:142 static-assert offsetof(eos_image_header_t, reserved) for a member that
became tlv_len + tlv_hash, and core/ed25519_verify.c redefines
point_is_identity -- two pairs of PRs that each merged clean and broke the
build together. eBoot #94 repairs it and waits on review; until it lands, no
change to any eos PR can make this job pass, and after it lands the next
broken eBoot master does the same thing again.

A floating `ref: master` makes every eos PR's CI depend on the moment-to-
moment state of another repository's default branch. Pinning the two
compiled-against checkouts makes the job a function of this repository plus
two recorded SHAs -- red means the PR broke something, green means it did not,
and a cross-repo bump is a reviewable one-line diff with CI on it.

  EBOOT_COMMIT  a172a6d6e1e6  the newest eBoot master ancestor that compiles;
                found by walking master back and building each candidate.
                Its child 8a015b2 already carries the ed25519 redefinition.
  EBUILD_COMMIT e5d8052f3e2c  ebuild's current master, frozen as-is. Both of
                its checkouts feed steps that are commented out (eFab does
                not exist), so this pin only removes the float.

The middleware matrix (10 repos) still floats; it runs those repos' own
tests rather than compiling eos against them, and 10 more pins deserve their
own decision. When eBoot #94 merges, bump EBOOT_COMMIT to the merge SHA.

Verified before pushing, master vs pin under the job's own configuration
(-DEBLDR_BOARD=qemu_arm64 -DEBLDR_VERIFY_STAGE1=OFF -DCMAKE_BUILD_TYPE=Release,
cross-compile mode): eBoot@master fails at the eos_image.h asserts, the exact
error in the job log; eBoot@a172a6d builds every target, rc=0. Both pinned
SHAs verified reachable from their repos' master. Workflow parses; the three
`ref:` edits and the env block are the whole diff. This PR's own simulation
run is the live test: it uses this branch's workflow and should go green
while every master-based PR stays red.

Carried onto this branch from #135 (001128d) so this PR's own simulation
job runs against a buildable eBoot: each pull request executes its own copy
of the workflow, so the fix greens a PR only once its branch contains it.
Byte-identical to #135's commit -- git drops it as already-upstream the
moment #135 merges. The pin itself is #135's to review.
Kartikey1306 added a commit to Kartikey1306/eos that referenced this pull request Sep 14, 2026
Nine PRs were merged into master within minutes on 09-08, each on the base it
was written against. Master has not compiled since, and the Python guards
that would have named the rest never ran because the C build failed first.

Compile (every C job, CodeQL, and the ARM64 kernel in the simulation):
- services/linux/src/linux_security.c: embeddedos-org#119 and embeddedos-org#132 each added an #else to
  the same #ifndef _WIN32 in eos_busybox_install_to_rootfs(), so master has
  "#else after #else". The embeddedos-org#132 arm (`(void)bb;`) is the one removed -- the
  embeddedos-org#119 arm already uses bb and reports the unsupported platform.

Guards from embeddedos-org#121 / embeddedos-org#93 that later merges walked back:
- ci.yml: embeddedos-org#132 added windows-test after embeddedos-org#121's gate; the gate did not wait
  for it, so "CI Gate" could be green with the MSVC leg red.
- test_ci_gate.py: embeddedos-org#129 gave eosim-sanity.yml and simulation-test.yml a
  path-filtered pull_request trigger (they test their own edits); a
  path-filtered check cannot be required, so both are recorded in
  NOT_REQUIRED with the book-build.yml reason.
- tests/CMakeLists.txt: test_linux_security_paths (embeddedos-org#119) and test_pkg_fetch
  (embeddedos-org#115) had no add_executable(). embeddedos-org#115's replay replaced embeddedos-org#119's block with
  its own, and embeddedos-org#118's replay replaced that; two suites compiled against
  nothing. Both registered again. 41 -> 43 suites.
- tests/test_kernel.c: four tests from embeddedos-org#130 and embeddedos-org#131 were defined and never
  called -- their RUN() lines did not survive the replay of main().

And the one that was not bookkeeping:
- kernel/src/task.c: embeddedos-org#130 was merged after embeddedos-org#131 from a base that predates it,
  and its copy of task.c replaced embeddedos-org#131's. embeddedos-org#131 had also flattened 574 CRLF
  line endings, so its 1166-line diff hid a 39/21 change and the replay took
  embeddedos-org#130's side wholesale. Master kept embeddedos-org#131's tests and lost its kernel: the
  idle task could be deleted and suspended, a half-initialised TCB was
  published to the scheduler before its stack existed, and eos_schedule()
  pointed g_current_sp at the outgoing task. test_idle_task_is_permanent
  fails on master the moment it is called. embeddedos-org#131's task.c diff re-applied
  on top of embeddedos-org#130's; the result is embeddedos-org#131's file plus embeddedos-org#130's wake_armed hunk
  and nothing else (verified by diff against 3a00bd9).

Verified locally (macOS, clang): Release build clean, 43/43 ctest; the
README default configuration builds; 47/47 pytest.

Not in this PR: the nightly "Upstream drift" job builds eBoot at master and
eBoot master is broken separately (its own repair PR); bump EBOOT_COMMIT in
eos-simulation.yml once that lands.
Kartikey1306 added a commit to Kartikey1306/eos that referenced this pull request Sep 14, 2026
The pin in the previous commit stops an eBoot or ebuild master that does not
compile from turning every eos pull request red. It also stops that break
being visible from here at all, which on its own is the worse half of the
trade: the next upstream breakage would be silent instead of loud, and the
pin would sit at a172a6d indefinitely because nothing would ever ask -- while
quietly dropping the six eBoot commits that follow it.

`.github/workflows/upstream-drift.yml` is the other half. It builds eBoot at
master and compares both pins against their master, so an upstream break
still surfaces, on a run that blocks no pull request and names the repository
responsible.

Design choices, each one avoiding a defect this repo has been filing against:

  Its own file, schedule + workflow_dispatch only. The first draft was a job
  inside eos-simulation.yml guarded by `if: github.event_name == 'schedule'`.
  That works, but the run showed it on the PR as a SKIPPED check -- and a
  required-check gate that treats any non-success as a failure, which is the
  design in embeddedos-org#121, would trip on exactly that. Moving it out removes the check
  from PR runs entirely rather than relying on every future gate to exempt it.

  Not `continue-on-error`. A job reporting success over a failure is the
  fail-open shape we keep finding in aggregating gates. This one may fail
  honestly precisely because it gates nothing.

  Failing is reserved for "upstream does not build". A pin merely behind a
  healthy master is a warning: actionable, not broken.

  The pins are read out of eos-simulation.yml, not copied. A second copy of a
  SHA is a second thing to forget. The read demands a 40-character hex value
  and fails closed if it cannot find one, so a renamed or reformatted key
  stops the job instead of silently letting it check a pin of "".

  The failure message is scoped to the build step's own outcome, so a
  checkout, apt or pin-read failure cannot report itself as evidence about
  eBoot's master. The summary says eBoot is built and ebuild is only
  compared, rather than implying both were checked.

Verified by execution, not by reading:

  eBoot @ a172a6d (the pin)   host build rc=0, 0 errors
  eBoot @ 22d8f8b (master)    host build rc=2, "no member named 'reserved'
                              in 'eos_image_header_t'" at eos_image.h:135
                              and :142 -- the same defect CI reports, so the
                              pin is demonstrably the difference

  pin extraction, run against the real file and two mutations of it:
    real workflow                -> both SHAs read correctly, and the eBoot
                                    one resolves to embeddedos-org#86's merge
    key renamed to EBOOT_SHA     -> fails closed, rc=1
    value unquoted and shortened -> fails closed, rc=1

  drift logic, run against the real repositories:
    pin 6 behind healthy master  -> "behind by 6" warning, no failure
    pin absent from master       -> "not an ancestor" warning, exit 0 under
                                    set -e (the first draft printed "pinned
                                    pin not found on master commit(s)")
    pin equal to master          -> no warning

  eos on this branch: cmake build rc=0, ctest 39/39, pytest 15/15 with the
  new workflow file present, test_crypto_ed25519_loworder registered and
  passing.

When eBoot embeddedos-org#94 lands, bump EBOOT_COMMIT to that merge; this job is what will
say whether it holds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants