Skip to content

Commit 6b0979c

Browse files
build: commit the rebuilt engines to main under engine/binaries/<lang>/<platform>/ — a checkout carries its own engines
The publish job places all four binaries of each rebuilt language under engine/binaries/<lang>/<platform>/, records the rule-set id in ENGINE_ID, and pushes one commit to main (rebased and retried if main moved during the build); a language missing a platform is left unchanged so the next merge retries it. The plan step now compares each language's id against the committed ENGINE_ID instead of a release. run-souffle.sh resolves the engine in order: the committed binary when its ENGINE_ID equals the checkout's id, a local compile when souffle is on PATH, then the release. The release stays as a secondary copy. engine-fetch-test.sh covers the committed path: used when the id matches, reported and skipped when it does not.
1 parent f11e603 commit 6b0979c

5 files changed

Lines changed: 158 additions & 58 deletions

File tree

‎.gitattributes‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
engine/binaries/** binary -text

‎.github/workflows/engine-binaries.yml‎

Lines changed: 85 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -3,20 +3,23 @@
33
#
44
# EVERY merge to main runs this. Each language's engine id (src/pipeline/run-souffle.sh
55
# --print-engine-id) is a hash of that language's rules alone, so the plan step decides per
6-
# language: an id that already has a release is unchanged and skipped; an id without one is
7-
# built on every platform. A Python-only merge therefore rebuilds Python on all four targets
8-
# and nothing else; a merge touching no rule set builds nothing and finishes in seconds. For
9-
# each language that needs building:
6+
# language: an id equal to the one recorded in engine/binaries/<lang>/ENGINE_ID is unchanged
7+
# and skipped; any other id is built on every platform. A Python-only merge therefore rebuilds
8+
# Python on all four targets and nothing else; a merge touching no rule set builds nothing and
9+
# finishes in seconds. For each language that needs building:
1010
# generate Ubuntu + the pinned Soufflé .deb: emit the program, `souffle -g` it to portable
1111
# C++ once, and capture the header tree the C++ compiles against.
1212
# build compile that C++ on every platform with the SAME flags the script uses locally
1313
# (-std=c++17 -O3, no OpenMP/zlib/sqlite defines — the binary links against nothing
1414
# but the C++ runtime). Portable target: no -march=native; macOS universal.
15-
# release one GitHub release per (language, id), tagged engine-<lang>-<id>, holding
16-
# axiom-engine-<lang>-<os>-<arch>[.exe], sha256sum.txt, and the generated .cpp.
17-
# run-souffle.sh computes the same id on the user's machine and fetches the matching asset.
18-
# The id is a function of the repository alone (see engine_id in the script), which is what
19-
# makes "no release for this id" mean exactly "these rules have not been merged and built".
15+
# publish COMMIT the binaries to main under engine/binaries/<lang>/<platform>/ with the id
16+
# they were built from in engine/binaries/<lang>/ENGINE_ID — so a checkout carries
17+
# its own engines and nothing is downloaded — and, as a secondary copy, a GitHub
18+
# release tagged engine-<lang>-<id> with the same files plus the generated .cpp.
19+
# run-souffle.sh computes the same id on the user's machine and uses the committed binary only
20+
# when the ids agree. The id is a function of the repository alone (see engine_id in the
21+
# script), which is what makes "id differs" mean exactly "these rules were edited after the
22+
# binary was built".
2023
# ─────────────────────────────────────────────────────────────────────────────
2124
name: engine-binaries
2225

@@ -62,10 +65,11 @@ jobs:
6265
for lang in $LANGUAGES; do
6366
id="$(bash src/pipeline/run-souffle.sh --language "$lang" --print-engine-id)"
6467
echo "$lang: $id"
65-
if [ "$FORCE" != "true" ] && [ "${{ github.event_name }}" != "pull_request" ] && gh release view "engine-$lang-$id" >/dev/null 2>&1; then
66-
echo " unchanged — release engine-$lang-$id exists"; continue
68+
have="$(tr -d '[:space:]' < "engine/binaries/$lang/ENGINE_ID" 2>/dev/null || true)"
69+
if [ "$FORCE" != "true" ] && [ "${{ github.event_name }}" != "pull_request" ] && [ "$have" = "$id" ]; then
70+
echo " unchanged — engine/binaries/$lang is at this id"; continue
6771
fi
68-
echo " changed — no release for this id; building on every platform"
72+
echo " changed — engine/binaries/$lang is at '${have:-<none>}'; building on every platform"
6973
printf '%s' "$id" > "out/$lang.id"
7074
langs="$langs \"$lang\""
7175
done
@@ -184,48 +188,89 @@ jobs:
184188
path: axiom-engine-*
185189
if-no-files-found: error
186190

187-
# Per language, and independent of the other languages' builds: `always()` lets this run
188-
# when some build job failed, and the step below publishes a language only if ALL FOUR of
189-
# its binaries exist. A language with a missing platform gets no release — so the next
190-
# merge retries it — while the others publish.
191-
release:
191+
# One job for every language built: `always()` lets it run when some build failed, and a
192+
# language is published only if ALL FOUR of its binaries exist — a language with a missing
193+
# platform is left as it was (the next merge retries it, since its ENGINE_ID still differs)
194+
# while the others go in. One commit on main carries everything that was rebuilt.
195+
publish:
192196
needs: [generate, build]
193197
if: always() && needs.generate.result == 'success' && needs.generate.outputs.langs != '[]' && github.event_name != 'pull_request'
194198
runs-on: ubuntu-24.04
195-
strategy:
196-
matrix:
197-
lang: ${{ fromJson(needs.generate.outputs.langs) }}
198199
steps:
200+
- uses: actions/checkout@v4
201+
with:
202+
ref: main
199203
- uses: actions/download-artifact@v4
200204
with:
201205
name: generated
202206
path: gen
203207
- uses: actions/download-artifact@v4
204208
with:
205-
pattern: engine-${{ matrix.lang }}-*
209+
pattern: engine-*
206210
path: bins
207211
merge-multiple: true
208212

209-
- name: Publish release engine-${{ matrix.lang }}-<id>
213+
- name: Place the binaries under engine/binaries/<lang>/<platform>/
214+
id: place
215+
run: |
216+
set -e
217+
echo "artifacts:"; ls -la bins
218+
published=""
219+
for lang in $LANGUAGES; do
220+
[ -f "gen/$lang.id" ] || continue
221+
id="$(cat "gen/$lang.id")"
222+
n="$(ls bins/axiom-engine-$lang-* 2>/dev/null | grep -vc '\.sha256$' || true)"
223+
if [ "$n" != 4 ]; then
224+
echo "::warning::$lang: only $n of 4 platform binaries were built — engine/binaries/$lang left unchanged; the next merge to main retries"
225+
continue
226+
fi
227+
rm -rf "engine/binaries/$lang"
228+
for f in bins/axiom-engine-$lang-*; do
229+
case "$f" in *.sha256) continue;; esac
230+
name="$(basename "$f")"
231+
platform="${name#axiom-engine-$lang-}"; platform="${platform%.exe}"
232+
mkdir -p "engine/binaries/$lang/$platform"
233+
cp "$f" "engine/binaries/$lang/$platform/$name"
234+
chmod +x "engine/binaries/$lang/$platform/$name"
235+
done
236+
cat bins/axiom-engine-$lang-*.sha256 | sed 's# .*/# #' > "engine/binaries/$lang/sha256sum.txt"
237+
printf '%s\n' "$id" > "engine/binaries/$lang/ENGINE_ID"
238+
printf 'Built by the engine-binaries workflow from %s.\nRule-set id: %s (src/pipeline/run-souffle.sh --language %s --print-engine-id).\nrun-souffle.sh uses these only while the rules hash to that id.\n' "${{ github.sha }}" "$id" "$lang" > "engine/binaries/$lang/BUILD.txt"
239+
published="$published $lang"
240+
done
241+
echo "published=${published# }" >> "$GITHUB_OUTPUT"
242+
git status --short engine/binaries | head -20
243+
244+
- name: Commit to main
245+
if: steps.place.outputs.published != ''
246+
run: |
247+
set -e
248+
git config user.name "engine-binaries"
249+
git config user.email "engine-binaries@users.noreply.github.com"
250+
git add -A engine/binaries
251+
git commit -m "engine: rebuilt binaries for ${{ steps.place.outputs.published }} from ${{ github.sha }}" \
252+
-m "Built on linux-x86_64, linux-arm64, darwin-universal and windows-x86_64 by the engine-binaries workflow. Each language's ENGINE_ID is the id of the rules it was compiled from; run-souffle.sh uses the binary only while the checkout's rules hash to that id."
253+
# main may have moved during the build; rebase our one commit on top and retry.
254+
for attempt in 1 2 3 4 5; do
255+
git push origin HEAD:main && exit 0
256+
echo "push rejected (attempt $attempt) — rebasing on the current main"
257+
git fetch origin main && git rebase origin/main
258+
done
259+
echo "::error::could not push the binaries to main after 5 attempts"; exit 1
260+
261+
- name: Secondary copy — a release per language
262+
if: steps.place.outputs.published != ''
210263
env:
211264
GH_TOKEN: ${{ github.token }}
212-
FORCE: ${{ inputs.force }}
213265
run: |
214266
set -e
215-
lang="${{ matrix.lang }}"
216-
id="$(cat "gen/$lang.id")"
217-
tag="engine-$lang-$id"
218-
n="$(ls bins/axiom-engine-$lang-* 2>/dev/null | grep -vc '\.sha256$')"
219-
if [ "$n" != 4 ]; then
220-
echo "::warning::$lang: only $n of 4 platform binaries were built — not publishing $tag; the next merge to main retries"
221-
exit 0
222-
fi
223-
cat bins/*.sha256 | sed 's# .*/# #' > bins/sha256sum.txt
224-
rm -f bins/*.sha256
225-
cp "gen/$lang.cpp" "bins/$lang.cpp"
226-
echo "assets:"; ls -la bins
227-
notes="Prebuilt engine for the **$lang** rule set at id \`$id\` (Soufflé $(cat gen/souffle.version), pinned in src/pipeline/engine.conf), built from ${{ github.sha }}.
228-
229-
\`run-souffle.sh --language $lang\` fetches the asset for its platform when \`souffle\` is not installed. \`$lang.cpp\` is the generated program for anyone who prefers to compile it themselves (needs only the Soufflé headers and a C++17 compiler)."
230-
if [ "$FORCE" = "true" ]; then gh release delete "$tag" -y 2>/dev/null || true; fi
231-
gh release create "$tag" --title "engine · $lang · ${id:0:12}" --notes "$notes" --target "${{ github.sha }}" bins/*
267+
for lang in ${{ steps.place.outputs.published }}; do
268+
id="$(cat "gen/$lang.id")"; tag="engine-$lang-$id"
269+
mkdir -p "rel/$lang"
270+
cp engine/binaries/$lang/*/axiom-engine-* engine/binaries/$lang/sha256sum.txt "rel/$lang/"
271+
cp "gen/$lang.cpp" "rel/$lang/$lang.cpp"
272+
gh release delete "$tag" -y 2>/dev/null || true
273+
gh release create "$tag" --title "engine · $lang · ${id:0:12}" --target main \
274+
--notes "Prebuilt engine for the **$lang** rule set at id \`$id\` (Soufflé $(cat gen/souffle.version), pinned in src/pipeline/engine.conf), built from ${{ github.sha }}. The same binaries are committed under engine/binaries/$lang/; \`$lang.cpp\` is the generated program for anyone who prefers to compile it (Soufflé headers + a C++17 compiler)." \
275+
rel/$lang/*
276+
done

‎README.md‎

Lines changed: 13 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -127,10 +127,11 @@ receiver is a lambda parameter. Restricted to files of 1,000 lines or more, d1 r
127127
## Run
128128

129129
Requirements: Node ≥ 22.5 and a POSIX shell with `awk` (Git Bash on Windows). **No Soufflé and
130-
no C++ compiler**: the rules compile to one self-contained executable, CI builds it for
131-
Linux (x86_64, arm64), macOS (universal) and Windows on every merge to `main`, and the script
132-
fetches the one for your platform on first use — verified by sha256, cached under the rule
133-
set's id. With `souffle` installed the script compiles locally instead, exactly as before.
130+
no C++ compiler**: the rules compile to one self-contained executable, and CI builds it for
131+
Linux (x86_64, arm64), macOS (universal) and Windows on every merge to `main` and commits it
132+
to `engine/binaries/<lang>/<platform>/` — so a checkout already contains the engine for every
133+
platform and nothing is downloaded. With `souffle` installed the script compiles locally
134+
instead, exactly as before.
134135

135136
```bash
136137
npm install && npm run build
@@ -146,11 +147,14 @@ bash src/pipeline/run-souffle.sh \
146147
--intermediate <scratch> --output <out>
147148
```
148149

149-
The prebuilt binaries live in GitHub releases tagged `engine-<lang>-<id>`, where `<id>` is
150-
`bash src/pipeline/run-souffle.sh --language <lang> --print-engine-id` — a sha256 of the rules
151-
and the pinned Soufflé version (`src/pipeline/engine.conf`), the same from any checkout. The
152-
repository is private, so the fetch authenticates through `gh` or a `GH_TOKEN`. A rule set that
153-
has not been merged has no release: edit rules with Soufflé installed, or merge first.
150+
`engine/binaries/<lang>/ENGINE_ID` records the rule-set id the committed binaries were built
151+
from — `bash src/pipeline/run-souffle.sh --language <lang> --print-engine-id`, a sha256 of the
152+
rules and the pinned Soufflé version (`src/pipeline/engine.conf`), the same from any checkout.
153+
The script uses a committed binary only while the checkout's rules hash to that id; rules you
154+
have edited locally need Soufflé installed to compile, or a merge to `main` so CI rebuilds and
155+
commits them. The same binaries are also published as a GitHub release tagged
156+
`engine-<lang>-<id>`, which the script falls back to fetching (sha256-verified) when neither
157+
a committed nor a locally compiled engine exists.
154158

155159
**Outputs — the same in every language** ([`src/bundle/SCHEMA.md`](src/bundle/SCHEMA.md))
156160

‎src/pipeline/run-souffle.sh‎

Lines changed: 29 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,13 @@
88
#
99
# NO SOUFFLÉ NEEDED TO RUN. The rules compile to one self-contained executable that is
1010
# project-independent; CI builds it for every platform on merge (engine-binaries.yml) and
11-
# publishes it under a release tagged engine-<lang>-<id>. When `souffle` is not on PATH this
12-
# script fetches that binary for the local platform (once, into the cache) and verifies its
13-
# sha256. With souffle installed it compiles locally as before. See src/pipeline/engine.conf.
11+
# COMMITS it under engine/binaries/<lang>/<platform>/ next to the id of the rules it was
12+
# built from, so a checkout carries its own engines. The binary is resolved in this order:
13+
# 1. engine/binaries/<lang>/<platform>/ — used only if its ENGINE_ID equals the id of the
14+
# rules in this checkout (edited rules never silently run a stale binary);
15+
# 2. a locally compiled engine, when `souffle` is on PATH (the cache under .souffle-cache);
16+
# 3. the GitHub release tagged engine-<lang>-<id> (sha256-verified), when it exists.
17+
# See src/pipeline/engine.conf.
1418
#
1519
# OUTPUT LAYOUT — the same in every language (src/bundle/SCHEMA.md):
1620
# $OUT/graph.sqlite the contract: core tables + ext_* tables + the schema catalog
@@ -390,7 +394,26 @@ fetch_engine(){
390394
echo "▶ verified sha256 $sum → $BIN"
391395
}
392396

393-
if [ -x "$BIN" ]; then
397+
# 1. the binary committed with the repository, if it was built from exactly these rules
398+
COMMITTED_DIR="$SRC/../engine/binaries/$LANG_ARG"
399+
COMMITTED=""
400+
if [ -f "$COMMITTED_DIR/ENGINE_ID" ]; then
401+
if [ "$(tr -d '[:space:]' < "$COMMITTED_DIR/ENGINE_ID")" = "$ENGINE_ID" ]; then
402+
platform="$(engine_platform 2>/dev/null || true)"
403+
cand="$COMMITTED_DIR/$platform/axiom-engine-$LANG_ARG-$platform$EXE"
404+
if [ -n "$platform" ] && [ -f "$cand" ]; then
405+
COMMITTED="$cand"; chmod +x "$COMMITTED" 2>/dev/null || true
406+
else
407+
echo " ! engine/binaries/$LANG_ARG matches these rules but has no binary for ${platform:-this platform}"
408+
fi
409+
else
410+
echo " ! engine/binaries/$LANG_ARG was built from $(cut -c1-12 "$COMMITTED_DIR/ENGINE_ID")…, these rules are ${ENGINE_ID:0:12}… — not using it"
411+
fi
412+
fi
413+
414+
if [ -n "$COMMITTED" ]; then
415+
BIN="$COMMITTED"; echo "▶ using committed engine ${COMMITTED#"$SRC/../"}"
416+
elif [ -x "$BIN" ]; then
394417
echo "▶ reusing cached binary"
395418
elif command -v souffle >/dev/null 2>&1; then
396419
echo "▶ compiling souffle program (cache miss)..."
@@ -421,9 +444,9 @@ elif command -v souffle >/dev/null 2>&1; then
421444
mv -f "$BIN.tmp.$$" "$BIN"
422445
else
423446
fetch_engine || {
424-
echo "❌ no engine for $LANG_ARG@$ENGINE_ID on this machine. Either:" >&2
447+
echo "❌ no engine for $LANG_ARG@${ENGINE_ID:0:12}… on this machine. Either:" >&2
425448
echo " • install souffle $SOUFFLE_VERSION to compile locally (macOS: brew install souffle; Ubuntu: the .deb from souffle-lang/souffle releases), or" >&2
426-
echo " • use a rule set CI has built: merge to main, wait for the engine-binaries workflow, then rerun." >&2
449+
echo " • use a rule set CI has built: merge to main, wait for the engine-binaries workflow to commit engine/binaries/$LANG_ARG, then pull and rerun." >&2
427450
exit 1
428451
}
429452
fi

‎test/tools/engine-fetch-test.sh‎

Lines changed: 30 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -11,11 +11,15 @@
1111
# ─────────────────────────────────────────────────────────────────────────────
1212
set -u
1313
ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
14-
RUN="$ROOT/src/pipeline/run-souffle.sh"
1514
fail=0; bad(){ echo " ✗ $*"; fail=$((fail+1)); }
1615
[ -x "$ROOT/node_modules/.bin/tsx" ] || { echo "engine-fetch: SKIP (no node_modules/.bin/tsx — run npm install)"; exit 0; }
1716
W="$(mktemp -d)"; trap 'rm -rf "$W"' EXIT
18-
mkdir -p "$W/bin" "$W/release" "$W/cache" "$W/ir" "$W/int" "$W/out"
17+
mkdir -p "$W/bin" "$W/release" "$W/cache" "$W/ir" "$W/int" "$W/out" "$W/tree"
18+
# a copy of the tree, so a committed engine can be planted under engine/binaries/ without
19+
# touching the repository; the bundler's node_modules and tsconfig are shared by link
20+
cp -R "$ROOT/src" "$W/tree/src"
21+
ln -s "$ROOT/node_modules" "$W/tree/node_modules"; ln -s "$ROOT/tsconfig.json" "$W/tree/tsconfig.json"; ln -s "$ROOT/package.json" "$W/tree/package.json"
22+
RUN="$W/tree/src/pipeline/run-souffle.sh"
1923
# a PATH with everything the driver and the bundler need, and no souffle
2024
for t in bash sh grep awk sed sort cut tr mktemp uname cat rm cp mv ls dirname basename shasum sha256sum stat date mkdir chmod head tail wc find ln printf tee env node git curl; do
2125
p="$(command -v "$t" 2>/dev/null)" && ln -sf "$p" "$W/bin/$t"
@@ -50,6 +54,29 @@ export FAKE_TAG="engine-$lang-$id" FAKE_RELEASE="$W/release"
5054

5155
run(){ PATH="$W/bin" AXIOM_SOUFFLE_CACHE="$W/cache" bash "$RUN" --language $lang --client-ir "$W/ir" --library "" --intermediate "$W/int" --output "$W/out" > "$W/log" 2>&1; }
5256

57+
# 0. a COMMITTED engine whose ENGINE_ID matches the rules is used directly — no download
58+
committed="$W/tree/engine/binaries/$lang"
59+
mkdir -p "$committed/$platform"; cp "$W/release/$asset" "$committed/$platform/$asset"; printf '%s\n' "$id" > "$committed/ENGINE_ID"
60+
export FAKE_TAG="engine-$lang-must-not-be-asked"
61+
if run; then
62+
grep -q "using committed engine" "$W/log" || bad "committed engine with a matching id was not used"
63+
grep -q "fetching prebuilt" "$W/log" && bad "a download was attempted although a matching committed engine exists"
64+
[ -f "$W/out/graph.sqlite" ] || [ -f "$W/out/graph/call_edges.csv" ] || bad "the committed-engine run did not reach the bundle stage"
65+
else
66+
bad "run with a matching committed engine failed:"; tail -8 "$W/log" | sed 's/^/ /'
67+
fi
68+
# 0b. a committed engine built from OTHER rules is ignored, with a message, and the fetch runs
69+
printf 'deadbeef%s\n' "${id:8}" > "$committed/ENGINE_ID"; rm -rf "$W/out"
70+
export FAKE_TAG="engine-$lang-$id"
71+
if run; then
72+
grep -q "not using it" "$W/log" || bad "a stale committed engine was not reported"
73+
grep -q "using committed engine" "$W/log" && bad "a committed engine with a different id was used"
74+
grep -q "verified sha256" "$W/log" || bad "after ignoring the stale committed engine, no fetch happened"
75+
else
76+
bad "run with a stale committed engine and a valid release failed:"; tail -8 "$W/log" | sed 's/^/ /'
77+
fi
78+
rm -rf "$W/tree/engine" "$W/cache"/* "$W/out"
79+
5380
# 1. a good release is fetched, verified, cached and run through to the bundle
5481
if run; then
5582
grep -q "verified sha256" "$W/log" || bad "no 'verified sha256' line in the log"
@@ -74,4 +101,4 @@ if run; then bad "a run with no release and no souffle succeeded"; else
74101
grep -q "install souffle" "$W/log" && grep -q "merge to main" "$W/log" || bad "the no-release error does not name both ways out"
75102
fi
76103

77-
if [ "$fail" -eq 0 ]; then echo "engine-fetch: ok (fetch, verify, cache, refuse tampered, explain absence)"; else echo "engine-fetch: $fail failure(s)"; exit 1; fi
104+
if [ "$fail" -eq 0 ]; then echo "engine-fetch: ok (committed engine by id, fetch, verify, cache, refuse tampered, explain absence)"; else echo "engine-fetch: $fail failure(s)"; exit 1; fi

0 commit comments

Comments
 (0)