Skip to content

Commit ec894e4

Browse files
build: a stopped query or index no longer loses the main graph (#1555)
A multi-language repository could be left with no .axiomcode/out/graph.sqlite after its first build: the other languages' graphs sat unpublished under .axiomcode/out/<lang>, every query said "no graph yet — building one", rebuilt every language, and lost the main graph again. The build a query starts ran in the query's process group. A caller that stops the query (a timeout, Ctrl-C, a host that ends the command's process group) stopped the build with it: before the main graph was published, or while it was being indexed, which the build took for a failure and answered by deleting the solved graph. On a repository whose first build outlasts the caller's timeout, every query repeated it. - ax_contract.ensure_graph starts the build in a session of its own, as the background refresher already is, so it finishes whatever becomes of the query that started it; a later query finds it running and waits for it. - axiomcode-build: a signal that arrives once the main language is solved is held until graph.sqlite points at that graph (the indexer runs in its own process group); the languages still unsolved are recorded in out/partial, so the graph reads as stale and the next build builds them. A signal before the main language is solved stops the build at once, as before. - A query waiting on a refresh no longer stops waiting when the main graph is published while the language that holds the edited file is still being solved: it answered from that language's previous graph. Tests: tests/multi_language.py gains the stopped-query and stopped-index checks (single-language and stopped-before-solve controls); its refresh and upgrade checks pass again. tests/refresh_races.py waits for the repair build before holding the lock and accepts the wait message #1555 introduced. Co-authored-by: axiomcode-bot[bot] <334110751+axiomcode-bot[bot]@users.noreply.github.com>
1 parent b271a22 commit ec894e4

6 files changed

Lines changed: 169 additions & 20 deletions

File tree

‎plugins/axiomcode/skills/axiomcode/scripts/ax_contract.py‎

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -452,8 +452,7 @@ def ensure_graph(repo, db):
452452
if nowait:
453453
os.makedirs(os.path.join(repo, '.axiomcode'), exist_ok=True)
454454
log = open(os.path.join(repo, '.axiomcode', 'first-build.log'), 'w')
455-
kw = dict(start_new_session=True) if os.name != 'nt' else dict(creationflags=0x00000008 | 0x00000200) # DETACHED | NEW_GROUP, as ax_fresh.kick
456-
p = subprocess.Popen([bash, build, repo], stdin=subprocess.DEVNULL, stdout=log, stderr=log, close_fds=True, env=env, **kw)
455+
p = _start_build([bash, build, repo], log, env)
457456
# until the build holds its lock a second query would see no build and start another, so that much is always waited
458457
up = time.time() + 10
459458
while p.poll() is None and not ax_fresh.building(repo) and time.time() < up: time.sleep(0.05)
@@ -469,7 +468,7 @@ def ensure_graph(repo, db):
469468
os.makedirs(os.path.join(repo, '.axiomcode'), exist_ok=True)
470469
logp = os.path.join(repo, '.axiomcode', 'first-build.log')
471470
with open(logp, 'w') as log:
472-
p = subprocess.Popen([bash, build, repo], stdin=subprocess.DEVNULL, stdout=log, stderr=log, close_fds=True, env=env)
471+
p = _start_build([bash, build, repo], log, env)
473472
shown = [0]
474473
def relay():
475474
try:
@@ -482,6 +481,25 @@ def relay():
482481
return p.returncode == 0 and os.path.exists(db)
483482

484483

484+
def _start_build(argv, log, env):
485+
"""the build a query starts, in a session (a process group) of its own, as ax_fresh.kick starts the refresher.
486+
487+
THE BUILD IS NOT THE QUERY'S (#1555). It goes on solving the other languages after the query has its answer, and a
488+
query is stopped all the time: a caller's timeout, Ctrl-C, an agent host that ends the command's process group when
489+
it returns or times out. Started in the query's group, the build was stopped with it -- on a repository in several
490+
languages, whose first build outlasts a two-minute timeout, before the main graph was published, or while it was
491+
being indexed, which deleted the solved graph as a failure. No graph.sqlite was left, the next query built again
492+
from nothing, was stopped again, and every query rebuilt forever. In its own session the build finishes whatever
493+
becomes of the query; a later query finds it running and waits for it rather than starting another."""
494+
if os.name != 'nt':
495+
return subprocess.Popen(argv, stdin=subprocess.DEVNULL, stdout=log, stderr=log, close_fds=True, env=env, start_new_session=True)
496+
# DETACHED | NEW_GROUP, and out of the caller's job object where the job allows it (see ax_fresh.kick)
497+
for flags in (0x00000008 | 0x00000200 | 0x01000000, 0x00000008 | 0x00000200):
498+
try: return subprocess.Popen(argv, stdin=subprocess.DEVNULL, stdout=log, stderr=log, close_fds=True, env=env, creationflags=flags)
499+
except OSError: continue
500+
return subprocess.Popen(argv, stdin=subprocess.DEVNULL, stdout=log, stderr=log, close_fds=True, env=env)
501+
502+
485503
def _published(repo):
486504
"""the main graph is out while the build goes on solving the repository's other languages: the answer is given now,
487505
and names on stderr (a `graph refresh:` line, which the MCP server carries into the answer) what it cannot see yet"""

‎plugins/axiomcode/skills/axiomcode/scripts/ax_fresh.py‎

Lines changed: 30 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -378,6 +378,15 @@ def pending(repo):
378378
old = [l for l in langs if os.path.isfile(os.path.join(repo, '.axiomcode', 'lang', l, 'out', 'graph.sqlite'))]
379379
return dict(pending=langs, pending_old=old, first=all(len(r) > 1 and r[1] == 'first' for r in rows))
380380

381+
def unbuilt(repo):
382+
"""the languages a build that was stopped after publishing the main graph left unbuilt (axiomcode-build writes them
383+
to out/partial), [] when none"""
384+
try: return open(os.path.join(out_dir(repo), 'partial')).read().split()
385+
except OSError: return []
386+
387+
def unbuilt_row(repo):
388+
return f".axiomcode/out/partial (the {', '.join(unbuilt(repo))} graph was not built: its build was stopped)"
389+
381390
def status(repo):
382391
if not has_graph(repo): return dict(state='no graph')
383392
if graph_broken(repo): return dict(state='building' if building(repo) else 'stale', changed=['.axiomcode/out/graph.sqlite (the pointer to the graph is broken)'], added=[], removed=[])
@@ -389,7 +398,8 @@ def status(repo):
389398
if not (changed or added or removed):
390399
# a build that already published this tree's main graph and is still solving other languages: the graph a query
391400
# reads is current, and waiting would be waiting for other languages' compiles
392-
return dict(state='building', **pending(repo)) if busy else dict(state='fresh')
401+
if busy: return dict(state='building', **pending(repo))
402+
return dict(state='stale', changed=[unbuilt_row(repo)], added=[], removed=[]) if unbuilt(repo) else dict(state='fresh')
393403
d = dict(state='building' if busy else 'stale', changed=changed, added=added, removed=removed)
394404
if not busy and st.get('failed_table') == change_key(c): d['failed'] = st.get('failed_log', ''); d['failed_reason'] = st.get('failed_reason', '')
395405
return d
@@ -451,6 +461,7 @@ def worker(repo):
451461
else:
452462
c = changes(repo, t)
453463
if graph_broken(repo) and not (c and any(c)): c = [['.axiomcode/out/graph.sqlite (broken pointer)'], [], []]
464+
elif unbuilt(repo) and not (c and any(c)): c = [[unbuilt_row(repo)], [], []]
454465
if (c is None or not any(c)) and not base_moved(repo):
455466
write_state(repo, state='fresh', checked=time.time(), checked_by=os.environ.get('AXIOMCODE_REFRESH_TRIGGER', '')); return 0
456467
c = c or [[], [], []]
@@ -718,13 +729,26 @@ def expected_left(repo):
718729
if st.get('state') == 'building' and st.get('started'): return secs - (time.time() - st['started'])
719730
return secs + float(os.environ.get('AXIOMCODE_REFRESH_DEBOUNCE') or 2)
720731

721-
def wait_fresh(repo, seconds, say=False):
732+
def _waits_on(s, files):
733+
"""the refresh has published the main graph and is still solving languages (#1555) of which one holds an edited file
734+
the wait is for (`files`; True: any language at all): the graph that answers for that edit is not out yet"""
735+
langs = set(s.get('pending') or ())
736+
if not langs or not files: return False
737+
if files is True: return True
738+
import ax_langs
739+
return any(l in langs for f in files for l in ax_langs.BY_EXT.get(os.path.splitext(f)[1].lower(), ()))
740+
741+
def wait_fresh(repo, seconds, say=False, files=()):
722742
"""wait until the graph matches the files again (the refresh swapped in its graph), a rebuild fails, or `seconds`
723-
pass. With `say`, a progress line on stderr every 10 s: a wait is never silent. True when the graph is current"""
743+
pass. With `say`, a progress line on stderr every 10 s: a wait is never silent. True when the graph is current.
744+
THE GRAPH OF THE EDIT, NOT THE MAIN ONE. A refresh publishes the main language's graph first and goes on with the
745+
others (#1555); from then on the file table matches, and a wait that ended there answered a question about a Python
746+
edit in a TypeScript repository from the previous Python graph: `nothing named` for the function just added. So a
747+
wait for `files` goes on while a language that holds one of them is still being solved (files=True: any language)."""
724748
end = t0 = time.time(); end += seconds; last = t0
725749
while True:
726750
s = status(repo)
727-
if s['state'] in ('fresh', 'no graph', 'unknown') or not edited(s): return True
751+
if (s['state'] in ('fresh', 'no graph', 'unknown') or not edited(s)) and not _waits_on(s, files): return True
728752
if s.get('failed'): return False
729753
if s['state'] == 'stale': kick(repo, 'a query')
730754
now = time.time()
@@ -768,7 +792,7 @@ def with_note(n): # the answer as it is, then
768792
left = expected_left(repo)
769793
print(f"waiting for the graph to refresh (--fresh): {len(edited(s))} file(s) changed since the graph was built" +
770794
(f", the last build took {int(build_seconds(repo)[0])} s" if left is not None else '') + " …", file=sys.stderr, flush=True)
771-
if wait_fresh(repo, float(os.environ.get('AXIOMCODE_FRESH_MAX') or 600), say=True): passthrough()
795+
if wait_fresh(repo, float(os.environ.get('AXIOMCODE_FRESH_MAX') or 600), say=True, files=True): passthrough()
772796
s = status(repo)
773797
stale = Stale(edited(s)); r = run()
774798
out = r.stdout.decode('utf-8', 'replace')
@@ -782,7 +806,7 @@ def with_note(n): # the answer as it is, then
782806
print(f"waiting for the graph to refresh: this answer touches {', '.join(stale.files[:3])}" + (' …' if len(stale.files) > 3 else '') +
783807
f", edited since the graph was built; up to {int(budget)} s" + (f" (the last build took {int(build_seconds(repo)[0])} s)" if left is not None else '') + " …",
784808
file=sys.stderr, flush=True)
785-
if wait_fresh(repo, budget, say=True): passthrough()
809+
if wait_fresh(repo, budget, say=True, files=stale.files): passthrough()
786810
s = status(repo); stale = Stale(edited(s))
787811
if edited(s):
788812
r = run(); out = r.stdout.decode('utf-8', 'replace')

‎plugins/axiomcode/skills/axiomcode/scripts/axiomcode-build‎

Lines changed: 29 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -242,7 +242,7 @@ LANGDIR="$REPO/.axiomcode/lang" # one graph directory per language other th
242242
# file the parser reads with its hash; when none differs the graph is current whatever git says: a dirty tree whose
243243
# edits are already indexed is not rebuilt, and a tree edited and restored, which git calls clean against a graph built
244244
# from the edit, is. A commit that changed no file moves the stamp, not the graph.
245-
if [ -f "$OUT/graph.sqlite" ] && [ -f "$OUT/stamp" ] && python3 "$H/ax_fresh.py" uptodate "$REPO" "$LANGS" "${AXIOMCODE_SRC:-}" "${STAMP#"$PREFIX"}"; then
245+
if [ -f "$OUT/graph.sqlite" ] && [ -f "$OUT/stamp" ] && [ ! -f "$OUT/partial" ] && python3 "$H/ax_fresh.py" uptodate "$REPO" "$LANGS" "${AXIOMCODE_SRC:-}" "${STAMP#"$PREFIX"}"; then
246246
if [ "$(cat "$OUT/stamp")" != "$STAMP" ]; then echo "$STAMP" > "$OUT/stamp"; INDEXED_TREE="$OLD_INDEXED"; set_base "$OLD_INDEXED"; PREV=""; keep_base_graph; commit_base; fi
247247
has_symbols "$OUT/graph.sqlite" || python3 "$H/axiomcode-index" "$REPO"
248248
[ -f "$REPO/.axiomcode/engine" ] || printf '%s\n' "$ENGINE" > "$REPO/.axiomcode/engine" || true
@@ -353,18 +353,34 @@ unpend(){ grep -v "^$1 " "$PENDING" > "$PENDING.new" 2>/dev/null || true; mv -f
353353
EPID=""
354354
stop_engine(){ if [ -n "$EPID" ] && kill -0 "$EPID" 2>/dev/null; then kill -TERM -- "-$EPID" 2>/dev/null || kill -TERM "$EPID" 2>/dev/null || true; fi; rm -f "$PROGRESS" "$PENDING" "$PENDING.new"; }
355355
trap stop_engine EXIT
356-
trap 'exit 130' INT; trap 'exit 143' TERM HUP
356+
# A BUILD STOPPED ONCE ITS MAIN LANGUAGE IS SOLVED PUBLISHES THAT GRAPH FIRST. The solved main graph is most of what the
357+
# build cost; stopped while it was being indexed (a caller's timeout ends the whole process group, the indexer included),
358+
# the build took the indexer's death for a failure, deleted the solved graph and left no graph.sqlite, so the next query
359+
# built everything again, was stopped again, and a repository in several languages rebuilt forever. A signal that
360+
# arrives after the engine names the main language solved, or while it is published, is held until graph.sqlite points
361+
# at it (and the indexer runs in a process group of its own, which a signal to the build's does not reach); the build
362+
# then stops, the other languages unpublished. One that arrives earlier stops the build at once, as it always did.
363+
SIG=""; SIGS=0; PUBLISHING=""; MAIN_DONE=""
364+
on_signal(){ SIGS=$((SIGS + 1)); SIG="$1"
365+
if [ -z "$MAIN_DONE" ] && { [ -n "$PUBLISHING" ] || grep -q "^$LANG_ARG ok" "$PROGRESS" 2>/dev/null; }; then return 0; fi
366+
exit "$1"; }
367+
trap 'on_signal 130' INT; trap 'on_signal 143' TERM HUP
368+
# run a command in a process group of its own and wait for it; a held signal interrupts `wait`, not the command
369+
shielded(){ local p rc n; set -m; "$@" & p=$!; set +m
370+
while :; do n=$SIGS; rc=0; wait "$p" || rc=$?; [ "$SIGS" != "$n" ] && [ "$rc" -gt 128 ] && kill -0 "$p" 2>/dev/null && continue; break; done
371+
return "$rc"; }
357372
# an engine older than this script has no --progress and refuses an option it does not know: it gets none, and its
358373
# graphs are published together when it ends, as they always were
359374
PROG=(); grep -q -- '--progress' "$ENGINE/bin/axiomcode" 2>/dev/null && PROG=(--progress "$PROGRESS")
360375
set -m
361376
"$ENGINE/bin/axiomcode" all "$SRC" "$OUT" --language "$SOLVE" ${PROG[@]+"${PROG[@]}"} ${LIB[@]+"${LIB[@]}"} < /dev/null > "$REPO/.axiomcode/build.log" 2>&1 &
362377
EPID=$!
363378
set +m
364-
MAIN_DONE=""; QUEUED=(); READY=(); FAILED=(); DONE=" "; SEEN=0
379+
QUEUED=(); READY=(); FAILED=(); DONE=" "; SEEN=0
365380
# THE MAIN LANGUAGE'S GRAPH BECOMES THE GRAPH. Everything a finished build did for it happens here, as soon as it is
366381
# solved: index, stamp, file table, baseline, pointer.
367382
publish_main(){
383+
PUBLISHING=1
368384
# the new database: a FILE under $OUT/$LANG_ARG. Matched as `*/$LANG_ARG/*` it was any path with the language's name
369385
# in it, the repository's own included: a repository at .../python/app matched the graph.sqlite SYMLINK, the link was
370386
# pointed at itself, the previous graph deleted, and every verb after it reported no graph.
@@ -374,7 +390,7 @@ publish_main(){
374390
# A QUERY NEVER READS A HALF-BUILT GRAPH. The new database gets its symbols, refs and views BEFORE graph.sqlite points
375391
# at it; until the pointer moves, every verb (and the hooks) keeps answering from the previous graph. Only then is the
376392
# old one removed: a reader that opened it keeps the file it opened.
377-
AXIOMCODE_INDEX_DB="$DB" python3 "$H/axiomcode-index" "$REPO" || { restore; echo "indexing the new graph failed — the graph that was there is untouched"; exit 1; }
393+
shielded env AXIOMCODE_INDEX_DB="$DB" python3 "$H/axiomcode-index" "$REPO" || { restore; echo "indexing the new graph failed — the graph that was there is untouched"; exit 1; }
378394
stamp_meta "$DB"
379395
echo "$STAMP" > "$OUT/stamp"
380396
# the engine that built this graph: the next build, the background refresh above all, starts its search there (see the top)
@@ -396,7 +412,13 @@ publish_main(){
396412
done
397413
commit_base
398414
rm -rf "$PREV"
399-
MAIN_DONE=1
415+
MAIN_DONE=1; PUBLISHING=""
416+
# the languages still unsolved are named in $OUT/partial: the graph is not up to date while it is there, so the next
417+
# build (a refresh the next query starts, or an index) builds them rather than calling the file table current
418+
if [ -n "$SIG" ]; then
419+
[ -s "$PENDING" ] && cut -d' ' -f1 "$PENDING" > "$OUT/partial"
420+
echo "$LANG_ARG graph published at $OUT/graph.sqlite; stopped before the other languages were built"; exit "$SIG"
421+
fi
400422
python3 - "$OUT/graph.sqlite" <<'PY'
401423
import sqlite3, sys
402424
rows = sqlite3.connect(sys.argv[1]).execute("SELECT tier, count(*) FROM call_edges GROUP BY tier ORDER BY 2 DESC").fetchall()
@@ -445,7 +467,7 @@ while :; do
445467
running=1; kill -0 "$EPID" 2>/dev/null || running=0
446468
drain
447469
[ "$running" = 1 ] || break
448-
sleep 1
470+
sleep 1 || true # a held signal ends the sleep, not the build
449471
done
450472
ENGINE_RC=0; wait "$EPID" || ENGINE_RC=$?; EPID=""
451473
drain
@@ -459,7 +481,7 @@ fi
459481
for l in ${OTHER_LANGS//,/ }; do
460482
case "$DONE" in *" $l "*) ;; *) publish_other "$l" ok;; esac
461483
done
462-
rm -f "$PROGRESS" "$PENDING"
484+
rm -f "$PROGRESS" "$PENDING" "$OUT/partial"
463485
if [ ${#FAILED[@]} -gt 0 ]; then
464486
for l in "${FAILED[@]}"; do
465487
if [ -f "$LANGDIR/$l/out/graph.sqlite" ]; then echo "⚠️ the $l graph did not build: $l questions are answered from its PREVIOUS graph, which predates this tree; see $REPO/.axiomcode/build.log"

‎tests/README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,8 @@ One check needs no graph and is its own script:
6161
unelevated Windows user (WinError 1314); indexes a case, so it needs the engine
6262
python3 tests/multi_language.py a repository in several languages is indexed in all of them and every query asks
6363
each graph: nothing dropped, a one-graph answer unchanged, an edit reported once,
64-
refreshed, upgraded from a one-language graph (builds real graphs, needs the engine)
64+
refreshed, upgraded from a one-language graph; a first query or an index stopped
65+
mid-build still leaves the main graph published (builds real graphs, needs the engine)
6566
python3 tests/latency.py what a query repeats on every call is done once, with the same answer: the non-source
6667
scan cached per graph, a repeated impact solve read back, one tree walk for every language,
6768
the verb and the Python found without starting programs, a first build that reports its

0 commit comments

Comments
 (0)