Skip to content

chore(aeval): bump to v0.3.1 (binary + aeval-data pin) - #125

Merged
guohai merged 1 commit into
mainfrom
chore/aeval-0.3.1
Aug 26, 2026
Merged

guohai merged 1 commit into
mainfrom
chore/aeval-0.3.1

Conversation

@guohai

@guohai guohai commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

Bumps aeval to v0.3.1, both halves per the contract documented in .github/workflows/docker.yml — AEVAL_VERSION selects the binary, the aeval-data submodule pins the corpus/config baked into the image:

  • vox_eval_agentd/Dockerfile: ARG AEVAL_VERSION=v0.3.0 → v0.3.1
  • vox_eval_agentd/aeval-data: 4ffb306 (v0.3.0) → cfc2b28 (v0.3.1)

Release content: updated elevenlabs platform config (new agent_url) plus refreshed examples/, corpus/, and release notes.

Verification

The image build only runs post-merge on main (no PR gate), so this was verified with a real local build — repo-root context, -f vox_eval_agentd/Dockerfile --target base (the stage that downloads aeval):

  • Build succeeds.
  • The baked binary's checksum matches the published v0.3.1 asset exactly: sha256 57c70374…05b711, 441515480 bytes.
  • npm run check clean; tests/aeval-seed.test.ts 27/27.

⚠️ Upstream bug found (not a blocker)

That verified-genuine v0.3.1 binary self-reports aeval 0.3.0 from --version. Traced the blast radius:

  • Daemons parse --version for their frameworkVersion metadata → they'll report v0.3.0. That's unchanged from today, so no regression.
  • The risk would be aeval-seed, which stamps the release-notes version (now v0.3.1) onto built-in eval sets, while the job version gate (routes.ts) hides any job whose frameworkVersion exceeds the agent's — v0.3.1-stamped jobs would be invisible to the whole fleet. However, both seed triggers (seedFromLocalAevalData, seedAevalVersion) currently have no live call site (routes.ts imports only compareVersions), so nothing creates such jobs.

Net: cosmetic/observability only today — but worth fixing upstream before that seeder is ever wired up, and worth knowing that operators' agent listings will read v0.3.0 while the image actually contains v0.3.1.

Generated with SMT smt@agora.build

Bumps both halves per the CI contract: AEVAL_VERSION selects the binary,
the aeval-data submodule pins the corpus/config baked into the image.

v0.3.1 carries an updated elevenlabs platform config (new agent_url) plus
refreshed examples/corpus.

Verified with a real local image build (repo-root context, --target base,
the stage that downloads aeval): build succeeds and the baked binary's
sha256 + size match the published v0.3.1 checksum exactly
(57c70374…b711 / 441515480).

UPSTREAM BUG FOUND: that verified-genuine v0.3.1 binary self-reports
"aeval 0.3.0" from `--version`. Consequences are cosmetic today — daemons
report frameworkVersion v0.3.0 (unchanged from before, so no regression),
and the aeval-seed auto-seeder that would stamp the release-notes version
(v0.3.1) onto built-in eval sets has no live call site, so no job can be
created that the version gate would hide from the fleet. Worth fixing
upstream before that seeder is ever wired up.

🤖 Built with SMT <smt@agora.build>
@github-actions

Copy link
Copy Markdown

🤖 Codex Code Review

No findings in the two-line PR diff.

Residual risk: I couldn’t inspect the aeval-data submodule contents because the target commits aren’t present in this checkout (4ffb306...cfc2b28, “commits not present”), and network is unavailable. The Dockerfile version bump itself is consistent with the documented binary/data split, and I don’t see a security, logic, performance, or maintainability regression in the visible changes.

@github-actions

Copy link
Copy Markdown

🤖 Claude Code Review

Review

The change itself is minimal and mechanically correct: AEVAL_VERSION is declared in the base stage (vox_eval_agentd/Dockerfile:68), so both the broker and daemon targets pick up v0.3.1 consistently. The CI path filter vox_eval_agentd/** (.github/workflows/docker.yml:69) matches both changed files, so the agentd/broker images will actually rebuild, and build-vox-eval-agentd checks out submodules recursively — the new pin gets baked in.

No security concerns. Nothing here touches credentials, secrets, or request handling.

One thing worth confirming before merge: the two version sources must agree

There are two independent version knobs, and this PR moves both — but nothing enforces that they land on the same version:

  1. AEVAL_VERSION=v0.3.1 → binary → aeval --version → the agent's reported frameworkVersion (vox_eval_agentd/vox-agentd.ts:266,306)
  2. The submodule's release/LATEST_RELEASE_NOTES.md → parsed by seedFromLocalAevalData() (server/aeval-seed.ts:284-292) → built-in eval sets seeded with config.frameworkVersion: <version>

If commit cfc2b28 declares a version newer than v0.3.1 in its release notes, the version gate at server/routes.ts:3358-3362 (compareVersions(jobVersion, agentVersion) > 0 → drop) will silently withhold every seeded built-in job from every v0.3.1 agent. The failure mode is bad: jobs sit pending indefinitely with no error surfaced anywhere. Since .gitmodules points aeval-data at the same Agora-Build/aeval.git repo as the binary, the check is simply that cfc2b28 is the commit tagged v0.3.1.

I could not verify this locally — the submodule is not initialized in this checkout, and remote lookups weren't available to me.

Two things that are fine, for the record:

  • The gate direction is safe for existing content: it only filters jobs requiring a newer version than the agent, so upgraded v0.3.1 agents still accept workflows/eval sets pinned to v0.3.0. No dispatch regression.
  • A missing/renamed release asset fails loudly — curl -fSL (Dockerfile:75) will break the build rather than ship a broken image.

Pre-existing, not introduced here

  • build-vox-service doesn't check out submodules (.github/workflows/docker.yml:89), while the Core Dockerfile does COPY . .. So the Core image ships an empty vox_eval_agentd/aeval-data/ → discoverScenarios() returns [] → both seeding triggers bail (aeval-seed.ts:143-146, :279-282). The lock is correctly released, so nothing gets stuck — but it does mean the aeval-data pin bump has no effect on Core-side built-in seeding in production. Worth a separate look if seeding is expected to work there.
  • No checksum on the downloaded binary. AEVAL_VERSION selects a mutable GitHub release asset with no sha256sum -c verification. Same shape as v0.3.0, so not a regression, but a sha256 pin would make the build reproducible and tamper-evident.
  • designs/2026-08-21-aeval-base-broker-split-{design,plan}.md still reference v0.3.0. These read as point-in-time design records, so leaving them is defensible.

@guohai
guohai merged commit f762f80 into main Aug 26, 2026
7 checks passed
@guohai
guohai deleted the chore/aeval-0.3.1 branch August 26, 2026 09:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant