serving: flip the blessed release to Qwen3.8-27B (multi-release plumbing, model-directory pin, thinking flag) - #1758
Merged
Merged
Conversation
Two releases in the loadout used to deadlock the second: baseline prompts
only ever named the primary, a miner on the other release refused them,
the refusals were misses in the primary's window, and after three rounds
the miner was dormant, never READY, never routed.
The validator now learns which release each miner serves from the miner
itself. A refusal of a baseline prompt names the release the miner does
run ("serving release X, not Y"); if X is in the loadout the round records
it and prompts X from then on, never a miss. A completion served for a
release declares it the same way. The mapping is persisted (miner_release
table) so a restart seeds probation on the right release.
audit_round settles each release over its own miners only: windows,
probation and the round report carry the miner's release instead of
whichever release ran last, and the round log prices tokens per release.
The gateway dispatches with the routed release's request_timeout.
Env: SERVING_REFERENCE_URL (and api key / attest sidecar) keep naming the
primary's reference; every other release's is SERVING_REFERENCE_URL__<ID>
with the release id upper-cased, so a validator hosts one reference per
release without committing endpoints. Miner-side SERVING_BASE_URL applies
to the release SERVING_RELEASE names.
Claude-Session: https://claude.ai/code/session_01VjhStJbNW6WzxucTcwuw4y
…ry too Qwen3.8-27B's blessed artifact is a compressed-tensors NVFP4 directory, not one GGUF. The image's entrypoint now takes MODEL_DIR_REPO + MODEL_DIR_REVISION (the HF commit: the repo is mutable, the revision is the pin) + MODEL_DIR_SHA256 (file=sha256 per weight shard), fetches the directory at that revision, refuses to start on any digest mismatch, and serves it (sparkinfer_server dispatches on -m <dir>). Without MODEL_DIR_* it is run.sh --download as before, so the 35B release is unchanged. scripts/serving_conformance_on_lium.sh measures a non-primary artifact with CONFORMANCE_MODEL_ID + CONFORMANCE_MODEL_ENV, for qualifying a candidate second release before it has a loadout entry. Claude-Session: https://claude.ai/code/session_01VjhStJbNW6WzxucTcwuw4y
…es both Qwen3.8-27B thinks by default and ships as a Hugging Face directory, so a release now carries `enable_thinking` (sent as-is by the miner's backend, the validator's live reference on both /v1/chat/completions and /v1/score, and the checker's --thinking flag, so served text and its teacher-forced score template identically) and a `model_dir` block (repo, pinned revision, per-shard sha256s) beside the GGUF fields. /v1/models surfaces model_dir for the release card. docker-compose.miner.yml and the validator's reference profile pass either artifact's pins plus TOK_REPO/MODEL_NAME/CTX through to docker/sparkinfer-entrypoint.sh; the env examples and README say which to copy. The Lium conformance script takes CONFORMANCE_CHECKER_ARGS and CONFORMANCE_SKIP_NODES. Claude-Session: https://claude.ai/code/session_01VjhStJbNW6WzxucTcwuw4y
anderdc
changed the title
…ing on every R8 request requests' Response is falsy for any 4xx/5xx, so the R8 failure line printed 'HTTP ?' instead of the status. Show status + body, and send the same enable_thinking on the R8 greedy generations and the verify_served reference as on the score call. Claude-Session: https://claude.ai/code/session_01VjhStJbNW6WzxucTcwuw4y
…e release Replaces Qwen3.6-35B-A3B in serving_loadout.json with qwen3.8-27b-nvfp4-sparkinfer-19ef39ec2: the gittensor-model-hub NVFP4 directory pinned at HF commit 8e2c0cd2 with both shard digests, image entrius/sparkinfer:19ef39ec2 by digest, thinking off, 64k prompts on a 131k KV pool, end-of-turn 248046. Measured on rented RTX 5090s (2026-09-08, scripts/check_serving_runtime.py, deterministic mode, logprobs on every request): every MUST passes, D1 1.000 / 0.0000, /v1/score exact; single-stream 99.4 tok/s, 16-concurrent aggregate ~338, per-request curve 1-16 in the entry; prefill 7.5-9.7k tok/s cold (carried as 7,500); 25 GiB resident -> attest reserves 26.4 GB. Full readout in the team run-log. Checker: verify_served now forces the runtime's own token ids (as the validator does) instead of re-tokenizing text; greedy() returns reference_token_ids. A runtime_env block on the release carries the container env (CTX, MODEL_NAME, TOK_REPO) for the release card. Tests repointed at the 27B curve; note that a dense 27B at 2 streams (92 vs 99 tok/s) no longer exposes a shared card by decode speed the way the MoE did - the same-instant attest fill carries that. Claude-Session: https://claude.ai/code/session_01VjhStJbNW6WzxucTcwuw4y
Pay is derived from the release's own measured speeds, so the flip already re-priced itself: $0.70 per card-hour is now ~$0.58 per million output tokens and ~$0.026 per million prompt tokens (one card flat out is ~1.2M output tokens or ~27M prompt tokens an hour). The constants only back a release without measurements, and they and their comments now describe the 27B: aggregate 338 tok/s, prefill 7,500, per-request curve 99.4/49.7/23.4 at 1/6/16, 26.4 GB resident, 64k context. The prompt-size DoS backstop doubles to 0.5 MB so a full 64k prompt fits. serving_rounds gains model_dir_sha256 and runtime_env (JSON) so the Current release card can show what a miner copies for a model-directory release (gittensor-db adds the columns). Claude-Session: https://claude.ai/code/session_01VjhStJbNW6WzxucTcwuw4y
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Flips the serving release from Qwen3.6-35B-A3B to Qwen3.8-27B (NVFP4, sparkinfer
19ef39ec2) as the single loadout entry, with everything the validator, miner image and compose files need for it, and re-prices the fallbacks so a card still earns $0.70 per hour. Decision 2026-09-07: flip outright; two models later.Commits, in order:
SERVING_REFERENCE_URL__<RELEASE_ID>; gateway timeout per release. Not needed for the flip, but it makes a second release later a loadout entry plus a reference.docker/sparkinfer-entrypoint.shserves either one GGUF (unchanged) or an HF model directory pinned by commit + per-shard sha256 (MODEL_DIR_*), refusing to start on any mismatch.enable_thinking(sent by the miner backend, the live reference on chat and/v1/score, and the checker),model_dir,runtime_env; both compose files pass either artifact's pins through; env examples and README updated.verify_servedforces the runtime's own token ids like production.qwen3.8-27b-nvfp4-sparkinfer-19ef39ec2:gittensor-model-hub/Qwen3.8-27B-NVFP4-RTX5090@8e2c0cd2(both shard digests),entrius/sparkinfer:19ef39ec2@sha256:d35719b0…, thinking off,context_tokens65536 on a 131k KV pool, end-of-turn 248046.serving_roundsgainsmodel_dir_sha256+runtime_envfor the release card.Measured (rented RTX 5090s, 2026-09-08,
scripts/check_serving_runtime.py, deterministic mode, logprobs on every request)/v1/scoreexact (max |Δ| 0), 429-not-queue at 16, attest 832 ms.ttft_ms: 7.6k tok/s at 10k tokens, 9.7k at 41k, 7.8k at 102k (carried as 7,500).vram_model_reserved_bytes26.4e9.Full readout in the team run-log (2026-09-08 entry).
Companion PRs (merge order)
Tests
805 passed; pre-commit + pre-push (ruff, format, pyright, vulture) clean.
Before mainnet
Testnet soak with real cards on the new image; miner announcement (flag day: new image, new artifact); validator reference switched to the 27B (
SPARKINFER_MODEL_DIR_*,SPARKINFER_CTXin.env).https://claude.ai/code/session_01VjhStJbNW6WzxucTcwuw4y