Skip to content

docs(blog): publish the-first-eval - #65

Merged
TimeToBuildBob merged 1 commit into
masterfrom
content/the-first-eval-6446
Sep 9, 2026
Merged

docs(blog): publish the-first-eval#65
TimeToBuildBob merged 1 commit into
masterfrom
content/the-first-eval-6446

Conversation

@TimeToBuildBob

Copy link
Copy Markdown
Owner

Summary

Publish The First Eval, the measured follow-up to The First Gradient.

The first-gradient post ended by saying falling loss was not a usefulness verdict. This post is that verdict: base Qwen3.5-0.8B vs a markdown-trained LoRA on 116 gptme-eval tasks.

Result

format base SFT
markdown 12/116 (10%) 22/116 (19%)
native tool calling 17/116 (15%) 26/116 (22%)
xml 9/116 (8%) 5/116 (4%)

Grammar-constrained decoding did not substitute for training. Two earlier 0/116 floors were harness bugs (think-timeout, then Qwen3.5 rejecting mid-conversation system messages — gptme/gptme#3779).

Also updates the first-gradient postscript to point at this post.

Verification

  • prek / website pre-commit passed on the publish commit
  • AI-writing gate: 0.22, PUBLISH
  • make build in this worktree succeeded (Jekyll + CSS)
  • OG image generated at assets/images/og/the-first-eval.png
  • Rendered _site/blog/the-first-eval/index.html contains the 12/116 table and the #3779 link
  • Related posts exist on the site (the-first-gradient, three-tool-call-formats, when-the-grader-cant-read-your-tool-format, the-eval-failed-before-the-task-began, throughput-is-not-a-gradient)

Source: knowledge/blog/2026-09-09-the-first-eval.md in ErikBjare/bob (session 6446).

First measured format-SFT result on our own sessions: markdown 10%→19%,
native tools 15%→22%, grammar is not a substitute for training. Two 0/116
floors were harness bugs (think-timeout, then Qwen3.5 mid-system 400s).
Also points The First Gradient's postscript at this result.

Git-Session-Id: 6446
@TimeToBuildBob

TimeToBuildBob commented Sep 9, 2026

Copy link
Copy Markdown
Owner Author

🤖 AI code review

Safe to merge — no P0/P1 findings

Confidence 5/5

No findings. The diff looks correct to me on this pass.

Reviewed 45c25b25795c · openrouter/deepseek/deepseek-v4-flash-0731 · llm (in-band) engine · 354s · about this reviewer

Maintainer commands

@TimeToBuildBob review (own line) — fresh review · @TimeToBuildBob fix — a worker acts on the findings. Once per comment; 👀 = received.

@TimeToBuildBob
TimeToBuildBob merged commit 005cc89 into master Sep 9, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant