Skip to content

Add live-status: command → one-sentence status model, pipeline and service - #17

Merged
Noisemaker111 merged 4 commits into
agentsfrom
feature/live-status-20260916
Sep 17, 2026
Merged

Noisemaker111 merged 4 commits into
agentsfrom
feature/live-status-20260916

Conversation

@Noisemaker111

Copy link
Copy Markdown
Owner

Adds live-status/, a local pipeline and service that turn shell/tool commands into one live status sentence.

Data. Mines Claude Code, Codex, OpenCode, OpenCode2, Cursor, Grok and PSReadLine transcripts (65,767 executions → 58,109 template groups), redacts secrets before any model call, and labels with Opus 5 teacher and judge passes. v1 splits: 3,113 train / 425 validation / 494 test. Families are leakage-controlled, and test is enriched with secret, injection and synthetic adversarial cases.

Grading. TypeSafe jev (through Vercel AI Gateway) grades outputs against the gold status. It agrees with Opus evaluations at AUC 0.92 (700 pairs) and takes seconds instead of hours.

Models (jev-accepted share of the 494 test statuses):

  • best untuned baseline: 23.5%
  • SmolLM2-135M, full fine-tune: 35.4%
  • SmolLM2-360M, full fine-tune: 52.8%
  • Qwen2.5-0.5B, LoRA: 58.3%
  • Qwen3-0.6B, LoRA: 69.4% (q4_K_M 67.8%, 397 MB, 100 ms GPU / 393 ms CPU p50)

Service. POST /v1/summarize-command: redaction, cache, model call, validation, then heuristic fallback. Auth/TLS/rate limits for remote use; commands are never logged.

Verification.

  • Drove the running service over HTTP and through api/client.py with real commands, a fake credential and an injection string. The credential became "with stored credentials", the injection was described as data, repeat requests hit the cache, and the log held only hashes and statuses.
  • New core tests (live-status/tests) are added to CI.
  • Private corpora, labels and weights stay in the ignored work/live-status.

Limits.

  • Failure-mining round 1 is partial (Opus cooldown after 104 of 723 judgements).
  • jev is stricter than Opus on good outputs, so compare runs only within one grader.
  • Two q8_0 runs echoed a redaction placeholder once each and were held back by the promotion gate.

🤖 Generated with Claude Code

Noisemaker111 and others added 4 commits September 16, 2026 21:35
…ing, training, service)

Mines local agent transcripts into deduplicated command groups, redacts
secrets before any teacher/judge call, labels with Opus 5 teacher and judge
passes, builds leakage-controlled splits, and adds SFT/DPO training, GGUF
export, evaluation and a local status service with heuristic fallback.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…raining memory fixes

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… results

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Noisemaker111
Noisemaker111 merged commit e016016 into agents Sep 17, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant