[engram-parser] GH#10 — safetensors: coordinate ecosystem / sibling parser extraction from corinth-canal (separate from GGUF #7; one-way copy, no dep on corinth-canal)

[rmems]

⚠ SUPERSEDED PLAN — read this first (2026-08-15)

Everything below describing a dedicated safetensors-parser crate and an "engram-parser charter remains GGUF-only" is reversed. Safetensors is ported into engram-parser behind an off-by-default safetensors cargo feature. No new repo. The body is preserved unedited below for history; the current decision and its rationale are in the comment thread and in engram-parser's README.md § *Origin / modularization (*#10).

Still true and unchanged: one-way copy from inspiration; no dep on corinth-canal and no corinth dep on engram-parser; corinth keeps an unmodified reference copy; config/map are not ported.

Also stale below: "Limen-Neural org" (both repos moved to rmems), "toward frozen" (module stays reference permanently), and "LIM-88" (archived/Duplicate — this issue is the live tracker).

The acceptance criteria below have been edited in place — ACs 5, 6, 9 and 10 needed their text changed, not just their boxes ticked. See the comment thread.

Source

Synced / linked to GitHub for 1:1 mirror (2026-07-26).

• GitHub: [engram-parser] GH#10 — safetensors: coordinate ecosystem / sibling parser extraction from corinth-canal (separate from GGUF #7; one-way copy, no dep on corinth-canal) #10 — OPEN
• Repo: Limen-Neural/engram-parser GH#10
• Match confidence: 1.15

---

Summary

Track the separate extraction (one-way copy of implementation as inspiration) of the reusable Safetensors header inspection, deterministic manifest generation, and MoE router/expert candidate discovery logic from the experimental reference implementation in rmems/corinth-canal (parallel to the GGUF work in this repo's #7).

This is the separate dedicated issue in engram-parser (per user request) for the Safetensors case. Companion/parallel to GGUF #7 / corinth#115. See corinth-canal#116 (the supporting/bootstrap issue in the source experimental repo) and Limen-Neural/cortex-tensor#9 (coord).

Important clarification (per modularization intent and prior reviews): This is one-way extraction/copying. We are not making corinth-canal depend on any new or target crate. corinth-canal remains self-contained as the experimental end-to-end reference (per its own PROMOTION_RULES.md: after "frozen", it keeps an unmodified reference copy of the logic). "we are not making corinth-canal dep. We are simply copying and implementing code from inspiration." "corinth-canal repo more of my experimental repo. Limen-Neural organization repos are more about modular reusable libraries. Ideally I am trying to extract useful code from corinth-canal repo."

Context

• The Safetensors support in corinth-canal (src/moe/safetensors.rs + safetensors/discovery.rs) is a self-contained pure-Rust implementation (header parse without the official safetensors crate for the header/inspect path, sharded/HF index/dir support, inspect_safetensors_checkpoint / write_safetensors_manifest, SafetensorsManifest with tensor records (name/dtype/shape/byte_size/offsets/shard) + metadata + checkpoint source, MoE candidate labeling via name/shape heuristics for routers/experts/groups (classify_tensor, discover_candidates, SafetensorsRouterCandidate, SafetensorsExpertGroup, layout family detection e.g. "generic_moe", "deepseek_v3_family", "phimoe" etc.), plus MappedSafetensorsCheckpoint for loading tensor bytes on demand).
• It is used for local model onboarding, examples/safetensors_manifest.rs, verification checklist, and dual-backend (with GGUF) in the Router/Checkpoint for experiments.
• Currently marked "reference" (implied under moe/checkpoint in MODULE_STATUS.md targeting rmems-moe).
• No Limen-Neural crate yet owns the reusable Safetensors parser + MoE discovery surface (unlike GGUF → engram-parser). Enram-parser remains GGUF-only per its charter (this issue is for ecosystem/sibling coordination of the parallel extraction).
• Phase 1 (GGUF) issues already created: engram#7 (primary in this repo), corinth#115. corinth#115 non-goals noted "Do not move Safetensors handling (separate module) unless trivial" — this work (tracked via the separate corinth#116 + this engram issue) makes it non-trivial.
• Parallel extractions (cortex-tensor from corinth) and the overall LIM-9 boundary matrix favor focused reusable libs under Limen-Neural. corinth-canal#116 exists as the source-side supporting issue (with note that primary work for a new crate will follow); this is the separate tracking issue here in engram-parser (the GGUF home) for the Safetensors extraction story.
• Supporting corinth-canal#116 and cortex#9 exist; this creates the requested separate issue in engram-parser.

Related

• rmems/corinth-canal#116 (supporting extraction issue in source repo; cross-link + update its comment)
• Phase 1 GGUF siblings: Limen-Neural/engram-parser#7 (this repo's GGUF primary; this issue is the Safetensors separate parallel), rmems/corinth-canal#115 (migration side), moe/gguf: coordinate with engram-parser extraction from corinth-canal; avoid overlapping parser work cortex-tensor#8 (GGUF coord) + #9 (Safetensors coord), Linear LIM-88 (extend), beads
• corinth docs/PROMOTION_RULES.md (frozen = reference copy kept unmodified in corinth), MODULE_STATUS.md (update safetensors layer), MODEL_SOURCE_VERIFICATION_CHECKLIST.md, configs/safetensors, examples/safetensors_manifest.rs
• LIM-9 (org boundary matrix)
• engram-parser README / scope (add ecosystem note for sibling parser extraction from corinth-canal)

Goal

Ensure the Safetensors extraction from corinth-canal (as inspiration, one-way copy, no dep) is tracked with a dedicated separate GitHub issue in this repo (engram-parser), parallel to #7. This provides visibility in the parser-related issues here, cross-links the full story (corinth source #116, cortex coord #9, future reusable crate work, LIM-9), and follows the precedent of explicit GitHub issues for such modularization.

(The actual reusable crate implementation can be a new Limen-Neural/safetensors-parser or placed per future decision; this issue tracks the extraction effort and ecosystem from engram's perspective.)

Non-goals

• Do not add a runtime or build dependency from corinth-canal's Cargo.toml or code paths.
• Do not change corinth-canal's Router/CheckpointBackend/Safetensors experiment paths, outputs, or examples (they continue using the local reference copy).
• Do not move corinth-specific extensions.
• This issue does not change engram-parser's GGUF-only implementation surface (engram charter preserved; this is coordination/ecosystem).
• No full matmul/dequant/CUDA (pure parser/extractor like the GGUF side).
• Do not create new repo in this plan (per user revision).

Acceptance criteria

• This separate GitHub issue created in Limen-Neural/engram-parser with full cross-links, "no dep on corinth-canal" language, and details.
• Core logic extraction tracked: the inspect, manifest, discovery/classify from corinth src/moe/safetensors* + discovery.rs (header-only, single/index/dir support, SafetensorsManifest, MoE router/expert groups/candidates, layout families).
• Cross-links and comments added/updated on corinth#116, cortex#9, this repo's #7.
• engram README updated with ecosystem/sibling note for the Safetensors parser extraction from corinth-canal (parallel to GGUF/engram).
• corinth-canal docs updated: docs/MODULE_STATUS.md + manifests/proven_components.toml retargeted to rmems/engram-parser (feature safetensors), status stays reference (toward frozen — a one-way copy is not a source-of-truth handoff), plus a docs/PROMOTION_RULES.md "One-way extractions" section; explicit "no dep" language on rmems/corinth-canal#115 / rmems/corinth-canal#116.
• Linear [engram-parser] GH#10 — safetensors: coordinate ecosystem / sibling parser extraction from corinth-canal (separate from GGUF #7; one-way copy, no dep on corinth-canal) #10 (live mirror) updated with the reversal. LIM-88 is archived/Duplicate — not the tracker; [engram-parser] GH#10 — safetensors: coordinate ecosystem / sibling parser extraction from corinth-canal (separate from GGUF #7; one-way copy, no dep on corinth-canal) #10 is its live successor.
• Beads created/updated in corinth-canal for the Safetensors (separate).
• Coordination acknowledged (e.g. comment in cortex #7 or #9).
• All places carry the clarifications: one-way copy from inspiration; rmems org for reusable libs (Limen-Neural — both repos were transferred); no dep on corinth-canal in either direction; engram-parser owns safetensors behind an off-by-default cargo feature (engram stays GGUF-only — reversed).
• Plan.md updated with the new issue number. No committable target: there is no root plan.md, and docs/superpowers/ is gitignored by design (.gitignore:15, per REVIEW.md §7). The committed record is README.md § *Origin / modularization (*[engram-parser] GH#10 — safetensors: coordinate ecosystem / sibling parser extraction from corinth-canal (separate from GGUF #7; one-way copy, no dep on corinth-canal) #10) + CHANGELOG.md.

Validation

• Manual review of new issue, all cross-comments, README, docs updates.
• Links render; language matches prior reviews ("no dep...", "copying and implementing code from inspiration").
• (Later, when port happens) New reusable surface produces equivalent manifests/candidates to corinth on test cases from corinth configs.

Suggested branch

docs/safetensors-sibling-extraction (or feat/safetensors-ecosystem if any engram updates).

References

• Primary plan: this engram-parser session plan.md
• corinth-canal#116 (bootstrap/supporting in source; the "extract ... from corinth-canal")
• Phase 1: engram-parser#7, corinth#115, cortex#8/#9, Linear LIM-88 + beads
• corinth src/moe/safetensors.rs + discovery.rs + examples + configs + docs/PROMOTION_RULES.md + MODULE_STATUS.md
• LIM-9

Created as the separate GitHub issue in engram-parser for the Safetensors extraction from corinth-canal (parallel to #7; per user request for the issue here, one-way copy, no dep on corinth-canal, following modularization pattern from LIM-9 and GGUF precedent).

---

Siblings (cross-linked/updated together):

• rmems/corinth-canal#116 (supporting; already existed, link + comment updated)
• Limen-Neural/cortex-tensor#9 (coord; link + comment updated)
• engram-parser#7 (GGUF primary in this repo; comment updated)
• Linear LIM-88, beads in corinth