Skip to content

[POC] Add scriptc-pm (spm): augment pnpm installs with verified upstream TypeScript (POC) - #1288

Draft
cramforce wants to merge 8 commits into
mainfrom
poc/spm
Draft

cramforce wants to merge 8 commits into
mainfrom
poc/spm

Conversation

@cramforce

Copy link
Copy Markdown
Contributor

Draft / proof of concept. Part of the scriptc-pm exploration (TypeScript-source overlays for npm packages). Not intended for merge as-is; see the usefulness findings below.

This adds packages/pm (@scriptc/pm, bins scriptc-pm and spm), a proof of concept of the scriptc-pm design.

spm wraps pnpm. After pnpm install it finds the upstream TypeScript for each installed package instance and checks that it corresponds to the published artifact. It writes per-instance source overlays plus .scriptc-pm/manifest.json (schema 1) for scriptc --sources. It has no npm dependencies, and it compiles to a native binary with scriptc that behaves identically to the Node build.

This branch also merges poc/spm-compiler, which adds --sources, for the end-to-end evaluation.

What it does

  • spm install [pnpm args]. Runs pnpm install, then spm sync.
  • spm sync. For every instance in node_modules/.pnpm:
    • Pre-filter. Skips types-only packages, native addons, and packages with no TS signal.
    • Locate the source, in this order: an override from scriptc-pm.json; complete shipped .ts or sourcesContent; npm provenance (the subject digest is checked, the signature is not); the registry gitHead plus repository; a version tag.
    • Fetch the exact commit. Either a blobless sparse git fetch, or the codeload archive with an in-process gunzip and tar reader.
    • Gate on content correspondence:
      • exact: byte equality with shipped sourcesContent or shipped .ts;
      • equivalent: re-emit with the project's TypeScript and a token-normalized per-module comparison;
      • structural: for bundles without maps;
      • plus an exported-name surface check and a closure check for codegen'd modules.
    • Write outputs. Overlays go to .scriptc-pm/overlay/<instance>/pkg, with a node_modules link to pnpm's resolved deps. Also writes scriptc-sources.lock.json and the manifest.
  • spm status [--json]. Augmented instances and fallbacks with reason codes.

Results on the public demo (examples/spm-demo)

Augmentation. 40 instances, 20 source candidates, 12 augmented: 3 exact, 9 equivalent, all with an identical surface. The 8 fallbacks carry reasons: 6 upstream-js, 1 codegen missing-module, and ts-api-utils at structural (a tsup bundle without maps). Every design MVP target reached the expected identity and content level, except ts-api-utils.

Cost. A cold spm sync takes about 5 s for this closure; a warm one takes 0.24 s natively. On a warm cache, spm install adds about 0.25 s over plain pnpm.

Native parity. scriptc coverage reports 100% static (1713 statements) and the binary is 2.7 MB. packages/pm/scripts/smoke.sh checks byte-identical stdout, manifest, lock and overlays between the Node and native lanes.

End to end. With scriptc --sources, 1 of 12 overlays compiles from source (@typescript-eslint/visitor-keys). The other 11 fall back to published JS:

  • 5 because of generic scriptc gaps that also block user code: unknown-typed class fields, the Error.cause redeclaration, array-typed destructuring and string-indexing shapes, export = under ESNext, and Temporal;
  • 2 because of class-field semantics (useDefineForClassFields: false packages);
  • 3 because of the overlay type environment: the package's ambient .d.ts is not loaded, and untyped devDependency imports fail;
  • 1 because of monorepo version skew.

No demo probe newly builds or newly matches Node, so spm is not useful end to end yet. The detailed table, findings, and what would change the verdict are in NOTES.md.

Contract notes

  • Manifest schema 1 is followed exactly; no schema change.
  • The overlay package.json type is module whenever the source uses ES module syntax. Copying the source type (DESIGN.md §5.3.4) made every module: commonjs package unusable with SC1013.
  • An additive per-instance ambientDeclarations field is proposed as a follow-up.

Not in scope (POC)

  • frozen and offline modes, why, doctor, fetch;
  • npmrc and private registries;
  • Sigstore verification;
  • npm tsconfig presets;
  • asset and codegen backfill;
  • a broad test suite.

All are listed in NOTES.md.

Validation

  • scripts/smoke.sh: real pnpm install, then Node vs native comparison, with cold caches.
  • oxlint --deny-warnings packages/pm and oxfmt --check on the package.
  • pnpm -r build.

🤖 Generated with Claude Code

cramforce and others added 7 commits October 17, 2026 19:33
spm runs pnpm, enumerates installed package instances from the isolated
node_modules/.pnpm layout, locates each instance's upstream TypeScript
(local override, tarball source or sourcesContent, npm provenance,
registry gitHead, version tag), fetches the package directory at that
commit (blobless sparse git, or the codeload archive with an in-process
gzip + tar reader), verifies it against the published artifact
(byte-exact sourcesContent, token-normalized re-emit equivalence, a
structural check for bundles, and an exported-name surface check), and
writes per-instance overlays plus .scriptc-pm/manifest.json (schema 1)
and scriptc-sources.lock.json for scriptc --sources.

It has no npm dependencies and compiles to a native binary with scriptc;
scripts/smoke.sh checks that the Node and native lanes produce identical
output on the public demo app in examples/spm-demo.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rces)

`scriptc build|run|coverage --sources[=<manifest>]` reads a scriptc-pm
schema-1 manifest (default: the nearest .scriptc-pm/manifest.json above
the entry) and compiles each augmented package instance from its
TypeScript overlay instead of its published JS.

- Instances are keyed by realpath(installedDir). The resolver redirects
  an import that lands in an instance's installed directory to the
  instance's exports inside its overlay (resolveSourceOverlayModule);
  overlay files compile as program modules.
- tsgo gets the pilot's "paths" injection, but only for names with a
  single augmented instance; other instances start dropped, and a second
  installed copy of a redirected name drops it too.
- Fallback per instance through the drop-and-reload loop: a preflight
  diagnostic in an overlay, a program type error naming it, a top-level
  require() edge to it, a lowering refusal inside it, or a program that
  only fails against the source surface (probe) drops the overlay and
  the package compiles from published JS. Reasons are reported on stderr
  and as "source"/"source-fallback" rows in coverage.
- Build caches key by the manifest's lockDigest and bytes instead of
  being bypassed; the network-fetching --provenance-sources pilot stays
  uncacheable and becomes one more producer of the same overlay set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Warm syncs reuse the content check result, keyed by the integrity-pinned
package and the commit-pinned tree. When a package's tsconfig cannot be
read, candidate targets are bounded by the syntax and downlevel helpers
in the published JS of the matching module format, and the target search
stops early when the published entry has no counterpart. Adds the demo's
measure.sh, which compiles every probe with and without extra scriptc
flags and records coverage, build result and Node parity.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…yntax

A module:commonjs package has no package.json type, so its overlay was
classified CommonJS and scriptc rejected the TypeScript's import/export
syntax (SC1013). Overlays now get type module unless the source uses
CommonJS-only TypeScript forms (export =, import x = require()); the
published format stays in the manifest as publishedFormat. Policy
fallback reasons now start with a reason code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@vercel

vercel Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
scriptc Ready Ready Preview, v0 Oct 18, 2026 4:28pm UTC

@socket-security

socket-security Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

@vercel-security-reviewer

Copy link
Copy Markdown

Security review details

Writes the additive schema-1 ambientDeclarations list that the compiler
side (stacked PR on poc/spm-compiler) loads for active overlays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active deployment
Preview — 54f8ca7f Deployed Oct 18, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant