Skip to content

Keep a lambda's resources once, apart from its versions - #118

Closed
MDA2AV wants to merge 1 commit into
mainfrom
fix/resources-kept-apart
Closed

MDA2AV wants to merge 1 commit into
mainfrom
fix/resources-kept-apart

Conversation

@MDA2AV

@MDA2AV MDA2AV commented Oct 13, 2026

Copy link
Copy Markdown
Owner

Saving or deploying a version with a lot of resources took the shared server hundreds of megabytes. Every resource was stored inside the version's JSON envelope, text as strings and everything else as base64. Each save and deploy then built and parsed the whole version as one string.

With this PR, a version names each resource by the SHA-256 of what it holds. The lambda's resource store keeps each one once, below {data}/code/{id}/resources/, for every version, feature and preview that ships it.

Measured

The same site was saved, deployed and then served once after a restart, on main and on this branch: HttpArena's, 15 MB in 1,768 files. The server's peak RSS rose by:

main this branch
saving + deploying the version +237 MB, +239 MB +46 MB, +50 MB
first request after a restart +111 MB, +141 MB +37 MB, +55 MB
on disk 23.1 MB 15.3 MB

What changed

  • LambdaFile holds content three ways. As text, as the bytes it was read as, or as a reference into the store. It turns one into another only when something asks for it.
    • Archives and pushes keep resources as bytes (FromBytes).
    • Archives and pushes now pass files to new Save and CheckAsync overloads on IMetaService and IFeatureService, rather than serializing them into one text first.
  • The two writers fill the store first. LambdaHistory.Append and FeatureService.Save put the resources into the store (LambdaSource.Keep) before writing the envelope that names them. That envelope is "version": 3.
  • Reading goes through the store.
    • Going online copies resources out of the store file to file.
    • Archives, exports, published sources and git read them through it (LambdaVersionContent.Files(), FeatureContent.Files(), IStorageService.ReadFiles).
    • Parsed without the store, a stored resource can still be named, counted and compared, but throws when read rather than reading as nothing.
  • Envelopes are written with relaxed escaping. The default encoder turned every <, >, & and ' in a page into six characters. An envelope of the second layout, which still carries its resources, is read from its file instead of into a string first.
  • Unused resources are collected. A resource that no version, feature or preview names any more is removed together with the version or feature that named it last, once it is an hour old. The hour covers a save in progress, whose resources are kept before the envelope naming them is written. An envelope that can't be read stops the collection.
  • The collector conserves memory. System.GC.ConserveMemory is set in the project file, so a spike from one lambda is not held for all of them.
  • No frontend change. The API still returns resources inline, as the editor expects.

Compatibility

Nothing stored is rewritten. Versions and features of the second layout are read as they are, resources inside, and use the new layout the next time they are saved. Demos are compared by what their files hold, so the seeder doesn't save them again on every start.

Tests

ResourceStoreTests adds six tests:

  • a version names its resources and serves them byte for byte, in pages, zips and the API;
  • a resource two versions ship unchanged is kept once;
  • a version of the second layout runs, and edits on top of it are saved in the new layout;
  • pruning removes what no version names any more;
  • deleting a feature removes what only it shipped;
  • seeding demos again saves nothing.

Two assertions that expected "version": 2 as today's layout now expect 3. The suite passes, 836 of 836.

Two tests are flaky on main as well, unchanged by this PR:

  • TheExportCarriesTheDatabase failed 2 of 4 runs. It scans the machine's temp directory for any recent genhttp-lambda-export-*.db, so it also finds copies from other tests exporting in parallel.
  • ABuildTheAgentDeclinedSaysWhyInTheWordsOfTheAgent failed 1 of 4.

Docs

  • README: the storage service, how resources are kept, the disk and memory budget, and the GC setting.
  • CLAUDE.md: the decisions (how resources are kept, conserving memory) and the layouts.
  • The resources.size line of platform_guide.

🤖 Generated with Claude Code

A version stored every resource inside its envelope - text as JSON strings,
everything else as base64 - and was read, parsed and rebuilt as one string
whenever it was saved or went online. A site of 15 MB in 1,768 files became a
46 MB string, and saving and deploying it took the shared server 237 MB
higher; the first request after a restart, 111 MB more.

Now a version names each resource by the SHA-256 of what it holds, and the
lambda's resource store keeps it once, below {data}/code/{id}/resources/, for
every version, feature and preview that ships it:

- LambdaFile holds what a file holds as text, as the bytes it was read as, or
  as a reference into the store, and turns one into another only when asked.
  Archives and pushes keep resources as bytes (FromBytes) and hand files to
  the new Save/CheckAsync overloads instead of serializing them into one text.
- LambdaHistory.Append and FeatureService.Save put the resources into the
  store (LambdaSource.Keep) before writing the envelope (version 3) that names
  them.
- Going online copies them out of the store file to file, and archives,
  exports, sources and git read them through it (content.Files(),
  Storage.ReadFiles). Read without the store, a stored resource can be named,
  counted and compared, and throws when read rather than reading as nothing.
- Envelopes are written without HTML escaping, and one of the second layout,
  which still carries its resources, is read from its file rather than into a
  string first. Nothing stored is rewritten.
- What no version, feature or preview names any more is removed with the
  version or feature that named it last, once it is an hour old.
- The server collector conserves memory (System.GC.ConserveMemory), since a
  spike of one lambda was held for all of them.

Measured with the same site: save and deploy +46 MB instead of +237, the
first request after a restart +37 MB instead of +111, on disk 15.3 MB instead
of 23.1.
@MDA2AV MDA2AV closed this Oct 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant