Repository navigation
Conversation
A version stored every resource inside its envelope - text as JSON strings,
everything else as base64 - and was read, parsed and rebuilt as one string
whenever it was saved or went online. A site of 15 MB in 1,768 files became a
46 MB string, and saving and deploying it took the shared server 237 MB
higher; the first request after a restart, 111 MB more.
Now a version names each resource by the SHA-256 of what it holds, and the
lambda's resource store keeps it once, below {data}/code/{id}/resources/, for
every version, feature and preview that ships it:
- LambdaFile holds what a file holds as text, as the bytes it was read as, or
as a reference into the store, and turns one into another only when asked.
Archives and pushes keep resources as bytes (FromBytes) and hand files to
the new Save/CheckAsync overloads instead of serializing them into one text.
- LambdaHistory.Append and FeatureService.Save put the resources into the
store (LambdaSource.Keep) before writing the envelope (version 3) that names
them.
- Going online copies them out of the store file to file, and archives,
exports, sources and git read them through it (content.Files(),
Storage.ReadFiles). Read without the store, a stored resource can be named,
counted and compared, and throws when read rather than reading as nothing.
- Envelopes are written without HTML escaping, and one of the second layout,
which still carries its resources, is read from its file rather than into a
string first. Nothing stored is rewritten.
- What no version, feature or preview names any more is removed with the
version or feature that named it last, once it is an hour old.
- The server collector conserves memory (System.GC.ConserveMemory), since a
spike of one lambda was held for all of them.
Measured with the same site: save and deploy +46 MB instead of +237, the
first request after a restart +37 MB instead of +111, on disk 15.3 MB instead
of 23.1.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Saving or deploying a version with a lot of resources took the shared server hundreds of megabytes. Every resource was stored inside the version's JSON envelope, text as strings and everything else as base64. Each save and deploy then built and parsed the whole version as one string.
With this PR, a version names each resource by the SHA-256 of what it holds. The lambda's resource store keeps each one once, below
{data}/code/{id}/resources/, for every version, feature and preview that ships it.Measured
The same site was saved, deployed and then served once after a restart, on
mainand on this branch: HttpArena's, 15 MB in 1,768 files. The server's peak RSS rose by:mainWhat changed
LambdaFileholds content three ways. As text, as the bytes it was read as, or as a reference into the store. It turns one into another only when something asks for it.FromBytes).SaveandCheckAsyncoverloads onIMetaServiceandIFeatureService, rather than serializing them into one text first.LambdaHistory.AppendandFeatureService.Saveput the resources into the store (LambdaSource.Keep) before writing the envelope that names them. That envelope is"version": 3.LambdaVersionContent.Files(),FeatureContent.Files(),IStorageService.ReadFiles).<,>,&and'in a page into six characters. An envelope of the second layout, which still carries its resources, is read from its file instead of into a string first.System.GC.ConserveMemoryis set in the project file, so a spike from one lambda is not held for all of them.Compatibility
Nothing stored is rewritten. Versions and features of the second layout are read as they are, resources inside, and use the new layout the next time they are saved. Demos are compared by what their files hold, so the seeder doesn't save them again on every start.
Tests
ResourceStoreTestsadds six tests:Two assertions that expected
"version": 2as today's layout now expect 3. The suite passes, 836 of 836.Two tests are flaky on
mainas well, unchanged by this PR:TheExportCarriesTheDatabasefailed 2 of 4 runs. It scans the machine's temp directory for any recentgenhttp-lambda-export-*.db, so it also finds copies from other tests exporting in parallel.ABuildTheAgentDeclinedSaysWhyInTheWordsOfTheAgentfailed 1 of 4.Docs
CLAUDE.md: the decisions (how resources are kept, conserving memory) and the layouts.resources.sizeline ofplatform_guide.🤖 Generated with Claude Code