Keep a lambda's resources once, apart from its versions

CLAUDE.md

@@ -315,14 +315,44 @@ build machinery for compatibility that nobody needs.
   leaves (`DatabaseVault.RoomOf`).
 - **Lambdas saved before keep working, untouched on disk.** A version or a
   feature is read in the layout it was saved in and translated as it is read
-  (`LambdaSource.Parse`: an envelope `"version": 1` is the old layout, 2 is
-  today's) - assets at the top move below `resources/`, `.lambda/docs/`,
+  (`LambdaSource.Parse`: an envelope `"version": 1` is the first layout, 2
+  the second - today's names, every resource inside it - and 3 today's) -
+  assets at the top move below `resources/`, `.lambda/docs/`,
   `.lambda/tests/` and `.lambda/build/` to `docs/`, `tests/` and `build/`;
   C# that was in `.lambda/` was never compiled, so it keeps a `.txt` ending
   (`tests/Smoke.cs.txt`) rather than be compiled into a lambda online.
   Never rewrite stored versions for it. Their git history keeps its commits;
   the move is a commit of its own on top (see Git). A top-level `assets/` or
   `.lambda/` is refused with a hint where it goes now.
+- **Resources are kept once, by what they hold - decided.** A version or a
+  feature names each resource by the SHA-256 of what it holds (`stored` and
+  `length` in the envelope) and the lambda's `IResourceStore`
+  (`{data}/code/{id}/resources/`) keeps it, once for all its versions,
+  features and previews. Carried inside, every save and every deployment of a
+  site took several copies of all of it as strings and base64 in one process
+  for every lambda: a fifteen megabyte site spiked the server by hundreds. So:
+  - **Never carry a resource as text when it can be bytes.** What an archive or
+    a push holds becomes a file with `LambdaFile.FromBytes`, and goes on as
+    files to the overloads that take them (`IMetaService.Save`,
+    `IFeatureService.Save`, `CheckAsync`) - never `LambdaSource.Serialize`d
+    into one text on its way there.
+  - **The two writers keep them** (`LambdaHistory.Append`,
+    `FeatureService.Save`, both with `LambdaSource.Keep`) before anything that
+    names them is written, so a version only ever names what is kept.
+  - **What needs what a resource holds reads it through the store**:
+    `LambdaVersionContent.Files()`, `FeatureContent.Files()`,
+    `IStorageService.ReadFiles`, or `LambdaSource.Parse(code, resources)`.
+    Parsed without one, a stored resource can be named, counted and compared
+    and throws when read, rather than reading as nothing. Going online copies
+    a resource out of the store file to file (`Materialize`), and an archive
+    streams it.
+  - Versions of the second layout are read as they are, resources inside, and
+    never rewritten.
+  - What nothing names any more goes with the version or the feature that
+    named it last (`DeleteVersion`, `DeleteFeature`), and only once it is an
+    hour old: the resources of a save are kept before what names them is
+    written, under a turn a pruning does not hold. An envelope that cannot be
+    read stops the collection rather than naming nothing.
 - Names: a C# file's own name as C# names (dots allowed), its folders as the
   rest of the code; a resource as files that are served
   always were (an extension, no dot names, six folders deep); the rest of the
@@ -756,6 +786,10 @@ rules that matter:
   interface (`IMetaService.GetFacts`, `ISourceService.Star`), not taken from
   the unit. An exception is an entry in the test with its reason; today only
   the composition root (`Application`) has one.
+- **The collector conserves memory - decided.** Server GC, with
+  `System.GC.ConserveMemory` set in the project file: every lambda shares the
+  process, and what one does now and then (a large version saved, an archive
+  packed) left the high water mark of large objects held for all of them.
 - Compiled lambdas run in the server process. The `CodeGuard` raises the cost of
   misbehaving; **it is not a sandbox.** Do not describe it as one.
 - Database: SQLite, EF Core, migrated by Evolve from `Data/Migrations/V<n>__*.sql`.
@@ -819,7 +853,9 @@ method is called - so the code is written the way it runs:
   binding with Roslyn, packing, copying a workspace or a database. Not in many
   small asynchronous steps (`File.ReadAllTextAsync` hops out and back per
   buffer), and not in place. A version's file is read and written in place:
-  it is bounded by its tier's allowance, and saving one is rare. `Task.Run` is
+  it names its resources rather than carrying them, is bounded by its tier's
+  allowance, and saving one is rare; one that still carries them is read from
+  its file as it is parsed, not into a string first. `Task.Run` is
   refused, so every hop is marked in `Offload` and can be given a scheduler of
   its own there; work started in the background rather than awaited (the
   scheduler's jobs, seeding the demos) is not a hop and suppresses RS0030

README.md

@@ -383,13 +383,28 @@ read in the layout it was saved in, so the old ones are translated as they are
 read - `site/app.js` becomes `resources/site/app.js`, `.lambda/docs/product.md`
 becomes `docs/product.md`, `.lambda/build/package.json` becomes
 `build/package.json` - and saved in the new one when they are next saved (an
-envelope with `"version": 2`, `LambdaSource.Parse`). C# that was in `.lambda/`
+envelope with `"version": 3` today, `LambdaSource.Parse`). C# that was in `.lambda/`
 was never compiled and would be now, so it keeps what it holds under a name
 that is not: `.lambda/tests/Smoke.cs` becomes `tests/Smoke.cs.txt`. A
 feature's files are read the same way. In git, a lambda's history keeps the commits its clones already
 have; the move is a commit of its own on top of each (see
 [Working with git](#working-with-git)).
 
+**Resources are kept once, apart from the versions.** A version names its
+resources by the SHA-256 of what they hold rather than carrying them, and the
+lambda keeps each of them once, below `{data}/code/{id}/resources/`, for every
+version, feature and preview that ships it unchanged. Going online copies them
+out of there, file to file, and an archive is packed by streaming them, so
+nothing reads a site's worth of pages into the server's memory to save it or
+put it online - which, when versions carried their resources as text and
+base64 in one string, took a few hundred megabytes of a process every lambda
+shares for a site of fifteen. A version saved before carries its resources
+inside, as an envelope of `"version": 2` did, and is read as it is - nothing
+on disk is rewritten - until it is saved again. A resource that no version,
+feature or preview names any more is removed with the version or the feature
+that named it last, once it is an hour old: the resources of a save are kept
+before what names them is written.
+
 ### What it is built from
 
 Some of a lambda may be made by a build tool rather than written as it is
@@ -1146,7 +1161,9 @@ its own (`IDatabaseVault`, `ISecretVault`, `IDomainRegistry`, `ILogBook`,
   maps the DTOs. `LambdaGuard` (who may change a lambda - not a demo) and
   `VersionInput` (what a version may hold) are rules other services apply too.
 - **Storage** (`Services/Storage`) - the code itself, on the file system, one
-  file per version. Never in the database.
+  file per version, and the resources of a lambda's versions and features in a
+  store of its own, each once, by what it holds (`ResourceStore`). Never in the
+  database.
 - **Data** (`Services/Data`) - which kinds of data a lambda keeps, switched on
   and off by its owner. Only the choice is stored: a lambda without a row for a
   kind has its default, so a kind added to `DataKinds` needs no migration to
@@ -1439,7 +1456,11 @@ Both are for the operator only, behind the administration token (see
 in the editor, and nobody sees everybody's. What it is for is
 the shape of the memory curve. A managed heap that climbs across gen 2
 collections is a leak; committed bytes climbing while the managed heap stays
-flat is the heap keeping pages it could return, which is not.
+flat is the heap keeping pages it could return, which is not. The collector is
+asked to conserve memory (`System.GC.ConserveMemory` in the project file), so
+after a spike - a large version saved, an archive packed - it compacts the
+large object heap and gives back what it no longer needs, rather than keeping
+the high water mark of the busiest lambda for all of them.
 
 ## TLS
 
@@ -1502,18 +1523,19 @@ version may come to, its code and its resources together, is a
 [limit](#limits) of the lambda's tier. How many files there are is not
 limited, and neither is the number of C# files.
 
-Budget the disk for the premium tier. A version is kept whole, its files as
-text or base64 in JSON, so each version saved at the limit takes up to half as
-much again on disk - 184 MB for 128 MB of pictures - and fifty of them are
-kept by default: at the defaults, a premium lambda that is saved over and over
-at its limit can come to around 9 GB of history.
-
-Budget the memory too. A version is read and written whole: with the 128 MB
-the premium tier allows, saving one took the server to 2.3 GB and deploying it
-to 2.8 GB. Even the 32 MB of a free lambda took a server idling at 350 MB to
-950 MB to save, and anybody can create a free lambda. Set what a version may
-come to, for both tiers, in the panel's **Limits** to what the machine can
-carry until resources are stored apart from the versions.
+Budget the disk for the premium tier. A version keeps its code whole, as text
+in JSON, and names its resources, which the lambda's store keeps once each, as
+the bytes they are: a version costs what it changed, and fifty of them are
+kept by default. A version saved before resources were kept apart carries
+them inside, as text or base64, up to half as much again as they are.
+
+Budget the memory too. Saving a version reads the archive or the request it
+came in, and its resources go into the store file by file; going online copies
+them out. A site of 15 MB in 1,768 files took the server 237 MB higher to save
+and deploy while versions carried their resources, and 46 MB since; its first
+request after a restart, 111 MB and 37. Set what a version may come to, for
+both tiers, in the panel's **Limits** to what the machine can carry - the
+request a version arrives in is still read whole.
 
 ### More than one hostname
 

src/GenHTTP.Lambda/Api/CodeResource.cs

@@ -28,7 +28,7 @@ public sealed class CodeResource(IMetaService meta)
     [ResourceMethod(Method.Post, "lambdas/:privateKey/code/check")]
     public async ValueTask<CompilationResponse> Check(string privateKey, CodeRequest request)
     {
-        var outcome = await meta.CheckAsync(privateKey, VersionResource.Serialize(request.Files));
+        var outcome = await meta.CheckAsync(privateKey, VersionResource.Checked(request.Files));
 
         return new CompilationResponse(outcome.Success, outcome.Diagnostics);
     }

src/GenHTTP.Lambda/Api/FeatureResource.cs

@@ -78,7 +78,7 @@ public FeatureContentResponse Get(string privateKey, string feature, string? fol
     {
         var found = features.Get(privateKey, feature);
 
-        return new FeatureContentResponse(Describe(found.Feature), VersionResource.Below(LambdaSource.Parse(found.Code), folder));
+        return new FeatureContentResponse(Describe(found.Feature), VersionResource.Below(found.Files(), folder));
     }
 
     /// <summary>
@@ -132,7 +132,7 @@ public async ValueTask<FeatureSavedResponse> Change(string privateKey, string fe
     {
         var read = features.Get(privateKey, feature);
 
-        var files = LambdaChanges.Apply(LambdaSource.Parse(read.Code), request.Files, request.Remove, request.Edits);
+        var files = LambdaChanges.Apply(read.Files(), request.Files, request.Remove, request.Edits);
 
         // made on top of what was read, and refused if something else was saved meanwhile
         return await SaveAsync(privateKey, feature, files, deploy, request.Specification, request.Change, read.Feature.Revision);
@@ -150,7 +150,7 @@ public IResponse GetArchive(string privateKey, string feature, string? layout, I
 
         var found = features.Get(privateKey, feature);
 
-        var zip = LambdaArchive.Pack(LambdaSource.Parse(found.Code), laid);
+        var zip = LambdaArchive.Pack(found.Files(), laid);
 
         logger.LogInformation("Downloaded feature '{Feature}' of lambda {Lambda} layout {Layout}", found.Feature.Name, meta.PublicKeyOf(privateKey), laid);
 
@@ -181,7 +181,7 @@ public async ValueTask<FeatureSavedResponse> PutArchive(string privateKey, strin
         var tier = Enum.Parse<LambdaTier>(lambda.Tier);
 
         var files = await VersionResource.UnpackAsync(body, limits.BuildOf(tier), laid,
-                                                      LambdaSource.Parse(features.Get(privateKey, feature).Code));
+                                                      features.Get(privateKey, feature).Files());
 
         return await SaveAsync(privateKey, feature, files, deploy, specification, change);
     }
@@ -328,7 +328,7 @@ public async ValueTask<Result<FeatureMergeResponse>> Merge(string privateKey, st
     private async ValueTask<FeatureSavedResponse> SaveAsync(string privateKey, string feature, IReadOnlyList<LambdaFile>? files, bool? deploy,
                                                             string? specification, string? change, int? after = null)
     {
-        var saved = features.Save(privateKey, feature, VersionResource.Serialize(files), new VersionNote(specification, change), after);
+        var saved = features.Save(privateKey, feature, VersionResource.Checked(files), new VersionNote(specification, change), after);
 
         var publicKey = meta.PublicKeyOf(privateKey);
 

src/GenHTTP.Lambda/Api/LambdaResource.cs

@@ -171,7 +171,7 @@ public async ValueTask<IResponse> Export(string privateKey, IRequest request)
             {
                 using var stream = File.Create(target);
 
-                ProjectPacker.Pack(exported, LambdaSource.Parse(content.Code), stream, database);
+                ProjectPacker.Pack(exported, content.Files(), stream, database);
 
                 return target;
             }