diff --git a/CLAUDE.md b/CLAUDE.md index 969888e2..09966c06 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -266,8 +266,8 @@ scripts and test data. this is said wherever they are told to write them. The demos have more, because they teach. This is said the same way in the MCP instructions, the tool descriptions and answers, `platform_guide` - (`documentationAndTests`), the build agent's brief, the guide and the README - - keep them the same. + (`documentationAndTests`), the build agent's brief, a repository's + `AGENTS.md`, the guide and the README - keep them the same. - The editor: **Documentation** is the second section after the overview, in both views; **Tests** is in the full view only. The simple view calls the documentation **About** and shows the product page alone, to be corrected by @@ -282,7 +282,8 @@ Graph), what the page is about in the HTML as served rather than only in what a script renders, and the domain as canonical where the lambda has one. A tool for a few people gets a title and nothing more. This is said in the MCP instructions (one line), `platform_guide` (`beingFound`), the build agent's -brief, the README and `/docs` - keep them the same. The platform writes none +brief, a repository's `AGENTS.md`, the README and `/docs` - keep them the same. +The platform writes none of it into a lambda's pages; the agent does. - `og:image` takes a full address - social networks do not resolve a relative @@ -314,7 +315,8 @@ server - server-sent events where the page only listens (`demo-live`), a websocket where it talks back (`demo-game`). A poll is a request whether anything changed or not, on a server every lambda shares. This is said in the MCP instructions (one line), `platform_guide` (`liveUpdates`), the build -agent's brief, the README and `/docs` - keep them the same. +agent's brief, a repository's `AGENTS.md`, the README and `/docs` - keep them +the same. ### The link back @@ -338,8 +340,8 @@ agent's brief, the README and `/docs` - keep them the same. script, nothing hidden. - The platform never writes it into a lambda, and nothing checks for it. - This is said in the MCP instructions (one line), `platform_guide` - (`backlink`), the build agent's brief, the README and `/docs` - keep them - the same. + (`backlink`), the build agent's brief, a repository's `AGENTS.md`, the README + and `/docs` - keep them the same. ### Features ("Drafts") @@ -357,7 +359,8 @@ New functionality is developed in a **feature** (a *draft* in the editor): the lambda's lock. - A lambda that exists is changed in a feature; a new lambda is written as versions. This is said the same way in the MCP instructions, the tool - descriptions and answers, the guide, and the README - keep them the same. + descriptions and answers, the guide, a repository's `AGENTS.md`, and the + README - keep them the same. ### Showcase, export, ownership of the code @@ -413,6 +416,60 @@ to read, star and download. - The demos are published under MIT by the installation, being there to be read and built on; nobody can change that. +### Git + +Every lambda is a git repository as well: `/editor/{privateKey}/{name}.git` +for its owner to clone and push to - the name only names the clone's folder, +the editor offers the public key - and `/source/{publicKey}.git` for anybody +to clone while its source is published. Served by GenHTTP.Modules.Git from +what the platform keeps (`Services/Git`); there is no repository on disk. + +- **One history per lambda - decided.** `main` is the newest version, every + version a commit tagged `v{n}`, every feature a branch. The published + source is the same commits without the features' branches, whose commits + it does not serve either. So nothing only the owner may know goes into a + commit the platform makes: no specification, no editor key - the message is + the change line. +- **A commit is made once and kept.** The platform makes the commits of + versions and of features' saves when the repository is read, not when they + are saved - saving stays as it was, and a lambda nobody clones costs + nothing - and keeps their bytes and their files below `/data/git/{lambda}`. + Never make one again from the version: a newer platform would make it + differently, and every clone would see its history rewritten. Pushed + commits are kept exactly as they came. +- **The tree is the project, the export's shape made for a repository.** What + the platform puts around the lambda (`Program.cs`, the project file, + `Platform/`, `Dockerfile`, the ignore files, `AGENTS.md`, `CLAUDE.md`, + `LICENSE` while published) depends on nothing a version decides - so the + project references SQLite, EF Core and Evolve always and the snippet is + `async Task BuildAsync()` always - and a push that leaves those files + alone is right whatever it changed. Keep it that way. A commit whose + lambda.cs did not change keeps the `Project.cs` before it. +- **The platform's files are refused, not ignored - decided.** A push that + changes, adds or removes one, or adds a file with no place in a lambda, is + refused saying which and how to undo it; ignoring it would leave a commit + that says what the lambda does not. +- **A commit pushed to main is a version, each one.** `main` only moves + forward, linear - no force, no merge commits, no deletion - and the newest + is compiled first and refused when it does not compile, as a merge is. + Pushes go through the services every other door uses (origin `git`), so + their rules hold. A feature whose tip reaches `main` is merged. +- **A branch is a feature.** Pushed, it starts or replaces one - based on + the newest version among its commits - and puts its preview online, which + no visitor sees; `-o merge` merges it, `-o deploy` puts a version online. + A feature's branch is named once (`features.branch`) and kept when it is + renamed. +- **The answers never hold the editor key.** Whoever pushed has it, and a + build log of theirs may be read by others. +- **For whoever works with code.** The clone popover is in the full view only + - the overview's header, the code section, a draft - and on `/source` in the + **Code** menu. Nothing about git is in the simple view or on `/build`. +- **`AGENTS.md` is a place the platform's rules are said** for an agent + working in a clone (`Resources/Export/Agents.md.txt`), in git's terms: keep + it saying what the MCP instructions and `platform_guide` say. +- A test clones every demo and builds the clone with the .NET SDK, as one + builds every export; keep it passing. + ### Demos The demos (`DemoCatalog`, files in `Resources/Templates`) exist to show agents @@ -756,7 +813,8 @@ touches all that apply: 2. `/docs` (`src/Frontend/src/locales/*/guide.tsx`) - for owners, in the vocabulary of the simple view, translated. 3. `platform_guide`, the MCP instructions and the tool descriptions - (`Api/Mcp`) - for agents. + (`Api/Mcp`) - for agents; and `AGENTS.md` in every lambda's repository + (`Resources/Export/Agents.md.txt`) for an agent working in a clone. 4. The API description (Scalar at `/api/v1/scalar/`, from the resources). The same rule is worded the same way in all of them. diff --git a/Directory.Packages.props b/Directory.Packages.props index 9b8c00c4..661b1f9b 100644 --- a/Directory.Packages.props +++ b/Directory.Packages.props @@ -10,6 +10,9 @@ + + + diff --git a/README.md b/README.md index 57c6894f..2ef48a6c 100644 --- a/README.md +++ b/README.md @@ -74,9 +74,11 @@ port, each against its own temporary data directory. | `/showcase` | the lambdas their owners chose to show | | `/source` | the lambdas whose owners published their code | | `/source/:publicKey` | the published code of one: its files, documentation, tests and changes, per version | +| `/source/:publicKey.git` | the published code as a git repository, read only | | `/enterprise`, `/terms`, `/privacy`, `/imprint` | the company pages | | `/editor/create` | the creation assistant | | `/editor/:privateKey`| the editor for one lambda | +| `/editor/:privateKey/:name.git` | the lambda as a git repository, to clone and push to (any name; the editor offers the public key) | | `/lambda/:publicKey` | the deployed handler | | `/features/:feature` | the preview of a feature, while it is online | | any path, at a lambda's own domain | the deployed handler of a premium lambda with that domain | @@ -694,6 +696,116 @@ The demos are published under MIT by the installation, being there to be read and built on; nobody else can change that, and their editor shows no Open source section. +### Working with git + +Every lambda is a git repository as well, so whoever works on it with their +own tools and agent clones it, changes it and pushes - and whoever reads a +published source clones every version of it. Both are served by the +[GenHTTP.Modules.Git][git-module] package from what the platform already +keeps: there is no repository on disk, only the commits the versions and the +features are. + +```bash +git clone https://genhttp.dev/editor//my-app.git +cd my-app && dotnet run # the app on http://localhost:8080/ +# change it, commit +git push -o deploy # the next version, online +``` + +The owner's address is the editor's with a name added, which only names the +folder a clone makes: any name answers, so a clone keeps working once the +public key changes. It holds the editor key like the editor's address does - +there are no accounts, and the key is what lets a push in. The published +source is `/source/.git`, read only, while it is published. Both +answer without the `.git`, and opened in a browser they lead to their page. +The editor offers the address under **Clone** - in the full view only, on the +overview and beside the code, and in a draft with its branch - and `/source` +in the **Code** menu beside the star, where the zip is too. + +**What is in it.** The project the lambda is exported as, made for a +repository: `Project.cs` is the snippet as the body of `BuildAsync()` - the +method the platform runs it in - with its types beside the class, the other +`.cs` files keep their code and are named the .NET way, `assets/` is what it +ships, and `docs/` and `tests/` are its `.lambda/docs/` and `.lambda/tests/`. +Around them are the platform's files: `Program.cs`, the project file, +`Platform/`, the `Dockerfile`, `.gitignore`, `.dockerignore`, `AGENTS.md` - +how an agent works in the repository, with the rules of the platform in git's +terms - `CLAUDE.md`, which points Claude Code to it, and `LICENSE` while the +source is published. Those are the same in every commit until the platform +changes them - nothing in them depends on the version, which is why the +project references SQLite, Entity Framework and Evolve whether the code uses +them or not, and makes the snippet asynchronous whether it awaits or not - so +a push that leaves them alone is always right. The data is never in it: no +database, no workspace, no secret. Nor is what only the owner may know - no +specification, no editor key - so the commits the owner reads are the commits +a published source shows. + +**The history.** `main` is the newest version, each version a commit tagged +`v1`, `v2` and so on, its message the change line; a version pruned is gone +from the history too, which a clone is told is shallow from there. Each +feature is a branch, named once when it starts - after the branch that was +pushed, or after its name in the words git allows (`dark-mode`) - and kept +when it is renamed (`features.branch`); a save of the feature is a commit on +it, and moving its base past what it holds is a merge commit bringing that +version in. The commits the platform makes are made when the repository is +read, not when a version is saved, so saving stays what it was and a lambda +nobody clones costs nothing; each is made once from what it stands for - the +files, the time it was saved, its change - and kept below +`/data/git/{lambda}` with the files it holds, by the id git gives them, so it +is the same commit for everybody who reads it later, a newer platform +included. A commit whose code did not change keeps the `Project.cs` of the +one before it, so a class somebody pushed in a way of their own is not +written over by a save in the editor. The published source is the same +history without the features: their branches are not there, and their +commits are not found by id either. + +**Pushing to main.** Every commit becomes the next version, in order, with +the origin `git`, the first line of its message as the change and the rest +as the specification. `main` only moves forward one commit after another - no +force push, no merge commit, no deletion - and a version saved meanwhile in +the editor or by an agent makes the push "fetch first". A push holding more +commits than the tier keeps versions is refused, asking for them to be +squashed. The newest is compiled first, the code guard included, and a push +that does not compile is refused - its problems come back with the file and +line in the repository, `Project.cs(14,9)`. `-o deploy` puts the newest +online. A feature whose branch is now in `main` is merged and goes, as +merging it would have done. + +**Pushing a branch.** A new branch starts a feature from the newest version +among its commits, with a copy of the lambda's data; pushing it again +replaces its files, a force push included, and a branch that holds none of +the versions is refused. Every push puts its preview online - nothing a +visitor sees - and answers with its address. `-o merge` with a push merges +it as one version, `-o deploy` beside it puts that online, and deleting the +branch deletes the feature. A feature changed in the editor since the client +fetched refuses the push rather than losing that change. + +**What is refused.** A push that changes, adds or removes one of the +platform's files, or adds a file that has no place in a lambda - a `README.md` +at the root, an editor's settings - is refused and says which and how to undo +it: changing them would change nothing of the lambda, and silently dropping +the change would leave the commit saying something the lambda does not. So is +an executable file or a link, a tag, `Project.cs` with something in `class +Project` besides `BuildAsync()`, and anything the services refuse a save - +names, sizes, a demo. Everything a push does goes through the services the +editor and the agents use, under the same rules. + +**What it says.** A push is answered as it goes, in the lines git prints after +`remote:`: the versions or the feature it became, problems the compiler +found, what went online and where, and what to do next. They never hold the +editor key - whoever pushed has it, and whoever reads their build log should +not. Options travel only with a push that changes something, so a version +already pushed is deployed in the editor or with +`POST /api/v1/lambdas/:privateKey/deployment/start`, which the answer names. + +**Threads and limits.** Reading a lambda for the first time reads every +version, so the commits are made on the pool, in the lambda's turn - a +semaphore, one request at a time per lambda. The server packs what a clone +asks for, and applies a push, while it writes its answer, so that answer is +written on the pool into a pipe the reactor copies to the connection +(`OffloadedContent`). A push is unpacked in memory, up to +`LAMBDA_GIT_MAX_PUSH_BYTES`, and only two are applied at once. + ## How it is put together A single .NET 11 project hosted by `GenHTTP.Full.Ioxide`, wired up in @@ -756,6 +868,13 @@ its own (`IDatabaseVault`, `ISecretVault`, `IDomainRegistry`, `ILogBook`, `/data/sources/{lambda}`, and `StarGuard` hands out the tickets a star is given with and remembers who starred what. Kept apart from Meta like the showcase: nothing about building a lambda reads or changes it. +- **Git** (`Services/Git`) - every lambda as a git repository (see + [Working with git](#working-with-git)). `GitService` finds the lambda and + takes its turn, `GitHistory` makes the commits the versions and features + are, `GitPushes` makes versions and features of what is pushed, `GitStore` + keeps the commits and their files below `/data/git/{lambda}`, and + `ProjectTree` says what a lambda is as a tree and back. The door is + `Api/Git/GitRoutes.cs`, beside the pages below `/editor/` and `/source/`. - **Building** (`Services/Building`) - the jobs of the build agent: builds asked for on `/build` and changes asked for in the editor's Change section. `BuildService` decides what may be asked for and in which order that is @@ -863,6 +982,7 @@ server's configuration and are set in the administration panel while it runs | `LAMBDA_DEVELOPMENT` | `false` | verbose error pages and debug logging | | `LAMBDA_MAINTENANCE_INTERVAL_HOURS` | `0.25` | how often expired lambdas are looked for | | `LAMBDA_SOURCE_CACHE_BYTES` | `2147483648` | what the published sources may take on disk, packed; the least recently read go first | +| `LAMBDA_GIT_MAX_PUSH_BYTES` | `268435456` | how large a `git push` may be; it is unpacked in memory | | `LAMBDA_MAX_CONCURRENCY` | `64` | lambda requests executed at once | | `LAMBDA_EXECUTION_TIMEOUT_SECONDS` | `15` | before an invocation is aborted | | `LAMBDA_TELEMETRY_INTERVAL_SECONDS` | `30` | how often a reading is taken | @@ -1228,6 +1348,15 @@ asking which license when they did not say. Once a lambda is published, then on is public, the versions before included - so an agent keeps keys, passwords and personal data out of the files, where they never belong anyway. +An agent that can run git works in a clone instead (see +[Working with git](#working-with-git)): `create_lambda` and `read_lambda` +answer with the `gitUrl`, features name their `branch`, the instructions say +so in a line and `platform_guide` how under `git`. In the clone, `AGENTS.md` is +its guide: what is what in the repository, how pushing to `main` and to a +branch works and what is refused, and the rules of the platform said in git's +terms - the same rules, worded the same way, as the instructions, the guide +and the build agent's brief. + `read_lambda` sends every file of the program while together they come to 30,000 characters, and names them with their lengths beyond that; `file` then fetches one in full, up to a megabyte, and anything larger is left to the zip of the @@ -1527,3 +1656,4 @@ existing ones are never edited. [genhttp]: https://genhttp.org/ [evolve]: https://evolve-db.netlify.app/ [efcore]: https://learn.microsoft.com/ef/core/ +[git-module]: https://github.com/Kaliumhexacyanoferrat/virtual-git-server diff --git a/src/Frontend/src/api.ts b/src/Frontend/src/api.ts index 27df8d21..0311f8c7 100644 --- a/src/Frontend/src/api.ts +++ b/src/Frontend/src/api.ts @@ -27,6 +27,8 @@ export interface Lambda { address: string; /** How its editor opens for somebody who has not chosen a view of their own: Full or Simple. See control/view.ts. */ view: string; + /** Where it is cloned with git and pushed to - holding the editor key, like editorPath. */ + gitPath: string; } /** The tiers there are. Only an administrator moves a lambda between them. */ @@ -51,7 +53,7 @@ export interface DomainState { } /** Which door something came through. */ -export type Origin = 'template' | 'api' | 'agent' | 'admin' | 'system'; +export type Origin = 'template' | 'api' | 'agent' | 'git' | 'admin' | 'system'; export interface VersionInfo { version: number; @@ -637,6 +639,8 @@ export interface Feature { /** What it is addressed by, and where its preview answers. */ key: string; name: string; + /** The branch it is in the lambda's git repository, kept when it is renamed. */ + branch: string; /** What the user wants from it and why. */ specification?: string | null; /** What it changes, in a line - what the version it becomes will say. */ @@ -952,6 +956,8 @@ export interface SourceProject { versions: SourceVersion[]; /** Sent with a star, good for a day from a second on. */ starTicket: string; + /** Where it is cloned with git: every version, read only. */ + gitPath: string; } /** What a file of a packed project is to somebody reading it. */ diff --git a/src/Frontend/src/components/Clone.tsx b/src/Frontend/src/components/Clone.tsx new file mode 100644 index 00000000..d0f70f9c --- /dev/null +++ b/src/Frontend/src/components/Clone.tsx @@ -0,0 +1,156 @@ +import { useEffect, useRef, useState, type ReactNode } from 'react'; + +import { IconCheck, IconChevronDown, IconCopy } from './Icons'; + +/** What copying says, in the words of the page it is on. */ +export interface CopyWords { + copy: string; + copied: string; +} + +/** + * Copies a text, and says so for a moment - nothing at all where the + * clipboard refuses, which is not worth an error. + */ +function useCopy(text: string) { + const [copied, setCopied] = useState(false); + + async function copy() { + try { + await navigator.clipboard.writeText(text); + } catch { + return; + } + + setCopied(true); + window.setTimeout(() => setCopied(false), 1500); + } + + return { copied, copy }; +} + +/** The address a repository is cloned from, on one line, with a way to copy it. */ +export function CloneAddress({ url, words }: { url: string; words: CopyWords }) { + const { copied, copy } = useCopy(url); + + return ( +
+ event.currentTarget.select()} + aria-label={url} + className="min-w-0 flex-1 bg-transparent px-3 py-2 font-mono text-[12px] text-ink-800 outline-none dark:text-slate-200" + /> + +
+ ); +} + +/** A command to type, with a way to copy it. */ +export function Command({ text, words }: { text: string; words: CopyWords }) { + const { copied, copy } = useCopy(text); + + // the button in a column of its own, so a long line scrolls beside it rather than under it + return ( +
+ 
{text}
+ +
+ ); +} + +/** + * A button that opens a panel below it, closed again by Escape or a click + * anywhere else - how a repository page offers its clone address. + * + * The panel lines up with the right edge of its button where there is room, + * and is moved in from either edge of the window where there is not - on a + * phone the button may stand anywhere in a row that wrapped. + */ +export function Popover({ label, title, button, width = 400, children }: { + /** What the button says, beside its icon. */ + label: ReactNode; + /** What the panel is, for whoever cannot see it. */ + title: string; + /** How the button looks. */ + button: string; + /** How wide the panel is where the window has room for it. */ + width?: number; + children: ReactNode; +}) { + const [place, setPlace] = useState<{ left: number; width: number } | null>(null); + const host = useRef(null); + + const open = place != null; + + function toggle() { + if (open || !host.current) { + setPlace(null); + return; + } + + const box = host.current.getBoundingClientRect(); + const margin = 16; + const wide = Math.min(width, window.innerWidth - 2 * margin); + + // its right edge under the button's, kept inside the window + const left = Math.min(Math.max(box.right - wide, margin), window.innerWidth - margin - wide); + + setPlace({ left: left - box.left, width: wide }); + } + + useEffect(() => { + if (!open) { + return; + } + + const onKey = (event: KeyboardEvent) => event.key === 'Escape' && setPlace(null); + + const onPointer = (event: PointerEvent) => { + if (!host.current?.contains(event.target as Node)) { + setPlace(null); + } + }; + + document.addEventListener('keydown', onKey); + document.addEventListener('pointerdown', onPointer); + + return () => { + document.removeEventListener('keydown', onKey); + document.removeEventListener('pointerdown', onPointer); + }; + }, [open]); + + return ( +
+ + + {place && ( +
+ {children} +
+ )} +
+ ); +} diff --git a/src/Frontend/src/components/Icons.tsx b/src/Frontend/src/components/Icons.tsx index 8009c423..986e08be 100644 --- a/src/Frontend/src/components/Icons.tsx +++ b/src/Frontend/src/components/Icons.tsx @@ -290,6 +290,11 @@ export const IconCode = (p: IconProps) => ( ); +/** A branch leaving a line and coming back, for git: cloning, branches. */ +export const IconBranch = (p: IconProps) => ( + +); + /** A magnifying glass, for searching. */ export const IconSearch = (p: IconProps) => ( diff --git a/src/Frontend/src/control/CloneMenu.tsx b/src/Frontend/src/control/CloneMenu.tsx new file mode 100644 index 00000000..eec3db9b --- /dev/null +++ b/src/Frontend/src/control/CloneMenu.tsx @@ -0,0 +1,77 @@ +import { absoluteAddress } from '../address'; +import { isDemo, type Feature, type Lambda } from '../api'; +import { CloneAddress, Command, Popover } from '../components/Clone'; +import { IconBranch, IconLock } from '../components/Icons'; +import { useEditorT } from '../i18n'; + +/** + * Where a lambda is cloned with git, the way a repository page offers it: a + * button that opens the address, the commands to start with, and what + * pushing does. + * + * For somebody who works with code, so the full view has it - on the + * overview, which is the lambda's front page, and beside its code. In a draft + * it says which branch the draft is. + * + * The address holds the editor key, as the editor's own does: the platform + * has no accounts, and the key is what lets a push in. So it says so. + */ +export function CloneMenu({ lambda, feature }: { lambda: Lambda; feature?: Feature | null }) { + const said = useEditorT().clone; + + const url = absoluteAddress(lambda.gitPath); + const demo = isDemo(lambda.tier); + + const words = { copy: said.copy, copied: said.copied }; + + const commands = feature + ? `git clone ${url}\ncd ${lambda.publicKey}\ngit switch ${feature.branch}` + : `git clone ${url}\ncd ${lambda.publicKey}\ndotnet run`; + + const code = 'rounded-sm bg-slate-100 px-1 py-px font-mono text-[12px] text-ink-800 dark:bg-ink-850 dark:text-slate-200'; + + return ( + {said.button}} + title={said.title} + button="btn-ghost !px-3 !py-1.5 text-[13px]" + width={416} + > +

+ + {said.title} +

+

{said.intro}

+ +
+ + {!demo && ( +

+ + {said.keyWarning} +

+ )} + + + + {feature && ( +

{said.draft({feature.branch})}

+ )} + +
+ {demo ? ( +

{said.readOnly}

+ ) : ( + <> +

{said.pushing}

+
    +
  • {said.toMain(git push -o deploy)}
    • +
  • {said.toBranch}
    • +
  • {said.agents(AGENTS.md)}
    • +
+ + )} +
+ + ); +} diff --git a/src/Frontend/src/control/FeatureTab.tsx b/src/Frontend/src/control/FeatureTab.tsx index cf1f833b..ad0cd0e3 100644 --- a/src/Frontend/src/control/FeatureTab.tsx +++ b/src/Frontend/src/control/FeatureTab.tsx @@ -1,9 +1,10 @@ import { useEffect, useState } from 'react'; import { ApiError, api, type LambdaFile } from '../api'; -import { IconAlert, IconChevronDown, IconPencil, IconSpark, IconSpinner } from '../components/Icons'; +import { IconAlert, IconBranch, IconChevronDown, IconPencil, IconSpark, IconSpinner } from '../components/Icons'; import { useEditorT } from '../i18n'; import { ChangeList } from './Changes'; +import { CloneMenu } from './CloneMenu'; import type { Control } from './context'; import { AgentMark, Ago, Quote, Section } from './ui'; import { isContext } from './written'; @@ -83,6 +84,7 @@ export function FeatureTab({ control, onNotes, onRebase }: { {said.editCode} )} + {!simple && } } > @@ -91,11 +93,21 @@ export function FeatureTab({ control, onNotes, onRebase }: { {said.started} - + {said.changed} + {/* where a clone has it, for whoever works on it with git */} + {!simple && ( + <> + + + + {feature.branch} + + + )}

{behind && ( diff --git a/src/Frontend/src/control/SummaryTab.tsx b/src/Frontend/src/control/SummaryTab.tsx index 4ce66875..ebee6e05 100644 --- a/src/Frontend/src/control/SummaryTab.tsx +++ b/src/Frontend/src/control/SummaryTab.tsx @@ -3,6 +3,7 @@ import { Link } from 'react-router-dom'; import { isDemo } from '../api'; import { IconAlert, IconCheck, IconDraft, IconGlobe, IconLock, IconSpinner } from '../components/Icons'; import { useEditorT } from '../i18n'; +import { CloneMenu } from './CloneMenu'; import type { Control } from './context'; import { ago, bytes, count, local, millis, percent, span } from './format'; import { AgentMark, Ago, Figure, LiveDot, Meter, Quote, Section, Sparkline } from './ui'; @@ -44,6 +45,7 @@ export function SummaryTab({ control }: { control: Control }) {
} > {documentation.about && (

@@ -112,7 +114,7 @@ export function SummaryTab({ control }: { control: Control }) {

{said.version(latest.version)} - + {latest.version !== lambda.activeVersion && {said.notOnline}}

diff --git a/src/Frontend/src/control/VersionsTab.tsx b/src/Frontend/src/control/VersionsTab.tsx index a65be189..8a44dd66 100644 --- a/src/Frontend/src/control/VersionsTab.tsx +++ b/src/Frontend/src/control/VersionsTab.tsx @@ -105,7 +105,7 @@ function Row({ {said.online} )} - + diff --git a/src/Frontend/src/control/Workbench.tsx b/src/Frontend/src/control/Workbench.tsx index 2404808d..ea6a4c23 100644 --- a/src/Frontend/src/control/Workbench.tsx +++ b/src/Frontend/src/control/Workbench.tsx @@ -10,6 +10,7 @@ import { IconAlert, IconPlay, IconSpinner } from '../components/Icons'; import { useToast } from '../components/Toast'; import { useEditorT } from '../i18n'; import { languageFor } from '../monaco'; +import { CloneMenu } from './CloneMenu'; import type { Control } from './context'; import { Section } from './ui'; import { CONTEXT, isCode } from './written'; @@ -337,6 +338,7 @@ export function Workbench({ control, onDirty }: { control: Control; onDirty: (di } actions={ <> +
+
+

{said.git(k)}

+ /my-app.git\ncd my-app\n# change, commit\ngit push -o deploy'} /> +

{said.git2(k)}

+ +
+

{said.open(k)}

{said.open2(k)}

@@ -447,6 +455,11 @@ function Sample({ code }: { code: string }) { ); } +/** Commands to type, as they are typed - no colours. */ +function Commands({ text }: { text: string }) { + return 
{text}
; +} + function Code({ children }: { children: React.ReactNode }) { return ( diff --git a/src/Frontend/src/source/Download.tsx b/src/Frontend/src/source/Download.tsx index deb042fa..ede77f3b 100644 --- a/src/Frontend/src/source/Download.tsx +++ b/src/Frontend/src/source/Download.tsx @@ -1,7 +1,8 @@ -import { useEffect, useRef, useState } from 'react'; +import { useState } from 'react'; import { ApiError, api } from '../api'; -import { IconCheck, IconChevronDown, IconCopy, IconDownload, IconSpinner } from '../components/Icons'; +import { CloneAddress, Command, Popover } from '../components/Clone'; +import { IconBranch, IconCode, IconDownload, IconSpinner } from '../components/Icons'; import { useToast } from '../components/Toast'; import { useSourceT } from '../i18n'; import { useFormat } from './format'; @@ -67,121 +68,74 @@ export function DownloadLink({ publicKey, version }: { publicKey: string; versio } /** - * Taking the code away: the version being read as a project, what is in it - * and what is not, and how to run it - the way out every lambda has, offered - * to everybody once its owner published it. + * Taking the code away, the way a repository page offers it: cloned with + * git - every version, as the commits of main - or the version being read + * as a project, with how to run it. The way out every lambda has, offered to + * everybody once its owner published it. */ -export function DownloadMenu({ publicKey, version, root, bytes }: { +export function CodeMenu({ publicKey, version, root, bytes, gitUrl, oldest, newest }: { publicKey: string; version: number; /** The folder the project unpacks into, once the version is known. */ root: string | null; /** How large the zip is, once it is known. */ bytes: number | null; + /** Where it is cloned from. */ + gitUrl: string; + /** The oldest version kept, and the newest - what a clone has as tags. */ + oldest: number; + newest: number; }) { - const said = useSourceT().download; + const t = useSourceT(); + const said = t.download; + const cloning = t.clone; const format = useFormat(); const { state, start } = useDownload(publicKey, version); - const [open, setOpen] = useState(false); - const host = useRef(null); + const folder = root ?? publicKey; + const tag = folder.toLowerCase(); - useEffect(() => { - if (!open) { - return; - } + const words = { copy: said.copy, copied: said.copied }; - const onKey = (event: KeyboardEvent) => event.key === 'Escape' && setOpen(false); + return ( + {cloning.button}} + title={cloning.title} + button="inline-flex h-9 items-center gap-1.5 rounded-full border border-slate-300 px-3.5 text-sm font-medium text-slate-700 hover:border-slate-400 hover:bg-slate-50 dark:border-ink-700 dark:text-slate-200 dark:hover:bg-ink-850" + > +

{cloning.title}

+

{cloning.what(oldest, newest)}

- const onPointer = (event: PointerEvent) => { - if (!host.current?.contains(event.target as Node)) { - setOpen(false); - } - }; +
- document.addEventListener('keydown', onKey); - document.addEventListener('pointerdown', onPointer); + - return () => { - document.removeEventListener('keydown', onKey); - document.removeEventListener('pointerdown', onPointer); - }; - }, [open]); +

{cloning.readOnly}

- const folder = root ?? publicKey; - const tag = folder.toLowerCase(); +
+

{said.title(version)}

+

{said.what}

- return ( -
- - - {open && ( -
-

{said.title(version)}

-

{said.what}

- - - - {state === 'slow' &&

{said.slow}

} - -
-

{said.run}

- -

{said.local}

- - -

{said.container}

- - -

{said.agent}

-
-
- )} -
- ); -} + -/** A command to type, with a way to copy it. */ -function Command({ text }: { text: string }) { - const said = useSourceT().download; - const [copied, setCopied] = useState(false); + {state === 'slow' &&

{said.slow}

} +
- return ( -
- 
-        {text}
-
- -
+
+

{said.run}

+ +

{said.local}

+ + +

{said.container}

+ + +

{said.agent}

+
+ ); } diff --git a/src/Frontend/src/source/Project.tsx b/src/Frontend/src/source/Project.tsx index 3415dc33..2c62c446 100644 --- a/src/Frontend/src/source/Project.tsx +++ b/src/Frontend/src/source/Project.tsx @@ -8,7 +8,7 @@ import { useLanguage, useSourceT } from '../i18n'; import { Link } from '../i18n/links'; import { pageMeta, usePageMeta } from '../meta'; import { CodeView } from './CodeView'; -import { DownloadMenu } from './Download'; +import { CodeMenu } from './Download'; import { useFormat } from './format'; import { ChangesView, OlderVersion, VersionPicker } from './History'; import { LambdaNotice, LicenseChip, StarButton } from './parts'; @@ -244,7 +244,15 @@ function Header({ project, ticketAt, version, tree }: {
- + {source.online && ( {said.project.openApp} diff --git a/src/GenHTTP.Lambda/Api/FeatureResource.cs b/src/GenHTTP.Lambda/Api/FeatureResource.cs index cfa42d3d..b0260429 100644 --- a/src/GenHTTP.Lambda/Api/FeatureResource.cs +++ b/src/GenHTTP.Lambda/Api/FeatureResource.cs @@ -343,8 +343,8 @@ private async ValueTask SaveAsync(string privateKey, strin } internal static FeatureResponse Describe(FeatureInfo feature) - => new(feature.Key, feature.Name, feature.Specification, feature.Change, feature.Base, feature.Newest, feature.Mergeable, feature.Origin, - feature.Created, feature.Modified, feature.Online, feature.Current, feature.Previewed, feature.Path, feature.Revision); + => new(feature.Key, feature.Name, feature.Branch, feature.Specification, feature.Change, feature.Base, feature.Newest, feature.Mergeable, + feature.Origin, feature.Created, feature.Modified, feature.Online, feature.Current, feature.Previewed, feature.Path, feature.Revision); private static FeaturePreviewResponse Describe(FeatureDeployment deployment) => new(deployment.Success, Describe(deployment.Feature), deployment.Diagnostics); diff --git a/src/GenHTTP.Lambda/Api/Git/GitRoutes.cs b/src/GenHTTP.Lambda/Api/Git/GitRoutes.cs new file mode 100644 index 00000000..d57b422b --- /dev/null +++ b/src/GenHTTP.Lambda/Api/Git/GitRoutes.cs @@ -0,0 +1,234 @@ +using GenHTTP.Api.Content; +using GenHTTP.Api.Infrastructure; +using GenHTTP.Api.Protocol; + +using GenHTTP.Lambda.Api.Infrastructure; +using GenHTTP.Lambda.Configuration; +using GenHTTP.Lambda.Infrastructure; +using GenHTTP.Lambda.Services.Git; +using GenHTTP.Lambda.Services.Meta; +using GenHTTP.Lambda.Web; + +using GenHTTP.Modules.Git; +using GenHTTP.Modules.IO; +using GenHTTP.Modules.Redirects; + +using Microsoft.Extensions.Logging; + +namespace GenHTTP.Lambda.Api.Git; + +/// +/// Every lambda as a git repository: at /editor/{privateKey}/{name}.git +/// for whoever holds its editor key, to clone and push to, and at +/// /source/{publicKey}.git for anybody, to clone, while its source is +/// published. +/// +/// +/// The address of the editor and of the published source with .git +/// added, the way repositories are addressed elsewhere: a clone is named +/// after what precedes it. Below the editor's that is any name - the public +/// key is what the platform offers - since it only names the folder, and a +/// clone made before the key changed goes on pushing to the same lambda. Both +/// answer without the .git as well, and opened in a browser they lead +/// to their page. +/// +/// Only the paths git asks for are answered here; anything else below +/// /editor/ and /source/ is the pages', and passes on to them. +/// A repository that is not there is answered as such - never with a page, +/// which git would take for a broken server. +/// +/// What the server writes - a pack, or the answer to a push, which is where +/// the push is applied - is written away from the reactor (see +/// ). Fetching is logged like a download, +/// pushing by the service, which knows what it did. +/// +public sealed class GitRoutes : IHandler +{ + private const string Chosen = "lambda.git.repository"; + + private static readonly string[][] Endpoints = [["info", "refs"], ["git-upload-pack"], ["git-receive-pack"]]; + + private readonly IHandler _git; + + private readonly IGitService _service; + + private readonly IMetaService _meta; + + private readonly ILogger _logger; + + #region Initialization + + public GitRoutes(IGitService service, IMetaService meta, LambdaOptions options, ILogger logger) + { + _service = service; + _meta = meta; + _logger = logger; + + _git = GitServer.Create() + .Repository(request => new ValueTask(request.Properties.TryGet(Chosen, out var chosen) ? chosen : null)) + .Agent("genhttp-lambda") + .MaximumPushSize(options.GitMaxPushBytes) + // read once more rather than held, beyond this, for every clone at once + .ContentCache(16 * 1024 * 1024) + .Build(); + } + + #endregion + + #region Functionality + + public ValueTask PrepareAsync(IServer server) => _git.PrepareAsync(server); + + public async ValueTask HandleAsync(IRequest request) + { + if (Parse(request.Header.Target) is not { } route) + { + return null; + } + + if (route.Skip == 0) + { + // the repository's own address, opened in a browser + return request.Header.Method == RequestMethod.Get || request.Header.Method == RequestMethod.Head + ? await Redirect.To(route.Owner ? $"/editor/{route.Key}" : $"/source/{route.Key}", true).Build().HandleAsync(request) + : null; + } + + var origin = RequestOrigin.Of(request); + + var repository = route.Owner + ? await _service.OpenAsync(route.Key, origin) + : await _service.ReadAsync(route.Key, origin); + + if (repository == null) + { + return Missing(request, route.Owner ? "There is no lambda with this editor key (or it has been deleted)." : "No source is published at this address."); + } + + request.Properties[Chosen] = repository; + + request.Header.Target.Advance(route.Skip); + + var response = await _git.HandleAsync(request); + + if (response == null) + { + return Missing(request, "This is no address of a git repository."); + } + + if (route.Fetching && request.Header.Query.GetEntry("service") == "git-upload-pack") + { + if (route.Owner) + { + _logger.LogInformation("Fetched lambda {Lambda} with git", _meta.PublicKeyOf(route.Key)); + } + else + { + _logger.LogInformation("Fetched source of lambda {Lambda} with git", route.Key); + } + } + + return response.Content is { } content + ? response.Rebuild().Content(new OffloadedContent(content)).Build() + : response; + } + + #endregion + + #region Routing + + /// + /// A request for a repository. + /// + /// Whether it is the editor's, read and written with the editor key + /// The editor key, or the public key of a published source + /// How many segments come before what git asks for - none for the repository's own address + /// Whether git asks what there is, which a clone, a fetch and a push begin with + private sealed record Route(bool Owner, string Key, int Skip, bool Fetching); + + private static Route? Parse(IRequestTarget target) + { + var area = target.Current?.Decode(); + + if (area is not ("editor" or "source") || target.Next(1)?.Decode() is not { Length: > 0 } second) + { + return null; + } + + var rest = new List(); + + for (var offset = 2; target.Next(offset) is { } segment && rest.Count < 4; offset++) + { + rest.Add(segment.Decode()); + } + + if (area == "source") + { + var named = second.EndsWith(".git", StringComparison.Ordinal); + + var key = named ? second[..^4] : second; + + if (named && rest.Count == 0) + { + return new Route(false, key, 0, false); + } + + return Endpoint(rest) is { } endpoint ? new Route(false, key, 2, endpoint == 0) : null; + } + + // below the editor's key: what git asks for, after a name or not + if (Endpoint(rest) is { } direct) + { + return new Route(true, second, 2, direct == 0); + } + + if (rest.Count == 1 && rest[0].EndsWith(".git", StringComparison.Ordinal)) + { + return new Route(true, second, 0, false); + } + + return rest.Count > 1 && Endpoint(rest[1..]) is { } named2 ? new Route(true, second, 3, named2 == 0) : null; + } + + /// + /// Which of the paths git asks for these segments are, if they are one. + /// + private static int? Endpoint(List segments) + { + for (var i = 0; i < Endpoints.Length; i++) + { + if (segments.SequenceEqual(Endpoints[i], StringComparer.Ordinal)) + { + return i; + } + } + + return null; + } + + private static IResponse Missing(IRequest request, string message) + => request.Respond() + .Status(ResponseStatus.NotFound) + .Content(Resource.FromString(message + "\n").Type(ContentType.TextPlain).Build()) + .Build(); + + #endregion + +} + +/// +/// Puts the repositories into the layout of the platform. +/// +public sealed class GitRoutesBuilder(IGitService service, IMetaService meta, LambdaOptions options, ILogger logger) : IHandlerBuilder +{ + private readonly List _concerns = []; + + public GitRoutesBuilder Add(IConcernBuilder concern) + { + _concerns.Add(concern); + return this; + } + + public IHandler Build() => Concerns.Chain(_concerns, new GitRoutes(service, meta, options, logger)); + +} diff --git a/src/GenHTTP.Lambda/Api/Mcp/McpHandler.cs b/src/GenHTTP.Lambda/Api/Mcp/McpHandler.cs index 31cbf5e4..54d62405 100644 --- a/src/GenHTTP.Lambda/Api/Mcp/McpHandler.cs +++ b/src/GenHTTP.Lambda/Api/Mcp/McpHandler.cs @@ -245,7 +245,10 @@ private static JsonObject Introduce(JsonObject parameters) "", "If you can make HTTP requests, the REST API (https://genhttp.dev/api/v1/openapi.json) does the same", "with fewer tokens: download a feature as a zip, edit locally, put it back ('zip -r ../f.zip .', so", - ".lambda/ comes along)." + ".lambda/ comes along).", + "", + "If you can run git, clone the lambda instead (read_lambda's gitUrl) - AGENTS.md in it says how: a", + "commit pushed to main is a version, a branch pushed is a feature - platform_guide, git." ]) }; } diff --git a/src/GenHTTP.Lambda/Api/Mcp/McpTools.cs b/src/GenHTTP.Lambda/Api/Mcp/McpTools.cs index 0122793b..78fdd712 100644 --- a/src/GenHTTP.Lambda/Api/Mcp/McpTools.cs +++ b/src/GenHTTP.Lambda/Api/Mcp/McpTools.cs @@ -556,6 +556,7 @@ private JsonObject Create(JsonObject arguments, string origin) privateKey = lambda.PrivateKey, publicUrl = $"{origin}/lambda/{lambda.PublicKey}/", editorUrl = $"{origin}/editor/{lambda.PrivateKey}", + gitUrl = $"{origin}{LambdaDescription.GitPath(lambda.PrivateKey, lambda.PublicKey)}", view = lambda.View, next = "write_code with deploy: true, and fix what needs fixing with change_code and deploy: true - a new lambda needs no feature. Send .lambda/docs/product.md, .lambda/docs/decisions.md and .lambda/tests/README.md with the code - short for a small app (platform_guide, documentationAndTests). Once people use it, make further changes in a feature (create_feature).", warning = "The editor key cannot be recovered. Give it to the user." @@ -961,6 +962,7 @@ private static bool LinksTo(string code, string publicKey) { feature = feature.Key, feature.Name, + feature.Branch, @base = feature.Base, newest = feature.Newest, mergeable = feature.Mergeable, @@ -1607,6 +1609,8 @@ private JsonObject Read(JsonObject arguments, string origin) lambda.PublicKey, publicUrl = $"{origin}/lambda/{lambda.PublicKey}/", domainUrl = DomainUrl(lambda), + // where it is cloned with git, if git is at hand: AGENTS.md in it says how to work there + gitUrl = $"{origin}{LambdaDescription.GitPath(lambda.PrivateKey, lambda.PublicKey)}", lambda.Tier, lambda.View, limits = Limits(Enum.Parse(lambda.Tier)), @@ -1617,7 +1621,7 @@ private JsonObject Read(JsonObject arguments, string origin) lambda.KeptUntil, // the work under way beside the lambda, which a change should // continue rather than start over where one fits - features = open.Select(f => new { feature = f.Key, f.Name, @base = f.Base, mergeable = f.Mergeable, previewOnline = f.Online, f.Modified }), + features = open.Select(f => new { feature = f.Key, f.Name, f.Branch, @base = f.Base, mergeable = f.Mergeable, previewOnline = f.Online, f.Modified }), workIn = lambda.ActiveVersion != null && feature == null ? "The lambda is online: make changes in a feature - create_feature, or continue one listed under features." : null, @@ -1998,6 +2002,16 @@ private JsonObject Guide(string origin) => McpProtocol.Say(new { ok = true, preferTheApi = "If you can make HTTP requests, the REST API at https://genhttp.dev/api/v1/openapi.json does the same as these tools and costs fewer tokens, because files are sent directly. GET /api/v1/lambdas/{privateKey}/versions/{version}/zip downloads a version; a feature is downloaded from and put back to /api/v1/lambdas/{privateKey}/features/{feature}/zip (GET, PUT) - so edit locally and push as often as it takes. POST /api/v1/lambdas/{privateKey}/versions/zip saves a zip as a new version. The zip holds the documentation and tests in .lambda/, a hidden folder: zip the contents with it ('zip -r ../feature.zip .', not '*'), or the version you save has none. Every endpoint that saves takes ?deploy=true. Many environments cannot reach it; then use these tools.", + git = new + { + what = "Every lambda is a git repository as well: read_lambda gives its gitUrl, /editor/{privateKey}/{publicKey}.git - the editor key is in it, so keep it to yourself and to the user. It holds the project the lambda is exported as: Project.cs is lambda.cs (the body of BuildAsync()), the other .cs files are its own, assets/ what it ships, docs/ and tests/ its .lambda/docs/ and .lambda/tests/ - and around them the platform's files, which make it run with dotnet run and are no part of the lambda. AGENTS.md in it says how to work there.", + when = "If you can run git, clone the lambda and work in the clone: it builds and runs with dotnet, and a push costs no tokens for the files it sends.", + versions = "Every commit pushed to main becomes the next version, tagged v1, v2 and so on: main moves forward one commit after another - never rewritten, no merge commits (git pull --rebase). The first line of a commit's message is the version's change, the rest its specification. A push to main is compiled first and refused when it does not compile; git push -o deploy puts the newest version online with it.", + features = "Every other branch is a feature: pushing one starts it - from the newest version among its commits, with a copy of the lambda's data - or replaces its files, and puts its preview online at the address the push answers with. git push origin :main merges it as a version per commit, -o merge with its last push merges it as one (-o deploy puts that online), and deleting the branch deletes the feature. A feature started anywhere else is a branch there too.", + refused = "A push that changes one of the platform's files (Program.cs, the .csproj, Platform/, AGENTS.md and the rest) or adds a file that has no place in a lambda is refused, and so is anything a save would refuse - the remote: lines of the push say why.", + published = "A published source is cloned read only from /source/{publicKey}.git: the same versions, without the features.", + data = "The data - the database, the workspace, the secrets - is never in the repository. Reach it with these tools or the REST API." + }, whatALambdaIs = "C# that returns a GenHTTP handler, served at /lambda/{publicKey}/. No Main and no project: the snippet is the program.", lifecycle = new { @@ -2246,7 +2260,7 @@ private JsonObject Guide(string origin) => McpProtocol.Say(new tree = "VirtualTree.Create().Add(\"app.css\", Resource.FromString(css).Type(new ContentType(\"text/css\"))) builds a tree in memory.", singlePage = "Content.From(Resource.FromString(html).Type(new ContentType(\"text/html; charset=utf-8\")))" }, - takingItAway = "GET /api/v1/lambdas/{privateKey}/export returns the newest version as a standalone zipped .NET 10 project with a Dockerfile: Program.cs hosts it, Project.cs is lambda.cs, the other files keep their code, the documentation and tests go to docs/ and tests/, and Platform/ stands in for Workspace, Assets, Secret, Database and the implicit imports. The database comes along as a SQLite file. It needs nothing from this platform. Worth telling the user.", + takingItAway = "GET /api/v1/lambdas/{privateKey}/export returns the newest version as a standalone zipped .NET 10 project with a Dockerfile: Program.cs hosts it, Project.cs is lambda.cs, the other files keep their code, the documentation and tests go to docs/ and tests/, and Platform/ stands in for Workspace, Assets, Secret, Database and the implicit imports. The database comes along as a SQLite file. It needs nothing from this platform. A clone of its gitUrl is the same project with every version, without the data. Worth telling the user.", importedForYou = ModuleCatalog.Imports, network = "A lambda can make outbound calls with HttpClient and sockets. System.Net.Http and System.Net.Sockets are not imported by default, so write the full type name or add a using. It runs in the shared server process, so give requests a timeout.", sayWhy = new @@ -2273,7 +2287,7 @@ private JsonObject Guide(string origin) => McpProtocol.Say(new neverPublished = "The data - the database, the workspace and the values of the secrets - what the owner asked for in their words (the specification), and anything about who uses it: traffic, logs, visitors.", thenPublic = "Once published, every version is public, the ones saved before included. Keys, passwords and personal data never go into files anyway; for a published lambda it matters at once.", licenses = SourceLicenses.All.Select(l => new { l.Id, l.Name, kind = l.Kind.ToString() }), - startingFromOne = "A published source downloads as a .NET project, not as a lambda. To make a lambda of it: Project.cs holds the snippet as the body of Build() with the types after the class - put the body back into lambda.cs with those types below it; the other .cs files stay as they are; assets/ is what goes at the root of the version; docs/ and tests/ go into .lambda/docs/ and .lambda/tests/; Platform/, Program.cs, the .csproj and the Dockerfile are the platform's and stay behind. Keep to the terms of its LICENSE." + startingFromOne = "A published source downloads as a .NET project, not as a lambda - or is cloned with git from /source/{publicKey}.git. To make a lambda of it with git: create_lambda, clone its gitUrl, copy Project.cs, the other .cs files, assets/, docs/ and tests/ over its own, commit and push. Without git: Project.cs holds the snippet as the body of Build() or BuildAsync() with the types after the class - put the body back into lambda.cs with those types below it; the other .cs files stay as they are; assets/ is what goes at the root of the version; docs/ and tests/ go into .lambda/docs/ and .lambda/tests/; Platform/, Program.cs, the .csproj and the Dockerfile are the platform's and stay behind. Keep to the terms of its LICENSE." }, afterDeploying = new { diff --git a/src/GenHTTP.Lambda/Api/Model/Features.cs b/src/GenHTTP.Lambda/Api/Model/Features.cs index 31c653e1..1d666ec4 100644 --- a/src/GenHTTP.Lambda/Api/Model/Features.cs +++ b/src/GenHTTP.Lambda/Api/Model/Features.cs @@ -48,6 +48,7 @@ public sealed record MergeFeatureRequest(bool? Deploy = null, string? Specificat /// A feature: a change being worked on beside the lambda. /// /// What it is addressed by in this API, and where its preview answers +/// The branch it is in the lambda's git repository - given when it starts, and kept when it is renamed /// The version it is based on /// The newest version of the lambda /// Whether it can be merged: it is based on the newest version @@ -59,6 +60,7 @@ public sealed record MergeFeatureRequest(bool? Deploy = null, string? Specificat public sealed record FeatureResponse( string Key, string Name, + string Branch, string? Specification, string? Change, int Base, diff --git a/src/GenHTTP.Lambda/Api/Model/Lambdas.cs b/src/GenHTTP.Lambda/Api/Model/Lambdas.cs index 81fc1aed..7f041cc7 100644 --- a/src/GenHTTP.Lambda/Api/Model/Lambdas.cs +++ b/src/GenHTTP.Lambda/Api/Model/Lambdas.cs @@ -31,6 +31,10 @@ public sealed record UpdateLambdaRequest(string? PublicKey, string? View = null) /// Whether it actually answers at that domain - it has one, and its tier includes it /// Where to link to it: its domain while that is served, its path otherwise /// How its editor opens for somebody who has not chosen a view of their own: Full or Simple +/// +/// Where it is cloned with git, and pushed to: its versions are the commits of main, its features the other +/// branches. Holds the editor key, like the editor's path +/// public sealed record LambdaResponse( string PublicKey, string PrivateKey, @@ -47,7 +51,8 @@ public sealed record LambdaResponse( string? Domain, bool DomainServed, string Address, - string View + string View, + string GitPath ); /// @@ -76,9 +81,16 @@ public static class LambdaDescription lambda.Domain, Serves(lambda.Tier, lambda.Domain), Address(lambda.PublicKey, lambda.Tier, lambda.Domain), - lambda.View + lambda.View, + GitPath(lambda.PrivateKey, lambda.PublicKey) ); + /// + /// Where whoever holds the editor key clones the lambda: the editor's + /// path, with a name for the folder a clone makes. + /// + public static string GitPath(string privateKey, string publicKey) => $"/editor/{privateKey}/{publicKey}.git"; + /// /// Whether a lambda of this tier may answer at a domain of its own. /// diff --git a/src/GenHTTP.Lambda/Api/Model/Sources.cs b/src/GenHTTP.Lambda/Api/Model/Sources.cs index 0880326c..2bbe8404 100644 --- a/src/GenHTTP.Lambda/Api/Model/Sources.cs +++ b/src/GenHTTP.Lambda/Api/Model/Sources.cs @@ -76,6 +76,7 @@ DateTime PublishedAt /// When the lambda was made /// Every version kept, newest first: what each changed, never what was asked for /// What starring it has to be sent with, good for a day from a second on +/// Where it is cloned with git - every version, read only public sealed record SourceProjectResponse( SourceEntryResponse Source, string? Author, @@ -83,11 +84,12 @@ public sealed record SourceProjectResponse( int? ActiveVersion, DateTime Created, IReadOnlyList Versions, - string StarTicket + string StarTicket, + string GitPath ); /// What it changed, in a line -/// Which door it came through: template, api or agent +/// Which door it came through: template, api, agent or git /// Whether this is the version online /// Where it is downloaded as a project public sealed record SourceVersionResponse(int Version, DateTime Created, string? Change, string? Origin, bool Online, string ZipPath); diff --git a/src/GenHTTP.Lambda/Api/SourceResource.cs b/src/GenHTTP.Lambda/Api/SourceResource.cs index c71ab18c..d2287904 100644 --- a/src/GenHTTP.Lambda/Api/SourceResource.cs +++ b/src/GenHTTP.Lambda/Api/SourceResource.cs @@ -101,7 +101,7 @@ public SourceProjectResponse Get(string publicKey) ZipPath(publicKey, v.Version))); return new SourceProjectResponse(entry, project.Author, SourceLicenses.Holder(project.Author, publicKey), project.ActiveVersion, - project.Created, [.. versions], sources.IssueTicket(publicKey)); + project.Created, [.. versions], sources.IssueTicket(publicKey), $"/source/{publicKey}.git"); } #endregion diff --git a/src/GenHTTP.Lambda/Application.cs b/src/GenHTTP.Lambda/Application.cs index cc77b5a8..461a3e48 100644 --- a/src/GenHTTP.Lambda/Application.cs +++ b/src/GenHTTP.Lambda/Application.cs @@ -1,6 +1,7 @@ using GenHTTP.Api.Content; using GenHTTP.Api.Infrastructure; +using GenHTTP.Lambda.Api.Git; using GenHTTP.Lambda.Api.Infrastructure; using GenHTTP.Lambda.Api.Mcp; using GenHTTP.Lambda.Configuration; @@ -11,6 +12,7 @@ using GenHTTP.Lambda.Services.Diagnostics; using GenHTTP.Lambda.Services.Execution; using GenHTTP.Lambda.Services.Features; +using GenHTTP.Lambda.Services.Git; using GenHTTP.Lambda.Services.Building; using GenHTTP.Lambda.Services.Data; using GenHTTP.Lambda.Services.Databases; @@ -158,6 +160,10 @@ private static ServiceProvider BuildServices(LambdaOptions options, ILoggerFacto services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); @@ -235,7 +241,11 @@ private static IHandler BuildPlatform(IServiceProvider services, LambdaOptions o // one path, for agents rather than for browsers .Add("mcp", new McpHandlerBuilder(services.GetRequiredService(), options.McpOrigins)) .Add("lambda", lambdas) - .Add("features", previews); + .Add("features", previews) + // the repositories below the editor and the published sources, asked + // before the pages that answer every other path there + .Add(new GitRoutesBuilder(services.GetRequiredService(), services.GetRequiredService(), options, + services.GetRequiredService>())); // Ahead of the application, so a miss here is a 404 rather than the // index page: a named route answers for itself and never falls through diff --git a/src/GenHTTP.Lambda/Configuration/LambdaOptions.cs b/src/GenHTTP.Lambda/Configuration/LambdaOptions.cs index e08f4249..f362f208 100644 --- a/src/GenHTTP.Lambda/Configuration/LambdaOptions.cs +++ b/src/GenHTTP.Lambda/Configuration/LambdaOptions.cs @@ -268,6 +268,19 @@ public sealed record LambdaOptions /// public long SourceCacheBytes { get; init; } = 2048L * 1024 * 1024; + /// + /// How large a single git push to a lambda may be. + /// + /// + /// A push is read into memory and unpacked there before anything is + /// decided about it, which is the git module's way and what makes it + /// atomic - so this protects the process, like a buffer, rather than + /// being a promise of a tier: what a version may hold is checked after + /// it was unpacked, against the tier of the lambda. Room for the first + /// push of a premium lambda that ships all the assets it may. + /// + public long GitMaxPushBytes { get; init; } = 256L * 1024 * 1024; + /// /// Requests per second a single client may send to the lambda routes. /// @@ -535,6 +548,13 @@ public sealed record LambdaOptions /// public string SourceDirectory => Path.Combine(DataDirectory, "sources"); + /// + /// Where the commits a lambda is read as with git are kept, one folder per + /// lambda: the commits, the files they hold, and which of them is which + /// version and feature. + /// + public string GitDirectory => Path.Combine(DataDirectory, "git"); + /// /// Where the key the secrets are sealed with is kept, when /// does not give one. @@ -608,6 +628,7 @@ public static LambdaOptions FromEnvironment() MaxVersions = ReadInt("LAMBDA_MAX_VERSIONS", defaults.MaxVersions), MaxFeatures = ReadInt("LAMBDA_MAX_FEATURES", defaults.MaxFeatures), SourceCacheBytes = ReadLong("LAMBDA_SOURCE_CACHE_BYTES", defaults.SourceCacheBytes), + GitMaxPushBytes = ReadLong("LAMBDA_GIT_MAX_PUSH_BYTES", defaults.GitMaxPushBytes), RateLimit = ReadInt("LAMBDA_RATE_LIMIT", defaults.RateLimit), MaxConcurrency = ReadInt("LAMBDA_MAX_CONCURRENCY", defaults.MaxConcurrency), ExecutionTimeout = TimeSpan.FromSeconds(ReadInt("LAMBDA_EXECUTION_TIMEOUT_SECONDS", (int)defaults.ExecutionTimeout.TotalSeconds)), diff --git a/src/GenHTTP.Lambda/Data/Entities/DeploymentEntity.cs b/src/GenHTTP.Lambda/Data/Entities/DeploymentEntity.cs index bebf916d..430ee964 100644 --- a/src/GenHTTP.Lambda/Data/Entities/DeploymentEntity.cs +++ b/src/GenHTTP.Lambda/Data/Entities/DeploymentEntity.cs @@ -51,6 +51,9 @@ public static class VersionOrigins /// Through the MCP endpoint, by an agent. public const string Agent = "agent"; + /// Pushed to the lambda's repository with git, a commit each. + public const string Git = "git"; + /// By the operator, through the administration panel. public const string Admin = "admin"; diff --git a/src/GenHTTP.Lambda/Data/Entities/FeatureEntity.cs b/src/GenHTTP.Lambda/Data/Entities/FeatureEntity.cs index bc39ab9f..48a8b144 100644 --- a/src/GenHTTP.Lambda/Data/Entities/FeatureEntity.cs +++ b/src/GenHTTP.Lambda/Data/Entities/FeatureEntity.cs @@ -28,6 +28,16 @@ public sealed class FeatureEntity /// public required string Name { get; set; } + /// + /// The branch it is in the lambda's git repository, unique per lambda. + /// + /// + /// Given once, when it starts - the branch that was pushed, or its name + /// as git allows it - and kept when it is renamed, since a clone knows + /// the branch by this. + /// + public required string Branch { get; set; } + /// /// What the user wants from it and why, which the version it is merged /// into keeps. diff --git a/src/GenHTTP.Lambda/Data/LambdaDbContext.cs b/src/GenHTTP.Lambda/Data/LambdaDbContext.cs index 4be836f0..aff9c42c 100644 --- a/src/GenHTTP.Lambda/Data/LambdaDbContext.cs +++ b/src/GenHTTP.Lambda/Data/LambdaDbContext.cs @@ -208,6 +208,7 @@ protected override void OnModelCreating(ModelBuilder builder) features.Property(f => f.LambdaId).HasColumnName("lambda_id"); features.Property(f => f.Key).HasColumnName("key"); features.Property(f => f.Name).HasColumnName("name"); + features.Property(f => f.Branch).HasColumnName("branch"); features.Property(f => f.Specification).HasColumnName("specification"); features.Property(f => f.Change).HasColumnName("change"); features.Property(f => f.BaseVersion).HasColumnName("base_version"); @@ -221,6 +222,7 @@ protected override void OnModelCreating(ModelBuilder builder) features.HasIndex(f => f.Key).IsUnique(); features.HasIndex(f => f.LambdaId); + features.HasIndex(f => new { f.LambdaId, f.Branch }).IsUnique(); features.HasOne(f => f.Lambda) .WithMany() diff --git a/src/GenHTTP.Lambda/Data/Migrations/V19__FeatureBranches.sql b/src/GenHTTP.Lambda/Data/Migrations/V19__FeatureBranches.sql new file mode 100644 index 00000000..fd62fda1 --- /dev/null +++ b/src/GenHTTP.Lambda/Data/Migrations/V19__FeatureBranches.sql @@ -0,0 +1,18 @@ +-- The branch a feature is in the lambda's git repository. +-- +-- Every lambda can be cloned with git: its versions are the commits of main, +-- and each feature is a branch. A branch has a name that git clients keep - +-- in their remote, in their checkouts - so it is decided once, when the +-- feature starts, and kept: renaming a feature changes what the editor calls +-- it, never where it is in a clone. A feature started by pushing a branch is +-- called what the branch was called; one started anywhere else gets its name +-- in the words git allows. +-- +-- The features open now never had one, and get one from their id: they are +-- few, and nobody has cloned them yet. + +ALTER TABLE features ADD COLUMN branch TEXT NULL; + +UPDATE features SET branch = 'feature-' || id WHERE branch IS NULL; + +CREATE UNIQUE INDEX ix_features_branch ON features (lambda_id, branch); diff --git a/src/GenHTTP.Lambda/GenHTTP.Lambda.csproj b/src/GenHTTP.Lambda/GenHTTP.Lambda.csproj index 5721f9f7..902ad322 100644 --- a/src/GenHTTP.Lambda/GenHTTP.Lambda.csproj +++ b/src/GenHTTP.Lambda/GenHTTP.Lambda.csproj @@ -35,6 +35,8 @@ GenHTTP.Full.Ioxide, which only carries it by way of GenHTTP.Testing --> + + diff --git a/src/GenHTTP.Lambda/Infrastructure/OffloadedContent.cs b/src/GenHTTP.Lambda/Infrastructure/OffloadedContent.cs new file mode 100644 index 00000000..a868ec42 --- /dev/null +++ b/src/GenHTTP.Lambda/Infrastructure/OffloadedContent.cs @@ -0,0 +1,94 @@ +using System.Buffers; +using System.IO.Pipelines; + +using GenHTTP.Api.Protocol; + +namespace GenHTTP.Lambda.Infrastructure; + +/// +/// The content of a response, written away from the reactor and copied to +/// the connection on it. +/// +/// +/// For content whose writing is the work: a git server packs what a clone +/// asks for - reading, hashing and compressing every file - and applies a +/// push - unpacking it and compiling what it holds - while it writes its +/// answer. Written in place, that is seconds in which every other connection +/// of the reactor waits. So the content is written on the pool, in one hop +/// (see ), into a pipe, and the reactor only copies what +/// comes out of it to the connection - flushed as it comes, so a line a push +/// says along the way reaches whoever pushed as it is said. +/// +/// The pipe holds a megabyte at most: a client reading slowly holds up the +/// writing, not the memory of the server. +/// +public sealed class OffloadedContent(IResponseContent content) : IResponseContent +{ + private static readonly PipeOptions Options = new(pauseWriterThreshold: 1024 * 1024, resumeWriterThreshold: 512 * 1024); + + public ulong? Length => content.Length; + + public ContentType? Type => content.Type; + + public ReadOnlyMemory? Encoding => content.Encoding; + + public ValueTask CalculateChecksumAsync() => content.CalculateChecksumAsync(); + + public async ValueTask WriteAsync(IResponseSink sink) + { + var pipe = new Pipe(Options); + + var writing = Offload.Run(async () => + { + try + { + await using var stream = pipe.Writer.AsStream(leaveOpen: true); + + await content.WriteAsync(new Sink(pipe.Writer, stream)); + + await pipe.Writer.CompleteAsync(); + } + catch (Exception e) + { + await pipe.Writer.CompleteAsync(e); + } + }); + + try + { + while (true) + { + var read = await pipe.Reader.ReadAsync(); + + foreach (var segment in read.Buffer) + { + await sink.Stream.WriteAsync(segment); + } + + await sink.Stream.FlushAsync(); + + pipe.Reader.AdvanceTo(read.Buffer.End); + + if (read.IsCompleted) + { + break; + } + } + } + finally + { + // a client that went away leaves nobody to read, which the writing notices + await pipe.Reader.CompleteAsync(); + + await writing; + } + } + + private sealed class Sink(IBufferWriter writer, Stream stream) : IResponseSink + { + public IBufferWriter Writer => writer; + + public Stream Stream => stream; + } + +} diff --git a/src/GenHTTP.Lambda/Resources/Export/Agents.md.txt b/src/GenHTTP.Lambda/Resources/Export/Agents.md.txt new file mode 100644 index 00000000..36e630c9 --- /dev/null +++ b/src/GenHTTP.Lambda/Resources/Export/Agents.md.txt @@ -0,0 +1,226 @@ +# {lambda} + +This repository is a lambda on GenHTTP Lambda ({home}/): a C# web service +that answers at {address}. It is also an ordinary .NET project - `dotnet run` +serves it at http://localhost:8080/ - and every commit of `main` is one of +the lambda's versions. + +This file is for the coding agent working on it. The platform writes it, and +it is the same in every commit until the platform changes it. + +## What is what + +| Here | On the platform | +|---|---| +| `Project.cs` | `lambda.cs`: the body of `BuildAsync()` is the code that returns the app's handler, the types below the class are its own | +| the other `*.cs` here | its other code files: types, no statements | +| `assets/` | what it ships and serves - the front end, its migrations, pictures | +| `docs/` | its documentation: `product.md`, `decisions.md` and more pages | +| `tests/` | how it is tested: `README.md`, the scripts and their data | +| `Program.cs`, `{project}.csproj`, `Platform/`, `Dockerfile`, `.gitignore`, `.dockerignore`, `AGENTS.md`, `CLAUDE.md`, `LICENSE` | the platform's: they make the lambda a project that runs here, and are no part of it | + +Change the lambda's files only. A push that changes, adds or removes one of +the platform's is refused, since it would change nothing of the lambda, and +so is a file anywhere else - a README at the root, an editor's settings: keep +those out of commits (`.git/info/exclude`). Leave `class Project` as it is +and work in the body of `BuildAsync()`; a helper goes into a type of its own. + +## Running it here + + dotnet run # then open http://localhost:8080/ + +`Platform/` stands in for what the platform provides: + +- `Workspace` is the folder `workspace/` +- `Assets` is `assets/`, copied beside the program when it is built +- `Secret.Read("NAME")` reads the environment variable `NAME` +- `Database.GetConnection()` opens `database/database.db`, made empty on first use + +None of it is the lambda's data on the platform, which is never in the +repository (see "Data"). + +## Versions: main + +Every commit pushed to `main` becomes the next version of the lambda, in the +order they were made, and is tagged `v1`, `v2`, ... by the platform. A +version never changes once it is saved, so: + +- `main` moves forward one commit after another: no `--force`, no merge + commits. When the platform saved a version meanwhile - the owner in the + editor, another agent - the push is refused; `git pull --rebase` and push + again. +- One commit per thing the user asked for, not one per attempt: squash + before you push. Each commit is a version the owner reads in the history, + and the oldest are removed once a lambda has many. +- The first line of the message is what the version changes, in a line, for + the owner: "Adds a leaderboard that keeps the ten best scores", not + "Update Project.cs". The rest of the message, if any, is kept as what the + user wanted and why, in their words where you can. +- A push to `main` is compiled first, and refused when it does not compile: + the errors come back as `remote:` lines, with the file and line. +- A version is not online until it is deployed: `git push -o deploy` puts the + newest online with the push. Without it, the owner or you deploy it later, + in the editor or through the API. + +A lambda nobody uses yet is written on `main`. A lambda that is online, or in +use, is changed in a feature. + +## Features: branches + +Every branch other than `main` is a feature - a "draft" in the editor: a +change worked on beside the lambda, with an address and a copy of the +lambda's data of its own, while visitors keep getting what is online. + + git switch -c leaderboard + # change, commit + git push -u origin leaderboard # starts the feature, its preview online + +- A new branch starts a feature from the newest version among its commits, + with a copy of the lambda's data as it is then. Pushing the branch again + replaces the feature's files; force pushes are fine here. +- Every push of a branch puts the feature's preview online, at the address + the push answers with (`/features/.../`), against its copy of the data. Try + it there and run the tests against it - never against the live lambda. +- `git push origin leaderboard:main` merges it: each of its commits becomes a + version, and the feature is gone with its preview and its copy of the data + (the lambda keeps its own). Add `-o deploy` to put the newest online. Or + push its last commit with `-o merge` - it becomes one version - and + `-o deploy` with it. Then `git switch main`, `git pull` and + `git fetch --prune`. +- Only a feature that holds the newest version is merged, so it cannot undo + a version saved since it began. When `main` moved on, bring it in - + `git rebase origin/main`, or merge `main` into the branch and merge with + `-o merge`, since `main` takes no merge commits - push, and merge then. + Nothing merges or rebases for you. +- `git push origin --delete leaderboard` throws the feature away. +- A feature started in the editor, or by another agent, is a branch here as + well: `git fetch` shows it. +- Options travel with a push that changes something: git sends nothing when + everything is up to date. To put a version online that is already pushed, + deploy it in the editor or through the API (see "Beyond git"). + +## What the platform answers + +Every push is answered with `remote:` lines: the versions or the feature it +became, whether it compiles, what went online and where. Read them - and +when a push is refused, the line says why. + +## A lambda + +- `Project.cs`'s `BuildAsync()` returns an `IHandler` or a builder of one: + `Inline.Create()`, `Layout.Create()`, `Content.From(...)`, + `Websocket.Functional()`. Routes cannot return anonymous types: declare a + record. The GenHTTP modules are imported in every file already + (`Platform/Usings.cs`), as on the platform. +- In other files, `Assets` is the Files module's type of that name: use + `LambdaEnvironment.Assets` there. `Workspace`, `Secret` and `Database` work + in every file. +- Links are relative: `"api/items"`, `"app.css"`, `"./"` - no leading slash, + never `/lambda/{lambda}/` or a full address. The same lambda answers at + `/lambda/{lambda}/`, at the root of a domain of its own, and as a feature at + `/features/.../`; a path starting with `/` leaves it, and from a preview + reaches the live lambda and its data. A page a level deeper needs + `"../api/items"`. `Redirect.To(...)` takes relative paths too. The only + full addresses in a page are `og:image`, `og:url` and the canonical link, + which name it for whoever reads it elsewhere. +- Never wait for a task: `.Result`, `.Wait()`, `.GetAwaiter().GetResult()`, + `Task.WaitAll`, `Task.WaitAny` and `SemaphoreSlim.Wait()` are refused - a + request runs on one thread per core, and a task finishes on the thread that + would be waiting for it. Await it, or call the synchronous method. +- Refused as well: reflection, `dynamic`, processes, the environment, the + file system outside `Workspace`, and anything else that reaches the host. + All lambdas share one process. Code that compiles here can still be refused + there - the push says so. +- Never poll: a page does not fetch on a timer to see whether something + changed. Push the change from the server - server-sent events + (`EventSource.Create()`, `new EventSource("events")` in the browser) when + the page only listens, a websocket when it talks back. + +## Data + +A version is the program - the code and the assets, the front end included. +Data is what the program keeps: the database, the workspace and the secrets. +It belongs to the lambda, is shared by every version, and no deploy, rollback +or merge touches it. A feature works on a copy. It is never in the +repository. + +- Records - entries, accounts, orders, votes - go in the database: a SQLite + file of the lambda's own, switched on once (see "Beyond git"). Its schema + is SQL migrations in `assets/migrations/` (`V1__Create_items.sql`), applied + by Evolve as the lambda starts: + + using (var connection = Database.GetConnection()) + { + new Evolve(connection) { Locations = [Assets.Root + "migrations"], IsEraseDisabled = true }.Migrate(); + } + + Never edit a migration that was applied - add the next one - and never + `EnsureCreated`, `Migrate` or Entity Framework's migrations, which are + refused. Keep the records with a `DbContext` of Entity Framework Core on the + connection it is handed - `new Records(Database.GetConnection())`, configured + with `options.UseSqlite(connection, contextOwnsConnection: true)` - one per + request, synchronously: `ToList`, `SaveChanges`, never their `Async` forms. + Every version reads the same database, and a rollback runs an older version + against a newer schema: change it additively where that costs little. +- Files users upload, and large data that is no program, go in the workspace + (`Workspace.WriteBytes(...)`), never in `assets/`: assets are read only and + replaced with every version. +- API keys and passwords are secrets, never code, assets or commits - a key + in a commit stays in every clone for good. Read one with + `Secret.Read("NAME")`, have secrets switched on, and let the owner enter the + value in the editor; here, set the environment variable. + +## Documentation and tests + +`docs/product.md` says what the app is - in a sentence or two first, which the +editor shows - who it is for and what they do with it, in the user's terms. +`docs/decisions.md` holds the technical decisions and why. `tests/README.md` +says how the app is tested: what has to keep working and how each is checked, +with the scripts beside it (`node tests/smoke.mjs
`). Read them +before changing the lambda, and update what a change affects in the same +commit. In proportion: a small lambda gets a short paragraph a page and one +quick check, not a test suite; they grow with the app. Run the tests against +a feature's preview before merging it. Write them in the user's language. + +## Pages meant to be found + +A page meant to be found or shared - a website, a shop, a landing page - gets +a ``, a meta description, `<html lang>`, a favicon as an asset linked +relatively, and a social preview (`og:title`, `og:description`, `og:image` +with a full address), with what it is about in the HTML as served rather than +only in what a script renders. A tool for a few people gets a title and +nothing more. + +## The link back + +We ask for a small "Made with GenHTTP Lambda" line at the foot of the pages +visitors see: a link to {home}/ on a lambda without a domain of its own, +plain text on one with a domain. The name is the whole of the link. Say that +you added it; leave it out or take it out when the user does not want it, and +note that in `docs/decisions.md`. Do not add one to a lambda that has none +unless the user asks. + +## Beyond git + +The repository holds the program. The rest - the data, the secrets, the logs, +what is online, the features' previews - is reached through the platform's +API (`{home}/api/v1/scalar/`) or its MCP tools (`{home}/mcp`, `platform_guide` +first), with the lambda's editor key: the segment after `/editor/` in the +address you cloned from (`git remote get-url origin`). + +- `GET {home}/api/v1/lambdas/<key>/logs` - what it has been doing, its + errors included; `.../features/<feature>/logs` for a preview +- `PUT {home}/api/v1/lambdas/<key>/data/database` - switches the database on; + `.../data/secrets` the secrets +- `GET {home}/api/v1/lambdas/<key>/secrets` - which secrets the code reads and + which have no value yet +- `POST {home}/api/v1/lambdas/<key>/deployment/start` - puts a version online + +Whoever has the editor key can change and delete the lambda: keep it out of +files, commits and logs. + +A clone of `{home}/source/{lambda}.git` is the published source, and read +only. To build on it, create a lambda of your own ({home}/ship), clone that, +copy the lambda's files - `Project.cs`, the other `*.cs`, `assets/`, `docs/`, +`tests/` - over its own and push; the platform's files stay as your clone has +them. Keep to the terms of `LICENSE`. diff --git a/src/GenHTTP.Lambda/Resources/Export/Claude.md.txt b/src/GenHTTP.Lambda/Resources/Export/Claude.md.txt new file mode 100644 index 00000000..8f243734 --- /dev/null +++ b/src/GenHTTP.Lambda/Resources/Export/Claude.md.txt @@ -0,0 +1,3 @@ +This repository is a lambda on GenHTTP Lambda. How to work on it: + +@AGENTS.md diff --git a/src/GenHTTP.Lambda/Services/Deployment/ProjectPacker.cs b/src/GenHTTP.Lambda/Services/Deployment/ProjectPacker.cs index 2be3c63d..f47a9d14 100644 --- a/src/GenHTTP.Lambda/Services/Deployment/ProjectPacker.cs +++ b/src/GenHTTP.Lambda/Services/Deployment/ProjectPacker.cs @@ -3,11 +3,6 @@ using System.Reflection; using System.Text; -using Microsoft.CodeAnalysis; -using Microsoft.CodeAnalysis.CSharp; -using Microsoft.CodeAnalysis.CSharp.Syntax; -using Microsoft.CodeAnalysis.Text; - using GenHTTP.Api.Content; using GenHTTP.Lambda.Services.Databases; @@ -154,15 +149,15 @@ public static void Pack(ExportedLambda lambda, IReadOnlyList<LambdaFile> files, { var name = Identifier(lambda.PublicKey); - var snippet = SourceBuilder.ParseSnippet(files.FirstOrDefault(f => f.Name == LambdaSource.EntryName)?.Code ?? string.Empty); + var snippet = files.FirstOrDefault(f => f.Name == LambdaSource.EntryName)?.Code ?? string.Empty; - var awaits = Awaits(snippet); + var awaits = ProjectSnippet.Awaits(snippet); var assets = files.Where(f => f.IsAsset).ToList(); var context = files.Where(f => f.IsContext).ToList(); - var folders = context.Select(f => Outside(f.Name).Split('/')[0]).Distinct().Order(StringComparer.Ordinal).ToList(); + var folders = context.Select(f => ProjectPaths.Of(f.Name).Split('/')[0]).Distinct().Order(StringComparer.Ordinal).ToList(); var code = files.Where(f => f.IsCode).ToList(); @@ -178,22 +173,22 @@ public static void Pack(ExportedLambda lambda, IReadOnlyList<LambdaFile> files, { Write(archive, $"{name}/{name}.csproj", Csproj(assets.Count > 0, folders, data, entities, evolve)); Write(archive, $"{name}/Program.cs", Program(lambda, name, awaits, folders, data ? database != null : null)); - Write(archive, $"{name}/Project.cs", Project(snippet, awaits)); + Write(archive, $"{name}/{ProjectPaths.Snippet}", ProjectSnippet.ForExport(snippet)); foreach (var file in files.Where(f => f.IsCode && f.Name != LambdaSource.EntryName)) { - Write(archive, $"{name}/{Capitalize(file.Name)}", file.Code); + Write(archive, $"{name}/{ProjectPaths.Of(file.Name)}", file.Code); } // assets keep their folders, because the code that serves them names those folders foreach (var file in assets) { - Write(archive, $"{name}/assets/{file.Name}", file.Bytes); + Write(archive, $"{name}/{ProjectPaths.Of(file.Name)}", file.Bytes); } foreach (var file in context) { - Write(archive, $"{name}/{Outside(file.Name)}", file.Bytes); + Write(archive, $"{name}/{ProjectPaths.Of(file.Name)}", file.Bytes); } Write(archive, $"{name}/Platform/Usings.cs", Usings(data, entities, evolve)); @@ -232,6 +227,72 @@ public static void Pack(ExportedLambda lambda, IReadOnlyList<LambdaFile> files, } } + /// <summary> + /// The files of the project a lambda's git repository holds: the lambda's + /// own where <see cref="ProjectPaths"/> puts them, and the platform's around them. + /// </summary> + /// <remarks> + /// The export as a repository needs it. What the platform puts around the + /// lambda is the same for every commit, whatever the lambda does - only + /// a newer platform, a new address or a published source change it - so a + /// push that leaves it alone is right however much it changed, and + /// somebody who changes it has changed nothing of the lambda. That is why + /// the project references SQLite, Entity Framework and Evolve whether + /// the code uses them or not, keeps the documentation and the tests out + /// of the build whether there are any or not, and makes the snippet + /// asynchronous whether it awaits anything or not: the commit that starts + /// to is somebody's push, and must not need a project of its own. + /// + /// There is no data in it, as in a published source, and nothing that + /// only the owner may know - the same commits are read by anybody once + /// the source is published. AGENTS.md tells an agent how to work in it, + /// and CLAUDE.md points the agent that reads that file instead to it. + /// </remarks> + public static IReadOnlyList<ProjectFile> Repository(RepositoryProject project, IReadOnlyList<LambdaFile> files) + { + var name = Identifier(project.PublicKey); + + var result = new List<ProjectFile> + { + Text($"{name}.csproj", Csproj(true, ["docs", "tests"], true, true, true, awaitsAlways: true)), + Text(ProjectPaths.Program, RepositoryProgram(project)), + Text(ProjectPaths.Snippet, ProjectSnippet.ForRepository(files.FirstOrDefault(f => f.Name == LambdaSource.EntryName)?.Code ?? string.Empty), LambdaSource.EntryName) + }; + + foreach (var file in files.Where(f => f.Name != LambdaSource.EntryName)) + { + result.Add(new ProjectFile(ProjectPaths.Of(file.Name), file.Bytes, file.Name)); + } + + result.Add(Text($"{ProjectPaths.Platform}Usings.cs", Usings(true, true, true))); + result.Add(Text($"{ProjectPaths.Platform}LambdaEnvironment.cs", Resource("LambdaEnvironment.cs"))); + result.Add(Text($"{ProjectPaths.Platform}Folder.cs", Resource("Folder.cs"))); + result.Add(Text($"{ProjectPaths.Platform}Secrets.cs", Resource("Secrets.cs"))); + result.Add(Text($"{ProjectPaths.Platform}Handlers.cs", Resource("Handlers.cs"))); + result.Add(Text($"{ProjectPaths.Platform}Database.cs", Resource("Database.cs"))); + + if (project.License is { } license) + { + result.Add(Text("LICENSE", SourceLicenses.Text(license.License, license.Year, license.Holder))); + } + + result.Add(Text("Dockerfile", Resource("Dockerfile").Replace("{assembly}", name))); + + const string ignored = "bin/\nobj/\nworkspace/\ndatabase/\n"; + + result.Add(Text(".gitignore", ignored)); + result.Add(Text(".dockerignore", $"{ignored}docs/\ntests/\n.git/\nAGENTS.md\nCLAUDE.md\n")); + + result.Add(Text("AGENTS.md", Resource("Agents.md").Replace("{lambda}", project.PublicKey) + .Replace("{project}", name) + .Replace("{address}", project.Address) + .Replace("{home}", project.Home))); + + result.Add(Text("CLAUDE.md", Resource("Claude.md"))); + + return result; + } + #endregion #region Parts @@ -251,7 +312,11 @@ public static void Pack(ExportedLambda lambda, IReadOnlyList<LambdaFile> files, /// <param name="data">Whether the code uses a database, which takes SQLite</param> /// <param name="entities">Whether it keeps its records with Entity Framework Core</param> /// <param name="evolve">Whether it migrates it with Evolve</param> - private static string Csproj(bool assets, IReadOnlyList<string> context, bool data, bool entities, bool evolve) + /// <param name="awaitsAlways"> + /// Whether the snippet is made asynchronous whether it waits for anything or not, as the project of a + /// repository has it - which the compiler only warns about, and need not + /// </param> + private static string Csproj(bool assets, IReadOnlyList<string> context, bool data, bool entities, bool evolve, bool awaitsAlways = false) { var copy = assets ? "\n\n <ItemGroup>\n <None Update=\"assets/**\" CopyToOutputDirectory=\"PreserveNewest\" />\n </ItemGroup>" : string.Empty; @@ -266,12 +331,15 @@ private static string Csproj(bool assets, IReadOnlyList<string> context, bool da var migrations = evolve ? $"\n <PackageReference Include=\"{EvolvePackage}\" Version=\"{EvolveVersion}\" />" : string.Empty; + // CS1998: an async method that awaits nothing + var quiet = awaitsAlways ? "\n <NoWarn>$(NoWarn);CS1998</NoWarn>" : string.Empty; + return $""" <Project Sdk="Microsoft.NET.Sdk"> <PropertyGroup> <OutputType>Exe</OutputType> - <TargetFramework>{Framework}</TargetFramework> + <TargetFramework>{Framework}</TargetFramework>{quiet} </PropertyGroup> <ItemGroup> @@ -390,69 +458,55 @@ await Host.Create() } /// <summary> - /// The snippet, as the body of the method that builds the app. + /// The host of a lambda's repository, and a word about where it is. /// </summary> /// <remarks> - /// The same split the platform makes. A snippet ends in a return, which at - /// the top of a file would end the program rather than produce a handler, - /// and it may end with a record or two, which cannot be declared inside a - /// method - so statements become the body and types sit beside the class. - /// Leading comments travel with what they stand in front of. + /// Without anything that changes from one version to the next - the + /// number, the change, the secrets the code reads - since it is the same + /// in every commit, including those a push makes. /// </remarks> - private static string Project(SyntaxTree snippet, bool awaits) + private static string RepositoryProgram(RepositoryProject project) { - var root = (CompilationUnitSyntax)snippet.GetRoot(); - - var text = snippet.GetText(); - - var builder = new StringBuilder(); - - foreach (var import in root.Usings) - { - builder.Append(import.NormalizeWhitespace().ToFullString()).Append('\n'); - } - - if (root.Usings.Count > 0) - { - builder.Append('\n'); - } - - builder.Append("public static class Project").Append('\n'); - builder.Append("{").Append('\n'); - builder.Append(" // Workspace comes from Platform/LambdaEnvironment.cs, for every file").Append('\n'); - builder.Append(" private static Platform.AssetFolder Assets => Platform.LambdaEnvironment.Assets;").Append('\n'); - builder.Append('\n'); + var name = Identifier(project.PublicKey); - if (awaits) - { - builder.Append(" public static async Task<IHandler> CreateAsync() => Platform.Handlers.From(await BuildAsync());").Append('\n'); - builder.Append('\n'); - builder.Append(" private static async Task<object> BuildAsync()").Append('\n'); - } - else - { - builder.Append(" public static IHandler Create() => Platform.Handlers.From(Build());").Append('\n'); - builder.Append('\n'); - builder.Append(" private static object Build()").Append('\n'); - } - - builder.Append(" {").Append('\n'); + var tag = name.ToLowerInvariant(); - foreach (var line in Body(root, text)) - { - builder.Append(line).Append('\n'); - } + return $""" + // This app is a lambda on GenHTTP Lambda ({project.Home}), where you describe + // an app - or let your coding agent write it - and it is online at an address + // of its own a moment later, with every version kept and a way back to each. + // + // Lambda {project.PublicKey} ({project.Address}) + // GenHTTP {FrameworkVersion} + // + // This repository is the lambda: every commit of main is one of its versions, + // every other branch a feature being worked on. AGENTS.md says how to work on it. + // + // It is served by GenHTTP, an embeddable web server for .NET. What it can do, + // and how: https://genhttp.org/documentation/ + // + // dotnet run then open http://localhost:8080/ + // + // docker build -t {tag} . + // docker run -p 8080:8080 -v {tag}-data:/app/workspace {tag} + // + // Project.cs holds the code of the lambda and the other .cs files here are its + // own. Platform/ stands in for what the platform provides: the Workspace the + // app writes to (workspace/), the Assets it ships with (assets/), the Secret it + // reads, from environment variables of the same name, and the Database it keeps + // its records in (database/database.db, made empty the first time it connects). + // docs/ says what the app is for and why it is built the way it is, and tests/ + // how it is tested. - builder.Append(" }").Append('\n'); - builder.Append("}").Append('\n'); + using GenHTTP.Engine.Internal; + using GenHTTP.Modules.Practices; - foreach (var member in root.Members.Where(IsType)) - { - builder.Append('\n'); - builder.Append(Public(member, text).Trim('\r', '\n')).Append('\n'); - } + await Host.Create() + .Handler(await Project.CreateAsync()) + .Defaults() + .RunAsync(); - return builder.ToString(); + """; } /// <summary> @@ -501,133 +555,6 @@ private static string Usings(bool data, bool entities, bool evolve) #endregion - #region Snippet - - private static bool IsType(MemberDeclarationSyntax member) => member is BaseTypeDeclarationSyntax or DelegateDeclarationSyntax; - - /// <summary> - /// Whether the statements of the snippet wait for something, which - /// decides whether the method they become has to be asynchronous. - /// </summary> - /// <remarks> - /// Most snippets do not, and a synchronous Project.Create() is the - /// simpler thing to read. Waiting inside a lambda or a local function - /// does not count, because that one is asynchronous on its own. - /// </remarks> - private static bool Awaits(SyntaxTree snippet) - { - var root = (CompilationUnitSyntax)snippet.GetRoot(); - - return root.Members.OfType<GlobalStatementSyntax>() - .SelectMany(s => s.DescendantNodesAndSelf(n => n is not (AnonymousFunctionExpressionSyntax or LocalFunctionStatementSyntax))) - .Any(n => n switch - { - AwaitExpressionSyntax => true, - CommonForEachStatementSyntax loop => loop.AwaitKeyword.IsKind(SyntaxKind.AwaitKeyword), - UsingStatementSyntax block => block.AwaitKeyword.IsKind(SyntaxKind.AwaitKeyword), - LocalDeclarationStatementSyntax declaration => declaration.AwaitKeyword.IsKind(SyntaxKind.AwaitKeyword), - _ => false - }); - } - - /// <summary> - /// The lines of the statements, moved eight spaces in to sit in the method. - /// </summary> - /// <remarks> - /// A line that starts inside a token - a verbatim or raw string spanning - /// lines - is left as it is, because its whitespace is part of the string. - /// </remarks> - private static List<string> Body(CompilationUnitSyntax root, SourceText text) - { - var numbers = new SortedSet<int>(); - - foreach (var member in root.Members.Where(m => !IsType(m))) - { - var first = text.Lines.GetLineFromPosition(member.FullSpan.Start).LineNumber; - var last = text.Lines.GetLineFromPosition(Math.Max(member.FullSpan.Start, member.FullSpan.End - 1)).LineNumber; - - for (var number = first; number <= last; number++) - { - numbers.Add(number); - } - } - - var lines = new List<string>(); - - foreach (var number in numbers) - { - var line = text.Lines[number]; - - var content = line.ToString(); - - var token = root.FindToken(line.Start); - - var inside = line.Start > token.SpanStart && line.Start < token.Span.End; - - lines.Add(inside ? content : content.Trim().Length == 0 ? string.Empty : " " + content); - } - - // the blank lines that stood between the usings, the statements and the types - while (lines.Count > 0 && lines[0].Length == 0) - { - lines.RemoveAt(0); - } - - while (lines.Count > 0 && lines[^1].Length == 0) - { - lines.RemoveAt(lines.Count - 1); - } - - return lines; - } - - /// <summary> - /// A type declared in the snippet, made public on the way. - /// </summary> - /// <remarks> - /// As the platform does: GenHTTP generates the code that invokes a handler - /// into an assembly of its own, so every type in the signature of a - /// handler has to be visible from outside this one. - /// </remarks> - private static string Public(MemberDeclarationSyntax member, SourceText text) - { - var original = text.ToString(member.FullSpan); - - var modifiers = member.Modifiers.Where(IsAccessibility).ToList(); - - if (modifiers.Any(m => m.IsKind(SyntaxKind.PublicKeyword))) - { - return original; - } - - var start = member.FullSpan.Start; - - var at = member.AttributeLists.Count > 0 - ? member.AttributeLists.Last().GetLastToken().GetNextToken().SpanStart - : member.GetFirstToken().SpanStart; - - // from the back, so every position stays where it was; at the same - // position the modifier goes before "public" comes in - var edits = modifiers.Select(m => (Start: m.SpanStart, Length: m.FullSpan.End - m.SpanStart, Text: string.Empty)) - .Append((Start: at, Length: 0, Text: "public ")) - .OrderByDescending(e => e.Start) - .ThenByDescending(e => e.Length); - - var builder = new StringBuilder(original); - - foreach (var (position, length, insert) in edits) - { - builder.Remove(position - start, length).Insert(position - start, insert); - } - - return builder.ToString(); - } - - private static bool IsAccessibility(SyntaxToken token) => token.Kind() is SyntaxKind.PublicKeyword - or SyntaxKind.InternalKeyword or SyntaxKind.PrivateKeyword or SyntaxKind.ProtectedKeyword or SyntaxKind.FileKeyword; - - #endregion - #region Plumbing /// <summary> @@ -653,22 +580,6 @@ private static string Identifier(string publicKey) return name.Length == 0 || !char.IsAsciiLetter(name[0]) ? $"lambda-{name}".TrimEnd('-') : name; } - /// <summary> - /// A code file named the way .NET names them: "store.cs" becomes - /// "Store.cs", and the folders it is in stay as they are. - /// </summary> - private static string Capitalize(string path) - { - var slash = path.LastIndexOf('/') + 1; - - return slash < path.Length ? path[..slash] + char.ToUpperInvariant(path[slash]) + path[(slash + 1)..] : path; - } - - /// <summary> - /// Where a file of the context goes in the project: .lambda/docs/x.md to docs/x.md. - /// </summary> - private static string Outside(string name) => name[LambdaSource.ContextFolder.Length..]; - private static string Day(DateTime value) => value.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture); private static string OneLine(string value) => string.Join(' ', value.Split(['\r', '\n'], StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries)); @@ -690,6 +601,8 @@ private static string Resource(string file) return reader.ReadToEnd().ReplaceLineEndings("\n"); } + private static ProjectFile Text(string path, string content, string? name = null) => new(path, Encoding.UTF8.GetBytes(content), name); + private static void Write(ZipArchive archive, string path, string content) => Write(archive, path, Encoding.UTF8.GetBytes(content)); @@ -737,3 +650,27 @@ public sealed record ExportedLambda(string PublicKey, int Version, DateTime Save /// <param name="Holder">Who holds the copyright, as the license names them</param> /// <param name="Page">Where the source is published, if the installation knows its own address</param> public sealed record ExportedLicense(SourceLicense License, string Holder, string? Page); + +/// <summary> +/// One file of the project a lambda's repository holds. +/// </summary> +/// <param name="Path">Where it is in the project</param> +/// <param name="Content">What it holds</param> +/// <param name="Name">The file of the lambda it is made from, or nothing for one the platform puts around it</param> +public sealed record ProjectFile(string Path, byte[] Content, string? Name); + +/// <summary> +/// What the project of a lambda's repository says about the lambda. +/// </summary> +/// <param name="PublicKey">The name of the lambda, which names the project</param> +/// <param name="Address">Where it runs</param> +/// <param name="Home">Where the installation is, which its files link to</param> +/// <param name="License">The license its source is published under, while it is</param> +public sealed record RepositoryProject(string PublicKey, string Address, string Home, RepositoryLicense? License); + +/// <summary> +/// The license the project of a repository is under, written into its LICENSE. +/// </summary> +/// <param name="Year">The year the license names, which is the year the commit was made in</param> +/// <param name="Holder">Who holds the copyright, as the license names them</param> +public sealed record RepositoryLicense(SourceLicense License, int Year, string Holder); diff --git a/src/GenHTTP.Lambda/Services/Deployment/ProjectPaths.cs b/src/GenHTTP.Lambda/Services/Deployment/ProjectPaths.cs new file mode 100644 index 00000000..47a117d5 --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Deployment/ProjectPaths.cs @@ -0,0 +1,157 @@ +using GenHTTP.Lambda.Services.Deployment.Model; + +namespace GenHTTP.Lambda.Services.Deployment; + +/// <summary> +/// Where the files of a lambda are in the .NET project it is exported as and +/// read as with git, and which file of the lambda a path of the project is. +/// </summary> +/// <remarks> +/// The snippet is <c>Project.cs</c>, the other code files keep their names the +/// way .NET writes them (<c>store.cs</c> is <c>Store.cs</c>), the assets go +/// into <c>assets/</c>, and the documentation and the tests leave +/// <c>.lambda/</c> for <c>docs/</c> and <c>tests/</c>, where a .NET project +/// keeps them. A code file that would be called what the project calls its +/// own - <c>program.cs</c> - is put beside it under a name of its own. +/// +/// What else the project holds is the platform's: the program that hosts the +/// lambda, its project file, what stands in for the platform in +/// <c>Platform/</c>, and the files around them. Every one of those is a +/// <see cref="Generated"/> path; anything that is neither is no part of a +/// lambda's project at all. +/// </remarks> +public static class ProjectPaths +{ + + /// <summary> + /// The file the snippet becomes. + /// </summary> + public const string Snippet = "Project.cs"; + + /// <summary> + /// The program that hosts what <see cref="Snippet"/> returns. + /// </summary> + public const string Program = "Program.cs"; + + /// <summary> + /// Where the files the lambda ships go. + /// </summary> + public const string Assets = "assets/"; + + /// <summary> + /// What stands in for the platform. + /// </summary> + public const string Platform = "Platform/"; + + /// <summary> + /// The files at the root of a project that are the platform's, besides + /// <see cref="Program"/> and the project file. + /// </summary> + private static readonly HashSet<string> Around = new(StringComparer.Ordinal) + { + "Dockerfile", ".gitignore", ".dockerignore", "LICENSE", "AGENTS.md", "CLAUDE.md" + }; + + /// <summary> + /// The names a code file of the lambda cannot have in the project, since + /// the project's own are called that. + /// </summary> + private static readonly HashSet<string> Taken = new(StringComparer.OrdinalIgnoreCase) { Snippet, Program }; + + #region Functionality + + /// <summary> + /// Where a file of the lambda is in the project. + /// </summary> + public static string Of(string name) + { + if (name == LambdaSource.EntryName) + { + return Snippet; + } + + if (LambdaSource.IsContext(name)) + { + return name[LambdaSource.ContextFolder.Length..]; + } + + if (LambdaSource.IsCode(name)) + { + var capitalized = Capitalize(name); + + // beside the project's own, under a name the lambda could not give it + return Taken.Contains(capitalized) ? $"{capitalized[..^3]}.Lambda.cs" : capitalized; + } + + return Assets + name; + } + + /// <summary> + /// What a path of the project is to the lambda. + /// </summary> + public static ProjectPath Classify(string path) + { + if (path == Snippet) + { + return new ProjectPath(ProjectPathKind.Lambda, LambdaSource.EntryName); + } + + if (path.StartsWith(Assets, StringComparison.Ordinal)) + { + return new ProjectPath(ProjectPathKind.Lambda, path[Assets.Length..]); + } + + if (path.StartsWith("docs/", StringComparison.Ordinal) || path.StartsWith("tests/", StringComparison.Ordinal)) + { + return new ProjectPath(ProjectPathKind.Lambda, LambdaSource.ContextFolder + path); + } + + if (path == Program || path.StartsWith(Platform, StringComparison.Ordinal) || Around.Contains(path) || IsProjectFile(path)) + { + return new ProjectPath(ProjectPathKind.Generated, null); + } + + if (!path.Contains('/') && path.EndsWith(".cs", StringComparison.Ordinal)) + { + return new ProjectPath(ProjectPathKind.Lambda, path); + } + + return new ProjectPath(ProjectPathKind.Foreign, null); + } + + /// <summary> + /// The name a code file of the lambda gets in the project: "store.cs" + /// becomes "Store.cs". + /// </summary> + public static string Capitalize(string name) + { + var slash = name.LastIndexOf('/') + 1; + + return slash < name.Length ? name[..slash] + char.ToUpperInvariant(name[slash]) + name[(slash + 1)..] : name; + } + + private static bool IsProjectFile(string path) => !path.Contains('/') && path.EndsWith(".csproj", StringComparison.Ordinal); + + #endregion + +} + +/// <summary> +/// What a path of a project is to the lambda it was made from. +/// </summary> +/// <param name="Name">The file of the lambda it is, where it is one - as the path says, which may differ from the name the lambda gives it in case</param> +public sealed record ProjectPath(ProjectPathKind Kind, string? Name); + +public enum ProjectPathKind +{ + + /// <summary>A file of the lambda: its code, an asset, its documentation or a test.</summary> + Lambda, + + /// <summary>What the platform puts around a lambda to make it a project.</summary> + Generated, + + /// <summary>Neither: nothing a lambda's project holds.</summary> + Foreign + +} diff --git a/src/GenHTTP.Lambda/Services/Deployment/ProjectSnippet.cs b/src/GenHTTP.Lambda/Services/Deployment/ProjectSnippet.cs new file mode 100644 index 00000000..670014b9 --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Deployment/ProjectSnippet.cs @@ -0,0 +1,506 @@ +using System.Text; + +using Microsoft.CodeAnalysis; +using Microsoft.CodeAnalysis.CSharp; +using Microsoft.CodeAnalysis.CSharp.Syntax; +using Microsoft.CodeAnalysis.Text; + +using GenHTTP.Lambda.Services.Deployment.Compilation; + +namespace GenHTTP.Lambda.Services.Deployment; + +/// <summary> +/// The snippet of a lambda as the class <c>Project</c> of a .NET project, and +/// that class as the snippet again. +/// </summary> +/// <remarks> +/// A snippet ends in a return, which at the top of a file would end the +/// program rather than produce a handler, and it may end with a record or two, +/// which cannot be declared inside a method - so its statements become the +/// body of a method of <c>Project</c> and its types sit beside the class. +/// Leading comments travel with what they stand in front of. +/// +/// The way back is what a push to the lambda's repository takes: whoever +/// changed <c>Project.cs</c> changed the snippet. It takes the method body +/// out again, moved back to where it stood, with the imports above it and +/// the types below - so a snippet put into the class and taken out once more +/// is what it was, and a class taken out and put back is the class it was. +/// </remarks> +public static class ProjectSnippet +{ + + /// <summary> + /// The name of the class the snippet becomes. + /// </summary> + public const string Class = "Project"; + + /// <summary> + /// How far the statements are moved in to sit in the method. + /// </summary> + private const int Indent = 8; + + /// <summary> + /// The members of <see cref="Class"/> that are the platform's rather than the snippet's. + /// </summary> + private static readonly HashSet<string> Frame = ["Assets", "Create", "CreateAsync"]; + + #region Wrapping + + /// <summary> + /// The snippet as the class of an exported project: <c>Create()</c>, or + /// <c>CreateAsync()</c> where its statements wait for something. + /// </summary> + public static string ForExport(string snippet) + { + var tree = SourceBuilder.ParseSnippet(snippet); + + return Wrap(tree, Awaits(tree)); + } + + /// <summary> + /// The snippet as the class of the project a lambda's repository holds: + /// always <c>CreateAsync()</c>, as the platform runs it. + /// </summary> + /// <remarks> + /// Always the same shape, whatever the statements do, because the rest of + /// the project - Program.cs above all - is the platform's and the same + /// for every commit: a push that starts awaiting something must not have + /// to change it. A method that waits for nothing is merely warned about, + /// and the project does not warn about that. + /// </remarks> + public static string ForRepository(string snippet) => Wrap(SourceBuilder.ParseSnippet(snippet), true); + + /// <summary> + /// Whether the statements of the snippet wait for something, which + /// decides whether the method they become has to be asynchronous. + /// </summary> + /// <remarks> + /// Most snippets do not, and a synchronous Project.Create() is the + /// simpler thing to read. Waiting inside a lambda or a local function + /// does not count, because that one is asynchronous on its own. + /// </remarks> + public static bool Awaits(string snippet) => Awaits(SourceBuilder.ParseSnippet(snippet)); + + private static bool Awaits(SyntaxTree snippet) + { + var root = (CompilationUnitSyntax)snippet.GetRoot(); + + return root.Members.OfType<GlobalStatementSyntax>() + .SelectMany(s => s.DescendantNodesAndSelf(n => n is not (AnonymousFunctionExpressionSyntax or LocalFunctionStatementSyntax))) + .Any(n => n switch + { + AwaitExpressionSyntax => true, + CommonForEachStatementSyntax loop => loop.AwaitKeyword.IsKind(SyntaxKind.AwaitKeyword), + UsingStatementSyntax block => block.AwaitKeyword.IsKind(SyntaxKind.AwaitKeyword), + LocalDeclarationStatementSyntax declaration => declaration.AwaitKeyword.IsKind(SyntaxKind.AwaitKeyword), + _ => false + }); + } + + private static string Wrap(SyntaxTree snippet, bool asynchronous) + { + var root = (CompilationUnitSyntax)snippet.GetRoot(); + + var text = snippet.GetText(); + + var builder = new StringBuilder(); + + foreach (var import in root.Usings) + { + builder.Append(import.NormalizeWhitespace().ToFullString()).Append('\n'); + } + + if (root.Usings.Count > 0) + { + builder.Append('\n'); + } + + builder.Append($"public static class {Class}").Append('\n'); + builder.Append("{").Append('\n'); + builder.Append(" // Workspace comes from Platform/LambdaEnvironment.cs, for every file").Append('\n'); + builder.Append(" private static Platform.AssetFolder Assets => Platform.LambdaEnvironment.Assets;").Append('\n'); + builder.Append('\n'); + + if (asynchronous) + { + builder.Append(" public static async Task<IHandler> CreateAsync() => Platform.Handlers.From(await BuildAsync());").Append('\n'); + builder.Append('\n'); + builder.Append(" private static async Task<object> BuildAsync()").Append('\n'); + } + else + { + builder.Append(" public static IHandler Create() => Platform.Handlers.From(Build());").Append('\n'); + builder.Append('\n'); + builder.Append(" private static object Build()").Append('\n'); + } + + builder.Append(" {").Append('\n'); + + foreach (var line in Body(root, text)) + { + builder.Append(line).Append('\n'); + } + + builder.Append(" }").Append('\n'); + builder.Append("}").Append('\n'); + + foreach (var member in root.Members.Where(IsType)) + { + builder.Append('\n'); + builder.Append(Public(member, text).Trim('\r', '\n')).Append('\n'); + } + + return builder.ToString(); + } + + private static bool IsType(MemberDeclarationSyntax member) => member is BaseTypeDeclarationSyntax or DelegateDeclarationSyntax; + + /// <summary> + /// The lines of the statements, moved eight spaces in to sit in the method. + /// </summary> + /// <remarks> + /// A line that starts inside a token - a verbatim or raw string spanning + /// lines - is left as it is, because its whitespace is part of the string. + /// </remarks> + private static List<string> Body(CompilationUnitSyntax root, SourceText text) + { + var numbers = new SortedSet<int>(); + + foreach (var member in root.Members.Where(m => !IsType(m))) + { + var first = text.Lines.GetLineFromPosition(member.FullSpan.Start).LineNumber; + var last = text.Lines.GetLineFromPosition(Math.Max(member.FullSpan.Start, member.FullSpan.End - 1)).LineNumber; + + for (var number = first; number <= last; number++) + { + numbers.Add(number); + } + } + + var lines = new List<string>(); + + foreach (var number in numbers) + { + var line = text.Lines[number]; + + var content = line.ToString(); + + lines.Add(Inside(root, line.Start) ? content : content.Trim().Length == 0 ? string.Empty : new string(' ', Indent) + content); + } + + // the blank lines that stood between the usings, the statements and the types + while (lines.Count > 0 && lines[0].Length == 0) + { + lines.RemoveAt(0); + } + + while (lines.Count > 0 && lines[^1].Length == 0) + { + lines.RemoveAt(lines.Count - 1); + } + + return lines; + } + + /// <summary> + /// A type declared in the snippet, made public on the way. + /// </summary> + /// <remarks> + /// As the platform does: GenHTTP generates the code that invokes a handler + /// into an assembly of its own, so every type in the signature of a + /// handler has to be visible from outside this one. + /// </remarks> + private static string Public(MemberDeclarationSyntax member, SourceText text) + { + var original = text.ToString(member.FullSpan); + + var modifiers = member.Modifiers.Where(IsAccessibility).ToList(); + + if (modifiers.Any(m => m.IsKind(SyntaxKind.PublicKeyword))) + { + return original; + } + + var start = member.FullSpan.Start; + + var at = member.AttributeLists.Count > 0 + ? member.AttributeLists.Last().GetLastToken().GetNextToken().SpanStart + : member.GetFirstToken().SpanStart; + + // from the back, so every position stays where it was; at the same + // position the modifier goes before "public" comes in + var edits = modifiers.Select(m => (Start: m.SpanStart, Length: m.FullSpan.End - m.SpanStart, Text: string.Empty)) + .Append((Start: at, Length: 0, Text: "public ")) + .OrderByDescending(e => e.Start) + .ThenByDescending(e => e.Length); + + var builder = new StringBuilder(original); + + foreach (var (position, length, insert) in edits) + { + builder.Remove(position - start, length).Insert(position - start, insert); + } + + return builder.ToString(); + } + + private static bool IsAccessibility(SyntaxToken token) => token.Kind() is SyntaxKind.PublicKeyword + or SyntaxKind.InternalKeyword or SyntaxKind.PrivateKeyword or SyntaxKind.ProtectedKeyword or SyntaxKind.FileKeyword; + + #endregion + + #region Unwrapping + + /// <summary> + /// The snippet a class <c>Project</c> holds: the imports above it, the + /// body of its <c>BuildAsync()</c> - or <c>Build()</c>, as an export has + /// it - moved back out, and the types beside the class below. + /// </summary> + /// <remarks> + /// The body is moved out by as much as it was moved in - eight spaces, or + /// less where all of it stands less far in - so the lines keep where they + /// stand to each other. A line that starts inside a string is left alone. + /// What the class holds besides the method - the assets, the method that + /// makes the handler - is the platform's and is left behind; anything + /// else in it has no place in a snippet, and is refused. + /// </remarks> + /// <returns>The snippet and where each of its lines came from, or why there is none</returns> + public static SnippetUnwrapped Unwrap(string project) + { + var tree = CSharpSyntaxTree.ParseText(project, SourceBuilder.RegularOptions, "Project.cs"); + + var root = (CompilationUnitSyntax)tree.GetRoot(); + + var text = tree.GetText(); + + var classes = root.Members.OfType<ClassDeclarationSyntax>().Where(c => c.Identifier.Text == Class).ToList(); + + if (classes.Count != 1) + { + return SnippetUnwrapped.Refused($"Project.cs keeps the class {Class}: its BuildAsync() holds the code of the lambda, what lambda.cs is on the platform."); + } + + var frame = classes[0]; + + var methods = frame.Members.OfType<MethodDeclarationSyntax>().Where(m => m.Identifier.Text is "BuildAsync" or "Build").ToList(); + + if (methods.Count != 1 || methods[0].Body is not { } body) + { + return SnippetUnwrapped.Refused($"The class {Class} in Project.cs keeps one method BuildAsync() with a body: that body is the code of the lambda."); + } + + foreach (var member in frame.Members) + { + if (member == methods[0] || Frame.Contains(NameOf(member))) + { + continue; + } + + return SnippetUnwrapped.Refused($"'{NameOf(member)}' cannot stay in the class {Class}: it only holds BuildAsync(), whose body is the code of the lambda. " + + "Declare it in BuildAsync() as a local, or in a type of its own below the class or in a file of its own."); + } + + if (root.AttributeLists.Count > 0 || root.Externs.Count > 0) + { + return SnippetUnwrapped.Refused("Project.cs cannot hold assembly attributes or extern aliases: the lambda has no place for them."); + } + + var lines = new List<string>(); + + var origins = new List<(int Line, int Removed)>(); + + if (root.Usings.Count > 0) + { + // with what is written above them - the comment a snippet opens with + var first = text.Lines.GetLineFromPosition(root.Usings[0].FullSpan.Start).LineNumber; + var last = text.Lines.GetLineFromPosition(root.Usings[^1].Span.End).LineNumber; + + for (var number = first; number <= last; number++) + { + var line = text.Lines[number].ToString().TrimEnd(); + + if (lines.Count > 0 || line.Length > 0) + { + lines.Add(line); + origins.Add((number, 0)); + } + } + + lines.Add(string.Empty); + origins.Add((-1, 0)); + } + + var statements = Statements(root, text, body); + + if (statements.Count > 0) + { + lines.AddRange(statements.Select(s => s.Text)); + origins.AddRange(statements.Select(s => (s.Line, s.Removed))); + } + + foreach (var member in root.Members.Where(m => m != frame)) + { + if (member is not (BaseTypeDeclarationSyntax or DelegateDeclarationSyntax)) + { + return SnippetUnwrapped.Refused("Beside the class Project, Project.cs holds types and nothing else: the code of the lambda goes into BuildAsync()."); + } + + var declared = text.ToString(member.FullSpan).Trim('\r', '\n'); + + // where the declaration starts, once the blank lines before it are gone + var start = member.FullSpan.Start + text.ToString(member.FullSpan).IndexOf(declared, StringComparison.Ordinal); + + var first = text.Lines.GetLineFromPosition(start).LineNumber; + + if (lines.Count > 0) + { + lines.Add(string.Empty); + origins.Add((-1, 0)); + } + + var parts = declared.Split('\n'); + + for (var i = 0; i < parts.Length; i++) + { + lines.Add(parts[i].TrimEnd('\r')); + origins.Add((first + i, 0)); + } + } + + // a line break is a line feed, whatever Project.cs was written with - + // which is what putting it back into the class writes, too + return new SnippetUnwrapped(string.Join('\n', lines) + "\n", null, origins); + } + + /// <summary> + /// The lines of the method's body, moved back out. + /// </summary> + private static List<(string Text, int Line, int Removed)> Statements(CompilationUnitSyntax root, SourceText text, BlockSyntax body) + { + var open = text.Lines.GetLineFromPosition(body.OpenBraceToken.SpanStart).LineNumber; + var close = text.Lines.GetLineFromPosition(body.CloseBraceToken.SpanStart).LineNumber; + + var lines = new List<(string Text, int Line, bool Inside)>(); + + if (open == close) + { + // all on one line: what stands between the braces + var between = text.ToString(TextSpan.FromBounds(body.OpenBraceToken.Span.End, body.CloseBraceToken.SpanStart)).Trim(); + + if (between.Length > 0) + { + lines.Add((between, open, false)); + } + } + else + { + // what follows the opening brace on its line, and what precedes the closing one on its + var after = text.ToString(TextSpan.FromBounds(body.OpenBraceToken.Span.End, text.Lines[open].End)); + + if (after.Trim().Length > 0) + { + lines.Add((after.Trim(), open, false)); + } + + for (var number = open + 1; number < close; number++) + { + var line = text.Lines[number]; + + lines.Add((line.ToString(), number, Inside(root, line.Start))); + } + + var before = text.ToString(TextSpan.FromBounds(text.Lines[close].Start, body.CloseBraceToken.SpanStart)); + + if (before.Trim().Length > 0) + { + lines.Add((before, close, Inside(root, text.Lines[close].Start))); + } + } + + // moved out by as much as all of it stands in, and no more than it was moved in + var common = lines.Where(l => !l.Inside && l.Text.Trim().Length > 0) + .Select(l => l.Text.Length - l.Text.TrimStart(' ', '\t').Length) + .DefaultIfEmpty(0) + .Min(); + + var removed = Math.Min(Indent, common); + + var result = lines.Select(l => l.Inside + ? (l.Text, l.Line, 0) + : l.Text.Trim().Length == 0 + ? (string.Empty, l.Line, 0) + : (l.Text[removed..], l.Line, removed)) + .ToList(); + + while (result.Count > 0 && result[0].Item1.Length == 0) + { + result.RemoveAt(0); + } + + while (result.Count > 0 && result[^1].Item1.Length == 0) + { + result.RemoveAt(result.Count - 1); + } + + return result; + } + + /// <summary> + /// Whether a position is inside a token rather than before one: a line + /// starting there continues a string written over several lines. + /// </summary> + private static bool Inside(SyntaxNode root, int position) + { + var token = root.FindToken(position); + + return position > token.SpanStart && position < token.Span.End; + } + + private static string NameOf(MemberDeclarationSyntax member) => member switch + { + MethodDeclarationSyntax method => method.Identifier.Text, + PropertyDeclarationSyntax property => property.Identifier.Text, + FieldDeclarationSyntax field => string.Join(", ", field.Declaration.Variables.Select(v => v.Identifier.Text)), + BaseTypeDeclarationSyntax type => type.Identifier.Text, + ConstructorDeclarationSyntax => "the constructor", + EventDeclarationSyntax e => e.Identifier.Text, + EventFieldDeclarationSyntax e => string.Join(", ", e.Declaration.Variables.Select(v => v.Identifier.Text)), + DelegateDeclarationSyntax d => d.Identifier.Text, + _ => member.Kind().ToString() + }; + + #endregion + +} + +/// <summary> +/// A snippet taken out of the class <c>Project</c>. +/// </summary> +/// <param name="Snippet">What lambda.cs holds, or nothing where it could not be taken out</param> +/// <param name="Complaint">Why it could not</param> +/// <param name="Origins"> +/// For every line of the snippet, the line of Project.cs it came from (zero based, -1 for a line the way back put +/// in) and how many characters it was moved out by - to say where in Project.cs a problem the compiler found is +/// </param> +public sealed record SnippetUnwrapped(string? Snippet, string? Complaint, IReadOnlyList<(int Line, int Removed)> Origins) +{ + + public static SnippetUnwrapped Refused(string complaint) => new(null, complaint, []); + + /// <summary> + /// Where in Project.cs a position in the snippet is, both one based - or + /// nothing where the line is not one of Project.cs. + /// </summary> + public (int Line, int Column)? Locate(int line, int column) + { + if (line < 1 || line > Origins.Count || Origins[line - 1].Line < 0) + { + return null; + } + + var (origin, removed) = Origins[line - 1]; + + return (origin + 1, column + removed); + } + +} diff --git a/src/GenHTTP.Lambda/Services/Features/FeatureBranches.cs b/src/GenHTTP.Lambda/Services/Features/FeatureBranches.cs new file mode 100644 index 00000000..069d2380 --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Features/FeatureBranches.cs @@ -0,0 +1,154 @@ +using System.Globalization; +using System.Text; + +using GenHTTP.Modules.Git; + +namespace GenHTTP.Lambda.Services.Features; + +/// <summary> +/// The branch a feature is in the lambda's git repository: what it may be +/// called, and what a feature started anywhere but git is called there. +/// </summary> +/// <remarks> +/// A branch is named once, when the feature starts, and kept: a clone knows +/// it by its name, so renaming the feature leaves it alone. A branch somebody +/// pushed keeps the name they gave it; any other feature is given its name +/// in the words git allows - "Dark mode" is <c>dark-mode</c> - and a number +/// where that is taken. +/// </remarks> +internal static class FeatureBranches +{ + + /// <summary> + /// The branch the versions are, which no feature may be. + /// </summary> + public const string Main = "main"; + + /// <summary> + /// How long the name of a branch may be. + /// </summary> + public const int MaxLength = 100; + + /// <summary> + /// How long a branch named after a feature gets before the rest is cut. + /// </summary> + private const int Shortened = 40; + + #region Functionality + + /// <summary> + /// Why a branch somebody wants could not be a feature's, or nothing when it could. + /// </summary> + public static string? Check(string branch) + { + if (branch.Length > MaxLength) + { + return $"A branch of a lambda may be called {MaxLength} characters at most."; + } + + if (string.Equals(branch, Main, StringComparison.OrdinalIgnoreCase) || string.Equals(branch, "HEAD", StringComparison.OrdinalIgnoreCase)) + { + return $"'{branch}' is not a name for a feature: main holds the versions."; + } + + if (!GitReference.IsValidName($"refs/heads/{branch}")) + { + return $"'{branch}' is not a name git allows for a branch."; + } + + return null; + } + + /// <summary> + /// The branch a feature called this is given, unless one of the taken is + /// called that - then the first with a number after it that is free. + /// </summary> + /// <param name="taken">The branches of the lambda's other features</param> + public static string For(string name, IEnumerable<string> taken) + { + var stem = Slug(name); + + var used = taken.ToHashSet(StringComparer.OrdinalIgnoreCase); + + if (!Clashes(stem, used)) + { + return stem; + } + + for (var number = 2; ; number++) + { + var candidate = $"{stem}-{number}"; + + if (!Clashes(candidate, used)) + { + return candidate; + } + } + } + + /// <summary> + /// Whether a branch could not be held beside the taken ones: one of the + /// same name, or one that would be a folder of it or it of one - + /// <c>dark-mode</c> and <c>dark-mode/menu</c> cannot both be in a clone. + /// </summary> + /// <remarks> + /// Without regard to case, since a clone on a file system that has none + /// keeps its branches as files. + /// </remarks> + public static bool Clashes(string branch, IEnumerable<string> taken) + { + foreach (var other in taken) + { + if (string.Equals(branch, other, StringComparison.OrdinalIgnoreCase) + || branch.StartsWith(other + "/", StringComparison.OrdinalIgnoreCase) + || other.StartsWith(branch + "/", StringComparison.OrdinalIgnoreCase)) + { + return true; + } + } + + return string.Equals(branch, Main, StringComparison.OrdinalIgnoreCase) || branch.StartsWith(Main + "/", StringComparison.OrdinalIgnoreCase); + } + + #endregion + + #region Helpers + + /// <summary> + /// A name in the words a branch may have: lower case letters and digits, + /// and a dash for everything else. + /// </summary> + private static string Slug(string name) + { + var builder = new StringBuilder(); + + foreach (var character in name.Normalize(NormalizationForm.FormD)) + { + if (char.IsAsciiLetterOrDigit(character)) + { + builder.Append(char.ToLowerInvariant(character)); + } + else if (CharUnicodeInfo.GetUnicodeCategory(character) == UnicodeCategory.NonSpacingMark) + { + // the accent of a letter, which leaves the letter as it is + continue; + } + else if (builder.Length > 0 && builder[^1] != '-') + { + builder.Append('-'); + } + } + + var slug = builder.ToString().Trim('-'); + + if (slug.Length > Shortened) + { + slug = slug[..Shortened].TrimEnd('-'); + } + + return slug.Length == 0 ? "feature" : slug; + } + + #endregion + +} diff --git a/src/GenHTTP.Lambda/Services/Features/FeatureModel.cs b/src/GenHTTP.Lambda/Services/Features/FeatureModel.cs index 9dbc26b4..9c3a0f09 100644 --- a/src/GenHTTP.Lambda/Services/Features/FeatureModel.cs +++ b/src/GenHTTP.Lambda/Services/Features/FeatureModel.cs @@ -8,6 +8,7 @@ namespace GenHTTP.Lambda.Services.Features; /// </summary> /// <param name="Key">What it is addressed by, and where its preview answers: <c>/features/{Key}/</c></param> /// <param name="Name">What it is, in a few words</param> +/// <param name="Branch">The branch it is in the lambda's git repository</param> /// <param name="Specification">What the user wants from it and why; the version it is merged into keeps it</param> /// <param name="Change">What it changes, in a line; the version it is merged into keeps it</param> /// <param name="Base">The version it is based on: the one it branched off, or a later one whose changes were brought in</param> @@ -21,6 +22,7 @@ namespace GenHTTP.Lambda.Services.Features; public sealed record FeatureInfo( string Key, string Name, + string Branch, string? Specification, string? Change, int Base, @@ -61,7 +63,8 @@ public sealed record FeatureContent(FeatureInfo Feature, string Code); /// <param name="Specification">What the user wants from it and why</param> /// <param name="Base">The version to branch off; the newest when left out</param> /// <param name="Origin">Which door it came through</param> -public sealed record FeatureDraft(string? Name, string? Specification = null, int? Base = null, string? Origin = null); +/// <param name="Branch">The branch it is to be in the lambda's git repository; left out, one named after it</param> +public sealed record FeatureDraft(string? Name, string? Specification = null, int? Base = null, string? Origin = null, string? Branch = null); /// <summary> /// What to change about a feature. What is left out stays as it is. diff --git a/src/GenHTTP.Lambda/Services/Features/FeatureService.cs b/src/GenHTTP.Lambda/Services/Features/FeatureService.cs index 739718e9..8a4fa54a 100644 --- a/src/GenHTTP.Lambda/Services/Features/FeatureService.cs +++ b/src/GenHTTP.Lambda/Services/Features/FeatureService.cs @@ -203,11 +203,14 @@ public async ValueTask<FeatureInfo> CreateAsync(string privateKey, FeatureDraft var now = DateTime.UtcNow; + var taken = database.Features.Where(f => f.LambdaId == lambdaId).Select(f => f.Branch).ToList(); + var entity = new FeatureEntity { LambdaId = lambdaId, Key = Convert.ToHexStringLower(RandomNumberGenerator.GetBytes(16)), Name = name, + Branch = Branch(draft.Branch, name, taken), Specification = VersionInput.Tidy(draft.Specification, VersionNote.MaxSpecification), BaseVersion = from, Origin = draft.Origin, @@ -221,7 +224,21 @@ public async ValueTask<FeatureInfo> CreateAsync(string privateKey, FeatureDraft // is not left to expire while they do lambda.Modified = now; - database.SaveChanges(); + try + { + database.SaveChanges(); + } + catch (DbUpdateException) when (draft.Branch == null) + { + // another feature took the same branch in the meantime: the next free one + entity.Branch = FeatureBranches.For(name, [.. database.Features.AsNoTracking().Where(f => f.LambdaId == lambdaId).Select(f => f.Branch)]); + + database.SaveChanges(); + } + catch (DbUpdateException) + { + throw LambdaException.Conflict($"Another feature of this lambda is at the branch '{draft.Branch}'."); + } try { @@ -743,7 +760,7 @@ internal static string Behind(FeatureEntity feature, int? newest) + $"(update_feature with base: {newest}, or PATCH the feature with base) and merge again."; private static FeatureInfo Describe(FeatureEntity feature, int? newest) - => new(feature.Key, feature.Name, feature.Specification, feature.Change, feature.BaseVersion, newest, feature.Origin, + => new(feature.Key, feature.Name, feature.Branch, feature.Specification, feature.Change, feature.BaseVersion, newest, feature.Origin, feature.Created, feature.Modified, feature.Previewed != null, feature.Previewed != null && feature.PreviewOf == feature.Revision, feature.Previewed, feature.Revision); @@ -754,6 +771,31 @@ private static FeatureInfo Describe(FeatureEntity feature, int? newest) internal static bool IsKey(string? key) => key is { Length: 32 } && key.All(c => c is >= '0' and <= '9' or >= 'a' and <= 'f'); + /// <summary> + /// The branch a new feature is in the lambda's repository: the one asked + /// for, which has to be free, or one named after it. + /// </summary> + /// <param name="taken">The branches of the lambda's other features</param> + private static string Branch(string? wanted, string name, IReadOnlyList<string> taken) + { + if (wanted == null) + { + return FeatureBranches.For(name, taken); + } + + if (FeatureBranches.Check(wanted) is { } complaint) + { + throw LambdaException.Invalid(complaint); + } + + if (FeatureBranches.Clashes(wanted, taken)) + { + throw LambdaException.Conflict($"Another feature of this lambda is at the branch '{wanted}', or at one that cannot sit beside it."); + } + + return wanted; + } + private static string Name(string? name) { var trimmed = name?.Trim(); diff --git a/src/GenHTTP.Lambda/Services/Git/GitHistory.cs b/src/GenHTTP.Lambda/Services/Git/GitHistory.cs new file mode 100644 index 00000000..0bae9182 --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Git/GitHistory.cs @@ -0,0 +1,320 @@ +using GenHTTP.Modules.Git; + +using GenHTTP.Lambda.Services.Deployment; +using GenHTTP.Lambda.Services.Deployment.Model; +using GenHTTP.Lambda.Services.Features; +using GenHTTP.Lambda.Services.Meta; +using GenHTTP.Lambda.Services.Meta.Model; + +using Microsoft.Extensions.Logging; + +namespace GenHTTP.Lambda.Services.Git; + +/// <summary> +/// Writes down the commits a lambda is read as: one for every version and +/// every save of a feature the platform made, when its repository is read. +/// </summary> +/// <remarks> +/// Made when somebody reads the repository rather than when a version is +/// saved, so saving - the editor, an agent, the API - stays as it was, and a +/// lambda nobody clones costs nothing. A commit is made from what it stands +/// for alone - the version, its time, its change - and kept from then on, so +/// it is the same commit for everybody who reads it later. +/// +/// A version's commit follows the commit of the version before it. A +/// feature's follows the feature's tip, or the version it starts from; one +/// whose base was moved past what its tip holds - its owner or an agent +/// brought a newer version in - follows that version as well, so the branch +/// says it holds it. A feature that holds what its version holds is that +/// version's commit, until it changes. +/// +/// Called in the lambda's turn to be read (see <see cref="GitService"/>), and +/// away from the reactor: the first read of a lambda with a long history +/// reads every version. +/// </remarks> +public sealed class GitHistory(GitStore store, IMetaService meta, IFeatureService features, ILogger<GitHistory> logger) +{ + + /// <summary> + /// Who the commits the platform makes are by. + /// </summary> + private const string Author = "GenHTTP Lambda"; + + #region Bringing up to date + + /// <summary> + /// Makes the commits that are missing, forgets those of versions and + /// features that are gone, and says which commit is what now. + /// </summary> + /// <param name="withFeatures">Whether the features are read too - only for whoever holds the editor key</param> + public async ValueTask<GitIndex> SyncAsync(GitLambda lambda, bool withFeatures) + { + var index = store.ReadIndex(lambda.Id); + + var changed = false; + + var dropped = false; + + var versions = meta.GetVersions(lambda.PrivateKey).OrderBy(v => v.Version).ToList(); + + var existing = versions.Select(v => v.Version).ToHashSet(); + + // the commits of versions pruned since, which the next commit still follows + var before = new SortedDictionary<int, string>(index.Versions); + + foreach (var gone in index.Versions.Keys.Where(v => !existing.Contains(v)).ToList()) + { + index.Versions.Remove(gone); + dropped = true; + } + + foreach (var version in versions) + { + if (index.Versions.ContainsKey(version.Version)) + { + continue; + } + + // the commit of the version before it - or, where that was pruned + // before anybody read it, of the newest before it that was read + var parent = before.Where(v => v.Key < version.Version).Select(v => v.Value).LastOrDefault(); + + string code; + + try + { + code = meta.GetVersion(lambda.PrivateKey, version.Version).Code; + } + catch (LambdaException) + { + // pruned since the versions were listed: it is no part of the history any more + continue; + } + + var previous = parent != null ? store.ReadCommit(lambda.Id, GitObjectId.Parse(parent)) : null; + + var made = await CommitAsync(lambda, LambdaSource.Parse(code), parent != null ? [parent] : [], previous, + OneLine(version.Change) ?? $"Version {version.Version}", version.Created, version.Version); + + index.Versions[version.Version] = made; + before[version.Version] = made; + + changed = true; + } + + if (withFeatures) + { + var open = features.List(lambda.PrivateKey); + + var keys = open.Select(f => f.Key).ToHashSet(StringComparer.Ordinal); + + foreach (var gone in index.Features.Keys.Where(k => !keys.Contains(k)).ToList()) + { + index.Features.Remove(gone); + dropped = true; + } + + foreach (var feature in open) + { + changed |= await SyncAsync(lambda, index, feature); + } + } + + if (changed || dropped) + { + store.WriteIndex(lambda.Id, index); + } + + if (dropped) + { + var removed = store.Collect(lambda.Id, index); + + if (removed > 0) + { + logger.LogDebug("Collected {Count} commits of lambda #{LambdaId}", removed, lambda.Id); + } + } + + return index; + } + + /// <summary> + /// Makes the commit a feature needs, if it needs one. + /// </summary> + /// <returns>Whether its tip moved</returns> + private async ValueTask<bool> SyncAsync(GitLambda lambda, GitIndex index, FeatureInfo feature) + { + index.Features.TryGetValue(feature.Key, out var tip); + + if (tip != null && tip.Revision == feature.Revision && tip.Base >= feature.Base) + { + return false; + } + + if (!index.Versions.TryGetValue(feature.Base, out var based)) + { + // the version it starts from is kept like the one online, so this + // is a version saved after the versions were read - the next read + return false; + } + + string code; + + try + { + code = features.Get(lambda.PrivateKey, feature.Key).Code; + } + catch (LambdaException) + { + // merged or deleted since the features were listed + return false; + } + + var own = tip != null && store.ReadCommit(lambda.Id, GitObjectId.Parse(tip.Commit)) is { Version: null }; + + List<string> parents; + + if (!own) + { + // nothing of its own yet: as long as it holds what its version + // holds, it is that version's commit + if (code == meta.GetVersion(lambda.PrivateKey, feature.Base).Code) + { + index.Features[feature.Key] = new FeatureTip(based, feature.Revision, feature.Base); + return true; + } + + parents = [based]; + } + else + { + parents = tip!.Base < feature.Base ? [tip.Commit, based] : [tip.Commit]; + } + + var previous = store.ReadCommit(lambda.Id, GitObjectId.Parse(parents[0])); + + var message = parents.Count > 1 + ? $"Brings version {feature.Base} into the feature '{feature.Name}'" + : OneLine(feature.Change) ?? feature.Name; + + var commit = await CommitAsync(lambda, LambdaSource.Parse(code), parents, previous, message, feature.Modified, null); + + index.Features[feature.Key] = new FeatureTip(commit, feature.Revision, feature.Base); + + return true; + } + + #endregion + + #region Commits + + /// <summary> + /// Makes a commit of the platform's, keeps it, and says which it is. + /// </summary> + private async ValueTask<string> CommitAsync(GitLambda lambda, IReadOnlyList<LambdaFile> files, IReadOnlyList<string> parents, StoredCommit? previous, + string message, DateTime when, int? version) + { + var at = new DateTimeOffset(DateTime.SpecifyKind(when, DateTimeKind.Utc)); + + // the year a published source's license names is the year of the commit + var project = lambda.Project.License is { } license ? lambda.Project with { License = license with { Year = at.Year } } : lambda.Project; + + var laid = ProjectTree.Lay(files, project, previous); + + foreach (var (blob, content) in laid.Contents) + { + store.WriteBlob(lambda.Id, GitObjectId.Parse(blob), content); + } + + var tree = await TreeOf(lambda.Id, laid.Entries).ComputeIdAsync(); + + var builder = GitCommit.Create() + .Tree(tree) + .Author(new GitSignature(Author, Email(lambda), at)) + .Message(message); + + foreach (var parent in parents) + { + builder.Parent(GitObjectId.Parse(parent)); + } + + var commit = builder.Build(); + + store.WriteCommit(lambda.Id, commit.Id, new StoredCommit(Convert.ToBase64String(commit.Data.Span), laid.Entries, laid.Files, version)); + + return commit.Id.ToString(); + } + + /// <summary> + /// Keeps a commit somebody pushed, exactly as it came. + /// </summary> + /// <param name="read">What the lambda makes of it, or nothing for one that was only on the way</param> + /// <param name="version">The version it was saved as, if it was</param> + public async ValueTask<StoredCommit> KeepAsync(long lambdaId, GitCommit commit, IReadOnlyList<GitFile> tree, ReadTree? read, int? version) + { + var entries = new List<StoredEntry>(tree.Count); + + foreach (var file in tree) + { + var id = file.Id ?? GitObjectId.ForBlob((await file.ReadAsync()).Span); + + if (!store.HasBlob(lambdaId, id)) + { + store.WriteBlob(lambdaId, id, (await file.ReadAsync()).Span); + } + + entries.Add(new StoredEntry(file.Path, id.ToString())); + } + + foreach (var (blob, content) in read?.Contents ?? new Dictionary<string, byte[]>()) + { + store.WriteBlob(lambdaId, GitObjectId.Parse(blob), content); + } + + var stored = new StoredCommit(Convert.ToBase64String(commit.Data.Span), entries, read?.Files != null ? read.Stored : null, version); + + store.WriteCommit(lambdaId, commit.Id, stored); + + return stored; + } + + /// <summary> + /// The files of a kept commit, as the server reads them. + /// </summary> + public GitTree TreeOf(long lambdaId, IReadOnlyList<StoredEntry> entries) + { + var tree = GitTree.Create(); + + foreach (var entry in entries) + { + var id = GitObjectId.Parse(entry.Blob); + + tree.Add(entry.Path, () => new ValueTask<ReadOnlyMemory<byte>>(store.ReadBlob(lambdaId, id)), GitFileMode.Regular, id); + } + + return tree.Build(); + } + + #endregion + + #region Helpers + + /// <summary> + /// The address the commits of the platform are by: the installation's. + /// </summary> + private static string Email(GitLambda lambda) + => $"lambda@{(Uri.TryCreate(lambda.Project.Home, UriKind.Absolute, out var home) ? home.Host : "localhost")}"; + + private static string? OneLine(string? text) + { + if (string.IsNullOrWhiteSpace(text)) + { + return null; + } + + return string.Join(' ', text.Split(['\r', '\n'], StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries)); + } + + #endregion + +} diff --git a/src/GenHTTP.Lambda/Services/Git/GitModel.cs b/src/GenHTTP.Lambda/Services/Git/GitModel.cs new file mode 100644 index 00000000..6b9bd3b2 --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Git/GitModel.cs @@ -0,0 +1,101 @@ +using GenHTTP.Modules.Git; + +using GenHTTP.Lambda.Services.Deployment; + +namespace GenHTTP.Lambda.Services.Git; + +/// <summary> +/// Which commit each version of a lambda is, and which is the tip of each of +/// its features - what the references of its repository are made from. +/// </summary> +/// <param name="Versions">The commit of each version, by its number</param> +/// <param name="Features">The tip of each feature, by its key</param> +public sealed record GitIndex(Dictionary<int, string> Versions, Dictionary<string, FeatureTip> Features) +{ + + public static GitIndex Empty() => new([], []); + + /// <summary> + /// The commit of the newest version. + /// </summary> + public GitObjectId? Newest => Versions.Count == 0 ? null : GitObjectId.Parse(Versions[Versions.Keys.Max()]); + + /// <summary> + /// The version a commit is, or nothing for one that is none. + /// </summary> + public int? VersionOf(GitObjectId commit) + { + var hex = commit.ToString(); + + foreach (var (version, id) in Versions) + { + if (id == hex) + { + return version; + } + } + + return null; + } + +} + +/// <summary> +/// The commit a feature's branch points to, and what of the feature it holds. +/// </summary> +/// <param name="Commit">The commit</param> +/// <param name="Revision">The save of the feature's files it holds - a newer save needs a commit of its own</param> +/// <param name="Base">The newest version it contains - a feature moved past it needs a commit bringing that in</param> +public sealed record FeatureTip(string Commit, int Revision, int Base); + +/// <summary> +/// A commit as it is kept: exactly as it was made or pushed, its tree, and +/// what each file of the tree is to the lambda. +/// </summary> +/// <param name="Data">The commit, base64 - its bytes are its identity, so they are never made again</param> +/// <param name="Tree">Every file of its tree</param> +/// <param name="Files"> +/// The files of the lambda it holds, in the lambda's order - or nothing for a commit somebody pushed that was no +/// state of the lambda, because it was only on the way to one +/// </param> +/// <param name="Version">The version it was saved as, if it was one - which ends what it keeps of the history below it</param> +public sealed record StoredCommit(string Data, IReadOnlyList<StoredEntry> Tree, IReadOnlyList<StoredFile>? Files, int? Version = null) +{ + + public GitCommit Parse() => GitCommit.Parse(Convert.FromBase64String(Data)); + + /// <summary> + /// The file of the tree at a path, if there is one. + /// </summary> + public StoredEntry? At(string path) => Tree.FirstOrDefault(e => e.Path == path); + +} + +/// <summary> +/// A file of a commit's tree. Always a regular file: a lambda holds nothing else. +/// </summary> +/// <param name="Path">Where it is</param> +/// <param name="Blob">The blob holding what it holds</param> +public sealed record StoredEntry(string Path, string Blob); + +/// <summary> +/// A file of the lambda, as a commit holds it. +/// </summary> +/// <param name="Name">What the lambda calls it</param> +/// <param name="Path">Where it is in the tree</param> +/// <param name="Blob"> +/// What it holds - for the snippet what lambda.cs holds, which is not in the tree: the tree holds it as Project.cs +/// </param> +/// <param name="Binary">Whether the lambda keeps it as base64 rather than as text</param> +public sealed record StoredFile(string Name, string Path, string Blob, bool Binary); + +/// <summary> +/// The lambda a repository is read or written as. +/// </summary> +/// <param name="Id">The identity its commits are kept under</param> +/// <param name="PrivateKey">Its editor key, to ask the other services with - never told anybody</param> +/// <param name="PublicKey">Its public key, to name it by</param> +/// <param name="Tier">Its tier</param> +/// <param name="Project">What the platform's files of a commit made now say</param> +/// <param name="Origin">The address it was reached at, for the addresses a push answers with</param> +public sealed record GitLambda(long Id, string PrivateKey, string PublicKey, string Tier, RepositoryProject Project, string Origin); diff --git a/src/GenHTTP.Lambda/Services/Git/GitPushes.cs b/src/GenHTTP.Lambda/Services/Git/GitPushes.cs new file mode 100644 index 00000000..c9a6c74b --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Git/GitPushes.cs @@ -0,0 +1,704 @@ +using GenHTTP.Modules.Git; + +using GenHTTP.Lambda.Data.Entities; +using GenHTTP.Lambda.Services.Deployment; +using GenHTTP.Lambda.Services.Deployment.Model; +using GenHTTP.Lambda.Services.Features; +using GenHTTP.Lambda.Services.Meta; +using GenHTTP.Lambda.Services.Meta.Model; +using GenHTTP.Lambda.Services.Settings; + +using Microsoft.Extensions.Logging; + +namespace GenHTTP.Lambda.Services.Git; + +/// <summary> +/// What a <c>git push</c> to a lambda does: commits pushed to main become its +/// versions, a branch pushed becomes a feature, and a branch deleted takes +/// its feature with it. +/// </summary> +/// <remarks> +/// Everything goes through the services every other door goes through - a +/// version is saved, a feature started, saved, previewed and merged the way +/// the editor or an agent does it - so no rule of theirs is skipped, and a +/// refusal says what they say. What is git's own is decided here: main moves +/// on one version at a time and is never rewritten, and the files the +/// platform puts around a lambda are not the lambda's to change (see +/// <see cref="ProjectTree"/>). +/// +/// What happened is told back as the push goes, in the lines git shows after +/// <c>remote:</c> - never with the editor key, which whoever pushed has, and +/// whoever reads a build log of theirs should not. Options a push carries +/// (<c>git push -o deploy</c>, <c>-o merge</c>) put what was pushed online. +/// </remarks> +public sealed class GitPushes(GitStore store, GitHistory history, IMetaService meta, IFeatureService features, ILimitsService limits, + ILogger<GitPushes> logger) +{ + private const string Deploy = "deploy"; + + private const string Merge = "merge"; + + /// <summary> + /// How many problems the compiler found are told before the rest is counted. + /// </summary> + private const int MostProblems = 15; + + #region Pushing + + /// <summary> + /// Applies a push, in the lambda's turn to be written. + /// </summary> + /// <param name="index">Which commit is what, as it stands at the start of the turn - kept up to date as it goes</param> + public async ValueTask PushAsync(GitLambda lambda, GitIndex index, GitPush push) + { + var said = new Said(push); + + foreach (var option in push.Options.Where(o => o is not (Deploy or Merge))) + { + await said.LineAsync($"There is no option '{option}' here: -o deploy and -o merge are."); + } + + var commits = new Commits(store, history, lambda.Id, push); + + foreach (var update in push.Updates) + { + try + { + if (update.IsTag) + { + Refuse(lambda, update, "Tags are the versions, and the platform sets them: v1, v2, and so on. Push a branch instead."); + } + else if (!update.IsBranch) + { + Refuse(lambda, update, "Only branches are pushed here: main for versions, any other for a feature."); + } + else if (update.ShortName == FeatureBranches.Main) + { + await MainAsync(lambda, index, update, commits, said, push.Options.Contains(Deploy)); + } + else + { + await FeatureAsync(lambda, index, update, commits, said, push.Options.Contains(Deploy), push.Options.Contains(Merge)); + } + } + catch (LambdaException e) + { + // what the services refuse is what the push is told, in their words + if (update.Status == GitUpdateStatus.Pending) + { + Refuse(lambda, update, e.Message); + } + else + { + await said.LineAsync(e.Message); + } + } + } + } + + /// <summary> + /// Refuses every update of a push to a lambda nobody may change. + /// </summary> + public void RefuseAll(GitLambda lambda, GitPush push, string reason) + { + foreach (var update in push.Updates) + { + Refuse(lambda, update, reason); + } + } + + #endregion + + #region Versions + + /// <summary> + /// Commits pushed to main: a version each, the newest compiled first. + /// </summary> + private async ValueTask MainAsync(GitLambda lambda, GitIndex index, GitReferenceUpdate update, Commits commits, Said said, bool deploy) + { + if (update.IsDelete) + { + Refuse(lambda, update, "main holds the versions of the lambda and cannot be deleted."); + return; + } + + if (index.Newest is not { } newest || update.OldId != newest) + { + Refuse(lambda, update, "A version was saved since you fetched - in the editor, or by an agent. git pull --rebase, and push again."); + return; + } + + if (!update.IsFastForward) + { + Refuse(lambda, update, "A version never changes, so main is never rewritten. git pull --rebase, and push again without --force."); + return; + } + + var allowed = limits.Of(Enum.Parse<LambdaTier>(lambda.Tier)).Versions; + + // from the old tip to the new one, one commit after another + var chain = new List<GitObjectId>(); + + for (var at = update.NewId; at != update.OldId;) + { + var commit = commits.Find(at); + + if (commit == null || index.VersionOf(at) != null) + { + Refuse(lambda, update, "This is not based on main as it is. git pull --rebase, and push again."); + return; + } + + if (commit.Parents.Count != 1) + { + Refuse(lambda, update, $"{Short(at)} is a merge commit, and main is one version after another. " + + "Rebase instead of merging (git pull --rebase, or git rebase origin/main), and push again."); + return; + } + + chain.Insert(0, at); + + if (chain.Count > allowed) + { + Refuse(lambda, update, $"Every commit pushed to main becomes a version, and a lambda keeps {allowed}: this push holds more. " + + "Squash them into fewer (git rebase -i origin/main), and push again."); + return; + } + + at = commit.Parents[0]; + } + + // each commit read as the lambda it holds, against the one before it + var steps = new List<(GitObjectId Id, GitCommit Commit, IReadOnlyList<GitFile> Tree, ReadTree Read)>(); + + var parent = store.ReadCommit(lambda.Id, update.OldId); + + foreach (var id in chain) + { + var tree = commits.TreeOf(id); + + var read = await ProjectTree.ReadAsync(tree, parent != null ? [parent] : [], blob => store.ReadBlob(lambda.Id, blob)); + + if (read.Files == null) + { + Refuse(lambda, update, chain.Count > 1 ? $"{Short(id)}: {read.Complaint}" : read.Complaint!); + return; + } + + steps.Add((id, commits.Find(id)!, tree, read)); + + parent = new StoredCommit(string.Empty, [.. tree.Select(f => new StoredEntry(f.Path, f.Id!.Value.ToString()))], read.Stored); + } + + // a version is only made of what compiles, as when a feature is merged + var tip = steps[^1]; + + var check = await meta.CheckAsync(lambda.PrivateKey, LambdaSource.Serialize(tip.Read.Files!)); + + if (!check.Success) + { + await ProblemsAsync(said, check.Diagnostics, tip.Read); + Refuse(lambda, update, "It does not compile, so it is no version. Fix what the lines above say, and push again."); + return; + } + + var saved = new List<LambdaVersionInfo>(); + + var after = index.Versions.Keys.Max(); + + foreach (var step in steps) + { + var (change, specification) = Message(step.Commit); + + LambdaVersionInfo version; + + try + { + version = meta.Save(lambda.PrivateKey, LambdaSource.Serialize(step.Read.Files!), new VersionNote(specification, change, VersionOrigins.Git), after); + } + catch (LambdaException e) when (saved.Count > 0) + { + // the versions before it are saved, and main is where they took it + await said.LineAsync($"Saved {Versions(saved)}, and stopped at {Short(step.Id)}: {e.Message}"); + Refuse(lambda, update, "Not all of it was saved. git pull --rebase, and push again."); + return; + } + + await history.KeepAsync(lambda.Id, step.Commit, step.Tree, step.Read, version.Version); + + index.Versions[version.Version] = step.Id.ToString(); + + store.WriteIndex(lambda.Id, index); + + saved.Add(version); + + after = version.Version; + } + + update.Accept(); + + logger.LogInformation("Pushed lambda {Lambda} versions {From} to {To}", lambda.PublicKey, saved[0].Version, saved[^1].Version); + + await said.LineAsync(saved.Count == 1 + ? $"Saved version {saved[0].Version}: {saved[0].Change ?? "(no message)"}" + : $"Saved {Versions(saved)}, the newest: {saved[^1].Change ?? "(no message)"}"); + + await MergedAsync(lambda, index, chain, saved[^1].Version, said); + + if (deploy) + { + await DeployAsync(lambda, saved[^1].Version, said); + } + else + { + await NotOnlineAsync(lambda, saved[^1].Version, said); + } + } + + /// <summary> + /// Removes the features main now holds: their branch was pushed to it, so + /// they are merged, as merging them would have done. + /// </summary> + private async ValueTask MergedAsync(GitLambda lambda, GitIndex index, List<GitObjectId> chain, int version, Said said) + { + var made = chain.Select(c => c.ToString()).ToHashSet(StringComparer.Ordinal); + + foreach (var (key, tip) in index.Features.Where(f => made.Contains(f.Value.Commit)).ToList()) + { + var name = features.Delete(lambda.PrivateKey, key); + + index.Features.Remove(key); + + store.WriteIndex(lambda.Id, index); + + logger.LogInformation("Merged feature '{Feature}' of lambda {Lambda} by push version {Version}", name, lambda.PublicKey, version); + + await said.LineAsync($"The feature '{name}' is in main now, so it is merged: its preview and its copy of the data are gone, the lambda keeps its own."); + } + } + + #endregion + + #region Features + + /// <summary> + /// A branch pushed: its feature started, saved or deleted. + /// </summary> + private async ValueTask FeatureAsync(GitLambda lambda, GitIndex index, GitReferenceUpdate update, Commits commits, Said said, bool deploy, bool merge) + { + var branch = update.ShortName; + + var feature = features.List(lambda.PrivateKey).FirstOrDefault(f => f.Branch == branch); + + if (update.IsDelete) + { + if (feature == null) + { + Refuse(lambda, update, $"There is no feature at '{branch}'."); + return; + } + + var name = features.Delete(lambda.PrivateKey, feature.Key); + + index.Features.Remove(feature.Key); + + store.WriteIndex(lambda.Id, index); + + update.Accept(); + + logger.LogInformation("Deleted feature '{Feature}' of lambda {Lambda} by push", name, lambda.PublicKey); + + await said.LineAsync($"Deleted the feature '{name}', with its preview and its copy of the data."); + return; + } + + var tipBefore = feature != null ? index.Features.GetValueOrDefault(feature.Key) : null; + + if (feature != null && (tipBefore == null || GitObjectId.Parse(tipBefore.Commit) != update.OldId)) + { + Refuse(lambda, update, $"The feature '{feature.Name}' was changed since you fetched - in the editor, or by an agent. git pull --rebase, and push again."); + return; + } + + if (Base(index, update.NewId, commits) is not { } based) + { + Refuse(lambda, update, "A branch starts from main or one of its versions, and this one does not. " + + $"git switch -c {branch} origin/main, bring your commits over, and push again."); + return; + } + + // what its commits make of the lambda, the new ones parents first + ReadTree? read = null; + + var kept = new List<(GitCommit Commit, IReadOnlyList<GitFile> Tree, ReadTree Read)>(); + + foreach (var revision in update.Revisions) + { + var tree = commits.TreeOf(revision.Commit.Id); + + var parents = revision.Commit.Parents.Select(p => commits.Stored(p)).OfType<StoredCommit>().ToList(); + + var mine = await ProjectTree.ReadAsync(tree, parents, blob => store.ReadBlob(lambda.Id, blob)); + + commits.Provide(revision.Commit.Id, tree, mine.Files != null ? mine.Stored : null); + + kept.Add((revision.Commit, tree, mine)); + + if (revision.Commit.Id == update.NewId) + { + read = mine; + } + } + + if (read == null) + { + // a commit the lambda has already: a version, or another feature's + var known = commits.Stored(update.NewId); + + read = known?.Files != null + ? await ProjectTree.ReadAsync(commits.TreeOf(update.NewId), [known], blob => store.ReadBlob(lambda.Id, blob)) + : ReadTree.Refused("This commit holds no state of the lambda."); + } + + if (read.Files == null) + { + Refuse(lambda, update, read.Complaint!); + return; + } + + var code = LambdaSource.Serialize(read.Files); + + var (change, _) = Message(commits.Find(update.NewId)!); + + var started = feature == null; + + if (feature == null) + { + feature = await features.CreateAsync(lambda.PrivateKey, new FeatureDraft(Name(branch), Base: based, Origin: VersionOrigins.Git, Branch: branch)); + } + else if (feature.Base != based) + { + feature = features.Update(lambda.PrivateKey, feature.Key, new FeatureUpdate(Base: based)); + } + + try + { + feature = features.Save(lambda.PrivateKey, feature.Key, code, new VersionNote(Change: feature.Change == null ? change : null, Origin: VersionOrigins.Git), + after: feature.Revision); + } + catch (LambdaException) when (started) + { + // refused, so the branch is not there: neither is the feature started for it + features.Delete(lambda.PrivateKey, feature.Key); + throw; + } + + if (started) + { + logger.LogInformation("Created feature '{Feature}' of lambda {Lambda} by push base {Version}", feature.Name, lambda.PublicKey, based); + + await said.LineAsync($"Started the feature '{feature.Name}' from version {based}, with a copy of the lambda's data to try it on."); + } + + foreach (var (commit, tree, mine) in kept) + { + await history.KeepAsync(lambda.Id, commit, tree, mine.Files != null ? mine : null, null); + } + + index.Features[feature.Key] = new FeatureTip(update.NewId.ToString(), feature.Revision, based); + + store.WriteIndex(lambda.Id, index); + + update.Accept(); + + logger.LogInformation("Pushed feature '{Feature}' of lambda {Lambda} base {Version}", feature.Name, lambda.PublicKey, based); + + if (!started) + { + await said.LineAsync($"Saved the feature '{feature.Name}', based on version {based}{(feature.Mergeable ? string.Empty : $" - the newest is {feature.Newest}")}."); + } + + if (merge) + { + await MergeAsync(lambda, index, feature, read, said, deploy); + } + else + { + // a preview is no visitor's business, so every push puts it online + await PreviewAsync(lambda, feature, read, said); + } + } + + /// <summary> + /// The newest version a commit holds - the one a feature at it is based + /// on - or nothing where it holds none. + /// </summary> + private static int? Base(GitIndex index, GitObjectId tip, Commits commits) + { + int? found = null; + + var seen = new HashSet<GitObjectId>(); + + var pending = new Stack<GitObjectId>([tip]); + + while (pending.TryPop(out var id) && seen.Count < 100_000) + { + if (!seen.Add(id)) + { + continue; + } + + if (index.VersionOf(id) is { } version) + { + // the versions below it are older + found = Math.Max(found ?? 0, version); + continue; + } + + foreach (var parent in commits.Find(id)?.Parents ?? []) + { + pending.Push(parent); + } + } + + return found; + } + + private async ValueTask PreviewAsync(GitLambda lambda, FeatureInfo feature, ReadTree read, Said said) + { + var result = await features.DeployAsync(lambda.PrivateKey, feature.Key); + + logger.LogInformation(result.Success ? "Deployed feature '{Feature}' of lambda {Lambda}" : "Failed to deploy feature '{Feature}' of lambda {Lambda}", + feature.Name, lambda.PublicKey); + + if (!result.Success) + { + await ProblemsAsync(said, result.Diagnostics, read); + await said.LineAsync("It does not compile yet, so its preview stays as it was. Fix what the lines above say, and push again."); + return; + } + + await said.LineAsync($"Its preview is online at {Preview(lambda, feature)}"); + + await said.LineAsync(feature.Mergeable + ? $"To make it the next version: git push origin {feature.Branch}:main (a version for each of its commits), -o merge with its next push (one version), or merge it in the editor." + : $"Before it can become a version, bring main in (git rebase origin/main) - version {feature.Newest} was saved since it began."); + } + + private async ValueTask MergeAsync(GitLambda lambda, GitIndex index, FeatureInfo feature, ReadTree read, Said said, bool deploy) + { + if (!feature.Mergeable) + { + await said.LineAsync($"It cannot be merged yet: it holds version {feature.Base}, and the newest is {feature.Newest}. " + + "Bring main in (git rebase origin/main, or git merge origin/main), and push it with -o merge again."); + return; + } + + var merged = await features.MergeAsync(lambda.PrivateKey, feature.Key, new VersionNote(Origin: VersionOrigins.Git), deploy); + + if (!merged.Merged) + { + await ProblemsAsync(said, merged.Diagnostics, read); + await said.LineAsync("It does not compile, so it was not merged. Fix what the lines above say, and push it with -o merge again."); + return; + } + + index.Features.Remove(feature.Key); + + store.WriteIndex(lambda.Id, index); + + logger.LogInformation("Merged feature '{Feature}' of lambda {Lambda} by push version {Version}", merged.Name, lambda.PublicKey, merged.Version!.Version); + + await said.LineAsync($"Merged it as version {merged.Version.Version}: the feature, its preview and its copy of the data are gone, the lambda keeps its own."); + + if (merged.Deployment is { } deployment) + { + await DeployedAsync(lambda, deployment, merged.Version.Version, said); + } + else + { + await NotOnlineAsync(lambda, merged.Version.Version, said); + } + + await said.LineAsync("git switch main, git pull and git fetch --prune bring your clone up to date."); + } + + #endregion + + #region Helpers + + /// <summary> + /// Says what is online instead of a version just made, and how it goes online. + /// </summary> + private async ValueTask NotOnlineAsync(GitLambda lambda, int version, Said said) + { + var online = meta.Get(lambda.PrivateKey)?.ActiveVersion; + + await said.LineAsync($"{(online == null ? "Nothing is online yet" : $"Version {online} is online")}. To put version {version} online, deploy it in the editor " + + $"or POST {lambda.Project.Home}/api/v1/lambdas/<editor key>/deployment/start; -o deploy does it with a push."); + } + + private async ValueTask DeployAsync(GitLambda lambda, int version, Said said) + { + var result = await meta.DeployAsync(lambda.PrivateKey, version, VersionOrigins.Git); + + await DeployedAsync(lambda, result, version, said); + } + + private async ValueTask DeployedAsync(GitLambda lambda, DeploymentResult result, int version, Said said) + { + logger.LogInformation(result.Success ? "Deployed lambda {Lambda} version {Version}" : "Failed to deploy lambda {Lambda} version {Version}", + lambda.PublicKey, version); + + if (result.Success) + { + await said.LineAsync($"Version {version} is online at {lambda.Project.Address}"); + } + else + { + await ProblemsAsync(said, result.Diagnostics, null); + await said.LineAsync($"Version {version} could not go online; what was online stays."); + } + } + + /// <summary> + /// What the compiler found, where it is in the repository. + /// </summary> + private static async ValueTask ProblemsAsync(Said said, IReadOnlyList<CompilationDiagnostic> diagnostics, ReadTree? read) + { + var errors = diagnostics.Where(d => d.Severity == "Error").ToList(); + + var unwrapped = read?.Project != null ? ProjectSnippet.Unwrap(read.Project) : null; + + foreach (var diagnostic in errors.Take(MostProblems)) + { + await said.LineAsync($" {Where(diagnostic, read, unwrapped)}: error {diagnostic.Id}: {diagnostic.Message}"); + } + + if (errors.Count > MostProblems) + { + await said.LineAsync($" and {errors.Count - MostProblems} more"); + } + } + + private static string Where(CompilationDiagnostic diagnostic, ReadTree? read, SnippetUnwrapped? unwrapped) + { + var file = diagnostic.File ?? LambdaSource.EntryName; + + if (diagnostic.Line <= 0) + { + return file == LambdaSource.EntryName ? ProjectPaths.Snippet : ProjectPaths.Of(file); + } + + if (file == LambdaSource.EntryName) + { + var (line, column) = unwrapped?.Locate(diagnostic.Line, diagnostic.Column) ?? (diagnostic.Line, diagnostic.Column); + + return $"{ProjectPaths.Snippet}({line},{column})"; + } + + var path = read?.Stored.FirstOrDefault(f => f.Name == file)?.Path ?? ProjectPaths.Of(file); + + return $"{path}({diagnostic.Line},{diagnostic.Column})"; + } + + /// <remarks> + /// Said once, by git, in the line it prints for the branch that was refused. + /// </remarks> + private void Refuse(GitLambda lambda, GitReferenceUpdate update, string reason) + { + update.Reject(OneLine(reason)); + + logger.LogInformation("Refused push of {Reference} to lambda {Lambda}", update.ShortName, lambda.PublicKey); + } + + /// <summary> + /// What a commit says it changes, and the rest of its message: what the + /// user wanted, kept with the version as its specification. + /// </summary> + private static (string? Change, string? Specification) Message(GitCommit commit) + { + var subject = commit.Subject.Trim(); + + var newline = commit.Message.IndexOf('\n'); + + var rest = newline < 0 ? null : commit.Message[(newline + 1)..].Trim(); + + return (subject.Length == 0 ? null : subject, string.IsNullOrEmpty(rest) ? null : rest); + } + + private static string Name(string branch) => branch.Length <= FeatureService.MaxName ? branch : branch[..FeatureService.MaxName]; + + private static string Preview(GitLambda lambda, FeatureInfo feature) => $"{lambda.Origin}{feature.Path}"; + + private static string Short(GitObjectId id) => id.ToString()[..7]; + + private static string Versions(List<LambdaVersionInfo> saved) => saved.Count == 1 ? $"version {saved[0].Version}" : $"versions {saved[0].Version} to {saved[^1].Version}"; + + private static string OneLine(string text) => string.Join(' ', text.Split(['\r', '\n'], StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries)); + + #endregion + + #region Types + + /// <summary> + /// What is told back to whoever pushes, a line at a time. + /// </summary> + private sealed class Said(GitPush push) + { + private bool _started; + + public async ValueTask LineAsync(string line) + { + if (!_started) + { + _started = true; + + await push.MessageAsync(string.Empty); + } + + await push.MessageAsync(line); + } + } + + /// <summary> + /// The commits a push reaches: those it brings, and those the lambda has. + /// </summary> + private sealed class Commits(GitStore store, GitHistory history, long lambdaId, GitPush push) + { + private readonly Dictionary<GitObjectId, GitRevision> _pushed = push.Updates.SelectMany(u => u.Revisions) + .DistinctBy(r => r.Commit.Id) + .ToDictionary(r => r.Commit.Id); + + private readonly Dictionary<GitObjectId, StoredCommit> _provided = []; + + public GitCommit? Find(GitObjectId id) + => _pushed.TryGetValue(id, out var revision) ? revision.Commit : store.ReadCommit(lambdaId, id)?.Parse(); + + public IReadOnlyList<GitFile> TreeOf(GitObjectId id) + { + if (_pushed.TryGetValue(id, out var revision)) + { + return revision.Tree.Files; + } + + var stored = store.ReadCommit(lambdaId, id) ?? throw new InvalidOperationException($"Commit {id} is not kept."); + + return history.TreeOf(lambdaId, stored.Tree).Files; + } + + /// <summary> + /// A commit as it is kept - or as it will be, for one this push brings. + /// </summary> + public StoredCommit? Stored(GitObjectId id) => _provided.GetValueOrDefault(id) ?? store.ReadCommit(lambdaId, id); + + /// <summary> + /// Remembers what the lambda makes of a commit this push brings, for the commits on top of it. + /// </summary> + public void Provide(GitObjectId id, IReadOnlyList<GitFile> tree, IReadOnlyList<StoredFile>? files) + => _provided[id] = new StoredCommit(string.Empty, [.. tree.Select(f => new StoredEntry(f.Path, f.Id!.Value.ToString()))], files); + } + + #endregion + +} diff --git a/src/GenHTTP.Lambda/Services/Git/GitService.cs b/src/GenHTTP.Lambda/Services/Git/GitService.cs new file mode 100644 index 00000000..a4774e85 --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Git/GitService.cs @@ -0,0 +1,162 @@ +using GenHTTP.Modules.Git; + +using GenHTTP.Lambda.Configuration; +using GenHTTP.Lambda.Data.Entities; +using GenHTTP.Lambda.Infrastructure; +using GenHTTP.Lambda.Services.Deployment; +using GenHTTP.Lambda.Services.Features; +using GenHTTP.Lambda.Services.Meta; +using GenHTTP.Lambda.Services.Meta.Model; +using GenHTTP.Lambda.Services.Source; + +namespace GenHTTP.Lambda.Services.Git; + +/// <summary> +/// Serves every lambda as a git repository. +/// </summary> +/// <remarks> +/// Finds the lambda, takes its turn, and leaves the rest to the units beside +/// it: <see cref="GitHistory"/> writes down the commits the versions and the +/// features are, <see cref="GitPushes"/> makes versions and features of what +/// is pushed, <see cref="GitStore"/> keeps the commits and their files, and +/// <see cref="ProjectTree"/> says what a lambda is as a tree and back. +/// +/// A lambda is read and written one request at a time: making its commits +/// and applying a push both decide what the next commit follows. The turn is +/// waited for asynchronously, since it is held across reading every version +/// of a lambda read for the first time, and across compiling what is pushed. +/// Pushes are few, but unpacked in memory and compiled, so only two are +/// applied at once on the whole server. +/// </remarks> +public sealed class GitService(IMetaService meta, IFeatureService features, ISourceService sources, GitStore store, GitHistory history, GitPushes pushes, + LambdaOptions options) : IGitService +{ + private readonly SemaphoreSlim[] _turns = [.. Enumerable.Range(0, 64).Select(_ => new SemaphoreSlim(1, 1))]; + + private readonly SemaphoreSlim _pushing = new(2, 2); + + #region Functionality + + public async ValueTask<IGitRepository?> OpenAsync(string privateKey, string origin, CancellationToken cancellation = default) + { + if (meta.Get(privateKey) is not { } info || meta.GetId(privateKey) is not { } id) + { + return null; + } + + var lambda = Describe(info, id, origin); + + var index = await SyncAsync(lambda, true, cancellation); + + var branches = features.List(privateKey).ToDictionary(f => f.Key, f => f.Branch); + + return new WritableLambdaRepository(store, history, id, index, key => branches.GetValueOrDefault(key), push => PushAsync(lambda, push)); + } + + public async ValueTask<IGitRepository?> ReadAsync(string publicKey, string origin, CancellationToken cancellation = default) + { + if (sources.GetProject(publicKey) == null || meta.GetPrivateKey(publicKey) is not { } privateKey) + { + return null; + } + + if (meta.Get(privateKey) is not { } info || meta.GetId(privateKey) is not { } id) + { + return null; + } + + var index = await SyncAsync(Describe(info, id, origin), false, cancellation); + + return new LambdaRepository(store, history, id, index, false, _ => null); + } + + #endregion + + #region Helpers + + /// <summary> + /// Brings the commits of a lambda up to date, in its turn and away from the reactor. + /// </summary> + private async ValueTask<GitIndex> SyncAsync(GitLambda lambda, bool withFeatures, CancellationToken cancellation) + { + var turn = Turn(lambda.Id); + + await turn.WaitAsync(cancellation); + + try + { + return await Offload.Run(() => history.SyncAsync(lambda, withFeatures), cancellation); + } + finally + { + turn.Release(); + } + } + + /// <summary> + /// Applies a push, in the lambda's turn and after the commits were + /// brought up to date in it - so it is checked against what is there now, + /// not against what was there when the client fetched. + /// </summary> + /// <remarks> + /// Already away from the reactor: the server applies a push where it + /// writes its answer, which the route hands to the pool. + /// </remarks> + private async ValueTask PushAsync(GitLambda lambda, GitPush push) + { + if (lambda.Tier == nameof(LambdaTier.Demo)) + { + pushes.RefuseAll(lambda, push, LambdaGuard.ReadOnly(lambda.PublicKey)); + return; + } + + await _pushing.WaitAsync(); + + try + { + var turn = Turn(lambda.Id); + + await turn.WaitAsync(); + + try + { + var index = await history.SyncAsync(lambda, true); + + await pushes.PushAsync(lambda, index, push); + } + finally + { + turn.Release(); + } + } + finally + { + _pushing.Release(); + } + } + + /// <summary> + /// The lambda as its repository says it: where it runs, where the + /// installation is, and the license its source is published under. + /// </summary> + private GitLambda Describe(LambdaInfo lambda, long id, string origin) + { + var home = (options.PublicUrl ?? origin).TrimEnd('/'); + + var address = lambda.Domain != null && lambda.Tier == nameof(LambdaTier.Premium) + ? $"https://{lambda.Domain}/" + : $"{home}/lambda/{lambda.PublicKey}/"; + + var license = sources.Get(lambda.PrivateKey) is { Published: true } source && SourceLicenses.Find(source.License) is { } found + ? new RepositoryLicense(found, DateTime.UtcNow.Year, SourceLicenses.Holder(source.Author, lambda.PublicKey)) + : null; + + return new GitLambda(id, lambda.PrivateKey, lambda.PublicKey, lambda.Tier, new RepositoryProject(lambda.PublicKey, address, home, license), + origin.TrimEnd('/')); + } + + private SemaphoreSlim Turn(long lambdaId) => _turns[(int)((ulong)lambdaId % (ulong)_turns.Length)]; + + #endregion + +} diff --git a/src/GenHTTP.Lambda/Services/Git/GitStore.cs b/src/GenHTTP.Lambda/Services/Git/GitStore.cs new file mode 100644 index 00000000..0a471c03 --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Git/GitStore.cs @@ -0,0 +1,250 @@ +using System.Text.Json; + +using GenHTTP.Modules.Git; + +using GenHTTP.Lambda.Configuration; + +namespace GenHTTP.Lambda.Services.Git; + +/// <summary> +/// Keeps the commits a lambda is read as with git, below +/// <c>{data}/git/{lambda}</c>: each commit as it was made or pushed, the +/// files they hold, and which commit is which version and feature. +/// </summary> +/// <remarks> +/// A commit is its bytes - its id is their hash - so it is made once and kept, +/// never made again from the version: a newer platform would make it a little +/// differently, and every clone would see its history rewritten. Pushed ones +/// are kept as they came, which is what a client that pushed them expects. +/// +/// The files are kept once each, by the id git gives them, so a version that +/// changes one file adds one file. They are what the versions hold already, +/// and kept here all the same: a commit has to be served as it was for as +/// long as it is reachable, and the files of a feature change in place. +/// +/// Written through a file beside the one it replaces, so a write cut short +/// leaves the one before. What nothing reaches any more is collected once a +/// version is pruned or a feature goes; until then it is merely kept. +/// </remarks> +public sealed class GitStore(LambdaOptions options) +{ + private static readonly JsonSerializerOptions Json = new(JsonSerializerDefaults.Web); + + #region Index + + public GitIndex ReadIndex(long lambda) + { + var file = Path.Combine(Root(lambda), "index.json"); + + if (!File.Exists(file)) + { + return GitIndex.Empty(); + } + + var stored = JsonSerializer.Deserialize<StoredIndex>(File.ReadAllText(file), Json); + + return new GitIndex(stored?.Versions ?? [], stored?.Features ?? []); + } + + public void WriteIndex(long lambda, GitIndex index) + => Write(Path.Combine(Root(lambda), "index.json"), JsonSerializer.SerializeToUtf8Bytes(new StoredIndex(index.Versions, index.Features), Json)); + + private sealed record StoredIndex(Dictionary<int, string>? Versions, Dictionary<string, FeatureTip>? Features); + + #endregion + + #region Commits + + public StoredCommit? ReadCommit(long lambda, GitObjectId id) + { + var file = CommitFile(lambda, id); + + try + { + return File.Exists(file) ? JsonSerializer.Deserialize<StoredCommit>(File.ReadAllText(file), Json) : null; + } + catch (FileNotFoundException) + { + // collected between looking and reading + return null; + } + } + + public void WriteCommit(long lambda, GitObjectId id, StoredCommit commit) + => Write(CommitFile(lambda, id), JsonSerializer.SerializeToUtf8Bytes(commit, Json)); + + #endregion + + #region Blobs + + public bool HasBlob(long lambda, GitObjectId id) => File.Exists(BlobFile(lambda, id)); + + /// <summary> + /// Keeps what a file holds, unless it is kept already. + /// </summary> + public void WriteBlob(long lambda, GitObjectId id, ReadOnlySpan<byte> content) + { + var file = BlobFile(lambda, id); + + if (!File.Exists(file)) + { + Write(file, content); + } + } + + /// <summary> + /// What a file holds, read in place: it is bounded by what a version may + /// hold, and read where the response is written, away from the reactor. + /// </summary> + public byte[] ReadBlob(long lambda, GitObjectId id) + { + try + { + return File.ReadAllBytes(BlobFile(lambda, id)); + } + catch (Exception e) when (e is FileNotFoundException or DirectoryNotFoundException) + { + throw new InvalidOperationException($"The file {id} of lambda #{lambda} is no longer kept.", e); + } + } + + #endregion + + #region Collection + + /// <summary> + /// Removes the commits and the files nothing reaches any more. + /// </summary> + /// <remarks> + /// Reached is a version that is still there, the tip of a feature that is + /// still there, and what lies below a tip up to the versions it starts + /// from. Below a version nothing is followed: the versions before it are + /// kept for as long as they are versions, and a version that was pruned is + /// gone from the history - a clone is told the history starts later. + /// </remarks> + /// <returns>How many commits were removed</returns> + public int Collect(long lambda, GitIndex index) + { + var root = Root(lambda); + + var commits = Path.Combine(root, "commits"); + + if (!Directory.Exists(commits)) + { + return 0; + } + + var versions = index.Versions.Values.ToHashSet(StringComparer.Ordinal); + + var live = new HashSet<string>(StringComparer.Ordinal); + + var blobs = new HashSet<string>(StringComparer.Ordinal); + + var pending = new Stack<string>(versions.Concat(index.Features.Values.Select(f => f.Commit))); + + while (pending.TryPop(out var id)) + { + if (!live.Add(id) || ReadCommit(lambda, GitObjectId.Parse(id)) is not { } commit) + { + continue; + } + + foreach (var entry in commit.Tree) + { + blobs.Add(entry.Blob); + } + + foreach (var file in commit.Files ?? []) + { + blobs.Add(file.Blob); + } + + if (commit.Version != null) + { + continue; + } + + foreach (var parent in commit.Parse().Parents) + { + var hex = parent.ToString(); + + // a version below a feature is reached as long as it is one + if (!versions.Contains(hex) && ReadCommit(lambda, parent) is { Version: not null }) + { + continue; + } + + pending.Push(hex); + } + } + + var removed = 0; + + foreach (var file in Directory.EnumerateFiles(commits, "*.json")) + { + if (!live.Contains(Path.GetFileNameWithoutExtension(file))) + { + Delete(file); + removed++; + } + } + + var stored = Path.Combine(root, "blobs"); + + if (Directory.Exists(stored)) + { + foreach (var file in Directory.EnumerateFiles(stored, "*", SearchOption.AllDirectories)) + { + var id = Path.GetFileName(Path.GetDirectoryName(file)) + Path.GetFileName(file); + + if (!blobs.Contains(id)) + { + Delete(file); + } + } + } + + return removed; + } + + #endregion + + #region Helpers + + private string Root(long lambda) => Path.Combine(options.GitDirectory, lambda.ToString()); + + private string CommitFile(long lambda, GitObjectId id) => Path.Combine(Root(lambda), "commits", $"{id}.json"); + + private string BlobFile(long lambda, GitObjectId id) + { + var hex = id.ToString(); + + return Path.Combine(Root(lambda), "blobs", hex[..2], hex[2..]); + } + + private static void Write(string file, ReadOnlySpan<byte> content) + { + Directory.CreateDirectory(Path.GetDirectoryName(file)!); + + var staging = $"{file}.{Guid.NewGuid():n}.writing"; + + File.WriteAllBytes(staging, content); + + File.Move(staging, file, true); + } + + private static void Delete(string file) + { + try + { + File.Delete(file); + } + catch (IOException) + { + // being read right now: the next collection takes it + } + } + + #endregion + +} diff --git a/src/GenHTTP.Lambda/Services/Git/IGitService.cs b/src/GenHTTP.Lambda/Services/Git/IGitService.cs new file mode 100644 index 00000000..469938ce --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Git/IGitService.cs @@ -0,0 +1,33 @@ +using GenHTTP.Modules.Git; + +namespace GenHTTP.Lambda.Services.Git; + +/// <summary> +/// Every lambda as a git repository: cloned, fetched and pushed to by whoever +/// holds its editor key, and cloned by anybody once its source is published. +/// </summary> +/// <remarks> +/// The repository is the project the lambda is exported as. Its versions are +/// the commits of main, tagged <c>v1</c>, <c>v2</c> and so on, and each of its +/// features is a branch. A commit pushed to main becomes a version, a branch +/// pushed a feature. The data is never in it - nor what only the owner may +/// know - so the commits the owner reads are those a published source shows. +/// </remarks> +public interface IGitService +{ + + /// <summary> + /// The repository of the lambda behind an editor key, to read and push to, + /// or nothing when no lambda has the key. + /// </summary> + /// <param name="origin">The address it was reached at, for the addresses a push answers with</param> + ValueTask<IGitRepository?> OpenAsync(string privateKey, string origin, CancellationToken cancellation = default); + + /// <summary> + /// The published source at a public key, to read, or nothing where none + /// is published. + /// </summary> + /// <param name="origin">The address it was reached at</param> + ValueTask<IGitRepository?> ReadAsync(string publicKey, string origin, CancellationToken cancellation = default); + +} diff --git a/src/GenHTTP.Lambda/Services/Git/LambdaRepository.cs b/src/GenHTTP.Lambda/Services/Git/LambdaRepository.cs new file mode 100644 index 00000000..18db8b88 --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Git/LambdaRepository.cs @@ -0,0 +1,78 @@ +using GenHTTP.Modules.Git; + +namespace GenHTTP.Lambda.Services.Git; + +/// <summary> +/// A lambda as git reads it: main is the newest version, a tag each version, +/// and - for whoever holds the editor key - a branch each feature. +/// </summary> +/// <remarks> +/// Made for one request, from which commit was what when it began; the +/// commits and their files are read from where they are kept as the server +/// asks for them. +/// +/// The published source is the same history without the features: they are +/// the owner's work in progress, so neither their branches nor their commits +/// are there - a commit is only found where it is a version. +/// </remarks> +public class LambdaRepository(GitStore store, GitHistory history, long lambdaId, GitIndex index, bool owner, Func<string, string?> branches) : IGitRepository +{ + + public ValueTask<GitReferences> GetReferencesAsync() + { + var references = new GitReferences().Head("main"); + + if (index.Newest is { } newest) + { + references.Branch("main", newest); + } + + foreach (var (version, commit) in index.Versions.OrderBy(v => v.Key)) + { + references.Tag($"v{version}", GitObjectId.Parse(commit)); + } + + if (owner) + { + foreach (var (key, tip) in index.Features) + { + if (branches(key) is { } branch) + { + references.Branch(branch, GitObjectId.Parse(tip.Commit)); + } + } + } + + return new(references); + } + + public ValueTask<GitCommit?> GetCommitAsync(GitObjectId id) + { + if (!owner && index.VersionOf(id) == null) + { + return new((GitCommit?)null); + } + + return new(store.ReadCommit(lambdaId, id)?.Parse()); + } + + public ValueTask<GitTree> GetTreeAsync(GitCommit commit) + { + var stored = store.ReadCommit(lambdaId, commit.Id) ?? throw new InvalidOperationException($"Commit {commit.Id} of lambda #{lambdaId} is no longer kept."); + + return new(history.TreeOf(lambdaId, stored.Tree)); + } + +} + +/// <summary> +/// The repository of a lambda as whoever holds its editor key reads it, and pushes to. +/// </summary> +public sealed class WritableLambdaRepository(GitStore store, GitHistory history, long lambdaId, GitIndex index, Func<string, string?> branches, + Func<GitPush, ValueTask> push) + : LambdaRepository(store, history, lambdaId, index, true, branches), IWritableGitRepository +{ + + public ValueTask PushAsync(GitPush changes) => push(changes); + +} diff --git a/src/GenHTTP.Lambda/Services/Git/ProjectTree.cs b/src/GenHTTP.Lambda/Services/Git/ProjectTree.cs new file mode 100644 index 00000000..0a0df482 --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Git/ProjectTree.cs @@ -0,0 +1,351 @@ +using System.Text; + +using GenHTTP.Modules.Git; + +using GenHTTP.Lambda.Services.Deployment; +using GenHTTP.Lambda.Services.Deployment.Model; + +namespace GenHTTP.Lambda.Services.Git; + +/// <summary> +/// A lambda as the tree of a commit, and the tree of a pushed commit as a +/// lambda again. +/// </summary> +/// <remarks> +/// The tree is the project the lambda is exported as, made for a repository +/// (see <see cref="ProjectPacker.Repository"/>): the lambda's files where +/// <see cref="ProjectPaths"/> puts them, and the platform's around them. +/// +/// The way back takes the lambda's files and checks the platform's: those +/// are the same in every commit until the platform changes them, so a pushed +/// commit has each as one of its parents has it - otherwise somebody changed +/// what has no effect on the lambda, and is told so rather than ignored. A +/// <c>Project.cs</c> that is a parent's comes back as the lambda.cs that +/// parent was made from, so a push that leaves the code alone leaves it alone +/// to the byte; one that changed it is taken out of the class again. +/// </remarks> +public static class ProjectTree +{ + private static readonly UTF8Encoding StrictUtf8 = new(false, true); + + #region Laying out + + /// <summary> + /// The tree of a commit the platform makes for the given files. + /// </summary> + /// <param name="previous"> + /// The commit it follows, whose Project.cs is kept where lambda.cs did not change - a push may have written it + /// in a way of its own, and putting the snippet back into the class again would change that for nothing + /// </param> + public static LaidTree Lay(IReadOnlyList<LambdaFile> files, RepositoryProject project, StoredCommit? previous) + { + var entries = new List<StoredEntry>(); + + var stored = new List<StoredFile>(); + + var contents = new Dictionary<string, byte[]>(StringComparer.Ordinal); + + var snippet = files.FirstOrDefault(f => f.Name == LambdaSource.EntryName)?.Code ?? string.Empty; + + var snippetBytes = Encoding.UTF8.GetBytes(snippet); + + var snippetBlob = GitObjectId.ForBlob(snippetBytes).ToString(); + + contents[snippetBlob] = snippetBytes; + + // the class it was put into before, if the snippet is what it was + var kept = previous?.Files?.FirstOrDefault(f => f.Name == LambdaSource.EntryName && f.Blob == snippetBlob) is { } same + ? previous.At(same.Path) + : null; + + var binary = files.Where(f => f.Encoding == "base64").Select(f => f.Name).ToHashSet(StringComparer.Ordinal); + + foreach (var file in ProjectPacker.Repository(project, files)) + { + string blob; + + if (file.Name == LambdaSource.EntryName && kept != null) + { + blob = kept.Blob; + } + else + { + blob = GitObjectId.ForBlob(file.Content).ToString(); + + contents[blob] = file.Content; + } + + entries.Add(new StoredEntry(file.Path, blob)); + + if (file.Name != null) + { + stored.Add(new StoredFile(file.Name, file.Path, file.Name == LambdaSource.EntryName ? snippetBlob : blob, binary.Contains(file.Name))); + } + } + + // the lambda's order, which the project does not keep + var order = files.Select((f, i) => (f.Name, i)).ToDictionary(x => x.Name, x => x.i, StringComparer.Ordinal); + + stored.Sort((x, y) => order[x.Name].CompareTo(order[y.Name])); + + entries.Sort((x, y) => string.CompareOrdinal(x.Path, y.Path)); + + return new LaidTree(entries, stored, contents); + } + + #endregion + + #region Reading back + + /// <summary> + /// The lambda a pushed tree holds, or why it holds none. + /// </summary> + /// <param name="tree">The files of the commit</param> + /// <param name="parents">The commits it follows, as far as they are kept</param> + /// <param name="read">Reads a kept file</param> + public static async ValueTask<ReadTree> ReadAsync(IReadOnlyList<GitFile> tree, IReadOnlyList<StoredCommit> parents, Func<GitObjectId, byte[]> read) + { + foreach (var file in tree) + { + if (file.Mode != GitFileMode.Regular) + { + var what = file.Mode == GitFileMode.Symlink ? "a link" : "executable"; + + return ReadTree.Refused($"'{file.Path}' is {what}, and a lambda holds plain files only. " + + (file.Mode == GitFileMode.Symlink ? "Commit the file itself instead." : $"git update-index --chmod=-x {file.Path}, and commit again.")); + } + + if (ProjectPaths.Classify(file.Path).Kind == ProjectPathKind.Foreign) + { + return ReadTree.Refused($"'{file.Path}' has no place in a lambda. Its code is Project.cs and the other .cs files beside it, what it ships " + + "is in assets/, its documentation in docs/ and its tests in tests/; the rest is the platform's. " + + "Keep other files out of commits (.git/info/exclude)."); + } + } + + if (Generated(tree, parents) is { } complaint) + { + return ReadTree.Refused(complaint); + } + + var parentFiles = parents.SelectMany(p => p.Files ?? []).ToList(); + + var files = new List<LambdaFile>(); + + var stored = new List<StoredFile>(); + + var contents = new Dictionary<string, byte[]>(StringComparer.Ordinal); + + string? project = null; + + foreach (var entry in tree) + { + var path = ProjectPaths.Classify(entry.Path); + + if (path.Kind != ProjectPathKind.Lambda) + { + continue; + } + + var blob = entry.Id!.Value.ToString(); + + var known = parentFiles.FirstOrDefault(f => f.Path == entry.Path && (f.Name == LambdaSource.EntryName || f.Blob == blob)) + ?? parentFiles.FirstOrDefault(f => f.Path == entry.Path); + + if (path.Name == LambdaSource.EntryName) + { + var bytes = (await entry.ReadAsync()).ToArray(); + + if (!TryText(bytes, out var text)) + { + return ReadTree.Refused("Project.cs is not UTF-8 text."); + } + + project = text; + + // a parent's Project.cs is that parent's lambda.cs, to the byte + var same = parents.Select(p => (Entry: p.At(ProjectPaths.Snippet), Snippet: p.Files?.FirstOrDefault(f => f.Name == LambdaSource.EntryName))) + .FirstOrDefault(p => p.Entry?.Blob == blob && p.Snippet != null); + + string snippet; + + if (same.Snippet != null) + { + snippet = Encoding.UTF8.GetString(read(GitObjectId.Parse(same.Snippet.Blob))); + } + else + { + var unwrapped = ProjectSnippet.Unwrap(text); + + if (unwrapped.Snippet == null) + { + return ReadTree.Refused(unwrapped.Complaint!); + } + + snippet = unwrapped.Snippet; + } + + var snippetBytes = Encoding.UTF8.GetBytes(snippet); + + var snippetBlob = GitObjectId.ForBlob(snippetBytes).ToString(); + + contents[snippetBlob] = snippetBytes; + + files.Add(new LambdaFile(LambdaSource.EntryName, snippet)); + stored.Add(new StoredFile(LambdaSource.EntryName, entry.Path, snippetBlob, false)); + + continue; + } + + // named as the lambda named it where a parent had it, as the path says otherwise + var name = known?.Name is { } before && before != LambdaSource.EntryName ? before : path.Name!; + + var content = (await entry.ReadAsync()).ToArray(); + + if (LambdaSource.IsCode(name) || entry.Path.EndsWith(".md", StringComparison.OrdinalIgnoreCase) && LambdaSource.IsContext(name)) + { + if (!TryText(content, out var text)) + { + return ReadTree.Refused($"'{entry.Path}' is not UTF-8 text."); + } + + files.Add(new LambdaFile(name, text)); + stored.Add(new StoredFile(name, entry.Path, blob, false)); + } + else if (known is { Binary: true } && known.Blob == blob || !TryText(content, out var asText)) + { + files.Add(new LambdaFile(name, Convert.ToBase64String(content), "base64")); + stored.Add(new StoredFile(name, entry.Path, blob, true)); + } + else + { + files.Add(new LambdaFile(name, asText)); + stored.Add(new StoredFile(name, entry.Path, blob, false)); + } + } + + if (project == null) + { + return ReadTree.Refused("Project.cs holds the code of the lambda and cannot be removed."); + } + + var ordered = Ordered(files, parents.FirstOrDefault()?.Files); + + if (LambdaSource.Validate(ordered) is { } invalid) + { + return ReadTree.Refused(invalid); + } + + var order = ordered.Select((f, i) => (f.Name, i)).ToDictionary(x => x.Name, x => x.i, StringComparer.Ordinal); + + stored.Sort((x, y) => order[x.Name].CompareTo(order[y.Name])); + + return new ReadTree(ordered, stored, contents, project, null); + } + + /// <summary> + /// Why the platform's files of a pushed tree are not those of a parent, if they are not. + /// </summary> + /// <remarks> + /// Each on its own, so a merge may take one from either side; a file that + /// is in no parent is one too many, and one missing that every parent has + /// is missing. + /// </remarks> + private static string? Generated(IReadOnlyList<GitFile> tree, IReadOnlyList<StoredCommit> parents) + { + if (parents.Count == 0) + { + return null; + } + + var paths = tree.Select(f => f.Path) + .Concat(parents.SelectMany(p => p.Tree.Select(e => e.Path))) + .Where(p => ProjectPaths.Classify(p).Kind == ProjectPathKind.Generated) + .Distinct(StringComparer.Ordinal) + .Order(StringComparer.Ordinal); + + foreach (var path in paths) + { + var mine = tree.FirstOrDefault(f => f.Path == path)?.Id?.ToString(); + + if (parents.Any(p => p.At(path)?.Blob == mine)) + { + continue; + } + + if (mine == null) + { + return $"'{path}' is the platform's and cannot be removed: it is no part of the lambda. git checkout origin/main -- {path}, and commit again."; + } + + return parents.All(p => p.At(path) == null) + ? $"'{path}' is no file of the lambda: the platform writes the files around it. Remove it (git rm --cached {path}), and commit again." + : $"'{path}' is the platform's: changing it changes nothing of the lambda, so the change is refused. Undo it (git checkout origin/main -- {path}), and commit again."; + } + + return null; + } + + /// <summary> + /// The files in the order the lambda had them: lambda.cs first, then the + /// files the commit before had in its order, then the new ones the way a + /// zip of a lambda is read - so an unchanged lambda is the same to the byte. + /// </summary> + private static List<LambdaFile> Ordered(List<LambdaFile> files, IReadOnlyList<StoredFile>? before) + { + var known = (before ?? []).Select((f, i) => (f.Name, i)).ToDictionary(x => x.Name, x => x.i, StringComparer.Ordinal); + + return files.OrderBy(f => f.Name == LambdaSource.EntryName ? 0 : known.ContainsKey(f.Name) ? 1 : 2) + .ThenBy(f => known.GetValueOrDefault(f.Name, int.MaxValue)) + .ThenBy(f => f.IsCode ? 0 : f.IsAsset ? 1 : 2) + .ThenBy(f => f.Name, StringComparer.Ordinal) + .ToList(); + } + + /// <summary> + /// Text where it is UTF-8 without NUL characters, kept as it is - a byte + /// order mark included, so it is written back as it came. + /// </summary> + private static bool TryText(byte[] bytes, out string text) + { + try + { + text = StrictUtf8.GetString(bytes); + } + catch (DecoderFallbackException) + { + text = string.Empty; + return false; + } + + // a byte order mark stays the character it stands for, so it is written back + return !text.Contains('\0'); + } + + #endregion + +} + +/// <summary> +/// The tree of a commit the platform makes. +/// </summary> +/// <param name="Entries">Every file of the tree</param> +/// <param name="Files">The lambda's files in it</param> +/// <param name="Contents">What the files hold that may not be kept yet, by their blob</param> +public sealed record LaidTree(IReadOnlyList<StoredEntry> Entries, IReadOnlyList<StoredFile> Files, IReadOnlyDictionary<string, byte[]> Contents); + +/// <summary> +/// The lambda a pushed tree holds. +/// </summary> +/// <param name="Files">The lambda's files, in its order - or nothing where the tree holds none</param> +/// <param name="Stored">The same as a commit keeps them</param> +/// <param name="Contents">What is kept for them besides the tree: the lambda.cs taken out of Project.cs</param> +/// <param name="Project">Project.cs as pushed, to say where in it a problem the compiler found is</param> +/// <param name="Complaint">Why the tree is no lambda</param> +public sealed record ReadTree(IReadOnlyList<LambdaFile>? Files, IReadOnlyList<StoredFile> Stored, IReadOnlyDictionary<string, byte[]> Contents, + string? Project, string? Complaint) +{ + + public static ReadTree Refused(string complaint) => new(null, [], new Dictionary<string, byte[]>(), null, complaint); + +} diff --git a/src/GenHTTP.Lambda/Services/Storage/FileSystemStorageService.cs b/src/GenHTTP.Lambda/Services/Storage/FileSystemStorageService.cs index 9477162f..2d89e208 100644 --- a/src/GenHTTP.Lambda/Services/Storage/FileSystemStorageService.cs +++ b/src/GenHTTP.Lambda/Services/Storage/FileSystemStorageService.cs @@ -109,6 +109,8 @@ public void Delete(long lambdaId) Remove(Path.Combine(Options.FeatureDirectory, lambdaId.ToString())); // what its published source was packed into, if it was published Remove(Path.Combine(Options.SourceDirectory, lambdaId.ToString())); + // and the commits it was read as with git, if anybody did + Remove(Path.Combine(Options.GitDirectory, lambdaId.ToString())); // the generated assembly stays: it cannot be unloaded and GenHTTP builds // its invocation code from the files behind the loaded assemblies, so diff --git a/src/GenHTTP.Lambda/Services/Storage/IStorageService.cs b/src/GenHTTP.Lambda/Services/Storage/IStorageService.cs index 82db1202..97c1f3c4 100644 --- a/src/GenHTTP.Lambda/Services/Storage/IStorageService.cs +++ b/src/GenHTTP.Lambda/Services/Storage/IStorageService.cs @@ -24,7 +24,7 @@ public interface IStorageService /// <summary> /// Removes everything stored for the given lambda, including its workspace, - /// its database and its features. + /// its database, its features and the commits it was read as with git. /// </summary> void Delete(long lambdaId); diff --git a/tests/GenHTTP.Lambda.Tests/Git/BranchTests.cs b/tests/GenHTTP.Lambda.Tests/Git/BranchTests.cs new file mode 100644 index 00000000..dae89e49 --- /dev/null +++ b/tests/GenHTTP.Lambda.Tests/Git/BranchTests.cs @@ -0,0 +1,227 @@ +using GenHTTP.Lambda.Services.Deployment.Model; +using GenHTTP.Lambda.Tests.Infrastructure; + +namespace GenHTTP.Lambda.Tests.Git; + +/// <summary> +/// Pushing a branch: a feature, with its own copy of the data and its preview +/// online, and merged once it is right - the way a change to a lambda in use +/// is made. +/// </summary> +[TestClass] +public sealed class BranchTests +{ + + [TestMethod] + public async Task ABranchPushedIsAFeatureWithItsPreviewOnline() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("branched"); + + await fixture.SaveAsync(lambda, "Says live", new LambdaFile(LambdaSource.EntryName, Repository.Says("live"))); + await fixture.DeployAsync(lambda.PrivateKey); + + await git.CloneAsync(fixture.EditorUrl(lambda), "branched"); + + await git.RunAsync("branched", "switch", "-c", "louder"); + git.Write("branched", "Project.cs", PushTests.ProjectOf(Repository.Says("LIVE"))); + await git.CommitAsync("branched", "Says it louder"); + + var pushed = await git.RunAsync("branched", "push", "-u", "origin", "louder"); + + Assert.Contains("Started the feature 'louder' from version 2", pushed.Said); + + var feature = (await fixture.FeaturesAsync(lambda)).Single(); + + Assert.AreEqual("louder", feature.Branch); + Assert.AreEqual("git", feature.Origin); + Assert.AreEqual(2, feature.Base); + Assert.AreEqual("Says it louder", feature.Change, "what its first commit says it changes"); + Assert.IsTrue(feature.Online); + + Assert.Contains($"Its preview is online at {fixture.Host.GetUrl(feature.PreviewPath)}", pushed.Said); + + Assert.AreEqual("LIVE", await fixture.CallAsync(feature.PreviewPath)); + Assert.AreEqual("live", await fixture.CallAsync("/lambda/branched/"), "the lambda goes on as it was"); + + // pushed again, its files are replaced - a rewrite included + git.Write("branched", "Project.cs", PushTests.ProjectOf(Repository.Says("LIVE!"))); + await git.RunAsync("branched", "add", "-A"); + await git.RunAsync("branched", "commit", "-q", "--amend", "--no-edit"); + await git.RunAsync("branched", "push", "--force"); + + Assert.AreEqual("LIVE!", await fixture.CallAsync(feature.PreviewPath)); + Assert.Contains("LIVE!", (await fixture.FeatureAsync(lambda, feature.Key)).Files.Single().Code); + } + + [TestMethod] + public async Task ABranchPushedWithMergeIsTheNextVersion() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("squashed"); + + await git.CloneAsync(fixture.EditorUrl(lambda), "squashed"); + + await git.RunAsync("squashed", "switch", "-c", "greeting"); + + git.Write("squashed", "assets/one.txt", "one"); + await git.CommitAsync("squashed", "Adds one"); + await git.RunAsync("squashed", "push", "-u", "origin", "greeting"); + + git.Write("squashed", "Project.cs", PushTests.ProjectOf(Repository.Says("merged"))); + await git.CommitAsync("squashed", "Says merged"); + + var pushed = await git.RunAsync("squashed", "push", "-o", "merge", "-o", "deploy"); + + Assert.Contains("Merged it as version 2", pushed.Said); + Assert.Contains("Version 2 is online", pushed.Said); + + Assert.IsEmpty(await fixture.FeaturesAsync(lambda), "the feature is gone, with its preview and its copy of the data"); + Assert.AreEqual("merged", await fixture.CallAsync("/lambda/squashed/")); + + Assert.HasCount(2, await fixture.VersionsAsync(lambda), "a feature merged is one version, however many commits it had"); + + await git.RunAsync("squashed", "switch", "main"); + await git.RunAsync("squashed", "pull"); + await git.RunAsync("squashed", "fetch", "--prune"); + + Assert.AreEqual(string.Empty, await git.ReadAsync("squashed", "branch", "-r", "--list", "origin/greeting")); + Assert.Contains("one", git.Read("squashed", "assets/one.txt")); + } + + [TestMethod] + public async Task ABranchPushedToMainIsMergedAVersionForEachCommit() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("forwarded"); + + await git.CloneAsync(fixture.EditorUrl(lambda), "forwarded"); + + await git.RunAsync("forwarded", "switch", "-c", "two-steps"); + + git.Write("forwarded", "assets/one.txt", "one"); + await git.CommitAsync("forwarded", "Adds one"); + + git.Write("forwarded", "assets/two.txt", "two"); + await git.CommitAsync("forwarded", "Adds two"); + + await git.RunAsync("forwarded", "push", "-u", "origin", "two-steps"); + + var pushed = await git.RunAsync("forwarded", "push", "origin", "two-steps:main"); + + Assert.Contains("Saved versions 2 to 3", pushed.Said); + Assert.Contains("The feature 'two-steps' is in main now, so it is merged", pushed.Said); + + Assert.IsEmpty(await fixture.FeaturesAsync(lambda)); + + CollectionAssert.AreEqual(new[] { "Adds two", "Adds one", "Started empty" }, (await fixture.VersionsAsync(lambda)).Select(v => v.Change).ToList()); + } + + [TestMethod] + public async Task ABranchDeletedTakesItsFeatureWithIt() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("pruned"); + + var feature = await fixture.CreateFeatureAsync(lambda, "Not wanted"); + + await git.CloneAsync(fixture.EditorUrl(lambda), "pruned"); + + var deleted = await git.RunAsync("pruned", "push", "origin", "--delete", feature.Branch); + + Assert.Contains("Deleted the feature 'Not wanted'", deleted.Said); + + Assert.IsEmpty(await fixture.FeaturesAsync(lambda)); + } + + [TestMethod] + public async Task ABranchThatStartsFromNoVersionIsRefused() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("orphaned"); + + await git.CloneAsync(fixture.EditorUrl(lambda), "orphaned"); + + await git.RunAsync("orphaned", "switch", "--orphan", "elsewhere"); + await git.RunAsync("orphaned", "checkout", "main", "--", "."); + await git.CommitAsync("orphaned", "From nowhere"); + + var pushed = await git.TryAsync("orphaned", "push", "origin", "elsewhere"); + + Assert.AreNotEqual(0, pushed.ExitCode); + Assert.Contains("A branch starts from main or one of its versions", pushed.Said); + + Assert.IsEmpty(await fixture.FeaturesAsync(lambda)); + } + + [TestMethod] + public async Task ABranchRefusedLeavesNoFeatureBehind() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var limits = fixture.Limits.Get(); + + fixture.Limits.Save(limits with { Free = limits.Free with { AssetBytes = 1024 } }); + + var lambda = await fixture.CreateLambdaAsync("toolarge"); + + await git.CloneAsync(fixture.EditorUrl(lambda), "toolarge"); + + await git.RunAsync("toolarge", "switch", "-c", "pictures"); + git.Write("toolarge", "assets/large.txt", new string('x', 4096)); + await git.CommitAsync("toolarge", "Adds something large"); + + var pushed = await git.TryAsync("toolarge", "push", "-u", "origin", "pictures"); + + Assert.AreNotEqual(0, pushed.ExitCode); + Assert.Contains("must not exceed", pushed.Said, "the tier's refusal, in its words"); + + Assert.IsEmpty(await fixture.FeaturesAsync(lambda), "the feature started for the branch went with the refusal"); + } + + [TestMethod] + public async Task AFeatureChangedElsewhereIsNotOverwrittenByAPush() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("raced"); + + var feature = await fixture.CreateFeatureAsync(lambda, "Shared"); + + await git.CloneAsync(fixture.EditorUrl(lambda), "raced"); + + await git.RunAsync("raced", "switch", feature.Branch); + + // the agent saves it meanwhile + await fixture.PutFeatureAsync(lambda, feature.Key, "The agent's change", new LambdaFile(LambdaSource.EntryName, Repository.Says("agent"))); + + git.Write("raced", "assets/mine.txt", "mine"); + await git.CommitAsync("raced", "Mine"); + + var pushed = await git.TryAsync("raced", "push"); + + Assert.AreNotEqual(0, pushed.ExitCode); + + Assert.Contains("agent", (await fixture.FeatureAsync(lambda, feature.Key)).Files.Single(f => f.Name == LambdaSource.EntryName).Code); + } + +} diff --git a/tests/GenHTTP.Lambda.Tests/Git/CloneTests.cs b/tests/GenHTTP.Lambda.Tests/Git/CloneTests.cs new file mode 100644 index 00000000..c6324533 --- /dev/null +++ b/tests/GenHTTP.Lambda.Tests/Git/CloneTests.cs @@ -0,0 +1,347 @@ +using System.Diagnostics; +using System.Net; + +using GenHTTP.Lambda.Services.Deployment.Model; +using GenHTTP.Lambda.Services.Meta; +using GenHTTP.Lambda.Tests.Infrastructure; + +namespace GenHTTP.Lambda.Tests.Git; + +/// <summary> +/// A lambda cloned with git: the project it is exported as, with every +/// version a commit of main and every feature a branch. +/// </summary> +[TestClass] +public sealed class CloneTests +{ + + #region The project + + [TestMethod] + public async Task ALambdaClonesAsTheProjectItIsExportedAs() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("shelf", "demo-crud"); + + var clone = await git.CloneAsync(fixture.EditorUrl(lambda), "shelf"); + + foreach (var wanted in (string[])["Project.cs", "Program.cs", "shelf.csproj", "Store.cs", "Tasks.cs", "Platform/Usings.cs", "Platform/Database.cs", + "assets/web/index.html", "assets/migrations/V1__Create_tasks.sql", "docs/product.md", "tests/README.md", + "AGENTS.md", "CLAUDE.md", "Dockerfile", ".gitignore"]) + { + Assert.IsTrue(File.Exists(Path.Combine(clone, wanted)), $"{wanted} is in the clone"); + } + + Assert.IsFalse(File.Exists(Path.Combine(clone, "lambda.cs")), "the snippet is Project.cs"); + Assert.IsFalse(Directory.Exists(Path.Combine(clone, ".lambda")), "the documentation and the tests are where a project keeps them"); + Assert.IsFalse(Directory.Exists(Path.Combine(clone, "database")), "no data is in a repository"); + + Assert.Contains("private static async Task<object> BuildAsync()", git.Read("shelf", "Project.cs"), "the snippet is the body of the method the platform runs"); + Assert.Contains(".Handler(await Project.CreateAsync())", git.Read("shelf", "Program.cs")); + Assert.Contains("@AGENTS.md", git.Read("shelf", "CLAUDE.md")); + } + + [TestMethod] + public async Task TheGuideForAgentsNamesTheLambdaAndNotItsKey() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("guided"); + + await git.CloneAsync(fixture.EditorUrl(lambda), "guided"); + + var guide = git.Read("guided", "AGENTS.md"); + + Assert.Contains("# guided", guide); + Assert.Contains("/lambda/guided/", guide, "where it runs"); + Assert.Contains("git push -o deploy", guide); + Assert.DoesNotContain("{lambda}", guide, "every placeholder is filled in"); + Assert.DoesNotContain("{home}", guide); + + foreach (var file in Directory.EnumerateFiles(git.PathOf("guided"), "*", SearchOption.AllDirectories).Where(f => !f.Contains($"{Path.DirectorySeparatorChar}.git{Path.DirectorySeparatorChar}"))) + { + Assert.DoesNotContain(lambda.PrivateKey, await File.ReadAllTextAsync(file), $"{file} never holds the editor key - the same commits are read by anybody once the source is published"); + } + } + + [TestMethod] + public async Task EveryDemoClonesToAProjectThatBuilds() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var builds = new List<Task<(string Demo, int Exit, string Output)>>(); + + foreach (var demo in DemoCatalog.All) + { + var lambda = await fixture.CreateLambdaAsync($"clone-{demo.Id}", demo.Id); + + var clone = await git.CloneAsync(fixture.EditorUrl(lambda), demo.Id); + + builds.Add(BuildAsync(demo.Id, clone)); + } + + foreach (var (demo, exit, output) in await Task.WhenAll(builds)) + { + Assert.AreEqual(0, exit, $"The clone of {demo} does not build:\n{output}"); + } + } + + #endregion + + #region History + + [TestMethod] + public async Task EveryVersionIsACommitOfMainTaggedWithItsNumber() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("history"); + + await fixture.SaveAsync(lambda, "Says one", new LambdaFile(LambdaSource.EntryName, Repository.Says("one"))); + await fixture.SaveAsync(lambda, "Says two", new LambdaFile(LambdaSource.EntryName, Repository.Says("two"))); + + await git.CloneAsync(fixture.EditorUrl(lambda), "history"); + + var log = await git.ReadAsync("history", "log", "--format=%s", "main"); + + CollectionAssert.AreEqual(new[] { "Says two", "Says one", "Started empty" }, log.Split('\n'), "the change of each version, newest first"); + + Assert.AreEqual("v1\nv2\nv3", await git.ReadAsync("history", "tag", "--list", "--sort=version:refname")); + + Assert.AreEqual(await git.ReadAsync("history", "rev-parse", "main"), await git.ReadAsync("history", "rev-parse", "v3^{commit}")); + Assert.AreEqual("GenHTTP Lambda", await git.ReadAsync("history", "log", "-1", "--format=%an")); + } + + [TestMethod] + public async Task ACommitIsTheSameCommitForEverybodyWhoReadsItLater() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("stable"); + + await git.CloneAsync(fixture.EditorUrl(lambda), "first"); + + await fixture.SaveAsync(lambda, "Says more", new LambdaFile(LambdaSource.EntryName, Repository.Says("more"))); + + await git.CloneAsync(fixture.EditorUrl(lambda), "second"); + + Assert.AreEqual(await git.ReadAsync("first", "rev-parse", "v1"), await git.ReadAsync("second", "rev-parse", "v1"), "a commit is made once and kept"); + + // the clone made before the version follows it without a forced update + var fetched = await git.RunAsync("first", "pull", "--ff-only"); + + Assert.AreEqual(await git.ReadAsync("second", "rev-parse", "main"), await git.ReadAsync("first", "rev-parse", "main"), fetched.Said); + } + + [TestMethod] + public async Task AVersionThatOnlyChangesAnAssetChangesOnlyThatFile() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("pictures"); + + var code = new LambdaFile(LambdaSource.EntryName, "return Assets.App(\"web\");\n"); + + await fixture.SaveAsync(lambda, "Has a page", code, new LambdaFile("web/index.html", "<h1>one</h1>")); + await fixture.SaveAsync(lambda, "Changes the page", code, new LambdaFile("web/index.html", "<h1>two</h1>")); + + await git.CloneAsync(fixture.EditorUrl(lambda), "pictures"); + + Assert.AreEqual("assets/web/index.html", await git.ReadAsync("pictures", "diff", "--name-only", "v2", "v3")); + } + + [TestMethod] + public async Task VersionsNoLongerKeptLeaveAShallowHistory() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var limits = fixture.Limits.Get(); + + fixture.Limits.Save(limits with { Free = limits.Free with { Versions = 2 } }); + + var lambda = await fixture.CreateLambdaAsync("pruned"); + + await git.CloneAsync(fixture.EditorUrl(lambda), "early"); + + foreach (var word in (string[])["one", "two", "three"]) + { + await fixture.SaveAsync(lambda, $"Says {word}", new LambdaFile(LambdaSource.EntryName, Repository.Says(word))); + } + + await git.CloneAsync(fixture.EditorUrl(lambda), "late"); + + Assert.AreEqual("true", await git.ReadAsync("late", "rev-parse", "--is-shallow-repository"), "the history starts where the versions kept do"); + Assert.AreEqual("Says three\nSays two", await git.ReadAsync("late", "log", "--format=%s")); + Assert.AreEqual("v3\nv4", await git.ReadAsync("late", "tag", "--list", "--sort=version:refname")); + + // a clone made before keeps what it has, and fetches the rest + await git.RunAsync("early", "pull", "--ff-only"); + + Assert.AreEqual(await git.ReadAsync("late", "rev-parse", "main"), await git.ReadAsync("early", "rev-parse", "main")); + + await git.RunAsync("early", "fsck", "--strict"); + } + + #endregion + + #region Features + + [TestMethod] + public async Task AFeatureIsABranchNamedAfterIt() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("drafts"); + + var feature = await fixture.CreateFeatureAsync(lambda, "Dark mode for the page"); + + Assert.AreEqual("dark-mode-for-the-page", feature.Branch); + + await git.CloneAsync(fixture.EditorUrl(lambda), "drafts"); + + Assert.AreEqual(await git.ReadAsync("drafts", "rev-parse", "origin/main"), await git.ReadAsync("drafts", "rev-parse", "origin/dark-mode-for-the-page"), + "a feature that holds what its version holds is that version"); + + await fixture.PutFeatureAsync(lambda, feature.Key, "Says it darkly", new LambdaFile(LambdaSource.EntryName, Repository.Says("dark"))); + + await git.RunAsync("drafts", "fetch"); + + Assert.AreEqual("Says it darkly", await git.ReadAsync("drafts", "log", "-1", "--format=%s", "origin/dark-mode-for-the-page")); + Assert.AreEqual(await git.ReadAsync("drafts", "rev-parse", "origin/main"), await git.ReadAsync("drafts", "rev-parse", "origin/dark-mode-for-the-page^"), + "its first commit follows the version it started from"); + + // renamed, it keeps its branch: a clone knows it by that + using var renamed = await fixture.SendAsync(HttpMethod.Patch, $"/api/v1/lambdas/{lambda.PrivateKey}/features/{feature.Key}", new { name = "Night mode" }); + + Assert.AreEqual(HttpStatusCode.OK, renamed.StatusCode); + + Assert.AreEqual("dark-mode-for-the-page", (await fixture.FeaturesAsync(lambda)).Single().Branch); + } + + [TestMethod] + public async Task AFeatureBroughtUpToDateFollowsTheVersionItWasBroughtTo() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("catchup"); + + var feature = await fixture.CreateFeatureAsync(lambda, "Louder"); + + await fixture.PutFeatureAsync(lambda, feature.Key, "Says it louder", new LambdaFile(LambdaSource.EntryName, Repository.Says("LOUD"))); + + await git.CloneAsync(fixture.EditorUrl(lambda), "catchup"); + + var newest = await fixture.SaveAsync(lambda, "Says something else", new LambdaFile(LambdaSource.EntryName, Repository.Says("else"))); + + // the agent brings it in, and says so by moving the base + using var moved = await fixture.SendAsync(HttpMethod.Patch, $"/api/v1/lambdas/{lambda.PrivateKey}/features/{feature.Key}", new { @base = newest }); + + Assert.AreEqual(HttpStatusCode.OK, moved.StatusCode); + + await git.RunAsync("catchup", "fetch"); + + var parents = await git.ReadAsync("catchup", "log", "-1", "--format=%P", "origin/louder"); + + Assert.Contains(await git.ReadAsync("catchup", "rev-parse", "origin/main"), parents, "the branch holds the version it was brought to"); + + Assert.AreEqual(0, (await git.TryAsync("catchup", "merge-base", "--is-ancestor", "origin/main", "origin/louder")).ExitCode); + } + + #endregion + + #region Addresses + + [TestMethod] + public async Task AnyNameBelowTheEditorKeyIsTheSameRepository() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("named"); + + await git.CloneAsync(fixture.Host.GetUrl($"/editor/{lambda.PrivateKey}/whatever.git"), "one"); + await git.CloneAsync(fixture.Host.GetUrl($"/editor/{lambda.PrivateKey}"), "two"); + + Assert.AreEqual(await git.ReadAsync("one", "rev-parse", "main"), await git.ReadAsync("two", "rev-parse", "main"), + "the name only names the folder, so a clone keeps working once the public key changes"); + } + + [TestMethod] + public async Task AKeyNobodyHasIsNoRepository() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var cloned = await git.TryAsync(null, "clone", fixture.Host.GetUrl("/editor/nobody-has-this-key/x.git"), "nothing"); + + Assert.AreNotEqual(0, cloned.ExitCode); + + using var answer = await fixture.GetAsync("/editor/nobody-has-this-key/x.git/info/refs?service=git-upload-pack"); + + Assert.AreEqual(HttpStatusCode.NotFound, answer.StatusCode, "never the page, which git would take for a broken server"); + } + + [TestMethod] + public async Task TheAddressOfARepositoryLeadsABrowserToItsPage() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + var lambda = await fixture.CreateLambdaAsync("browsed"); + + using var editor = await fixture.GetAsync($"/editor/{lambda.PrivateKey}/browsed.git"); + + Assert.AreEqual(HttpStatusCode.TemporaryRedirect, editor.StatusCode); + Assert.EndsWith($"/editor/{lambda.PrivateKey}", editor.Headers.Location?.OriginalString ?? string.Empty); + + using var page = await fixture.GetAsync($"/editor/{lambda.PrivateKey}/code"); + + Assert.AreEqual(HttpStatusCode.OK, page.StatusCode, "the editor's own pages are still the editor's"); + } + + #endregion + + #region Helpers + + private static async Task<(string, int, string)> BuildAsync(string demo, string directory) + { + var start = new ProcessStartInfo("dotnet", "build -c Release -nologo -clp:ErrorsOnly") + { + WorkingDirectory = directory, + RedirectStandardOutput = true, + RedirectStandardError = true + }; + + using var process = Process.Start(start)!; + + var output = process.StandardOutput.ReadToEndAsync(); + var error = process.StandardError.ReadToEndAsync(); + + await process.WaitForExitAsync(); + + return (demo, process.ExitCode, await output + await error); + } + + #endregion + +} diff --git a/tests/GenHTTP.Lambda.Tests/Git/ProjectSnippetTests.cs b/tests/GenHTTP.Lambda.Tests/Git/ProjectSnippetTests.cs new file mode 100644 index 00000000..34285340 --- /dev/null +++ b/tests/GenHTTP.Lambda.Tests/Git/ProjectSnippetTests.cs @@ -0,0 +1,144 @@ +using GenHTTP.Lambda.Services.Deployment; +using GenHTTP.Lambda.Services.Deployment.Model; +using GenHTTP.Lambda.Services.Meta; + +namespace GenHTTP.Lambda.Tests.Git; + +/// <summary> +/// The snippet of a lambda as the class Project.cs holds, and back: what a +/// push to its repository changed is what lambda.cs changes. +/// </summary> +[TestClass] +public sealed class ProjectSnippetTests +{ + + [TestMethod] + public void EveryDemoTakenOutOfTheClassIsPutBackAsItWas() + { + foreach (var demo in DemoCatalog.All) + { + var snippet = LambdaSource.Parse(TemplateCatalog.ForKey(demo.Id, demo.Key, demo: true)).Single(f => f.Name == LambdaSource.EntryName).Code; + + var project = ProjectSnippet.ForRepository(snippet); + + var unwrapped = ProjectSnippet.Unwrap(project); + + Assert.IsNull(unwrapped.Complaint, demo.Id); + + Assert.AreEqual(project, ProjectSnippet.ForRepository(unwrapped.Snippet!), $"{demo.Id}: a class taken out and put back is the class it was"); + + Assert.AreEqual(Statements(snippet), Statements(unwrapped.Snippet!), $"{demo.Id}: the statements are what they were"); + } + } + + [TestMethod] + public void TheBodyIsMovedBackOutAndAStringKeepsItsOwnSpaces() + { + const string project = """" + using System.Text; + + public static class Project + { + public static async Task<IHandler> CreateAsync() => Platform.Handlers.From(await BuildAsync()); + + private static async Task<object> BuildAsync() + { + var text = """ + kept as it is + """; + + if (text.Length > 0) + { + return Content.From(Resource.FromString(text)); + } + + return Content.From(Resource.FromString("empty")); + } + } + + public record Note(string Text); + """"; + + var unwrapped = ProjectSnippet.Unwrap(project); + + Assert.AreEqual("""" + using System.Text; + + var text = """ + kept as it is + """; + + if (text.Length > 0) + { + return Content.From(Resource.FromString(text)); + } + + return Content.From(Resource.FromString("empty")); + + public record Note(string Text); + + """", unwrapped.Snippet); + + Assert.AreEqual((9, 9), unwrapped.Locate(3, 1), "the third line of the snippet is the ninth of Project.cs, eight columns further in"); + Assert.AreEqual((22, 1), unwrapped.Locate(14, 1), "and the types are where they were"); + } + + [TestMethod] + public void ABodyWrittenLessFarInKeepsItsShape() + { + var unwrapped = ProjectSnippet.Unwrap(""" + public static class Project + { + static async Task<object> BuildAsync() + { + if (true) + { + return 1; + } + } + } + """); + + Assert.AreEqual("if (true)\n{\n return 1;\n}\n", unwrapped.Snippet); + } + + [TestMethod] + public void WhatHasNoPlaceInASnippetIsRefused() + { + var member = ProjectSnippet.Unwrap(""" + public static class Project + { + private static readonly int Answer = 42; + + private static async Task<object> BuildAsync() + { + return Answer; + } + } + """); + + Assert.Contains("'Answer' cannot stay in the class Project", member.Complaint ?? string.Empty); + + var none = ProjectSnippet.Unwrap("public static class Something { }"); + + Assert.Contains("keeps the class Project", none.Complaint ?? string.Empty); + } + + [TestMethod] + public void AnExportedProjectIsTakenOutAsWell() + { + var unwrapped = ProjectSnippet.Unwrap(ProjectSnippet.ForExport("return Content.From(Resource.FromString(\"x\"));\n")); + + Assert.AreEqual("return Content.From(Resource.FromString(\"x\"));\n", unwrapped.Snippet, "Build(), as the export has it, is the snippet as well"); + } + + /// <summary> + /// The lines of a snippet that say something, a type's made public or not. + /// </summary> + private static string Statements(string snippet) + => string.Join('\n', snippet.Split('\n') + .Select(l => l.TrimEnd()) + .Where(l => l.Length > 0) + .Select(l => l.StartsWith("public ", StringComparison.Ordinal) ? l["public ".Length..] : l)); + +} diff --git a/tests/GenHTTP.Lambda.Tests/Git/PushTests.cs b/tests/GenHTTP.Lambda.Tests/Git/PushTests.cs new file mode 100644 index 00000000..b5a2cfb5 --- /dev/null +++ b/tests/GenHTTP.Lambda.Tests/Git/PushTests.cs @@ -0,0 +1,360 @@ +using GenHTTP.Lambda.Services.Deployment.Model; +using GenHTTP.Lambda.Tests.Infrastructure; + +namespace GenHTTP.Lambda.Tests.Git; + +/// <summary> +/// Pushing to main: every commit becomes the next version of the lambda, and +/// what is no part of the lambda is refused rather than lost. +/// </summary> +[TestClass] +public sealed class PushTests +{ + + #region Versions + + [TestMethod] + public async Task ACommitPushedToMainIsTheNextVersion() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("pushed", "demo-crud"); + + await git.CloneAsync(fixture.EditorUrl(lambda), "pushed"); + + git.Change("pushed", "Project.cs", "var store = new TaskStore(limit: 200);", "var store = new TaskStore(limit: 300);"); + + await git.CommitAsync("pushed", "Keeps three hundred tasks\n\nThe list ran full in a week."); + + var pushed = await git.RunAsync("pushed", "push"); + + Assert.Contains("Saved version 2: Keeps three hundred tasks", pushed.Said); + + var version = (await fixture.VersionsAsync(lambda)).First(); + + Assert.AreEqual(2, version.Version); + Assert.AreEqual("Keeps three hundred tasks", version.Change, "the first line of the message is the change"); + Assert.AreEqual("The list ran full in a week.", version.Specification, "the rest is what the user wanted"); + Assert.AreEqual("git", version.Origin); + + var before = (await fixture.VersionAsync(lambda, 1)).Files; + var after = (await fixture.VersionAsync(lambda, 2)).Files; + + CollectionAssert.AreEqual(before.Select(f => f.Name).ToList(), after.Select(f => f.Name).ToList(), "the files keep their names and their order"); + + var code = after.Single(f => f.Name == LambdaSource.EntryName).Code; + + Assert.AreEqual(before.Single(f => f.Name == LambdaSource.EntryName).Code.Replace("limit: 200", "limit: 300"), code, + "the body of BuildAsync is lambda.cs again, and only what changed changed"); + + Assert.AreEqual(await git.ReadAsync("pushed", "rev-parse", "main"), (await git.ReadAsync("pushed", "ls-remote", "origin", "refs/tags/v2")).Split('\t')[0], + "the commit pushed is the version's commit"); + } + + [TestMethod] + public async Task ACommitThatLeavesProjectCsAloneLeavesTheCodeAsItWas() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("untouched"); + + // written in a way putting it into the class and back would not keep + const string written = "using System.Text;\n\n // indented oddly\nreturn Content.From(Resource.FromString(new StringBuilder(\"odd\").ToString()));\n\nrecord Unused(int X);\n"; + + await fixture.SaveAsync(lambda, "Is odd", new LambdaFile(LambdaSource.EntryName, written), new LambdaFile("store.cs", "public class Shelf { }\n")); + + await git.CloneAsync(fixture.EditorUrl(lambda), "untouched"); + + Assert.IsTrue(File.Exists(Path.Combine(git.PathOf("untouched"), "Store.cs")), "named the .NET way"); + + git.Write("untouched", "assets/readme.txt", "hello"); + git.Change("untouched", "Store.cs", "Shelf", "Bookshelf"); + + await git.CommitAsync("untouched", "Adds a file"); + await git.RunAsync("untouched", "push"); + + var files = (await fixture.VersionAsync(lambda, 3)).Files; + + Assert.AreEqual(written, files.Single(f => f.Name == LambdaSource.EntryName).Code, "to the byte"); + Assert.AreEqual("public class Bookshelf { }\n", files.Single(f => f.Name == "store.cs").Code, "under the name the lambda gave it"); + Assert.AreEqual("hello", files.Single(f => f.Name == "readme.txt").Code); + } + + [TestMethod] + public async Task EveryCommitOfAPushIsAVersion() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("several"); + + await git.CloneAsync(fixture.EditorUrl(lambda), "several"); + + foreach (var word in (string[])["one", "two", "three"]) + { + git.Write("several", "assets/word.txt", word); + await git.CommitAsync("several", $"Says {word}"); + } + + var pushed = await git.RunAsync("several", "push"); + + Assert.Contains("Saved versions 2 to 4", pushed.Said); + + CollectionAssert.AreEqual(new[] { "Says three", "Says two", "Says one", "Started empty" }, + (await fixture.VersionsAsync(lambda)).Select(v => v.Change).ToList()); + + await git.RunAsync("several", "fetch", "--tags"); + + Assert.AreEqual(await git.ReadAsync("several", "rev-parse", "main~1"), await git.ReadAsync("several", "rev-parse", "v3^{commit}")); + } + + [TestMethod] + public async Task APushWithDeployPutsTheNewestVersionOnline() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("deployed"); + + await git.CloneAsync(fixture.EditorUrl(lambda), "deployed"); + + git.Write("deployed", "Project.cs", ProjectOf(Repository.Says("pushed and online"))); + + await git.CommitAsync("deployed", "Says it is online"); + + var pushed = await git.RunAsync("deployed", "push", "-o", "deploy"); + + Assert.Contains("Version 2 is online at", pushed.Said); + + Assert.AreEqual("pushed and online", await fixture.CallAsync("/lambda/deployed/")); + } + + [TestMethod] + public async Task APushWithoutDeploySaysWhatIsOnline() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("offline"); + + await git.CloneAsync(fixture.EditorUrl(lambda), "offline"); + + git.Write("offline", "assets/note.txt", "x"); + + await git.CommitAsync("offline", "Adds a note"); + + var pushed = await git.RunAsync("offline", "push"); + + Assert.Contains("Nothing is online yet", pushed.Said); + Assert.Contains("-o deploy", pushed.Said); + Assert.DoesNotContain(lambda.PrivateKey, pushed.Said.Replace(fixture.EditorUrl(lambda), string.Empty), "the editor key is never said back"); + } + + #endregion + + #region Refused + + [TestMethod] + public async Task APushThatDoesNotCompileIsRefusedWithWhereInProjectCs() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("broken", "demo-crud"); + + await git.CloneAsync(fixture.EditorUrl(lambda), "broken"); + + git.Change("broken", "Project.cs", "new TaskStore(limit: 200)", "new TaskStore(limit: \"200\")"); + + await git.CommitAsync("broken", "Breaks it"); + + var pushed = await git.TryAsync("broken", "push"); + + Assert.AreNotEqual(0, pushed.ExitCode); + + var line = git.Read("broken", "Project.cs").Split('\n').ToList().FindIndex(l => l.Contains("limit: \"200\"")) + 1; + + Assert.Contains($"Project.cs({line},", pushed.Said, "where the problem is, in the file that was pushed"); + Assert.Contains("does not compile", pushed.Said); + + Assert.HasCount(1, await fixture.VersionsAsync(lambda), "no version is made of it"); + } + + [TestMethod] + public async Task AChangeToAFileOfThePlatformIsRefused() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("platform"); + + await git.CloneAsync(fixture.EditorUrl(lambda), "platform"); + + git.Write("platform", "Program.cs", git.Read("platform", "Program.cs") + "// mine\n"); + + await git.CommitAsync("platform", "Changes the program"); + + var pushed = await git.TryAsync("platform", "push"); + + Assert.AreNotEqual(0, pushed.ExitCode); + Assert.Contains("'Program.cs' is the platform's", pushed.Said); + + await git.RunAsync("platform", "reset", "--hard", "origin/main"); + + git.Write("platform", "README.md", "# Mine"); + + await git.CommitAsync("platform", "Adds a readme"); + + var foreign = await git.TryAsync("platform", "push"); + + Assert.AreNotEqual(0, foreign.ExitCode); + Assert.Contains("'README.md' has no place in a lambda", foreign.Said); + + Assert.HasCount(1, await fixture.VersionsAsync(lambda)); + } + + [TestMethod] + public async Task MainIsNeverRewritten() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("rewritten"); + + await fixture.SaveAsync(lambda, "Says two", new LambdaFile(LambdaSource.EntryName, Repository.Says("two"))); + + await git.CloneAsync(fixture.EditorUrl(lambda), "rewritten"); + + await git.RunAsync("rewritten", "reset", "--hard", "HEAD~1"); + + git.Write("rewritten", "assets/other.txt", "other"); + + await git.CommitAsync("rewritten", "Goes another way"); + + var forced = await git.TryAsync("rewritten", "push", "--force"); + + Assert.AreNotEqual(0, forced.ExitCode); + Assert.Contains("never rewritten", forced.Said); + + var deleted = await git.TryAsync("rewritten", "push", "origin", "--delete", "main"); + + Assert.AreNotEqual(0, deleted.ExitCode); + } + + [TestMethod] + public async Task AMergeCommitHasNoPlaceOnMain() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("merged"); + + await git.CloneAsync(fixture.EditorUrl(lambda), "merged"); + + await git.RunAsync("merged", "switch", "-c", "side"); + git.Write("merged", "assets/side.txt", "side"); + await git.CommitAsync("merged", "On the side"); + + await git.RunAsync("merged", "switch", "main"); + git.Write("merged", "assets/main.txt", "main"); + await git.CommitAsync("merged", "On main"); + + await git.RunAsync("merged", "merge", "--no-edit", "side"); + + var pushed = await git.TryAsync("merged", "push", "origin", "main"); + + Assert.AreNotEqual(0, pushed.ExitCode); + Assert.Contains("merge commit", pushed.Said); + } + + [TestMethod] + public async Task APushBehindAVersionSavedElsewhereIsRefused() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("behind"); + + await git.CloneAsync(fixture.EditorUrl(lambda), "behind"); + + await fixture.SaveAsync(lambda, "Saved in the editor", new LambdaFile(LambdaSource.EntryName, Repository.Says("editor"))); + + git.Write("behind", "assets/mine.txt", "mine"); + await git.CommitAsync("behind", "Mine"); + + var pushed = await git.TryAsync("behind", "push"); + + Assert.AreNotEqual(0, pushed.ExitCode, "nothing saved elsewhere is lost"); + + await git.RunAsync("behind", "pull", "--rebase"); + await git.RunAsync("behind", "push"); + + var versions = await fixture.VersionsAsync(lambda); + + Assert.AreEqual("Mine", versions[0].Change); + Assert.AreEqual("Saved in the editor", versions[1].Change); + } + + [TestMethod] + public async Task TagsAreTheVersionsAndCannotBePushed() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("tagged"); + + await git.CloneAsync(fixture.EditorUrl(lambda), "tagged"); + + await git.RunAsync("tagged", "tag", "release"); + + var pushed = await git.TryAsync("tagged", "push", "origin", "release"); + + Assert.AreNotEqual(0, pushed.ExitCode); + Assert.Contains("Tags are the versions", pushed.Said); + } + + [TestMethod] + public async Task ADemoIsReadOnlyThroughGitToo() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + await fixture.SeedDemosAsync(); + + using var git = new GitClient(); + + await git.CloneAsync(fixture.Host.GetUrl("/editor/demo-crud/demo-crud.git"), "demo"); + + git.Write("demo", "assets/mine.txt", "mine"); + await git.CommitAsync("demo", "Mine"); + + var pushed = await git.TryAsync("demo", "push"); + + Assert.AreNotEqual(0, pushed.ExitCode); + Assert.Contains("is a demo and read only", pushed.Said); + } + + #endregion + + #region Helpers + + /// <summary> + /// Project.cs holding the given snippet, the way the repository writes it. + /// </summary> + internal static string ProjectOf(string snippet) => GenHTTP.Lambda.Services.Deployment.ProjectSnippet.ForRepository(snippet); + + #endregion + +} diff --git a/tests/GenHTTP.Lambda.Tests/Git/Repository.cs b/tests/GenHTTP.Lambda.Tests/Git/Repository.cs new file mode 100644 index 00000000..1c6ac0a0 --- /dev/null +++ b/tests/GenHTTP.Lambda.Tests/Git/Repository.cs @@ -0,0 +1,89 @@ +using System.Net; + +using GenHTTP.Lambda.Api.Model; +using GenHTTP.Lambda.Services.Deployment.Model; +using GenHTTP.Lambda.Tests.Infrastructure; + +using GenHTTP.Testing; + +namespace GenHTTP.Lambda.Tests.Git; + +/// <summary> +/// What the tests of a lambda's repository do through the API, beside git. +/// </summary> +internal static class Repository +{ + + /// <summary> + /// Where whoever holds the editor key clones the lambda from. + /// </summary> + public static string EditorUrl(this LambdaFixture fixture, LambdaResponse lambda) => fixture.Host.GetUrl($"/editor/{lambda.PrivateKey}/{lambda.PublicKey}.git"); + + /// <summary> + /// Where anybody clones a published source from. + /// </summary> + public static string SourceUrl(this LambdaFixture fixture, string publicKey) => fixture.Host.GetUrl($"/source/{publicKey}.git"); + + /// <summary> + /// A lambda that says something, and nothing else. + /// </summary> + public static string Says(string text) => $"return Content.From(Resource.FromString(\"{text}\"));\n"; + + /// <summary> + /// Saves a version through the API, the way the editor does. + /// </summary> + public static async Task<int> SaveAsync(this LambdaFixture fixture, LambdaResponse lambda, string change, params LambdaFile[] files) + { + using var saved = await fixture.SendAsync(HttpMethod.Post, $"/api/v1/lambdas/{lambda.PrivateKey}/versions", new VersionRequest(files, null, change)); + + Assert.AreEqual(HttpStatusCode.Created, saved.StatusCode, await saved.Content.ReadAsStringAsync()); + + return (await saved.GetContentAsync<VersionResponse>()).Version; + } + + public static async Task<List<VersionResponse>> VersionsAsync(this LambdaFixture fixture, LambdaResponse lambda) + => await (await fixture.GetAsync($"/api/v1/lambdas/{lambda.PrivateKey}/versions")).GetContentAsync<List<VersionResponse>>(); + + public static async Task<VersionContentResponse> VersionAsync(this LambdaFixture fixture, LambdaResponse lambda, int version) + => await (await fixture.GetAsync($"/api/v1/lambdas/{lambda.PrivateKey}/versions/{version}")).GetContentAsync<VersionContentResponse>(); + + public static async Task<List<FeatureResponse>> FeaturesAsync(this LambdaFixture fixture, LambdaResponse lambda) + => await (await fixture.GetAsync($"/api/v1/lambdas/{lambda.PrivateKey}/features")).GetContentAsync<List<FeatureResponse>>(); + + public static async Task<FeatureContentResponse> FeatureAsync(this LambdaFixture fixture, LambdaResponse lambda, string key) + => await (await fixture.GetAsync($"/api/v1/lambdas/{lambda.PrivateKey}/features/{key}")).GetContentAsync<FeatureContentResponse>(); + + public static async Task<FeatureResponse> CreateFeatureAsync(this LambdaFixture fixture, LambdaResponse lambda, string name) + { + using var created = await fixture.SendAsync(HttpMethod.Post, $"/api/v1/lambdas/{lambda.PrivateKey}/features", new CreateFeatureRequest(name)); + + Assert.AreEqual(HttpStatusCode.Created, created.StatusCode, await created.Content.ReadAsStringAsync()); + + return await created.GetContentAsync<FeatureResponse>(); + } + + public static async Task PutFeatureAsync(this LambdaFixture fixture, LambdaResponse lambda, string key, string change, params LambdaFile[] files) + { + using var put = await fixture.SendAsync(HttpMethod.Put, $"/api/v1/lambdas/{lambda.PrivateKey}/features/{key}/files", new FeatureFilesRequest(files, Change: change)); + + Assert.AreEqual(HttpStatusCode.OK, put.StatusCode, await put.Content.ReadAsStringAsync()); + } + + public static async Task PublishAsync(this LambdaFixture fixture, LambdaResponse lambda, string? license = null) + { + using var published = await fixture.SendAsync(HttpMethod.Put, $"/api/v1/lambdas/{lambda.PrivateKey}/source", new SourceRequest(license, null)); + + Assert.AreEqual(HttpStatusCode.OK, published.StatusCode, await published.Content.ReadAsStringAsync()); + } + + /// <summary> + /// What the lambda answers right now. + /// </summary> + public static async Task<string> CallAsync(this LambdaFixture fixture, string path) + { + using var response = await fixture.GetAsync(path); + + return await response.Content.ReadAsStringAsync(); + } + +} diff --git a/tests/GenHTTP.Lambda.Tests/Git/SourceRepositoryTests.cs b/tests/GenHTTP.Lambda.Tests/Git/SourceRepositoryTests.cs new file mode 100644 index 00000000..24541c37 --- /dev/null +++ b/tests/GenHTTP.Lambda.Tests/Git/SourceRepositoryTests.cs @@ -0,0 +1,136 @@ +using System.Net; + +using GenHTTP.Lambda.Services.Deployment.Model; +using GenHTTP.Lambda.Tests.Infrastructure; + +namespace GenHTTP.Lambda.Tests.Git; + +/// <summary> +/// A published source cloned by anybody: the same versions the owner reads, +/// none of the owner's features, nothing written back - and never the data. +/// </summary> +[TestClass] +public sealed class SourceRepositoryTests +{ + + [TestMethod] + public async Task APublishedSourceIsTheOwnersHistoryWithoutTheFeatures() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("opened"); + + await fixture.SaveAsync(lambda, "Says hello", new LambdaFile(LambdaSource.EntryName, Repository.Says("hello"))); + + var feature = await fixture.CreateFeatureAsync(lambda, "Secret plans"); + + await fixture.PutFeatureAsync(lambda, feature.Key, "Plans", new LambdaFile(LambdaSource.EntryName, Repository.Says("plans"))); + + await git.CloneAsync(fixture.EditorUrl(lambda), "owner"); + + await fixture.PublishAsync(lambda); + + await git.CloneAsync(fixture.SourceUrl("opened"), "public"); + + Assert.AreEqual(await git.ReadAsync("owner", "rev-parse", "main"), await git.ReadAsync("public", "rev-parse", "main"), "the same commits"); + + Assert.AreEqual("origin/HEAD -> origin/main\norigin/main", string.Join('\n', (await git.ReadAsync("public", "branch", "-r")).Split('\n').Select(l => l.Trim()))); + + var plans = await git.ReadAsync("owner", "rev-parse", $"origin/{feature.Branch}"); + + var fetched = await git.TryAsync("public", "fetch", "origin", plans); + + Assert.AreNotEqual(0, fetched.ExitCode, "a feature's commit is not found by its id either"); + } + + [TestMethod] + public async Task ACommitMadeWhileTheSourceIsPublishedCarriesItsLicense() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("licensed"); + + await fixture.PublishAsync(lambda, "Apache-2.0"); + + await git.CloneAsync(fixture.SourceUrl("licensed"), "licensed"); + + Assert.Contains("Apache License", git.Read("licensed", "LICENSE")); + } + + [TestMethod] + public async Task ASourceNotPublishedIsNoRepository() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + await fixture.CreateLambdaAsync("closed"); + + var cloned = await git.TryAsync(null, "clone", fixture.SourceUrl("closed"), "closed"); + + Assert.AreNotEqual(0, cloned.ExitCode); + + using var answer = await fixture.GetAsync("/source/closed.git/info/refs?service=git-upload-pack"); + + Assert.AreEqual(HttpStatusCode.NotFound, answer.StatusCode, "the same as a key nobody has"); + } + + [TestMethod] + public async Task APublishedSourceIsNotPushedTo() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("readonly"); + + await fixture.PublishAsync(lambda); + + await git.CloneAsync(fixture.SourceUrl("readonly"), "readonly"); + + git.Write("readonly", "assets/mine.txt", "mine"); + await git.CommitAsync("readonly", "Mine"); + + var pushed = await git.TryAsync("readonly", "push"); + + Assert.AreNotEqual(0, pushed.ExitCode); + + Assert.HasCount(1, await fixture.VersionsAsync(lambda)); + } + + [TestMethod] + public async Task TheDataIsNeverInTheRepository() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + using var git = new GitClient(); + + var lambda = await fixture.CreateLambdaAsync("private", "demo-crud"); + + await fixture.DeployAsync(lambda.PrivateKey); + + using var added = await fixture.SendAsync(HttpMethod.Post, "/lambda/private/tasks/", new { title = "a-record-nobody-else-sees" }); + + Assert.IsTrue(added.IsSuccessStatusCode, await added.Content.ReadAsStringAsync()); + + using var secrets = await fixture.SendAsync(HttpMethod.Put, $"/api/v1/lambdas/{lambda.PrivateKey}/data/secrets"); + using var secret = await fixture.SendAsync(HttpMethod.Put, $"/api/v1/lambdas/{lambda.PrivateKey}/secrets/API_KEY", new { value = "a-value-nobody-else-sees" }); + + Assert.AreEqual(HttpStatusCode.OK, secret.StatusCode, await secret.Content.ReadAsStringAsync()); + + var clone = await git.CloneAsync(fixture.EditorUrl(lambda), "private"); + + foreach (var file in Directory.EnumerateFiles(clone, "*", SearchOption.AllDirectories)) + { + var content = await File.ReadAllTextAsync(file); + + Assert.DoesNotContain("a-record-nobody-else-sees", content, file); + Assert.DoesNotContain("a-value-nobody-else-sees", content, file); + } + } + +} diff --git a/tests/GenHTTP.Lambda.Tests/Infrastructure/GitClient.cs b/tests/GenHTTP.Lambda.Tests/Infrastructure/GitClient.cs new file mode 100644 index 00000000..4bb8d796 --- /dev/null +++ b/tests/GenHTTP.Lambda.Tests/Infrastructure/GitClient.cs @@ -0,0 +1,217 @@ +using System.Diagnostics; +using System.Text; + +namespace GenHTTP.Lambda.Tests.Infrastructure; + +/// <summary> +/// What running git came to. +/// </summary> +internal sealed record GitResult(int ExitCode, string Output, string Error) +{ + + public bool Success => ExitCode == 0; + + /// <summary> + /// What git printed, both streams - a push says what the server said on the second. + /// </summary> + public string Said => Output + Error; + + public override string ToString() => $"exit code {ExitCode}\n--- stdout ---\n{Output}\n--- stderr ---\n{Error}"; + +} + +/// <summary> +/// The git command line client, in a home of its own, so a test neither +/// depends on nor changes how the machine it runs on has git configured. +/// </summary> +/// <remarks> +/// It checks every object it receives (<c>transfer.fsckObjects</c>), so a +/// commit or a tree the platform serves wrongly fails the test that fetched it. +/// </remarks> +internal sealed class GitClient : IDisposable +{ + private static readonly TimeSpan Timeout = TimeSpan.FromMinutes(2); + + #region Get-/Setters + + /// <summary> + /// Where the clones of this client are made. + /// </summary> + public string Root { get; } + + private string Home { get; } + + #endregion + + #region Initialization + + public GitClient() + { + Root = Path.Combine(Path.GetTempPath(), "genhttp-lambda-git", Guid.NewGuid().ToString("n")); + Home = Path.Combine(Root, ".home"); + + Directory.CreateDirectory(Home); + + File.WriteAllText(Path.Combine(Home, ".gitconfig"), """ + [user] + name = Somebody + email = somebody@example.com + [init] + defaultBranch = main + [transfer] + fsckObjects = true + [core] + autocrlf = false + [advice] + detachedHead = false + [commit] + gpgsign = false + [gc] + auto = 0 + [pull] + rebase = true + """); + } + + #endregion + + #region Functionality + + /// <summary> + /// Where a clone of this client is. + /// </summary> + public string PathOf(string clone) => Path.Combine(Root, clone); + + /// <summary> + /// Runs git, failing the test where it fails. + /// </summary> + public async Task<GitResult> RunAsync(string? clone, params string[] arguments) + { + var result = await TryAsync(clone, arguments); + + if (!result.Success) + { + Assert.Fail($"git {string.Join(' ', arguments)} failed with {result}"); + } + + return result; + } + + /// <summary> + /// Runs git, whatever it comes to. + /// </summary> + public async Task<GitResult> TryAsync(string? clone, params string[] arguments) + { + var start = new ProcessStartInfo("git") + { + WorkingDirectory = clone != null ? PathOf(clone) : Root, + RedirectStandardOutput = true, + RedirectStandardError = true, + UseShellExecute = false, + StandardOutputEncoding = Encoding.UTF8, + StandardErrorEncoding = Encoding.UTF8 + }; + + foreach (var argument in arguments) + { + start.ArgumentList.Add(argument); + } + + start.Environment["HOME"] = Home; + start.Environment["XDG_CONFIG_HOME"] = Home; + start.Environment["GIT_CONFIG_NOSYSTEM"] = "1"; + start.Environment["GIT_TERMINAL_PROMPT"] = "0"; + start.Environment["LANG"] = "C"; + start.Environment["LC_ALL"] = "C"; + + using var process = Process.Start(start) ?? throw new InvalidOperationException("git did not start"); + + var output = process.StandardOutput.ReadToEndAsync(); + var error = process.StandardError.ReadToEndAsync(); + + using var cancellation = new CancellationTokenSource(Timeout); + + try + { + await process.WaitForExitAsync(cancellation.Token); + } + catch (OperationCanceledException) + { + process.Kill(true); + Assert.Fail($"git {string.Join(' ', arguments)} did not finish within {Timeout}"); + } + + return new GitResult(process.ExitCode, await output, await error); + } + + /// <summary> + /// Clones a repository and checks everything it received. + /// </summary> + public async Task<string> CloneAsync(string url, string clone) + { + await RunAsync(null, "clone", url, clone); + + await RunAsync(clone, "fsck", "--strict"); + + return PathOf(clone); + } + + /// <summary> + /// What a file of a clone holds. + /// </summary> + public string Read(string clone, string path) => File.ReadAllText(Path.Combine(PathOf(clone), path)); + + /// <summary> + /// Writes a file of a clone, its folders included. + /// </summary> + public void Write(string clone, string path, string content) + { + var file = Path.Combine(PathOf(clone), path); + + Directory.CreateDirectory(Path.GetDirectoryName(file)!); + + File.WriteAllText(file, content); + } + + /// <summary> + /// Changes a file of a clone: the text found, replaced. + /// </summary> + public void Change(string clone, string path, string find, string replace) + { + var content = Read(clone, path); + + Assert.Contains(find, content, $"{path} holds what is to be changed"); + + Write(clone, path, content.Replace(find, replace)); + } + + /// <summary> + /// Commits everything that changed in a clone, new files and removed ones included. + /// </summary> + public async Task<GitResult> CommitAsync(string clone, string message) + { + await RunAsync(clone, "add", "-A"); + + return await RunAsync(clone, "commit", "-q", "-m", message); + } + + /// <summary> + /// What a command printed, trimmed - a commit id, a list of names. + /// </summary> + public async Task<string> ReadAsync(string clone, params string[] arguments) => (await RunAsync(clone, arguments)).Output.Trim(); + + public void Dispose() + { + try + { + Directory.Delete(Root, true); + } + catch (Exception) + { + // the temporary folder of the machine is cleaned up by the system + } + } + + #endregion + +}