Skip to content

feat(functions-compiler): every build records a returned 5xx as one _b44_error line - #678

Open
egidijusj wants to merge 8 commits into
feat/functions-compiler-source-mapsfrom
feat/functions-compiler-error-records
Open

egidijusj wants to merge 8 commits into
feat/functions-compiler-source-mapsfrom
feat/functions-compiler-error-records

Conversation

@egidijusj

@egidijusj egidijusj commented Oct 23, 2026 •

Copy link
Copy Markdown
Collaborator

Note

Description

A backend function that catches its own error and returns a 5xx leaves Cloudflare nothing to record — the Tail Worker sees a status and no reason. This PR adds a small, self-contained recorder to every generated Worker entry (single-function and per-app; not actors) that logs exactly one _b44_error line for a handled 5xx, carrying the message, an optional error name and a minified stack, with secrets scrubbed. There is no build option: the off switch is the request, so the recorder is inert unless the call carries x-b44-feed-context, which the backend sends only for production calls of opted-in apps. An app is therefore ready the moment it opts in, with no rebuild — including apps compiled by the public CLI’s own COMPILE_MODES path.

Related Issue

None. Stacked on #677 (source maps) — this PR’s base is feat/functions-compiler-source-maps, so the diff shows only its own changes; GitHub retargets it to main when #677 merges. Replaces #669 (opened from a fork, where CI’s secret-dependent jobs cannot run); the earlier review discussion is there.

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactoring (no functional changes)
  • Other (please describe):

Changes Made

  • New packages/functions-compiler/src/error-records.ts — ERROR_RECORDS_PATCH, a dependency-free JS prelude inlined into the generated entry after the console patch. It provides _b44ErrorStore (a per-request store created only when the header is present), a console.* wrapper that remembers the last 8 logged Errors, and _b44RecordError(res), which returns res untouched and does all its work on a clone inside ctx.waitUntil.
  • Message selection, in order: a logged Error whose message the JSON body repeats (the newest match among the last 8, so a later unrelated warning cannot hide it) → the JSON body itself (≤ 2,048 bytes read within 50 ms; first non-empty error, message, detail, details, error.message, skipping error when it is exactly "Internal server error" and details is present) → the newest logged Error. No message means no record.
  • Scrubbing and bounds: every env-binding / runtime-secret string of 8+ characters — including strings nested in JSON-valued ones, leading whitespace allowed — becomes [redacted], longest first, before anything is cut. name ≤ 64 chars, message ≤ 500 UTF-8 bytes (toWellFormed(), cut on a code point), stack ≤ 4 KB (cut at a line end). The stack is minified by design; a consumer remaps it with the version’s source_map from feat(functions-compiler): opt-in external source maps with app paths #677.
  • worker-entry.ts: both buildEntrySource and buildAppEntrySource now inline ERROR_RECORDS_PATCH, spread ..._b44ErrorStore(request) into the request store, and route the handler through a shared HANDLER_CALL (_b44RecordError(await handler(request, info))) so the recorder sees the handler’s own response, inside telemetry and invocation logs. No new build flag and no banner field — the banner keys stay functions, telemetry, runtimeSecrets, compiler.
  • Crash path untouched: an uncaught throw is still logged and rethrown for Cloudflare to capture; a test pins that block’s exact text.
  • Tests: new test/error-records.e2e.test.ts (550 lines) running every case in real workerd via Miniflare and reading records through a real Tail Worker; test/worker-entry.test.ts gains a recorder suite; the four pinned entry snapshots are regenerated (each gains the prelude plus two changed lines).
  • README: a new "Error records for a handled 5xx" section documenting the consumer contract, plus corrections to the banner / byte-stability notes (the single-function lane is no longer byte-compared against a second engine, since apper’s bundler service now runs this package).

Testing

  • I have tested these changes locally
  • I have added/updated tests as needed
  • All tests pass (npm test)

Checklist

  • My code follows the project’s style guidelines
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation (if applicable)
  • My changes generate no new warnings
  • I have updated docs/ (AGENTS.md) if I made architectural changes

Additional Notes

What a consumer actually sees. The recorder calls console.error({ _b44_error: record }), which goes through the console patch — so a Tail Worker does not see that object. It gets one log entry whose level is "log" and whose message is a single argument, the patch’s wrapper: { _b44_env, _b44_function?, level: "error", message: "<JSON string>" }, where message is the JSON of { "_b44_error": { v: 1, kind: "response", status, name?, message, stack? } }. A consumer filters on the wrapper’s level === "error" and reads JSON.parse(wrapper.message)._b44_error. apper’s error tail (workers/base44-log-tail/src/error-tail.ts) parses the raw object today, so it drops every record until it is updated — harmless to merge before that; records are simply dropped.

Size and release impact. About +3.0 KB raw / +1.2 KB gzipped per module (single-function 41,400 → 44,422; per-app 43,159 → 46,191). Without the header the cost is one headers.has per request and one context lookup per console.* call. Every function’s bytes change once on its next deploy after apper’s bundler picks up this version; where a version’s identity is the hash of the compiled output (the CLI’s versions path), that mints one new version per app on its next publish — which a package-version bump would do anyway.

Invocation logs are unaffected: the line is logged inside waitUntil, after the response has gone back, so it never lands in the X-B44-Invocation-Logs header. It is still an ordinary stamped line, so per-function log views show it as an error line.

No record is written for: responses the wrapper builds rather than the handler (the per-app "failed to initialize" 500, the 503/404 routing answers), the per-app crash response, and actor entries.

Open question for review: the recorder treats the header’s presence as the switch. Please confirm the dispatcher strips a client-supplied x-b44-feed-context before invoking a function. The impact of a spoof is small — it only records a 5xx the function really returned, scrubbed, into that app’s own logs.

Conflict note: #662 (edge errors on outbound fetches) also edits worker-entry.ts and its snapshots; whichever lands second needs a rebase and a snapshot regeneration.

Version/changelog: deliberately none — the Manual Functions Compiler Publish workflow bumps this package’s version in its own release commit, and the root CHANGELOG.md is the CLI’s.

Reported results (from the author’s run; not re-executed while generating this description): full compiler suite 490/490; typecheck, lint and knip clean; the CLI’s tests/core/version-functions.spec.ts 4/4.


🤖 Generated by Claude | 2026-10-23 15:09 UTC | b8808dc

egidijusj and others added 8 commits October 20, 2026 10:22
`errorRecords: true` on `bundle`/`bundleApp` bakes a recorder into both
generated entries. It is inert unless the request carries
`x-b44-feed-context`. For a handler `Response` with status >= 500 it
returns that same response at once and, on a clone inside
ctx.waitUntil, logs one
`{ _b44_error: { v: 1, kind: "response", status, name?, message, stack? } }`
through the console patch.

The message comes from an Error logged during the request whose message
the JSON body repeats, else the body's first non-empty error / message /
detail / details / error.message (2 KB, 50 ms; the SSO
"Internal server error" placeholder skipped when details is present),
else the last logged Error. The stack comes only from that Error. Env and
runtime secrets of 8+ characters are scrubbed, longest first, before
name (64 chars), message (500 bytes) and stack (4 KB) are cut. Any
failure writes nothing; the crash path is unchanged.

Off by default; the public CLI never sets it, and flag-off entries and
banners keep their bytes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It is the raw Error.stack read inside the Worker, and Cloudflare remaps
only the exceptions it captures, never a logged string. Say so, and that
a consumer remaps it with the same version's source_map.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y recent logged Error

Two review findings on the error recorder:

- A JSON-valued binding with leading whitespace (`\n{"db":{...}}`) was
  not parsed, so the secrets nested in it were logged unredacted.
  Detect JSON after trimStart(); the original string stays a candidate.
- A single saved Error was overwritten on every log, so a later
  unrelated warning lost the matching Error's name and stack. Keep the
  last 8 Errors logged in the request, use the newest one whose message
  the body repeats, and the newest one only as the fallback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The error recorder now replaces a matched secret with `[redacted]`
instead of `[secret]`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop the `errorRecords` option. Both generated entries now always append
the recorder, route the handler's response through `_b44RecordError`
and seed `_b44ErrorStore(request)`, including the public CLI's own shard
builds and bundleApp's probe compiles. An app is ready for error
reporting the moment it opts in, with no rebuild.

The off switch is the request: the recorder does nothing unless it
carries `x-b44-feed-context`, which the backend sends only for
production calls of opted-in apps while its tail switch is on. Runtime
behaviour is unchanged, crash path included.

The banner loses its `errorRecords` field; `compiler` says which builds
carry the record. This changes every build's output once: about +3 KB
raw (+1.2 KB gzipped) per module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cord

The record is logged through the console patch, not raw: a Tail Worker
gets one entry with level "log" whose single argument is the patch's
wrapper `{ _b44_env, _b44_function?, level: "error", message }`, and
`message` is the JSON string of `{ "_b44_error": {...} }`. apper's error
tail is the consumer. Pin that entry for both entries, so a change to
the console patch fails here instead of silently in production, and
state the shape in the README and the module doc. No change to what is
emitted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Package Preview Available!


Install this PR's preview build with npm:

npm i @base44-preview/cli@0.2.2-pr.678.b8808dc

Prefer not to change any import paths? Install using npm alias so your code still imports base44:

npm i "base44@npm:@base44-preview/cli@0.2.2-pr.678.b8808dc"

Or add it to your package.json dependencies:

{
  "dependencies": {
    "base44": "npm:@base44-preview/cli@0.2.2-pr.678.b8808dc"
  }
}

Preview published to npm registry — try new features instantly!

@egidijusj

Copy link
Copy Markdown
Collaborator Author

Description

Every build of a backend function now carries a small recorder in its generated Worker entry. When a function catches its own error and returns a 5xx, the recorder logs one _b44_error line saying what went wrong, so the Production Agent's Tail Worker has a reason to file, not just a status.

There is no build option. The off switch is the request: the recorder does nothing unless the request carries x-b44-feed-context. The backend sends that header only for production calls of apps that opted in, while its tail switch is on. Turning that switch off silences the record in every bundle, whatever built it.

Stacked on base44/cli#677 (source maps). This PR's base is #677's branch, so its diff shows only its own changes. When #677 merges, GitHub retargets this PR to main. It replaces base44/cli#669, which was opened from a fork, where CI's secret-dependent jobs can't run; the earlier review discussion is there.

Why it's in every build

  • An app is ready the moment it opts in, with no rebuild. Behind a build flag, only builds that set the flag would carry the record.
  • The public CLI's own builds get it too. The CLI compiles shards locally with its own COMPILE_MODES (packages/cli/src/core/version/functions.ts), which would never have set a flag.
  • It costs little. About +3.0 KB raw / +1.2 KB gzipped per module (single-function: 41,400 → 44,422 bytes; per-app: 43,159 → 46,191). Without the header, it costs one headers.has per request and one context lookup per console.* call. Source maps (feat(functions-compiler): opt-in external source maps with app paths #677) keep their flag, because they cost compile time and memory and are useless unless the caller stores them.

Request and response changes

None. An earlier revision of this PR added an errorRecords option; it's gone. Nothing shipped with it, so no caller sends it and no compatibility shim is needed. The build banner is unchanged from main: the errorRecords field is gone, and the banner's compiler version tells a consumer whether a build carries the record.

What a consumer sees (the contract)

The recorder calls console.error({ _b44_error: record }). That goes through the console patch, so a Tail Worker does not see that object. It gets one log entry:

Field Value
TailItem log level "log". The patch writes every level through the original console.log
message an array with one argument: the patch's wrapper object
wrapper's keys _b44_env, _b44_function (per-app bundles only), level, message
wrapper's level "error"
wrapper's message the JSON string of { "_b44_error": { v: 1, kind: "response", status, name?, message, stack? } } (name/stack omitted when absent)

A consumer keeps entries whose wrapper level is "error" and reads JSON.parse(wrapper.message)._b44_error. apper's error tail (workers/base44-log-tail/src/error-tail.ts) is that consumer, and today it parses the raw object, so it drops every record until it's changed (level === "error" on the wrapper, then JSON-parse message). Merging this before that change is harmless: records are simply dropped until then. The README and the module doc state this shape, and a test pins it.

Invocation logs. The record is logged inside waitUntil, after the response went back, so it never lands in the X-B44-Invocation-Logs header owners get:

  • the single-function entry has no capture store at all;
  • the per-app entry pushes the line into the capture buffer after _b44AttachInvocationLogs has already written the header.

Confirmed in workerd. It is still an ordinary stamped line, so views built on stamped Workers log lines (such as per-function log queries) show it as an error line whose message is the JSON. That applies only to production calls of opted-in apps, with secrets scrubbed.

Behaviour (src/error-records.ts), unchanged by this revision

  • Inert unless the request carries x-b44-feed-context. Presence is enough; the value isn't checked.

  • Handler Response with status ≥ 500: the same object is returned at once, never rebuilt, delayed or given a header. The recorder clones it and does the rest inside ctx.waitUntil. It sees the handler's own response, before telemetry or invocation logs re-wrap it.

  • Message, in order:

    1. an Error logged (any console.*) during this request whose message the JSON body repeats anywhere. The recorder keeps the last 8 Errors logged in the request and uses the newest one that matches, so a later unrelated warning doesn't hide it;
    2. otherwise the body: JSON only, from the clone, at most 2,048 bytes within 50 ms, then the reader is cancelled (over 2,048 bytes yields nothing). The first non-empty string among error, message, detail, details, error.message; error is skipped when details is present and error is exactly "Internal server error";
    3. otherwise the newest Error logged during this request.

    No message means no record.

  • Stack only from the logged Error whose message was chosen. It's minified: it's that Error's raw stack, read inside the Worker, and Cloudflare remaps only the exceptions it captures. The Production Agent remaps it with the same version's source_map from feat(functions-compiler): opt-in external source maps with app paths #677.

  • Scrub before cutting. Every string of 8+ characters from the request's env bindings and runtime secrets, longest first, becomes [redacted] in the name, message and stack. This includes strings nested in JSON-valued bindings, also when they start with whitespace (such as the private-data-sources manifest, which carries DB passwords). Binding objects (KV, services, Hyperdrive) are skipped.

  • Bounds: name ≤ 64 characters; message ≤ 500 UTF-8 bytes after toWellFormed(), cut on a code point; stack ≤ 4 KB, cut at a line end.

  • Never throws into the request. Any failure writes nothing.

  • Crash path untouched. The per-app entry's catch still does console.error(e) then throw e. A test pins that block's exact text.

This changes every build's output once

Snapshot change, in one line: each of the four pinned worker-entry snapshots (single-function and per-app, telemetry on and off) gains the recorder prelude. In each, exactly two existing lines change: the request store gains ..._b44ErrorStore(request), and the return becomes _b44RecordError(await handler(...)).

For the release: every build's bytes change once, by about +3 KB raw and +1.2 KB gzipped per module. So every function gets new bytes on its next deploy after apper's bundler picks up this version. Where a version's identity is the hash of the compiled output, as in the CLI's versions path, that mints one new version per app on its next publish. A package-version bump does the same anyway.

Byte comparison of the single-function lane

The README said bundle() stays bannerless "which keeps that lane byte-comparable with the engine apper still runs". I found nothing that still compares it.

  • apper's bundler service (infra/base44-userapp-bundler) pins and runs this package (0.1.7), so there's no second engine to compare against.
  • apper's own "byte-identical" notes (function_bundle.py) are about the bundle input, the flat main.ts submission, not the compiled output.
  • I found no hashing of the compiled module in apper's deploy path that decides redeploys.
  • bundle-banner.e2e.test.ts only asserts that bundle() has no banner, which still holds.

I kept bundle() bannerless, since it compiles one function and has nothing to name, and reworded the README. The caveat: this is from a local apper checkout I didn't update, so a newer apper commit could differ. No flag was kept for it.

Open question: client-sent x-b44-feed-context

The recorder treats the header's presence as the switch. Now that every app carries it, it matters whether the dispatcher strips a client-supplied x-b44-feed-context before calling a function. I couldn't find the header anywhere in the local apper checkout, so I can't verify this. The impact of a spoofed header is small: it only records a 5xx the function really returned, with secrets scrubbed, into that app's own logs. Please confirm the dispatcher drops it from client requests.

Tests

test/error-records.e2e.test.ts runs every case in workerd (Miniflare) and reads records through a real Tail Worker, the way production reads them.

  • The raw Tail log entry is pinned (added after review), for both entries. The test checks the log's level ("log"), the argument count (1), the wrapper's exact keys and level ("error"), and that JSON.parse(wrapper.message)._b44_error equals the record. Its comment names apper's error tail as the consumer, so a change to the console patch fails here, not silently in production. With no build option, the header is the only switch: every positive case also asserts there's no record without the header, and the cases that compared an option-off build now compare header-absent vs header-present on the same build.

  • 5xx JSON in each field shape: error, message, detail, details, error.message, first non-empty wins. Both SSO shapes: {error:"Internal server error", details} gives details, and {error:"Internal server error"} gives the placeholder. A near-miss isn't skipped. A 503 reports 503. The per-app entry works too and stamps _b44_function;

  • no record for: a 5xx text body, a 4xx, a 2xx, a 5xx with no message field, a non-Response return, a body over 2 KB, and a streamed body that misses the 50 ms deadline. The client still gets the whole body in the last two;

  • a recorder failure (a Response subclass whose clone() throws): the response stands and no record is written;

  • logged Errors: a matching one gives name, message and stack; it still wins when an unrelated warning is logged after it; the 8-Error bound is pinned (7 later Errors keep the match, 8 push it out); an unrelated one is ignored; the newest is the fallback for a text body; an Error from an earlier request doesn't leak;

  • non-ASCII cut to ≤ 500 bytes on a code point; a lone surrogate becomes U+FFFD; env secrets scrubbed in message and stack, longest first, with short values kept; JSON bindings scrubbed, including whitespace-led ones; stack ≤ 4 KB on a line end; name ≤ 64;

  • the response is identical (status, headers, body) with and without the header, and its headers arrive while the handler's stream is still open;

  • an uncaught throw, in both entries: the same status, body and Tail exceptions with and without the header, and no record.

test/worker-entry.test.ts:

  • the recorder is in both entries under every telemetry × runtimeSecrets combination;
  • it sits inside telemetry and invocation logs;
  • the crash block is unchanged;
  • the banner keys are exactly functions, telemetry, runtimeSecrets, compiler;
  • the four pinned snapshots are regenerated.

Results:

  • full compiler suite: 490/490;
  • typecheck across all packages: clean;
  • lint and knip: clean;
  • the CLI's tests/core/version-functions.spec.ts, its function-compiling publish path: 4/4.

Notes for review

  • Version and changelog. No version bump or changelog entry in this PR, per this package's convention: the Manual Functions Compiler Publish workflow bumps the version in its own release commit, and the root CHANGELOG.md is the CLI's. The release notes should say: every build now carries the _b44_error record, inert until the backend sends x-b44-feed-context. The README says the same.
  • What gets no record:
    • responses the wrapper builds rather than the handler: the per-app "failed to initialize" 500 and the 503/404 routing answers;
    • the per-app crash response that X-B44-Capture-Logs turns an uncaught throw into, which is the crash path;
    • actor entries.
  • Overlap with base44/cli#662 (edge errors on outbound fetches): it also edits worker-entry.ts and its snapshots, so whichever lands second needs a rebase and a snapshot regeneration.
  • "Not delayed" in Miniflare. Miniflare's dispatchFetch hands a streamed body to the caller only once it closes, so the test times the headers.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant