Skip to content

Show owners who visits their lambda - #114

Merged
Kaliumhexacyanoferrat merged 2 commits into
mainfrom
analytics
Oct 12, 2026
Merged

Kaliumhexacyanoferrat merged 2 commits into
mainfrom
analytics

Conversation

@Kaliumhexacyanoferrat

@Kaliumhexacyanoferrat Kaliumhexacyanoferrat commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Owners can now see who visits their lambda: an Analytics section in the editor (full view under Run, and in the simple view), opt-in, with visitors, page views and requests per day and the pages, referring sites, countries, devices, browsers, systems and crawlers behind them.

Screenshots

After the review: the crawler card names Google, Bing, OpenAI, Apple and Perplexity only when they were confirmed by their published addresses (check mark), and groups every other bot by its kind.

Light Dark German

The first round:

Premium, 30 days (light) Page views picked, hovering a day (dark)
Free lambda in the simple view: 7 days shown, the rest kept The same in German
Before it is switched on Switching off Phone

The figures at the top are tabs for the chart below them. Each one shows the change against the period before, where the tier shows that period too. The chart has a column per day (per week for 12 months), with a hover tooltip and its values as a table one click away. The breakdowns are ranked bars. Devices are one bar of shares with a legend. Its four colours were checked for colour blindness in light and dark mode. The tab does not poll: it reads again when the page becomes visible.

How it works

  • One visit per request, read by everything. LambdaVisitConcern (new folder Services/Visits) builds a LambdaVisit once per request to a lambda or a preview (status, timing, bytes, path, domain, and the caller's country and agent from CallerInfo) and only leaves it in the request's properties. Every reader takes it from there: the stats and overview figures (LambdaActivityConcern, now a reader that skips previews), the request log (CallerConcern, which no longer measures lambda requests itself), and the analytics. The overview and stats look exactly as before. TelemetryConcern still counts every request to the server, the platform's included, so it has no visit to read.
  • Collected on the request, written by a job. AnalyticsConcern reads the visit from the request and, only for lambdas with analytics on and never for previews, adds it to AnalyticsQueue (a lock-free ConcurrentQueue with an interlocked counter, capped at 250k; anything dropped is logged). AnalyticsJob runs every LAMBDA_ANALYTICS_INTERVAL_SECONDS (default 300). It takes what is queued, sums it into one row per lambda/day/value, and writes everything in one transaction. Referrer host, device, browser, system and crawler are worked out there, never on the request. The Referer and Sec-Fetch-Dest headers are only read for lambdas that count visits, and they are read before the body is bound.
  • analytics.db, a star schema. lambdas, plus fact tables days, pages, referrers, countries, devices, browsers, systems and crawlers, each keyed (lambda, year, month, day[, value]) and WITHOUT ROWID. Reads and retention use key ranges. It has its own Evolve migrations (Services/Analytics/Migrations) and uses plain synchronous commands. Only the switch lives in lambda.db (V21: lambdas.analytics_since), and ResolvedLambda carries it, so serving a request never asks a database.
  • What counts. A hit is any request. A view is a page being opened: a browser navigation (Sec-Fetch-Dest: document, including a 304), or a successful GET that returns HTML to a client that doesn't send that header. Paths are kept only for answered requests, capped at 200 per day (the rest go under (other)), so scanners can't fill a day. A referrer is stored as its host only, without www., and only when it is another site.
  • Visitors. A visitor is a person: a hash of address + agent under a random key made per day and per lambda. The hash is held in memory for today and yesterday and is never written down, so it counts visitors per day. Bots are never visitors. No cookies.
  • Bots, hardened. Programs are recognised with crawler-user-agents (MIT, about 1,500 crawlers, last updated 2026-09-29). The release ships a copy, and CrawlerJob downloads a fresh one daily. Bots are counted by the list's kinds: search engine, AI crawler, link preview, SEO, monitoring, scanner, scripts…
  • Named crawlers are confirmed. Google, Bing, OpenAI, Apple and Perplexity are only named when the connection came from an address range they publish (also fetched daily). Anything else claiming to be them counts as "unconfirmed". The check uses the connection's own address, never X-Forwarded-For, which anybody can set. Production terminates TLS itself, so that address is the real client; behind a proxy nothing would be confirmed.
  • Browsers, systems and devices. These come from the user agent and, for Chromium browsers, from the client hints (Sec-CH-UA*), because Chromium's user agent no longer says which phone or OS it runs on. Hints are only believed alongside a Chromium user agent, and a value is only taken when it is a known one, so nobody can write their own words into an owner's figures. In-app browsers (Instagram, Facebook, TikTok…) are named.
  • Hostile user agents stay cheap. Matching takes time in proportion to the user agent's length: the list's 1,465 plain-text patterns are searched in one pass (SearchValues), the 25 real expressions run without backtracking, and at most 500 characters are read. Results are cached per distinct user agent.
  • Window and retention. One year is kept (AnalyticsStore.Kept = 366 days). How much the owner sees is a new tier limit in the panel, Analytics shown: 7 days on free, 365 on premium. A lambda that becomes premium therefore sees its history.
  • Not counted: feature previews and demos.
  • Deleting a lambda removes its rows, through IAnalyticsStore.

API: GET / PUT / DELETE /api/v1/lambdas/:privateKey/analytics (?days=). MCP: a new analytics tool that reads, and switches analytics on with enable: true only when the user asks. platform_guide has an analytics entry telling agents not to build a counter of their own. Documented in the README (new Analytics section, API table, limits, configuration, services, database), /docs (Who visits it), the privacy page (new section about the visitors of lambdas), and CLAUDE.md. Translated into all 12 languages with the translator agents; the section's name is added to glossary.json.

Decisions to confirm

These are mine and are recorded in CLAUDE.md. Please push back on any of them:

  1. Switching off deletes what was counted, like switching off a kind of data. That is why agents can switch analytics on but never off.

  2. It is in the simple view too. It is about the audience, not the code.

  3. Daily unique visitors via an in-memory salted hash, people only. On an installation with LAMBDA_LOG_CLIENT_ADDRESS=false there is no address to hash, so visitors stay at 0.

  4. The caller's country is now looked up even when addresses aren't logged. It names nobody and analytics need it. As a side effect, log lines on such installations now show a country.

  5. The default range is 30 days (7 on free).

  6. CLAUDE.md's PR-screenshot rule now describes the single pr-assets branch, matching your earlier decision.

  7. Crawler data is downloaded at runtime. The server fetches the list and the five address lists daily (LAMBDA_ANALYTICS_CRAWLER_LISTS=false turns this off). Like the geo data, it only downloads; nothing about visitors is sent. The privacy page says so.

Tests

  • New VisitorTests: all 2,120 example agents that ship with the crawler list come out as bots; 17 real browser agents come out with the right browser, system and device (Cubot phones, in-app browsers and Chrome's frozen Android agent included); hints are only believed from Chromium and only with known values; a claimed Googlebot is named only from a published address, and never through a forwarding header; a broken download keeps the previous list; a 260 kB hostile agent stays cheap.
  • New AnalyticsTests (12 tests): off until switched on; off deletes; demos refused; page/referrer/country/device/browser/crawler classification over real requests; views vs. hits; the daily path cap; previews not counted; free shows 7 days and premium shows the history; one-year expiry; deleting a lambda deletes its rows; the MCP tool.
  • One new test in LimitsTests for the new limit.
  • dotnet test: 830 passed. npm run build passes. translations check reports 0 errors.

I ran the app locally (Kestrel, Vite) with live requests and 120 days of seeded history to make the screenshots.

🤖 Generated with Claude Code

Kaliumhexacyanoferrat and others added 2 commits October 12, 2026 14:26
An Analytics section in the editor, in both views: visitors, page views
and requests per day, and the pages, referring sites, countries, devices,
browsers, systems and crawlers behind them. Opt-in; switching it off
deletes what was counted.

Every request to a lambda is made into one LambdaVisit by the activity
concern, which counts the overview's figures from it and leaves it on the
request; the analytics concern reads it from there and puts it on a
lock-free, bounded queue. A job writes the queue every five minutes into
analytics.db, a star schema of its own (lambdas, and a fact table per
statistic keyed by lambda, year, month and day), so nothing is written
while a visitor waits.

A year is kept; a free lambda shows seven days of it and a premium one
all of it - a tier limit in the panel. Visitors are told apart for a day
by a hash under a random daily key, held in memory only. Documented in
the README, /docs, platform_guide, the new analytics MCP tool and the
privacy page; translated into every language.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…s apart

One concern, LambdaVisitConcern, makes the LambdaVisit of a request to a
lambda or a preview and does nothing else. The overview's figures, the
request log and the analytics read it off the request and leave out what is
not theirs; nothing measures a request to a lambda twice.

Bots are told from people by crawler-user-agents (MIT, maintained), kept in
the release and fetched again daily, and counted by its kinds. Google, Bing,
OpenAI, Apple and Perplexity are only named where the connection came from
an address they publish - never a forwarding header's - and counted as an
unconfirmed claim otherwise. Visitors are people only. Browsers, systems and
devices come from the agent's tokens and Chromium's client hints, taken only
where they name something known. Matching stays linear in an agent's length:
the list's plain text is searched in one pass, its few expressions run
without backtracking, and only 500 characters are read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Kaliumhexacyanoferrat
Kaliumhexacyanoferrat merged commit 0755076 into main Oct 12, 2026
3 checks passed
@Kaliumhexacyanoferrat
Kaliumhexacyanoferrat deleted the analytics branch October 12, 2026 16:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant