diff --git a/CLAUDE.md b/CLAUDE.md index 674a1a7..bfe381d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -428,6 +428,67 @@ of it into a lambda's pages; the agent does. pages of the host it is served from. Do not bring it back on the site's host. +### Analytics + +An owner can have the visits of their lambda counted - visitors, page views +and requests per day, and the pages, referring sites, countries, devices, +browsers, systems and crawlers behind them - in the editor's **Analytics** +section. + +- **Opt-in - decided by the owner.** Nothing is counted until the owner + switches it on. Switching it off deletes what was counted, like switching + off a kind of data, so an agent may switch it on when the user asks + (`analytics` with `enable`) and never off. A demo and a feature's preview + are not counted. +- **By day, kept for a year, shown per tier - decided by the owner.** Free + shows seven days, premium the year; the window is a tier limit in the + panel (`analytics-days`), the year is `AnalyticsStore.Kept`. What is kept + beyond the window is shown once the lambda is premium, which is the point. +- **One visit per request, read by whoever needs it - decided by the + owner.** `LambdaVisitConcern` makes the `LambdaVisit` once (status, time, + bytes, path, caller's country and agent), for previews too, and does + nothing else; the overview's figures (`LambdaActivityConcern`), the + request log (`CallerConcern`) and the analytics (`AnalyticsConcern`) read + it off the request and decide for themselves what is theirs. Anything new + that wants to know about requests to lambdas reads the visit, and what is + missing is added to it - nothing measures a request to a lambda again. + What only analytics need (the referrer, whether a page was opened, the + client hints) is read only for a lambda that counts visits; what is made of + it (the referrer's host, the kind of device) is worked out by the job, + never on the request. +- **Collected on the request, written by a job - decided by the owner.** A + lock-free queue (`AnalyticsQueue`, bounded) and a job every five minutes + that sums it and writes one transaction. Losing what is queued on a restart + is accepted. Do not write to a database while a request is served. +- **A database of its own, a star - decided by the owner.** `analytics.db`, + `lambdas` and a fact table per statistic keyed by lambda, year, month, day + (and value). Plain commands, synchronous; migrated by Evolve from + `Services/Analytics/Migrations`, numbered on their own. Only the switch is + in the platform's database, so serving reads it from memory. +- **Crawlers by a maintained list and by their addresses, not by guessing - + decided with the owner.** Bots are told from people by + crawler-user-agents (MIT, refreshed daily, the release's copy as fallback) + and counted by its kinds; Google, Bing, OpenAI, Apple and Perplexity are + named only when the connection came from an address they publish - never + a forwarding header's - and an unconfirmed claim otherwise. Visitors are + people only. Browsers, systems and devices come from the agent's tokens + and Chromium's client hints, and a value the sender wrote is only taken + where it is one of a known few. Matching stays linear in the length of an + agent: no backtracking expression ever reads one. +- **Nothing that names a visitor is kept.** Counts only; the referrer as its + host; visitors told apart for a day by a hash under a random key per day + and lambda, held in memory and never written. No cookie, no script in the + lambda. The privacy page says so; keep it saying so. +- **In both views, in the words of the simple one.** It is about the + audience, not the program, so unlike the stats it is in the simple view as + well (after the domain). Charts, not tables: the figures at the top pick + what the columns below show, the breakdowns are ranked bars, the devices + one bar of shares; every chart has its values as a table one click away. + It does not poll - it reads again when the page is looked at again. +- This is said in the README (Analytics), `/docs` (Who visits it), + `platform_guide` (`analytics`), the tool's description and the privacy + page - keep them the same. + ### Pushing, not polling Agents are told never to poll: a page does not fetch on a timer to see @@ -972,5 +1033,7 @@ is in `platform_guide` and the tool descriptions; do not repeat it there. - Tailwind for styling, both light and dark themes, phones included. - Check a change you cannot test in the browser by running the app (see above) and looking at it; type checks and tests are not proof that a UI works. -- PR screenshots: an orphan branch `pr-assets/` with the images at its - root, linked from the PR body by commit-pinned raw URLs. +- PR screenshots: one branch `pr-assets`, unrelated to `main`, with a folder + per topic (`/.png`), a commit added on top for each PR and + linked from the PR body by commit-pinned raw URLs - decided by the owner; + no branch per topic. diff --git a/README.md b/README.md index c7d5670..66642ae 100644 --- a/README.md +++ b/README.md @@ -146,6 +146,7 @@ path. | `GET / PUT / DELETE /lambdas/:privateKey/showcase` | its entry on the showcase: read, list it (title, description, picture), take it off | | `GET /lambdas/:privateKey/domain`, `PUT / DELETE` | the domain of a premium lambda, whether it is served, and what it resolves to | | `GET /showcases`, `GET /showcases/:publicKey/image` | what `/showcase` lists, and a picture | +| `GET / PUT / DELETE /lambdas/:privateKey/analytics` | who visits it, by day, over the last `days` (at most what its tier shows); switches counting on; switches it off and deletes what was counted, see [Analytics](#analytics) | | `GET / PUT / DELETE /lambdas/:privateKey/source` | whether its code is published and under which license, the licenses on offer; publishes it or changes the license (`license`, `author`); takes it down | | `GET /sources` | what `/source` lists (`search`, `order`: `stars`, `updated`, `published`, `skip`, `take`) | | `GET /sources/:publicKey` | a published source: what it is, its license, where it runs, every version and what it changed, a ticket to star it with | @@ -793,6 +794,83 @@ has chosen for themselves, for that lambda, which is kept in their browser and never sent anywhere - so an operator looking at somebody's lambda in the full view leaves it simple for its owner. +### Analytics + +An owner can have the visits of a lambda counted, and reads them in the +editor's **Analytics** section, in both views: visitors, page views and +requests per day, and the pages, referring sites, countries, devices, +browsers, operating systems and crawlers behind them - search engines, AI +assistants, chats fetching a preview. Opt-in: nothing is counted until the +owner switches it on (`PUT …/analytics`, or the `analytics` tool when the +user asks an agent), and switching it off (`DELETE …/analytics`) deletes +what was counted. A demo's visitors are not counted, and neither is a +feature's preview. + +- **By day, in UTC, kept for a year.** How much of the year the owner sees is + a limit of the tier - seven days for a free lambda, all of it for a premium + one (see [Limits](#limits)) - so a lambda that becomes premium shows what was + counted before it was. A range of the same length before is summed for a + comparison only where the tier shows it as well. +- **Counted in front of the lambda, written behind it.** A request to a + lambda or a preview is made into one visit (`LambdaVisit`: its status, its + time, its bytes, its path, its caller's country and agent) by + `LambdaVisitConcern`, which only makes it and leaves it on the request. + Whatever wants to know about requests to lambdas reads it rather than + measuring again: the overview's figures (`LambdaActivityConcern`), the + request log (`CallerConcern`), and `AnalyticsConcern`, which puts it on a + lock-free queue where the owner counts visits. Each leaves out what is not + its own - the figures and the analytics previews. `AnalyticsJob` empties the queue every + `LAMBDA_ANALYTICS_INTERVAL_SECONDS` (five minutes), sums it into a row + per lambda, day and value, and writes those in one transaction. Nothing is + written while a visitor waits; visits not yet written are lost when the + server stops. The queue holds 250,000 visits; what does not fit is dropped + and logged. +- **What a visit is counted as.** A *hit* is every request answered; a *view* + is somebody opening a page - a browser's navigation (`Sec-Fetch-Dest: + document`, a revalidation included) or, from a program that does not send + that header, a successful `GET` answered with HTML. A path is counted where + it was answered without an error, at most 200 a day, and the rest under + `(other)`, so a scanner cannot fill a day. A referrer is kept as its host, + without `www.`, and only where it is another site than the one visited (100 + a day). Country, device, browser and system come from the address and the + `User-Agent` the request line already has, and from the client hints + (`Sec-CH-UA`, `-Mobile`, `-Platform`) where a Chromium browser sends them - + its agent no longer says which phone or system it is on. Hints are only + believed from an agent that is Chromium's, and only where they name a + browser or system that is known, so nobody writes words of their own into + an owner's figures (`VisitorClassifier`). +- **Crawlers, hardened.** A program is told from a person by + [crawler-user-agents](https://github.com/monperrus/crawler-user-agents) + (MIT, `Resources/Crawlers`), a maintained list of about 1,500 crawlers with a + kind each - search engine, AI crawler, link preview, monitoring, scanner, + HTTP library and so on - and counted under that kind, never as a browser + or a visitor. Google, Bing, OpenAI, Apple and Perplexity publish the + addresses their crawlers come from; a visit is only counted as theirs when + the connection came from one of them, and as an unconfirmed claim otherwise + - anybody can send `Googlebot`. The address checked is the connection's, + never a forwarding header's, so an installation behind a proxy confirms + none. `CrawlerJob` fetches the list and the addresses daily into + `/data/crawlers` (`LAMBDA_ANALYTICS_CRAWLER_LISTS`, + `LAMBDA_ANALYTICS_CRAWLER_REFRESH_HOURS`), the list of the release + standing in until it has. Matching is linear in the length of the agent + whatever it says: the list's plain text is looked for in one pass + (`SearchValues`), its few expressions run without backtracking, and only + 500 characters of an agent are read. +- **Visitors without keeping anybody.** A visitor is their address and their + agent, hashed with a key made at random per day and lambda, held in memory + for today and yesterday only (`DailyVisitors`), and only a person is one. + So a visitor counts once a day; a restart counts somebody again who comes back the same day, and an + installation that keeps no addresses (`LAMBDA_LOG_CLIENT_ADDRESS=false`) + counts no visitors. No cookie, no script in the lambda. +- **A database of its own.** `analytics.db` beside `lambda.db`, a star: + `lambdas`, and a table of facts per statistic - `days` (the totals), + `pages`, `referrers`, `countries`, `devices`, `browsers`, `systems`, + `crawlers` - keyed by lambda, year, month and day (and value), so a + lambda's days are a range of the key. Migrated by Evolve from + `Services/Analytics/Migrations`. Only the switch is in `lambda.db` + (`lambdas.analytics_since`), read with everything else that serves a + request. Deleting a lambda deletes its rows. + ### Taking a lambda away The code belongs to whoever made the lambda, so `GET …/export` (the editor's @@ -1110,6 +1188,14 @@ its own (`IDatabaseVault`, `ISecretVault`, `IDomainRegistry`, `ILogBook`, keeps the commits and their files below `/data/git/{lambda}`, and `ProjectTree` says what a lambda is as a tree and back. The door is `Api/Git/GitRoutes.cs`, beside the pages below `/editor/` and `/source/`. +- **Analytics** (`Services/Analytics`) - the visits counted for an owner who + switched analytics on (see [Analytics](#analytics)). `AnalyticsService` is + what the API and MCP call and what writes a batch down, `AnalyticsQueue` + holds the visits until then, `VisitTally` sums them into rows, + `VisitorClassifier` says what a visitor was - with `CrawlerList` and + `CrawlerRanges`, kept current by `CrawlerJob` - `DailyVisitors` who, and + `AnalyticsStore` keeps `analytics.db` - `IAnalyticsStore` for Meta, which + forgets a deleted lambda's rows. - **Building** (`Services/Building`) - the jobs of the build agent: builds asked for on `/build` and changes asked for in the editor's Change section. `BuildService` decides what may be asked for and in which order that is @@ -1231,6 +1317,9 @@ server's configuration and are set in the administration panel while it runs | `LAMBDA_EXECUTION_TIMEOUT_SECONDS` | `15` | before an invocation is aborted | | `LAMBDA_TELEMETRY_INTERVAL_SECONDS` | `30` | how often a reading is taken | | `LAMBDA_TELEMETRY_SAMPLES` | `2880` | how many readings are kept | +| `LAMBDA_ANALYTICS_INTERVAL_SECONDS` | `300` | how often the visits counted for the owners' analytics are written down | +| `LAMBDA_ANALYTICS_CRAWLER_LISTS` | `true` | fetch the list of crawlers and the addresses Google, Bing, OpenAI, Apple and Perplexity crawl from | +| `LAMBDA_ANALYTICS_CRAWLER_REFRESH_HOURS` | `24` | how often they are fetched again | | `LAMBDA_ADMIN_TOKEN` | - | enables the panel, and closes the figures | | `LAMBDA_LOG_HISTORY` | `4000` | log lines the panel can read back, to 10^6 | | `LAMBDA_LOG_MEMORY_MB` | derived | what their text may cost; whichever runs out first | @@ -1573,7 +1662,8 @@ The tools are the shape of the job: `create_lambda`, `update_lambda`, `write_cod data tools `upload_file`, `list_files`, `delete_file`, `enable_data`, `read_database`, `set_secret`, `list_secrets` and `delete_secret`, `showcase` for listing a lambda on the public showcase and `open_source` for publishing its code - -only when its owner asks for either - and +only when its owner asks for either - `analytics` for reading who visits a +lambda, and switching that on when the user asks, and `list_demos` for reading something that already works. `platform_guide` is the one to call first - it opens with how versions, features and data live, then says what a snippet has to return, what is imported, what is refused, and the handful of things @@ -1661,6 +1751,14 @@ it was not wanted, so the next agent does not put it back. A change does not add one to a lambda that has none. The instructions say so in a line, `platform_guide` under `backlink`, and the build agent's brief the same. +`analytics` answers what the owner sees in the editor's Analytics section - +the totals, the days and each breakdown - over `days`, at most what the tier +shows; while it is off it says so and that the owner switches it on. +`enable: true` switches it on, only when the user asks; there is no way for an +agent to switch it off, since that deletes what was counted. `platform_guide` +says under `analytics` that the platform counts this in front of the lambda, +so an agent does not build a counter of its own. + `open_source` publishes a lambda's code, changes its license or takes it down; with only the editor key it says how things are and changes nothing. It is not part of building, and its description says so: only when the user asks, and @@ -1772,6 +1870,7 @@ no tier in a block of their own below it. | Data | 512 MB | 4 GB | what a lambda keeps: its database and its workspace, together | | Versions kept | 50 | 50 | older ones are removed, never the one online | | Features open | 10 | 10 | each holds a copy of the workspace and database | +| Analytics shown | 7 days | 365 days | how far back the owner sees what was counted of their visitors; a year is kept of every lambda, so at most 366 | A free lambda also goes offline after 720 hours without visits or edits and is removed after 2160; a premium one stays online and is kept. Three limits are @@ -1983,6 +2082,11 @@ files in `Data/Migrations`. To change the schema, add the next `V__.sql` next to them - they are embedded resources, and the existing ones are never edited. +The analytics are kept apart, in `analytics.db`, migrated the same way from +`Services/Analytics/Migrations` and spoken to with plain commands. Back it up +with the data volume, or leave it out: nothing else depends on it, and an +installation without it starts counting again from nothing. + [genhttp]: https://genhttp.org/ [evolve]: https://evolve-db.netlify.app/ [efcore]: https://learn.microsoft.com/ef/core/ diff --git a/src/Frontend/src/api.ts b/src/Frontend/src/api.ts index d53024d..9540e0b 100644 --- a/src/Frontend/src/api.ts +++ b/src/Frontend/src/api.ts @@ -81,6 +81,67 @@ export interface Activation { seconds: number; } +/** What was counted of a lambda's visits over some days, summed. */ +export interface AnalyticsTotals { + /** Every request answered - pages, the files they load, calls of an API. */ + hits: number; + /** The requests that were somebody opening a page. */ + views: number; + /** Different visitors, each counted once a day. */ + visitors: number; + failed: number; + rejected: number; + bytes: number; + averageMillis: number; +} + +/** One day of a lambda's visits, in UTC. */ +export interface AnalyticsDay { + /** yyyy-mm-dd */ + date: string; + totals: AnalyticsTotals; +} + +/** One value of what visits are counted by - a path, a host, a country code, a name. */ +export interface AnalyticsValue { + value: string; + hits: number; + views: number; +} + +/** How a lambda's visitors reach it, as its owner is shown. */ +export interface LambdaAnalytics { + enabled: boolean; + since: string | null; + tier: string; + /** How many days the answer covers, today included. */ + days: number; + /** The most days the tier shows. */ + window: number; + /** How many days are kept of every lambda. */ + kept: number; + from: string; + to: string; + /** The first day anything was counted on - before `from` where more is kept than shown. */ + earliest: string | null; + totals: AnalyticsTotals | null; + /** The same number of days before, where the tier shows those too. */ + previous: AnalyticsTotals | null; + daily: AnalyticsDay[] | null; + pages: AnalyticsValue[] | null; + referrers: AnalyticsValue[] | null; + countries: AnalyticsValue[] | null; + devices: AnalyticsValue[] | null; + browsers: AnalyticsValue[] | null; + systems: AnalyticsValue[] | null; + crawlers: AnalyticsValue[] | null; + /** When visits were last written down. */ + updated: string | null; + intervalSeconds: number; + /** The operators whose crawlers are confirmed by their published addresses; in `crawlers` their names stand for confirmed visits. */ + confirmedCrawlers: string[]; +} + /** One interval of a lambda's traffic. */ export interface TrafficPoint { at: string; @@ -1045,6 +1106,8 @@ export interface TierLimits { versions: number; /** How many features it may have open at once. */ features: number; + /** How many days of its analytics its owner is shown, of the year that is kept. */ + analyticsDays: number; } /** Every limit of the product, which the panel edits. */ @@ -1313,6 +1376,10 @@ export const api = { summary: (privateKey: string) => request(`/lambdas/${privateKey}/summary`), traffic: (privateKey: string) => request(`/lambdas/${privateKey}/traffic`), + analytics: (privateKey: string, days?: number) => + request(`/lambdas/${privateKey}/analytics${days ? `?days=${days}` : ''}`), + enableAnalytics: (privateKey: string) => request(`/lambdas/${privateKey}/analytics`, { method: 'PUT' }), + disableAnalytics: (privateKey: string) => request(`/lambdas/${privateKey}/analytics`, { method: 'DELETE' }), /** The lambda's own log. No cursor answers with the tail. */ lambdaLogs: (privateKey: string, options: { since?: number; level?: string; limit?: number } = {}) => { diff --git a/src/Frontend/src/console/LimitsSection.tsx b/src/Frontend/src/console/LimitsSection.tsx index 3e290c8..63e1ad9 100644 --- a/src/Frontend/src/console/LimitsSection.tsx +++ b/src/Frontend/src/console/LimitsSection.tsx @@ -29,6 +29,8 @@ const TIER: (Limit & { key: keyof TierLimits })[] = [ note: 'Older ones are removed when a new one is saved, never the one online.' }, { key: 'features', title: 'Features open at once', unit: '', note: 'Each holds a copy of the files, the workspace and the database.' }, + { key: 'analyticsDays', title: 'Analytics shown', unit: 'days', + note: 'How far back the owner sees what was counted of their visitors. A year is kept of every lambda, so at most 366; moved to a tier that shows more, a lambda shows what was counted before.' }, ]; /** Only the free tier's: a premium lambda is kept online and kept. */ diff --git a/src/Frontend/src/control/AnalyticsTab.tsx b/src/Frontend/src/control/AnalyticsTab.tsx new file mode 100644 index 0000000..ff76a0d --- /dev/null +++ b/src/Frontend/src/control/AnalyticsTab.tsx @@ -0,0 +1,819 @@ +import { useEffect, useMemo, useRef, useState, type ReactNode } from 'react'; + +import { absoluteAddress } from '../address'; +import { ApiError, api, isDemo, type AnalyticsTotals, type AnalyticsValue, type LambdaAnalytics } from '../api'; +import { CONTACT_MAIL } from '../contact'; +import { Dialog } from '../components/Dialog'; +import { IconArrowDown, IconArrowUp, IconCheck, IconGlobe, IconLock, IconMail, IconSpinner } from '../components/Icons'; +import { useToast } from '../components/Toast'; +import { useEditorT, useLanguage } from '../i18n'; +import { tagOf } from '../i18n/languages'; +import type { Control } from './context'; +import { count, percent } from './format'; +import { Section, Sparkline, pill } from './ui'; + +/** The time ranges offered; those longer than the tier shows are offered locked. */ +const RANGES = [7, 30, 90, 365]; + +type Measure = 'visitors' | 'views' | 'hits'; + +/* + * The accent of the editor for the one measure the chart shows, and for the + * devices a fixed categorical order - validated as a set against both + * surfaces (light needs its labels, which the legend gives). "Other" is the + * neutral grey, since it is not an identity of its own. + */ +const ACCENT: [string, string] = ['#1a73e8', '#4285f4']; + +const DEVICES: Record = { + desktop: ['#1a73e8', '#4285f4'], + mobile: ['#eb6834', '#d95926'], + tablet: ['#1baf7a', '#199e70'], + bot: ['#eda100', '#c98500'], + other: ['#9aa0a6', '#5f6368'], +}; + +const DEVICE_ORDER = ['desktop', 'mobile', 'tablet', 'bot', 'other']; + +/** + * Who visits the lambda: how many people come each day, which pages they + * open, which sites send them, from which countries and on what. + * + * Off until the owner switches it on - then counted by the platform, in + * front of the lambda, by day, and written down every few minutes. The same + * in both views and in the words of the simple one: an owner who never reads + * the code wants to know who comes as much as one who does. + * + * The figures at the top pick what the chart under them shows, so the page + * answers one question at a time - how many came - before the cards below + * answer the others: to what, from where, on what. + */ +export function AnalyticsTab({ control }: { control: Control }) { + const { privateKey, lambda } = control; + + const t = useEditorT(); + const said = t.analytics; + const title = t.frame.sections.analytics; + const tag = tagOf(useLanguage()); + const toast = useToast(); + + // a month to begin with - a week where the tier shows no more, which the + // server says - rather than all of a year, of which the last weeks are + // what an owner opens the page for + const [days, setDays] = useState(30); + const [state, setState] = useState(null); + const [failure, setFailure] = useState(null); + const [measure, setMeasure] = useState('visitors'); + const [switching, setSwitching] = useState(false); + const [confirming, setConfirming] = useState(false); + + /* + * Read when opened and when the range changes, and again when the page is + * looked at after a while away - not on a timer: the figures move every few + * minutes, not every few seconds. + */ + useEffect(() => { + let alive = true; + + const load = () => + api + .analytics(privateKey, days ?? undefined) + .then((found) => { + if (alive) { + setState(found); + setFailure(null); + } + }) + .catch((error) => alive && setFailure(error instanceof ApiError ? error.message : said.loadFailed)); + + load(); + + const back = () => document.visibilityState === 'visible' && load(); + + document.addEventListener('visibilitychange', back); + + return () => { + alive = false; + document.removeEventListener('visibilitychange', back); + }; + }, [privateKey, days, said]); + + async function switchOn() { + setSwitching(true); + + try { + setState(await api.enableAnalytics(privateKey)); + toast(said.switchedOn, 'success'); + } catch (error) { + toast(error instanceof ApiError ? error.message : said.switchOnFailed, 'error'); + } finally { + setSwitching(false); + } + } + + async function switchOff() { + setSwitching(true); + + try { + await api.disableAnalytics(privateKey); + setConfirming(false); + setState(await api.analytics(privateKey)); + toast(said.switchedOff, 'success'); + } catch (error) { + toast(error instanceof ApiError ? error.message : said.switchOffFailed, 'error'); + } finally { + setSwitching(false); + } + } + + if (!state) { + return ( +
+ {failure ? ( +

{failure}

+ ) : ( +
{said.loading}
+ )} +
+ ); + } + + if (!state.enabled || !state.totals || !state.daily) { + return ( +
+ +
+ ); + } + + const dark = control.theme === 'dark'; + const date = (day: string, options: Intl.DateTimeFormatOptions = { day: 'numeric', month: 'short' }) => + new Date(`${day}T00:00:00Z`).toLocaleDateString(tag, { ...options, timeZone: 'UTC' }); + + // pages opened where there are any; an app that only answers calls is + // described by what it was asked for + const by: 'views' | 'hits' = state.totals.views > 0 ? 'views' : 'hits'; + const hidden = state.earliest != null && state.earliest < state.from; + + const ranges = ( +
+ {RANGES.map((range) => { + const locked = range > state.window; + const active = range === state.days || (range === 365 && state.days >= 365); + + return ( + + ); + })} +
+ ); + + return ( +
+ {hidden && lambda.tier !== 'Premium' && ( + + )} + + + + + +
+ + p[by] > 0)} + by={by} + mono + name={(value) => (value === '(other)' ? said.other : value)} + /> + + + + r[by] > 0)} + by={by} + name={(value) => (value === '(other)' ? said.other : value)} + icon={} + rest={by === 'views' ? { label: said.direct, value: Math.max(0, state.totals.views - sum(state.referrers, 'views')) } : undefined} + /> + + + + c[by] > 0)} by={by} tag={tag} /> + + + + + + + + + + {(state.crawlers ?? []).length === 0 ? ( +

{said.noCrawlers}

+ ) : ( + } /> + )} + +
+ +
+ + {said.footer(Math.max(1, Math.round(state.intervalSeconds / 60)))}{' '} + {state.updated && said.updated(new Date(state.updated).toLocaleTimeString(tag, { hour: '2-digit', minute: '2-digit' }))}{' '} + {state.since && said.since(new Date(state.since).toLocaleDateString(tag, { day: 'numeric', month: 'long', year: 'numeric' }))} + + +
+ + setConfirming(false)} + footer={ + <> + + + + } + > +

{said.switchOffText}

+ +
+ ); +} + +/** + * What analytics are, before they are on: what is counted, and what is not - + * the second as important as the first, since it is about the owner's + * visitors. + */ +function Off({ window, demo, busy, onSwitch }: { window: number; demo: boolean; busy: boolean; onSwitch: () => void }) { + const said = useEditorT().analytics; + + return ( +
+ + +
+

{said.offTitle}

+

{said.offText}

+ +
    + {[said.offPrivate, said.offAddresses, said.offKept(window)].map((line) => ( +
  • + + {line} +
    • + ))} +
+ + {demo ? ( +

{said.demo}

+ ) : ( + + )} +
+
+ ); +} + +/** A quiet picture of what is to come, behind the words: a week of columns. */ +function Preview() { + const heights = [34, 52, 41, 68, 59, 83, 74]; + + return ( + + ); +} + +/** + * For a free app that counted more than its tier shows: where the rest is. + * The address of the app in the mail, never its editor link. + */ +function Nudge({ since, address }: { since: string; address: string }) { + const said = useEditorT().analytics; + + const mail = `mailto:${CONTACT_MAIL}?subject=${encodeURIComponent(said.premiumSubject)}&body=${encodeURIComponent(said.premiumBody(address))}`; + + return ( +
+ {said.moreKept(since)} + + + {said.premiumAsk} + +
+ ); +} + +/** + * The three figures, each a tab for the chart below: its total over the + * range, how it changed against the days before where those are shown, and + * its days in small. + */ +function Figures({ state, measure, onPick, dark }: { state: LambdaAnalytics; measure: Measure; onPick: (m: Measure) => void; dark: boolean }) { + const said = useEditorT().analytics; + + const figures: { id: Measure; label: string; hint: string }[] = [ + { id: 'visitors', label: said.visitors, hint: said.visitorsHint }, + { id: 'views', label: said.views, hint: said.viewsHint }, + { id: 'hits', label: said.hits, hint: said.hitsHint }, + ]; + + return ( +
+ {figures.map((figure) => { + const active = figure.id === measure; + const value = state.totals![figure.id]; + const before = state.previous?.[figure.id]; + + return ( + + ); + })} +
+ ); +} + +/** + * How a figure changed against the same number of days before - with an + * arrow and a word, never colour alone. + */ +function Change({ now, before, days }: { now: number; before: number | undefined; days: number }) { + const said = useEditorT().analytics; + + if (before === undefined || before === 0) { + return
{said.inDays(days)}
; + } + + const change = (now - before) / before; + const shown = `${Math.abs(Math.round(change * 100))}%`; + + if (Math.round(change * 100) === 0) { + return
{said.noChange} · {said.versus(days)}
; + } + + const up = change > 0; + + return ( +
+ + {up ? : } + {shown} + + {said.versus(days)} +
+ ); +} + +/** + * The measure picked, a column a day - or a week, over a year, where a + * column a day would be thinner than the gap between two. Columns rather + * than a line: each counts what happened in its day. + */ +function Columns({ state, measure, dark, date }: { + state: LambdaAnalytics; + measure: Measure; + dark: boolean; + date: (day: string, options?: Intl.DateTimeFormatOptions) => string; +}) { + const said = useEditorT().analytics; + const [hover, setHover] = useState(null); + const [table, setTable] = useState(false); + const [frame, width] = useWidth(720); + + const weekly = state.days > 92; + + const points = useMemo(() => { + const days = state.daily!; + + if (!weekly) { + return days.map((d) => ({ start: d.date, value: d.totals[measure] })); + } + + // whole weeks back from today, so the newest column is never a stub + const weeks: { start: string; value: number }[] = []; + + for (let end = days.length; end > 0; end -= 7) { + const slice = days.slice(Math.max(0, end - 7), end); + weeks.unshift({ start: slice[0].date, value: slice.reduce((total, d) => total + d.totals[measure], 0) }); + } + + return weeks; + }, [state.daily, measure, weekly]); + + const total = points.reduce((all, p) => all + p.value, 0); + const max = Math.max(1, ...points.map((p) => p.value)); + const nice = niceCeiling(max); + + // drawn at the width it is given, so its labels stay the size of text on + // a phone rather than shrinking with a picture made for a desk + const height = width < 480 ? 180 : 260; + const pad = { top: 12, right: 8, bottom: 26, left: 48 }; + const plotW = width - pad.left - pad.right; + const plotH = height - pad.top - pad.bottom; + const slot = plotW / points.length; + const gap = Math.min(4, Math.max(1, slot * 0.25)); + const barW = Math.max(1, slot - gap); + const y = (v: number) => plotH - (v / nice) * plotH; + + const ink = dark ? '#9aa0a6' : '#5f6368'; + const grid = dark ? '#3c4043' : '#e8eaed'; + const colour = ACCENT[dark ? 1 : 0]; + + const heading = (weekly ? said.perWeek : said.perDay)[measure]; + const label = (start: string) => (weekly ? `${said.week} ${date(start)}` : date(start, { weekday: 'short', day: 'numeric', month: 'short' })); + + // the first, the last, and a few between - never one under every column + const every = Math.max(1, Math.ceil(points.length / (width < 480 ? 3 : 6))); + + return ( +
+
+
{heading}
+ +
+ + {total === 0 ? ( +

+ {state.earliest == null ? said.nobodyYet : said.nobody} +

+ ) : table ? ( +
+ + + + + + + + + {[...points].reverse().map((p) => ( + + + + + ))} + +
{weekly ? said.week : said.day}{heading}
{label(p.start)}{count(p.value)}
+
+ ) : ( +
+ {hover !== null && ( +
+
{label(points[hover].start)}
+
{count(points[hover].value)}
+
+ )} + + setHover(null)} + onMouseMove={(event) => { + const box = event.currentTarget.getBoundingClientRect(); + const at = ((event.clientX - box.left) / box.width) * width - pad.left; + setHover(Math.min(points.length - 1, Math.max(0, Math.floor(at / slot)))); + }} + > + + {[0, nice / 2, nice].map((tick) => ( + + + {count(tick)} + + ))} + + {points.map((p, i) => { + const h = p.value > 0 ? Math.max(2, plotH - y(p.value)) : 0; + const x = i * slot + gap / 2; + + return ( + + {/* the whole slot answers the pointer, not only the column */} + + {h > 0 && ( + + )} + + ); + })} + + {points.map((p, i) => + i % every === 0 || i === points.length - 1 ? ( + i !== points.length - 1 && points.length - 1 - i < every / 2 ? null : ( + + {date(p.start)} + + ) + ) : null)} + + +
+ )} +
+ ); +} + +/** The width an element is laid out at, measured as it changes. */ +function useWidth(initial: number) { + const ref = useRef(null); + const [width, setWidth] = useState(initial); + + useEffect(() => { + const element = ref.current; + + if (!element || typeof ResizeObserver === 'undefined') { + return; + } + + const observer = new ResizeObserver(([entry]) => setWidth(Math.max(280, Math.round(entry.contentRect.width)))); + + observer.observe(element); + + return () => observer.disconnect(); + }, []); + + return [ref, width] as const; +} + +/** A card of the grid under the chart. */ +function Card({ title, hint, children }: { title: string; hint?: string; children: ReactNode }) { + return ( +
+

{title}

+ {hint &&

{hint}

} +
{children}
+
+ ); +} + +/** + * A list of values, the largest first, each over a bar as long as its share + * of the largest - so the list reads as a chart at a glance and as numbers + * when read. Eight at first, the rest a click away. + */ +function Ranked({ values, by, name, mono = false, icon, rest }: { + values: AnalyticsValue[]; + by: 'views' | 'hits'; + name: (value: string) => ReactNode; + mono?: boolean; + icon?: ReactNode; + /** A row for what the values leave over - the visitors no site sent. */ + rest?: { label: string; value: number }; +}) { + const said = useEditorT().analytics; + const [all, setAll] = useState(false); + + const rows = [ + ...values.map((v) => ({ key: v.value, label: name(v.value), value: v[by], muted: false, icon })), + ...(rest && rest.value > 0 ? [{ key: '\u0000rest', label: rest.label, value: rest.value, muted: true, icon: undefined }] : []), + ].sort((a, b) => b.value - a.value); + + if (rows.length === 0) { + return

{said.none}

; + } + + const largest = Math.max(1, ...rows.map((r) => r.value)); + const whole = rows.reduce((total, r) => total + r.value, 0); + const shown = all ? rows : rows.slice(0, 8); + + return ( + <> +
    + {shown.map((row) => ( +
  • +
    • + ))} +
+ + {rows.length > 8 && ( + + )} + + ); +} + +/** The countries by their names in the reader's language, their codes beside them. */ +function Countries({ values, by, tag }: { values: AnalyticsValue[]; by: 'views' | 'hits'; tag: string }) { + const said = useEditorT().analytics; + + const names = useMemo(() => { + try { + return new Intl.DisplayNames([tag], { type: 'region' }); + } catch { + return null; + } + }, [tag]); + + const named = (code: string) => { + if (code === '') { + return said.unknownCountry; + } + + try { + return names?.of(code) ?? code; + } catch { + return code; + } + }; + + return ( + ( + <> + {code || '··'} + {named(code)} + + )} + /> + ); +} + +/** + * The devices as one bar of their shares, part of a whole, with a legend + * that names each and says its share - identity never rests on colour. + */ +function Devices({ values, by, dark }: { values: AnalyticsValue[]; by: 'views' | 'hits'; dark: boolean }) { + const said = useEditorT().analytics; + + const parts = DEVICE_ORDER.map((id) => ({ id, value: values.find((v) => v.value === id)?.[by] ?? 0 })).filter((p) => p.value > 0); + const whole = parts.reduce((total, p) => total + p.value, 0); + + if (whole === 0) { + return

{said.none}

; + } + + const word = (id: string) => said.device[id as keyof typeof said.device] ?? id; + + return ( +
+
`${word(p.id)} ${percent(p.value, whole)}`).join(', ')}> + {parts.map((p) => ( + + ))} +
+ +
    + {parts.map((p) => ( +
  • +
    • + ))} +
+
+ ); +} + +/** + * A crawler by its operator's name, marked as confirmed, where it came from + * the addresses the operator publishes - or by the kind of program it is. + */ +function Crawler({ value, confirmed }: { value: string; confirmed: string[] }) { + const said = useEditorT().analytics; + const kinds: Record = said.crawlerKinds; + + if (confirmed.includes(value)) { + return ( + + {value} + + {said.confirmed(value)} + + ); + } + + return <>{kinds[value] ?? value}; +} + +/** The browsers and the operating systems of people, one card with a pill for each. */ +function Agents({ browsers, systems, by }: { browsers: AnalyticsValue[]; systems: AnalyticsValue[]; by: 'views' | 'hits' }) { + const said = useEditorT().analytics; + const [showing, setShowing] = useState<'browsers' | 'systems'>('browsers'); + + const values = (showing === 'browsers' ? browsers : systems).filter((v) => v[by] > 0); + + return ( +
+
+ {(['browsers', 'systems'] as const).map((id) => ( + + ))} +
+
+ (value === 'other' ? said.device.other : value)} /> +
+
+ ); +} + +/** A column with rounded ends at the top, square on the baseline. */ +function column(x: number, y: number, w: number, h: number, r: number) { + const radius = Math.min(r, h); + + return `M ${x},${y + h} L ${x},${y + radius} Q ${x},${y} ${x + radius},${y} L ${x + w - radius},${y} Q ${x + w},${y} ${x + w},${y + radius} L ${x + w},${y + h} Z`; +} + +/** A round number at or above the largest value, so the axis reads 0, 50, 100 rather than 0, 43.5, 87. */ +function niceCeiling(value: number) { + // small counts keep whole numbers at the half way line + if (value <= 10) { + return Math.max(2, Math.ceil(value / 2) * 2); + } + + const power = 10 ** Math.floor(Math.log10(value)); + + for (const step of [1, 2, 2.5, 5, 10]) { + if (step * power >= value) { + return step * power; + } + } + + return 10 * power; +} + +const sum = (values: AnalyticsValue[] | null, by: keyof Pick) => + (values ?? []).reduce((total, v) => total + v[by], 0); diff --git a/src/Frontend/src/locales/de/editor/analytics.tsx b/src/Frontend/src/locales/de/editor/analytics.tsx new file mode 100644 index 0000000..d6bfe27 --- /dev/null +++ b/src/Frontend/src/locales/de/editor/analytics.tsx @@ -0,0 +1,117 @@ +import type { EditorMessages } from '../../en/editor'; + +export const analytics: EditorMessages['analytics'] = { + hint: 'Von der Plattform für Sie gezählt: keine Cookies, kein Code in Ihrer App und nichts, was Ihre Besucher akzeptieren müssten.', + loading: 'Besuche werden gelesen …', + loadFailed: 'Die Besuche konnten nicht gelesen werden.', + + offTitle: 'Sehen Sie, wer Ihre App besucht', + offText: + 'Wie viele Personen jeden Tag kommen, welche Seiten sie öffnen, welche Websites sie schicken, aus welchen Ländern und auf welchen Geräten – tageweise gezählt, ab dem Moment, in dem Sie es einschalten.', + offPrivate: 'Keine Cookies und kein Banner, das Ihre Besucher akzeptieren müssen', + offAddresses: 'Es werden keine Adressen gespeichert – nur, wie viele Besuche es pro Tag gab', + offKept: (days) => + days >= 365 + ? 'Ein Jahr lang gespeichert, und Sie sehen alles davon' + : `Ein Jahr lang gespeichert – angezeigt werden die letzten ${days} Tage, mit Premium alles`, + switchOn: 'Analyse einschalten', + switchedOn: 'Die Analyse ist eingeschaltet. Die ersten Besuche erscheinen innerhalb weniger Minuten.', + switchOnFailed: 'Die Analyse konnte nicht eingeschaltet werden.', + demo: 'Die Besucher einer Demo werden nicht gezählt. Eine eigene Kopie kann ihre Besucher zählen.', + + range: 'Zeitraum', + days: (days) => (days >= 365 ? '12 Monate' : `${days} Tage`), + premiumRange: 'Premium zeigt bis zu einem Jahr', + + visitors: 'Besucher', + visitorsHint: 'Verschiedene Personen, jede einmal pro Tag gezählt. Suchmaschinen und andere Programme sind keine Besucher.', + views: 'Seitenaufrufe', + viewsHint: 'Jedes Mal, wenn jemand eine Seite geöffnet hat.', + hits: 'Requests', + hitsHint: 'Alles, was bei Ihrer App angefragt wurde: ihre Seiten sowie die Bilder und Skripte, die sie laden.', + versus: (days) => `gegenüber den ${days} Tagen davor`, + noChange: 'unverändert', + inDays: (days) => (days >= 365 ? 'in den letzten 12 Monaten' : `in den letzten ${days} Tagen`), + + perDay: { + visitors: 'Besucher pro Tag', + views: 'Seitenaufrufe pro Tag', + hits: 'Requests pro Tag', + }, + perWeek: { + visitors: 'Besucher pro Woche', + views: 'Seitenaufrufe pro Woche', + hits: 'Requests pro Woche', + }, + showValues: 'Werte anzeigen', + showChart: 'Diagramm anzeigen', + day: 'Tag', + week: 'Woche ab', + + nobody: 'In diesen Tagen hat niemand die App besucht.', + nobodyYet: 'Seit dem Einschalten der Analyse hat niemand die App besucht. Die ersten Besuche erscheinen innerhalb weniger Minuten.', + + pages: 'Seiten', + pagesHint: 'Die Seiten, die geöffnet wurden – die am häufigsten geöffneten zuerst.', + requested: 'Am häufigsten angefragt', + requestedHint: 'Alles, was bei Ihrer App angefragt wurde, nach Adresse – sie hat keine Seiten, die Personen öffnen.', + sources: 'Woher die Besucher kommen', + sourcesHint: 'Die Websites, die Personen hierher geschickt haben, nach der Seite, auf der sie waren. Bei Suchmaschinen steht nur, dass sie es waren, nicht wonach gesucht wurde.', + direct: 'Direkt eingegeben, Lesezeichen oder App', + countries: 'Länder', + countriesHint: 'Wo die Internetadressen der Besucher registriert sind – meist, aber nicht immer, dort, wo sie sich befinden.', + unknownCountry: 'Unbekannt', + devices: 'Geräte', + device: { + desktop: 'Computer', + mobile: 'Smartphone', + tablet: 'Tablet', + bot: 'Bots', + other: 'Andere', + }, + browsers: 'Browser', + systems: 'Betriebssysteme', + crawlers: 'Suchmaschinen, KI und andere Programme', + crawlersHint: 'Programme, die vorbeigeschaut haben, nach Art geordnet. Google, Bing, OpenAI, Apple und Perplexity werden nur genannt, wenn der Besuch von einer Adresse kam, die sie veröffentlichen – jeder kann behaupten, sie zu sein.', + noCrawlers: 'In diesen Tagen hat keine Suchmaschine, keine KI und kein anderes Programm vorbeigeschaut.', + confirmed: (name) => `Kam von einer Adresse, die ${name} für seine Crawler veröffentlicht`, + crawlerKinds: { + 'search-engine': 'Suchmaschinen', + 'ai-crawler': 'KI-Crawler', + 'social-preview': 'Link-Vorschauen in Chats und sozialen Netzwerken', + advertising: 'Werbung', + seo: 'SEO-Tools', + monitoring: 'Überwachung', + 'feed-reader': 'Feed-Reader', + archiver: 'Web-Archive', + academic: 'Forschung', + scanner: 'Sicherheits-Scanner', + 'browser-automation': 'Automatisierte Browser', + 'http-library': 'Skripte und Apps', + other: 'Andere Programme', + unconfirmed: 'Gab sich als bekannter Crawler aus, nicht bestätigt', + }, + other: 'Andere', + showAll: (count) => `Alle ${count} anzeigen`, + showFewer: 'Weniger anzeigen', + share: (part, whole) => `${part} von ${whole}`, + none: 'Noch nichts.', + + moreKept: (since) => `Gezählt seit ${since} – Premium zeigt alles davon, bis zu einem Jahr.`, + premiumAsk: 'Nach Premium fragen', + premiumSubject: 'Premium für meine App', + premiumBody: (address) => `Guten Tag,\n\nich möchte die gesamte Analyse meiner App sehen: ${address}\n\n`, + + footer: (minutes) => + minutes === 1 ? 'Pro Tag gezählt, in UTC. Wird jede Minute aktualisiert.' : `Pro Tag gezählt, in UTC. Wird alle ${minutes} Minuten aktualisiert.`, + updated: (time) => `Letzte Aktualisierung um ${time}.`, + since: (date) => `Gezählt wird seit ${date}.`, + + switchOff: 'Analyse ausschalten', + switchOffTitle: 'Analyse ausschalten?', + switchOffText: 'Das Zählen endet, und alles, was bisher gezählt wurde, wird gelöscht. Das lässt sich nicht rückgängig machen.', + keep: 'Weiter zählen', + switchOffConfirm: 'Ausschalten und löschen', + switchedOff: 'Die Analyse ist ausgeschaltet, und was gezählt wurde, ist gelöscht.', + switchOffFailed: 'Die Analyse konnte nicht ausgeschaltet werden.', +}; diff --git a/src/Frontend/src/locales/de/editor/frame.tsx b/src/Frontend/src/locales/de/editor/frame.tsx index bd61e84..c2acb13 100644 --- a/src/Frontend/src/locales/de/editor/frame.tsx +++ b/src/Frontend/src/locales/de/editor/frame.tsx @@ -16,6 +16,7 @@ export const frame: EditorMessages['frame'] = { history: 'Verlauf', deployments: 'Deployments', stats: 'Statistik', + analytics: 'Analyse', logs: 'Logs', code: 'Code', }, diff --git a/src/Frontend/src/locales/de/editor/index.ts b/src/Frontend/src/locales/de/editor/index.ts index ba11d8a..8a7ab01 100644 --- a/src/Frontend/src/locales/de/editor/index.ts +++ b/src/Frontend/src/locales/de/editor/index.ts @@ -1,4 +1,5 @@ import type { EditorMessages } from '../../en/editor'; +import { analytics } from './analytics'; import { change } from './change'; import { clone } from './clone'; import { code } from './code'; @@ -33,6 +34,7 @@ export const editor: EditorMessages = { versions, deployments, stats, + analytics, logs, showcase, openSource, diff --git a/src/Frontend/src/locales/de/guide.tsx b/src/Frontend/src/locales/de/guide.tsx index b0f86cc..43f4e84 100644 --- a/src/Frontend/src/locales/de/guide.tsx +++ b/src/Frontend/src/locales/de/guide.tsx @@ -26,6 +26,7 @@ export const guide: Messages['guide'] = { away: 'Alles mitnehmen', git: 'Mit git arbeiten', open: 'Den Code veröffentlichen', + visitors: 'Wer sie besucht', agents: 'Mit einem Agenten arbeiten', }, @@ -105,6 +106,7 @@ export const guide: Messages['guide'] = { ['Versionen', () => <>Was jede Version geändert hat, worum gebeten wurde und der Diff zur vorherigen. Von hier aus deployen oder zurückrollen – oder aus jeder von ihnen einen Entwurf beginnen.], ['Deployments', () => <>Was wann online war und warum es offline ging.], ['Statistik', () => <>Requests, Fehler, Antwortzeiten und die meistgefragten Pfade der letzten Stunde oder des letzten Tages.], + ['Analyse', () => <>Wer die App besucht, sobald Sie sie einschalten: Besucher, Seitenaufrufe und Requests pro Tag, die geöffneten Seiten, die Websites, die Besucher schicken, ihre Länder und Geräte sowie die Suchmaschinen und KI, die vorbeigeschaut haben.], ['Logs', () => <>Requests, Ausgaben und Stacktraces von allem, was schiefging – live.], [ 'Code', @@ -128,7 +130,7 @@ export const guide: Messages['guide'] = { ), editorAside: - 'Traffic und Log liegen im Arbeitsspeicher. Sie sind zum Beobachten da, nicht zum Aufbewahren: Nach einem Neustart des Servers beginnen sie von vorn. Versionen und der Deployment-Verlauf werden gespeichert.', + 'Traffic und Log liegen im Arbeitsspeicher. Sie sind zum Beobachten da, nicht zum Aufbewahren: Nach einem Neustart des Servers beginnen sie von vorn. Versionen, der Deployment-Verlauf und die Analyse werden gespeichert.', why: (k) => ( <> @@ -504,6 +506,27 @@ export const guide: Messages['guide'] = { ), openAside: 'Alles im Code wird öffentlich, auch die früheren Versionen. Ein Schlüssel oder Passwort gehört zu den Schlüsseln und Passwörtern unter Daten, nie in den Code – ob veröffentlicht oder nicht.', + visitors: ((k) => ( + <> + Öffnen Sie im Kontrollzentrum {k.b('Analyse')} und schalten Sie sie ein, um zu sehen, wer Ihre App besucht: wie viele + Personen jeden Tag kommen, welche Seiten sie öffnen, welche Websites sie schicken – eine Suchmaschine, ein Beitrag, ein + Newsletter –, aus welchen Ländern, auf welchen Geräten und in welchen Browsern, und welche Suchmaschinen und + KI-Assistenten ihre Seiten lesen. Wählen Sie oben eine Zahl, um sie Tag für Tag zu sehen. + + )), + visitors2: ((k) => ( + <> + Gezählt wird von der Plattform, vor Ihrer App – Ihre App braucht dafür nichts: keine Cookies, kein Script, kein Banner, + das Ihre Besucher akzeptieren müssten. Gespeichert werden nur Zahlen pro Tag – nie eine Adresse, und von der Seite, von + der jemand kam, nur die Website. Ein Besucher ist eine Person, einmal pro Tag gezählt: Suchmaschinen, KI-Crawler und + andere Programme werden getrennt gezählt, nach Art, und Google, Bing, OpenAI, Apple und Perplexity nur dann mit Namen, + wenn der Besuch von einer Adresse kam, die sie veröffentlichen – jeder kann behaupten, sie zu sein. Ein Jahr wird + aufbewahrt; eine kostenlose App zeigt davon die letzten sieben Tage und {k.b('Premium')} alles, auch was vorher gezählt + wurde. Wer die Analyse ausschaltet, löscht, was gezählt wurde. + + )), + visitorsAside: + 'Die Zahlen werden alle paar Minuten festgehalten, die letzten Besuche erscheinen also etwas später. Ein Entwurf wird nie gezählt: Eine Änderung auszuprobieren ist kein Besuch.', agents: (k) => ( <> diff --git a/src/Frontend/src/locales/de/privacy.tsx b/src/Frontend/src/locales/de/privacy.tsx index 1676472..079af20 100644 --- a/src/Frontend/src/locales/de/privacy.tsx +++ b/src/Frontend/src/locales/de/privacy.tsx @@ -17,6 +17,11 @@ export const privacy: Messages['privacy'] = { 'So finden wir Fehler, so führen wir einen überlasteten Server auf die Ursache zurück, und so gehen wir Missbrauch nach, der uns gemeldet wird. Außerdem nutzt der Server die IP-Adresse, nur im Arbeitsspeicher, um zu begrenzen, wie viele Anfragen und Builds eine Person auslösen kann. Ohne diese Angaben lässt sich eine Anfrage nicht beantworten. Rechtsgrundlage ist unser berechtigtes Interesse daran, den Dienst zu betreiben und sicher zu halten (Art. 6 Abs. 1 lit. f DSGVO).', 'Administratoren können alles davon lesen. Wer ein Lambda besitzt, sieht bei jeder Anfrage an dieses Lambda das Land und den Browser, aber nicht die IP-Adresse.', ], + analyticsTitle: 'Besuche, die für den Inhaber eines Lambdas gezählt werden', + analytics: [ + 'Der Inhaber eines Lambdas kann dafür die Analyse einschalten. Dann zählt der Server die Besuche dieses Lambdas pro Tag: wie viele Requests und Seitenaufrufe es gab, für welche Seiten sie waren, von welcher Website ein Besucher kam (nur deren Name, nicht die Adresse der Seite), das Land der IP-Adresse des Besuchers sowie die Art des Geräts, des Browsers und des Betriebssystems, die der Browser angibt. Nur diese Zahlen werden gespeichert – nicht die IP-Adresse und nichts anderes, das einen Besucher benennt – in einer Datenbank auf dem Server, für ein Jahr. Der Inhaber des Lambdas sieht sie in seinem Editor, und Administratoren können sie lesen.', + 'Um jeden Besucher einmal pro Tag zu zählen, werden die IP-Adresse und der Browser mit einem Schlüssel, der für jeden Tag und jedes Lambda zufällig erzeugt wird, in einen Code umgewandelt, der sich nicht zurückrechnen lässt. Die Codes und die Schlüssel liegen nur im Arbeitsspeicher, werden nie aufgeschrieben und nach einem Tag vergessen. Es wird kein Cookie gesetzt und nichts im Browser des Besuchers gespeichert. Gibt ein Besuch an, vom Crawler von Google, Bing, OpenAI, Apple oder Perplexity zu stammen, wird seine IP-Adresse im Arbeitsspeicher mit den Adressen verglichen, die diese für ihre Crawler veröffentlichen. Um Crawler von Menschen zu unterscheiden, ruft der Server einmal pro Tag eine öffentliche Liste von Crawlern und diese veröffentlichten Adressen ab; dafür wird nichts über Besucher irgendwohin gesendet. Rechtsgrundlage ist das berechtigte Interesse des Inhabers, zu wissen, wie sein Lambda genutzt wird (Art. 6 Abs. 1 lit. f DSGVO).', + ], logsTitle: 'Wie lange das Log aufbewahrt wird', logs: 'Das Log liegt an zwei Stellen: im Arbeitsspeicher des Servers, der bei jedem Neustart geleert wird, und in der Konsolenausgabe des Servers, die bei jedem Update gelöscht wird. Beide haben eine feste Größe. Jede neue Zeile verdrängt also die älteste, und wie lange eine Zeile erhalten bleibt, hängt davon ab, wie viel auf der Seite los ist. Nichts aus dem Log wird archiviert.', @@ -44,14 +49,14 @@ export const privacy: Messages['privacy'] = { lambdasTitle: 'Was ein Lambda tut, entscheidet sein Besitzer', lambdas: - 'Ein Lambda wird von der Person geschrieben, die den Editor-Link hat – nicht von uns. Was es von seinen Besuchern wissen will und was es damit macht, entscheidet diese Person. Diese Erklärung gilt dafür nicht, mit Ausnahme des Anfrage-Logs oben, das der Server für jedes Lambda führt. Die Nutzungsbedingungen verbieten, mit einem Lambda personenbezogene Daten anderer Menschen zu sammeln. Wenn Sie auf eines stoßen, das es trotzdem tut, melden Sie es bitte.', + 'Ein Lambda wird von der Person geschrieben, die den Editor-Link hat – nicht von uns. Was es von seinen Besuchern wissen will und was es damit macht, entscheidet diese Person. Diese Erklärung gilt dafür nicht, mit Ausnahme des Anfrage-Logs oben, das der Server für jedes Lambda führt, und der Besuche, die er für Inhaber zählt, die die Analyse eingeschaltet haben. Die Nutzungsbedingungen verbieten, mit einem Lambda personenbezogene Daten anderer Menschen zu sammeln. Wenn Sie auf eines stoßen, das es trotzdem tut, melden Sie es bitte.', mailTitle: 'Wenn Sie uns schreiben', mail: 'Wenn Sie uns schreiben – um Missbrauch zu melden oder aus einem anderen Grund –, nutzen wir Ihre Adresse und Ihre Nachricht, um Ihnen zu antworten und Ihr Anliegen zu bearbeiten. Sobald wir sie dafür nicht mehr brauchen, löschen wir sie (Art. 6 Abs. 1 lit. f DSGVO).', storageTitle: 'Cookies und Ihr Browser', storage: - 'Es gibt ein einziges Cookie. Es heißt lang, merkt sich die Sprache, die Sie gewählt haben, damit sich Adressen ohne Sprachangabe in dieser Sprache öffnen, und hält ein Jahr. Im Speicher Ihres Browsers liegen außerdem das helle oder dunkle Design, ein paar Einstellungen der Seiten, die Sie nutzen, und bei Administratoren ihr Token. Nichts davon dient dazu, Sie zu verfolgen, und nichts davon geht an andere: Es gibt keine Analyse-Tools, keine Werbung, und nichts wird von fremden Seiten geladen, nicht einmal Schriftarten. Weil all das nur tut, was Sie selbst wollen, ist keine Einwilligung nötig (§ 25 Abs. 2 Nr. 2 TDDDG).', + 'Es gibt ein einziges Cookie. Es heißt lang, merkt sich die Sprache, die Sie gewählt haben, damit sich Adressen ohne Sprachangabe in dieser Sprache öffnen, und hält ein Jahr. Im Speicher Ihres Browsers liegen außerdem das helle oder dunkle Design, ein paar Einstellungen der Seiten, die Sie nutzen, und bei Administratoren ihr Token. Nichts davon dient dazu, Sie zu verfolgen, und nichts davon geht an andere: Es gibt keine Analyse dieser Website, keine Werbung, und nichts wird von fremden Seiten geladen, nicht einmal Schriftarten. Weil all das nur tut, was Sie selbst wollen, ist keine Einwilligung nötig (§ 25 Abs. 2 Nr. 2 TDDDG).', hostingTitle: 'Wo die Daten liegen', hosting: diff --git a/src/Frontend/src/locales/en/editor/analytics.tsx b/src/Frontend/src/locales/en/editor/analytics.tsx new file mode 100644 index 0000000..2ebf492 --- /dev/null +++ b/src/Frontend/src/locales/en/editor/analytics.tsx @@ -0,0 +1,134 @@ +/** + * The analytics of a lambda: who visits it, from where, to which pages - in + * both views, so in the words of the simple view. + */ +export const analytics = { + /** Next to the section's title, on hover. */ + hint: 'Counted for you by the platform: no cookies, no code in your app, and nothing for your visitors to accept.', + loading: 'Reading the visits…', + loadFailed: 'The visits could not be read.', + + /** The page shown while analytics are off. */ + offTitle: 'See who visits your app', + offText: + 'How many people come each day, which pages they open, which sites send them, from which countries and on which devices - counted day by day, from the moment you switch it on.', + offPrivate: 'No cookies, and no banner your visitors have to accept', + offAddresses: 'No addresses are kept - only how many visits there were each day', + offKept: (days: number) => + days >= 365 + ? 'Kept for a year, and all of it shown to you' + : `Kept for a year - the last ${days} days are shown, all of it with Premium`, + switchOn: 'Switch on analytics', + switchedOn: 'Analytics are on. The first visits show up within a few minutes.', + switchOnFailed: 'Analytics could not be switched on.', + demo: 'The visitors of a demo are not counted. A copy of your own can count its visitors.', + + range: 'Time range', + days: (days: number) => (days >= 365 ? '12 months' : `${days} days`), + /** On a range the tier does not show, on hover. */ + premiumRange: 'Premium shows up to a year', + + /** The figures at the top; each one picks what the chart below shows. */ + visitors: 'Visitors', + visitorsHint: 'Different people, each counted once a day. Search engines and other programs are not visitors.', + views: 'Page views', + viewsHint: 'Every time somebody opened a page.', + hits: 'Requests', + hitsHint: 'Everything your app was asked for: its pages, and the pictures and scripts they load.', + /** Under a figure: how it changed against the days before the ones shown. */ + versus: (days: number) => `against the ${days} days before`, + noChange: 'unchanged', + /** Under a figure when there is nothing to compare with. */ + inDays: (days: number) => (days >= 365 ? 'in the last 12 months' : `in the last ${days} days`), + + /** Above the chart, for the figure picked. */ + perDay: { + visitors: 'Visitors per day', + views: 'Page views per day', + hits: 'Requests per day', + }, + perWeek: { + visitors: 'Visitors per week', + views: 'Page views per week', + hits: 'Requests per week', + }, + showValues: 'Show the values', + showChart: 'Show the chart', + day: 'Day', + week: 'Week of', + + nobody: 'Nobody visited in these days.', + nobodyYet: 'Nobody has visited since analytics were switched on. The first visits show up within a few minutes.', + + pages: 'Pages', + pagesHint: 'The pages people opened, the most opened first.', + requested: 'Most requested', + requestedHint: 'Everything your app was asked for, by address - it has no pages people open.', + sources: 'Where visitors come from', + sourcesHint: 'The sites that sent people here, by the page they were on. Search engines only say that it was them, not what was searched.', + direct: 'Typed in, a bookmark or an app', + countries: 'Countries', + countriesHint: 'Where the visitors\' internet addresses are registered - usually, not always, where they are.', + unknownCountry: 'Unknown', + devices: 'Devices', + device: { + desktop: 'Computer', + mobile: 'Phone', + tablet: 'Tablet', + bot: 'Bots', + other: 'Other', + }, + browsers: 'Browsers', + systems: 'Systems', + crawlers: 'Search engines, AI and other programs', + crawlersHint: + 'Programs that came by, by what they are. Google, Bing, OpenAI, Apple and Perplexity are named only when the visit came from an address they publish - anybody can claim to be them.', + noCrawlers: 'No search engine, AI or other program has come by in these days.', + /** On hover, next to the name of a crawler that was confirmed. */ + confirmed: (name: string) => `Came from an address ${name} publishes for its crawlers`, + /** The kinds of programs that are counted without a name of their own. */ + crawlerKinds: { + 'search-engine': 'Search engines', + 'ai-crawler': 'AI crawlers', + 'social-preview': 'Link previews in chats and social networks', + advertising: 'Advertising', + seo: 'SEO tools', + monitoring: 'Monitoring', + 'feed-reader': 'Feed readers', + archiver: 'Web archives', + academic: 'Research', + scanner: 'Security scanners', + 'browser-automation': 'Automated browsers', + 'http-library': 'Scripts and apps', + other: 'Other programs', + /** Programs that called themselves Google, Bing and so on, but did not come from their addresses. */ + unconfirmed: 'Claimed to be a known crawler, not confirmed', + }, + /** What did not fit in a day's list, gathered in one row. */ + other: 'Others', + showAll: (count: number) => `Show all ${count}`, + showFewer: 'Show fewer', + /** Next to a share in a list, on hover. */ + share: (part: string, whole: string) => `${part} of ${whole}`, + none: 'Nothing yet.', + + /** For a free app that counted more than it shows. */ + moreKept: (since: string) => `Counted since ${since} - Premium shows all of it, up to a year.`, + premiumAsk: 'Ask about Premium', + premiumSubject: 'Premium for my app', + premiumBody: (address: string) => `Hello,\n\nI would like to see all of my app's analytics: ${address}\n\n`, + + /** At the foot of the page. */ + footer: (minutes: number) => + minutes === 1 ? 'Counted per day, in UTC. Updated every minute.' : `Counted per day, in UTC. Updated every ${minutes} minutes.`, + updated: (time: string) => `Last update at ${time}.`, + since: (date: string) => `Counting since ${date}.`, + + switchOff: 'Switch off analytics', + switchOffTitle: 'Switch off analytics?', + switchOffText: 'Counting stops, and everything counted so far is deleted. This cannot be undone.', + keep: 'Keep counting', + switchOffConfirm: 'Switch off and delete', + switchedOff: 'Analytics are off, and what was counted is deleted.', + switchOffFailed: 'Analytics could not be switched off.', +}; diff --git a/src/Frontend/src/locales/en/editor/frame.tsx b/src/Frontend/src/locales/en/editor/frame.tsx index d17663e..d031014 100644 --- a/src/Frontend/src/locales/en/editor/frame.tsx +++ b/src/Frontend/src/locales/en/editor/frame.tsx @@ -19,6 +19,8 @@ export const frame = { history: 'History', deployments: 'Deployments', stats: 'Stats', + /** Who visits the lambda - from where, to which pages, on what - once its owner switched it on. */ + analytics: 'Analytics', logs: 'Logs', /** Every file of a version - its code and its resources - to read and to change, in the full view only. */ code: 'Code', diff --git a/src/Frontend/src/locales/en/editor/index.ts b/src/Frontend/src/locales/en/editor/index.ts index bcdf877..ed23087 100644 --- a/src/Frontend/src/locales/en/editor/index.ts +++ b/src/Frontend/src/locales/en/editor/index.ts @@ -1,3 +1,4 @@ +import { analytics } from './analytics'; import { change } from './change'; import { clone } from './clone'; import { code } from './code'; @@ -38,6 +39,7 @@ export const editor = { versions, deployments, stats, + analytics, logs, showcase, openSource, diff --git a/src/Frontend/src/locales/en/guide.tsx b/src/Frontend/src/locales/en/guide.tsx index d2ea43c..65c5d2d 100644 --- a/src/Frontend/src/locales/en/guide.tsx +++ b/src/Frontend/src/locales/en/guide.tsx @@ -30,6 +30,7 @@ export const guide = { away: 'Taking it away', git: 'Working on it with git', open: 'Publishing the code', + visitors: 'Who visits it', agents: 'Letting an agent do it', }, @@ -110,6 +111,7 @@ export const guide = { ['Versions', () => <>What each version changed and what was asked for, and the difference to the one before. Deploy or roll back from here, or start a draft from any of them.], ['Deployments', () => <>What was online when, and what took it down.], ['Stats', () => <>Requests, failures, response times and the most asked-for paths, over the last hour or day.], + ['Analytics', () => <>Who visits the app, once you switch it on: visitors, page views and requests per day, the pages they open, the sites that send them, their countries and devices, and the search engines and AIs that came by.], ['Logs', () => <>Its requests, what it printed, and the stack trace of anything that went wrong, as it happens.], [ 'Code', @@ -133,7 +135,7 @@ export const guide = { )) as Text, editorAside: - 'The traffic and the log are held in memory, for watching rather than keeping: a restart of the server begins them again. Versions and the deployment history are stored.', + 'The traffic and the log are held in memory, for watching rather than keeping: a restart of the server begins them again. Versions, the deployment history and the analytics are stored.', why: ((k) => ( <> @@ -500,6 +502,28 @@ export const guide = { openAside: 'Everything in the code becomes public, the earlier versions included. A key or a password belongs with the keys and passwords under Data, never in the code - published or not.', + visitors: ((k) => ( + <> + Open {k.b('Analytics')} in the control center and switch it on to see who visits your app: how many people + come each day, which pages they open, which sites send them - a search engine, a post, a newsletter - from + which countries, on which devices and browsers, and which search engines and AI assistants read its pages. + Pick a figure at the top to see it day by day. + + )) as Text, + visitors2: ((k) => ( + <> + It is counted by the platform, in front of your app, so your app needs nothing for it: no cookies, no script, + no banner for your visitors to accept. Only counts per day are kept - never an address, and of the page somebody + came from only the site. A visitor is a person, counted once a day: search engines, AI crawlers and other + programs are counted apart, by what they are, and Google, Bing, OpenAI, Apple and Perplexity only by name when + the visit came from an address they publish - anybody can claim to be them. A year is kept; a free app shows the last seven + days of it and {k.b('Premium')} all of it, what was counted before included. Switching it off deletes what was + counted. + + )) as Text, + visitorsAside: + 'The figures are written down every few minutes, so the last visits show up a little later. A draft is never counted: trying a change is not a visit.', + agents: ((k) => ( <> There is an MCP endpoint at {k.code('/mcp')}. Point an agent at it and it can do everything the editor does: diff --git a/src/Frontend/src/locales/en/privacy.tsx b/src/Frontend/src/locales/en/privacy.tsx index 2fa2b91..bbf6e86 100644 --- a/src/Frontend/src/locales/en/privacy.tsx +++ b/src/Frontend/src/locales/en/privacy.tsx @@ -20,6 +20,12 @@ export const privacy = { 'Administrators can read all of it. The owner of a lambda sees the country and the browser of each request to their lambda, but not the IP address.', ], + analyticsTitle: 'Visits counted for the owner of a lambda', + analytics: [ + "The owner of a lambda can switch on analytics for it. Then the server counts the visits to that lambda by day: how many requests and page views there were, which pages they were for, the site a visitor came from (its name only, not the address of the page), the country of the visitor's IP address, and the kind of device, browser and operating system the browser names. Only these counts are kept - not the IP address and nothing else that names a visitor - in a database on the server, for a year. The owner of the lambda sees them in its editor, and administrators can read them.", + "To count each visitor once a day, the IP address and the browser are turned into a code that cannot be turned back, with a key made at random for each day and lambda. The codes and the keys are held in memory only, never written down, and forgotten after a day. No cookie is set and nothing is stored in the visitor's browser. Where a visit says it comes from the crawler of Google, Bing, OpenAI, Apple or Perplexity, its IP address is compared, in memory, with the addresses these publish for their crawlers. To tell crawlers from people, the server fetches a public list of crawlers and these published addresses once a day; nothing about visitors is sent anywhere for it. The legal basis is the owner's legitimate interest in knowing how their lambda is used (Art. 6(1)(f) GDPR).", + ], + logsTitle: 'How long the log is kept', logs: "The log is kept in two places: in the server's memory, which is emptied whenever the server restarts, and in the console output of the server, which is removed whenever the server is updated. Both have a fixed size, so every new line pushes out the oldest one, and how long a line lasts depends on how busy the site is. Nothing from the log is copied into an archive.", @@ -44,14 +50,14 @@ export const privacy = { lambdasTitle: 'What a lambda does is up to its owner', lambdas: - 'A lambda is written by whoever holds its editor link, not by us. What it asks its visitors for and what it does with that is up to them, and this page does not cover it - apart from the request log above, which the server keeps for every lambda. The terms do not allow using a lambda to collect personal data about other people; if you come across one that does, please report it.', + 'A lambda is written by whoever holds its editor link, not by us. What it asks its visitors for and what it does with that is up to them, and this page does not cover it - apart from the request log above, which the server keeps for every lambda, and the visits it counts for an owner who switched analytics on. The terms do not allow using a lambda to collect personal data about other people; if you come across one that does, please report it.', mailTitle: 'When you write to us', mail: 'If you write to us, to report abuse or about anything else, we use your address and your message to answer you and to deal with what you wrote, and delete them once they are no longer needed for that (Art. 6(1)(f) GDPR).', storageTitle: 'Cookies and your browser', storage: - "There is one cookie, called lang. It remembers the language you picked, so that addresses without one open in it, and it lasts a year. The browser's own storage remembers light or dark mode, a few settings of the pages you use and, for administrators, their token. None of it is used to follow you and none of it goes to anybody else: there is no analytics, no advertising, and nothing is loaded from other sites, not even fonts. Since all of it only does what you asked for, it needs no consent (§ 25(2) no. 2 TDDDG).", + "There is one cookie, called lang. It remembers the language you picked, so that addresses without one open in it, and it lasts a year. The browser's own storage remembers light or dark mode, a few settings of the pages you use and, for administrators, their token. None of it is used to follow you and none of it goes to anybody else: there is no analytics of this site, no advertising, and nothing is loaded from other sites, not even fonts. Since all of it only does what you asked for, it needs no consent (§ 25(2) no. 2 TDDDG).", hostingTitle: 'Where it is kept', hosting: diff --git a/src/Frontend/src/locales/es/editor/analytics.tsx b/src/Frontend/src/locales/es/editor/analytics.tsx new file mode 100644 index 0000000..b348612 --- /dev/null +++ b/src/Frontend/src/locales/es/editor/analytics.tsx @@ -0,0 +1,117 @@ +import type { EditorMessages } from '../../en/editor'; + +export const analytics: EditorMessages['analytics'] = { + hint: 'Lo cuenta la plataforma por ti: sin cookies, sin código en tu app y sin nada que tus visitantes tengan que aceptar.', + loading: 'Leyendo las visitas…', + loadFailed: 'No se pudieron leer las visitas.', + + offTitle: 'Descubre quién visita tu app', + offText: + 'Cuánta gente llega cada día, qué páginas abre, qué sitios la envían, desde qué países y en qué dispositivos: se cuenta día a día, desde el momento en que lo activas.', + offPrivate: 'Sin cookies y sin ningún aviso que tus visitantes tengan que aceptar', + offAddresses: 'No se guardan direcciones: solo cuántas visitas hubo cada día', + offKept: (days) => + days >= 365 + ? 'Se guarda durante un año y se te muestra todo' + : `Se guarda durante un año: se muestran los últimos ${days} días, y todo con Premium`, + switchOn: 'Activar la analítica', + switchedOn: 'La analítica está activada. Las primeras visitas aparecen en unos minutos.', + switchOnFailed: 'No se pudo activar la analítica.', + demo: 'Los visitantes de una demo no se cuentan. Una copia tuya sí puede contar a sus visitantes.', + + range: 'Periodo', + days: (days) => (days >= 365 ? '12 meses' : `${days} días`), + premiumRange: 'Premium muestra hasta un año', + + visitors: 'Visitantes', + visitorsHint: 'Personas distintas, cada una contada una vez al día. Los buscadores y otros programas no son visitantes.', + views: 'Visualizaciones de página', + viewsHint: 'Cada vez que alguien abrió una página.', + hits: 'Peticiones', + hitsHint: 'Todo lo que se pidió a tu app: sus páginas y también las imágenes y los scripts que cargan.', + versus: (days) => `frente a los ${days} días anteriores`, + noChange: 'sin cambios', + inDays: (days) => (days >= 365 ? 'en los últimos 12 meses' : `en los últimos ${days} días`), + + perDay: { + visitors: 'Visitantes por día', + views: 'Visualizaciones de página por día', + hits: 'Peticiones por día', + }, + perWeek: { + visitors: 'Visitantes por semana', + views: 'Visualizaciones de página por semana', + hits: 'Peticiones por semana', + }, + showValues: 'Mostrar los valores', + showChart: 'Mostrar el gráfico', + day: 'Día', + week: 'Semana del', + + nobody: 'Nadie visitó la app en estos días.', + nobodyYet: 'Nadie la ha visitado desde que se activó la analítica. Las primeras visitas aparecen en unos minutos.', + + pages: 'Páginas', + pagesHint: 'Las páginas que abrió la gente, las más abiertas primero.', + requested: 'Lo más pedido', + requestedHint: 'Todo lo que se pidió a tu app, por dirección: no tiene páginas que la gente abra.', + sources: 'De dónde vienen los visitantes', + sourcesHint: 'Los sitios que enviaron gente aquí, según la página en la que estaban. Los buscadores solo indican que fueron ellos, no qué se buscó.', + direct: 'Escrito a mano, un marcador o una app', + countries: 'Países', + countriesHint: 'Donde están registradas las direcciones IP de los visitantes: normalmente, aunque no siempre, donde se encuentran.', + unknownCountry: 'Desconocido', + devices: 'Dispositivos', + device: { + desktop: 'Ordenador', + mobile: 'Móvil', + tablet: 'Tableta', + bot: 'Bots', + other: 'Otros', + }, + browsers: 'Navegadores', + systems: 'Sistemas', + crawlers: 'Buscadores, IA y otros programas', + crawlersHint: 'Programas que pasaron por aquí, según lo que son. Google, Bing, OpenAI, Apple y Perplexity solo se nombran cuando la visita llegó desde una dirección que ellos publican: cualquiera puede decir que es uno de ellos.', + noCrawlers: 'Ningún buscador, IA ni otro programa ha pasado por aquí en estos días.', + confirmed: (name) => `Llegó desde una dirección que ${name} publica para sus rastreadores`, + crawlerKinds: { + 'search-engine': 'Buscadores', + 'ai-crawler': 'Rastreadores de IA', + 'social-preview': 'Vistas previas de enlaces en chats y redes sociales', + advertising: 'Publicidad', + seo: 'Herramientas de SEO', + monitoring: 'Monitorización', + 'feed-reader': 'Lectores de feeds', + archiver: 'Archivos web', + academic: 'Investigación', + scanner: 'Escáneres de seguridad', + 'browser-automation': 'Navegadores automatizados', + 'http-library': 'Scripts y apps', + other: 'Otros programas', + unconfirmed: 'Dijeron ser un rastreador conocido, sin confirmar', + }, + other: 'Otros', + showAll: (count) => `Mostrar los ${count}`, + showFewer: 'Mostrar menos', + share: (part, whole) => `${part} de ${whole}`, + none: 'Todavía nada.', + + moreKept: (since) => `Contado desde ${since}: Premium muestra todo, hasta un año.`, + premiumAsk: 'Preguntar por Premium', + premiumSubject: 'Premium para mi app', + premiumBody: (address) => `Hola:\n\nMe gustaría ver toda la analítica de mi app: ${address}\n\n`, + + footer: (minutes) => + minutes === 1 ? 'Se cuenta por día, en UTC. Se actualiza cada minuto.' : `Se cuenta por día, en UTC. Se actualiza cada ${minutes} minutos.`, + updated: (time) => `Última actualización a las ${time}.`, + since: (date) => `Contando desde ${date}.`, + + switchOff: 'Desactivar la analítica', + switchOffTitle: '¿Desactivar la analítica?', + switchOffText: 'Se deja de contar y se borra todo lo contado hasta ahora. No se puede deshacer.', + keep: 'Seguir contando', + switchOffConfirm: 'Desactivar y borrar', + switchedOff: 'La analítica está desactivada y lo contado se ha borrado.', + switchOffFailed: 'No se pudo desactivar la analítica.', +}; diff --git a/src/Frontend/src/locales/es/editor/frame.tsx b/src/Frontend/src/locales/es/editor/frame.tsx index af7b9ae..221ad09 100644 --- a/src/Frontend/src/locales/es/editor/frame.tsx +++ b/src/Frontend/src/locales/es/editor/frame.tsx @@ -16,6 +16,7 @@ export const frame: EditorMessages['frame'] = { history: 'Historial', deployments: 'Despliegues', stats: 'Estadísticas', + analytics: 'Analítica', logs: 'Logs', code: 'Código', }, diff --git a/src/Frontend/src/locales/es/editor/index.ts b/src/Frontend/src/locales/es/editor/index.ts index 3068bd3..af55eae 100644 --- a/src/Frontend/src/locales/es/editor/index.ts +++ b/src/Frontend/src/locales/es/editor/index.ts @@ -1,4 +1,5 @@ import type { EditorMessages } from '../../en/editor'; +import { analytics } from './analytics'; import { change } from './change'; import { clone } from './clone'; import { code } from './code'; @@ -33,6 +34,7 @@ export const editor: EditorMessages = { versions, deployments, stats, + analytics, logs, showcase, openSource, diff --git a/src/Frontend/src/locales/es/guide.tsx b/src/Frontend/src/locales/es/guide.tsx index c23bb7d..2ac92b7 100644 --- a/src/Frontend/src/locales/es/guide.tsx +++ b/src/Frontend/src/locales/es/guide.tsx @@ -26,6 +26,7 @@ export const guide: Messages['guide'] = { away: 'Llévate tu código', git: 'Trabajar con git', open: 'Publicar el código', + visitors: 'Quién la visita', agents: 'Que lo haga un agente', }, @@ -106,6 +107,7 @@ export const guide: Messages['guide'] = { ['Versiones', () => <>Qué cambió cada versión, qué se pidió y la diferencia con la anterior. Desde aquí despliegas o vuelves atrás, o empiezas un borrador a partir de cualquiera de ellas.], ['Despliegues', () => <>Qué estuvo en línea y cuándo, y qué lo desconectó.], ['Estadísticas', () => <>Peticiones, fallos, tiempos de respuesta y las rutas más pedidas, en la última hora o el último día.], + ['Analítica', () => <>Quién visita la app, en cuanto la activas: visitantes, visualizaciones de página y peticiones por día, las páginas que abren, los sitios que los envían, sus países y dispositivos, y los buscadores y las IA que han pasado por aquí.], ['Logs', () => <>Sus peticiones, lo que imprimió y el stack trace de cualquier error, en tiempo real.], [ 'Código', @@ -129,7 +131,7 @@ export const guide: Messages['guide'] = { ), editorAside: - 'El tráfico y los logs se guardan en memoria, para mirarlos, no para conservarlos: si el servidor se reinicia, empiezan de cero. Las versiones y el historial de despliegues sí se guardan.', + 'El tráfico y los logs se guardan en memoria, para mirarlos, no para conservarlos: si el servidor se reinicia, empiezan de cero. Las versiones, el historial de despliegues y la analítica sí se guardan.', why: (k) => ( <> @@ -494,6 +496,27 @@ export const guide: Messages['guide'] = { ), openAside: 'Todo lo que hay en el código se hace público, incluidas las versiones anteriores. Una clave o una contraseña va en Claves y contraseñas, dentro de Datos, nunca en el código, esté publicado o no.', + visitors: (k) => ( + <> + Abre {k.b('Analítica')} en el centro de control y actívala para ver quién visita tu app: cuánta gente llega cada + día, qué páginas abre, qué sitios la envían (un buscador, una publicación, un boletín), desde qué países, en qué + dispositivos y navegadores, y qué buscadores y asistentes de IA leen sus páginas. Elige una cifra arriba para + verla día a día. + + ), + visitors2: (k) => ( + <> + La cuenta la lleva la plataforma, delante de tu app, así que tu app no necesita nada para ello: ni cookies, ni + scripts, ni avisos que tus visitantes tengan que aceptar. Solo se guardan recuentos por día, nunca una dirección, + y de la página de la que alguien viene solo se guarda el sitio. Un visitante es una persona, contada una vez al + día: los buscadores, los rastreadores de IA y otros programas se cuentan aparte, según lo que son, y Google, Bing, + OpenAI, Apple y Perplexity solo se nombran cuando la visita llegó desde una dirección que ellos publican: cualquiera + puede decir que es uno de ellos. Se guarda un año: una app gratuita muestra los últimos siete días y {k.b('Premium')} + todo, incluido lo contado antes. Al desactivarla se borra lo contado. + + ), + visitorsAside: + 'Las cifras se anotan cada pocos minutos, así que las últimas visitas aparecen un poco más tarde. Un borrador nunca se cuenta: probar un cambio no es una visita.', agents: (k) => ( <> diff --git a/src/Frontend/src/locales/es/privacy.tsx b/src/Frontend/src/locales/es/privacy.tsx index f299a64..791127f 100644 --- a/src/Frontend/src/locales/es/privacy.tsx +++ b/src/Frontend/src/locales/es/privacy.tsx @@ -20,6 +20,11 @@ export const privacy: Messages['privacy'] = { 'Así se encuentran los fallos, se averigua qué está sobrecargando el servidor y se localizan los abusos que nos denuncian. La dirección IP también se usa, solo en memoria, para limitar cuántas peticiones puede hacer y cuántas apps puede crear un mismo visitante. Sin estos datos no se puede responder a una petición. La base jurídica es nuestro interés legítimo en prestar el servicio y mantenerlo seguro (art. 6.1.f) RGPD).', 'Los administradores pueden verlo todo. El dueño de una lambda ve el país y el navegador de cada petición a su lambda, pero no la dirección IP.', ], + analyticsTitle: 'Visitas contadas para el propietario de una lambda', + analytics: [ + "El propietario de una lambda puede activar la analítica para ella. Entonces el servidor cuenta por día las visitas a esa lambda: cuántas peticiones y visualizaciones de página hubo, para qué páginas, el sitio del que venía el visitante (solo su nombre, no la dirección de la página), el país de la dirección IP del visitante y el tipo de dispositivo, navegador y sistema operativo que indica el navegador. Solo se guardan estos recuentos, no la dirección IP ni nada más que identifique a un visitante, en una base de datos del servidor, durante un año. El propietario de la lambda los ve en su editor y los administradores pueden leerlos.", + "Para contar a cada visitante una vez al día, la dirección IP y el navegador se convierten en un código que no se puede revertir, con una clave generada al azar para cada día y cada lambda. Los códigos y las claves solo se mantienen en memoria, nunca se escriben y se olvidan al cabo de un día. No se instala ninguna cookie ni se guarda nada en el navegador del visitante. Cuando una visita dice provenir del rastreador de Google, Bing, OpenAI, Apple o Perplexity, su dirección IP se compara, en memoria, con las direcciones que estas empresas publican para sus rastreadores. Para distinguir los rastreadores de las personas, el servidor descarga una vez al día una lista pública de rastreadores y estas direcciones publicadas; para ello no se envía a ningún sitio nada sobre los visitantes. La base jurídica es el interés legítimo del propietario en saber cómo se usa su lambda (art. 6, apdo. 1, letra f del RGPD).", + ], logsTitle: 'Cuánto tiempo se guarda el registro', logs: 'El registro se guarda en dos sitios: en la memoria del servidor, que se vacía cada vez que el servidor se reinicia, y en la salida de consola del servidor, que se borra cada vez que se actualiza. Los dos tienen un tamaño fijo, así que cada línea nueva desplaza a la más antigua, y cuánto dura una línea depende de cuánto movimiento tenga el sitio. Nada del registro se copia para archivarlo.', @@ -45,14 +50,14 @@ export const privacy: Messages['privacy'] = { lambdasTitle: 'Lo que hace una lambda lo decide su dueño', lambdas: - 'Una lambda la escribe quien tiene su enlace de edición, no nosotros. Qué pide a sus visitantes y qué hace con ello lo decide esa persona, y esta página no lo cubre, salvo el registro de peticiones de arriba, que el servidor lleva para todas las lambdas. Las condiciones del servicio no permiten usar una lambda para recoger datos personales de otras personas. Si encuentras una que lo hace, denúnciala, por favor.', + 'Una lambda la escribe quien tiene su enlace de edición, no nosotros. Qué pide a sus visitantes y qué hace con ello lo decide esa persona, y esta página no lo cubre, salvo el registro de peticiones de arriba, que el servidor lleva para todas las lambdas, y las visitas que cuenta para un propietario que activó la analítica. Las condiciones del servicio no permiten usar una lambda para recoger datos personales de otras personas. Si encuentras una que lo hace, denúnciala, por favor.', mailTitle: 'Cuando nos escribes', mail: 'Si nos escribes, para denunciar un abuso o por cualquier otro motivo, usamos tu dirección de correo y tu mensaje para responderte y ocuparnos de lo que nos cuentas, y los borramos cuando ya no hacen falta para eso (art. 6.1.f) RGPD).', storageTitle: 'Cookies y tu navegador', storage: - 'Hay una sola cookie, llamada lang. Recuerda el idioma que elegiste, para que las direcciones que no indican idioma se abran en él, y dura un año. El almacenamiento local del navegador recuerda el modo claro u oscuro, algunos ajustes de las páginas que usas y, en el caso de los administradores, su token. Nada de esto se usa para seguirte y nada llega a nadie más: no hay analítica, ni publicidad, y no se carga nada de otros sitios, ni siquiera las fuentes. Como todo sirve solo para lo que tú pediste, no hace falta tu consentimiento (§ 25, apdo. 2, n.º 2 de la ley alemana TDDDG).', + 'Hay una sola cookie, llamada lang. Recuerda el idioma que elegiste, para que las direcciones que no indican idioma se abran en él, y dura un año. El almacenamiento local del navegador recuerda el modo claro u oscuro, algunos ajustes de las páginas que usas y, en el caso de los administradores, su token. Nada de esto se usa para seguirte y nada llega a nadie más: no hay analítica de este sitio, ni publicidad, y no se carga nada de otros sitios, ni siquiera las fuentes. Como todo sirve solo para lo que tú pediste, no hace falta tu consentimiento (§ 25, apdo. 2, n.º 2 de la ley alemana TDDDG).', hostingTitle: 'Dónde se guarda', hosting: diff --git a/src/Frontend/src/locales/fr/editor/analytics.tsx b/src/Frontend/src/locales/fr/editor/analytics.tsx new file mode 100644 index 0000000..493f3f7 --- /dev/null +++ b/src/Frontend/src/locales/fr/editor/analytics.tsx @@ -0,0 +1,117 @@ +import type { EditorMessages } from '../../en/editor'; + +export const analytics: EditorMessages['analytics'] = { + hint: 'Compté pour vous par la plateforme : pas de cookies, pas de code dans votre application, et rien à accepter pour vos visiteurs.', + loading: 'Lecture des visites…', + loadFailed: 'Les visites n’ont pas pu être lues.', + + offTitle: 'Voyez qui visite votre application', + offText: + 'Combien de personnes viennent chaque jour, quelles pages elles ouvrent, quels sites les envoient, de quels pays et sur quels appareils - comptés jour par jour, dès que vous l’activez.', + offPrivate: 'Aucun cookie, et aucune bannière à faire accepter à vos visiteurs', + offAddresses: 'Aucune adresse n’est conservée - seulement le nombre de visites de chaque jour', + offKept: (days) => + days >= 365 + ? 'Conservé un an, et tout vous est montré' + : `Conservé un an - les ${days} derniers jours sont affichés, tout avec Premium`, + switchOn: 'Activer la mesure d’audience', + switchedOn: 'La mesure d’audience est activée. Les premières visites apparaissent d’ici quelques minutes.', + switchOnFailed: 'La mesure d’audience n’a pas pu être activée.', + demo: 'Les visiteurs d’une démo ne sont pas comptés. Une copie à vous peut compter ses visiteurs.', + + range: 'Période', + days: (days) => (days >= 365 ? '12 mois' : `${days} jours`), + premiumRange: 'Premium affiche jusqu’à un an', + + visitors: 'Visiteurs', + visitorsHint: 'Personnes différentes, chacune comptée une fois par jour. Les moteurs de recherche et les autres programmes ne sont pas des visiteurs.', + views: 'Pages vues', + viewsHint: 'Chaque fois que quelqu’un a ouvert une page.', + hits: 'Requêtes', + hitsHint: 'Tout ce qui a été demandé à votre application : ses pages, et les images et scripts qu’elles chargent.', + versus: (days) => `par rapport aux ${days} jours précédents`, + noChange: 'inchangé', + inDays: (days) => (days >= 365 ? 'sur les 12 derniers mois' : `sur les ${days} derniers jours`), + + perDay: { + visitors: 'Visiteurs par jour', + views: 'Pages vues par jour', + hits: 'Requêtes par jour', + }, + perWeek: { + visitors: 'Visiteurs par semaine', + views: 'Pages vues par semaine', + hits: 'Requêtes par semaine', + }, + showValues: 'Afficher les valeurs', + showChart: 'Afficher le graphique', + day: 'Jour', + week: 'Semaine du', + + nobody: 'Personne n’est venu ces jours-ci.', + nobodyYet: 'Personne n’est venu depuis l’activation de la mesure d’audience. Les premières visites apparaissent d’ici quelques minutes.', + + pages: 'Pages', + pagesHint: 'Les pages ouvertes, les plus ouvertes en premier.', + requested: 'Les plus demandées', + requestedHint: 'Tout ce qui a été demandé à votre application, par adresse - elle n’a pas de pages que l’on ouvre.', + sources: 'D’où viennent les visiteurs', + sourcesHint: 'Les sites qui ont envoyé des gens ici, avec la page où ils se trouvaient. Les moteurs de recherche indiquent seulement que c’était eux, pas ce qui a été cherché.', + direct: 'Saisie directe, favori ou application', + countries: 'Pays', + countriesHint: 'Où sont enregistrées les adresses internet des visiteurs - généralement, mais pas toujours, là où ils se trouvent.', + unknownCountry: 'Inconnu', + devices: 'Appareils', + device: { + desktop: 'Ordinateur', + mobile: 'Téléphone', + tablet: 'Tablette', + bot: 'Robots', + other: 'Autre', + }, + browsers: 'Navigateurs', + systems: 'Systèmes', + crawlers: 'Moteurs de recherche, IA et autres programmes', + crawlersHint: 'Programmes passés par là, classés selon ce qu’ils sont. Google, Bing, OpenAI, Apple et Perplexity ne sont nommés que si la visite provient d’une adresse qu’ils publient : n’importe qui peut se faire passer pour eux.', + noCrawlers: 'Aucun moteur de recherche, aucune IA ni aucun autre programme n’est passé ces jours-ci.', + confirmed: (name) => `Provient d’une adresse que ${name} publie pour ses robots d’exploration`, + crawlerKinds: { + 'search-engine': 'Moteurs de recherche', + 'ai-crawler': 'Robots d’exploration IA', + 'social-preview': 'Aperçus de liens dans les messageries et les réseaux sociaux', + advertising: 'Publicité', + seo: 'Outils SEO', + monitoring: 'Surveillance', + 'feed-reader': 'Lecteurs de flux', + archiver: 'Archives du web', + academic: 'Recherche', + scanner: 'Scanners de sécurité', + 'browser-automation': 'Navigateurs automatisés', + 'http-library': 'Scripts et applications', + other: 'Autres programmes', + unconfirmed: 'Se présente comme un robot connu, sans confirmation', + }, + other: 'Autres', + showAll: (count) => `Tout afficher (${count})`, + showFewer: 'Afficher moins', + share: (part, whole) => `${part} sur ${whole}`, + none: 'Rien pour l’instant.', + + moreKept: (since) => `Décompte depuis le ${since} - Premium affiche tout, jusqu’à un an.`, + premiumAsk: 'Se renseigner sur Premium', + premiumSubject: 'Premium pour mon application', + premiumBody: (address) => `Bonjour,\n\nJe voudrais voir toutes les statistiques d’audience de mon application : ${address}\n\n`, + + footer: (minutes) => + minutes === 1 ? 'Décompte par jour, en UTC. Actualisé chaque minute.' : `Décompte par jour, en UTC. Actualisé toutes les ${minutes} minutes.`, + updated: (time) => `Dernière mise à jour à ${time}.`, + since: (date) => `Décompte depuis le ${date}.`, + + switchOff: 'Désactiver la mesure d’audience', + switchOffTitle: 'Désactiver la mesure d’audience ?', + switchOffText: 'Le comptage s’arrête, et tout ce qui a été compté jusqu’ici est supprimé. Cette action est irréversible.', + keep: 'Continuer à compter', + switchOffConfirm: 'Désactiver et supprimer', + switchedOff: 'La mesure d’audience est désactivée, et ce qui avait été compté est supprimé.', + switchOffFailed: 'La mesure d’audience n’a pas pu être désactivée.', +}; diff --git a/src/Frontend/src/locales/fr/editor/frame.tsx b/src/Frontend/src/locales/fr/editor/frame.tsx index 875de9a..8bd44dc 100644 --- a/src/Frontend/src/locales/fr/editor/frame.tsx +++ b/src/Frontend/src/locales/fr/editor/frame.tsx @@ -16,6 +16,7 @@ export const frame: EditorMessages['frame'] = { history: 'Historique', deployments: 'Déploiements', stats: 'Stats', + analytics: 'Audience', logs: 'Logs', code: 'Code', }, diff --git a/src/Frontend/src/locales/fr/editor/index.ts b/src/Frontend/src/locales/fr/editor/index.ts index b9f3fa7..2e3b137 100644 --- a/src/Frontend/src/locales/fr/editor/index.ts +++ b/src/Frontend/src/locales/fr/editor/index.ts @@ -1,4 +1,5 @@ import type { EditorMessages } from '../../en/editor'; +import { analytics } from './analytics'; import { change } from './change'; import { clone } from './clone'; import { code } from './code'; @@ -33,6 +34,7 @@ export const editor: EditorMessages = { versions, deployments, stats, + analytics, logs, showcase, openSource, diff --git a/src/Frontend/src/locales/fr/guide.tsx b/src/Frontend/src/locales/fr/guide.tsx index 2692560..11689e3 100644 --- a/src/Frontend/src/locales/fr/guide.tsx +++ b/src/Frontend/src/locales/fr/guide.tsx @@ -26,6 +26,7 @@ export const guide: Messages['guide'] = { away: 'Repartir avec votre code', git: 'Travailler avec git', open: 'Publier le code', + visitors: 'Voir qui la visite', agents: 'Laisser faire un agent', }, @@ -106,6 +107,7 @@ export const guide: Messages['guide'] = { ['Versions', () => <>Ce que chaque version a changé, ce qui avait été demandé, et la différence avec la précédente. C’est ici qu’on déploie ou qu’on revient en arrière, ou qu’on démarre un brouillon à partir de n’importe quelle version.], ['Déploiements', () => <>Ce qui était en ligne, quand, et ce qui l’a arrêté.], ['Stats', () => <>Requêtes, échecs, temps de réponse et chemins les plus demandés, sur la dernière heure ou les dernières 24 heures.], + ['Audience', () => <>Qui visite l’application, une fois que vous l’avez activée : visiteurs, pages vues et requêtes par jour, les pages ouvertes, les sites qui envoient les visiteurs, leurs pays et leurs appareils, et les moteurs de recherche et IA qui sont passés.], ['Logs', () => <>Ses requêtes, ce qu’elle affiche, et la stack trace de tout ce qui plante, en direct.], [ 'Code', @@ -129,7 +131,7 @@ export const guide: Messages['guide'] = { ), editorAside: - 'Le trafic et les logs sont gardés en mémoire, pour suivre ce qui se passe, pas pour archiver : un redémarrage du serveur les remet à zéro. Les versions et l’historique des déploiements, eux, sont enregistrés.', + 'Le trafic et les logs sont gardés en mémoire, pour suivre ce qui se passe, pas pour archiver : un redémarrage du serveur les remet à zéro. Les versions, l’historique des déploiements et l’audience, eux, sont enregistrés.', why: (k) => ( <> @@ -506,6 +508,29 @@ export const guide: Messages['guide'] = { ), openAside: 'Tout ce qui est dans le code devient public, les versions précédentes comprises. Une clé ou un mot de passe a sa place avec les clés et mots de passe, sous Données, jamais dans le code – publié ou non.', + visitors: ((k) => ( + <> + Ouvrez {k.b('Audience')} dans le tableau de bord et activez-la pour voir qui visite votre application : combien + de personnes viennent chaque jour, quelles pages elles ouvrent, quels sites les envoient - un moteur de + recherche, une publication, une newsletter - depuis quels pays, sur quels appareils et navigateurs, et quels + moteurs de recherche et assistants IA lisent ses pages. Choisissez un chiffre en haut pour le voir jour par jour. + + )), + visitors2: ((k) => ( + <> + Le comptage est fait par la plateforme, en amont de votre application, qui n’a donc besoin de rien : pas de + cookies, pas de script, pas de bannière à faire accepter à vos visiteurs. Seuls des totaux par jour sont + conservés - jamais une adresse, et de la page d’où quelqu’un est venu, seulement le site. Un visiteur est une + personne, comptée une fois par jour : les moteurs de recherche, les robots d’exploration IA et les autres + programmes sont comptés à part, selon ce qu’ils sont, et Google, Bing, OpenAI, Apple et Perplexity ne sont + nommés que si la visite provient d’une adresse qu’ils publient : n’importe qui peut se faire passer pour eux. + Un an est conservé ; une application gratuite en montre les sept derniers jours et{' '} + {k.b('Premium')} la totalité, y compris ce qui a été compté avant. Désactiver la mesure d’audience supprime ce + qui a été compté. + + )), + visitorsAside: + 'Les chiffres sont enregistrés toutes les quelques minutes, les dernières visites apparaissent donc un peu plus tard. Un brouillon n’est jamais compté : essayer une modification n’est pas une visite.', agents: (k) => ( <> diff --git a/src/Frontend/src/locales/fr/privacy.tsx b/src/Frontend/src/locales/fr/privacy.tsx index f511d1e..39bc142 100644 --- a/src/Frontend/src/locales/fr/privacy.tsx +++ b/src/Frontend/src/locales/fr/privacy.tsx @@ -20,6 +20,11 @@ export const privacy: Messages['privacy'] = { 'C’est ce qui permet de trouver les pannes, de remonter jusqu’à ce qui surcharge le serveur et de retrouver les abus qui nous sont signalés. L’adresse IP sert aussi, uniquement en mémoire, à limiter le nombre de requêtes et de créations d’un même visiteur. Sans ces informations, impossible de répondre à une requête. La base légale est notre intérêt légitime à faire fonctionner le service et à le sécuriser (art. 6.1.f du RGPD).', 'Les administrateurs peuvent tout consulter. Le propriétaire d’une lambda voit le pays et le navigateur de chaque requête vers sa lambda, mais pas l’adresse IP.', ], + analyticsTitle: 'Visites comptées pour le propriétaire d’une lambda', + analytics: [ + 'Le propriétaire d’une lambda peut activer la mesure d’audience pour celle-ci. Le serveur compte alors les visites de cette lambda par jour : le nombre de requêtes et de pages vues, les pages concernées, le site d’où vient un visiteur (son nom seulement, pas l’adresse de la page), le pays de l’adresse IP du visiteur, ainsi que le type d’appareil, de navigateur et de système d’exploitation que le navigateur indique. Seuls ces totaux sont conservés - ni l’adresse IP, ni rien d’autre qui désigne un visiteur - dans une base de données du serveur, pendant un an. Le propriétaire de la lambda les consulte dans son éditeur, et les administrateurs peuvent les lire.', + 'Pour compter chaque visiteur une fois par jour, l’adresse IP et le navigateur sont transformés en un code irréversible, avec une clé générée au hasard pour chaque jour et chaque lambda. Les codes et les clés ne sont gardés qu’en mémoire, jamais écrits, et oubliés au bout d’un jour. Aucun cookie n’est déposé et rien n’est stocké dans le navigateur du visiteur. Lorsqu’une visite indique provenir du robot d’exploration de Google, Bing, OpenAI, Apple ou Perplexity, son adresse IP est comparée, en mémoire, aux adresses que ces sociétés publient pour leurs robots. Pour distinguer les robots des personnes, le serveur récupère une fois par jour une liste publique de robots et ces adresses publiées ; aucune donnée sur les visiteurs n’est envoyée à l’extérieur pour cela. La base juridique est l’intérêt légitime du propriétaire à savoir comment sa lambda est utilisée (art. 6, par. 1, point f du RGPD).', + ], logsTitle: 'Combien de temps le journal est conservé', logs: 'Le journal est conservé à deux endroits : dans la mémoire du serveur, qui est vidée à chaque redémarrage, et dans la sortie console du serveur, qui est effacée à chaque mise à jour. Les deux ont une taille fixe : chaque nouvelle ligne chasse la plus ancienne, et la durée de vie d’une ligne dépend de l’activité du site. Rien n’en est archivé.', @@ -46,14 +51,14 @@ export const privacy: Messages['privacy'] = { lambdasTitle: 'Ce que fait une lambda dépend de son propriétaire', lambdas: - 'Une lambda est écrite par la personne qui a son lien d’édition, pas par nous. Ce qu’elle demande à ses visiteurs et ce qu’elle en fait ne dépend que de cette personne, et cette page ne le couvre pas, à part le journal des requêtes décrit plus haut, que le serveur tient pour chaque lambda. Les conditions d’utilisation interdisent d’utiliser une lambda pour collecter des données personnelles sur d’autres personnes. Si vous en trouvez une qui le fait, merci de la signaler.', + 'Une lambda est écrite par la personne qui a son lien d’édition, pas par nous. Ce qu’elle demande à ses visiteurs et ce qu’elle en fait ne dépend que de cette personne, et cette page ne le couvre pas, à part le journal des requêtes décrit plus haut, que le serveur tient pour chaque lambda, et les visites qu’il compte pour un propriétaire qui a activé la mesure d’audience. Les conditions d’utilisation interdisent d’utiliser une lambda pour collecter des données personnelles sur d’autres personnes. Si vous en trouvez une qui le fait, merci de la signaler.', mailTitle: 'Quand vous nous écrivez', mail: 'Si vous nous écrivez, pour signaler un abus ou pour autre chose, nous utilisons votre adresse e-mail et votre message pour vous répondre et traiter ce que vous nous écrivez, puis nous les supprimons dès qu’ils ne sont plus nécessaires pour cela (art. 6.1.f du RGPD).', storageTitle: 'Cookies et navigateur', storage: - 'Il y a un seul cookie, appelé lang. Il retient la langue que vous avez choisie, pour que les adresses qui n’en indiquent pas s’ouvrent dans cette langue, et il dure un an. Le stockage local du navigateur retient le mode clair ou sombre, quelques réglages des pages que vous utilisez et, pour les administrateurs, leur jeton. Rien de tout cela ne sert à vous suivre et rien n’est transmis à qui que ce soit : pas de mesure d’audience, pas de publicité, et rien n’est chargé depuis d’autres sites, pas même les polices. Comme tout cela sert uniquement à ce que vous avez demandé, aucun consentement n’est nécessaire (§ 25, al. 2, n° 2 de la loi allemande TDDDG).', + 'Il y a un seul cookie, appelé lang. Il retient la langue que vous avez choisie, pour que les adresses qui n’en indiquent pas s’ouvrent dans cette langue, et il dure un an. Le stockage local du navigateur retient le mode clair ou sombre, quelques réglages des pages que vous utilisez et, pour les administrateurs, leur jeton. Rien de tout cela ne sert à vous suivre et rien n’est transmis à qui que ce soit : pas de mesure d’audience de ce site, pas de publicité, et rien n’est chargé depuis d’autres sites, pas même les polices. Comme tout cela sert uniquement à ce que vous avez demandé, aucun consentement n’est nécessaire (§ 25, al. 2, n° 2 de la loi allemande TDDDG).', hostingTitle: 'Où tout cela est stocké', hosting: diff --git a/src/Frontend/src/locales/glossary.json b/src/Frontend/src/locales/glossary.json index a177c48..39828a5 100644 --- a/src/Frontend/src/locales/glossary.json +++ b/src/Frontend/src/locales/glossary.json @@ -105,6 +105,11 @@ "en": "Showcase", "id": "Showcase", "de": "Showcase", "es": "Galería", "fr": "Vitrine", "it": "Vetrina", "nl": "Showcase", "pl": "Galeria", "pt": "Vitrine", "pt-pt": "Montra", "tr": "Vitrin", "ja": "ショーケース", "ko": "쇼케이스" }, + { + "en": "Analytics", + "note": "the editor's section that counts who visits a lambda, once its owner switched it on", + "id": "Analitik", "de": "Analyse", "es": "Analítica", "fr": "Audience", "it": "Analytics", "nl": "Analytics", "pl": "Analityka", "pt": "Analytics", "pt-pt": "Análise de visitas", "tr": "Analitik", "ja": "アクセス解析", "ko": "분석" + }, { "en": "open source", "id": "open source", "de": "Open Source", "es": "código abierto", "fr": "open source", "it": "open source", "nl": "open source", "pl": "open source", "pt": "código aberto", "pt-pt": "código aberto", "tr": "açık kaynak", "ja": "オープンソース", "ko": "오픈 소스" diff --git a/src/Frontend/src/locales/id/editor/analytics.tsx b/src/Frontend/src/locales/id/editor/analytics.tsx new file mode 100644 index 0000000..0b7d6b2 --- /dev/null +++ b/src/Frontend/src/locales/id/editor/analytics.tsx @@ -0,0 +1,117 @@ +import type { EditorMessages } from '../../en/editor'; + +export const analytics: EditorMessages['analytics'] = { + hint: 'Dihitung untuk Anda oleh platform: tanpa cookie, tanpa kode di aplikasi Anda, dan tanpa sesuatu yang harus disetujui pengunjung.', + loading: 'Membaca kunjungan…', + loadFailed: 'Kunjungan tidak dapat dibaca.', + + offTitle: 'Lihat siapa yang mengunjungi aplikasi Anda', + offText: + 'Berapa banyak orang yang datang setiap hari, halaman apa yang mereka buka, situs mana yang mengarahkan mereka, dari negara mana dan dengan perangkat apa - dihitung per hari, sejak Anda mengaktifkannya.', + offPrivate: 'Tanpa cookie, dan tanpa banner yang harus disetujui pengunjung Anda', + offAddresses: 'Tidak ada alamat yang disimpan - hanya jumlah kunjungan setiap hari', + offKept: (days) => + days >= 365 + ? 'Disimpan selama setahun, dan semuanya ditampilkan kepada Anda' + : `Disimpan selama setahun - ${days} hari terakhir ditampilkan, semuanya dengan Premium`, + switchOn: 'Aktifkan analitik', + switchedOn: 'Analitik aktif. Kunjungan pertama muncul dalam beberapa menit.', + switchOnFailed: 'Analitik tidak dapat diaktifkan.', + demo: 'Pengunjung sebuah demo tidak dihitung. Salinan milik Anda sendiri dapat menghitung pengunjungnya.', + + range: 'Rentang waktu', + days: (days) => (days >= 365 ? '12 bulan' : `${days} hari`), + premiumRange: 'Premium menampilkan hingga setahun', + + visitors: 'Pengunjung', + visitorsHint: 'Orang yang berbeda, masing-masing dihitung sekali sehari. Mesin pencari dan program lain bukan pengunjung.', + views: 'Tampilan halaman', + viewsHint: 'Setiap kali seseorang membuka sebuah halaman.', + hits: 'Request', + hitsHint: 'Semua yang diminta dari aplikasi Anda: halamannya, serta gambar dan skrip yang dimuat halaman itu.', + versus: (days) => `dibandingkan ${days} hari sebelumnya`, + noChange: 'tidak berubah', + inDays: (days) => (days >= 365 ? 'dalam 12 bulan terakhir' : `dalam ${days} hari terakhir`), + + perDay: { + visitors: 'Pengunjung per hari', + views: 'Tampilan halaman per hari', + hits: 'Request per hari', + }, + perWeek: { + visitors: 'Pengunjung per minggu', + views: 'Tampilan halaman per minggu', + hits: 'Request per minggu', + }, + showValues: 'Tampilkan nilai', + showChart: 'Tampilkan grafik', + day: 'Hari', + week: 'Minggu', + + nobody: 'Tidak ada yang berkunjung pada hari-hari ini.', + nobodyYet: 'Belum ada yang berkunjung sejak analitik diaktifkan. Kunjungan pertama muncul dalam beberapa menit.', + + pages: 'Halaman', + pagesHint: 'Halaman yang dibuka orang, yang paling sering dibuka di atas.', + requested: 'Paling sering diminta', + requestedHint: 'Semua yang diminta dari aplikasi Anda, menurut alamat - aplikasi ini tidak punya halaman yang dibuka orang.', + sources: 'Asal pengunjung', + sourcesHint: 'Situs yang mengarahkan orang ke sini, menurut halaman tempat mereka berada. Mesin pencari hanya menyebutkan bahwa mereka asalnya, bukan apa yang dicari.', + direct: 'Diketik langsung, bookmark, atau aplikasi', + countries: 'Negara', + countriesHint: 'Tempat alamat internet pengunjung terdaftar - biasanya, tetapi tidak selalu, tempat mereka berada.', + unknownCountry: 'Tidak diketahui', + devices: 'Perangkat', + device: { + desktop: 'Komputer', + mobile: 'Ponsel', + tablet: 'Tablet', + bot: 'Bot', + other: 'Lainnya', + }, + browsers: 'Browser', + systems: 'Sistem', + crawlers: 'Mesin pencari, AI, dan program lain', + crawlersHint: 'Program yang datang berkunjung, menurut jenisnya. Google, Bing, OpenAI, Apple, dan Perplexity hanya disebut namanya jika kunjungan berasal dari alamat yang mereka publikasikan - siapa pun bisa mengaku sebagai mereka.', + noCrawlers: 'Tidak ada mesin pencari, AI, atau program lain yang datang pada hari-hari ini.', + confirmed: (name) => `Berasal dari alamat yang dipublikasikan ${name} untuk crawler-nya`, + crawlerKinds: { + 'search-engine': 'Mesin pencari', + 'ai-crawler': 'Crawler AI', + 'social-preview': 'Pratinjau link di chat dan jejaring sosial', + advertising: 'Iklan', + seo: 'Alat SEO', + monitoring: 'Pemantauan', + 'feed-reader': 'Pembaca feed', + archiver: 'Arsip web', + academic: 'Penelitian', + scanner: 'Pemindai keamanan', + 'browser-automation': 'Browser otomatis', + 'http-library': 'Skrip dan aplikasi', + other: 'Program lainnya', + unconfirmed: 'Mengaku sebagai crawler yang dikenal, tidak terkonfirmasi', + }, + other: 'Lainnya', + showAll: (count) => `Tampilkan semua ${count}`, + showFewer: 'Tampilkan lebih sedikit', + share: (part, whole) => `${part} dari ${whole}`, + none: 'Belum ada.', + + moreKept: (since) => `Dihitung sejak ${since} - Premium menampilkan semuanya, hingga setahun.`, + premiumAsk: 'Tanya soal Premium', + premiumSubject: 'Premium untuk aplikasi saya', + premiumBody: (address) => `Halo,\n\nSaya ingin melihat semua analitik aplikasi saya: ${address}\n\n`, + + footer: (minutes) => + minutes === 1 ? 'Dihitung per hari, dalam UTC. Diperbarui setiap menit.' : `Dihitung per hari, dalam UTC. Diperbarui setiap ${minutes} menit.`, + updated: (time) => `Pembaruan terakhir pukul ${time}.`, + since: (date) => `Dihitung sejak ${date}.`, + + switchOff: 'Nonaktifkan analitik', + switchOffTitle: 'Nonaktifkan analitik?', + switchOffText: 'Penghitungan berhenti, dan semua yang telah dihitung sejauh ini dihapus. Tindakan ini tidak dapat dibatalkan.', + keep: 'Tetap hitung', + switchOffConfirm: 'Nonaktifkan dan hapus', + switchedOff: 'Analitik nonaktif, dan yang telah dihitung dihapus.', + switchOffFailed: 'Analitik tidak dapat dinonaktifkan.', +}; diff --git a/src/Frontend/src/locales/id/editor/frame.tsx b/src/Frontend/src/locales/id/editor/frame.tsx index bc58060..bcb39db 100644 --- a/src/Frontend/src/locales/id/editor/frame.tsx +++ b/src/Frontend/src/locales/id/editor/frame.tsx @@ -16,6 +16,7 @@ export const frame: EditorMessages['frame'] = { history: 'Riwayat', deployments: 'Deployment', stats: 'Statistik', + analytics: 'Analitik', logs: 'Log', code: 'Kode', }, diff --git a/src/Frontend/src/locales/id/editor/index.ts b/src/Frontend/src/locales/id/editor/index.ts index a25fad9..314e467 100644 --- a/src/Frontend/src/locales/id/editor/index.ts +++ b/src/Frontend/src/locales/id/editor/index.ts @@ -1,4 +1,5 @@ import type { EditorMessages } from '../../en/editor'; +import { analytics } from './analytics'; import { change } from './change'; import { clone } from './clone'; import { code } from './code'; @@ -33,6 +34,7 @@ export const editor: EditorMessages = { versions, deployments, stats, + analytics, logs, showcase, openSource, diff --git a/src/Frontend/src/locales/id/guide.tsx b/src/Frontend/src/locales/id/guide.tsx index 8338ace..4a0487e 100644 --- a/src/Frontend/src/locales/id/guide.tsx +++ b/src/Frontend/src/locales/id/guide.tsx @@ -26,6 +26,7 @@ export const guide: Messages['guide'] = { away: 'Membawa kode Anda keluar', git: 'Mengerjakannya dengan git', open: 'Memublikasikan kode', + visitors: 'Siapa yang berkunjung', agents: 'Menyerahkannya ke agen', }, @@ -108,6 +109,7 @@ export const guide: Messages['guide'] = { ['Versi', () => <>Apa yang diubah setiap versi dan apa yang diminta, serta bedanya dengan versi sebelumnya. Deploy atau rollback dari sini, atau mulai draf dari versi mana pun.], ['Deployment', () => <>Apa yang online dan kapan, dan apa yang membuatnya berhenti.], ['Statistik', () => <>Request, kegagalan, waktu respons, dan path yang paling sering diminta, selama satu jam atau satu hari terakhir.], + ['Analitik', () => <>Siapa yang mengunjungi aplikasi, setelah Anda mengaktifkannya: pengunjung, tampilan halaman, dan request per hari, halaman yang mereka buka, situs yang mengarahkan mereka, negara dan perangkat mereka, serta mesin pencari dan AI yang datang.], ['Log', () => <>Request yang masuk, apa yang dicetak lambda, dan stack trace dari setiap error, secara langsung.], [ 'Kode', @@ -131,7 +133,7 @@ export const guide: Messages['guide'] = { ), editorAside: - 'Trafik dan log hanya ada di memori, untuk dipantau, bukan untuk diarsipkan: restart server memulainya dari awal. Versi dan riwayat deployment disimpan permanen.', + 'Trafik dan log hanya ada di memori, untuk dipantau, bukan untuk diarsipkan: restart server memulainya dari awal. Versi, riwayat deployment, dan analitik disimpan permanen.', why: (k) => ( <> @@ -503,6 +505,29 @@ export const guide: Messages['guide'] = { ), openAside: 'Semua yang ada di kode menjadi publik, termasuk versi-versi sebelumnya. Kunci atau kata sandi tempatnya di kunci dan kata sandi di bagian Data, tidak pernah di kode - dipublikasikan atau tidak.', + visitors: ((k) => ( + <> + Buka {k.b('Analitik')} di pusat kontrol dan aktifkan untuk melihat siapa yang mengunjungi aplikasi Anda: berapa + banyak orang yang datang setiap hari, halaman apa yang mereka buka, situs mana yang mengarahkan mereka - mesin + pencari, sebuah postingan, sebuah newsletter - dari negara mana, dengan perangkat dan browser apa, dan mesin + pencari serta asisten AI mana yang membaca halamannya. Pilih salah satu angka di bagian atas untuk melihatnya + hari demi hari. + + )), + visitors2: ((k) => ( + <> + Penghitungan dilakukan oleh platform, di depan aplikasi Anda, sehingga aplikasi Anda tidak memerlukan apa pun + untuk itu: tanpa cookie, tanpa skrip, tanpa banner yang harus disetujui pengunjung Anda. Hanya jumlah per hari + yang disimpan - tidak pernah alamat, dan dari halaman asal seorang pengunjung hanya situsnya. Pengunjung adalah + seseorang yang dihitung sekali sehari: mesin pencari, crawler AI, dan program lain dihitung terpisah menurut + jenisnya, dan Google, Bing, OpenAI, Apple, dan Perplexity hanya disebut namanya jika kunjungan berasal dari + alamat yang mereka publikasikan - siapa pun bisa mengaku sebagai mereka. Data disimpan selama setahun; aplikasi + gratis menampilkan tujuh hari terakhir darinya dan {k.b('Premium')} menampilkan semuanya, termasuk yang dihitung + sebelumnya. Menonaktifkannya menghapus apa yang telah dihitung. + + )), + visitorsAside: + 'Angka-angkanya dicatat setiap beberapa menit, jadi kunjungan terakhir muncul sedikit lebih lambat. Draf tidak pernah dihitung: mencoba perubahan bukanlah kunjungan.', agents: (k) => ( <> diff --git a/src/Frontend/src/locales/id/privacy.tsx b/src/Frontend/src/locales/id/privacy.tsx index de03ea4..eafe6e2 100644 --- a/src/Frontend/src/locales/id/privacy.tsx +++ b/src/Frontend/src/locales/id/privacy.tsx @@ -17,6 +17,11 @@ export const privacy: Messages['privacy'] = { 'Dengan cara inilah gangguan ditemukan, server yang kelebihan beban dilacak ke penyebabnya, dan penyalahgunaan yang dilaporkan kepada kami ditelusuri. Alamat IP juga dipakai, hanya di memori, untuk membatasi berapa banyak permintaan dan build yang bisa dibuat satu pengunjung. Permintaan tidak bisa dijawab tanpa data ini. Dasar hukumnya adalah kepentingan yang sah dari kami untuk menjalankan layanan ini dan menjaganya tetap aman (Pasal 6 ayat (1) huruf f GDPR).', 'Administrator bisa membaca semuanya. Pemilik lambda bisa melihat negara dan browser dari setiap permintaan ke lambda miliknya, tapi tidak bisa melihat alamat IP-nya.', ], + analyticsTitle: 'Kunjungan yang dihitung untuk pemilik lambda', + analytics: [ + "Pemilik sebuah lambda dapat mengaktifkan analitik untuknya. Server kemudian menghitung kunjungan ke lambda itu per hari: berapa banyak request dan tampilan halaman, halaman mana yang diminta, situs asal pengunjung (hanya namanya, bukan alamat halamannya), negara dari alamat IP pengunjung, serta jenis perangkat, browser, dan sistem operasi yang disebutkan browser. Hanya hitungan ini yang disimpan - bukan alamat IP dan bukan hal lain yang menyebut nama pengunjung - dalam database di server, selama setahun. Pemilik lambda melihatnya di editornya, dan administrator dapat membacanya.", + "Untuk menghitung setiap pengunjung sekali sehari, alamat IP dan browser diubah menjadi kode yang tidak dapat dikembalikan, dengan kunci yang dibuat secara acak untuk setiap hari dan setiap lambda. Kode dan kunci itu hanya disimpan di memori, tidak pernah dituliskan, dan dilupakan setelah satu hari. Tidak ada cookie yang dipasang dan tidak ada yang disimpan di browser pengunjung. Jika sebuah kunjungan menyatakan berasal dari crawler Google, Bing, OpenAI, Apple, atau Perplexity, alamat IP-nya dibandingkan, di memori, dengan alamat yang mereka publikasikan untuk crawler mereka. Untuk membedakan crawler dari manusia, server mengambil daftar publik crawler dan alamat yang dipublikasikan tersebut sekali sehari; tidak ada data tentang pengunjung yang dikirim ke mana pun untuk keperluan ini. Dasar hukumnya adalah kepentingan sah pemilik untuk mengetahui bagaimana lambdanya digunakan (Pasal 6 ayat (1) huruf f GDPR).", + ], logsTitle: 'Berapa lama log disimpan', logs: 'Log disimpan di dua tempat: di memori server, yang dikosongkan setiap kali server dimulai ulang, dan di output konsol server, yang dihapus setiap kali server diperbarui. Ukuran keduanya tetap, jadi setiap baris baru menggeser baris yang paling lama. Berapa lama satu baris bertahan tergantung seberapa sibuk situs ini. Tidak ada isi log yang disalin ke arsip.', @@ -43,14 +48,14 @@ export const privacy: Messages['privacy'] = { lambdasTitle: 'Apa yang dilakukan lambda ditentukan pemiliknya', lambdas: - 'Lambda ditulis oleh pemegang link editornya, bukan oleh kami. Apa yang diminta lambda dari pengunjungnya dan apa yang dilakukannya dengan data itu ditentukan oleh mereka, dan halaman ini tidak mencakupnya. Satu-satunya pengecualian adalah log permintaan di atas, yang disimpan server untuk setiap lambda. Ketentuan layanan melarang penggunaan lambda untuk mengumpulkan data pribadi orang lain. Kalau Anda menemukan lambda yang melakukannya, tolong laporkan.', + 'Lambda ditulis oleh pemegang link editornya, bukan oleh kami. Apa yang diminta lambda dari pengunjungnya dan apa yang dilakukannya dengan data itu ditentukan oleh mereka, dan halaman ini tidak mencakupnya. Pengecualiannya adalah log permintaan di atas, yang disimpan server untuk setiap lambda, dan kunjungan yang dihitungnya untuk pemilik yang mengaktifkan analitik. Ketentuan layanan melarang penggunaan lambda untuk mengumpulkan data pribadi orang lain. Kalau Anda menemukan lambda yang melakukannya, tolong laporkan.', mailTitle: 'Kalau Anda menghubungi kami', mail: 'Kalau Anda mengirim email kepada kami, untuk melaporkan penyalahgunaan atau untuk hal lain, kami memakai alamat dan pesan Anda untuk membalas dan menindaklanjuti isinya. Keduanya kami hapus begitu tidak diperlukan lagi untuk itu (Pasal 6 ayat (1) huruf f GDPR).', storageTitle: 'Cookie dan browser Anda', storage: - 'Hanya ada satu cookie, namanya lang. Cookie ini mengingat bahasa yang Anda pilih, supaya alamat yang tidak mencantumkan bahasa terbuka dalam bahasa itu, dan berlaku selama satu tahun. Penyimpanan bawaan browser mengingat mode terang atau gelap, beberapa pengaturan di halaman yang Anda pakai, dan, bagi administrator, token mereka. Tidak ada yang dipakai untuk melacak Anda, dan tidak ada yang dikirim ke pihak lain: tidak ada analitik, tidak ada iklan, dan tidak ada yang dimuat dari situs lain, bahkan font sekalipun. Karena semuanya hanya menjalankan apa yang Anda minta, tidak diperlukan persetujuan (§ 25 ayat (2) angka 2 TDDDG, undang-undang Jerman).', + 'Hanya ada satu cookie, namanya lang. Cookie ini mengingat bahasa yang Anda pilih, supaya alamat yang tidak mencantumkan bahasa terbuka dalam bahasa itu, dan berlaku selama satu tahun. Penyimpanan bawaan browser mengingat mode terang atau gelap, beberapa pengaturan di halaman yang Anda pakai, dan, bagi administrator, token mereka. Tidak ada yang dipakai untuk melacak Anda, dan tidak ada yang dikirim ke pihak lain: tidak ada analitik untuk situs ini, tidak ada iklan, dan tidak ada yang dimuat dari situs lain, bahkan font sekalipun. Karena semuanya hanya menjalankan apa yang Anda minta, tidak diperlukan persetujuan (§ 25 ayat (2) angka 2 TDDDG, undang-undang Jerman).', hostingTitle: 'Tempat semuanya disimpan', hosting: diff --git a/src/Frontend/src/locales/it/editor/analytics.tsx b/src/Frontend/src/locales/it/editor/analytics.tsx new file mode 100644 index 0000000..035f098 --- /dev/null +++ b/src/Frontend/src/locales/it/editor/analytics.tsx @@ -0,0 +1,117 @@ +import type { EditorMessages } from '../../en/editor'; + +export const analytics: EditorMessages['analytics'] = { + hint: 'Contate per te dalla piattaforma: niente cookie, niente codice nella tua app e niente da accettare per i tuoi visitatori.', + loading: 'Lettura delle visite…', + loadFailed: 'Non è stato possibile leggere le visite.', + + offTitle: 'Scopri chi visita la tua app', + offText: + 'Quante persone arrivano ogni giorno, quali pagine aprono, da quali siti arrivano, da quali paesi e con quali dispositivi: contati giorno per giorno, dal momento in cui lo attivi.', + offPrivate: 'Niente cookie e nessun banner che i visitatori debbano accettare', + offAddresses: 'Non si conserva nessun indirizzo, solo quante visite ci sono state ogni giorno', + offKept: (days) => + days >= 365 + ? 'Conservati per un anno e mostrati per intero' + : `Conservati per un anno: vengono mostrati gli ultimi ${days} giorni, tutto con Premium`, + switchOn: 'Attiva Analytics', + switchedOn: 'Analytics è attivo. Le prime visite compaiono entro pochi minuti.', + switchOnFailed: 'Non è stato possibile attivare Analytics.', + demo: 'I visitatori di una demo non vengono contati. Una copia tutta tua può contare i suoi visitatori.', + + range: 'Intervallo di tempo', + days: (days) => (days >= 365 ? '12 mesi' : `${days} giorni`), + premiumRange: 'Premium mostra fino a un anno', + + visitors: 'Visitatori', + visitorsHint: 'Persone diverse, ognuna contata una volta al giorno. I motori di ricerca e gli altri programmi non sono visitatori.', + views: 'Visualizzazioni di pagina', + viewsHint: 'Ogni volta che qualcuno ha aperto una pagina.', + hits: 'Richieste', + hitsHint: 'Tutto ciò che è stato chiesto alla tua app: le sue pagine, e le immagini e gli script che caricano.', + versus: (days) => `rispetto ai ${days} giorni precedenti`, + noChange: 'invariato', + inDays: (days) => (days >= 365 ? 'negli ultimi 12 mesi' : `negli ultimi ${days} giorni`), + + perDay: { + visitors: 'Visitatori al giorno', + views: 'Visualizzazioni di pagina al giorno', + hits: 'Richieste al giorno', + }, + perWeek: { + visitors: 'Visitatori a settimana', + views: 'Visualizzazioni di pagina a settimana', + hits: 'Richieste a settimana', + }, + showValues: 'Mostra i valori', + showChart: 'Mostra il grafico', + day: 'Giorno', + week: 'Settimana del', + + nobody: 'In questi giorni non ha visitato nessuno.', + nobodyYet: 'Nessuno ha visitato da quando Analytics è stato attivato. Le prime visite compaiono entro pochi minuti.', + + pages: 'Pagine', + pagesHint: 'Le pagine aperte dalle persone, le più aperte per prime.', + requested: 'Più richieste', + requestedHint: 'Tutto ciò che è stato chiesto alla tua app, per indirizzo: non ha pagine che le persone aprono.', + sources: 'Da dove arrivano i visitatori', + sourcesHint: 'I siti che hanno mandato qui le persone, per la pagina in cui si trovavano. I motori di ricerca dicono solo di essere stati loro, non cosa è stato cercato.', + direct: 'Digitato, un segnalibro o un’app', + countries: 'Paesi', + countriesHint: 'Dove sono registrati gli indirizzi internet dei visitatori: di solito, non sempre, dove si trovano.', + unknownCountry: 'Sconosciuto', + devices: 'Dispositivi', + device: { + desktop: 'Computer', + mobile: 'Telefono', + tablet: 'Tablet', + bot: 'Bot', + other: 'Altro', + }, + browsers: 'Browser', + systems: 'Sistemi', + crawlers: 'Motori di ricerca, IA e altri programmi', + crawlersHint: 'Programmi passati da qui, divisi per ciò che sono. Google, Bing, OpenAI, Apple e Perplexity vengono nominati solo quando la visita arriva da un indirizzo che pubblicano: chiunque può dichiararsi uno di loro.', + noCrawlers: 'In questi giorni non è passato nessun motore di ricerca, IA o altro programma.', + confirmed: (name) => `Arrivato da un indirizzo che ${name} pubblica per i suoi crawler`, + crawlerKinds: { + 'search-engine': 'Motori di ricerca', + 'ai-crawler': 'Crawler IA', + 'social-preview': 'Anteprime di link nelle chat e nei social network', + advertising: 'Pubblicità', + seo: 'Strumenti SEO', + monitoring: 'Monitoraggio', + 'feed-reader': 'Lettori di feed', + archiver: 'Archivi web', + academic: 'Ricerca', + scanner: 'Scanner di sicurezza', + 'browser-automation': 'Browser automatizzati', + 'http-library': 'Script e app', + other: 'Altri programmi', + unconfirmed: 'Si è dichiarato un crawler noto, ma non è confermato', + }, + other: 'Altri', + showAll: (count) => `Mostra tutti (${count})`, + showFewer: 'Mostra meno', + share: (part, whole) => `${part} su ${whole}`, + none: 'Ancora niente.', + + moreKept: (since) => `Contate dal ${since}: con Premium vedi tutto, fino a un anno.`, + premiumAsk: 'Chiedi di Premium', + premiumSubject: 'Premium per la mia app', + premiumBody: (address) => `Ciao,\n\nvorrei vedere tutte le analisi della mia app: ${address}\n\n`, + + footer: (minutes) => + minutes === 1 ? 'Conteggio al giorno, in UTC. Aggiornato ogni minuto.' : `Conteggio al giorno, in UTC. Aggiornato ogni ${minutes} minuti.`, + updated: (time) => `Ultimo aggiornamento alle ${time}.`, + since: (date) => `Conteggio dal ${date}.`, + + switchOff: 'Disattiva Analytics', + switchOffTitle: 'Disattivare Analytics?', + switchOffText: 'Il conteggio si ferma e tutto ciò che è stato contato finora viene eliminato. Non si può annullare.', + keep: 'Continua a contare', + switchOffConfirm: 'Disattiva ed elimina', + switchedOff: 'Analytics è disattivato e ciò che era stato contato è stato eliminato.', + switchOffFailed: 'Non è stato possibile disattivare Analytics.', +}; diff --git a/src/Frontend/src/locales/it/editor/frame.tsx b/src/Frontend/src/locales/it/editor/frame.tsx index 3cd3088..5b0855d 100644 --- a/src/Frontend/src/locales/it/editor/frame.tsx +++ b/src/Frontend/src/locales/it/editor/frame.tsx @@ -16,6 +16,7 @@ export const frame: EditorMessages['frame'] = { history: 'Cronologia', deployments: 'Deployment', stats: 'Statistiche', + analytics: 'Analytics', logs: 'Log', code: 'Codice', }, diff --git a/src/Frontend/src/locales/it/editor/index.ts b/src/Frontend/src/locales/it/editor/index.ts index 1e72506..cd36d2b 100644 --- a/src/Frontend/src/locales/it/editor/index.ts +++ b/src/Frontend/src/locales/it/editor/index.ts @@ -1,4 +1,5 @@ import type { EditorMessages } from '../../en/editor'; +import { analytics } from './analytics'; import { change } from './change'; import { clone } from './clone'; import { code } from './code'; @@ -33,6 +34,7 @@ export const editor: EditorMessages = { versions, deployments, stats, + analytics, logs, showcase, openSource, diff --git a/src/Frontend/src/locales/it/guide.tsx b/src/Frontend/src/locales/it/guide.tsx index 652477b..414c497 100644 --- a/src/Frontend/src/locales/it/guide.tsx +++ b/src/Frontend/src/locales/it/guide.tsx @@ -26,6 +26,7 @@ export const guide: Messages['guide'] = { away: 'Portarla via', git: 'Lavorare con git', open: 'Pubblicare il codice', + visitors: 'Chi la visita', agents: 'Lasciar fare a un agente', }, @@ -106,6 +107,7 @@ export const guide: Messages['guide'] = { ['Versioni', () => <>Cosa ha cambiato ogni versione, cosa era stato chiesto e le differenze rispetto alla precedente. Da qui fai il deploy o torni indietro, oppure avvii una bozza da una qualsiasi di esse.], ['Deployment', () => <>Cosa è stato online e quando, e cosa l’ha fermato.], ['Statistiche', () => <>Richieste, errori, tempi di risposta e i percorsi più richiesti, nell’ultima ora o nelle ultime 24 ore.], + ['Analytics', () => <>Chi visita l’app, una volta che lo attivi: visitatori, visualizzazioni di pagina e richieste al giorno, le pagine che aprono, i siti che li mandano, i loro paesi e dispositivi, e i motori di ricerca e le IA passati da qui.], ['Log', () => <>Le richieste, cosa ha stampato e lo stack trace di ogni errore, in tempo reale.], [ 'Codice', @@ -129,7 +131,7 @@ export const guide: Messages['guide'] = { ), editorAside: - 'Traffico e log restano in memoria: servono per tenere d’occhio le cose, non per conservarle. Un riavvio del server li azzera. Le versioni e la cronologia dei deployment invece vengono salvate.', + 'Traffico e log restano in memoria: servono per tenere d’occhio le cose, non per conservarle. Un riavvio del server li azzera. Le versioni, la cronologia dei deployment e Analytics invece vengono salvati.', why: (k) => ( <> @@ -495,6 +497,27 @@ export const guide: Messages['guide'] = { ), openAside: 'Tutto quello che c’è nel codice diventa pubblico, comprese le versioni precedenti. Una chiave o una password va tra le chiavi e password in Dati, mai nel codice, che sia pubblicato o no.', + visitors: ((k) => ( + <> + Apri {k.b('Analytics')} nel pannello di controllo e attivalo per vedere chi visita la tua app: quante persone + arrivano ogni giorno, quali pagine aprono, quali siti le mandano (un motore di ricerca, un post, una newsletter), + da quali paesi, con quali dispositivi e browser, e quali motori di ricerca e assistenti IA leggono le sue pagine. + Scegli una cifra in alto per vederla giorno per giorno. + + )), + visitors2: ((k) => ( + <> + Lo conta la piattaforma, davanti alla tua app, quindi alla tua app non serve nulla: niente cookie, niente script, + nessun banner che i visitatori debbano accettare. Si conservano solo i conteggi per giorno: mai un indirizzo e, + della pagina da cui qualcuno è arrivato, solo il sito. Un visitatore è una persona, contata una volta al giorno: i + motori di ricerca, i crawler IA e gli altri programmi vengono contati a parte, per ciò che sono, e Google, Bing, + OpenAI, Apple e Perplexity solo per nome quando la visita arriva da un indirizzo che pubblicano: chiunque può + dichiararsi uno di loro. Si conserva un anno: un’app gratuita ne mostra gli ultimi sette giorni e {k.b('Premium')} + tutto, anche ciò che è stato contato prima. Disattivandolo, ciò che è stato contato viene eliminato. + + )), + visitorsAside: + 'Le cifre vengono registrate ogni pochi minuti, quindi le ultime visite compaiono con un po’ di ritardo. Una bozza non viene mai contata: provare una modifica non è una visita.', agents: (k) => ( <> diff --git a/src/Frontend/src/locales/it/privacy.tsx b/src/Frontend/src/locales/it/privacy.tsx index b926510..eae9829 100644 --- a/src/Frontend/src/locales/it/privacy.tsx +++ b/src/Frontend/src/locales/it/privacy.tsx @@ -17,6 +17,11 @@ export const privacy: Messages['privacy'] = { 'Serve a trovare i problemi, a capire cosa sta sovraccaricando il server e a rintracciare gli abusi che ci vengono segnalati. L’indirizzo IP viene usato anche, solo in memoria, per limitare quante richieste può fare e quante app può creare un singolo visitatore. Senza questi dati non si può rispondere a una richiesta. La base giuridica è il nostro legittimo interesse a gestire il servizio e a mantenerlo sicuro (art. 6, par. 1, lett. f) GDPR).', 'Gli amministratori possono vedere tutto. Il proprietario di una lambda vede il paese e il browser di ogni richiesta alla sua lambda, ma non l’indirizzo IP.', ], + analyticsTitle: 'Visite contate per il proprietario di una lambda', + analytics: [ + 'Il proprietario di una lambda può attivare Analytics per quella lambda. Da quel momento il server conta le visite alla lambda giorno per giorno: quante richieste e visualizzazioni di pagina ci sono state, per quali pagine, il sito da cui è arrivato un visitatore (solo il nome, non l’indirizzo della pagina), il paese dell’indirizzo IP del visitatore e il tipo di dispositivo, browser e sistema operativo indicato dal browser. Si conservano solo questi conteggi, non l’indirizzo IP né altro che identifichi un visitatore, in un database sul server, per un anno. Il proprietario della lambda li vede nel suo editor e gli amministratori possono leggerli.', + 'Per contare ogni visitatore una volta al giorno, l’indirizzo IP e il browser vengono trasformati in un codice che non si può invertire, con una chiave generata a caso per ogni giorno e per ogni lambda. I codici e le chiavi restano solo in memoria, non vengono mai scritti e dopo un giorno vengono dimenticati. Non viene impostato alcun cookie e nel browser del visitatore non viene memorizzato nulla. Quando una visita dichiara di provenire dal crawler di Google, Bing, OpenAI, Apple o Perplexity, il suo indirizzo IP viene confrontato, in memoria, con gli indirizzi che questi pubblicano per i propri crawler. Per distinguere i crawler dalle persone, il server scarica una volta al giorno un elenco pubblico di crawler e questi indirizzi pubblicati; per questo non viene inviato altrove nulla sui visitatori. La base giuridica è il legittimo interesse del proprietario a sapere come viene usata la sua lambda (art. 6, par. 1, lett. f GDPR).', + ], logsTitle: 'Per quanto tempo si conserva il log', logs: 'Il log si trova in due posti: nella memoria del server, che si svuota a ogni riavvio, e nell’output della console del server, che viene cancellato a ogni aggiornamento. Entrambi hanno una dimensione fissa, quindi ogni nuova riga spinge fuori la più vecchia, e quanto dura una riga dipende da quanto è trafficato il sito. Nessuna parte del log viene archiviata.', @@ -43,14 +48,14 @@ export const privacy: Messages['privacy'] = { lambdasTitle: 'Cosa fa una lambda lo decide il suo proprietario', lambdas: - 'Una lambda la scrive chi ha il suo link di modifica, non noi. Cosa chiede ai suoi visitatori e cosa ne fa lo decide quella persona, e questa pagina non lo copre, a parte il log delle richieste descritto sopra, che il server tiene per ogni lambda. I termini di servizio non consentono di usare una lambda per raccogliere dati personali di altre persone: se ne trovi una che lo fa, segnalacela.', + 'Una lambda la scrive chi ha il suo link di modifica, non noi. Cosa chiede ai suoi visitatori e cosa ne fa lo decide quella persona, e questa pagina non lo copre, a parte il log delle richieste descritto sopra, che il server tiene per ogni lambda, e le visite che conta per il proprietario che ha attivato Analytics. I termini di servizio non consentono di usare una lambda per raccogliere dati personali di altre persone: se ne trovi una che lo fa, segnalacela.', mailTitle: 'Quando ci scrivi', mail: 'Se ci scrivi, per segnalare un abuso o per qualsiasi altro motivo, usiamo il tuo indirizzo email e il tuo messaggio per risponderti e occuparci di quello che ci hai scritto, e li cancelliamo quando non servono più a questo (art. 6, par. 1, lett. f) GDPR).', storageTitle: 'I cookie e il tuo browser', storage: - 'C’è un solo cookie, che si chiama lang. Ricorda la lingua che hai scelto, così gli indirizzi che non ne indicano una si aprono in quella lingua, e dura un anno. La memoria locale del browser ricorda il tema chiaro o scuro, alcune impostazioni delle pagine che usi e, per gli amministratori, il loro token. Niente di tutto questo serve a seguirti e niente va a qualcun altro: niente strumenti di analisi, niente pubblicità, e niente viene caricato da altri siti, nemmeno i font. Visto che tutto serve solo a fare quello che hai chiesto tu, non serve il consenso (§ 25, comma 2, n. 2 della legge tedesca TDDDG).', + 'C’è un solo cookie, che si chiama lang. Ricorda la lingua che hai scelto, così gli indirizzi che non ne indicano una si aprono in quella lingua, e dura un anno. La memoria locale del browser ricorda il tema chiaro o scuro, alcune impostazioni delle pagine che usi e, per gli amministratori, il loro token. Niente di tutto questo serve a seguirti e niente va a qualcun altro: nessuno strumento di analisi di questo sito, niente pubblicità, e niente viene caricato da altri siti, nemmeno i font. Visto che tutto serve solo a fare quello che hai chiesto tu, non serve il consenso (§ 25, comma 2, n. 2 della legge tedesca TDDDG).', hostingTitle: 'Dove sono conservati i dati', hosting: diff --git a/src/Frontend/src/locales/ja/editor/analytics.tsx b/src/Frontend/src/locales/ja/editor/analytics.tsx new file mode 100644 index 0000000..c052204 --- /dev/null +++ b/src/Frontend/src/locales/ja/editor/analytics.tsx @@ -0,0 +1,128 @@ +import type { EditorMessages } from '../../en/editor'; + +export const analytics: EditorMessages['analytics'] = { + /** Next to the section's title, on hover. */ + hint: 'プラットフォームが代わりに数えています。Cookieも、アプリへのコードの追加も、訪問者の同意も必要ありません。', + loading: '訪問数を読み込んでいます…', + loadFailed: '訪問数を読み込めませんでした。', + + /** The page shown while analytics are off. */ + offTitle: 'アプリを訪れた人を確認しましょう', + offText: + '毎日どれくらいの人が訪れるか、どのページを開くか、どのサイトから来るか、どの国から、どのデバイスで来るか。オンにした時点から、日ごとに集計します。', + offPrivate: 'Cookieも、訪問者が同意する必要のあるバナーもありません', + offAddresses: 'IPアドレスは保存せず、日ごとの訪問数だけを保存します', + offKept: (days) => + days >= 365 + ? '1年間保存され、そのすべてを表示します' + : `1年間保存されます。表示するのは直近${days}日間で、すべてを見るにはプレミアムが必要です`, + switchOn: 'アクセス解析をオンにする', + switchedOn: 'アクセス解析をオンにしました。最初の訪問は数分以内に表示されます。', + switchOnFailed: 'アクセス解析をオンにできませんでした。', + demo: 'デモの訪問者は数えません。ご自身のコピーなら訪問者を数えられます。', + + range: '期間', + days: (days) => (days >= 365 ? '12か月' : `${days}日間`), + /** On a range the tier does not show, on hover. */ + premiumRange: 'プレミアムなら最大1年分を表示します', + + /** The figures at the top; each one picks what the chart below shows. */ + visitors: '訪問者', + visitorsHint: 'ユニークな訪問者数です。同じ人は1日に1回だけ数えます。検索エンジンやその他のプログラムは訪問者に含めません。', + views: 'ページビュー', + viewsHint: 'ページが開かれた回数です。', + hits: 'リクエスト', + hitsHint: 'アプリに寄せられたすべてのリクエストです。ページに加え、ページが読み込む画像やスクリプトも含みます。', + /** Under a figure: how it changed against the days before the ones shown. */ + versus: (days) => `その前の${days}日間と比較`, + noChange: '変化なし', + /** Under a figure when there is nothing to compare with. */ + inDays: (days) => (days >= 365 ? '直近12か月' : `直近${days}日間`), + + /** Above the chart, for the figure picked. */ + perDay: { + visitors: '1日あたりの訪問者', + views: '1日あたりのページビュー', + hits: '1日あたりのリクエスト', + }, + perWeek: { + visitors: '1週間あたりの訪問者', + views: '1週間あたりのページビュー', + hits: '1週間あたりのリクエスト', + }, + showValues: '数値を表示', + showChart: 'グラフを表示', + day: '日', + week: '週の開始日', + + nobody: 'この期間の訪問者はいません。', + nobodyYet: 'アクセス解析をオンにしてから、まだ訪問者はいません。最初の訪問は数分以内に表示されます。', + + pages: 'ページ', + pagesHint: '開かれたページを、多い順に表示します。', + requested: 'リクエストの多いもの', + requestedHint: 'アプリに寄せられたすべてのリクエストを、パスごとに表示します。人が開くページはありません。', + sources: '訪問元', + sourcesHint: '訪問者を送ってきたサイトを、元のページごとに表示します。検索エンジンについては、検索エンジンから来たことだけがわかり、検索語はわかりません。', + direct: '直接入力、ブックマーク、またはアプリ', + countries: '国', + countriesHint: '訪問者のIPアドレスが登録されている国です。多くの場合、実際にいる国と一致しますが、そうとは限りません。', + unknownCountry: '不明', + devices: 'デバイス', + device: { + desktop: 'パソコン', + mobile: 'スマートフォン', + tablet: 'タブレット', + bot: 'ボット', + other: 'その他', + }, + browsers: 'ブラウザ', + systems: 'OS', + crawlers: '検索エンジン、AI、その他のプログラム', + crawlersHint: '訪れたプログラムを、種類別に示します。Google、Bing、OpenAI、Apple、Perplexityは、それぞれが公開しているアドレスからの訪問だった場合にのみ名前を表示します。誰でもこれらを名乗れるためです。', + noCrawlers: 'この期間に検索エンジン、AI、その他のプログラムは訪れていません。', + confirmed: (name) => `${name}がクローラー用に公開しているアドレスからの訪問です`, + crawlerKinds: { + 'search-engine': '検索エンジン', + 'ai-crawler': 'AIクローラー', + 'social-preview': 'チャットやSNSでのリンクプレビュー', + advertising: '広告', + seo: 'SEOツール', + monitoring: '監視', + 'feed-reader': 'フィードリーダー', + archiver: 'ウェブアーカイブ', + academic: '研究', + scanner: 'セキュリティスキャナー', + 'browser-automation': '自動化されたブラウザ', + 'http-library': 'スクリプトとアプリ', + other: 'その他のプログラム', + unconfirmed: '既知のクローラーを名乗っていますが、確認できませんでした', + }, + /** What did not fit in a day's list, gathered in one row. */ + other: 'その他', + showAll: (count) => `${count}件すべてを表示`, + showFewer: '表示を減らす', + /** Next to a share in a list, on hover. */ + share: (part, whole) => `${whole}のうち${part}`, + none: 'まだありません。', + + /** For a free app that counted more than it shows. */ + moreKept: (since) => `${since}から集計しています。プレミアムなら最大1年分をすべて表示します。`, + premiumAsk: 'プレミアムについて問い合わせる', + premiumSubject: '私のアプリのプレミアムについて', + premiumBody: (address) => `お世話になっております。\n\n私のアプリのアクセス解析をすべて見たいと考えています：${address}\n\n`, + + /** At the foot of the page. */ + footer: (minutes) => + minutes === 1 ? '日ごとにUTCで集計しています。1分ごとに更新されます。' : `日ごとにUTCで集計しています。${minutes}分ごとに更新されます。`, + updated: (time) => `最終更新：${time}`, + since: (date) => `${date}から集計しています。`, + + switchOff: 'アクセス解析をオフにする', + switchOffTitle: 'アクセス解析をオフにしますか？', + switchOffText: '集計を停止し、これまでに集計したものはすべて削除されます。この操作は元に戻せません。', + keep: '集計を続ける', + switchOffConfirm: 'オフにして削除する', + switchedOff: 'アクセス解析をオフにし、集計したものを削除しました。', + switchOffFailed: 'アクセス解析をオフにできませんでした。', +}; diff --git a/src/Frontend/src/locales/ja/editor/frame.tsx b/src/Frontend/src/locales/ja/editor/frame.tsx index 78495e6..c40c383 100644 --- a/src/Frontend/src/locales/ja/editor/frame.tsx +++ b/src/Frontend/src/locales/ja/editor/frame.tsx @@ -16,6 +16,7 @@ export const frame: EditorMessages['frame'] = { history: '履歴', deployments: 'デプロイ履歴', stats: '統計', + analytics: 'アクセス解析', logs: 'ログ', code: 'コード', }, diff --git a/src/Frontend/src/locales/ja/editor/index.ts b/src/Frontend/src/locales/ja/editor/index.ts index 54217ee..72d9c6c 100644 --- a/src/Frontend/src/locales/ja/editor/index.ts +++ b/src/Frontend/src/locales/ja/editor/index.ts @@ -1,4 +1,5 @@ import type { EditorMessages } from '../../en/editor'; +import { analytics } from './analytics'; import { change } from './change'; import { clone } from './clone'; import { code } from './code'; @@ -39,6 +40,7 @@ export const editor: EditorMessages = { versions, deployments, stats, + analytics, logs, showcase, openSource, diff --git a/src/Frontend/src/locales/ja/guide.tsx b/src/Frontend/src/locales/ja/guide.tsx index 18573c6..426b8c8 100644 --- a/src/Frontend/src/locales/ja/guide.tsx +++ b/src/Frontend/src/locales/ja/guide.tsx @@ -31,6 +31,7 @@ export const guide: Messages['guide'] = { away: 'まるごと持ち出す', git: 'gitで作業する', open: 'コードを公開する', + visitors: '訪問者を確認する', agents: 'エージェントに任せる', }, @@ -107,6 +108,7 @@ export const guide: Messages['guide'] = { ['バージョン', () => <>各バージョンの変更点と依頼内容、前のバージョンとの差分。ここからデプロイやロールバックができ、どのバージョンからでも下書きを作れます。], ['デプロイ履歴', () => <>いつ何がオンラインだったか、何が原因で止まったか。], ['統計', () => <>直近1時間または24時間のリクエスト数、失敗数、応答時間、よくアクセスされるパス。], + ['アクセス解析', () => <>オンにすると、アプリを訪れた人がわかります。1日ごとの訪問者数、ページビュー数、リクエスト数に加え、開かれたページ、訪問元のサイト、国とデバイス、訪れた検索エンジンとAIも確認できます。], ['ログ', () => <>リクエスト、出力された内容、エラーのスタックトレースをリアルタイムで。], [ 'コード', @@ -126,7 +128,7 @@ export const guide: Messages['guide'] = { ), editorAside: - 'トラフィックとログはメモリ上にあり、保存するためではなく、様子を見るためのものです。サーバーが再起動するとリセットされます。バージョンとデプロイ履歴は保存されます。', + 'トラフィックとログはメモリ上にあり、保存するためではなく、様子を見るためのものです。サーバーが再起動するとリセットされます。バージョン、デプロイ履歴、アクセス解析は保存されます。', why: (k) => ( <> @@ -462,6 +464,18 @@ export const guide: Messages['guide'] = { ), openAside: 'コードに書かれているものは、以前のバージョンも含めてすべて公開されます。キーやパスワードは、公開するかどうかにかかわらずコードには書かず、「データ」のキーとパスワードに入れてください。', + visitors: ((k) => ( + <> + 管理画面で{k.b('アクセス解析')}を開いてオンにすると、アプリを訪れた人がわかります。毎日どれくらいの人が来るか、どのページを開くか、どのサイト（検索エンジン、投稿、ニュースレターなど）から来るか、どの国から、どのデバイスやブラウザで来るか、そしてどの検索エンジンやAIアシスタントがページを読んだかを確認できます。上部の数値を選ぶと、日ごとに見られます。 + + )), + visitors2: ((k) => ( + <> + 集計はアプリの手前でプラットフォームが行うので、アプリ側で必要なことはありません。Cookieも、スクリプトも、訪問者が同意するバナーも不要です。保存するのは日ごとの数だけで、IPアドレスは保存せず、訪問元のページについてもサイトだけを記録します。訪問者とは人のことで、1日に1回だけ数えます。検索エンジン、AIクローラー、その他のプログラムは、種類別に分けて別に集計します。Google、Bing、OpenAI、Apple、Perplexityは、それぞれが公開しているアドレスからの訪問だった場合にのみ名前で表示します。誰でもこれらを名乗れるためです。保存期間は1年で、無料のアプリでは直近7日間を表示し、{k.b('プレミアム')}ならそれ以前に集計した分も含めてすべて表示します。オフにすると、集計したものは削除されます。 + + )), + visitorsAside: + '数値は数分おきに書き込まれるため、直近の訪問は少し遅れて表示されます。下書きは数えません。変更を試すことは訪問ではないためです。', agents: (k) => ( <> diff --git a/src/Frontend/src/locales/ja/privacy.tsx b/src/Frontend/src/locales/ja/privacy.tsx index 443bd41..6c7d29d 100644 --- a/src/Frontend/src/locales/ja/privacy.tsx +++ b/src/Frontend/src/locales/ja/privacy.tsx @@ -15,6 +15,11 @@ export const privacy: Messages['privacy'] = { 'これらの記録は、障害を見つけるため、サーバーが過負荷になったときに原因を突き止めるため、そして報告された不正利用を追跡するために使います。また、1人の訪問者が送れるリクエストの数と、作成を依頼できる回数を制限するためにも、IPアドレスを使います（この用途では、メモリ上に置くだけです）。これらの情報がなければ、リクエストに応答できません。法的根拠は、サービスを運営し安全に保つという私たちの正当な利益です（GDPR第6条1項(f)）。', '管理者はすべてを閲覧できます。lambdaの所有者は、自分のlambdaへの各リクエストの国とブラウザを見られますが、IPアドレスは見られません。', ], + analyticsTitle: 'lambdaの所有者のために数える訪問', + analytics: [ + 'lambdaの所有者は、そのlambdaのアクセス解析をオンにできます。オンにすると、サーバーはそのlambdaへの訪問を日ごとに数えます。数えるのは、リクエストとページビューの数、どのページへのものか、訪問者が来たサイト（サイト名のみで、ページのアドレスは含みません）、訪問者のIPアドレスの国、そしてブラウザが名乗るデバイス、ブラウザ、オペレーティングシステムの種類です。保存するのはこれらの数だけで、IPアドレスや訪問者を特定できるものは保存しません。数はサーバー上のデータベースに1年間保存されます。lambdaの所有者は編集画面で確認でき、管理者も読むことができます。', + '訪問者を1日に1回だけ数えるために、IPアドレスとブラウザは、lambdaごと・日ごとにランダムに作られる鍵を使って、元に戻せないコードに変換されます。コードと鍵はメモリ上にのみ保持され、書き込まれることはなく、1日たつと破棄されます。Cookieは設定せず、訪問者のブラウザには何も保存しません。訪問がGoogle、Bing、OpenAI、Apple、Perplexityのクローラーからのものだと名乗っている場合は、そのIPアドレスを、これらが自社のクローラー用に公開しているアドレスとメモリ上で照合します。クローラーと人を見分けるために、サーバーは公開されているクローラーの一覧と、これらの公開アドレスを1日に1回取得します。そのために訪問者に関する情報がどこかへ送られることはありません。法的根拠は、lambdaがどのように利用されているかを知るという所有者の正当な利益です（GDPR第6条第1項(f)）。', + ], logsTitle: 'ログの保存期間', logs: 'ログは2か所に保存されます。サーバーのメモリ（サーバーが再起動するたびに消去されます）と、サーバーのコンソール出力（サーバーが更新されるたびに削除されます）です。どちらも容量が決まっているため、新しい行が書き込まれるたびに最も古い行が押し出されます。そのため、1行が残る期間はサイトの混み具合によって変わります。ログの内容をアーカイブにコピーすることはありません。', @@ -36,14 +41,14 @@ export const privacy: Messages['privacy'] = { lambdasTitle: 'lambdaが何をするかは所有者しだい', lambdas: - 'lambdaを書くのは、私たちではなく、その編集用リンクを持っている人です。lambdaが訪問者に何を求め、それをどう扱うかはその人が決めることで、このページの対象外です。ただし、上で説明したリクエストのログは例外で、サーバーがすべてのlambdaについて記録しています。利用規約では、lambdaを使って他人の個人データを集めることを禁止しています。そうしたlambdaを見つけたら、お知らせください。', + 'lambdaを書くのは、私たちではなく、その編集用リンクを持っている人です。lambdaが訪問者に何を求め、それをどう扱うかはその人が決めることで、このページの対象外です。ただし、上で説明したリクエストのログ（サーバーがすべてのlambdaについて記録しています）と、アクセス解析をオンにした所有者のために数えている訪問は例外です。利用規約では、lambdaを使って他人の個人データを集めることを禁止しています。そうしたlambdaを見つけたら、お知らせください。', mailTitle: '私たちに連絡したとき', mail: '不正利用の報告でもそれ以外の用件でも、私たちに連絡をいただいた場合は、返信と用件への対応のために、あなたのメールアドレスとメッセージを使います。その目的に必要なくなったら削除します（GDPR第6条1項(f)）。', storageTitle: 'Cookieとブラウザ', storage: - 'Cookieは「lang」の1つだけです。選んだ言語を覚えておき、言語の入っていないURLをその言語で開くためのもので、有効期間は1年です。ブラウザ自体のストレージには、ライトモードかダークモードか、使っているページのいくつかの設定、そして管理者の場合はそのトークンが保存されます。どれもあなたを追跡するためには使わず、ほかの誰にも渡しません。アクセス解析も広告もなく、ほかのサイトから読み込むものもありません。フォントさえもです。どれもあなたが求めたことをするためだけのものなので、同意は必要ありません（ドイツの法律であるTDDDG第25条2項2号）。', + 'Cookieは「lang」の1つだけです。選んだ言語を覚えておき、言語の入っていないURLをその言語で開くためのもので、有効期間は1年です。ブラウザ自体のストレージには、ライトモードかダークモードか、使っているページのいくつかの設定、そして管理者の場合はそのトークンが保存されます。どれもあなたを追跡するためには使わず、ほかの誰にも渡しません。このサイト自体のアクセス解析も広告もなく、ほかのサイトから読み込むものもありません。フォントさえもです。どれもあなたが求めたことをするためだけのものなので、同意は必要ありません（ドイツの法律であるTDDDG第25条2項2号）。', hostingTitle: '保存場所', hosting: diff --git a/src/Frontend/src/locales/ko/editor/analytics.tsx b/src/Frontend/src/locales/ko/editor/analytics.tsx new file mode 100644 index 0000000..ca49d5b --- /dev/null +++ b/src/Frontend/src/locales/ko/editor/analytics.tsx @@ -0,0 +1,117 @@ +import type { EditorMessages } from '../../en/editor'; + +export const analytics: EditorMessages['analytics'] = { + hint: '플랫폼이 대신 집계해요. 쿠키도, 앱에 넣을 코드도 필요 없고, 방문자가 동의할 것도 없어요.', + loading: '방문 기록을 불러오는 중…', + loadFailed: '방문 기록을 불러오지 못했어요.', + + offTitle: '내 앱에 누가 방문하는지 확인하세요', + offText: + '하루에 몇 명이 오는지, 어떤 페이지를 여는지, 어떤 사이트에서 오는지, 어느 나라에서 어떤 기기로 오는지를 켠 순간부터 하루 단위로 집계해요.', + offPrivate: '쿠키도 없고, 방문자가 동의해야 하는 배너도 없어요', + offAddresses: '주소는 저장하지 않고, 하루에 방문이 몇 번이었는지만 기록해요', + offKept: (days) => + days >= 365 + ? '1년 동안 보관하고, 전부 보여 드려요' + : `1년 동안 보관하며, 최근 ${days}일만 보여 드려요. 전체는 프리미엄에서 볼 수 있어요`, + switchOn: '분석 켜기', + switchedOn: '분석을 켰어요. 첫 방문은 몇 분 안에 나타나요.', + switchOnFailed: '분석을 켜지 못했어요.', + demo: '데모의 방문자는 집계하지 않아요. 직접 만든 복사본은 방문자를 집계할 수 있어요.', + + range: '기간', + days: (days) => (days >= 365 ? '12개월' : `${days}일`), + premiumRange: '프리미엄에서는 최대 1년까지 볼 수 있어요', + + visitors: '방문자', + visitorsHint: '서로 다른 사람을 하루에 한 번씩 세어요. 검색 엔진과 그 밖의 프로그램은 방문자가 아니에요.', + views: '페이지 조회수', + viewsHint: '누군가 페이지를 열 때마다 세어요.', + hits: '요청', + hitsHint: '앱이 받은 모든 요청이에요. 페이지와, 페이지가 불러오는 그림과 스크립트를 포함해요.', + versus: (days) => `직전 ${days}일 대비`, + noChange: '변화 없음', + inDays: (days) => (days >= 365 ? '최근 12개월 동안' : `최근 ${days}일 동안`), + + perDay: { + visitors: '일별 방문자', + views: '일별 페이지 조회수', + hits: '일별 요청', + }, + perWeek: { + visitors: '주별 방문자', + views: '주별 페이지 조회수', + hits: '주별 요청', + }, + showValues: '값 보기', + showChart: '차트 보기', + day: '날짜', + week: '주 시작일', + + nobody: '이 기간에는 방문자가 없었어요.', + nobodyYet: '분석을 켠 뒤로 방문자가 없어요. 첫 방문은 몇 분 안에 나타나요.', + + pages: '페이지', + pagesHint: '사람들이 연 페이지예요. 가장 많이 열린 것부터 보여 줘요.', + requested: '가장 많이 요청된 것', + requestedHint: '앱이 받은 모든 요청을 주소별로 보여 줘요. 이 앱에는 사람들이 여는 페이지가 없어요.', + sources: '방문 경로', + sourcesHint: '사람들을 보낸 사이트를, 방문자가 있던 페이지별로 보여 줘요. 검색 엔진은 검색 엔진에서 왔다는 것만 알려 주고, 검색어는 알려 주지 않아요.', + direct: '직접 입력, 북마크 또는 앱', + countries: '국가', + countriesHint: '방문자의 인터넷 주소가 등록된 곳이에요. 대개 실제 위치와 같지만, 항상 그런 것은 아니에요.', + unknownCountry: '알 수 없음', + devices: '기기', + device: { + desktop: '컴퓨터', + mobile: '휴대폰', + tablet: '태블릿', + bot: '봇', + other: '기타', + }, + browsers: '브라우저', + systems: '운영체제', + crawlers: '검색 엔진, AI 및 기타 프로그램', + crawlersHint: '찾아온 프로그램을 종류별로 보여 줘요. Google, Bing, OpenAI, Apple, Perplexity는 해당 업체가 공개한 주소에서 방문했을 때만 이름으로 표시해요. 누구나 이들을 사칭할 수 있기 때문이에요.', + noCrawlers: '이 기간에는 검색 엔진, AI, 그 밖의 프로그램이 찾아오지 않았어요.', + confirmed: (name) => `${name}의 크롤러용 공개 주소에서 왔어요`, + crawlerKinds: { + 'search-engine': '검색 엔진', + 'ai-crawler': 'AI 크롤러', + 'social-preview': '채팅과 소셜 네트워크의 링크 미리보기', + advertising: '광고', + seo: 'SEO 도구', + monitoring: '모니터링', + 'feed-reader': '피드 리더', + archiver: '웹 아카이브', + academic: '연구', + scanner: '보안 스캐너', + 'browser-automation': '자동화된 브라우저', + 'http-library': '스크립트와 앱', + other: '기타 프로그램', + unconfirmed: '알려진 크롤러라고 주장했지만 확인되지 않음', + }, + other: '그 외', + showAll: (count) => `${count}개 모두 보기`, + showFewer: '적게 보기', + share: (part, whole) => `${whole} 중 ${part}`, + none: '아직 없어요.', + + moreKept: (since) => `${since}부터 집계했어요. 프리미엄에서는 최대 1년까지 전부 볼 수 있어요.`, + premiumAsk: '프리미엄 문의', + premiumSubject: '내 앱의 프리미엄', + premiumBody: (address) => `안녕하세요.\n\n제 앱의 분석을 모두 보고 싶습니다: ${address}\n\n`, + + footer: (minutes) => + minutes === 1 ? '하루 단위로, UTC 기준으로 집계해요. 1분마다 갱신돼요.' : `하루 단위로, UTC 기준으로 집계해요. ${minutes}분마다 갱신돼요.`, + updated: (time) => `마지막 갱신: ${time}.`, + since: (date) => `${date}부터 집계 중이에요.`, + + switchOff: '분석 끄기', + switchOffTitle: '분석을 끌까요?', + switchOffText: '집계가 멈추고, 지금까지 집계한 내용이 모두 삭제돼요. 되돌릴 수 없어요.', + keep: '계속 집계', + switchOffConfirm: '끄고 삭제', + switchedOff: '분석을 껐고, 집계한 내용을 삭제했어요.', + switchOffFailed: '분석을 끄지 못했어요.', +}; diff --git a/src/Frontend/src/locales/ko/editor/frame.tsx b/src/Frontend/src/locales/ko/editor/frame.tsx index 84cc599..f2c3c3a 100644 --- a/src/Frontend/src/locales/ko/editor/frame.tsx +++ b/src/Frontend/src/locales/ko/editor/frame.tsx @@ -16,6 +16,7 @@ export const frame: EditorMessages['frame'] = { history: '기록', deployments: '배포 기록', stats: '통계', + analytics: '분석', logs: '로그', code: '코드', }, diff --git a/src/Frontend/src/locales/ko/editor/index.ts b/src/Frontend/src/locales/ko/editor/index.ts index 0079c4e..69e5734 100644 --- a/src/Frontend/src/locales/ko/editor/index.ts +++ b/src/Frontend/src/locales/ko/editor/index.ts @@ -1,4 +1,5 @@ import type { EditorMessages } from '../../en/editor'; +import { analytics } from './analytics'; import { change } from './change'; import { clone } from './clone'; import { code } from './code'; @@ -33,6 +34,7 @@ export const editor: EditorMessages = { versions, deployments, stats, + analytics, logs, showcase, openSource, diff --git a/src/Frontend/src/locales/ko/guide.tsx b/src/Frontend/src/locales/ko/guide.tsx index 21cfeee..84016c6 100644 --- a/src/Frontend/src/locales/ko/guide.tsx +++ b/src/Frontend/src/locales/ko/guide.tsx @@ -26,6 +26,7 @@ export const guide: Messages['guide'] = { away: '가지고 나가기', git: 'git으로 작업하기', open: '코드 공개하기', + visitors: '방문자 확인하기', agents: '에이전트에게 맡기기', }, @@ -104,6 +105,7 @@ export const guide: Messages['guide'] = { ['버전', () => <>버전마다 무엇이 바뀌었고 무엇을 요청했는지, 그리고 이전 버전과의 차이. 여기서 배포하거나 롤백하고, 어느 버전에서든 초안을 만들 수 있어요.], ['배포 기록', () => <>언제 무엇이 온라인이었는지, 무엇 때문에 내려갔는지.], ['통계', () => <>최근 1시간 또는 하루 동안의 요청, 실패, 응답 시간, 가장 많이 요청된 경로.], + ['분석', () => <>켜 두면 앱에 누가 방문하는지 보여 줘요. 하루 단위의 방문자, 페이지 조회수, 요청 수와 방문자가 여는 페이지, 방문자를 보낸 사이트, 국가와 기기, 찾아온 검색 엔진과 AI까지.], ['로그', () => <>요청, 람다가 출력한 내용, 문제가 생긴 곳의 스택 트레이스를 실시간으로.], [ '코드', @@ -127,7 +129,7 @@ export const guide: Messages['guide'] = { ), editorAside: - '트래픽과 로그는 보관용이 아니라 지켜보기용이라 메모리에만 있어요. 서버가 재시작되면 처음부터 다시 쌓여요. 버전과 배포 기록은 저장돼요.', + '트래픽과 로그는 보관용이 아니라 지켜보기용이라 메모리에만 있어요. 서버가 재시작되면 처음부터 다시 쌓여요. 버전, 배포 기록, 분석은 저장돼요.', why: (k) => ( <> @@ -480,6 +482,26 @@ export const guide: Messages['guide'] = { ), openAside: '코드에 있는 모든 것이 이전 버전까지 포함해 공개돼요. 키나 비밀번호는 공개 여부와 상관없이 코드에 넣지 말고, ‘데이터’의 키와 비밀번호에 두세요.', + visitors: ((k) => ( + <> + 관리 화면에서 {k.b('분석')}을 열고 켜면 앱에 누가 방문하는지 볼 수 있어요. 하루에 몇 명이 오는지, 어떤 + 페이지를 여는지, 검색 엔진이나 게시글, 뉴스레터처럼 어떤 사이트에서 오는지, 어느 나라에서 어떤 기기와 + 브라우저로 오는지, 그리고 어떤 검색 엔진과 AI 어시스턴트가 페이지를 읽어 가는지 알려 줘요. + 맨 위에서 수치를 고르면 날짜별로 볼 수 있어요. + + )), + visitors2: ((k) => ( + <> + 집계는 플랫폼이 앱 앞에서 하기 때문에 앱에는 아무것도 필요 없어요. 쿠키도, 스크립트도, 방문자가 동의할 + 배너도 없어요. 하루 단위의 횟수만 저장하고, 주소는 저장하지 않으며, 방문자가 어디에서 왔는지도 사이트만 + 남겨요. 방문자는 한 사람을 뜻하고 하루에 한 번만 세어요. 검색 엔진, AI 크롤러, 그 밖의 프로그램은 종류별로 + 따로 세고, Google, Bing, OpenAI, Apple, Perplexity는 해당 업체가 공개한 주소에서 방문했을 때만 이름으로 + 표시해요. 누구나 이들을 사칭할 수 있기 때문이에요. 1년 치를 보관하고, 무료 앱은 그중 최근 7일을, + {k.b('프리미엄')}은 그 전에 집계한 것까지 전부 보여 줘요. 끄면 집계한 내용이 삭제돼요. + + )), + visitorsAside: + '수치는 몇 분마다 기록되기 때문에 가장 최근 방문은 조금 늦게 나타나요. 초안은 집계하지 않아요. 변경을 시험하는 것은 방문이 아니기 때문이에요.', agents: (k) => ( <> diff --git a/src/Frontend/src/locales/ko/privacy.tsx b/src/Frontend/src/locales/ko/privacy.tsx index 96f423b..025a6f8 100644 --- a/src/Frontend/src/locales/ko/privacy.tsx +++ b/src/Frontend/src/locales/ko/privacy.tsx @@ -17,6 +17,11 @@ export const privacy: Messages['privacy'] = { '이 기록으로 오류를 찾고, 서버에 과부하가 걸렸을 때 원인을 추적하고, 신고된 악용을 찾아내요. IP 주소는 방문자 한 명이 보낼 수 있는 요청 수와 빌드 횟수를 제한하는 데도 쓰이는데, 이때는 메모리에서만 다뤄요. 이 정보 없이는 요청에 응답할 수 없어요. 법적 근거는 서비스를 운영하고 안전하게 지키려는 저희의 정당한 이익이에요(GDPR 제6조 제1항 (f)).', '관리자는 이 모든 기록을 볼 수 있어요. 람다 소유자는 자기 람다로 들어온 요청마다 국가와 브라우저를 볼 수 있지만, IP 주소는 볼 수 없어요.', ], + analyticsTitle: '람다 소유자를 위해 집계하는 방문', + analytics: [ + '람다 소유자는 자신의 람다에 대해 분석을 켤 수 있어요. 그러면 서버가 그 람다에 대한 방문을 하루 단위로 집계해요. 요청과 페이지 조회가 몇 번이었는지, 어떤 페이지에 대한 것이었는지, 방문자가 온 사이트(페이지 주소가 아니라 사이트 이름만), 방문자 IP 주소의 국가, 그리고 브라우저가 알려 주는 기기, 브라우저, 운영체제의 종류를 집계해요. 이 집계값만 서버의 데이터베이스에 1년 동안 저장하고, IP 주소를 비롯해 방문자를 특정할 수 있는 다른 정보는 저장하지 않아요. 람다 소유자는 에디터에서 이 집계를 볼 수 있고, 관리자도 읽을 수 있어요.', + '방문자를 하루에 한 번만 세기 위해, IP 주소와 브라우저 정보를 날짜와 람다마다 무작위로 만든 키로 되돌릴 수 없는 코드로 바꿔요. 이 코드와 키는 메모리에만 있고 어디에도 기록되지 않으며, 하루가 지나면 잊혀요. 쿠키는 설정하지 않고, 방문자의 브라우저에도 아무것도 저장하지 않아요. 방문이 Google, Bing, OpenAI, Apple 또는 Perplexity의 크롤러에서 왔다고 밝히면, 그 IP 주소를 이들이 크롤러용으로 공개한 주소와 메모리에서 비교해요. 크롤러와 사람을 구분하기 위해 서버는 공개된 크롤러 목록과 이렇게 공개된 주소를 하루에 한 번 가져오며, 이를 위해 방문자에 관한 정보는 어디로도 보내지 않아요. 법적 근거는 자신의 람다가 어떻게 쓰이는지 알고자 하는 소유자의 정당한 이익이에요(GDPR 제6조 제1항 f호).', + ], logsTitle: '로그 보관 기간', logs: '로그는 두 군데에 남아요. 서버 메모리에 남은 로그는 서버가 재시작될 때마다 비워지고, 서버의 콘솔 출력에 남은 로그는 서버가 업데이트될 때마다 지워져요. 둘 다 크기가 정해져 있어서 새 줄이 들어올 때마다 가장 오래된 줄이 밀려나요. 그래서 한 줄이 얼마나 남아 있는지는 사이트가 얼마나 바쁜지에 따라 달라요. 로그를 따로 보관용으로 복사하지는 않아요.', @@ -42,14 +47,14 @@ export const privacy: Messages['privacy'] = { lambdasTitle: '람다가 하는 일은 소유자가 정해요', lambdas: - '람다는 저희가 아니라 에디터 링크를 가진 사람이 작성해요. 람다가 방문자에게 무엇을 요구하고 그걸로 무엇을 하는지는 그 사람이 정하는 일이고, 이 페이지에서 다루지 않아요. 다만 위에서 설명한 요청 로그는 예외로, 서버가 모든 람다에 대해 남겨요. 이용약관은 람다로 다른 사람의 개인정보를 수집하는 것을 금지해요. 그런 람다를 발견하면 신고해 주세요.', + '람다는 저희가 아니라 에디터 링크를 가진 사람이 작성해요. 람다가 방문자에게 무엇을 요구하고 그걸로 무엇을 하는지는 그 사람이 정하는 일이고, 이 페이지에서 다루지 않아요. 다만 위에서 설명한 요청 로그는 예외로, 서버가 모든 람다에 대해 남겨요. 분석을 켠 소유자를 위해 집계하는 방문도 마찬가지예요. 이용약관은 람다로 다른 사람의 개인정보를 수집하는 것을 금지해요. 그런 람다를 발견하면 신고해 주세요.', mailTitle: '저희에게 연락할 때', mail: '악용 신고든 다른 일이든 저희에게 연락하면, 답장하고 연락한 내용을 처리하는 데 이메일 주소와 메시지를 사용해요. 그리고 그 일에 더 이상 필요 없어지면 삭제해요(GDPR 제6조 제1항 (f)).', storageTitle: '쿠키와 브라우저', storage: - '쿠키는 lang 하나뿐이에요. 고른 언어를 기억해서, 언어가 들어 있지 않은 주소를 그 언어로 열어 주고, 1년 동안 유지돼요. 브라우저 자체 저장소에는 라이트 모드인지 다크 모드인지, 쓰고 있는 페이지의 몇 가지 설정, 그리고 관리자라면 관리자 토큰이 저장돼요. 이 중 어느 것도 이용자를 추적하는 데 쓰이지 않고, 다른 누구에게도 넘어가지 않아요. 분석 도구도, 광고도 없고, 다른 사이트에서 불러오는 것도 없어요. 폰트조차도요. 모두 이용자가 요청한 일을 하는 데만 쓰이기 때문에 동의가 필요하지 않아요(독일 법률인 TDDDG 제25조 제2항 제2호).', + '쿠키는 lang 하나뿐이에요. 고른 언어를 기억해서, 언어가 들어 있지 않은 주소를 그 언어로 열어 주고, 1년 동안 유지돼요. 브라우저 자체 저장소에는 라이트 모드인지 다크 모드인지, 쓰고 있는 페이지의 몇 가지 설정, 그리고 관리자라면 관리자 토큰이 저장돼요. 이 중 어느 것도 이용자를 추적하는 데 쓰이지 않고, 다른 누구에게도 넘어가지 않아요. 이 사이트의 분석 도구도, 광고도 없고, 다른 사이트에서 불러오는 것도 없어요. 폰트조차도요. 모두 이용자가 요청한 일을 하는 데만 쓰이기 때문에 동의가 필요하지 않아요(독일 법률인 TDDDG 제25조 제2항 제2호).', hostingTitle: '저장 위치', hosting: diff --git a/src/Frontend/src/locales/nl/editor/analytics.tsx b/src/Frontend/src/locales/nl/editor/analytics.tsx new file mode 100644 index 0000000..77ff6df --- /dev/null +++ b/src/Frontend/src/locales/nl/editor/analytics.tsx @@ -0,0 +1,117 @@ +import type { EditorMessages } from '../../en/editor'; + +export const analytics: EditorMessages['analytics'] = { + hint: 'Door het platform voor je geteld: geen cookies, geen code in je app en niets wat je bezoekers hoeven te accepteren.', + loading: 'De bezoeken worden opgehaald…', + loadFailed: 'De bezoeken konden niet worden opgehaald.', + + offTitle: 'Zie wie je app bezoekt', + offText: + 'Hoeveel mensen er per dag komen, welke pagina’s ze openen, welke sites hen sturen, uit welke landen en op welke apparaten – dag voor dag geteld, vanaf het moment dat je het aanzet.', + offPrivate: 'Geen cookies en geen banner die je bezoekers moeten accepteren', + offAddresses: 'Er worden geen adressen bewaard – alleen hoeveel bezoeken er per dag waren', + offKept: (days) => + days >= 365 + ? 'Een jaar bewaard en alles wordt aan jou getoond' + : `Een jaar bewaard – de laatste ${days} dagen worden getoond, alles met Premium`, + switchOn: 'Analytics aanzetten', + switchedOn: 'Analytics staat aan. De eerste bezoeken verschijnen binnen een paar minuten.', + switchOnFailed: 'Analytics kon niet worden aangezet.', + demo: 'De bezoekers van een demo worden niet geteld. Een eigen kopie kan haar bezoekers wel tellen.', + + range: 'Periode', + days: (days) => (days >= 365 ? '12 maanden' : `${days} dagen`), + premiumRange: 'Premium toont tot een jaar', + + visitors: 'Bezoekers', + visitorsHint: 'Verschillende mensen, elk één keer per dag geteld. Zoekmachines en andere programma’s zijn geen bezoekers.', + views: 'Paginaweergaven', + viewsHint: 'Elke keer dat iemand een pagina opende.', + hits: 'Requests', + hitsHint: 'Alles wat er van je app gevraagd werd: de pagina’s, en de plaatjes en scripts die ze laden.', + versus: (days) => `ten opzichte van de ${days} dagen ervoor`, + noChange: 'ongewijzigd', + inDays: (days) => (days >= 365 ? 'in de afgelopen 12 maanden' : `in de afgelopen ${days} dagen`), + + perDay: { + visitors: 'Bezoekers per dag', + views: 'Paginaweergaven per dag', + hits: 'Requests per dag', + }, + perWeek: { + visitors: 'Bezoekers per week', + views: 'Paginaweergaven per week', + hits: 'Requests per week', + }, + showValues: 'Waarden tonen', + showChart: 'Grafiek tonen', + day: 'Dag', + week: 'Week van', + + nobody: 'In deze dagen heeft niemand de app bezocht.', + nobodyYet: 'Sinds analytics aanstaat heeft nog niemand de app bezocht. De eerste bezoeken verschijnen binnen een paar minuten.', + + pages: 'Pagina’s', + pagesHint: 'De pagina’s die mensen openden, de meest geopende eerst.', + requested: 'Meest opgevraagd', + requestedHint: 'Alles wat er van je app gevraagd werd, per adres – er zijn geen pagina’s die mensen openen.', + sources: 'Waar bezoekers vandaan komen', + sourcesHint: 'De sites die mensen hierheen stuurden, per pagina waarop ze zaten. Zoekmachines laten alleen weten dát zij het waren, niet wat er gezocht is.', + direct: 'Ingetypt, een bladwijzer of een app', + countries: 'Landen', + countriesHint: 'Waar de internetadressen van de bezoekers geregistreerd staan – meestal, niet altijd, waar ze zijn.', + unknownCountry: 'Onbekend', + devices: 'Apparaten', + device: { + desktop: 'Computer', + mobile: 'Telefoon', + tablet: 'Tablet', + bot: 'Bots', + other: 'Overig', + }, + browsers: 'Browsers', + systems: 'Besturingssystemen', + crawlers: 'Zoekmachines, AI en andere programma’s', + crawlersHint: 'Programma’s die langskwamen, per soort. Google, Bing, OpenAI, Apple en Perplexity worden alleen bij naam genoemd als het bezoek van een door hen gepubliceerd adres kwam – iedereen kan zich als hen voordoen.', + noCrawlers: 'In deze dagen is er geen zoekmachine, AI of ander programma langs geweest.', + confirmed: (name) => `Kwam van een adres dat ${name} publiceert voor zijn crawlers`, + crawlerKinds: { + 'search-engine': 'Zoekmachines', + 'ai-crawler': 'AI-crawlers', + 'social-preview': 'Linkvoorbeelden in chats en sociale netwerken', + advertising: 'Advertenties', + seo: 'SEO-tools', + monitoring: 'Monitoring', + 'feed-reader': 'Feedreaders', + archiver: 'Webarchieven', + academic: 'Onderzoek', + scanner: 'Beveiligingsscanners', + 'browser-automation': 'Geautomatiseerde browsers', + 'http-library': 'Scripts en apps', + other: 'Overige programma’s', + unconfirmed: 'Deed zich voor als een bekende crawler, niet bevestigd', + }, + other: 'Overige', + showAll: (count) => `Alle ${count} tonen`, + showFewer: 'Minder tonen', + share: (part, whole) => `${part} van ${whole}`, + none: 'Nog niets.', + + moreKept: (since) => `Geteld sinds ${since} – Premium toont alles, tot een jaar.`, + premiumAsk: 'Vraag naar Premium', + premiumSubject: 'Premium voor mijn app', + premiumBody: (address) => `Hallo,\n\nIk wil graag alle analytics van mijn app zien: ${address}\n\n`, + + footer: (minutes) => + minutes === 1 ? 'Per dag geteld, in UTC. Elke minuut bijgewerkt.' : `Per dag geteld, in UTC. Elke ${minutes} minuten bijgewerkt.`, + updated: (time) => `Laatst bijgewerkt om ${time}.`, + since: (date) => `Geteld sinds ${date}.`, + + switchOff: 'Analytics uitzetten', + switchOffTitle: 'Analytics uitzetten?', + switchOffText: 'Het tellen stopt en alles wat tot nu toe geteld is, wordt verwijderd. Dit kan niet ongedaan worden gemaakt.', + keep: 'Blijven tellen', + switchOffConfirm: 'Uitzetten en verwijderen', + switchedOff: 'Analytics staat uit en wat geteld was, is verwijderd.', + switchOffFailed: 'Analytics kon niet worden uitgezet.', +}; diff --git a/src/Frontend/src/locales/nl/editor/frame.tsx b/src/Frontend/src/locales/nl/editor/frame.tsx index 8bc5041..3555b4e 100644 --- a/src/Frontend/src/locales/nl/editor/frame.tsx +++ b/src/Frontend/src/locales/nl/editor/frame.tsx @@ -16,6 +16,7 @@ export const frame: EditorMessages['frame'] = { history: 'Geschiedenis', deployments: 'Deployments', stats: 'Statistieken', + analytics: 'Analytics', logs: 'Logs', code: 'Code', }, diff --git a/src/Frontend/src/locales/nl/editor/index.ts b/src/Frontend/src/locales/nl/editor/index.ts index 265a740..cc44a22 100644 --- a/src/Frontend/src/locales/nl/editor/index.ts +++ b/src/Frontend/src/locales/nl/editor/index.ts @@ -1,4 +1,5 @@ import type { EditorMessages } from '../../en/editor'; +import { analytics } from './analytics'; import { change } from './change'; import { clone } from './clone'; import { code } from './code'; @@ -33,6 +34,7 @@ export const editor: EditorMessages = { versions, deployments, stats, + analytics, logs, showcase, openSource, diff --git a/src/Frontend/src/locales/nl/guide.tsx b/src/Frontend/src/locales/nl/guide.tsx index 98e09ba..d8803ef 100644 --- a/src/Frontend/src/locales/nl/guide.tsx +++ b/src/Frontend/src/locales/nl/guide.tsx @@ -26,6 +26,7 @@ export const guide: Messages['guide'] = { away: 'Alles meenemen', git: 'Werken met git', open: 'De code publiceren', + visitors: 'Wie je app bezoekt', agents: 'Het aan een agent overlaten', }, @@ -106,6 +107,7 @@ export const guide: Messages['guide'] = { ['Versies', () => <>Wat elke versie veranderde en wat er gevraagd werd, en het verschil met de vorige. Van hieruit deploy je of zet je een versie terug, en vanuit elke versie kun je een concept starten.], ['Deployments', () => <>Wat wanneer online stond, en waardoor het offline ging.], ['Statistieken', () => <>Requests, fouten, responstijden en de meest opgevraagde paden, over het afgelopen uur of de afgelopen dag.], + ['Analytics', () => <>Wie de app bezoekt, zodra je het aanzet: bezoekers, paginaweergaven en requests per dag, de pagina’s die ze openen, de sites die hen sturen, hun landen en apparaten, en de zoekmachines en AI’s die langskwamen.], ['Logs', () => <>Requests, output en de stacktrace van alles wat misging, live.], [ 'Code', @@ -129,7 +131,7 @@ export const guide: Messages['guide'] = { ), editorAside: - 'Het verkeer en de logs staan in het geheugen. Ze zijn om mee te kijken, niet om te bewaren: na een herstart van de server beginnen ze opnieuw. Versies en de deploygeschiedenis worden wel opgeslagen.', + 'Het verkeer en de logs staan in het geheugen. Ze zijn om mee te kijken, niet om te bewaren: na een herstart van de server beginnen ze opnieuw. Versies, de deploygeschiedenis en de analytics worden wel opgeslagen.', why: (k) => ( <> @@ -503,6 +505,27 @@ export const guide: Messages['guide'] = { ), openAside: 'Alles in de code wordt openbaar, de eerdere versies inbegrepen. Een sleutel of wachtwoord hoort bij de sleutels en wachtwoorden onder Data, nooit in de code – gepubliceerd of niet.', + visitors: ((k) => ( + <> + Open {k.b('Analytics')} in het dashboard en zet het aan om te zien wie je app bezoekt: hoeveel mensen er per dag + komen, welke pagina’s ze openen, welke sites hen sturen – een zoekmachine, een bericht, een nieuwsbrief – uit welke + landen, op welke apparaten en browsers, en welke zoekmachines en AI-assistenten de pagina’s lezen. Kies bovenaan + een cijfer om het dag voor dag te zien. + + )), + visitors2: ((k) => ( + <> + Het wordt geteld door het platform, vóór je app, dus je app heeft er niets voor nodig: geen cookies, geen script, + geen banner die je bezoekers moeten accepteren. Alleen aantallen per dag worden bewaard – nooit een adres, en van + de pagina waar iemand vandaan kwam alleen de site. Een bezoeker is een persoon, één keer per dag geteld: + zoekmachines, AI-crawlers en andere programma’s worden apart geteld, per soort, en Google, Bing, OpenAI, Apple en + Perplexity alleen bij naam als het bezoek van een door hen gepubliceerd adres kwam – iedereen kan zich als hen + voordoen. Een jaar wordt bewaard; een gratis app toont daarvan de laatste zeven dagen en {k.b('Premium')} alles, + ook wat eerder geteld is. Als je het uitzet, wordt wat geteld was verwijderd. + + )), + visitorsAside: + 'De cijfers worden om de paar minuten opgeschreven, dus de laatste bezoeken verschijnen iets later. Een concept wordt nooit geteld: een wijziging uitproberen is geen bezoek.', agents: (k) => ( <> diff --git a/src/Frontend/src/locales/nl/privacy.tsx b/src/Frontend/src/locales/nl/privacy.tsx index b0afe06..d49e0f3 100644 --- a/src/Frontend/src/locales/nl/privacy.tsx +++ b/src/Frontend/src/locales/nl/privacy.tsx @@ -17,6 +17,11 @@ export const privacy: Messages['privacy'] = { 'Zo vinden we fouten, zo zien we waardoor een server overbelast raakt, en zo sporen we misbruik op dat bij ons gemeld wordt. Het IP-adres wordt ook gebruikt, alleen in het geheugen, om te beperken hoeveel verzoeken en builds één bezoeker kan doen. Zonder deze gegevens kan een verzoek niet beantwoord worden. De rechtsgrond is ons gerechtvaardigd belang om de dienst te laten draaien en veilig te houden (art. 6 lid 1 sub f AVG).', 'Beheerders kunnen alles inzien. De eigenaar van een lambda ziet bij elk verzoek aan die lambda het land en de browser, maar niet het IP-adres.', ], + analyticsTitle: 'Bezoeken die voor de eigenaar van een lambda worden geteld', + analytics: [ + "De eigenaar van een lambda kan analytics ervoor aanzetten. Dan telt de server de bezoeken aan die lambda per dag: hoeveel requests en paginaweergaven er waren, voor welke pagina’s, de site waar een bezoeker vandaan kwam (alleen de naam, niet het adres van de pagina), het land van het IP-adres van de bezoeker, en het soort apparaat, de browser en het besturingssysteem die de browser opgeeft. Alleen deze aantallen worden bewaard – niet het IP-adres en niets anders waaruit een bezoeker blijkt – in een database op de server, een jaar lang. De eigenaar van de lambda ziet ze in zijn editor, en beheerders kunnen ze inzien.", + "Om elke bezoeker één keer per dag te tellen, worden het IP-adres en de browser omgezet in een code die niet terug te rekenen is, met een sleutel die voor elke dag en elke lambda willekeurig wordt gemaakt. De codes en de sleutels staan alleen in het geheugen, worden nooit opgeschreven en na een dag vergeten. Er wordt geen cookie geplaatst en er wordt niets opgeslagen in de browser van de bezoeker. Als een bezoek zegt van de crawler van Google, Bing, OpenAI, Apple of Perplexity te komen, wordt het IP-adres in het geheugen vergeleken met de adressen die zij voor hun crawlers publiceren. Om crawlers van mensen te onderscheiden haalt de server één keer per dag een openbare lijst van crawlers en deze gepubliceerde adressen op; daarvoor wordt niets over bezoekers ergens naartoe gestuurd. De rechtsgrond is het gerechtvaardigd belang van de eigenaar om te weten hoe zijn lambda wordt gebruikt (art. 6 lid 1 sub f AVG).", + ], logsTitle: 'Hoe lang het log bewaard blijft', logs: 'Het log staat op twee plekken: in het geheugen van de server, dat bij elke herstart wordt leeggemaakt, en in de console-uitvoer van de server, die bij elke update wordt verwijderd. Allebei hebben ze een vaste grootte, dus elke nieuwe regel duwt de oudste eruit. Hoe lang een regel blijft staan, hangt af van hoe druk het op de site is. Niets uit het log wordt in een archief bewaard.', @@ -43,14 +48,14 @@ export const privacy: Messages['privacy'] = { lambdasTitle: 'Wat een lambda doet, bepaalt de eigenaar', lambdas: - 'Een lambda wordt geschreven door wie de editorlink heeft, niet door ons. Wat hij aan bezoekers vraagt en wat hij daarmee doet, bepaalt die persoon, en dat valt buiten dit privacybeleid. Alleen het verzoeklog hierboven, dat de server voor elke lambda bijhoudt, valt er wel onder. Volgens de gebruiksvoorwaarden mag je een lambda niet gebruiken om persoonsgegevens van anderen te verzamelen. Kom je er een tegen die dat wel doet? Meld het dan.', + 'Een lambda wordt geschreven door wie de editorlink heeft, niet door ons. Wat hij aan bezoekers vraagt en wat hij daarmee doet, bepaalt die persoon, en dat valt buiten dit privacybeleid. Alleen het verzoeklog hierboven, dat de server voor elke lambda bijhoudt, en de bezoeken die de server telt voor een eigenaar die analytics heeft aangezet, vallen er wel onder. Volgens de gebruiksvoorwaarden mag je een lambda niet gebruiken om persoonsgegevens van anderen te verzamelen. Kom je er een tegen die dat wel doet? Meld het dan.', mailTitle: 'Als je ons mailt', mail: 'Als je ons schrijft, om misbruik te melden of over iets anders, gebruiken we je adres en je bericht om je te antwoorden en af te handelen wat je schreef. We verwijderen ze zodra we ze daarvoor niet meer nodig hebben (art. 6 lid 1 sub f AVG).', storageTitle: 'Cookies en je browser', storage: - "Er is één cookie, met de naam lang. Die onthoudt welke taal je hebt gekozen, zodat adressen zonder taal in die taal openen, en blijft een jaar bewaard. De opslag van je browser zelf onthoudt of je de lichte of donkere modus gebruikt, een paar instellingen van de pagina's die je gebruikt en, voor beheerders, hun token. Niets daarvan wordt gebruikt om je te volgen en niets gaat naar anderen: er zijn geen analytics en geen advertenties, en er wordt niets van andere sites geladen, zelfs geen lettertypen. Omdat dit allemaal alleen doet waar je zelf om vraagt, is er geen toestemming nodig (§ 25 lid 2 nr. 2 van de Duitse TDDDG).", + "Er is één cookie, met de naam lang. Die onthoudt welke taal je hebt gekozen, zodat adressen zonder taal in die taal openen, en blijft een jaar bewaard. De opslag van je browser zelf onthoudt of je de lichte of donkere modus gebruikt, een paar instellingen van de pagina's die je gebruikt en, voor beheerders, hun token. Niets daarvan wordt gebruikt om je te volgen en niets gaat naar anderen: er zijn geen analytics van deze site en geen advertenties, en er wordt niets van andere sites geladen, zelfs geen lettertypen. Omdat dit allemaal alleen doet waar je zelf om vraagt, is er geen toestemming nodig (§ 25 lid 2 nr. 2 van de Duitse TDDDG).", hostingTitle: 'Waar het wordt bewaard', hosting: diff --git a/src/Frontend/src/locales/pl/editor/analytics.tsx b/src/Frontend/src/locales/pl/editor/analytics.tsx new file mode 100644 index 0000000..745ce7d --- /dev/null +++ b/src/Frontend/src/locales/pl/editor/analytics.tsx @@ -0,0 +1,117 @@ +import type { EditorMessages } from '../../en/editor'; + +export const analytics: EditorMessages['analytics'] = { + hint: 'Liczone za ciebie przez platformę: bez plików cookie, bez kodu w twojej aplikacji i bez niczego, co twoi odwiedzający musieliby akceptować.', + loading: 'Wczytuję wizyty…', + loadFailed: 'Nie udało się wczytać wizyt.', + + offTitle: 'Zobacz, kto odwiedza twoją aplikację', + offText: + 'Ilu ludzi przychodzi każdego dnia, jakie strony otwierają, które strony ich kierują, z jakich krajów i na jakich urządzeniach – liczone dzień po dniu, od chwili włączenia.', + offPrivate: 'Bez plików cookie i bez banera, który odwiedzający musieliby zaakceptować', + offAddresses: 'Nie przechowujemy żadnych adresów – tylko liczbę wizyt każdego dnia', + offKept: (days) => + days >= 365 + ? 'Przechowywane przez rok, a pokazujemy ci całość' + : `Przechowywane przez rok – pokazujemy ostatnie ${days} dni, całość w Premium`, + switchOn: 'Włącz analitykę', + switchedOn: 'Analityka jest włączona. Pierwsze wizyty pojawią się w ciągu kilku minut.', + switchOnFailed: 'Nie udało się włączyć analityki.', + demo: 'Odwiedzający dema nie są liczeni. Twoja własna kopia może liczyć swoich odwiedzających.', + + range: 'Zakres czasu', + days: (days) => (days >= 365 ? '12 miesięcy' : `${days} dni`), + premiumRange: 'Premium pokazuje do roku', + + visitors: 'Odwiedzający', + visitorsHint: 'Różne osoby, każda liczona raz dziennie. Wyszukiwarki i inne programy nie są odwiedzającymi.', + views: 'Odsłony stron', + viewsHint: 'Za każdym razem, gdy ktoś otworzył stronę.', + hits: 'Żądania', + hitsHint: 'Wszystko, o co poproszono twoją aplikację: jej strony oraz obrazy i skrypty, które ładują.', + versus: (days) => `względem poprzednich ${days} dni`, + noChange: 'bez zmian', + inDays: (days) => (days >= 365 ? 'w ostatnich 12 miesiącach' : `w ostatnich ${days} dniach`), + + perDay: { + visitors: 'Odwiedzający dziennie', + views: 'Odsłony stron dziennie', + hits: 'Żądania dziennie', + }, + perWeek: { + visitors: 'Odwiedzający tygodniowo', + views: 'Odsłony stron tygodniowo', + hits: 'Żądania tygodniowo', + }, + showValues: 'Pokaż wartości', + showChart: 'Pokaż wykres', + day: 'Dzień', + week: 'Tydzień od', + + nobody: 'W tych dniach nikt nie odwiedził aplikacji.', + nobodyYet: 'Od włączenia analityki nikt jej nie odwiedził. Pierwsze wizyty pojawią się w ciągu kilku minut.', + + pages: 'Strony', + pagesHint: 'Strony otwierane przez ludzi, od najczęściej otwieranych.', + requested: 'Najczęściej żądane', + requestedHint: 'Wszystko, o co poproszono twoją aplikację, według adresu – nie ma w niej stron otwieranych przez ludzi.', + sources: 'Skąd przychodzą odwiedzający', + sourcesHint: 'Strony, które skierowały tu ludzi, według strony, na której byli. Wyszukiwarki podają tylko, że to one, a nie czego szukano.', + direct: 'Wpisany adres, zakładka lub aplikacja', + countries: 'Kraje', + countriesHint: 'Gdzie zarejestrowane są adresy internetowe odwiedzających – zwykle, choć nie zawsze, tam, gdzie się znajdują.', + unknownCountry: 'Nieznany', + devices: 'Urządzenia', + device: { + desktop: 'Komputer', + mobile: 'Telefon', + tablet: 'Tablet', + bot: 'Boty', + other: 'Inne', + }, + browsers: 'Przeglądarki', + systems: 'Systemy', + crawlers: 'Wyszukiwarki, AI i inne programy', + crawlersHint: 'Programy, które zajrzały, według tego, czym są. Google, Bing, OpenAI, Apple i Perplexity są podawane z nazwy tylko wtedy, gdy wizyta przyszła z adresu, który publikują – każdy może się za nie podać.', + noCrawlers: 'W tych dniach nie zajrzała żadna wyszukiwarka, AI ani inny program.', + confirmed: (name) => `Wizyta z adresu, który ${name} publikuje dla swoich robotów`, + crawlerKinds: { + 'search-engine': 'Wyszukiwarki', + 'ai-crawler': 'Roboty AI', + 'social-preview': 'Podglądy linków w czatach i sieciach społecznościowych', + advertising: 'Reklama', + seo: 'Narzędzia SEO', + monitoring: 'Monitoring', + 'feed-reader': 'Czytniki kanałów', + archiver: 'Archiwa internetu', + academic: 'Badania', + scanner: 'Skanery bezpieczeństwa', + 'browser-automation': 'Zautomatyzowane przeglądarki', + 'http-library': 'Skrypty i aplikacje', + other: 'Inne programy', + unconfirmed: 'Podały się za znanego robota, bez potwierdzenia', + }, + other: 'Pozostałe', + showAll: (count) => `Pokaż wszystkie (${count})`, + showFewer: 'Pokaż mniej', + share: (part, whole) => `${part} z ${whole}`, + none: 'Jeszcze nic.', + + moreKept: (since) => `Liczone od ${since} – Premium pokazuje całość, do roku.`, + premiumAsk: 'Zapytaj o Premium', + premiumSubject: 'Premium dla mojej aplikacji', + premiumBody: (address) => `Dzień dobry,\n\nproszę o dostęp do pełnej analityki mojej aplikacji: ${address}\n\n`, + + footer: (minutes) => + minutes === 1 ? 'Liczone dziennie, w UTC. Odświeżane co minutę.' : `Liczone dziennie, w UTC. Odświeżane co ${minutes} min.`, + updated: (time) => `Ostatnia aktualizacja: ${time}.`, + since: (date) => `Liczone od ${date}.`, + + switchOff: 'Wyłącz analitykę', + switchOffTitle: 'Wyłączyć analitykę?', + switchOffText: 'Liczenie zostanie zatrzymane, a wszystko, co dotąd policzono, zostanie usunięte. Tego nie można cofnąć.', + keep: 'Liczyć dalej', + switchOffConfirm: 'Wyłącz i usuń', + switchedOff: 'Analityka jest wyłączona, a to, co policzono, zostało usunięte.', + switchOffFailed: 'Nie udało się wyłączyć analityki.', +}; diff --git a/src/Frontend/src/locales/pl/editor/frame.tsx b/src/Frontend/src/locales/pl/editor/frame.tsx index 67d56c5..6813055 100644 --- a/src/Frontend/src/locales/pl/editor/frame.tsx +++ b/src/Frontend/src/locales/pl/editor/frame.tsx @@ -16,6 +16,7 @@ export const frame: EditorMessages['frame'] = { history: 'Historia', deployments: 'Wdrożenia', stats: 'Statystyki', + analytics: 'Analityka', logs: 'Logi', code: 'Kod', }, diff --git a/src/Frontend/src/locales/pl/editor/index.ts b/src/Frontend/src/locales/pl/editor/index.ts index 3e1b706..a4b0923 100644 --- a/src/Frontend/src/locales/pl/editor/index.ts +++ b/src/Frontend/src/locales/pl/editor/index.ts @@ -1,4 +1,5 @@ import type { EditorMessages } from '../../en/editor'; +import { analytics } from './analytics'; import { change } from './change'; import { clone } from './clone'; import { code } from './code'; @@ -33,6 +34,7 @@ export const editor: EditorMessages = { versions, deployments, stats, + analytics, logs, showcase, openSource, diff --git a/src/Frontend/src/locales/pl/guide.tsx b/src/Frontend/src/locales/pl/guide.tsx index 0d46143..95cd882 100644 --- a/src/Frontend/src/locales/pl/guide.tsx +++ b/src/Frontend/src/locales/pl/guide.tsx @@ -26,6 +26,7 @@ export const guide: Messages['guide'] = { away: 'Zabierz kod ze sobą', git: 'Praca z git', open: 'Publikowanie kodu', + visitors: 'Kto ją odwiedza', agents: 'Niech zrobi to agent', }, @@ -105,6 +106,7 @@ export const guide: Messages['guide'] = { ['Wersje', () => <>Co zmieniła każda wersja, o co proszono i czym różni się od poprzedniej. Stąd wdrażasz wersję albo wracasz do starszej – albo tworzysz szkic na bazie dowolnej z nich.], ['Wdrożenia', () => <>Co i kiedy było online – i co to wyłączyło.], ['Statystyki', () => <>Żądania, błędy, czasy odpowiedzi i najczęściej odwiedzane ścieżki z ostatniej godziny albo ostatnich 24 godzin.], + ['Analityka', () => <>Kto odwiedza aplikację, gdy ją włączysz: odwiedzający, odsłony stron i żądania dziennie, otwierane strony, strony, które kierują ludzi do aplikacji, ich kraje i urządzenia oraz wyszukiwarki i AI, które zajrzały.], ['Logi', () => <>Żądania, to, co lambda wypisała, i stack trace każdego błędu – na bieżąco.], [ 'Kod', @@ -127,7 +129,7 @@ export const guide: Messages['guide'] = { ), editorAside: - 'Ruch i log są trzymane w pamięci – do podglądu, nie do archiwizacji: po restarcie serwera zaczynają się od zera. Wersje i historia wdrożeń są zapisywane na stałe.', + 'Ruch i log są trzymane w pamięci – do podglądu, nie do archiwizacji: po restarcie serwera zaczynają się od zera. Wersje, historia wdrożeń i analityka są zapisywane na stałe.', why: (k) => ( <> @@ -489,6 +491,27 @@ export const guide: Messages['guide'] = { ), openAside: 'Wszystko, co jest w kodzie, staje się publiczne, łącznie z wcześniejszymi wersjami. Miejsce klucza czy hasła jest wśród kluczy i haseł w sekcji Dane, nigdy w kodzie – opublikowanym czy nie.', + visitors: (k) => ( + <> + Otwórz {k.b('Analityka')} w centrum sterowania i włącz ją, aby zobaczyć, kto odwiedza twoją aplikację: ilu ludzi + przychodzi każdego dnia, jakie strony otwierają, które strony ich kierują – wyszukiwarka, post, newsletter – z + jakich krajów, na jakich urządzeniach i w jakich przeglądarkach oraz które wyszukiwarki i asystenci AI czytają + jej strony. Wybierz liczbę u góry, aby zobaczyć ją dzień po dniu. + + ), + visitors2: (k) => ( + <> + Liczy to platforma, przed twoją aplikacją, więc aplikacja nie potrzebuje do tego niczego: ani plików cookie, ani + skryptu, ani banera do zaakceptowania przez odwiedzających. Przechowujemy tylko liczby dzień po dniu – nigdy + adresu, a ze strony, z której ktoś przyszedł, tylko nazwę witryny. Odwiedzający to osoba, liczona raz dziennie: + wyszukiwarki, roboty AI i inne programy są liczone osobno, według tego, czym są, a Google, Bing, OpenAI, Apple + i Perplexity tylko z nazwy, gdy wizyta przyszła z adresu, który publikują – każdy może się za nie podać. + Dane są przechowywane przez rok; bezpłatna aplikacja pokazuje z nich ostatnie siedem dni, a {k.b('Premium')} + całość, także to, co policzono wcześniej. Wyłączenie analityki usuwa to, co policzono. + + ), + visitorsAside: + 'Liczby są zapisywane co kilka minut, więc ostatnie wizyty pojawiają się z niewielkim opóźnieniem. Szkic nigdy nie jest liczony: wypróbowywanie zmiany nie jest wizytą.', agents: (k) => ( <> diff --git a/src/Frontend/src/locales/pl/privacy.tsx b/src/Frontend/src/locales/pl/privacy.tsx index ac1846e..ab3bf35 100644 --- a/src/Frontend/src/locales/pl/privacy.tsx +++ b/src/Frontend/src/locales/pl/privacy.tsx @@ -17,6 +17,11 @@ export const privacy: Messages['privacy'] = { 'Dzięki temu znajdujemy błędy, ustalamy, co przeciąża serwer, i namierzamy zgłoszone nam nadużycia. Adres IP służy też – wyłącznie w pamięci – do ograniczania liczby żądań i zadań budowania, jakie może zlecić jeden odwiedzający. Bez tych danych nie da się odpowiedzieć na żądanie. Podstawą prawną jest nasz prawnie uzasadniony interes w prowadzeniu usługi i dbaniu o jej bezpieczeństwo (art. 6 ust. 1 lit. f RODO).', 'Administratorzy serwisu mogą zobaczyć wszystko. Właściciel lambdy widzi przy każdym żądaniu do niej kraj i przeglądarkę, ale nie adres IP.', ], + analyticsTitle: 'Wizyty liczone dla właściciela lambdy', + analytics: [ + 'Właściciel lambdy może włączyć dla niej analitykę. Wtedy serwer liczy wizyty w tej lambdzie dzień po dniu: ile było żądań i odsłon stron, jakich stron dotyczyły, z jakiej witryny przyszedł odwiedzający (tylko jej nazwa, bez adresu strony), kraj adresu IP odwiedzającego oraz rodzaj urządzenia, przeglądarki i systemu operacyjnego, który podaje przeglądarka. Przechowujemy wyłącznie te liczby – bez adresu IP i bez niczego innego, co wskazuje odwiedzającego – w bazie danych na serwerze, przez rok. Właściciel lambdy widzi je w swoim edytorze, a administratorzy mogą je odczytać.', + 'Aby policzyć każdego odwiedzającego raz dziennie, adres IP i przeglądarka są zamieniane na kod, którego nie da się odwrócić, za pomocą klucza tworzonego losowo dla każdego dnia i każdej lambdy. Kody i klucze są trzymane tylko w pamięci, nigdy nie są zapisywane i po dobie są zapominane. Nie jest ustawiany żaden plik cookie i nic nie jest zapisywane w przeglądarce odwiedzającego. Jeśli wizyta podaje, że pochodzi od robota Google, Bing, OpenAI, Apple lub Perplexity, jej adres IP jest porównywany, w pamięci, z adresami, które te firmy publikują dla swoich robotów. Aby odróżnić roboty od ludzi, serwer raz dziennie pobiera publiczną listę robotów i te opublikowane adresy; nic o odwiedzających nie jest w tym celu nigdzie wysyłane. Podstawą prawną jest uzasadniony interes właściciela w wiedzy o tym, jak jego lambda jest używana (art. 6 ust. 1 lit. f RODO).', + ], logsTitle: 'Jak długo przechowujemy log', logs: 'Log jest przechowywany w dwóch miejscach: w pamięci serwera, która jest czyszczona przy każdym restarcie, oraz w wyjściu konsoli serwera, które jest usuwane przy każdej aktualizacji. Oba mają stały rozmiar, więc każdy nowy wpis wypycha najstarszy, a to, jak długo wpis przetrwa, zależy od ruchu na stronie. Nic z logu nie trafia do archiwum.', @@ -42,14 +47,14 @@ export const privacy: Messages['privacy'] = { lambdasTitle: 'O tym, co robi lambda, decyduje jej właściciel', lambdas: - 'Lambdę pisze ten, kto ma link do edytora, a nie my. O co pyta odwiedzających i co robi z tymi danymi, zależy od tej osoby, a ta polityka tego nie obejmuje – poza opisanym wyżej logiem żądań, który serwer prowadzi dla każdej lambdy. Regulamin zabrania używania lambdy do zbierania danych osobowych innych ludzi. Jeśli trafisz na taką, która to robi, zgłoś ją.', + 'Lambdę pisze ten, kto ma link do edytora, a nie my. O co pyta odwiedzających i co robi z tymi danymi, zależy od tej osoby, a ta polityka tego nie obejmuje – poza opisanym wyżej logiem żądań, który serwer prowadzi dla każdej lambdy, oraz wizytami, które liczy dla właściciela, który włączył analitykę. Regulamin zabrania używania lambdy do zbierania danych osobowych innych ludzi. Jeśli trafisz na taką, która to robi, zgłoś ją.', mailTitle: 'Gdy do nas piszesz', mail: 'Jeśli do nas napiszesz – żeby zgłosić nadużycie albo w innej sprawie – używamy twojego adresu i wiadomości, żeby ci odpowiedzieć i zająć się sprawą. Usuwamy je, gdy nie są już do tego potrzebne (art. 6 ust. 1 lit. f RODO).', storageTitle: 'Pliki cookie i twoja przeglądarka', storage: - 'Jest jeden plik cookie o nazwie lang. Zapamiętuje wybrany przez ciebie język, żeby adresy bez języka otwierały się właśnie w nim, i jest ważny przez rok. Pamięć samej przeglądarki przechowuje jasny lub ciemny motyw, kilka ustawień stron, z których korzystasz, a w przypadku administratorów serwisu – ich token. Nic z tego nie służy do śledzenia cię i nic nie trafia do nikogo innego: nie ma analityki ani reklam, a nic nie jest ładowane z innych stron, nawet czcionki. Wszystko to służy wyłącznie temu, o co prosisz, więc nie wymaga zgody (§ 25 ust. 2 pkt 2 niemieckiej ustawy TDDDG).', + 'Jest jeden plik cookie o nazwie lang. Zapamiętuje wybrany przez ciebie język, żeby adresy bez języka otwierały się właśnie w nim, i jest ważny przez rok. Pamięć samej przeglądarki przechowuje jasny lub ciemny motyw, kilka ustawień stron, z których korzystasz, a w przypadku administratorów serwisu – ich token. Nic z tego nie służy do śledzenia cię i nic nie trafia do nikogo innego: nie ma analityki tego serwisu ani reklam, a nic nie jest ładowane z innych stron, nawet czcionki. Wszystko to służy wyłącznie temu, o co prosisz, więc nie wymaga zgody (§ 25 ust. 2 pkt 2 niemieckiej ustawy TDDDG).', hostingTitle: 'Gdzie są przechowywane dane', hosting: diff --git a/src/Frontend/src/locales/pt-pt/editor/analytics.tsx b/src/Frontend/src/locales/pt-pt/editor/analytics.tsx new file mode 100644 index 0000000..f65d497 --- /dev/null +++ b/src/Frontend/src/locales/pt-pt/editor/analytics.tsx @@ -0,0 +1,117 @@ +import type { EditorMessages } from '../../en/editor'; + +export const analytics: EditorMessages['analytics'] = { + hint: 'Contado para ti pela plataforma: sem cookies, sem código na tua app e sem nada para os teus visitantes aceitarem.', + loading: 'A ler as visitas…', + loadFailed: 'Não foi possível ler as visitas.', + + offTitle: 'Vê quem visita a tua app', + offText: + 'Quantas pessoas chegam por dia, que páginas abrem, que sites as enviam, de que países e em que dispositivos - contado dia a dia, a partir do momento em que ligares.', + offPrivate: 'Sem cookies e sem banners para os teus visitantes aceitarem', + offAddresses: 'Não se guardam endereços - apenas quantas visitas houve em cada dia', + offKept: (days) => + days >= 365 + ? 'Guardado durante um ano, e tudo é mostrado a ti' + : `Guardado durante um ano - são mostrados os últimos ${days} dias, e tudo com o Premium`, + switchOn: 'Ligar a análise de visitas', + switchedOn: 'A análise de visitas está ligada. As primeiras visitas aparecem dentro de alguns minutos.', + switchOnFailed: 'Não foi possível ligar a análise de visitas.', + demo: 'Os visitantes de uma demo não são contados. Uma cópia tua pode contar os seus visitantes.', + + range: 'Período', + days: (days) => (days >= 365 ? '12 meses' : `${days} dias`), + premiumRange: 'O Premium mostra até um ano', + + visitors: 'Visitantes', + visitorsHint: 'Pessoas diferentes, cada uma contada uma vez por dia. Motores de pesquisa e outros programas não são visitantes.', + views: 'Visualizações de páginas', + viewsHint: 'Cada vez que alguém abriu uma página.', + hits: 'Pedidos', + hitsHint: 'Tudo o que foi pedido à tua app: as suas páginas e as imagens e scripts que elas carregam.', + versus: (days) => `em relação aos ${days} dias anteriores`, + noChange: 'sem alteração', + inDays: (days) => (days >= 365 ? 'nos últimos 12 meses' : `nos últimos ${days} dias`), + + perDay: { + visitors: 'Visitantes por dia', + views: 'Visualizações de páginas por dia', + hits: 'Pedidos por dia', + }, + perWeek: { + visitors: 'Visitantes por semana', + views: 'Visualizações de páginas por semana', + hits: 'Pedidos por semana', + }, + showValues: 'Mostrar os valores', + showChart: 'Mostrar o gráfico', + day: 'Dia', + week: 'Semana de', + + nobody: 'Ninguém visitou nestes dias.', + nobodyYet: 'Ninguém visitou desde que a análise de visitas foi ligada. As primeiras visitas aparecem dentro de alguns minutos.', + + pages: 'Páginas', + pagesHint: 'As páginas que as pessoas abriram, as mais abertas primeiro.', + requested: 'Mais pedidos', + requestedHint: 'Tudo o que foi pedido à tua app, por endereço - ela não tem páginas que as pessoas abram.', + sources: 'De onde vêm os visitantes', + sourcesHint: 'Os sites que enviaram pessoas para aqui, pela página em que estavam. Os motores de pesquisa só indicam que foram eles, não o que foi pesquisado.', + direct: 'Endereço escrito, favorito ou app', + countries: 'Países', + countriesHint: 'Onde estão registados os endereços de internet dos visitantes - normalmente, mas nem sempre, onde eles estão.', + unknownCountry: 'Desconhecido', + devices: 'Dispositivos', + device: { + desktop: 'Computador', + mobile: 'Telemóvel', + tablet: 'Tablet', + bot: 'Bots', + other: 'Outro', + }, + browsers: 'Browsers', + systems: 'Sistemas', + crawlers: 'Motores de pesquisa, IA e outros programas', + crawlersHint: 'Programas que passaram por aqui, por aquilo que são. Google, Bing, OpenAI, Apple e Perplexity só são indicados pelo nome quando a visita veio de um endereço que eles publicam - qualquer um pode afirmar que é um deles.', + noCrawlers: 'Nenhum motor de pesquisa, IA ou outro programa passou por cá nestes dias.', + confirmed: (name) => `Veio de um endereço que ${name} publica para os seus crawlers`, + crawlerKinds: { + 'search-engine': 'Motores de pesquisa', + 'ai-crawler': 'Crawlers de IA', + 'social-preview': 'Pré-visualizações de links em chats e redes sociais', + advertising: 'Publicidade', + seo: 'Ferramentas de SEO', + monitoring: 'Monitorização', + 'feed-reader': 'Leitores de feeds', + archiver: 'Arquivos da web', + academic: 'Investigação', + scanner: 'Verificadores de segurança', + 'browser-automation': 'Browsers automatizados', + 'http-library': 'Scripts e apps', + other: 'Outros programas', + unconfirmed: 'Diziam ser um crawler conhecido, sem confirmação', + }, + other: 'Outros', + showAll: (count) => `Mostrar todos os ${count}`, + showFewer: 'Mostrar menos', + share: (part, whole) => `${part} de ${whole}`, + none: 'Ainda nada.', + + moreKept: (since) => `Contado desde ${since} - o Premium mostra tudo, até um ano.`, + premiumAsk: 'Pergunta pelo Premium', + premiumSubject: 'Premium para a minha app', + premiumBody: (address) => `Olá,\n\nGostaria de ver toda a análise de visitas da minha app: ${address}\n\n`, + + footer: (minutes) => + minutes === 1 ? 'Contado por dia, em UTC. Atualizado a cada minuto.' : `Contado por dia, em UTC. Atualizado a cada ${minutes} minutos.`, + updated: (time) => `Última atualização às ${time}.`, + since: (date) => `Contagem desde ${date}.`, + + switchOff: 'Desligar a análise de visitas', + switchOffTitle: 'Desligar a análise de visitas?', + switchOffText: 'A contagem pára e tudo o que foi contado até agora é apagado. Isto não pode ser desfeito.', + keep: 'Continuar a contar', + switchOffConfirm: 'Desligar e apagar', + switchedOff: 'A análise de visitas está desligada e o que foi contado foi apagado.', + switchOffFailed: 'Não foi possível desligar a análise de visitas.', +}; diff --git a/src/Frontend/src/locales/pt-pt/editor/frame.tsx b/src/Frontend/src/locales/pt-pt/editor/frame.tsx index 92ba6d1..ba9c863 100644 --- a/src/Frontend/src/locales/pt-pt/editor/frame.tsx +++ b/src/Frontend/src/locales/pt-pt/editor/frame.tsx @@ -16,6 +16,7 @@ export const frame: EditorMessages['frame'] = { history: 'Histórico', deployments: 'Deploys', stats: 'Estatísticas', + analytics: 'Análise de visitas', logs: 'Logs', code: 'Código', }, diff --git a/src/Frontend/src/locales/pt-pt/editor/index.ts b/src/Frontend/src/locales/pt-pt/editor/index.ts index eee31a2..d5103e2 100644 --- a/src/Frontend/src/locales/pt-pt/editor/index.ts +++ b/src/Frontend/src/locales/pt-pt/editor/index.ts @@ -1,4 +1,5 @@ import type { EditorMessages } from '../../en/editor'; +import { analytics } from './analytics'; import { change } from './change'; import { clone } from './clone'; import { code } from './code'; @@ -33,6 +34,7 @@ export const editor: EditorMessages = { versions, deployments, stats, + analytics, logs, showcase, openSource, diff --git a/src/Frontend/src/locales/pt-pt/guide.tsx b/src/Frontend/src/locales/pt-pt/guide.tsx index 56fc5e1..1aef636 100644 --- a/src/Frontend/src/locales/pt-pt/guide.tsx +++ b/src/Frontend/src/locales/pt-pt/guide.tsx @@ -26,6 +26,7 @@ export const guide: Messages['guide'] = { away: 'Levar o código contigo', git: 'Trabalhar nela com git', open: 'Publicar o código', + visitors: 'Quem a visita', agents: 'Deixar um agente fazer o trabalho', }, @@ -106,6 +107,7 @@ export const guide: Messages['guide'] = { ['Versões', () => <>O que cada versão mudou, o que foi pedido e a diferença para a anterior. Faz deploy ou reverte a partir daqui, ou começa um rascunho a partir de qualquer uma delas.], ['Deploys', () => <>O que esteve online e quando, e o que o pôs offline.], ['Estatísticas', () => <>Pedidos, falhas, tempos de resposta e os caminhos mais pedidos, na última hora ou nas últimas 24 horas.], + ['Análise de visitas', () => <>Quem visita a app, depois de a ligares: visitantes, visualizações de páginas e pedidos por dia, as páginas que abrem, os sites que os enviam, os países e os dispositivos, e os motores de pesquisa e as IA que passaram por cá.], ['Logs', () => <>Os pedidos, o que a lambda escreveu na consola e o stack trace de tudo o que correu mal, em tempo real.], [ 'Código', @@ -129,7 +131,7 @@ export const guide: Messages['guide'] = { ), editorAside: - 'O tráfego e os logs ficam em memória, para acompanhar e não para guardar: um reinício do servidor põe-nos a zero. As versões e o histórico de deploys ficam guardados.', + 'O tráfego e os logs ficam em memória, para acompanhar e não para guardar: um reinício do servidor põe-nos a zero. As versões, o histórico de deploys e a análise de visitas ficam guardados.', why: (k) => ( <> @@ -493,6 +495,27 @@ export const guide: Messages['guide'] = { ), openAside: 'Tudo o que está no código fica público, incluindo as versões anteriores. Uma chave ou uma palavra-passe fica junto das chaves e palavras-passe em Dados, nunca no código, publicado ou não.', + visitors: ((k) => ( + <> + Abre {k.b('Análise de visitas')} no painel de controlo e liga-a para ver quem visita a tua app: quantas pessoas + chegam por dia, que páginas abrem, que sites as enviam - um motor de pesquisa, uma publicação, uma newsletter - e + de que países, em que dispositivos e browsers, e que motores de pesquisa e assistentes de IA leem as suas páginas. + Escolhe um valor no topo para o ver dia a dia. + + )), + visitors2: ((k) => ( + <> + É contado pela plataforma, à frente da tua app, por isso a tua app não precisa de nada para isso: sem cookies, + sem script e sem banner para os teus visitantes aceitarem. Só se guardam contagens por dia - nunca um endereço + e, da página de onde alguém veio, apenas o site. Um visitante é uma pessoa, contada uma vez por dia: motores de + pesquisa, crawlers de IA e outros programas são contados à parte, por aquilo que são, e Google, Bing, OpenAI, + Apple e Perplexity só pelo nome quando a visita veio de um endereço que eles publicam - qualquer um pode + afirmar que é um deles. Guarda-se um ano; uma app gratuita mostra os últimos sete dias e o {k.b('Premium')} + mostra tudo, incluindo o que foi contado antes. Ao desligar, apaga-se o que foi contado. + + )), + visitorsAside: + 'Os valores são registados de poucos em poucos minutos, por isso as últimas visitas aparecem um pouco mais tarde. Um rascunho nunca é contado: experimentar uma alteração não é uma visita.', agents: (k) => ( <> diff --git a/src/Frontend/src/locales/pt-pt/privacy.tsx b/src/Frontend/src/locales/pt-pt/privacy.tsx index 9799073..ef8c118 100644 --- a/src/Frontend/src/locales/pt-pt/privacy.tsx +++ b/src/Frontend/src/locales/pt-pt/privacy.tsx @@ -17,6 +17,11 @@ export const privacy: Messages['privacy'] = { 'É assim que se encontram falhas, que se descobre o que está a sobrecarregar o servidor e que se investigam os abusos que nos são denunciados. O endereço IP também é usado, só em memória, para limitar quantos pedidos um mesmo visitante pode fazer e quantas apps pode mandar criar. Sem estes dados, não é possível responder a um pedido. O fundamento jurídico é o nosso interesse legítimo em manter o serviço a funcionar e em segurança (art. 6.º, n.º 1, alínea f), do RGPD).', 'Os administradores podem ver tudo isto. Quem é dono de uma lambda vê o país e o browser de cada pedido feito a essa lambda, mas não o endereço IP.', ], + analyticsTitle: 'Visitas contadas para o proprietário de uma lambda', + analytics: [ + 'O proprietário de uma lambda pode ligar a análise de visitas dessa lambda. Nesse caso, o servidor conta as visitas a essa lambda por dia: quantos pedidos e visualizações de páginas houve, para que páginas eram, o site de onde veio o visitante (apenas o nome, não o endereço da página), o país do endereço IP do visitante e o tipo de dispositivo, de browser e de sistema operativo que o browser indica. Guardam-se apenas estas contagens - não o endereço IP nem mais nada que identifique um visitante - numa base de dados no servidor, durante um ano. O proprietário da lambda vê-as no seu editor e os administradores podem lê-las.', + 'Para contar cada visitante uma vez por dia, o endereço IP e o browser são transformados num código que não pode ser revertido, com uma chave gerada aleatoriamente para cada dia e cada lambda. Os códigos e as chaves existem apenas em memória, nunca são escritos e são esquecidos ao fim de um dia. Não é definido nenhum cookie e nada é guardado no browser do visitante. Quando uma visita diz vir do crawler da Google, do Bing, da OpenAI, da Apple ou da Perplexity, o seu endereço IP é comparado, em memória, com os endereços que estes publicam para os seus crawlers. Para distinguir crawlers de pessoas, o servidor vai buscar uma vez por dia uma lista pública de crawlers e estes endereços publicados; nada sobre os visitantes é enviado para lado nenhum para isso. O fundamento jurídico é o interesse legítimo do proprietário em saber como a sua lambda é usada (art. 6.º, n.º 1, alínea f), do RGPD).', + ], logsTitle: 'Durante quanto tempo o log é guardado', logs: 'O log fica em dois sítios: na memória do servidor, que é esvaziada sempre que o servidor reinicia, e na saída da consola do servidor, que é apagada sempre que o servidor é atualizado. Ambos têm um tamanho fixo, por isso cada linha nova empurra para fora a mais antiga, e o tempo que uma linha lá fica depende do movimento que o site tiver. Nada do log é copiado para um arquivo.', @@ -43,14 +48,14 @@ export const privacy: Messages['privacy'] = { lambdasTitle: 'Quem decide o que uma lambda faz é o dono dela', lambdas: - 'Uma lambda é escrita por quem tem o link de edição, não por nós. O que pede aos visitantes e o que faz com isso é decisão dessa pessoa, e esta página não trata disso, à exceção do log de pedidos descrito acima, que o servidor guarda para todas as lambdas. Os termos de utilização não permitem usar uma lambda para recolher dados pessoais de outras pessoas. Se encontrares uma que o faça, denuncia-a.', + 'Uma lambda é escrita por quem tem o link de edição, não por nós. O que pede aos visitantes e o que faz com isso é decisão dessa pessoa, e esta página não trata disso, à exceção do log de pedidos descrito acima, que o servidor guarda para todas as lambdas, e das visitas que conta para um proprietário que ligou a análise de visitas. Os termos de utilização não permitem usar uma lambda para recolher dados pessoais de outras pessoas. Se encontrares uma que o faça, denuncia-a.', mailTitle: 'Quando nos escreves', mail: 'Se nos escreveres, para denunciar um abuso ou por qualquer outro motivo, usamos o teu endereço de email e a tua mensagem para te responder e tratar do assunto, e apagamo-los quando deixarem de ser necessários para isso (art. 6.º, n.º 1, alínea f), do RGPD).', storageTitle: 'Cookies e o teu browser', storage: - 'Há um único cookie, chamado lang. Guarda o idioma que escolheste, para que os endereços que não indicam idioma abram nesse idioma, e dura um ano. O armazenamento do próprio browser guarda o modo claro ou escuro, algumas definições das páginas que usas e, no caso dos administradores, o respetivo token. Nada disto é usado para te seguir, e nada vai para mais ninguém: não há ferramentas de análise, nem publicidade, e nada é carregado de outros sites, nem sequer os tipos de letra. Como tudo isto só serve para fazer o que pediste, não é necessário consentimento (§ 25(2), n.º 2, da lei alemã TDDDG).', + 'Há um único cookie, chamado lang. Guarda o idioma que escolheste, para que os endereços que não indicam idioma abram nesse idioma, e dura um ano. O armazenamento do próprio browser guarda o modo claro ou escuro, algumas definições das páginas que usas e, no caso dos administradores, o respetivo token. Nada disto é usado para te seguir, e nada vai para mais ninguém: não há ferramentas de análise deste site, nem publicidade, e nada é carregado de outros sites, nem sequer os tipos de letra. Como tudo isto só serve para fazer o que pediste, não é necessário consentimento (§ 25(2), n.º 2, da lei alemã TDDDG).', hostingTitle: 'Onde fica guardado', hosting: diff --git a/src/Frontend/src/locales/pt/editor/analytics.tsx b/src/Frontend/src/locales/pt/editor/analytics.tsx new file mode 100644 index 0000000..48dbe51 --- /dev/null +++ b/src/Frontend/src/locales/pt/editor/analytics.tsx @@ -0,0 +1,117 @@ +import type { EditorMessages } from '../../en/editor'; + +export const analytics: EditorMessages['analytics'] = { + hint: 'Contadas para você pela plataforma: sem cookies, sem código no seu app e sem nada para os seus visitantes aceitarem.', + loading: 'Lendo as visitas…', + loadFailed: 'Não foi possível ler as visitas.', + + offTitle: 'Veja quem visita o seu app', + offText: + 'Quantas pessoas chegam por dia, quais páginas abrem, quais sites as enviam, de quais países e em quais dispositivos - contado dia a dia, a partir do momento em que você ativa.', + offPrivate: 'Sem cookies e sem banner para os seus visitantes aceitarem', + offAddresses: 'Nenhum endereço é guardado, apenas quantas visitas houve por dia', + offKept: (days) => + days >= 365 + ? 'Guardado por um ano, e tudo é mostrado para você' + : `Guardado por um ano - os últimos ${days} dias são mostrados, tudo com o Premium`, + switchOn: 'Ativar o analytics', + switchedOn: 'O analytics está ativado. As primeiras visitas aparecem em alguns minutos.', + switchOnFailed: 'Não foi possível ativar o analytics.', + demo: 'Os visitantes de uma demo não são contados. Uma cópia sua pode contar os visitantes.', + + range: 'Período', + days: (days) => (days >= 365 ? '12 meses' : `${days} dias`), + premiumRange: 'O Premium mostra até um ano', + + visitors: 'Visitantes', + visitorsHint: 'Pessoas diferentes, cada uma contada uma vez por dia. Mecanismos de busca e outros programas não são visitantes.', + views: 'Visualizações de página', + viewsHint: 'Cada vez que alguém abriu uma página.', + hits: 'Requisições', + hitsHint: 'Tudo o que o seu app recebeu: as páginas e as imagens e scripts que elas carregam.', + versus: (days) => `em relação aos ${days} dias anteriores`, + noChange: 'sem alteração', + inDays: (days) => (days >= 365 ? 'nos últimos 12 meses' : `nos últimos ${days} dias`), + + perDay: { + visitors: 'Visitantes por dia', + views: 'Visualizações de página por dia', + hits: 'Requisições por dia', + }, + perWeek: { + visitors: 'Visitantes por semana', + views: 'Visualizações de página por semana', + hits: 'Requisições por semana', + }, + showValues: 'Mostrar os valores', + showChart: 'Mostrar o gráfico', + day: 'Dia', + week: 'Semana de', + + nobody: 'Ninguém visitou nesses dias.', + nobodyYet: 'Ninguém visitou desde que o analytics foi ativado. As primeiras visitas aparecem em alguns minutos.', + + pages: 'Páginas', + pagesHint: 'As páginas que as pessoas abriram, das mais abertas para as menos abertas.', + requested: 'Mais requisitados', + requestedHint: 'Tudo o que o seu app recebeu, por endereço - ele não tem páginas que as pessoas abrem.', + sources: 'De onde vêm os visitantes', + sourcesHint: 'Os sites que enviaram pessoas para cá, pela página em que estavam. Os mecanismos de busca só informam que foram eles, não o que foi pesquisado.', + direct: 'Digitado, um favorito ou um app', + countries: 'Países', + countriesHint: 'Onde os endereços de internet dos visitantes estão registrados - em geral, mas nem sempre, onde eles estão.', + unknownCountry: 'Desconhecido', + devices: 'Dispositivos', + device: { + desktop: 'Computador', + mobile: 'Celular', + tablet: 'Tablet', + bot: 'Bots', + other: 'Outros', + }, + browsers: 'Navegadores', + systems: 'Sistemas', + crawlers: 'Mecanismos de busca, IA e outros programas', + crawlersHint: 'Programas que passaram por aqui, conforme o que são. Google, Bing, OpenAI, Apple e Perplexity só são citados pelo nome quando a visita veio de um endereço que eles publicam - qualquer um pode dizer que é eles.', + noCrawlers: 'Nenhum mecanismo de busca, IA ou outro programa passou por aqui nesses dias.', + confirmed: (name) => `Veio de um endereço que ${name} publica para os seus rastreadores`, + crawlerKinds: { + 'search-engine': 'Mecanismos de busca', + 'ai-crawler': 'Rastreadores de IA', + 'social-preview': 'Prévias de links em chats e redes sociais', + advertising: 'Publicidade', + seo: 'Ferramentas de SEO', + monitoring: 'Monitoramento', + 'feed-reader': 'Leitores de feeds', + archiver: 'Arquivos da web', + academic: 'Pesquisa', + scanner: 'Scanners de segurança', + 'browser-automation': 'Navegadores automatizados', + 'http-library': 'Scripts e apps', + other: 'Outros programas', + unconfirmed: 'Disse ser um rastreador conhecido, sem confirmação', + }, + other: 'Outros', + showAll: (count) => `Mostrar todos os ${count}`, + showFewer: 'Mostrar menos', + share: (part, whole) => `${part} de ${whole}`, + none: 'Nada ainda.', + + moreKept: (since) => `Contado desde ${since} - o Premium mostra tudo, até um ano.`, + premiumAsk: 'Perguntar sobre o Premium', + premiumSubject: 'Premium para o meu app', + premiumBody: (address) => `Olá,\n\nGostaria de ver todas as análises do meu app: ${address}\n\n`, + + footer: (minutes) => + minutes === 1 ? 'Contado por dia, em UTC. Atualizado a cada minuto.' : `Contado por dia, em UTC. Atualizado a cada ${minutes} minutos.`, + updated: (time) => `Última atualização às ${time}.`, + since: (date) => `Contando desde ${date}.`, + + switchOff: 'Desativar o analytics', + switchOffTitle: 'Desativar o analytics?', + switchOffText: 'A contagem para e tudo o que foi contado até agora é apagado. Isso não pode ser desfeito.', + keep: 'Continuar contando', + switchOffConfirm: 'Desativar e apagar', + switchedOff: 'O analytics está desativado, e o que foi contado foi apagado.', + switchOffFailed: 'Não foi possível desativar o analytics.', +}; diff --git a/src/Frontend/src/locales/pt/editor/frame.tsx b/src/Frontend/src/locales/pt/editor/frame.tsx index bb2c829..f27418a 100644 --- a/src/Frontend/src/locales/pt/editor/frame.tsx +++ b/src/Frontend/src/locales/pt/editor/frame.tsx @@ -16,6 +16,7 @@ export const frame: EditorMessages['frame'] = { history: 'Histórico', deployments: 'Deploys', stats: 'Métricas', + analytics: 'Analytics', logs: 'Logs', code: 'Código', }, diff --git a/src/Frontend/src/locales/pt/editor/index.ts b/src/Frontend/src/locales/pt/editor/index.ts index 980a660..0ee0dc3 100644 --- a/src/Frontend/src/locales/pt/editor/index.ts +++ b/src/Frontend/src/locales/pt/editor/index.ts @@ -1,4 +1,5 @@ import type { EditorMessages } from '../../en/editor'; +import { analytics } from './analytics'; import { change } from './change'; import { clone } from './clone'; import { code } from './code'; @@ -33,6 +34,7 @@ export const editor: EditorMessages = { versions, deployments, stats, + analytics, logs, showcase, openSource, diff --git a/src/Frontend/src/locales/pt/guide.tsx b/src/Frontend/src/locales/pt/guide.tsx index cdb4ee7..4ecae0e 100644 --- a/src/Frontend/src/locales/pt/guide.tsx +++ b/src/Frontend/src/locales/pt/guide.tsx @@ -26,6 +26,7 @@ export const guide: Messages['guide'] = { away: 'Levando seu código embora', git: 'Trabalhando com git', open: 'Publicando o código', + visitors: 'Vendo quem visita', agents: 'Deixando um agente fazer', }, @@ -106,6 +107,7 @@ export const guide: Messages['guide'] = { ['Versões', () => <>O que cada versão mudou, o que foi pedido e a diferença para a anterior. Faça deploy ou volte uma versão por aqui, ou comece um rascunho a partir de qualquer uma delas.], ['Deploys', () => <>O que esteve no ar e quando, e o que tirou do ar.], ['Métricas', () => <>Requisições, falhas, tempos de resposta e os caminhos mais acessados, na última hora ou nas últimas 24 horas.], + ['Analytics', () => <>Quem visita o app, depois que você ativa: visitantes, visualizações de página e requisições por dia, as páginas que eles abrem, os sites que os enviam, os países e dispositivos deles, e os mecanismos de busca e IAs que passaram por aqui.], ['Logs', () => <>As requisições, o que ela imprimiu e o stack trace de qualquer erro, em tempo real.], [ 'Código', @@ -129,7 +131,7 @@ export const guide: Messages['guide'] = { ), editorAside: - 'O tráfego e o log ficam em memória, para acompanhar, não para guardar: reiniciar o servidor zera tudo. As versões e o histórico de deploys ficam salvos.', + 'O tráfego e o log ficam em memória, para acompanhar, não para guardar: reiniciar o servidor zera tudo. As versões, o histórico de deploys e o analytics ficam salvos.', why: (k) => ( <> @@ -495,6 +497,27 @@ export const guide: Messages['guide'] = { ), openAside: 'Tudo o que está no código fica público, inclusive as versões anteriores. Uma chave ou uma senha fica junto das chaves e senhas em Dados, nunca no código, publicado ou não.', + visitors: ((k) => ( + <> + Abra {k.b('Analytics')} no painel de controle e ative para ver quem visita o seu app: quantas pessoas chegam + por dia, quais páginas abrem, quais sites as enviam - um mecanismo de busca, uma postagem, uma newsletter -, de + quais países, em quais dispositivos e navegadores, e quais mecanismos de busca e assistentes de IA leem as + páginas dele. Escolha um número no alto para vê-lo dia a dia. + + )), + visitors2: ((k) => ( + <> + A contagem é feita pela plataforma, na frente do seu app, então ele não precisa de nada para isso: sem cookies, + sem script, sem banner para os seus visitantes aceitarem. Só as contagens por dia são guardadas - nunca um + endereço, e da página de onde alguém veio, apenas o site. Um visitante é uma pessoa, contada uma vez por dia: + mecanismos de busca, rastreadores de IA e outros programas são contados à parte, conforme o que são, e Google, + Bing, OpenAI, Apple e Perplexity só pelo nome quando a visita veio de um endereço que eles publicam - qualquer + um pode dizer que é eles. Um ano fica guardado; um app gratuito mostra os últimos sete dias dele e o {k.b('Premium')} + mostra tudo, inclusive o que foi contado antes. Desativar apaga o que foi contado. + + )), + visitorsAside: + 'Os números são registrados a cada poucos minutos, então as últimas visitas aparecem um pouco depois. Um rascunho nunca é contado: testar uma mudança não é uma visita.', agents: (k) => ( <> diff --git a/src/Frontend/src/locales/pt/privacy.tsx b/src/Frontend/src/locales/pt/privacy.tsx index f0ef311..96d28fc 100644 --- a/src/Frontend/src/locales/pt/privacy.tsx +++ b/src/Frontend/src/locales/pt/privacy.tsx @@ -17,6 +17,11 @@ export const privacy: Messages['privacy'] = { 'É assim que encontramos falhas, descobrimos o que está sobrecarregando o servidor e investigamos os abusos denunciados a nós. O endereço IP também é usado, só em memória, para limitar quantas requisições e quantos pedidos de criação um mesmo visitante pode fazer. Sem esses dados, não dá para responder a uma requisição. A base legal é o nosso legítimo interesse em manter o serviço funcionando e seguro (art. 6º, 1, f, do RGPD).', 'Os administradores podem ver tudo isso. O dono de uma lambda vê o país e o navegador de cada requisição feita à lambda dele, mas não o endereço IP.', ], + analyticsTitle: 'Visitas contadas para o dono de uma lambda', + analytics: [ + 'O dono de uma lambda pode ativar o analytics para ela. Então o servidor conta as visitas a essa lambda por dia: quantas requisições e visualizações de página houve, para quais páginas foram, o site de onde o visitante veio (apenas o nome, não o endereço da página), o país do endereço IP do visitante e o tipo de dispositivo, navegador e sistema operacional que o navegador informa. Só essas contagens são guardadas - não o endereço IP nem qualquer outra coisa que identifique um visitante - num banco de dados no servidor, por um ano. O dono da lambda as vê no editor, e os administradores podem lê-las.', + 'Para contar cada visitante uma vez por dia, o endereço IP e o navegador são transformados num código que não pode ser revertido, com uma chave gerada ao acaso para cada dia e cada lambda. Os códigos e as chaves ficam apenas na memória, nunca são gravados e são esquecidos depois de um dia. Nenhum cookie é definido e nada é armazenado no navegador do visitante. Quando uma visita diz vir do rastreador do Google, Bing, OpenAI, Apple ou Perplexity, o endereço IP dela é comparado, na memória, com os endereços que essas empresas publicam para os seus rastreadores. Para distinguir rastreadores de pessoas, o servidor busca uma vez por dia uma lista pública de rastreadores e esses endereços publicados; nada sobre os visitantes é enviado a lugar nenhum para isso. A base legal é o interesse legítimo do dono em saber como a lambda dele é usada (art. 6(1)(f) do GDPR).', + ], logsTitle: 'Por quanto tempo o log fica guardado', logs: 'O log fica em dois lugares: na memória do servidor, que é apagada toda vez que o servidor reinicia, e na saída de console do servidor, que é apagada toda vez que o servidor é atualizado. Os dois têm tamanho fixo, então cada linha nova empurra a mais antiga para fora. Por isso, quanto tempo uma linha dura depende do movimento do site. Nada do log é arquivado em outro lugar.', @@ -42,14 +47,14 @@ export const privacy: Messages['privacy'] = { lambdasTitle: 'O que uma lambda faz depende do dono dela', lambdas: - 'Uma lambda é escrita por quem tem o link de edição dela, não por nós. O que ela pede aos visitantes e o que faz com isso é decisão dessa pessoa, e esta página não cobre isso, a não ser pelo log de requisições descrito acima, que o servidor mantém para todas as lambdas. Os termos de uso não permitem usar uma lambda para coletar dados pessoais de outras pessoas. Se você encontrar uma que faça isso, denuncie.', + 'Uma lambda é escrita por quem tem o link de edição dela, não por nós. O que ela pede aos visitantes e o que faz com isso é decisão dessa pessoa, e esta página não cobre isso, a não ser pelo log de requisições descrito acima, que o servidor mantém para todas as lambdas, e pelas visitas que ele conta para quem ativou o analytics. Os termos de uso não permitem usar uma lambda para coletar dados pessoais de outras pessoas. Se você encontrar uma que faça isso, denuncie.', mailTitle: 'Quando você escreve para a gente', mail: 'Se você escrever para a gente, para denunciar um abuso ou sobre qualquer outro assunto, usamos o seu e-mail e a sua mensagem para responder e para resolver o que você escreveu. Depois apagamos tudo, quando não for mais necessário para isso (art. 6º, 1, f, do RGPD).', storageTitle: 'Cookies e o seu navegador', storage: - 'Existe um único cookie, chamado lang. Ele guarda o idioma que você escolheu, para que os endereços que não indicam idioma abram nele, e dura um ano. O armazenamento do próprio navegador guarda o tema claro ou escuro, algumas configurações das páginas que você usa e, no caso dos administradores, o token deles. Nada disso é usado para rastrear você, e nada vai para mais ninguém: não há ferramentas de análise, nem anúncios, e nada é carregado de outros sites, nem mesmo fontes. Como tudo isso só serve para fazer o que você pediu, não é preciso consentimento (§ 25(2) nº 2 da lei alemã TDDDG).', + 'Existe um único cookie, chamado lang. Ele guarda o idioma que você escolheu, para que os endereços que não indicam idioma abram nele, e dura um ano. O armazenamento do próprio navegador guarda o tema claro ou escuro, algumas configurações das páginas que você usa e, no caso dos administradores, o token deles. Nada disso é usado para rastrear você, e nada vai para mais ninguém: não há ferramentas de análise neste site, nem anúncios, e nada é carregado de outros sites, nem mesmo fontes. Como tudo isso só serve para fazer o que você pediu, não é preciso consentimento (§ 25(2) nº 2 da lei alemã TDDDG).', hostingTitle: 'Onde tudo fica guardado', hosting: diff --git a/src/Frontend/src/locales/tr/editor/analytics.tsx b/src/Frontend/src/locales/tr/editor/analytics.tsx new file mode 100644 index 0000000..beed438 --- /dev/null +++ b/src/Frontend/src/locales/tr/editor/analytics.tsx @@ -0,0 +1,117 @@ +import type { EditorMessages } from '../../en/editor'; + +export const analytics: EditorMessages['analytics'] = { + hint: 'Platform sizin için sayar: çerez yok, uygulamanızda kod yok ve ziyaretçilerinizin kabul etmesi gereken hiçbir şey yok.', + loading: 'Ziyaretler okunuyor…', + loadFailed: 'Ziyaretler okunamadı.', + + offTitle: 'Uygulamanızı kimlerin ziyaret ettiğini görün', + offText: + 'Her gün kaç kişinin geldiği, hangi sayfaları açtıkları, onları hangi sitelerin yönlendirdiği, hangi ülkelerden ve hangi cihazlarla geldikleri - açtığınız andan itibaren gün gün sayılır.', + offPrivate: 'Çerez yok ve ziyaretçilerinizin kabul etmesi gereken bir banner yok', + offAddresses: 'Hiçbir adres saklanmaz - yalnızca her gün kaç ziyaret olduğu saklanır', + offKept: (days) => + days >= 365 + ? 'Bir yıl saklanır ve tamamı size gösterilir' + : `Bir yıl saklanır - son ${days} gün gösterilir, tamamı Premium ile`, + switchOn: 'Analitiği aç', + switchedOn: 'Analitik açık. İlk ziyaretler birkaç dakika içinde görünür.', + switchOnFailed: 'Analitik açılamadı.', + demo: 'Bir demonun ziyaretçileri sayılmaz. Kendi kopyanız ziyaretçilerini sayabilir.', + + range: 'Zaman aralığı', + days: (days) => (days >= 365 ? '12 ay' : `${days} gün`), + premiumRange: 'Premium bir yıla kadar gösterir', + + visitors: 'Ziyaretçiler', + visitorsHint: 'Farklı kişiler, her biri günde bir kez sayılır. Arama motorları ve diğer programlar ziyaretçi sayılmaz.', + views: 'Sayfa görüntülemeleri', + viewsHint: 'Birinin bir sayfayı her açışı.', + hits: 'İstekler', + hitsHint: 'Uygulamanızdan istenen her şey: sayfaları ve onların yüklediği resimler ile scriptler.', + versus: (days) => `önceki ${days} güne göre`, + noChange: 'değişmedi', + inDays: (days) => (days >= 365 ? 'son 12 ayda' : `son ${days} günde`), + + perDay: { + visitors: 'Günlük ziyaretçiler', + views: 'Günlük sayfa görüntülemeleri', + hits: 'Günlük istekler', + }, + perWeek: { + visitors: 'Haftalık ziyaretçiler', + views: 'Haftalık sayfa görüntülemeleri', + hits: 'Haftalık istekler', + }, + showValues: 'Değerleri göster', + showChart: 'Grafiği göster', + day: 'Gün', + week: 'Hafta başlangıcı', + + nobody: 'Bu günlerde kimse ziyaret etmedi.', + nobodyYet: 'Analitik açıldığından beri kimse ziyaret etmedi. İlk ziyaretler birkaç dakika içinde görünür.', + + pages: 'Sayfalar', + pagesHint: 'İnsanların açtığı sayfalar, en çok açılan başta.', + requested: 'En çok istenenler', + requestedHint: 'Uygulamanızdan istenen her şey, adrese göre - insanların açtığı sayfaları yok.', + sources: 'Ziyaretçiler nereden geliyor', + sourcesHint: 'İnsanları buraya yönlendiren siteler, bulundukları sayfaya göre. Arama motorları yalnızca kendilerinin yönlendirdiğini söyler, neyin arandığını söylemez.', + direct: 'Adres çubuğu, yer imi ya da bir uygulama', + countries: 'Ülkeler', + countriesHint: 'Ziyaretçilerin internet adreslerinin kayıtlı olduğu yer - çoğunlukla, ama her zaman değil, bulundukları yer.', + unknownCountry: 'Bilinmiyor', + devices: 'Cihazlar', + device: { + desktop: 'Bilgisayar', + mobile: 'Telefon', + tablet: 'Tablet', + bot: 'Botlar', + other: 'Diğer', + }, + browsers: 'Tarayıcılar', + systems: 'Sistemler', + crawlers: 'Arama motorları, yapay zekâ ve diğer programlar', + crawlersHint: 'Uğrayan programlar, ne olduklarına göre. Google, Bing, OpenAI, Apple ve Perplexity yalnızca ziyaret, yayımladıkları bir adresten geldiyse adıyla anılır - herkes onlardan biri olduğunu iddia edebilir.', + noCrawlers: 'Bu günlerde hiçbir arama motoru, yapay zekâ ya da başka bir program uğramadı.', + confirmed: (name) => `${name} tarafından tarayıcıları için yayımlanan bir adresten geldi`, + crawlerKinds: { + 'search-engine': 'Arama motorları', + 'ai-crawler': 'Yapay zekâ tarayıcıları', + 'social-preview': 'Sohbetlerde ve sosyal ağlarda link önizlemeleri', + advertising: 'Reklam', + seo: 'SEO araçları', + monitoring: 'İzleme', + 'feed-reader': 'Besleme okuyucular', + archiver: 'Web arşivleri', + academic: 'Araştırma', + scanner: 'Güvenlik tarayıcıları', + 'browser-automation': 'Otomatik tarayıcılar', + 'http-library': 'Script’ler ve uygulamalar', + other: 'Diğer programlar', + unconfirmed: 'Bilinen bir tarayıcı olduğunu iddia etti, doğrulanmadı', + }, + other: 'Diğerleri', + showAll: (count) => `Tümünü göster (${count})`, + showFewer: 'Daha az göster', + share: (part, whole) => `${whole} içinden ${part}`, + none: 'Henüz hiçbir şey yok.', + + moreKept: (since) => `${since} tarihinden beri sayılıyor - Premium, bir yıla kadar tamamını gösterir.`, + premiumAsk: 'Premium hakkında sorun', + premiumSubject: 'Uygulamam için Premium', + premiumBody: (address) => `Merhaba,\n\nUygulamamın tüm analitik verilerini görmek istiyorum: ${address}\n\n`, + + footer: (minutes) => + minutes === 1 ? 'Günlük sayılır, UTC’ye göre. Her dakika güncellenir.' : `Günlük sayılır, UTC’ye göre. Her ${minutes} dakikada bir güncellenir.`, + updated: (time) => `Son güncelleme: ${time}.`, + since: (date) => `Sayım başlangıcı: ${date}.`, + + switchOff: 'Analitiği kapat', + switchOffTitle: 'Analitik kapatılsın mı?', + switchOffText: 'Sayım durur ve şimdiye kadar sayılan her şey silinir. Bu geri alınamaz.', + keep: 'Saymaya devam et', + switchOffConfirm: 'Kapat ve sil', + switchedOff: 'Analitik kapalı ve sayılanlar silindi.', + switchOffFailed: 'Analitik kapatılamadı.', +}; diff --git a/src/Frontend/src/locales/tr/editor/frame.tsx b/src/Frontend/src/locales/tr/editor/frame.tsx index 8422ded..7c90792 100644 --- a/src/Frontend/src/locales/tr/editor/frame.tsx +++ b/src/Frontend/src/locales/tr/editor/frame.tsx @@ -16,6 +16,7 @@ export const frame: EditorMessages['frame'] = { history: 'Geçmiş', deployments: 'Yayın geçmişi', stats: 'İstatistikler', + analytics: 'Analitik', logs: 'Loglar', code: 'Kod', }, diff --git a/src/Frontend/src/locales/tr/editor/index.ts b/src/Frontend/src/locales/tr/editor/index.ts index 58f6f63..e125d5d 100644 --- a/src/Frontend/src/locales/tr/editor/index.ts +++ b/src/Frontend/src/locales/tr/editor/index.ts @@ -1,4 +1,5 @@ import type { EditorMessages } from '../../en/editor'; +import { analytics } from './analytics'; import { change } from './change'; import { clone } from './clone'; import { code } from './code'; @@ -33,6 +34,7 @@ export const editor: EditorMessages = { versions, deployments, stats, + analytics, logs, showcase, openSource, diff --git a/src/Frontend/src/locales/tr/guide.tsx b/src/Frontend/src/locales/tr/guide.tsx index 73d1d0e..053b4f3 100644 --- a/src/Frontend/src/locales/tr/guide.tsx +++ b/src/Frontend/src/locales/tr/guide.tsx @@ -26,6 +26,7 @@ export const guide: Messages['guide'] = { away: 'Kodunuzu alıp gitmek', git: 'git ile üzerinde çalışmak', open: 'Kodu yayımlamak', + visitors: 'Kimlerin ziyaret ettiği', agents: 'İşi bir ajana bırakmak', }, @@ -106,6 +107,7 @@ export const guide: Messages['guide'] = { ['Sürümler', () => <>Her sürümün neyi değiştirdiği, ne istendiği ve bir öncekinden farkı. Buradan yayına alabilir, eski bir sürüme dönebilir ya da herhangi bir sürümden bir taslak başlatabilirsiniz.], ['Yayın geçmişi', () => <>Ne zaman neyin yayında olduğu ve neden yayından kalktığı.], ['İstatistikler', () => <>Son bir saatin ya da günün istekleri, hataları, yanıt süreleri ve en çok istenen yolları.], + ['Analitik', () => <>Uygulamanızı kimlerin ziyaret ettiği, açtığınızda: günlük ziyaretçiler, sayfa görüntülemeleri ve istekler, açtıkları sayfalar, onları yönlendiren siteler, ülkeleri ve cihazları, ayrıca uğrayan arama motorları ve yapay zekâlar.], ['Loglar', () => <>İstekler, lambdanın yazdırdıkları ve ters giden her şeyin stack trace’i, anında.], [ 'Kod', @@ -129,7 +131,7 @@ export const guide: Messages['guide'] = { ), editorAside: - 'Trafik ve log bellekte tutulur. Saklamak için değil, izlemek içindir: sunucu yeniden başlarsa sıfırdan başlarlar. Sürümler ve yayın geçmişi ise kalıcı olarak saklanır.', + 'Trafik ve log bellekte tutulur. Saklamak için değil, izlemek içindir: sunucu yeniden başlarsa sıfırdan başlarlar. Sürümler, yayın geçmişi ve analitik ise kalıcı olarak saklanır.', why: (k) => ( <> @@ -499,6 +501,28 @@ export const guide: Messages['guide'] = { ), openAside: 'Koddaki her şey herkese açık hâle gelir, önceki sürümler de dahil. Bir anahtarın ya da parolanın yeri, Veriler altındaki anahtarlar ve parolalardır; yayımlansın ya da yayımlanmasın, asla kod değildir.', + visitors: ((k) => ( + <> + Uygulamanızı kimlerin ziyaret ettiğini görmek için kontrol panelinde {k.b('Analitik')} bölümünü açın ve + açın: her gün kaç kişinin geldiğini, hangi sayfaları açtığını, onları hangi sitelerin - bir arama motoru, bir + paylaşım, bir bülten - yönlendirdiğini, hangi ülkelerden, hangi cihaz ve tarayıcılarla geldiğini ve hangi arama + motorlarının ve yapay zekâ asistanlarının sayfalarını okuduğunu görürsünüz. Gün gün görmek için üstten bir + rakam seçin. + + )), + visitors2: ((k) => ( + <> + Sayımı platform yapar, uygulamanızın önünde; bu yüzden uygulamanızın bunun için hiçbir şeye ihtiyacı yoktur: + çerez yok, script yok, ziyaretçilerinizin kabul etmesi gereken banner yok. Yalnızca günlük sayılar saklanır - hiçbir + adres saklanmaz, birinin geldiği sayfadan da yalnızca site saklanır. Ziyaretçi bir kişidir ve günde bir kez + sayılır: arama motorları, yapay zekâ tarayıcıları ve diğer programlar ne olduklarına göre ayrıca sayılır; Google, + Bing, OpenAI, Apple ve Perplexity ise yalnızca ziyaret, yayımladıkları bir adresten geldiyse adıyla sayılır - herkes + onlardan biri olduğunu iddia edebilir. Bir yıl saklanır; ücretsiz bir uygulama bunun son yedi gününü, + {k.b('Premium')} ise öncesinde sayılanlar dahil tamamını gösterir. Analitiği kapatmak, sayılanları siler. + + )), + visitorsAside: + 'Rakamlar birkaç dakikada bir yazılır, bu yüzden son ziyaretler biraz geç görünür. Bir taslak asla sayılmaz: bir değişikliği denemek ziyaret değildir.', agents: (k) => ( <> diff --git a/src/Frontend/src/locales/tr/privacy.tsx b/src/Frontend/src/locales/tr/privacy.tsx index ff70997..9056bc9 100644 --- a/src/Frontend/src/locales/tr/privacy.tsx +++ b/src/Frontend/src/locales/tr/privacy.tsx @@ -19,6 +19,11 @@ export const privacy: Messages['privacy'] = { 'Hataları bu sayede buluruz. Sunucuyu aşırı yükleyen şeyi bu sayede tespit ederiz. Bize bildirilen kötüye kullanımın izini de bu sayede süreriz. IP adresi ayrıca, yalnızca bellekte, bir ziyaretçinin kaç istek gönderip kaç uygulama oluşturabileceğini sınırlamak için kullanılır. Bu bilgiler olmadan bir isteğe yanıt verilemez. Hukuki dayanak, hizmeti işletme ve güvende tutma konusundaki meşru menfaatimizdir (GDPR md. 6/1-f).', 'Yöneticiler bunların hepsini görebilir. Bir lambdanın sahibi, kendi lambdasına gelen her isteğin ülkesini ve tarayıcısını görür ama IP adresini göremez.', ], + analyticsTitle: 'Bir lambdanın sahibi için sayılan ziyaretler', + analytics: [ + 'Bir lambdanın sahibi, o lambda için analitiği açabilir. Bu durumda sunucu, o lambdaya yapılan ziyaretleri güne göre sayar: kaç istek ve sayfa görüntülemesi olduğu, hangi sayfalar için olduğu, ziyaretçinin geldiği site (yalnızca adı, sayfanın adresi değil), ziyaretçinin IP adresinin ülkesi ve tarayıcının bildirdiği cihaz, tarayıcı ve işletim sistemi türü. Yalnızca bu sayılar saklanır - IP adresi ve bir ziyaretçiyi belirten başka hiçbir şey saklanmaz - sunucudaki bir veritabanında, bir yıl boyunca. Lambdanın sahibi bunları editöründe görür, yöneticiler de okuyabilir.', + 'Her ziyaretçiyi günde bir kez saymak için IP adresi ve tarayıcı, her gün ve her lambda için rastgele üretilen bir anahtarla, geri çevrilemeyen bir koda dönüştürülür. Kodlar ve anahtarlar yalnızca bellekte tutulur, hiçbir zaman yazılmaz ve bir gün sonra unutulur. Hiçbir çerez ayarlanmaz ve ziyaretçinin tarayıcısında hiçbir şey saklanmaz. Bir ziyaret Google, Bing, OpenAI, Apple veya Perplexity tarayıcısından geldiğini söylüyorsa, IP adresi bellekte, bunların tarayıcıları için yayımladığı adreslerle karşılaştırılır. Tarayıcıları insanlardan ayırmak için sunucu, herkese açık bir tarayıcı listesini ve bu yayımlanmış adresleri günde bir kez indirir; bunun için ziyaretçilerle ilgili hiçbir şey hiçbir yere gönderilmez. Hukuki dayanak, sahibin lambdasının nasıl kullanıldığını bilmeye yönelik meşru menfaatidir (GDPR madde 6(1)(f)).', + ], logsTitle: 'Log ne kadar süre saklanır', logs: 'Log iki yerde tutulur: sunucunun belleğinde ve sunucunun konsol çıktısında. Bellek sunucu her yeniden başladığında boşalır, konsol çıktısı da sunucu her güncellendiğinde silinir. İkisinin de boyutu sabittir, yani her yeni satır en eski satırı dışarı iter. Bu yüzden bir satırın ne kadar kalacağı, sitenin ne kadar yoğun olduğuna bağlıdır. Logdaki hiçbir şey bir arşive kopyalanmaz.', @@ -44,14 +49,14 @@ export const privacy: Messages['privacy'] = { lambdasTitle: 'Bir lambdanın ne yaptığına sahibi karar verir', lambdas: - 'Bir lambdayı biz değil, editör linkine sahip olan kişi yazar. Lambdanın ziyaretçilerinden ne istediği ve bununla ne yaptığı o kişiye bağlıdır. Bu sayfa bunları kapsamaz. Tek istisna, yukarıda anlatılan ve sunucunun her lambda için tuttuğu istek logudur. Kullanım koşulları, bir lambdanın başkaları hakkında kişisel veri toplamak için kullanılmasına izin vermez. Bunu yapan bir lambdaya rastlarsanız lütfen bildirin.', + 'Bir lambdayı biz değil, editör linkine sahip olan kişi yazar. Lambdanın ziyaretçilerinden ne istediği ve bununla ne yaptığı o kişiye bağlıdır. Bu sayfa bunları kapsamaz. İstisnalar, yukarıda anlatılan ve sunucunun her lambda için tuttuğu istek logu ile analitiği açan bir sahip için saydığı ziyaretlerdir. Kullanım koşulları, bir lambdanın başkaları hakkında kişisel veri toplamak için kullanılmasına izin vermez. Bunu yapan bir lambdaya rastlarsanız lütfen bildirin.', mailTitle: 'Bize yazdığınızda', mail: 'Kötüye kullanımı bildirmek ya da başka bir konu için bize yazarsanız, e-posta adresinizi ve mesajınızı size yanıt vermek ve yazdığınız konuyla ilgilenmek için kullanırız. Bu iş için artık gerekmediklerinde onları sileriz (GDPR md. 6/1-f).', storageTitle: 'Çerezler ve tarayıcınız', storage: - 'Tek bir çerez var, adı lang. Seçtiğiniz dili hatırlar, böylece dil belirtmeyen adresler o dilde açılır. Bir yıl geçerlidir. Tarayıcının kendi depolama alanı ise açık ya da koyu temayı, kullandığınız sayfaların birkaç ayarını ve yöneticiler için onların erişim anahtarını hatırlar. Bunların hiçbiri sizi takip etmek için kullanılmaz ve hiçbiri başka birine gitmez: analiz aracı yok, reklam yok, başka sitelerden hiçbir şey yüklenmez, yazı tipleri bile. Bunların hepsi yalnızca sizin istediğiniz şeyi yaptığı için onay gerekmez (Alman TDDDG yasası, § 25(2) no. 2).', + 'Tek bir çerez var, adı lang. Seçtiğiniz dili hatırlar, böylece dil belirtmeyen adresler o dilde açılır. Bir yıl geçerlidir. Tarayıcının kendi depolama alanı ise açık ya da koyu temayı, kullandığınız sayfaların birkaç ayarını ve yöneticiler için onların erişim anahtarını hatırlar. Bunların hiçbiri sizi takip etmek için kullanılmaz ve hiçbiri başka birine gitmez: bu sitenin ziyaret analizi yok, reklam yok, başka sitelerden hiçbir şey yüklenmez, yazı tipleri bile. Bunların hepsi yalnızca sizin istediğiniz şeyi yaptığı için onay gerekmez (Alman TDDDG yasası, § 25(2) no. 2).', hostingTitle: 'Nerede saklanır', hosting: diff --git a/src/Frontend/src/pages/Editor.tsx b/src/Frontend/src/pages/Editor.tsx index bc5739d..1223c63 100644 --- a/src/Frontend/src/pages/Editor.tsx +++ b/src/Frontend/src/pages/Editor.tsx @@ -26,6 +26,7 @@ import { } from '../components/Icons'; import { useToast } from '../components/Toast'; import { AdminTab } from '../control/AdminTab'; +import { AnalyticsTab } from '../control/AnalyticsTab'; import { useAgent } from '../control/agent'; import { ChangeTab } from '../control/ChangeTab'; import { CodeTab } from '../control/CodeTab'; @@ -60,7 +61,7 @@ interface Props { } type SectionId = - | 'overview' | 'docs' | 'change' | 'features' | 'code' | 'tests' | 'data' | 'versions' | 'deployments' | 'stats' + | 'overview' | 'docs' | 'change' | 'features' | 'code' | 'tests' | 'data' | 'versions' | 'deployments' | 'stats' | 'analytics' | 'logs' | 'showcase' | 'source' | 'domain' | 'history' | 'admin'; /* @@ -95,7 +96,7 @@ const GROUPS: { id: GroupId | null; sections: SectionId[] }[] = [ { id: 'sharing', sections: ['showcase', 'source', 'domain'] }, { id: 'develop', sections: ['change', 'features', 'code', 'tests'] }, { id: 'program', sections: ['data', 'versions'] }, - { id: 'run', sections: ['deployments', 'stats', 'logs'] }, + { id: 'run', sections: ['deployments', 'stats', 'analytics', 'logs'] }, ]; const SECTIONS: SectionId[] = GROUPS.flatMap((group) => group.sections); @@ -127,8 +128,11 @@ const WRITABLE = ['code', 'docs', 'tests']; * what it saved, and the keys it uses - and not while there is nothing in it. * Publishing its code is there, next to the showcase: it is the owner's to * decide whoever built the app, and said in the same words in both views. + * And who visits it, after how people find it: an owner who never reads the + * code wants to know who comes as much as one who does - it is about the + * audience, not the program, so unlike the figures it is in both. */ -const SIMPLE_SECTIONS: SectionId[] = ['overview', 'docs', 'change', 'features', 'history', 'data', 'showcase', 'source', 'domain']; +const SIMPLE_SECTIONS: SectionId[] = ['overview', 'docs', 'change', 'features', 'history', 'data', 'showcase', 'source', 'domain', 'analytics']; /** A draft, in the simple view, is what it does and where to try it - not its code, its data or its log. */ const SIMPLE_FEATURE_VIEWS: FeatureView[] = ['overview']; @@ -506,7 +510,7 @@ export function Editor({ theme }: Props) { // of them is nothing to look at until then const absent = (id: SectionId) => (hidden && id === 'domain') - || (demo && (id === 'showcase' || id === 'source' || id === 'change' || id === 'features')) + || (demo && (id === 'showcase' || id === 'source' || id === 'change' || id === 'features' || id === 'analytics')) || (id === 'features' && section !== 'features' && (features?.length ?? 0) === 0) // the simple view has the data once the app keeps any, or waits for a key || (id === 'data' && simple && section !== 'data' && !keeps) @@ -1054,6 +1058,8 @@ export function Editor({ theme }: Props) { ) : section === 'deployments' ? ( + ) : section === 'analytics' ? ( + ) : section === 'stats' ? ( ) : section === 'logs' ? ( diff --git a/src/Frontend/src/pages/Guide.tsx b/src/Frontend/src/pages/Guide.tsx index cdb57fb..6b0ccda 100644 --- a/src/Frontend/src/pages/Guide.tsx +++ b/src/Frontend/src/pages/Guide.tsx @@ -34,6 +34,7 @@ const PARTS = [ 'away', 'git', 'open', + 'visitors', 'agents', ] as const; @@ -436,6 +437,12 @@ return Layout.Create().Add("chat", socket);`} /> +
+

{said.visitors(k)}

+

{said.visitors2(k)}

+ +
+

{said.agents(k)}

{said.agents2(k)}

diff --git a/src/Frontend/src/pages/Privacy.tsx b/src/Frontend/src/pages/Privacy.tsx index 4411744..5a29d91 100644 --- a/src/Frontend/src/pages/Privacy.tsx +++ b/src/Frontend/src/pages/Privacy.tsx @@ -47,6 +47,12 @@ export function Privacy() { ))} + + {part.analytics.map((paragraph) => ( +

{paragraph}

+ ))} + +

{part.logs}

 diff --git a/src/GenHTTP.Lambda/Api/AdminResource.cs b/src/GenHTTP.Lambda/Api/AdminResource.cs index 5234dbb..9d6e810 100644 --- a/src/GenHTTP.Lambda/Api/AdminResource.cs +++ b/src/GenHTTP.Lambda/Api/AdminResource.cs @@ -271,10 +271,10 @@ public LimitsModel ChangeLimits(LimitsModel body) } private static TierLimits Tier(TierLimitsModel model) - => new(model.BuildBytes, model.DataBytes, model.Versions, model.Features); + => new(model.BuildBytes, model.DataBytes, model.Versions, model.Features, model.AnalyticsDays); private static TierLimitsModel Describe(TierLimits limits) - => new(limits.BuildBytes, limits.DataBytes, limits.Versions, limits.Features); + => new(limits.BuildBytes, limits.DataBytes, limits.Versions, limits.Features, limits.AnalyticsDays); private static LimitsModel Describe(ProductLimits limits) => new(Describe(limits.Free), Describe(limits.Premium), (int)limits.OfflineAfter.TotalHours, (int)limits.RemovedAfter.TotalHours, diff --git a/src/GenHTTP.Lambda/Api/Infrastructure/ApiLayout.cs b/src/GenHTTP.Lambda/Api/Infrastructure/ApiLayout.cs index 98eb2bd..d478492 100644 --- a/src/GenHTTP.Lambda/Api/Infrastructure/ApiLayout.cs +++ b/src/GenHTTP.Lambda/Api/Infrastructure/ApiLayout.cs @@ -43,6 +43,7 @@ public static IHandlerBuilder Create(LambdaOptions options) .Add(Resource(mode)) .Add(Resource(mode)) .Add(Resource(mode)) + .Add(Resource(mode)) .Add(Resource(mode)) .Add(Resource(mode)) .AddScalar(title: "GenHTTP Lambda API") diff --git a/src/GenHTTP.Lambda/Api/LambdaAnalyticsResource.cs b/src/GenHTTP.Lambda/Api/LambdaAnalyticsResource.cs new file mode 100644 index 0000000..65d20f3 --- /dev/null +++ b/src/GenHTTP.Lambda/Api/LambdaAnalyticsResource.cs @@ -0,0 +1,61 @@ +using GenHTTP.Lambda.Api.Model; +using GenHTTP.Lambda.Services.Analytics; + +using GenHTTP.Modules.Reflection; +using GenHTTP.Modules.Webservices; + +using Microsoft.Extensions.Logging; + +namespace GenHTTP.Lambda.Api; + +/// +/// How a lambda's visitors reach it, for its owner. +/// +/// +/// Behind the editor key like everything else about a lambda: what is +/// counted of its visitors is the owner's to see, and nobody else's. +/// Counting is the owner's choice and never starts on its own. +/// +public sealed class LambdaAnalyticsResource(IAnalyticsService analytics, ILogger logger) +{ + + /// + /// Whether the lambda's visits are counted, and what was counted over the + /// last days: by day, page, referrer, country, device, browser, system and + /// crawler. + /// + /// + /// Counted by day in UTC, and written down every few minutes, so the + /// visits of the last minutes are not in it yet. A year is kept of every + /// lambda; a free one shows its owner the last seven days of it, a premium + /// one all of it. + /// + /// How many days, today included - left out, as many as its tier shows, which is also the most + [ResourceMethod("lambdas/:privateKey/analytics")] + public AnalyticsResponse Get(string privateKey, int? days = null) => AnalyticsResponse.Of(analytics.Get(privateKey, days)); + + /// + /// Starts counting the lambda's visits, from the next request on. + /// + [ResourceMethod(Method.Put, "lambdas/:privateKey/analytics")] + public AnalyticsResponse Put(string privateKey) + { + var enabled = analytics.Enable(privateKey); + + logger.LogInformation("Switched on analytics of lambda {Lambda}", enabled.PublicKey); + + return AnalyticsResponse.Of(enabled); + } + + /// + /// Stops counting the lambda's visits, and deletes what was counted. + /// + [ResourceMethod(Method.Delete, "lambdas/:privateKey/analytics")] + public void Delete(string privateKey) + { + var publicKey = analytics.Disable(privateKey); + + logger.LogInformation("Switched off analytics of lambda {Lambda}", publicKey); + } + +} diff --git a/src/GenHTTP.Lambda/Api/Mcp/McpTools.cs b/src/GenHTTP.Lambda/Api/Mcp/McpTools.cs index 0b7315d..4c1c3b4 100644 --- a/src/GenHTTP.Lambda/Api/Mcp/McpTools.cs +++ b/src/GenHTTP.Lambda/Api/Mcp/McpTools.cs @@ -4,6 +4,7 @@ using GenHTTP.Lambda.Api.Model; using GenHTTP.Lambda.Configuration; using GenHTTP.Lambda.Data.Entities; +using GenHTTP.Lambda.Services.Analytics; using GenHTTP.Lambda.Services.Data; using GenHTTP.Lambda.Services.Databases; using GenHTTP.Lambda.Services.Deployment.Compilation; @@ -64,7 +65,8 @@ namespace GenHTTP.Lambda.Api.Mcp; /// for what is left out. /// public sealed class McpTools(IMetaService meta, IWorkspaceService workspace, IDataService data, IFeatureService features, ISecretService secrets, - IDatabaseService databases, IShowcaseService showcases, ISourceService sources, ITelemetryService telemetry, ILogBook book, + IDatabaseService databases, IShowcaseService showcases, ISourceService sources, IAnalyticsService analytics, + ITelemetryService telemetry, ILogBook book, LambdaOptions options, ILimitsService tiers, ILambdaAddresses addresses, ILogger logger) { @@ -472,6 +474,20 @@ public JsonArray Describe() => ["required"] = new JsonArray("privateKey") }), + Tool("analytics", "Read who visits a lambda", Effect.Replace, + "How people reach a lambda, as its owner sees it in the editor: visitors, pages opened and requests per day, and the pages, referring sites, countries, devices, browsers, operating systems and crawlers (search engines, AIs, link previews) behind them. Counted by day in UTC and written down every few minutes; a year is kept, of which a free lambda shows the last 7 days and a premium one all. Off until switched on - nothing is counted before. enable: true switches it on, only when the user asks; switching off deletes what was counted and is the owner's to do in the editor. Use it to answer what the user asks about their visitors, or to see which pages matter before changing them.", + new JsonObject + { + ["type"] = "object", + ["properties"] = new JsonObject + { + ["privateKey"] = Field("string", "The editor key."), + ["days"] = Field("integer", "How many days, today included. Left out, as many as the lambda's tier shows, which is also the most."), + ["enable"] = Field("boolean", "Switch analytics on, counting from the next request - only when the user asked for it.") + }, + ["required"] = new JsonArray("privateKey") + }), + Tool("list_demos", "List the demos", Effect.Read, "Demos this platform keeps online, each a finished lambda showing one way to build something: a REST API over records, registration and login, a websocket game, uploads, live updates. Their keys are public and read only: read the closest one with read_lambda (and list_files, read_logs) before writing similar code. create_lambda with a demo's id as template starts from a copy.", new JsonObject { ["type"] = "object", ["properties"] = new JsonObject() }), @@ -517,6 +533,7 @@ public async ValueTask CallAsync(string name, JsonObject arguments, "delete_secret" => DeleteSecret(arguments), "showcase" => Showcase(arguments, origin), "open_source" => OpenSource(arguments, origin), + "analytics" => Analytics(arguments), "list_demos" => ListDemos(), "platform_guide" => ReadGuide(origin), _ => McpProtocol.Refuse($"There is no tool called '{name}'.") @@ -1991,6 +2008,54 @@ private JsonObject Showcase(JsonObject arguments, string origin) /// One tool, like the showcase, for the same reason: it is used rarely, /// and every tool listed is read by every agent in every conversation. /// + private JsonObject Analytics(JsonObject arguments) + { + var privateKey = Required(arguments, "privateKey"); + + var days = Number(arguments, "days"); + + LambdaAnalytics found; + + if (Flag(arguments, "enable") == true) + { + found = analytics.Enable(privateKey); + + logger.LogInformation("Switched on analytics of lambda {Lambda}", found.PublicKey); + + if (days != null) + { + found = analytics.Get(privateKey, days); + } + } + else + { + found = analytics.Get(privateKey, days); + + logger.LogInformation("Read analytics of lambda {Lambda} days {Days}", found.PublicKey, found.Shown); + } + + if (found.Since == null) + { + return McpProtocol.Say(new + { + ok = true, + enabled = false, + note = "Analytics are off for this lambda, so nothing about its visitors was counted. The owner switches them on in the editor (Analytics); pass enable: true only when the user asks for it." + }); + } + + var minutes = Math.Max(1, (int)found.Interval.TotalMinutes); + + return McpProtocol.Say(new + { + ok = true, + analytics = AnalyticsResponse.Of(found), + note = found.Shown < found.Kept && found.Earliest is { } earliest && earliest < found.From + ? $"Counted by day in UTC, written down every {minutes} minutes. More is kept than this tier shows: the owner sees all of the year with premium." + : $"Counted by day in UTC, written down every {minutes} minutes - the visits of the last minutes are not in it yet. Views are pages opened, hits every request; visitors are counted once a day." + }); + } + private JsonObject OpenSource(JsonObject arguments, string origin) { var privateKey = Required(arguments, "privateKey"); @@ -2431,6 +2496,12 @@ private JsonObject Guide(string origin) => McpProtocol.Say(new licenses = SourceLicenses.All.Select(l => new { l.Id, l.Name, kind = l.Kind.ToString() }), startingFromOne = "A published source downloads as a .NET project, not as a lambda - or is cloned with git from /source/{publicKey}.git. To make a lambda of it with git: create_lambda, clone its gitUrl, copy the lambda's files - Project.cs, the other .cs files at the top, resources/ and every other folder and file of its code - over its own, commit and push. Without git: Project.cs holds the snippet as the body of Build() or BuildAsync() with the types after the class - put the body back into lambda.cs with those types below it; every other file of the lambda keeps its name and folder; Program.cs, the .csproj, Platform/, the Dockerfile and the ignore files are the platform's and stay behind. Keep to the terms of its LICENSE." }, + analytics = new + { + what = "The owner can have a lambda's visits counted: visitors, pages opened and requests per day, and the pages, referring sites, countries, devices, browsers, operating systems and crawlers behind them. They read it in the editor, and the analytics tool reads the same.", + when = "Off until the owner switches it on, in the editor or by asking you (analytics with enable: true). Switching off deletes what was counted, and is theirs to do. A feature's preview is never counted.", + how = "Counted by the platform, in front of the lambda, by day in UTC and written down every few minutes; a year is kept, of which a free lambda shows the last 7 days and a premium one all. Only the host of a referring site is kept, never an address, and a visitor - a person, never a program - is told apart for a day by a hash that is never stored. Crawlers are counted by their kind from a maintained list; Google, Bing, OpenAI, Apple and Perplexity by name only where the visit came from an address they publish. The lambda needs no code for it - do not build a counter of your own for what this counts." + }, afterDeploying = new { what = "read_logs shows what the lambda has been doing: each request and its status, what it printed, and any exception it threw with the stack trace, plus its traffic over the last hour and day. With feature, what the feature's preview did.", diff --git a/src/GenHTTP.Lambda/Api/Model/Admin.cs b/src/GenHTTP.Lambda/Api/Model/Admin.cs index bc7deeb..6796ef9 100644 --- a/src/GenHTTP.Lambda/Api/Model/Admin.cs +++ b/src/GenHTTP.Lambda/Api/Model/Admin.cs @@ -109,4 +109,5 @@ int BuildsPerDay /// The room its data may take: its database and its workspace together /// How many of its versions are kept /// How many features it may have open at once -public sealed record TierLimitsModel(long BuildBytes, long DataBytes, int Versions, int Features); +/// How many days of its analytics its owner is shown, of the year that is kept +public sealed record TierLimitsModel(long BuildBytes, long DataBytes, int Versions, int Features, int AnalyticsDays); diff --git a/src/GenHTTP.Lambda/Api/Model/Analytics.cs b/src/GenHTTP.Lambda/Api/Model/Analytics.cs new file mode 100644 index 0000000..7c9bc78 --- /dev/null +++ b/src/GenHTTP.Lambda/Api/Model/Analytics.cs @@ -0,0 +1,125 @@ +using GenHTTP.Lambda.Services.Analytics; + +namespace GenHTTP.Lambda.Api.Model; + +/// +/// How a lambda's visitors reach it, as its owner is shown. +/// +/// Whether its visits are counted; nothing below is there while they are not +/// When counting was switched on +/// Free, Premium or Demo - what decides how many days are shown +/// How many days this answer covers, today included +/// The most days its tier shows +/// How many days are kept of every lambda, whatever its tier shows +/// The first day of the answer, in UTC +/// The last day of the answer: today, in UTC +/// The first day anything was counted on - before where more is kept than shown +/// The days of the answer, summed +/// The same number of days before, summed - only where the tier shows those as well +/// Each day of the answer, oldest first, days without visits included +/// The paths that were answered, the most viewed first - a page opened is a view, every request a hit +/// The sites visitors came from, by host +/// The countries their addresses were allocated to, by ISO code; empty where not known +/// desktop, mobile, tablet, bot or other +/// The browsers of people - bots are not counted here +/// The operating systems of people +/// Whose bots came: search engines, AIs, chats showing a preview +/// When visits were last written down; visits since then are not in this answer yet +/// How often they are written down +/// +/// The operators whose crawlers are confirmed by the addresses they publish - Google, Bing, OpenAI, Apple, Perplexity - +/// as far as those could be fetched. In their names stand for confirmed visits, +/// "unconfirmed" for claims to be theirs from elsewhere, and every other value for a kind of crawler. +/// +public sealed record AnalyticsResponse( + bool Enabled, + DateTime? Since, + string Tier, + int Days, + int Window, + int Kept, + DateOnly From, + DateOnly To, + DateOnly? Earliest, + AnalyticsTotalsResponse? Totals, + AnalyticsTotalsResponse? Previous, + IReadOnlyList? Daily, + IReadOnlyList? Pages, + IReadOnlyList? Referrers, + IReadOnlyList? Countries, + IReadOnlyList? Devices, + IReadOnlyList? Browsers, + IReadOnlyList? Systems, + IReadOnlyList? Crawlers, + DateTime? Updated, + int IntervalSeconds, + IReadOnlyList ConfirmedCrawlers +) +{ + + public static AnalyticsResponse Of(LambdaAnalytics analytics) + { + var report = analytics.Report; + + IReadOnlyList? Of(AnalyticsDimension dimension) + => report?.Breakdowns[dimension].Select(v => new AnalyticsValueResponse(v.Value, v.Hits, v.Views)).ToList(); + + return new AnalyticsResponse( + analytics.Since != null, + analytics.Since, + analytics.Tier, + analytics.Shown, + analytics.Window, + analytics.Kept, + analytics.From, + analytics.To, + analytics.Earliest, + report == null ? null : AnalyticsTotalsResponse.Of(report.Totals), + analytics.Previous == null ? null : AnalyticsTotalsResponse.Of(analytics.Previous), + report?.Days.Select(d => new AnalyticsDayResponse(d.Date, AnalyticsTotalsResponse.Of(d.Totals))).ToList(), + Of(AnalyticsDimension.Pages), + Of(AnalyticsDimension.Referrers), + Of(AnalyticsDimension.Countries), + Of(AnalyticsDimension.Devices), + Of(AnalyticsDimension.Browsers), + Of(AnalyticsDimension.Systems), + Of(AnalyticsDimension.Crawlers), + analytics.Updated, + (int)analytics.Interval.TotalSeconds, + analytics.Confirmed + ); + } + +} + +/// +/// What was counted over some days, summed. +/// +/// Every request answered - pages, the files they load, calls of an API +/// The requests that were somebody opening a page +/// Different visitors, each counted once a day - summed over days, one who came back is counted again +/// Answered with a server error +/// Answered with a client error, not found included +/// What the responses carried, where that was known +/// The mean time to answer a request +public sealed record AnalyticsTotalsResponse(long Hits, long Views, long Visitors, long Failed, long Rejected, long Bytes, double AverageMillis) +{ + + public static AnalyticsTotalsResponse Of(AnalyticsTotals totals) + => new(totals.Hits, totals.Views, totals.Visitors, totals.Failed, totals.Rejected, totals.Bytes, + totals.Hits > 0 ? Math.Round(totals.Millis / totals.Hits, 2) : 0); + +} + +/// +/// One day, in UTC. +/// +public sealed record AnalyticsDayResponse(DateOnly Date, AnalyticsTotalsResponse Totals); + +/// +/// One value of what visits are counted by. +/// +/// The path, the host, the country code or the name; "(other)" gathers what did not fit in a day +/// Requests it was part of +/// Pages opened it was part of +public sealed record AnalyticsValueResponse(string Value, long Hits, long Views); diff --git a/src/GenHTTP.Lambda/Application.cs b/src/GenHTTP.Lambda/Application.cs index 5015b4e..b2b6ec2 100644 --- a/src/GenHTTP.Lambda/Application.cs +++ b/src/GenHTTP.Lambda/Application.cs @@ -7,6 +7,7 @@ using GenHTTP.Lambda.Configuration; using GenHTTP.Lambda.Data; using GenHTTP.Lambda.Infrastructure; +using GenHTTP.Lambda.Services.Analytics; using GenHTTP.Lambda.Services.Background; using GenHTTP.Lambda.Services.Deployment; using GenHTTP.Lambda.Services.Diagnostics; @@ -25,6 +26,7 @@ using GenHTTP.Lambda.Services.Source; using GenHTTP.Lambda.Services.Storage; using GenHTTP.Lambda.Services.Telemetry; +using GenHTTP.Lambda.Services.Visits; using GenHTTP.Lambda.Services.Workspace; using GenHTTP.Lambda.Web; @@ -187,6 +189,15 @@ private static ServiceProvider BuildServices(LambdaOptions options, ILoggerFacto services.AddSingleton(); services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(provider => provider.GetRequiredService()); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); + /* * The concerns that need nothing but services, resolved per request * from the scope the outermost concern opens (see Configure). @@ -197,12 +208,16 @@ private static ServiceProvider BuildServices(LambdaOptions options, ILoggerFacto services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); + services.AddSingleton(); services.AddSingleton(); + services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); services.AddSingleton(); + services.AddSingleton(); + services.AddSingleton(); services.AddSingleton(); return services.BuildServiceProvider(); diff --git a/src/GenHTTP.Lambda/Configuration/LambdaOptions.cs b/src/GenHTTP.Lambda/Configuration/LambdaOptions.cs index ecd5953..ad47049 100644 --- a/src/GenHTTP.Lambda/Configuration/LambdaOptions.cs +++ b/src/GenHTTP.Lambda/Configuration/LambdaOptions.cs @@ -288,6 +288,25 @@ public sealed record LambdaOptions /// public int TelemetrySamples { get; init; } = 2880; + /// + /// How often the visits counted for the owners' analytics are written + /// down. Visits not yet written are lost when the server stops. + /// + public TimeSpan AnalyticsInterval { get; init; } = TimeSpan.FromMinutes(5); + + /// + /// Whether the list of crawlers and the addresses Google, Bing, OpenAI, + /// Apple and Perplexity crawl from are fetched, to tell crawlers from people + /// and confirm the crawlers that say whose they are. Off, the list of the + /// release is used and no crawler is confirmed. + /// + public bool CrawlerLists { get; init; } = true; + + /// + /// How often they are fetched again. + /// + public TimeSpan CrawlerRefresh { get; init; } = TimeSpan.FromDays(1); + /// /// How many log lines are kept for the panel to read back. /// @@ -506,6 +525,11 @@ public sealed record LambdaOptions public string ConnectionString => $"Data Source={DatabaseFile};Default Timeout=30;Pooling=True"; + /// + /// The database the owners' analytics are kept in, apart from the platform's. + /// + public string AnalyticsFile => Path.Combine(DataDirectory, "analytics.db"); + public string CodeDirectory => Path.Combine(DataDirectory, "code"); public string WorkspaceDirectory => Path.Combine(DataDirectory, "workspaces"); @@ -656,6 +680,9 @@ public static LambdaOptions FromEnvironment() AdminToken = ReadOptional("LAMBDA_ADMIN_TOKEN"), TelemetryInterval = TimeSpan.FromSeconds(ReadInt("LAMBDA_TELEMETRY_INTERVAL_SECONDS", (int)defaults.TelemetryInterval.TotalSeconds)), TelemetrySamples = ReadInt("LAMBDA_TELEMETRY_SAMPLES", defaults.TelemetrySamples), + AnalyticsInterval = TimeSpan.FromSeconds(ReadInt("LAMBDA_ANALYTICS_INTERVAL_SECONDS", (int)defaults.AnalyticsInterval.TotalSeconds)), + CrawlerLists = ReadBool("LAMBDA_ANALYTICS_CRAWLER_LISTS", defaults.CrawlerLists), + CrawlerRefresh = TimeSpan.FromHours(ReadInt("LAMBDA_ANALYTICS_CRAWLER_REFRESH_HOURS", (int)defaults.CrawlerRefresh.TotalHours)), LogHistory = ReadInt("LAMBDA_LOG_HISTORY", defaults.LogHistory), LogMemory = ReadInt("LAMBDA_LOG_MEMORY_MB", defaults.LogMemory), CaptureLambdaOutput = ReadBool("LAMBDA_LOG_LAMBDA_OUTPUT", defaults.CaptureLambdaOutput), diff --git a/src/GenHTTP.Lambda/Data/Entities/LambdaEntity.cs b/src/GenHTTP.Lambda/Data/Entities/LambdaEntity.cs index 18ccdfb..e2d4527 100644 --- a/src/GenHTTP.Lambda/Data/Entities/LambdaEntity.cs +++ b/src/GenHTTP.Lambda/Data/Entities/LambdaEntity.cs @@ -76,6 +76,16 @@ public sealed class LambdaEntity /// public byte[]? SecretSalt { get; set; } + /// + /// Since when its visits are counted for its owner, or nothing while + /// analytics are switched off. + /// + /// + /// Only the switch: the counts are kept in a database of their own (see + /// AnalyticsStore), which serving a request never waits for. + /// + public DateTime? AnalyticsSince { get; set; } + public List Deployments { get; set; } = []; } diff --git a/src/GenHTTP.Lambda/Data/LambdaDbContext.cs b/src/GenHTTP.Lambda/Data/LambdaDbContext.cs index 935eb45..13e34a0 100644 --- a/src/GenHTTP.Lambda/Data/LambdaDbContext.cs +++ b/src/GenHTTP.Lambda/Data/LambdaDbContext.cs @@ -72,6 +72,7 @@ protected override void OnModelCreating(ModelBuilder builder) lambdas.Property(l => l.Modified).HasColumnName("modified"); lambdas.Property(l => l.Deployed).HasColumnName("deployed"); lambdas.Property(l => l.LastSeen).HasColumnName("last_seen"); + lambdas.Property(l => l.AnalyticsSince).HasColumnName("analytics_since"); lambdas.Property(l => l.Domain).HasColumnName("domain"); lambdas.Property(l => l.View).HasColumnName("editor_view").HasConversion(); lambdas.Property(l => l.SecretSalt).HasColumnName("secret_salt"); diff --git a/src/GenHTTP.Lambda/Data/Migrations/V21__Analytics.sql b/src/GenHTTP.Lambda/Data/Migrations/V21__Analytics.sql new file mode 100644 index 0000000..377b69d --- /dev/null +++ b/src/GenHTTP.Lambda/Data/Migrations/V21__Analytics.sql @@ -0,0 +1,11 @@ +-- Whether the owner switched analytics on, and since when. +-- +-- Analytics count how a lambda is visited - by day, page, country, referrer +-- and device - and are opt-in: nothing is counted for a lambda until its +-- owner asks for it. The counts themselves are not kept here but in a +-- database of their own (analytics.db), written in batches by a background +-- job, so serving a request never writes to this one. What is kept here is +-- only the switch, because it is read with everything else that decides how a +-- request to the lambda is served. + +ALTER TABLE lambdas ADD COLUMN analytics_since TEXT NULL; diff --git a/src/GenHTTP.Lambda/GenHTTP.Lambda.csproj b/src/GenHTTP.Lambda/GenHTTP.Lambda.csproj index 902ad32..11d4753 100644 --- a/src/GenHTTP.Lambda/GenHTTP.Lambda.csproj +++ b/src/GenHTTP.Lambda/GenHTTP.Lambda.csproj @@ -60,6 +60,9 @@ + + + diff --git a/src/GenHTTP.Lambda/Resources/Crawlers/crawler-user-agents.LICENSE.txt b/src/GenHTTP.Lambda/Resources/Crawlers/crawler-user-agents.LICENSE.txt new file mode 100644 index 0000000..b684617 --- /dev/null +++ b/src/GenHTTP.Lambda/Resources/Crawlers/crawler-user-agents.LICENSE.txt @@ -0,0 +1,20 @@ +The MIT License (MIT) + +Copyright (c) 2017 Martin Monperrus + +Permission is hereby granted, free of charge, to any person obtaining a copy of +this software and associated documentation files (the "Software"), to deal in +the Software without restriction, including without limitation the rights to +use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of +the Software, and to permit persons to whom the Software is furnished to do so, +subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS +FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR +COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER +IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN +CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. diff --git a/src/GenHTTP.Lambda/Resources/Crawlers/crawler-user-agents.json b/src/GenHTTP.Lambda/Resources/Crawlers/crawler-user-agents.json new file mode 100644 index 0000000..1539407 --- /dev/null +++ b/src/GenHTTP.Lambda/Resources/Crawlers/crawler-user-agents.json @@ -0,0 +1,18404 @@ +[ + { + "pattern": "Googlebot\\/", + "url": "http://www.google.com/bot.html", + "instances": [ + "Googlebot/2.1 (+http://www.google.com/bot.html)", + "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", + "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", + "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", + "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", + "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", + "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)", + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Safari/537.36" + ], + "description": "Google's main web crawling bot for search indexing", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Googlebot-Mobile", + "instances": [ + "DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)", + "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)", + "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)", + "Nokia6820/2.0 (4.83) Profile/MIDP-1.0 Configuration/CLDC-1.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)", + "SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)" + ], + "description": "Google's legacy mobile crawler for Google Search indexing", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Googlebot-Image", + "instances": [ + "Googlebot-Image/1.0" + ], + "description": "Google's image-specific web crawling bot for image search indexing", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Googlebot-News", + "instances": [ + "Googlebot-News" + ], + "description": "Google's news-specific web crawling bot for Google News indexing", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Googlebot-Video", + "instances": [ + "Googlebot-Video/1.0" + ], + "description": "Google's video crawler for video-related Google Search features", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "AdsBot-Google([^-]|$)", + "url": "https://support.google.com/webmasters/answer/1061943?hl=en", + "instances": [ + "AdsBot-Google (+http://www.google.com/adsbot.html)" + ], + "description": "Google's Ads bot for checking web page ad quality", + "tags": [ + "advertising" + ] + }, + { + "pattern": "AdsBot-Google-Mobile", + "addition_date": "2017/08/21", + "url": "https://support.google.com/adwords/answer/2404197", + "instances": [ + "AdsBot-Google-Mobile-Apps", + "Mozilla/5.0 (Linux; Android 5.0; SM-G920A) AppleWebKit (KHTML, like Gecko) Chrome Mobile Safari (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)", + "Mozilla/5.0 (iPhone; CPU iPhone OS 9_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13B143 Safari/601.1 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html)" + ], + "description": "Google's mobile Ads bot for crawling mobile pages to serve targeted ads", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Feedfetcher-Google", + "addition_date": "2018/06/27", + "url": "https://support.google.com/webmasters/answer/178852", + "instances": [ + "Feedfetcher-Google; (+http://www.google.com/feedfetcher.html; 1 subscribers; feed-id=728742641706423)" + ], + "description": "Google's feed fetcher bot for fetching RSS and Atom feeds for Google services", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Mediapartners-Google", + "url": "https://support.google.com/webmasters/answer/1061943?hl=en", + "instances": [ + "Mediapartners-Google", + "Mozilla/5.0 (compatible; MSIE or Firefox mutant; not on Windows server;) Daumoa/4.0 (Following Mediapartners-Google)", + "Mozilla/5.0 (iPhone; U; CPU iPhone OS 10_0 like Mac OS X; en-us) AppleWebKit/602.1.38 (KHTML, like Gecko) Version/10.0 Mobile/14A5297c Safari/602.1 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)", + "Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_1 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8B117 Safari/6531.22.7 (compatible; Mediapartners-Google/2.1; +http://www.google.com/bot.html)" + ], + "description": "Google's Mediapartners bot for AdSense and AdMob crawling", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Mediapartners \\(Googlebot\\)", + "addition_date": "2017/08/08", + "url": "https://support.google.com/webmasters/answer/1061943?hl=en", + "instances": [], + "description": "Google's Mediapartners bot variant for AdSense and AdMob crawling", + "tags": [ + "advertising" + ] + }, + { + "pattern": "APIs-Google", + "addition_date": "2017/08/08", + "url": "https://support.google.com/webmasters/answer/1061943?hl=en", + "instances": [ + "APIs-Google (+https://developers.google.com/webmasters/APIs-Google.html)" + ], + "description": "Google's APIs bot for crawling API documentation and services", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Google-InspectionTool", + "url": "https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers", + "instances": [ + "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-InspectionTool/1.0)", + "Mozilla/5.0 (compatible; Google-InspectionTool/1.0)" + ], + "description": "Google's inspection tool bot for testing and debugging search indexing", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Storebot-Google", + "url": "https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36", + "Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Mobile Safari/537.36" + ], + "description": "Google's Storebot for crawling product and e-commerce pages", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "GoogleOther", + "url": "https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers", + "instances": [ + "GoogleOther" + ], + "description": "Google's other bots and services for various Google search features", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "bingbot", + "url": "http://www.bing.com/bingbot.htm", + "instances": [ + "Mozilla/5.0 (Windows Phone 8.1; ARM; Trident/7.0; Touch; rv:11.0; IEMobile/11.0; NOKIA; Lumia 530) like Gecko (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)", + "Mozilla/5.0 (compatible; adidxbot/2.0; http://www.bing.com/bingbot.htm)", + "Mozilla/5.0 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)", + "Mozilla/5.0 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)", + "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm", + "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)", + "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) SitemapProbe", + "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; http://www.bing.com/bingbot.htm)", + "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; adidxbot/2.0; +http://www.bing.com/bingbot.htm)", + "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; http://www.bing.com/bingbot.htm)", + "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)", + "Mozilla/5.0 (seoanalyzer; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)", + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Safari/537.36", + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/103.0.5060.134 Safari/537.36" + ], + "description": "Microsoft's web crawling bot for Bing search indexing", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Slurp", + "url": "http://help.yahoo.com/help/us/ysearch/slurp", + "instances": [ + "Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)", + "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)", + "Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)" + ], + "description": "Yahoo's web crawling bot for Yahoo search indexing", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "[wW]get", + "instances": [ + "WGETbot/1.0 (+http://wget.alanreed.org)", + "Wget/1.14 (linux-gnu)", + "Wget/1.20.3 (linux-gnu)" + ], + "description": "GNU Wget command-line tool for downloading web content", + "tags": [ + "http-library" + ] + }, + { + "pattern": "LinkedInBot", + "instances": [ + "LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/3.1 +http://www.linkedin.com)", + "LinkedInBot/1.0 (compatible; Mozilla/5.0; Jakarta Commons-HttpClient/4.3 +http://www.linkedin.com)", + "LinkedInBot/1.0 (compatible; Mozilla/5.0; Apache-HttpClient +http://www.linkedin.com)" + ], + "description": "LinkedIn's bot for crawling professional content and profiles", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Python-urllib", + "instances": [ + "Python-urllib/1.17", + "Python-urllib/2.5", + "Python-urllib/2.6", + "Python-urllib/2.7", + "Python-urllib/3.1", + "Python-urllib/3.2", + "Python-urllib/3.3", + "Python-urllib/3.4", + "Python-urllib/3.5", + "Python-urllib/3.6", + "Python-urllib/3.7" + ], + "description": "Python's built-in URL library for HTTP requests", + "tags": [ + "http-library" + ] + }, + { + "pattern": "python-requests", + "addition_date": "2018/05/27", + "instances": [ + "python-requests/2.9.2", + "python-requests/2.11.1", + "python-requests/2.18.4", + "python-requests/2.19.1", + "python-requests/2.20.0", + "python-requests/2.21.0", + "python-requests/2.22.0" + ], + "description": "Popular Python HTTP library for making web requests", + "tags": [ + "http-library" + ] + }, + { + "pattern": "aiohttp", + "addition_date": "2019/12/23", + "instances": [ + "Python/3.9 aiohttp/3.7.3", + "Python/3.8 aiohttp/3.7.2", + "Python/3.7 aiohttp/3.6.2a2" + ], + "url": "https://docs.aiohttp.org/en/stable/", + "description": "Asynchronous HTTP client library for Python", + "tags": [ + "http-library" + ] + }, + { + "pattern": "httpx", + "addition_date": "2019/12/23", + "instances": [ + "python-httpx/0.16.1", + "python-httpx/0.13.0.dev1" + ], + "url": "https://www.python-httpx.org", + "description": "Modern Python HTTP client with async support", + "tags": [ + "http-library" + ] + }, + { + "pattern": "libwww-perl", + "instances": [ + "2Bone_LinkChecker/1.0 libwww-perl/6.03", + "2Bone_LinkChkr/1.0 libwww-perl/6.03", + "amibot - http://www.amidalla.de - tech@amidalla.com libwww-perl/5.831" + ], + "description": "Perl library for making HTTP requests and web crawling", + "tags": [ + "http-library" + ] + }, + { + "pattern": "httpunit", + "instances": [ + "httpunit/1.x" + ], + "description": "Java library for automated web application testing", + "tags": [ + "http-library" + ] + }, + { + "pattern": "Nutch", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/605.1.16 (KHTML, like Gecko; compatible; Friendly_Crawler/2.0) Chrome/120.0.6099.217 Safari/605.1.15/Nutch-1.20-SNAPSHOT", + "NutchCVS/0.7.1 (Nutch; http://lucene.apache.org/nutch/bot.html; nutch-agent@lucene.apache.org)", + "istellabot-nutch/Nutch-1.10" + ], + "description": "Apache Nutch open-source web crawler framework", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Go-http-client", + "addition_date": "2016/03/26", + "url": "https://golang.org/pkg/net/http/", + "instances": [ + "Go-http-client/1.1", + "Go-http-client/2.0" + ], + "description": "Go programming language HTTP client library", + "tags": [ + "http-library" + ] + }, + { + "pattern": "phpcrawl", + "addition_date": "2012/09/17", + "url": "http://phpcrawl.cuab.de/", + "instances": [ + "phpcrawl" + ], + "description": "PHP web crawler library for scraping websites", + "tags": [ + "http-library" + ] + }, + { + "pattern": "msnbot", + "url": "http://search.msn.com/msnbot.htm", + "instances": [ + "adidxbot/1.1 (+http://search.msn.com/msnbot.htm)", + "adidxbot/2.0 (+http://search.msn.com/msnbot.htm)", + "librabot/1.0 (+http://search.msn.com/msnbot.htm)", + "librabot/2.0 (+http://search.msn.com/msnbot.htm)", + "msnbot-NewsBlogs/2.0b (+http://search.msn.com/msnbot.htm)", + "msnbot-UDiscovery/2.0b (+http://search.msn.com/msnbot.htm)", + "msnbot-media/1.0 (+http://search.msn.com/msnbot.htm)", + "msnbot-media/1.1 (+http://search.msn.com/msnbot.htm)", + "msnbot-media/2.0b (+http://search.msn.com/msnbot.htm)", + "msnbot/1.0 (+http://search.msn.com/msnbot.htm)", + "msnbot/1.1 (+http://search.msn.com/msnbot.htm)", + "msnbot/2.0b (+http://search.msn.com/msnbot.htm)", + "msnbot/2.0b (+http://search.msn.com/msnbot.htm).", + "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" + ], + "description": "Microsoft's search engine bot for web indexing", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "jyxobot", + "instances": [], + "description": "Jyxo search engine bot for web crawling", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "FAST-WebCrawler", + "instances": [ + "FAST-WebCrawler/3.6/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)", + "FAST-WebCrawler/3.7 (atw-crawler at fast dot no; http://fast.no/support/crawler.asp)", + "FAST-WebCrawler/3.7/FirstPage (atw-crawler at fast dot no;http://fast.no/support/crawler.asp)", + "FAST-WebCrawler/3.8" + ], + "description": "FAST search engine web crawler for indexing", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "FAST Enterprise Crawler", + "instances": [ + "FAST Enterprise Crawler 6 / Scirus scirus-crawler@fast.no; http://www.scirus.com/srsapp/contactus/", + "FAST Enterprise Crawler 6 used by Schibsted (webcrawl@schibstedsok.no)" + ], + "description": "FAST enterprise-grade web crawler for search", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "BIGLOTRON", + "instances": [ + "BIGLOTRON (Beta 2;GNU/Linux)" + ], + "description": "Biglotron search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Teoma", + "instances": [ + "Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://sp.ask.com/docs/about/tech_crawling.html)", + "Mozilla/2.0 (compatible; Ask Jeeves/Teoma; +http://about.ask.com/en/docs/about/webmasters.shtml)" + ], + "url": "http://about.ask.com/en/docs/about/webmasters.shtml", + "description": "Ask Jeeves Teoma search engine web crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "convera", + "instances": [ + "ConveraCrawler/0.9e (+http://ews.converasearch.com/crawl.htm)" + ], + "url": "http://ews.converasearch.com/crawl.htm", + "description": "Convera search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "^Seekbot", + "instances": [ + "Seekbot/1.0 (http://www.seekbot.net/bot.html) RobotsTxtFetcher/1.2" + ], + "url": "http://www.seekbot.net/bot.html", + "description": "Seekbot search engine web crawler for indexing", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Gigabot", + "instances": [ + "Gigabot/1.0", + "Gigabot/2.0 (http://www.gigablast.com/spider.html)" + ], + "url": "http://www.gigablast.com/spider.html", + "description": "Gigablast search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Gigablast", + "instances": [ + "GigablastOpenSource/1.0" + ], + "url": "https://github.com/gigablast/open-source-search-engine", + "description": "Gigablast open-source search engine crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "exabot", + "instances": [ + "Mozilla/5.0 (compatible; Alexabot/1.0; +http://www.alexa.com/help/certifyscan; certifyscan@alexa.com)", + "Mozilla/5.0 (compatible; Exabot PyExalead/3.0; +http://www.exabot.com/go/robot)", + "Mozilla/5.0 (compatible; Exabot-Images/3.0; +http://www.exabot.com/go/robot)", + "Mozilla/5.0 (compatible; Exabot/3.0 (BiggerBetter); +http://www.exabot.com/go/robot)", + "Mozilla/5.0 (compatible; Exabot/3.0; +http://www.exabot.com/go/robot)", + "Mozilla/5.0 (compatible; Exabot/3.0; http://www.exabot.com/go/robot)" + ], + "description": "Exabot search engine web crawler for indexing", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "ia_archiver", + "instances": [ + "ia_archiver (+http://www.alexa.com/site/help/webmasters; crawler@alexa.com)", + "ia_archiver-web.archive.org" + ], + "description": "Internet Archive Wayback Machine web crawler", + "tags": [ + "archiver" + ] + }, + { + "pattern": "GingerCrawler", + "instances": [ + "GingerCrawler/1.0 (Language Assistant for Dyslexics; www.gingersoftware.com/crawler_agent.htm; support at ginger software dot com)" + ], + "description": "Ginger Software's language assistant web crawler", + "tags": [ + "seo" + ] + }, + { + "pattern": "webmon ", + "instances": [], + "description": "Webmon website monitoring and crawling bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "HTTrack", + "instances": [ + "Mozilla/4.5 (compatible; HTTrack 3.0x; Windows 98)" + ], + "description": "HTTrack website copier for offline browsing", + "tags": [ + "http-library" + ] + }, + { + "pattern": "grub\\.org", + "instances": [ + "Mozilla/4.0 (compatible; grub-client-0.3.0; Crawl your own stuff with http://grub.org)", + "Mozilla/4.0 (compatible; grub-client-1.0.4; Crawl your own stuff with http://grub.org)", + "Mozilla/4.0 (compatible; grub-client-1.0.5; Crawl your own stuff with http://grub.org)", + "Mozilla/4.0 (compatible; grub-client-1.0.6; Crawl your own stuff with http://grub.org)", + "Mozilla/4.0 (compatible; grub-client-1.0.7; Crawl your own stuff with http://grub.org)", + "Mozilla/4.0 (compatible; grub-client-1.1.1; Crawl your own stuff with http://grub.org)", + "Mozilla/4.0 (compatible; grub-client-1.2.1; Crawl your own stuff with http://grub.org)", + "Mozilla/4.0 (compatible; grub-client-1.3.1; Crawl your own stuff with http://grub.org)", + "Mozilla/4.0 (compatible; grub-client-1.3.7; Crawl your own stuff with http://grub.org)", + "Mozilla/4.0 (compatible; grub-client-1.4.3; Crawl your own stuff with http://grub.org)", + "Mozilla/4.0 (compatible; grub-client-1.5.3; Crawl your own stuff with http://grub.org)" + ], + "description": "Grub search engine web crawler for indexing", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "UsineNouvelleCrawler", + "instances": [], + "description": "Usine Nouvelle news site web crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "antibot", + "instances": [], + "description": "Antibot web crawler for content discovery", + "tags": [ + "scanner" + ] + }, + { + "pattern": "netresearchserver", + "instances": [], + "description": "Net Research Server web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "speedy", + "instances": [ + "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)", + "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) Speedy Spider for SpeedyAds (http://www.entireweb.com/about/search_tech/speedy_spider/)", + "Mozilla/5.0 (compatible; Speedy Spider; http://www.entireweb.com/about/search_tech/speedy_spider/)", + "Speedy Spider (Entireweb; Beta/1.2; http://www.entireweb.com/about/search_tech/speedyspider/)", + "Speedy Spider (http://www.entireweb.com/about/search_tech/speedy_spider/)" + ], + "description": "Entireweb Speedy Spider web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "fluffy", + "instances": [], + "description": "Fluffy search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "findlink", + "instances": [ + "findlinks/1.0 (+http://wortschatz.uni-leipzig.de/findlinks/)", + "findlinks/1.1.3-beta8 (+http://wortschatz.uni-leipzig.de/findlinks/)", + "findlinks/1.1.3-beta9 (+http://wortschatz.uni-leipzig.de/findlinks/)", + "findlinks/1.1.5-beta7 (+http://wortschatz.uni-leipzig.de/findlinks/)", + "findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/)", + "findlinks/1.1.6-beta1 (+http://wortschatz.uni-leipzig.de/findlinks/; YaCy 0.1; yacy.net)", + "findlinks/1.1.6-beta2 (+http://wortschatz.uni-leipzig.de/findlinks/)", + "findlinks/1.1.6-beta3 (+http://wortschatz.uni-leipzig.de/findlinks/)", + "findlinks/1.1.6-beta4 (+http://wortschatz.uni-leipzig.de/findlinks/)", + "findlinks/1.1.6-beta5 (+http://wortschatz.uni-leipzig.de/findlinks/)", + "findlinks/1.1.6-beta6 (+http://wortschatz.uni-leipzig.de/findlinks/)", + "findlinks/2.0 (+http://wortschatz.uni-leipzig.de/findlinks/)", + "findlinks/2.0.1 (+http://wortschatz.uni-leipzig.de/findlinks/)", + "findlinks/2.0.2 (+http://wortschatz.uni-leipzig.de/findlinks/)", + "findlinks/2.0.4 (+http://wortschatz.uni-leipzig.de/findlinks/)", + "findlinks/2.0.5 (+http://wortschatz.uni-leipzig.de/findlinks/)", + "findlinks/2.0.9 (+http://wortschatz.uni-leipzig.de/findlinks/)", + "findlinks/2.1 (+http://wortschatz.uni-leipzig.de/findlinks/)", + "findlinks/2.1.3 (+http://wortschatz.uni-leipzig.de/findlinks/)", + "findlinks/2.1.5 (+http://wortschatz.uni-leipzig.de/findlinks/)", + "findlinks/2.2 (+http://wortschatz.uni-leipzig.de/findlinks/)", + "findlinks/2.5 (+http://wortschatz.uni-leipzig.de/findlinks/)", + "findlinks/2.6 (+http://wortschatz.uni-leipzig.de/findlinks/)" + ], + "description": "Findlinks web crawler for link discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "msrbot", + "instances": [], + "description": "Microsoft Research web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "panscient", + "instances": [ + "panscient.com" + ], + "description": "Panscient web crawler for content analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "yacybot", + "instances": [ + "yacybot (/global; amd64 FreeBSD 10.3-RELEASE; java 1.8.0_77; GMT/en) http://yacy.net/bot.html", + "yacybot (/global; amd64 FreeBSD 10.3-RELEASE-p7; java 1.7.0_95; GMT/en) http://yacy.net/bot.html", + "yacybot (-global; amd64 FreeBSD 9.2-RELEASE-p10; java 1.7.0_65; Europe/en) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 2.6.32-042stab093.4; java 1.7.0_65; Etc/en) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 2.6.32-042stab094.8; java 1.7.0_79; America/en) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 2.6.32-042stab108.8; java 1.7.0_91; America/en) http://yacy.net/bot.html", + "yacybot (-global; amd64 Linux 2.6.32-042stab111.11; java 1.7.0_79; Europe/en) http://yacy.net/bot.html", + "yacybot (-global; amd64 Linux 2.6.32-042stab116.1; java 1.7.0_79; Europe/en) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 2.6.32-573.3.1.el6.x86_64; java 1.7.0_85; Europe/en) http://yacy.net/bot.html", + "yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html", + "yacybot (-global; amd64 Linux 3.10.0-229.4.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.10.0-229.7.2.el7.x86_64; java 1.8.0_45; Europe/en) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.10.0-327.22.2.el7.x86_64; java 1.7.0_101; Etc/en) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.11.10-21-desktop; java 1.7.0_51; America/en) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.12.1; java 1.7.0_65; Europe/en) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/de) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.13.0-042stab093.4; java 1.7.0_79; Europe/en) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.13.0-45-generic; java 1.7.0_75; Europe/en) http://yacy.net/bot.html", + "yacybot (-global; amd64 Linux 3.13.0-61-generic; java 1.7.0_79; Europe/en) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.13.0-74-generic; java 1.7.0_91; Europe/en) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/de) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.13.0-83-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_101; Europe/en) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.13.0-85-generic; java 1.7.0_95; Europe/en) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.13.0-88-generic; java 1.7.0_101; Europe/en) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.14-0.bpo.1-amd64; java 1.7.0_55; Europe/de) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html", + "yacybot (-global; amd64 Linux 3.14.32-xxxx-grs-ipv6-64; java 1.8.0_111; Europe/de) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_111; Europe/de) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; America/en) http://yacy.net/bot.html", + "yacybot (-global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_75; Europe/en) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/de) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_79; Europe/en) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_91; Europe/de) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.7.0_95; Europe/en) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.16.0-4-amd64; java 1.8.0_111; Europe/en) http://yacy.net/bot.html", + "yacybot (/global; amd64 Linux 3.16-0.bpo.2-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html", + "yacybot (-global; amd64 Linux 3.19.0-15-generic; java 1.8.0_45-internal; Europe/de) http://yacy.net/bot.html", + "yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_65; Europe/en) http://yacy.net/bot.html", + "yacybot (-global; amd64 Linux 3.2.0-4-amd64; java 1.7.0_67; Europe/en) http://yacy.net/bot.html", + "yacybot (-global; amd64 Linux 4.4.0-57-generic; java 9-internal; Europe/en) http://yacy.net/bot.html", + "yacybot (-global; amd64 Windows 8.1 6.3; java 1.7.0_55; Europe/de) http://yacy.net/bot.html", + "yacybot (-global; amd64 Windows 8 6.2; java 1.7.0_55; Europe/de) http://yacy.net/bot.html", + "yacybot (-global; amd64 Linux 5.2.8-Jinsol; java 12.0.2; Europe/en) http://yacy.net/bot.html", + "yacybot (-global; amd64 Linux 5.2.9-Jinsol; java 12.0.2; Europe/en) http://yacy.net/bot.html", + "yacybot (-global; amd64 Linux 5.2.11-Jinsol; java 12.0.2; Europe/en) http://yacy.net/bot.html" + ], + "description": "YaCy decentralized search engine web crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "AISearchBot", + "instances": [], + "description": "AI-powered search engine web crawler bot", + "tags": [ + "search-engine", + "ai-crawler" + ] + }, + { + "pattern": "ips-agent", + "instances": [ + "BlackBerry9000/4.6.0.167 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 ips-agent", + "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.12; ips-agent) Gecko/20050922 Fedora/1.0.7-1.1.fc4 Firefox/1.0.7", + "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.1.3; ips-agent) Gecko/20090824 Fedora/1.0.7-1.1.fc4 Firefox/3.5.3", + "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.24; ips-agent) Gecko/20111107 Ubuntu/10.04 (lucid) Firefox/3.6.24", + "Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:14.0; ips-agent) Gecko/20100101 Firefox/14.0.1" + ], + "description": "IPS agent web crawler for content indexing", + "tags": [ + "seo" + ] + }, + { + "pattern": "tagoobot", + "instances": [], + "description": "Tagoo search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "MJ12bot", + "instances": [ + "MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)", + "Mozilla/5.0 (compatible; MJ12bot/v1.2.1; http://www.majestic12.co.uk/bot.php?+)", + "Mozilla/5.0 (compatible; MJ12bot/v1.2.3; http://www.majestic12.co.uk/bot.php?+)", + "Mozilla/5.0 (compatible; MJ12bot/v1.2.4; http://www.majestic12.co.uk/bot.php?+)", + "Mozilla/5.0 (compatible; MJ12bot/v1.2.5; http://www.majestic12.co.uk/bot.php?+)", + "Mozilla/5.0 (compatible; MJ12bot/v1.3.0; http://www.majestic12.co.uk/bot.php?+)", + "Mozilla/5.0 (compatible; MJ12bot/v1.3.1; http://www.majestic12.co.uk/bot.php?+)", + "Mozilla/5.0 (compatible; MJ12bot/v1.3.2; http://www.majestic12.co.uk/bot.php?+)", + "Mozilla/5.0 (compatible; MJ12bot/v1.3.3; http://www.majestic12.co.uk/bot.php?+)", + "Mozilla/5.0 (compatible; MJ12bot/v1.4.0; http://www.majestic12.co.uk/bot.php?+)", + "Mozilla/5.0 (compatible; MJ12bot/v1.4.1; http://www.majestic12.co.uk/bot.php?+)", + "Mozilla/5.0 (compatible; MJ12bot/v1.4.2; http://www.majestic12.co.uk/bot.php?+)", + "Mozilla/5.0 (compatible; MJ12bot/v1.4.3; http://www.majestic12.co.uk/bot.php?+)", + "Mozilla/5.0 (compatible; MJ12bot/v1.4.4 (domain ownership verifier); http://www.majestic12.co.uk/bot.php?+)", + "Mozilla/5.0 (compatible; MJ12bot/v1.4.4; http://www.majestic12.co.uk/bot.php?+)", + "Mozilla/5.0 (compatible; MJ12bot/v1.4.5; http://www.majestic12.co.uk/bot.php?+)", + "Mozilla/5.0 (compatible; MJ12bot/v1.4.6; http://mj12bot.com/)", + "Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)", + "Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://www.majestic12.co.uk/bot.php?+)", + "Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)" + ], + "description": "Majestic-12 search engine web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "woriobot", + "instances": [ + "Mozilla/5.0 (compatible; woriobot +http://worio.com)", + "Mozilla/5.0 (compatible; woriobot support [at] zite [dot] com +http://zite.com)" + ], + "description": "Worio search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "yanga", + "instances": [ + "Yanga WorldSearch Bot v1.1/beta (http://www.yanga.co.uk/)" + ], + "description": "Yanga search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "buzzbot", + "instances": [ + "Buzzbot/1.0 (Buzzbot; http://www.buzzstream.com; buzzbot@buzzstream.com)" + ], + "description": "Buzzstream web crawler for link research", + "tags": [ + "seo" + ] + }, + { + "pattern": "mlbot", + "instances": [ + "MLBot (www.metadatalabs.com/mlbot)" + ], + "description": "Metadata Labs web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "yandex\\.com\\/bots", + "url": "https://yandex.ru/support/webmaster/robot-workings/check-yandex-robots.html#robot-in-logs", + "instances": [ + "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexBot/3.0; MirrorDetector; +http://yandex.com/bots)", + "Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexBot/3.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexImages/3.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexAccessibilityBot/3.0; +http://yandex.com/bots", + "Mozilla/5.0 (compatible; YandexUserproxy; robot; +http://yandex.com/bots", + "Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B411 Safari/600.1.4 (compatible; YandexMobileBot/3.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexMetrika/2.0; +http://yandex.com/bots yabs01)", + "Mozilla/5.0 (compatible; YandexMetrika/3.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexMetrika/4.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexTurbo/1.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexVideoParser/1.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexVideo/3.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexImageResizer/2.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexAdNet/1.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexBlogs/0.99; robot; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexCalendar/1.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexDirect/3.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexDirectDyn/1.0; +http://yandex.com/bots", + "Mozilla/5.0 (compatible; YandexFavicons/1.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YaDirectFetcher/1.0; Dyatel; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexForDomain/1.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexMarket/1.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexMarket/2.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexMedia/3.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexMobileScreenShotBot/1.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexNews/4.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexOntoDB/1.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexOntoDBAPI/1.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexPagechecker/1.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexPartner/3.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexRCA/1.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexSearchShop/1.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexSitelinks; Dyatel; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexSpravBot/1.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexTracker/1.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexVertis/3.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexVerticals/1.0; +http://yandex.com/bots)", + "Mozilla/5.0 (compatible; YandexWebmaster/2.0; +http://yandex.com/bots)", + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36 (compatible; YandexScreenshotBot/3.0; +http://yandex.com/bots)" + ], + "addition_date": "2015/04/14", + "description": "Yandex search engine web crawler bots", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "purebot", + "addition_date": "2010/01/19", + "instances": [], + "description": "Pure web crawler for content discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Linguee Bot", + "addition_date": "2010/01/26", + "url": "http://www.linguee.com/bot", + "instances": [ + "Linguee Bot (http://www.linguee.com/bot)", + "Linguee Bot (http://www.linguee.com/bot; bot@linguee.com)" + ], + "description": "Linguee translation web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "CyberPatrol", + "addition_date": "2010/02/11", + "url": "http://www.cyberpatrol.com/cyberpatrolcrawler.asp", + "instances": [ + "CyberPatrol SiteCat Webbot (http://www.cyberpatrol.com/cyberpatrolcrawler.asp)" + ], + "description": "CyberPatrol web content filtering bot", + "tags": [ + "scanner" + ] + }, + { + "pattern": "voilabot", + "addition_date": "2010/05/18", + "instances": [ + "Mozilla/5.0 (Windows NT 5.1; U; Win64; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)", + "Mozilla/5.0 (Windows; U; Windows NT 5.1; fr; rv:1.8.1) VoilaBot BETA 1.2 (support.voilabot@orange-ftgroup.com)" + ], + "description": "Voila search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Baiduspider", + "addition_date": "2010/07/15", + "url": "http://www.baidu.jp/spider/", + "instances": [ + "Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)", + "Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)" + ], + "description": "Baidu search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "citeseerxbot", + "addition_date": "2010/07/17", + "instances": [], + "description": "CiteSeerX academic web crawler bot", + "tags": [ + "academic" + ] + }, + { + "pattern": "spbot", + "addition_date": "2010/07/31", + "url": "http://www.seoprofiler.com/bot", + "instances": [ + "Mozilla/5.0 (compatible; spbot/1.0; +http://www.seoprofiler.com/bot/ )", + "Mozilla/5.0 (compatible; spbot/1.1; +http://www.seoprofiler.com/bot/ )", + "Mozilla/5.0 (compatible; spbot/1.2; +http://www.seoprofiler.com/bot/ )", + "Mozilla/5.0 (compatible; spbot/2.0.1; +http://www.seoprofiler.com/bot/ )", + "Mozilla/5.0 (compatible; spbot/2.0.2; +http://www.seoprofiler.com/bot/ )", + "Mozilla/5.0 (compatible; spbot/2.0.3; +http://www.seoprofiler.com/bot/ )", + "Mozilla/5.0 (compatible; spbot/2.0.4; +http://www.seoprofiler.com/bot )", + "Mozilla/5.0 (compatible; spbot/2.0; +http://www.seoprofiler.com/bot/ )", + "Mozilla/5.0 (compatible; spbot/2.1; +http://www.seoprofiler.com/bot )", + "Mozilla/5.0 (compatible; spbot/3.0; +http://www.seoprofiler.com/bot )", + "Mozilla/5.0 (compatible; spbot/3.1; +http://www.seoprofiler.com/bot )", + "Mozilla/5.0 (compatible; spbot/4.0.1; +http://www.seoprofiler.com/bot )", + "Mozilla/5.0 (compatible; spbot/4.0.2; +http://www.seoprofiler.com/bot )", + "Mozilla/5.0 (compatible; spbot/4.0.3; +http://www.seoprofiler.com/bot )", + "Mozilla/5.0 (compatible; spbot/4.0.4; +http://www.seoprofiler.com/bot )", + "Mozilla/5.0 (compatible; spbot/4.0.5; +http://www.seoprofiler.com/bot )", + "Mozilla/5.0 (compatible; spbot/4.0.6; +http://www.seoprofiler.com/bot )", + "Mozilla/5.0 (compatible; spbot/4.0.7; +http://OpenLinkProfiler.org/bot )", + "Mozilla/5.0 (compatible; spbot/4.0.7; +https://www.seoprofiler.com/bot )", + "Mozilla/5.0 (compatible; spbot/4.0.8; +http://OpenLinkProfiler.org/bot )", + "Mozilla/5.0 (compatible; spbot/4.0.9; +http://OpenLinkProfiler.org/bot )", + "Mozilla/5.0 (compatible; spbot/4.0; +http://www.seoprofiler.com/bot )", + "Mozilla/5.0 (compatible; spbot/4.0a; +http://www.seoprofiler.com/bot )", + "Mozilla/5.0 (compatible; spbot/4.0b; +http://www.seoprofiler.com/bot )", + "Mozilla/5.0 (compatible; spbot/4.1.0; +http://OpenLinkProfiler.org/bot )", + "Mozilla/5.0 (compatible; spbot/4.2.0; +http://OpenLinkProfiler.org/bot )", + "Mozilla/5.0 (compatible; spbot/4.3.0; +http://OpenLinkProfiler.org/bot )", + "Mozilla/5.0 (compatible; spbot/4.4.0; +http://OpenLinkProfiler.org/bot )", + "Mozilla/5.0 (compatible; spbot/4.4.1; +http://OpenLinkProfiler.org/bot )", + "Mozilla/5.0 (compatible; spbot/4.4.2; +http://OpenLinkProfiler.org/bot )", + "Mozilla/5.0 (compatible; spbot/5.0.1; +http://OpenLinkProfiler.org/bot )", + "Mozilla/5.0 (compatible; spbot/5.0.2; +http://OpenLinkProfiler.org/bot )", + "Mozilla/5.0 (compatible; spbot/5.0.3; +http://OpenLinkProfiler.org/bot )", + "Mozilla/5.0 (compatible; spbot/5.0; +http://OpenLinkProfiler.org/bot )" + ], + "description": "SEO Profiler web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "twengabot", + "addition_date": "2010/08/03", + "url": "http://www.twenga.com/bot.html", + "instances": [], + "description": "Twenga shopping web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "postrank", + "addition_date": "2010/08/03", + "url": "http://www.postrank.com", + "instances": [ + "PostRank/2.0 (postrank.com)", + "PostRank/2.0 (postrank.com; 1 subscribers)" + ], + "description": "PostRank web crawler for content ranking", + "tags": [ + "seo" + ] + }, + { + "pattern": "Turnitin", + "addition_date": "2010/09/26", + "url": "http://www.turnitin.com", + "instances": [ + "TurnitinBot (https://turnitin.com/robot/crawlerinfo.html)", + "Turnitin (https://bit.ly/2UvnfoQ)" + ], + "description": "Turnitin plagiarism detection web crawler", + "tags": [ + "academic" + ] + }, + { + "pattern": "scribdbot", + "addition_date": "2010/09/28", + "url": "http://www.scribd.com", + "instances": [], + "description": "Scribd document web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "page2rss", + "addition_date": "2010/10/07", + "url": "http://www.page2rss.com", + "instances": [ + "Mozilla/5.0 (compatible; Page2RSS/0.7; +http://page2rss.com/)" + ], + "description": "Page2RSS web crawler for RSS conversion", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "sitebot", + "addition_date": "2010/12/15", + "url": "http://www.sitebot.org", + "instances": [ + "Mozilla/5.0 (compatible; Whoiswebsitebot/0.1; +http://www.whoiswebsite.net)" + ], + "description": "Sitebot web crawler for site analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "linkdex", + "addition_date": "2011/01/06", + "url": "http://www.linkdex.com", + "instances": [ + "Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/about/bots/)", + "Mozilla/5.0 (compatible; linkdexbot/2.0; +http://www.linkdex.com/bots/)", + "Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/about/bots/)", + "Mozilla/5.0 (compatible; linkdexbot/2.1; +http://www.linkdex.com/bots/)", + "Mozilla/5.0 (compatible; linkdexbot/2.2; +http://www.linkdex.com/bots/)", + "linkdex.com/v2.0", + "linkdexbot/Nutch-1.0-dev (http://www.linkdex.com/; crawl at linkdex dot com)" + ], + "description": "Linkdex SEO tool web crawler for link analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Adidxbot", + "url": "https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0", + "instances": [], + "description": "Bing's advertising index web crawler bot", + "tags": [ + "advertising", + "search-engine" + ] + }, + { + "pattern": "ezooms", + "addition_date": "2011/04/27", + "url": "http://www.phpbb.com/community/viewtopic.php?f=64&t=935605&start=450#p12948289", + "instances": [ + "Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com)" + ], + "description": "Ezooms search engine web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "dotbot", + "addition_date": "2011/04/27", + "instances": [ + "Mozilla/5.0 (compatible; DotBot/1.1; http://www.opensiteexplorer.org/dotbot, help@moz.com)", + "dotbot" + ], + "description": "Moz DotBot web crawler for SEO analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Mail\\.RU_Bot", + "addition_date": "2011/04/27", + "instances": [ + "Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/help/robots)", + "Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/2.0; +http://go.mail.ru/", + "Mozilla/5.0 (compatible; Mail.RU_Bot/2.0; +http://go.mail.ru/", + "Mozilla/5.0 (compatible; Linux x86_64; Mail.RU_Bot/Robots/2.0; +http://go.mail.ru/help/robots)" + ], + "description": "Mail.RU search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "discobot", + "addition_date": "2011/05/03", + "url": "http://discoveryengine.com/discobot.html", + "instances": [ + "Mozilla/5.0 (compatible; discobot/1.0; +http://discoveryengine.com/discobot.html)", + "Mozilla/5.0 (compatible; discobot/2.0; +http://discoveryengine.com/discobot.html)", + "mozilla/5.0 (compatible; discobot/1.1; +http://discoveryengine.com/discobot.html)" + ], + "description": "Discovery Engine web crawler for content discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "heritrix", + "addition_date": "2011/06/21", + "url": "https://github.com/internetarchive/heritrix3/wiki", + "instances": [ + "Mozilla/5.0 (compatible; heritrix/1.12.1 +http://www.webarchiv.cz)", + "Mozilla/5.0 (compatible; heritrix/1.12.1b +http://netarkivet.dk/website/info.html)", + "Mozilla/5.0 (compatible; heritrix/1.14.2 +http://rjpower.org)", + "Mozilla/5.0 (compatible; heritrix/1.14.2 +http://www.webarchiv.cz)", + "Mozilla/5.0 (compatible; heritrix/1.14.3 +http://archive.org)", + "Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.accelobot.com)", + "Mozilla/5.0 (compatible; heritrix/1.14.3 +http://www.webarchiv.cz)", + "Mozilla/5.0 (compatible; heritrix/1.14.3.r6601 +http://www.buddybuzz.net/yptrino)", + "Mozilla/5.0 (compatible; heritrix/1.14.4 +http://parsijoo.ir)", + "Mozilla/5.0 (compatible; heritrix/1.14.4 +http://www.exif-search.com)", + "Mozilla/5.0 (compatible; heritrix/2.0.2 +http://aihit.com)", + "Mozilla/5.0 (compatible; heritrix/2.0.2 +http://seekda.com)", + "Mozilla/5.0 (compatible; heritrix/3.0.0-SNAPSHOT-20091120.021634 +http://crawler.archive.org)", + "Mozilla/5.0 (compatible; heritrix/3.1.0-RC1 +http://boston.lti.cs.cmu.edu/crawler_12/)", + "Mozilla/5.0 (compatible; heritrix/3.1.1 +http://places.tomtom.com/crawlerinfo)", + "Mozilla/5.0 (compatible; heritrix/3.1.1 +http://www.mixdata.com)", + "Mozilla/5.0 (compatible; heritrix/3.1.1; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)", + "Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.crim.ca)", + "Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.exif-search.com)", + "Mozilla/5.0 (compatible; heritrix/3.2.0 +http://www.mixdata.com)", + "Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20160309-0050; UniLeipzigASV +http://corpora.informatik.uni-leipzig.de/crawler_faq.html)", + "Mozilla/5.0 (compatible; sukibot_heritrix/3.1.1 +http://suki.ling.helsinki.fi/eng/webmasters.html)" + ], + "description": "Internet Archive's Heritrix web crawler framework", + "tags": [ + "archiver" + ] + }, + { + "pattern": "findthatfile", + "addition_date": "2011/06/21", + "url": "http://www.findthatfile.com/", + "instances": [], + "description": "FindThatFile web crawler for file discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "europarchive\\.org", + "addition_date": "2011/06/21", + "url": "", + "instances": [ + "Mozilla/5.0 (compatible; MSIE 7.0 +http://www.europarchive.org)" + ], + "description": "European Archive web crawler for preservation", + "tags": [ + "archiver" + ] + }, + { + "pattern": "NerdByNature\\.Bot", + "addition_date": "2011/07/12", + "url": "http://www.nerdbynature.net/bot", + "instances": [ + "Mozilla/5.0 (compatible; NerdByNature.Bot; http://www.nerdbynature.net/bot)" + ], + "description": "NerdByNature web crawler for content indexing", + "tags": [ + "seo" + ] + }, + { + "pattern": "(sistrix|SISTRIX) [cC]rawler", + "addition_date": "2011/08/02", + "url": "https://www.sistrix.com/tutorials/crawling-errors-in-the-optimizer/", + "instances": [ + "Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)" + ], + "description": "SISTRIX SEO tool web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Ahrefs(Bot|SiteAudit)", + "addition_date": "2011/08/28", + "instances": [ + "Mozilla/5.0 (compatible; AhrefsBot/6.1; +http://ahrefs.com/robot/)", + "Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/)", + "Mozilla/5.0 (compatible; AhrefsBot/5.2; News; +http://ahrefs.com/robot/)", + "Mozilla/5.0 (compatible; AhrefsBot/5.2; +http://ahrefs.com/robot/)", + "Mozilla/5.0 (compatible; AhrefsSiteAudit/5.2; +http://ahrefs.com/robot/)", + "Mozilla/5.0 (compatible; AhrefsBot/6.1; News; +http://ahrefs.com/robot/)", + "Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)" + ], + "description": "Ahrefs SEO tool web crawler for link analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "fuelbot", + "addition_date": "2018/06/28", + "instances": [ + "fuelbot" + ], + "description": "Fuel web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "^CrunchBot", + "addition_date": "2018/06/28", + "instances": [ + "CrunchBot/1.0 (+http://www.leadcrunch.com/crunchbot)" + ], + "description": "LeadCrunch web crawler for lead generation", + "tags": [ + "seo" + ] + }, + { + "pattern": "IndeedBot", + "addition_date": "2018/06/28", + "instances": [ + "Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0 (IndeedBot 1.1)" + ], + "description": "Indeed job search web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "mappydata", + "addition_date": "2018/06/28", + "instances": [ + "Mozilla/5.0 (compatible; Mappy/1.0; +http://mappydata.net/bot/)" + ], + "description": "Mappy web crawler for mapping data", + "tags": [ + "seo" + ] + }, + { + "pattern": "woobot", + "addition_date": "2018/06/28", + "instances": [ + "woobot" + ], + "description": "Woo web crawler for content analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "ZoominfoBot", + "addition_date": "2018/06/28", + "instances": [ + "ZoominfoBot (zoominfobot at zoominfo dot com)" + ], + "description": "ZoomInfo web crawler for business intelligence", + "tags": [ + "seo" + ] + }, + { + "pattern": "PrivacyAwareBot", + "addition_date": "2018/06/28", + "instances": [ + "Mozilla/5.0 (compatible; PrivacyAwareBot/1.1; +http://www.privacyaware.org)" + ], + "description": "PrivacyAware web crawler for privacy analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Multiviewbot", + "addition_date": "2018/06/28", + "instances": [ + "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Multiviewbot" + ], + "description": "Multiview web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "SWIMGBot", + "addition_date": "2018/06/28", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36 SWIMGBot" + ], + "description": "SWIMG web crawler for image discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "Grobbot", + "addition_date": "2018/06/28", + "instances": [ + "Mozilla/5.0 (compatible; Grobbot/2.2; +https://grob.it)" + ], + "description": "Grob web crawler for content analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "eright", + "addition_date": "2018/06/28", + "instances": [ + "Mozilla/5.0 (compatible; eright/1.0; +bot@eright.com)" + ], + "description": "Eright web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "Apercite", + "addition_date": "2018/06/28", + "instances": [ + "Mozilla/5.0 (compatible; Apercite; +http://www.apercite.fr/robot/index.html)" + ], + "description": "Apercite web crawler for content analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "semanticbot", + "addition_date": "2018/06/28", + "instances": [ + "semanticbot", + "semanticbot (info@semanticaudience.com)" + ], + "description": "Semantic web crawler for content analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Aboundex", + "addition_date": "2011/09/28", + "url": "http://www.aboundex.com/crawler/", + "instances": [ + "Aboundex/0.2 (http://www.aboundex.com/crawler/)", + "Aboundex/0.3 (http://www.aboundex.com/crawler/)" + ], + "description": "Aboundex web crawler for content discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "domaincrawler", + "addition_date": "2011/10/21", + "instances": [ + "CipaCrawler/3.0 (info@domaincrawler.com; http://www.domaincrawler.com/www.example.com)" + ], + "description": "Domain Crawler web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "wbsearchbot", + "addition_date": "2011/12/21", + "url": "http://www.warebay.com/bot.html", + "instances": [], + "description": "Warebay search web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "summify", + "addition_date": "2012/01/04", + "url": "http://summify.com", + "instances": [ + "Summify (Summify/1.0.1; +http://summify.com)" + ], + "description": "Summify web crawler for content summarization", + "tags": [ + "seo" + ] + }, + { + "pattern": "CCBot", + "addition_date": "2012/02/05", + "url": "http://www.commoncrawl.org/bot.html", + "instances": [ + "CCBot/2.0 (http://commoncrawl.org/faq/)", + "CCBot/2.0 (https://commoncrawl.org/faq/)" + ], + "description": "Common Crawl web crawler for indexing", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "edisterbot", + "addition_date": "2012/02/25", + "instances": [], + "description": "Edister web crawler for content discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "SeznamBot", + "addition_date": "2012/03/14", + "instances": [ + "Mozilla/5.0 (compatible; SeznamBot/3.2-test1-1; +http://napoveda.seznam.cz/en/seznambot-intro/)", + "Mozilla/5.0 (compatible; SeznamBot/3.2-test1; +http://napoveda.seznam.cz/en/seznambot-intro/)", + "Mozilla/5.0 (compatible; SeznamBot/3.2-test2; +http://napoveda.seznam.cz/en/seznambot-intro/)", + "Mozilla/5.0 (compatible; SeznamBot/3.2-test4; +http://napoveda.seznam.cz/en/seznambot-intro/)", + "Mozilla/5.0 (compatible; SeznamBot/3.2; +http://napoveda.seznam.cz/en/seznambot-intro/)", + "Mozilla/5.0 (compatible; SeznamBot/4.0; +http://napoveda.seznam.cz/seznambot-intro/)" + ], + "description": "Seznam search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "ec2linkfinder", + "addition_date": "2012/03/22", + "instances": [ + "ec2linkfinder" + ], + "description": "EC2 link finder web crawler bot", + "tags": [ + "scanner" + ] + }, + { + "pattern": "gslfbot", + "addition_date": "2012/04/03", + "instances": [], + "description": "GSLFBOT web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "aiHitBot", + "addition_date": "2012/04/16", + "instances": [ + "Mozilla/5.0 (compatible; aiHitBot/2.9; +https://www.aihitdata.com/about)" + ], + "description": "AiHit web crawler for data collection", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "intelium_bot", + "addition_date": "2012/05/07", + "instances": [], + "description": "Intelium web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "facebookexternalhit", + "addition_date": "2012/05/07", + "instances": [ + "facebookexternalhit/1.0 (+http://www.facebook.com/externalhit_uatext.php)", + "facebookexternalhit/1.1", + "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" + ], + "url": "https://developers.facebook.com/docs/sharing/webmasters/crawler/", + "description": "Facebook external hit web crawler bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Yeti", + "addition_date": "2012/05/07", + "url": "http://naver.me/bot", + "instances": [ + "Mozilla/5.0 (compatible; Yeti/1.1; +http://naver.me/bot)" + ], + "description": "Naver Yeti search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "RetrevoPageAnalyzer", + "addition_date": "2012/05/07", + "instances": [ + "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; RetrevoPageAnalyzer; +http://www.retrevo.com/content/about-us)" + ], + "description": "Retrevo page analyzer web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "lb-spider", + "addition_date": "2012/05/07", + "instances": [], + "description": "LB spider web crawler for content discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Sogou", + "addition_date": "2012/05/13", + "url": "http://www.sogou.com/docs/help/webmasters.htm#07", + "instances": [ + "Sogou News Spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)", + "Sogou Pic Spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)", + "Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" + ], + "description": "Sogou search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "lssbot", + "addition_date": "2012/05/15", + "url": "https://www.lssbot.com/", + "instances": [], + "description": "LSS web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "careerbot", + "addition_date": "2012/05/23", + "url": "http://www.career-x.de/bot.html", + "instances": [], + "description": "Career-X web crawler for job discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "wotbox", + "addition_date": "2012/06/12", + "url": "http://www.wotbox.com", + "instances": [ + "Wotbox/2.0 (bot@wotbox.com; http://www.wotbox.com)", + "Wotbox/2.01 (+http://www.wotbox.com/bot/)" + ], + "description": "Wotbox web crawler for content discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "wocbot", + "addition_date": "2012/07/25", + "url": "http://www.wocodi.com/crawler", + "instances": [], + "description": "Wocodi web crawler for content discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "ichiro", + "addition_date": "2012/08/28", + "url": "http://help.goo.ne.jp/help/article/1142", + "instances": [ + "DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)", + "DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)", + "DoCoMo/2.0 P900i(c100;TB;W24H11) (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)", + "DoCoMo/2.0 P900i(c100;TB;W24H11)(compatible; ichiro/mobile goo;+http://help.goo.ne.jp/door/crawler.html)", + "DoCoMo/2.0 P901i(c100;TB;W24H11) (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/door/crawler.html)", + "KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +http://help.goo.ne.jp/help/article/1142/)", + "KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo; +http://search.goo.ne.jp/option/use/sub4/sub4-1/)", + "KDDI-CA31 UP.Browser/6.2.0.7.3.129 (GUI) MMP/2.0 (compatible; ichiro/mobile goo;+http://search.goo.ne.jp/option/use/sub4/sub4-1/)", + "ichiro/2.0 (http://help.goo.ne.jp/door/crawler.html)", + "ichiro/2.0 (ichiro@nttr.co.jp)", + "ichiro/3.0 (http://help.goo.ne.jp/door/crawler.html)", + "ichiro/3.0 (http://help.goo.ne.jp/help/article/1142)", + "ichiro/3.0 (http://search.goo.ne.jp/option/use/sub4/sub4-1/)", + "ichiro/4.0 (http://help.goo.ne.jp/door/crawler.html)", + "ichiro/5.0 (http://help.goo.ne.jp/door/crawler.html)" + ], + "description": "Goo ichiro search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "DuckDuckBot", + "addition_date": "2012/09/19", + "url": "http://duckduckgo.com/duckduckbot.html", + "instances": [ + "DuckDuckBot/1.0; (+http://duckduckgo.com/duckduckbot.html)", + "DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)", + "Mozilla/5.0 (compatible; DuckDuckBot-Https/1.1; https://duckduckgo.com/duckduckbot)", + "'Mozilla/5.0 (compatible; DuckDuckBot-Https/1.1; https://duckduckgo.com/duckduckbot)'" + ], + "description": "DuckDuckGo search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "lssrocketcrawler", + "addition_date": "2012/09/24", + "instances": [], + "description": "LSS Rocket web crawler for content", + "tags": [ + "seo" + ] + }, + { + "pattern": "drupact", + "addition_date": "2012/09/27", + "url": "http://www.arocom.de/drupact", + "instances": [ + "drupact/0.7; http://www.arocom.de/drupact" + ], + "description": "Drupact web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "webcompanycrawler", + "addition_date": "2012/10/03", + "instances": [], + "description": "Web Company web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "acoonbot", + "addition_date": "2012/10/07", + "url": "http://www.acoon.de/robot.asp", + "instances": [], + "description": "Acoon web crawler for content discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "openindexspider", + "addition_date": "2012/10/26", + "url": "http://www.openindex.io/en/webmasters/spider.html", + "instances": [], + "description": "OpenIndex web crawler for indexing", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "gnam gnam spider", + "addition_date": "2012/10/31", + "instances": [], + "description": "Gnam Gnam web crawler for discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "web-archive-net\\.com\\.bot", + "instances": [], + "description": "Web Archive web crawler for preservation", + "tags": [ + "archiver" + ] + }, + { + "pattern": "backlinkcrawler", + "addition_date": "2013/01/04", + "url": "http://www.backlinktest.com/crawler.html", + "instances": [], + "description": "Backlink Crawler web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "coccoc", + "addition_date": "2013/01/04", + "url": "http://help.coccoc.vn/", + "instances": [ + "Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/)", + "Mozilla/5.0 (compatible; coccoc/1.0; +http://help.coccoc.com/searchengine)", + "Mozilla/5.0 (compatible; coccocbot-image/1.0; +http://help.coccoc.com/searchengine)", + "Mozilla/5.0 (compatible; coccocbot-web/1.0; +http://help.coccoc.com/searchengine)", + "Mozilla/5.0 (compatible; image.coccoc/1.0; +http://help.coccoc.com/)", + "Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/)", + "Mozilla/5.0 (compatible; imagecoccoc/1.0; +http://help.coccoc.com/searchengine)", + "coccoc", + "coccoc/1.0 ()", + "coccoc/1.0 (http://help.coccoc.com/)", + "coccoc/1.0 (http://help.coccoc.vn/)" + ], + "description": "Coccoc search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "integromedb", + "addition_date": "2013/01/10", + "url": "http://www.integromedb.org/Crawler", + "instances": [ + "www.integromedb.org/Crawler" + ], + "description": "IntegromeDB web crawler for research", + "tags": [ + "academic" + ] + }, + { + "pattern": "content crawler spider", + "addition_date": "2013/01/11", + "instances": [], + "description": "Content Crawler web crawler for discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "toplistbot", + "addition_date": "2013/02/05", + "instances": [], + "description": "TopList web crawler for ranking", + "tags": [ + "seo" + ] + }, + { + "pattern": "it2media-domain-crawler", + "addition_date": "2013/03/12", + "instances": [ + "it2media-domain-crawler/1.0 on crawler-prod.it2media.de", + "it2media-domain-crawler/2.0" + ], + "description": "IT2Media domain web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "ip-web-crawler\\.com", + "addition_date": "2013/03/22", + "instances": [], + "description": "IP Web Crawler web crawler bot", + "tags": [ + "scanner" + ] + }, + { + "pattern": "siteexplorer\\.info", + "addition_date": "2013/05/01", + "instances": [ + "Mozilla/5.0 (compatible; SiteExplorer/1.0b; +http://siteexplorer.info/)", + "Mozilla/5.0 (compatible; SiteExplorer/1.1b; +http://siteexplorer.info/Backlink-Checker-Spider/)" + ], + "description": "Site Explorer web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "elisabot", + "addition_date": "2013/06/27", + "instances": [], + "description": "Elisa web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "proximic", + "addition_date": "2013/09/12", + "url": "http://www.proximic.com/info/spider.php", + "instances": [ + "Mozilla/5.0 (compatible; proximic; +http://www.proximic.com)", + "Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)" + ], + "description": "Proximic web crawler for content analysis", + "tags": [ + "advertising" + ] + }, + { + "pattern": "changedetection", + "addition_date": "2013/09/13", + "url": "http://www.changedetection.com/bot.html", + "instances": [ + "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; http://www.changedetection.com/bot.html )" + ], + "description": "ChangeDetection web crawler for monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "arabot", + "addition_date": "2013/10/09", + "instances": [], + "description": "Arabot web crawler for content discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "WeSEE:Search", + "addition_date": "2013/11/18", + "instances": [ + "WeSEE:Search", + "WeSEE:Search/0.1 (Alpha, http://www.wesee.com/en/support/bot/)" + ], + "description": "WeSEE search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "niki-bot", + "addition_date": "2014/01/01", + "instances": [], + "description": "Niki web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "CrystalSemanticsBot", + "addition_date": "2014/02/17", + "url": "http://www.crystalsemantics.com/user-agent/", + "instances": [], + "description": "Crystal Semantics web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "rogerbot", + "addition_date": "2014/02/28", + "url": "http://moz.com/help/pro/what-is-rogerbot-", + "instances": [ + "Mozilla/5.0 (compatible; rogerBot/1.0; UrlCrawler; http://www.seomoz.org/dp/rogerbot)", + "rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+partager@moz.com)", + "rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+shiny@moz.com)", + "rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-wherecat@moz.com", + "rogerbot/1.0 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-wherecat@moz.com)", + "rogerbot/1.0 (http://www.moz.com/dp/rogerbot, rogerbot-crawler@moz.com)", + "rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler+shiny@seomoz.org)", + "rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-crawler@seomoz.org)", + "rogerbot/1.0 (http://www.seomoz.org/dp/rogerbot, rogerbot-wherecat@moz.com)", + "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr2-crawler-05@moz.com)", + "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr4-crawler-11@moz.com)", + "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler+pr4-crawler-15@moz.com)", + "rogerbot/1.2 (http://moz.com/help/pro/what-is-rogerbot-, rogerbot-crawler+phaser-testing-crawler-01@moz.com)" + ], + "description": "Moz RogerBot web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "360Spider", + "addition_date": "2014/03/14", + "url": "http://needs-be.blogspot.co.uk/2013/02/how-to-block-spider360.html", + "instances": [ + "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider", + "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1; 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)", + "Mozilla/5.0 (Windows NT 6.2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 QIHU 360SE; 360Spider", + "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider", + "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11; 360Spider", + "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Firefox/1.5.0.11 360Spider;", + "Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; rv:1.8.0.11) Gecko/20070312 Firefox/1.5.0.11; 360Spider", + "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider", + "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; http://www.haosou.com/help/help_3_2.html)", + "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36; 360Spider" + ], + "description": "360 search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "psbot", + "addition_date": "2014/03/31", + "url": "http://www.picsearch.com/bot.html", + "instances": [ + "psbot-image (+http://www.picsearch.com/bot.html)", + "psbot-page (+http://www.picsearch.com/bot.html)", + "psbot/0.1 (+http://www.picsearch.com/bot.html)" + ], + "description": "PicSearch web crawler for image discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "InterfaxScanBot", + "addition_date": "2014/03/31", + "url": "http://scan-interfax.ru", + "instances": [], + "description": "Interfax scan web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "CC Metadata Scaper", + "addition_date": "2014/04/01", + "url": "http://wiki.creativecommons.org/Metadata_Scraper", + "instances": [ + "CC Metadata Scaper http://wiki.creativecommons.org/Metadata_Scraper" + ], + "description": "Creative Commons metadata web crawler", + "tags": [ + "archiver" + ] + }, + { + "pattern": "g00g1e\\.net", + "addition_date": "2014/04/01", + "url": "http://www.g00g1e.net/", + "instances": [], + "description": "G00g1e web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "GrapeshotCrawler", + "addition_date": "2014/04/01", + "url": "http://www.grapeshot.co.uk/crawler.php", + "instances": [ + "Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)" + ], + "description": "Grapeshot web crawler for content analysis", + "tags": [ + "advertising" + ] + }, + { + "pattern": "urlappendbot", + "addition_date": "2014/05/10", + "url": "http://www.profound.net/urlappendbot.html", + "instances": [ + "Mozilla/5.0 (compatible; URLAppendBot/1.0; +http://www.profound.net/urlappendbot.html)" + ], + "description": "URL Append web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "brainobot", + "addition_date": "2014/06/24", + "instances": [], + "description": "Braino web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "fr-crawler", + "addition_date": "2014/07/31", + "instances": [ + "Mozilla/5.0 (compatible; fr-crawler/1.1)" + ], + "description": "FR Crawler web crawler for content discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "binlar", + "addition_date": "2014/09/12", + "instances": [ + "binlar_2.6.3 binlar2.6.3@unspecified.mail", + "binlar_2.6.3 binlar_2.6.3@unspecified.mail", + "binlar_2.6.3 larbin2.6.3@unspecified.mail", + "binlar_2.6.3 phanendra_kalapala@McAfee.com", + "binlar_2.6.3 test@mgmt.mic" + ], + "description": "Binlar web crawler for content discovery", + "tags": [ + "scanner" + ] + }, + { + "pattern": "SimpleCrawler", + "addition_date": "2014/09/12", + "instances": [ + "SimpleCrawler/0.1" + ], + "description": "Simple Crawler web crawler framework", + "tags": [ + "http-library" + ] + }, + { + "pattern": "Twitterbot", + "addition_date": "2014/09/12", + "url": "https://dev.twitter.com/cards/getting-started", + "instances": [ + "Twitterbot/0.1", + "Twitterbot/1.0" + ], + "description": "Twitter web crawler for link previews", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "cXensebot", + "addition_date": "2014/10/05", + "instances": [ + "cXensebot/1.1a" + ], + "url": "http://www.cxense.com/bot.html", + "description": "CXense web crawler for content analysis", + "tags": [ + "advertising" + ] + }, + { + "pattern": "smtbot", + "addition_date": "2014/10/04", + "instances": [ + "Mozilla/5.0 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)", + "SMTBot (similartech.com/smtbot)", + "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)", + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36 (compatible; SMTBot/1.0; +http://www.similartech.com/smtbot)", + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36 (compatible; SMTBot/1.0; http://www.similartech.com/smtbot)" + ], + "url": "http://www.similartech.com/smtbot", + "description": "SimilarTech web crawler for technology detection", + "tags": [ + "seo" + ] + }, + { + "pattern": "bnf\\.fr_bot", + "addition_date": "2014/11/18", + "url": "http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html", + "instances": [ + "Mozilla/5.0 (compatible; bnf.fr_bot; +http://bibnum.bnf.fr/robot/bnf.html)", + "Mozilla/5.0 (compatible; bnf.fr_bot; +http://www.bnf.fr/fr/outils/a.dl_web_capture_robot.html)" + ], + "description": "BNF French National Library web crawler", + "tags": [ + "archiver" + ] + }, + { + "pattern": "A6-Indexer", + "addition_date": "2014/12/05", + "url": "http://www.a6corp.com/a6-web-scraping-policy/", + "instances": [ + "A6-Indexer" + ], + "description": "A6 Corporation web crawler for indexing", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "ADmantX", + "addition_date": "2014/12/05", + "url": "http://www.admantx.com", + "instances": [ + "ADmantX Platform Semantic Analyzer - ADmantX Inc. - www.admantx.com - support@admantx.com" + ], + "description": "ADmantX semantic analyzer web crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Facebot", + "url": "https://developers.facebook.com/docs/sharing/best-practices#crawl", + "addition_date": "2014/12/30", + "instances": [ + "Facebot/1.0" + ], + "description": "Facebook's web crawler for social sharing", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "OrangeBot\\/", + "instances": [ + "Mozilla/5.0 (compatible; OrangeBot/2.0; support.orangebot@orange.com" + ], + "addition_date": "2015/01/12", + "description": "Orange search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "memorybot", + "url": "http://mignify.com/bot.htm", + "instances": [ + "Mozilla/5.0 (compatible; memorybot/1.21.14 +http://mignify.com/bot.html)" + ], + "addition_date": "2015/02/01", + "description": "Mignify memory web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "AdvBot", + "url": "http://advbot.net/bot.html", + "instances": [ + "Mozilla/5.0 (compatible; AdvBot/2.0; +http://advbot.net/bot.html)" + ], + "addition_date": "2015/02/01", + "description": "AdvBot web crawler for advertising analysis", + "tags": [ + "advertising" + ] + }, + { + "pattern": "MegaIndex", + "url": "https://www.megaindex.ru/?tab=linkAnalyze", + "instances": [ + "Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +https://www.megaindex.ru/?tab=linkAnalyze)", + "Mozilla/5.0 (compatible; MegaIndex.ru/2.0; +http://megaindex.com/crawler)" + ], + "addition_date": "2015/03/28", + "description": "MegaIndex SEO tool web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "SemanticScholarBot", + "url": "https://www.semanticscholar.org/crawler", + "instances": [ + "SemanticScholarBot/1.0 (+http://s2.allenai.org/bot.html)", + "Mozilla/5.0 (compatible) SemanticScholarBot (+https://www.semanticscholar.org/crawler)" + ], + "addition_date": "2015/03/28", + "description": "Semantic Scholar web crawler for academic content", + "tags": [ + "academic" + ] + }, + { + "pattern": "ltx71", + "url": "http://ltx71.com/", + "instances": [ + "ltx71 - (http://ltx71.com/)" + ], + "addition_date": "2015/04/04", + "description": "LTX71 web crawler for content discovery", + "tags": [ + "scanner" + ] + }, + { + "pattern": "nerdybot", + "url": "http://nerdybot.com/", + "instances": [ + "nerdybot" + ], + "addition_date": "2015/04/05", + "description": "NerdyBot web crawler for content analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "xovibot", + "url": "http://www.xovibot.net/", + "instances": [ + "Mozilla/5.0 (compatible; XoviBot/2.0; +http://www.xovibot.net/)" + ], + "addition_date": "2015/04/05", + "description": "Xovi SEO tool web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "BUbiNG", + "url": "http://law.di.unimi.it/BUbiNG.html", + "instances": [ + "BUbiNG (+http://law.di.unimi.it/BUbiNG.html)" + ], + "addition_date": "2015/04/06", + "description": "BUbiNG web crawler framework for research", + "tags": [ + "academic" + ] + }, + { + "pattern": "Qwantify", + "url": "https://www.qwant.com/", + "instances": [ + "Mozilla/5.0 (compatible; Qwantify/2.0n; +https://www.qwant.com/)/*", + "Mozilla/5.0 (compatible; Qwantify/2.4w; +https://www.qwant.com/)/2.4w", + "Mozilla/5.0 (compatible; Qwantify/Bleriot/1.1; +https://help.qwant.com/bot)", + "Mozilla/5.0 (compatible; Qwantify/Bleriot/1.2.1; +https://help.qwant.com/bot)" + ], + "addition_date": "2015/04/06", + "description": "Qwant search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "archive\\.org_bot", + "url": "http://www.archive.org/details/archive.org_bot", + "depends_on": [ + "heritrix" + ], + "instances": [ + "Mozilla/5.0 (compatible; heritrix/3.1.1-SNAPSHOT-20120116.200628 +http://www.archive.org/details/archive.org_bot)", + "Mozilla/5.0 (compatible; archive.org_bot/heritrix-1.15.4 +http://www.archive.org)", + "Mozilla/5.0 (compatible; heritrix/3.3.0-SNAPSHOT-20140702-2247 +http://archive.org/details/archive.org_bot)", + "Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)", + "Mozilla/5.0 (compatible; archive.org_bot +http://archive.org/details/archive.org_bot)", + "Mozilla/5.0 (compatible; special_archiver/3.1.1 +http://www.archive.org/details/archive.org_bot)" + ], + "addition_date": "2015/04/14", + "description": "Archive.org web crawler for preservation", + "tags": [ + "archiver" + ] + }, + { + "pattern": "Applebot", + "url": "http://www.apple.com/go/applebot", + "addition_date": "2015/04/15", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)", + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)", + "Mozilla/5.0 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)", + "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Applebot/0.3; +http://www.apple.com/go/applebot)", + "Mozilla/5.0 (iPhone; CPU iPhone OS 8_1 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12B410 Safari/600.1.4 (Applebot/0.1; +http://www.apple.com/go/applebot)" + ], + "description": "Apple's web crawler for Siri and search", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "TweetmemeBot", + "url": "http://datasift.com/bot.html", + "instances": [ + "Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0" + ], + "addition_date": "2015/04/15", + "description": "TweetMeme web crawler for social content", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "crawler4j", + "url": "https://github.com/yasserg/crawler4j", + "instances": [ + "crawler4j (http://code.google.com/p/crawler4j/)", + "crawler4j (https://github.com/yasserg/crawler4j/)" + ], + "addition_date": "2015/05/07", + "description": "Crawler4j Java web crawler framework", + "tags": [ + "http-library" + ] + }, + { + "pattern": "findxbot", + "url": "http://www.findxbot.com", + "instances": [ + "Mozilla/5.0 (compatible; Findxbot/1.0; +http://www.findxbot.com)" + ], + "addition_date": "2015/05/07", + "description": "FindX web crawler for content discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "S[eE][mM]rushBot", + "url": "http://www.semrush.com/bot.html", + "instances": [ + "Mozilla/5.0 (compatible; SemrushBot-SA/0.97; +http://www.semrush.com/bot.html)", + "Mozilla/5.0 (compatible; SemrushBot-SI/0.97; +http://www.semrush.com/bot.html)", + "Mozilla/5.0 (compatible; SemrushBot/3~bl; +http://www.semrush.com/bot.html)", + "Mozilla/5.0 (compatible; SemrushBot/0.98~bl; +http://www.semrush.com/bot.html)", + "Mozilla/5.0 (compatible; SemrushBot-BA; +http://www.semrush.com/bot.html)", + "Mozilla/5.0 (compatible; SemrushBot/6~bl; +http://www.semrush.com/bot.html)", + "Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)", + "SEMrushBot" + ], + "addition_date": "2015/05/26", + "description": "Semrush SEO tool web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "yoozBot", + "url": "http://yooz.ir", + "instances": [ + "Mozilla/5.0 (compatible; yoozBot-2.2; http://yooz.ir; info@yooz.ir)" + ], + "addition_date": "2015/05/26", + "description": "Yooz web crawler for content analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "lipperhey", + "url": "http://www.lipperhey.com/", + "instances": [ + "Mozilla/5.0 (compatible; Lipperhey Link Explorer; http://www.lipperhey.com/)", + "Mozilla/5.0 (compatible; Lipperhey SEO Service; http://www.lipperhey.com/)", + "Mozilla/5.0 (compatible; Lipperhey Site Explorer; http://www.lipperhey.com/)", + "Mozilla/5.0 (compatible; Lipperhey-Kaus-Australis/5.0; +https://www.lipperhey.com/en/about/)" + ], + "addition_date": "2015/08/26", + "description": "Lipperhey SEO tool web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Y!J", + "url": "https://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/~/%E3%82%A6%E3%82%A7%E3%83%96%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AB%E3%82%A2%E3%82%AF%E3%82%BB%E3%82%B9%E3%81%99%E3%82%8B%E3%82%B7%E3%82%B9%E3%83%86%E3%83%A0%E3%81%AE%E3%83%A6%E3%83%BC%E3%82%B6%E3%83%BC%E3%82%A8%E3%83%BC%E3%82%B8%E3%82%A7%E3%83%B3%E3%83%88%E3%81%AB%E3%81%A4%E3%81%84%E3%81%A6", + "instances": [ + "Y!J-ASR/0.1 crawler (http://www.yahoo-help.jp/app/answers/detail/p/595/a_id/42716/)", + "Y!J-BRJ/YATS crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)", + "Y!J-PSC/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)", + "Y!J-BRW/1.0 crawler (http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)", + "Mozilla/5.0 (iPhone; Y!J-BRY/YATSH crawler; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html)", + "Mozilla/5.0 (compatible; Y!J SearchMonkey/1.0 (Y!J-AGENT; http://help.yahoo.co.jp/help/jp/search/indexing/indexing-15.html))" + ], + "addition_date": "2015/05/26", + "description": "Yahoo Japan search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Domain Re-Animator Bot", + "url": "http://domainreanimator.com", + "instances": [ + "Domain Re-Animator Bot (http://domainreanimator.com) - support@domainreanimator.com" + ], + "addition_date": "2015/04/14", + "description": "Domain Re-Animator web crawler for domain analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "AddThis", + "url": "https://www.addthis.com", + "instances": [ + "AddThis.com robot tech.support@clearspring.com" + ], + "addition_date": "2015/06/02", + "description": "AddThis social sharing web crawler bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Screaming Frog SEO Spider", + "url": "http://www.screamingfrog.co.uk/seo-spider", + "instances": [ + "Screaming Frog SEO Spider/5.1" + ], + "addition_date": "2016/01/08", + "description": "Screaming Frog SEO tool web crawler", + "tags": [ + "seo" + ] + }, + { + "pattern": "MetaURI", + "url": "http://www.useragentstring.com/MetaURI_id_17683.php", + "instances": [ + "MetaURI API/2.0 +metauri.com" + ], + "addition_date": "2016/01/02", + "description": "MetaURI API web crawler for metadata", + "tags": [ + "seo" + ] + }, + { + "pattern": "Scrapy", + "url": "http://scrapy.org/", + "instances": [ + "Scrapy/1.0.3 (+http://scrapy.org)" + ], + "addition_date": "2016/01/02", + "description": "Scrapy Python web crawler framework", + "tags": [ + "http-library" + ] + }, + { + "pattern": "Livelap[bB]ot", + "url": "http://site.livelap.com/crawler", + "instances": [ + "LivelapBot/0.2 (http://site.livelap.com/crawler)", + "Livelapbot/0.1" + ], + "addition_date": "2016/01/02", + "description": "Livelap web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "OpenHoseBot", + "url": "http://www.openhose.org/bot.html", + "instances": [ + "Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)" + ], + "addition_date": "2016/01/02", + "description": "OpenHose web crawler for content analysis", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "CapsuleChecker", + "url": "http://www.capsulink.com/about", + "instances": [ + "CapsuleChecker (http://www.capsulink.com/)" + ], + "addition_date": "2016/01/02", + "description": "Capsule web crawler for link checking", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "collection@infegy\\.com", + "url": "http://infegy.com/", + "instances": [ + "Mozilla/5.0 (compatible) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.73 Safari/537.36 collection@infegy.com" + ], + "addition_date": "2016/01/03", + "description": "Infegy web crawler for social listening", + "tags": [ + "seo" + ] + }, + { + "pattern": "IstellaBot", + "url": "http://www.tiscali.it/", + "instances": [ + "Mozilla/5.0 (compatible; IstellaBot/1.23.15 +http://www.tiscali.it/)" + ], + "addition_date": "2016/01/09", + "description": "Istella web crawler for search indexing", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "DeuSu\\/", + "addition_date": "2016/01/23", + "url": "https://deusu.de/robot.html", + "instances": [ + "Mozilla/5.0 (compatible; DeuSu/0.1.0; +https://deusu.org)", + "Mozilla/5.0 (compatible; DeuSu/5.0.2; +https://deusu.de/robot.html)" + ], + "description": "DeuSu web crawler for search indexing", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "betaBot", + "addition_date": "2016/01/23", + "instances": [], + "description": "BetaBot web crawler for testing", + "tags": [ + "seo" + ] + }, + { + "pattern": "Cliqzbot\\/", + "addition_date": "2016/01/23", + "url": "http://cliqz.com/company/cliqzbot", + "instances": [ + "Mozilla/5.0 (compatible; Cliqzbot/2.0; +http://cliqz.com/company/cliqzbot)", + "Cliqzbot/0.1 (+http://cliqz.com +cliqzbot@cliqz.com)", + "Cliqzbot/0.1 (+http://cliqz.com/company/cliqzbot)", + "Mozilla/5.0 (compatible; Cliqzbot/0.1 +http://cliqz.com/company/cliqzbot)", + "Mozilla/5.0 (compatible; Cliqzbot/1.0 +http://cliqz.com/company/cliqzbot)" + ], + "description": "Cliqz search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "MojeekBot\\/", + "addition_date": "2016/01/23", + "url": "https://www.mojeek.com/bot.html", + "instances": [ + "MojeekBot/0.2 (archi; http://www.mojeek.com/bot.html)", + "Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html#relaunch)", + "Mozilla/5.0 (compatible; MojeekBot/0.2; http://www.mojeek.com/bot.html)", + "Mozilla/5.0 (compatible; MojeekBot/0.5; http://www.mojeek.com/bot.html)", + "Mozilla/5.0 (compatible; MojeekBot/0.6; +https://www.mojeek.com/bot.html)", + "Mozilla/5.0 (compatible; MojeekBot/0.6; http://www.mojeek.com/bot.html)" + ], + "description": "Mojeek search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "netEstate NE Crawler", + "addition_date": "2016/01/23", + "url": "http://www.website-datenbank.de/", + "instances": [ + "netEstate NE Crawler (+http://www.sengine.info/)", + "netEstate NE Crawler (+http://www.website-datenbank.de/)" + ], + "description": "NetEstate web crawler for domain analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "SafeSearch microdata crawler", + "addition_date": "2016/01/23", + "url": "https://safesearch.avira.com", + "instances": [ + "SafeSearch microdata crawler (https://safesearch.avira.com, safesearch-abuse@avira.com)" + ], + "description": "Avira SafeSearch web crawler for safety", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Gluten Free Crawler\\/", + "addition_date": "2016/01/23", + "url": "http://glutenfreepleasure.com/", + "instances": [ + "Mozilla/5.0 (compatible; Gluten Free Crawler/1.0; +http://glutenfreepleasure.com/)" + ], + "description": "Gluten Free Pleasure web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Sonic", + "addition_date": "2016/02/08", + "url": "http://www.yama.info.waseda.ac.jp/~crawler/info.html", + "instances": [ + "Mozilla/5.0 (compatible; RankSonicSiteAuditor/1.0; +https://ranksonic.com/ranksonic_sab.html)", + "Mozilla/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)", + "Mozzila/5.0 (compatible; Sonic/1.0; http://www.yama.info.waseda.ac.jp/~crawler/info.html)" + ], + "description": "Sonic web crawler for ranking analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Sysomos", + "addition_date": "2016/02/08", + "url": "http://www.sysomos.com", + "instances": [ + "Mozilla/5.0 (compatible; Sysomos/1.0; +http://www.sysomos.com/; Sysomos)" + ], + "description": "Sysomos web crawler for social media analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Trove", + "addition_date": "2016/02/08", + "url": "http://www.trove.com", + "instances": [], + "description": "Trove web crawler for content discovery", + "tags": [ + "archiver" + ] + }, + { + "pattern": "deadlinkchecker", + "addition_date": "2016/02/08", + "url": "http://www.deadlinkchecker.com", + "instances": [ + "www.deadlinkchecker.com Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36", + "www.deadlinkchecker.com XMLHTTP/1.0", + "www.deadlinkchecker.com XMLHTTP/1.0 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36" + ], + "description": "Dead Link Checker web crawler for link validation", + "tags": [ + "seo" + ] + }, + { + "pattern": "Slack-ImgProxy", + "addition_date": "2016/04/25", + "url": "https://api.slack.com/robots", + "instances": [ + "Slack-ImgProxy (+https://api.slack.com/robots)", + "Slack-ImgProxy 0.59 (+https://api.slack.com/robots)", + "Slack-ImgProxy 0.66 (+https://api.slack.com/robots)", + "Slack-ImgProxy 1.106 (+https://api.slack.com/robots)", + "Slack-ImgProxy 1.138 (+https://api.slack.com/robots)", + "Slack-ImgProxy 149 (+https://api.slack.com/robots)" + ], + "description": "Slack's image proxy web crawler bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Embedly", + "addition_date": "2016/04/25", + "url": "http://support.embed.ly", + "instances": [ + "Embedly +support@embed.ly", + "Mozilla/5.0 (compatible; Embedly/0.2; +http://support.embed.ly/)", + "Mozilla/5.0 (compatible; Embedly/0.2; snap; +http://support.embed.ly/)" + ], + "description": "Embedly web crawler for content embedding", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "RankActiveLinkBot", + "addition_date": "2016/06/20", + "url": "https://rankactive.com/resources/rankactive-linkbot", + "instances": [ + "Mozilla/5.0 (compatible; RankActiveLinkBot; +https://rankactive.com/resources/rankactive-linkbot)" + ], + "description": "RankActive web crawler for link analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "iskanie", + "addition_date": "2016/09/02", + "url": "http://www.iskanie.com", + "instances": [ + "iskanie (+http://www.iskanie.com)" + ], + "description": "Iskanie web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "SafeDNSBot", + "addition_date": "2016/09/10", + "url": "https://www.safedns.com/searchbot", + "instances": [ + "SafeDNSBot (https://www.safedns.com/searchbot)" + ], + "description": "SafeDNS web crawler for security analysis", + "tags": [ + "scanner" + ] + }, + { + "pattern": "SkypeUriPreview", + "addition_date": "2016/10/10", + "instances": [ + "Mozilla/5.0 (Windows NT 6.1; WOW64) SkypeUriPreview Preview/0.5" + ], + "description": "Skype's URI preview web crawler bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Veoozbot", + "addition_date": "2016/11/03", + "url": "http://www.veooz.com/veoozbot.html", + "instances": [ + "Mozilla/5.0 (compatible; Veoozbot/1.0; +http://www.veooz.com/veoozbot.html)" + ], + "description": "Veooz web crawler for marketing analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Slackbot", + "addition_date": "2016/11/03", + "url": "https://api.slack.com/robots", + "instances": [ + "Slackbot-LinkExpanding (+https://api.slack.com/robots)", + "Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)", + "Slackbot 1.0 (+https://api.slack.com/robots)" + ], + "description": "Slack's link expansion web crawler bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "redditbot", + "addition_date": "2016/11/03", + "url": "http://www.reddit.com/feedback", + "instances": [ + "Mozilla/5.0 (compatible; redditbot/1.0; +http://www.reddit.com/feedback)" + ], + "description": "Reddit's web crawler for content sharing", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "datagnionbot", + "addition_date": "2016/11/03", + "url": "http://www.datagnion.com/bot.html", + "instances": [ + "datagnionbot (+http://www.datagnion.com/bot.html)" + ], + "description": "Datagnion web crawler for data analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Google-Adwords-Instant", + "addition_date": "2016/11/03", + "url": "http://www.google.com/adsbot.html", + "instances": [ + "Google-Adwords-Instant (+http://www.google.com/adsbot.html)" + ], + "description": "Google AdWords instant web crawler bot", + "tags": [ + "advertising" + ] + }, + { + "pattern": "adbeat_bot", + "addition_date": "2016/11/04", + "instances": [ + "Mozilla/5.0 (compatible; adbeat_bot; +support@adbeat.com; support@adbeat.com)", + "adbeat_bot" + ], + "description": "AdBeat web crawler for advertising analysis", + "tags": [ + "advertising" + ] + }, + { + "pattern": "WhatsApp", + "addition_date": "2016/11/15", + "url": "https://www.whatsapp.com/", + "instances": [ + "WhatsApp", + "WhatsApp/0.3.4479 N", + "WhatsApp/0.3.4679 N", + "WhatsApp/0.3.4941 N", + "WhatsApp/2.12.15/i", + "WhatsApp/2.12.16/i", + "WhatsApp/2.12.17/i", + "WhatsApp/2.12.449 A", + "WhatsApp/2.12.453 A", + "WhatsApp/2.12.510 A", + "WhatsApp/2.12.540 A", + "WhatsApp/2.12.548 A", + "WhatsApp/2.12.555 A", + "WhatsApp/2.12.556 A", + "WhatsApp/2.16.1/i", + "WhatsApp/2.16.13 A", + "WhatsApp/2.16.2/i", + "WhatsApp/2.16.42 A", + "WhatsApp/2.16.57 A", + "WhatsApp/2.19.92 i", + "WhatsApp/2.19.175 A", + "WhatsApp/2.19.244 A", + "WhatsApp/2.19.258 A", + "WhatsApp/2.19.308 A", + "WhatsApp/2.19.330 A" + ], + "description": "WhatsApp's web crawler for link previews", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "contxbot", + "addition_date": "2017/02/25", + "instances": [ + "Mozilla/5.0 (compatible;contxbot/1.0)" + ], + "description": "Contx web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "pinterest\\.com\\/bot", + "addition_date": "2017/03/03", + "instances": [ + "Mozilla/5.0 (compatible; Pinterestbot/1.0; +http://www.pinterest.com/bot.html)", + "Pinterest/0.2 (+http://www.pinterest.com/bot.html)" + ], + "url": "http://www.pinterest.com/bot.html", + "description": "Pinterest web crawler for content discovery", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "electricmonk", + "addition_date": "2017/03/04", + "instances": [ + "Mozilla/5.0 (compatible; electricmonk/3.2.0 +https://www.duedil.com/our-crawler/)" + ], + "url": "https://www.duedil.com/our-crawler/", + "description": "DueDil electricmonk web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "GarlikCrawler", + "addition_date": "2017/03/18", + "instances": [ + "GarlikCrawler/1.2 (http://garlik.com/, crawler@garlik.com)" + ], + "url": "http://garlik.com/", + "description": "Garlik web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "BingPreview\\/", + "addition_date": "2017/04/23", + "url": "https://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0", + "instances": [ + "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b", + "Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; rv:11.0; BingPreview/1.0b) like Gecko", + "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.2; Trident/6.0; WOW64; Trident/6.0; BingPreview/1.0b)", + "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; WOW64; Trident/5.0; BingPreview/1.0b)", + "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Version/7.0 Mobile/11A465 Safari/9537.53 BingPreview/1.0b" + ], + "description": "Bing preview web crawler bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "vebidoobot", + "addition_date": "2017/05/08", + "instances": [ + "Mozilla/5.0 (compatible; vebidoobot/1.0; +https://blog.vebidoo.de/vebidoobot/" + ], + "url": "https://blog.vebidoo.de/vebidoobot/", + "description": "Vebidoo web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "FemtosearchBot", + "addition_date": "2017/05/16", + "instances": [ + "Mozilla/5.0 (compatible; FemtosearchBot/1.0; http://femtosearch.com)" + ], + "url": "http://femtosearch.com", + "description": "Femtosearch web crawler for content discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Yahoo Link Preview", + "addition_date": "2017/06/28", + "instances": [ + "Mozilla/5.0 (compatible; Yahoo Link Preview; https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html)" + ], + "url": "https://help.yahoo.com/kb/mail/yahoo-link-preview-SLN23615.html", + "description": "Yahoo link preview web crawler bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "MetaJobBot", + "addition_date": "2017/08/16", + "instances": [ + "Mozilla/5.0 (compatible; MetaJobBot; http://www.metajob.de/crawler)" + ], + "url": "http://www.metajob.de/the/crawler", + "description": "MetaJob web crawler for job discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "DomainStatsBot", + "addition_date": "2017/08/16", + "instances": [ + "DomainStatsBot/1.0 (http://domainstats.io/our-bot)" + ], + "url": "http://domainstats.io/our-bot", + "description": "DomainStats web crawler for domain analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "mindUpBot", + "addition_date": "2017/08/16", + "instances": [ + "mindUpBot (datenbutler.de)" + ], + "url": "http://www.datenbutler.de/", + "description": "MindUp web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "Daum\\/", + "addition_date": "2017/08/16", + "instances": [ + "Mozilla/5.0 (compatible; Daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqId=28966)" + ], + "url": "http://cs.daum.net/faq/15/4118.html?faqId=28966", + "description": "Daum search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Jugendschutzprogramm-Crawler", + "addition_date": "2017/08/16", + "instances": [ + "Jugendschutzprogramm-Crawler; Info: http://www.jugendschutzprogramm.de" + ], + "url": "http://www.jugendschutzprogramm.de", + "description": "Jugendschutzprogramm web crawler bot", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Xenu Link Sleuth", + "addition_date": "2017/08/19", + "instances": [ + "Xenu Link Sleuth/1.3.8" + ], + "url": "http://home.snafu.de/tilman/xenulink.html", + "description": "Xenu link checker web crawler tool", + "tags": [ + "seo" + ] + }, + { + "pattern": "Pcore-HTTP", + "addition_date": "2017/08/19", + "instances": [ + "Pcore-HTTP/v0.40.3", + "Pcore-HTTP/v0.44.0" + ], + "url": "https://bitbucket.org/softvisio/pcore/overview", + "description": "Pcore HTTP web crawler library", + "tags": [ + "http-library" + ] + }, + { + "pattern": "moatbot", + "addition_date": "2017/09/16", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.111 Safari/537.36 moatbot", + "Mozilla/5.0 (iPhone; CPU iPhone OS 8_0 like Mac OS X) AppleWebKit/600.1.3 (KHTML, like Gecko) Version/8.0 Mobile/12A4345d Safari/600.1.4 moatbot" + ], + "url": "https://moat.com", + "description": "Moat web crawler for advertising analysis", + "tags": [ + "advertising" + ] + }, + { + "pattern": "KosmioBot", + "addition_date": "2017/09/16", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.36 (compatible; KosmioBot/1.0; +http://kosm.io/bot.html)" + ], + "url": "http://kosm.io/bot.html", + "description": "Kosmio web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "[pP]ingdom", + "addition_date": "2017/09/16", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/59.0.3071.109 Chrome/59.0.3071.109 Safari/537.36 PingdomPageSpeed/1.0 (pingbot/2.0; +http://www.pingdom.com/)", + "Mozilla/5.0 (compatible; pingbot/2.0; +http://www.pingdom.com/)", + "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)", + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/61.0.3163.100 Chrome/61.0.3163.100 Safari/537.36 PingdomPageSpeed/1.0 (pingbot/2.0; +http://www.pingdom.com/)", + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) browser/2020.2.1 Chrome/78.0.3904.130 Electron/7.3.2 Safari/537.36 PingdomTMS/2020.2", + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) browser/2020.2.5 Chrome/78.0.3904.130 Electron/7.3.15 Safari/537.36 PingdomTMS/2020.2", + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) browser/2020.2.0 Chrome/78.0.3904.130 Electron/7.1.7 Safari/537.36 PingdomTMS/2020.2", + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) renderer/2020.2.0 Chrome/78.0.3904.130 Electron/7.1.7 Safari/537.36 PingdomTMS/2020.2", + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/61.0.3163.100 Chrome/61.0.3163.100 Safari/537.36 PingdomPageSpeed/1.0 (pingbot/2.0; http://www.pingdom.com/)" + ], + "url": "http://www.pingdom.com", + "description": "Pingdom website monitoring web crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "AppInsights", + "addition_date": "2019/03/09", + "instances": [ + "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0; AppInsights)" + ], + "url": "https://docs.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview", + "description": "Microsoft AppInsights web crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "PhantomJS", + "addition_date": "2017/09/18", + "instances": [ + "Mozilla/5.0 (Unknown; Linux x86_64) AppleWebKit/538.1 (KHTML, like Gecko) PhantomJS/2.1.1 Safari/538.1 bl.uk_lddc_renderbot/2.0.0 (+ http://www.bl.uk/aboutus/legaldeposit/websites/websites/faqswebmaster/index.html)" + ], + "url": "http://phantomjs.org/", + "description": "PhantomJS headless browser web crawler", + "tags": [ + "browser-automation" + ] + }, + { + "pattern": "Gowikibot", + "addition_date": "2017/10/26", + "instances": [ + "Mozilla/5.0 (compatible; Gowikibot/1.0; +http://www.gowikibot.com)" + ], + "url": "http://www.gowikibot.com", + "description": "GoWiki web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "PiplBot", + "addition_date": "2017/10/30", + "instances": [ + "PiplBot (+http://www.pipl.com/bot/)", + "Mozilla/5.0+(compatible;+PiplBot;+http://www.pipl.com/bot/)" + ], + "url": "http://www.pipl.com/bot/", + "description": "Pipl web crawler for people search", + "tags": [ + "seo" + ] + }, + { + "pattern": "Discordbot", + "addition_date": "2017/09/22", + "url": "https://discordapp.com", + "instances": [ + "Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)" + ], + "description": "Discord web crawler for link previews", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "TelegramBot", + "addition_date": "2017/10/01", + "instances": [ + "TelegramBot (like TwitterBot)" + ], + "description": "Telegram web crawler for link previews", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Jetslide", + "addition_date": "2017/09/27", + "url": "http://jetsli.de/crawler", + "instances": [ + "Mozilla/5.0 (compatible; Jetslide; +http://jetsli.de/crawler)" + ], + "description": "Jetslide web crawler for content discovery", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "newsharecounts", + "addition_date": "2017/09/30", + "url": "http://newsharecounts.com/crawler", + "instances": [ + "Mozilla/5.0 (compatible; NewShareCounts.com/1.0; +http://newsharecounts.com/crawler)" + ], + "description": "NewShareCounts web crawler for sharing", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "James BOT", + "addition_date": "2017/10/12", + "url": "http://cognitiveseo.com/bot.html", + "instances": [ + "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6 - James BOT - WebCrawler http://cognitiveseo.com/bot.html" + ], + "description": "CognitiveSEO James web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "Bark[rR]owler", + "addition_date": "2017/10/09", + "url": "http://www.exensa.com/crawl", + "instances": [ + "Barkrowler/0.5.1 (experimenting / debugging - sorry for your logs ) http://www.exensa.com/crawl - admin@exensa.com -- based on BuBiNG", + "Barkrowler/0.7 (+http://www.exensa.com/crawl)", + "BarkRowler/0.7 (+http://www.exensa.com/crawling)", + "Barkrowler/0.9 (+http://www.exensa.com/crawl)" + ], + "description": "Barkrowler web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "TinEye", + "addition_date": "2017/10/14", + "url": "http://www.tineye.com/crawler.html", + "instances": [ + "Mozilla/5.0 (compatible; TinEye-bot/1.31; +http://www.tineye.com/crawler.html)", + "TinEye/1.1 (http://tineye.com/crawler.html)" + ], + "description": "TinEye reverse image search web crawler", + "tags": [ + "seo" + ] + }, + { + "pattern": "SocialRankIOBot", + "addition_date": "2017/10/19", + "url": "http://socialrank.io/about", + "instances": [ + "SocialRankIOBot; http://socialrank.io/about" + ], + "description": "SocialRank web crawler for social analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "trendictionbot", + "addition_date": "2017/10/30", + "url": "http://www.trendiction.de/bot", + "instances": [ + "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.0; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20071127 Firefox/3.0.0.11", + "Mozilla/5.0 (Windows NT 10.0; Win64; x64; trendictionbot0.5.0; trendiction search; http://www.trendiction.de/bot; please let us know of any problems; web at trendiction.com) Gecko/20170101 Firefox/67.0" + ], + "description": "Trendiction web crawler for trend analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Ocarinabot", + "addition_date": "2017/09/27", + "instances": [ + "Ocarinabot" + ], + "description": "Ocarina web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "epicbot", + "addition_date": "2017/10/31", + "url": "http://www.epictions.com/epicbot", + "instances": [ + "Mozilla/5.0 (compatible; epicbot; +http://www.epictions.com/epicbot)" + ], + "description": "Epic web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "Primalbot", + "addition_date": "2017/09/27", + "url": "https://www.primal.com", + "instances": [ + "Mozilla/5.0 (compatible; Primalbot; +https://www.primal.com;)" + ], + "description": "Primal web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "DuckDuckGo-Favicons-Bot", + "addition_date": "2017/10/06", + "url": "http://duckduckgo.com", + "instances": [ + "Mozilla/5.0 (compatible; DuckDuckGo-Favicons-Bot/1.0; +http://duckduckgo.com)" + ], + "description": "DuckDuckGo favicon web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "GnowitNewsbot", + "addition_date": "2017/10/30", + "url": "http://www.gnowit.com", + "instances": [ + "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:49.0) Gecko/20100101 Firefox/49.0 / GnowitNewsbot / Contact information at http://www.gnowit.com" + ], + "description": "Gnowit news web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "Leikibot", + "addition_date": "2017/09/24", + "url": "http://www.leiki.com", + "instances": [ + "Mozilla/5.0 (Windows NT 6.3;compatible; Leikibot/1.0; +http://www.leiki.com)" + ], + "description": "Leiki web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "LinkArchiver", + "addition_date": "2017/09/24", + "url": "https://github.com/thisisparker/linkarchiver", + "instances": [ + "@LinkArchiver twitter bot" + ], + "description": "LinkArchiver web crawler for archiving", + "tags": [ + "archiver" + ] + }, + { + "pattern": "YaK\\/", + "addition_date": "2017/09/25", + "url": "http://linkfluence.com", + "instances": [ + "Mozilla/5.0 (compatible; YaK/1.0; http://linkfluence.com/; bot@linkfluence.com)" + ], + "description": "Linkfluence YaK web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "PaperLiBot", + "addition_date": "2017/09/25", + "url": "http://support.paper.li/entries/20023257-what-is-paper-li", + "instances": [ + "Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)", + "Mozilla/5.0 (compatible; PaperLiBot/2.1; https://support.paper.li/entries/20023257-what-is-paper-li)" + ], + "description": "Paper.li web crawler for content curation", + "tags": [ + "seo" + ] + }, + { + "pattern": "Digg Deeper", + "addition_date": "2017/09/26", + "url": "http://digg.com/about", + "instances": [ + "Digg Deeper/v1 (http://digg.com/about)" + ], + "description": "Digg web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "^dcrawl", + "addition_date": "2017/09/22", + "url": "https://github.com/kgretzky/dcrawl", + "instances": [ + "dcrawl/1.0" + ], + "description": "dcrawl web crawler framework", + "tags": [ + "seo" + ] + }, + { + "pattern": "Snacktory", + "addition_date": "2017/09/23", + "url": "https://github.com/karussell/snacktory", + "instances": [ + "Mozilla/5.0 (compatible; Snacktory; +https://github.com/karussell/snacktory)" + ], + "description": "Snacktory web crawler for content extraction", + "tags": [ + "seo" + ] + }, + { + "pattern": "AndersPinkBot", + "addition_date": "2017/09/24", + "url": "http://anderspink.com/bot.html", + "instances": [ + "Mozilla/5.0 (compatible; AndersPinkBot/1.0; +http://anderspink.com/bot.html)" + ], + "description": "AndersPink web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "Fyrebot", + "addition_date": "2017/09/22", + "instances": [ + "Fyrebot/1.0" + ], + "description": "Fyre web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "EveryoneSocialBot", + "addition_date": "2017/09/22", + "url": "http://everyonesocial.com", + "instances": [ + "Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; support@everyonesocial.com http://everyonesocial.com/)" + ], + "description": "EveryoneSocial web crawler for sharing", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Mediatoolkitbot", + "addition_date": "2017/10/06", + "url": "http://mediatoolkit.com", + "instances": [ + "Mediatoolkitbot (complaints@mediatoolkit.com)" + ], + "description": "Mediatoolkit web crawler for monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Luminator-robots", + "addition_date": "2017/09/22", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.13 (KHTML, like Gecko) Chrome/30.0.1599.66 Safari/537.13 Luminator-robots/2.0" + ], + "description": "Luminator web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "ExtLinksBot", + "addition_date": "2017/11/02", + "url": "https://extlinks.com/Bot.html", + "instances": [ + "Mozilla/5.0 (compatible; ExtLinksBot/1.5 +https://extlinks.com/Bot.html)" + ], + "description": "ExtLinks web crawler for link analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "SurveyBot", + "addition_date": "2017/11/02", + "instances": [ + "Mozilla/5.0 (Windows; U; Windows NT 5.1; en; rv:1.9.0.13) Gecko/2009073022 Firefox/3.5.2 (.NET CLR 3.5.30729) SurveyBot/2.3 (DomainTools)" + ], + "description": "DomainTools survey web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "NING\\/", + "addition_date": "2017/11/02", + "instances": [ + "NING/1.0" + ], + "description": "Ning web crawler for content discovery", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "okhttp", + "addition_date": "2017/11/02", + "instances": [ + "okhttp/2.5.0", + "okhttp/2.7.5", + "okhttp/3.2.0", + "okhttp/3.5.0", + "okhttp/4.1.0" + ], + "description": "OkHttp Java HTTP client library", + "tags": [ + "http-library" + ] + }, + { + "pattern": "Nuzzel", + "addition_date": "2017/11/02", + "instances": [ + "Nuzzel" + ], + "description": "Nuzzel web crawler for news discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "omgili", + "addition_date": "2017/11/02", + "url": "http://omgili.com", + "instances": [ + "omgili/0.5 +http://omgili.com" + ], + "description": "Omgili web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "PocketParser", + "addition_date": "2017/11/02", + "url": "https://getpocket.com/pocketparser_ua", + "instances": [ + "PocketParser/2.0 (+https://getpocket.com/pocketparser_ua)" + ], + "description": "Pocket web crawler for content parsing", + "tags": [ + "seo" + ] + }, + { + "pattern": "YisouSpider", + "addition_date": "2017/11/02", + "instances": [ + "YisouSpider", + "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.81 YisouSpider/5.0 Safari/537.36" + ], + "description": "Yisou search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "um-LN", + "addition_date": "2017/11/02", + "instances": [ + "Mozilla/5.0 (compatible; um-LN/1.0; mailto: techinfo@ubermetrics-technologies.com)" + ], + "description": "Ubermetrics web crawler for monitoring", + "tags": [ + "seo" + ] + }, + { + "pattern": "ToutiaoSpider", + "addition_date": "2017/11/02", + "url": "http://web.toutiao.com/media_cooperation/", + "instances": [ + "Mozilla/5.0 (compatible; ToutiaoSpider/1.0; http://web.toutiao.com/media_cooperation/;)" + ], + "description": "Toutiao news platform web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "MuckRack", + "addition_date": "2017/11/02", + "url": "http://muckrack.com", + "instances": [ + "Mozilla/5.0 (compatible; MuckRack/1.0; +http://muckrack.com)" + ], + "description": "MuckRack journalist database web crawler", + "tags": [ + "seo" + ] + }, + { + "pattern": "Jamie's Spider", + "addition_date": "2017/11/02", + "url": "http://jamiembrown.com/", + "instances": [ + "Jamie's Spider (http://jamiembrown.com/)" + ], + "description": "Jamie Brown's personal web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "AHC\\/", + "addition_date": "2017/11/02", + "url": "https://github.com/AsyncHttpClient/async-http-client", + "instances": [ + "AHC/2.0" + ], + "description": "Async HTTP Client Java library for HTTP requests", + "tags": [ + "http-library" + ] + }, + { + "pattern": "NetcraftSurveyAgent", + "addition_date": "2017/11/02", + "instances": [ + "Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com)" + ], + "description": "Netcraft survey web crawler bot", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Laserlikebot", + "addition_date": "2017/11/02", + "instances": [ + "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Laserlikebot/0.1)" + ], + "description": "Laserlike web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "^Apache-HttpClient", + "addition_date": "2017/11/02", + "instances": [ + "Apache-HttpClient/4.2.3 (java 1.5)", + "Apache-HttpClient/4.2.5 (java 1.5)", + "Apache-HttpClient/4.3.1 (java 1.5)", + "Apache-HttpClient/4.3.3 (java 1.5)", + "Apache-HttpClient/4.3.5 (java 1.5)", + "Apache-HttpClient/4.4.1 (Java/1.8.0_65)", + "Apache-HttpClient/4.5.2 (Java/1.8.0_65)", + "Apache-HttpClient/4.5.2 (Java/1.8.0_151)", + "Apache-HttpClient/4.5.2 (Java/1.8.0_161)", + "Apache-HttpClient/4.5.2 (Java/1.8.0_181)", + "Apache-HttpClient/4.5.3 (Java/1.8.0_121)", + "Apache-HttpClient/4.5.3-SNAPSHOT (Java/1.8.0_152)", + "Apache-HttpClient/4.5.7 (Java/11.0.3)", + "Apache-HttpClient/4.5.10 (Java/1.8.0_201)" + ], + "description": "Apache HTTP Client Java library for requests", + "tags": [ + "http-library" + ] + }, + { + "pattern": "AppEngine-Google", + "addition_date": "2017/11/02", + "instances": [ + "AppEngine-Google; (+http://code.google.com/appengine; appid: example)", + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36 AppEngine-Google; (+http://code.google.com/appengine; appid: s~feedly-nikon3)" + ], + "description": "Google App Engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Jetty", + "addition_date": "2017/11/02", + "instances": [ + "Jetty/9.3.z-SNAPSHOT" + ], + "description": "Jetty web server HTTP client library", + "tags": [ + "http-library" + ] + }, + { + "pattern": "Upflow", + "addition_date": "2017/11/02", + "instances": [ + "Upflow/1.0" + ], + "description": "Upflow web crawler for content analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Thinklab", + "addition_date": "2017/11/02", + "url": "thinklab.com", + "instances": [ + "Thinklab (thinklab.com)" + ], + "description": "Thinklab web crawler for research", + "tags": [ + "academic" + ] + }, + { + "pattern": "Traackr\\.com", + "addition_date": "2017/11/02", + "url": "https://www.traackr.com/", + "instances": [ + "Traackr.com" + ], + "description": "Traackr influencer marketing web crawler", + "tags": [ + "seo" + ] + }, + { + "pattern": "Twurly", + "addition_date": "2017/11/02", + "url": "http://twurly.org", + "instances": [ + "Ruby, Twurly v1.1 (http://twurly.org)" + ], + "description": "Twurly Ruby web crawler framework", + "tags": [ + "seo" + ] + }, + { + "pattern": "Mastodon", + "addition_date": "2017/11/02", + "instances": [ + "http.rb/2.2.2 (Mastodon/1.5.1; +https://example-masto-instance.org/)" + ], + "description": "Mastodon social network web crawler bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "http_get", + "addition_date": "2017/11/02", + "instances": [ + "http_get" + ], + "description": "HTTP GET command-line tool for requests", + "tags": [ + "http-library" + ] + }, + { + "pattern": "DnyzBot", + "addition_date": "2017/11/20", + "instances": [ + "Mozilla/5.0 (compatible; DnyzBot/1.0)" + ], + "description": "Dnyz web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "botify", + "addition_date": "2018/02/01", + "instances": [ + "Mozilla/5.0 (compatible; botify; http://botify.com)" + ], + "description": "Botify SEO tool web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "007ac9 Crawler", + "addition_date": "2018/02/09", + "instances": [ + "Mozilla/5.0 (compatible; 007ac9 Crawler; http://crawler.007ac9.net/)" + ], + "description": "007ac9 web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "BehloolBot", + "addition_date": "2018/02/09", + "instances": [ + "Mozilla/5.0 (compatible; BehloolBot/beta; +http://www.webeaver.com/bot)" + ], + "description": "Behlool web crawler for content analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "BrandVerity", + "addition_date": "2018/02/27", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:41.0) Gecko/20100101 Firefox/55.0 BrandVerity/1.0 (http://www.brandverity.com/why-is-brandverity-visiting-me)", + "Mozilla/5.0 (iPhone; CPU iPhone OS 7_0 like Mac OS X) AppleWebKit/537.51.1 (KHTML, like Gecko) Mobile/11A465 Twitter for iPhone BrandVerity/1.0 (http://www.brandverity.com/why-is-brandverity-visiting-me)" + ], + "url": "http://www.brandverity.com/why-is-brandverity-visiting-me", + "description": "BrandVerity web crawler for brand monitoring", + "tags": [ + "seo" + ] + }, + { + "pattern": "check_http", + "addition_date": "2018/02/09", + "instances": [ + "check_http/v2.2.1 (nagios-plugins 2.2.1)" + ], + "description": "Nagios check_http monitoring plugin", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "BDCbot", + "addition_date": "2018/02/09", + "instances": [ + "Mozilla/5.0 (Windows NT 6.1; compatible; BDCbot/1.0; +http://bigweb.bigdatacorp.com.br/faq.aspx) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36", + "Mozilla/5.0 (Windows NT 10.0; Win64; x64; BDCbot/1.0; +http://bigweb.bigdatacorp.com.br/faq.aspx) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36" + ], + "description": "BigDataCorp web crawler for data collection", + "tags": [ + "seo" + ] + }, + { + "pattern": "ZumBot", + "addition_date": "2018/02/09", + "instances": [ + "Mozilla/5.0 (compatible; ZumBot/1.0; http://help.zum.com/inquiry)" + ], + "description": "Zum search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "EZID", + "addition_date": "2018/02/09", + "instances": [ + "EZID (EZID link checker; https://ezid.cdlib.org/)" + ], + "description": "EZID link checker web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "ICC-Crawler", + "addition_date": "2018/02/28", + "instances": [ + "ICC-Crawler/2.0 (Mozilla-compatible; ; http://ucri.nict.go.jp/en/icccrawler.html)" + ], + "url": "http://ucri.nict.go.jp/en/icccrawler.html", + "description": "ICC web crawler for language research", + "tags": [ + "academic" + ] + }, + { + "pattern": "ArchiveBot", + "addition_date": "2018/02/28", + "instances": [ + "ArchiveTeam ArchiveBot/20170106.02 (wpull 2.0.2)" + ], + "url": "https://github.com/ArchiveTeam/ArchiveBot", + "description": "Archive Team web crawler for preservation", + "tags": [ + "archiver" + ] + }, + { + "pattern": "^LCC ", + "addition_date": "2018/02/28", + "instances": [ + "LCC (+http://corpora.informatik.uni-leipzig.de/crawler_faq.html)" + ], + "url": "http://corpora.informatik.uni-leipzig.de/crawler_faq.html", + "description": "Leipzig Corpora Collection web crawler", + "tags": [ + "academic" + ] + }, + { + "pattern": "filterdb\\.iss\\.net\\/crawler", + "addition_date": "2018/03/16", + "instances": [ + "Mozilla/5.0 (compatible; oBot/2.3.1; +http://filterdb.iss.net/crawler/)" + ], + "url": "http://filterdb.iss.net/crawler/", + "description": "ISS filter database web crawler", + "tags": [ + "scanner" + ] + }, + { + "pattern": "BLP_bbot", + "addition_date": "2018/03/27", + "instances": [ + "BLP_bbot/0.1" + ], + "description": "BLP web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "BomboraBot", + "addition_date": "2018/03/27", + "instances": [ + "Mozilla/5.0 (compatible; BomboraBot/1.0; +http://www.bombora.com/bot)" + ], + "url": "http://www.bombora.com/bot", + "description": "Bombora web crawler for business intelligence", + "tags": [ + "seo" + ] + }, + { + "pattern": "Buck\\/", + "addition_date": "2018/03/27", + "instances": [ + "Buck/2.2; (+https://app.hypefactors.com/media-monitoring/about.html)" + ], + "url": "https://app.hypefactors.com/media-monitoring/about.html", + "description": "Hypefactors media monitoring web crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Companybook-Crawler", + "addition_date": "2018/03/27", + "instances": [ + "Companybook-Crawler (+https://www.companybooknetworking.com/)" + ], + "url": "https://www.companybooknetworking.com/", + "description": "Companybook networking web crawler", + "tags": [ + "seo" + ] + }, + { + "pattern": "Genieo", + "addition_date": "2018/03/27", + "instances": [ + "Mozilla/5.0 (compatible; Genieo/1.0 http://www.genieo.com/webfilter.html)" + ], + "url": "http://www.genieo.com/webfilter.html", + "description": "Genieo web filter web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "magpie-crawler", + "addition_date": "2018/03/27", + "instances": [ + "magpie-crawler/1.1 (U; Linux amd64; en-GB; +http://www.brandwatch.net)" + ], + "url": "http://www.brandwatch.net", + "description": "Brandwatch magpie web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "MeltwaterNews", + "addition_date": "2018/03/27", + "instances": [ + "MeltwaterNews www.meltwater.com" + ], + "url": "http://www.meltwater.com", + "description": "Meltwater news monitoring web crawler", + "tags": [ + "seo" + ] + }, + { + "pattern": "Moreover", + "addition_date": "2018/03/27", + "instances": [ + "Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com)" + ], + "url": "http://www.moreover.com", + "description": "Moreover news aggregation web crawler", + "tags": [ + "seo" + ] + }, + { + "pattern": "newspaper\\/", + "addition_date": "2018/03/27", + "instances": [ + "newspaper/0.1.0.7", + "newspaper/0.2.5", + "newspaper/0.2.6", + "newspaper/0.2.8" + ], + "description": "Newspaper Python web scraping library", + "tags": [ + "http-library" + ] + }, + { + "pattern": "ScoutJet", + "addition_date": "2018/03/27", + "instances": [ + "Mozilla/5.0 (compatible; ScoutJet; +http://www.scoutjet.com/)" + ], + "url": "http://www.scoutjet.com/", + "description": "ScoutJet web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "(^| )sentry\\/", + "addition_date": "2018/03/27", + "instances": [ + "sentry/8.22.0 (https://sentry.io)" + ], + "url": "https://sentry.io", + "description": "Sentry error tracking web crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "StorygizeBot", + "addition_date": "2018/03/27", + "instances": [ + "Mozilla/5.0 (compatible; StorygizeBot; http://www.storygize.com)" + ], + "url": "http://www.storygize.com", + "description": "Storygize web crawler for content analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "UptimeRobot", + "addition_date": "2018/03/27", + "instances": [ + "Mozilla/5.0+(compatible; UptimeRobot/2.0; http://www.uptimerobot.com/)" + ], + "url": "http://www.uptimerobot.com/", + "description": "UptimeRobot website monitoring web crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "OutclicksBot", + "addition_date": "2018/04/21", + "instances": [ + "OutclicksBot/2 +https://www.outclicks.net/agent/VjzDygCuk4ubNmg40ZMbFqT0sIh7UfOKk8s8ZMiupUR", + "OutclicksBot/2 +https://www.outclicks.net/agent/gIYbZ38dfAuhZkrFVl7sJBFOUhOVct6J1SvxgmBZgCe", + "OutclicksBot/2 +https://www.outclicks.net/agent/PryJzTl8POCRHfvEUlRN5FKtZoWDQOBEvFJ2wh6KH5J", + "OutclicksBot/2 +https://www.outclicks.net/agent/p2i4sNUh7eylJF1S6SGgRs5mP40ExlYvsr9GBxVQG6h" + ], + "url": "https://www.outclicks.net", + "description": "Outclicks web crawler for link tracking", + "tags": [ + "advertising" + ] + }, + { + "pattern": "seoscanners", + "addition_date": "2018/05/27", + "instances": [ + "Mozilla/5.0 (compatible; seoscanners.net/1; +spider@seoscanners.net)" + ], + "url": "https://github.com/monperrus/crawler-user-agents/issues/384#issuecomment-2575367162", + "description": "SEO Scanners web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Hatena", + "addition_date": "2018/05/29", + "instances": [ + "Hatena Antenna/0.3", + "Hatena::Russia::Crawler/0.01", + "Hatena-Favicon/2 (http://www.hatena.ne.jp/faq/)", + "Hatena::Scissors/0.01", + "HatenaBookmark/4.0 (Hatena::Bookmark; Analyzer)", + "Hatena::Fetcher/0.01 (master) Furl/3.13" + ], + "description": "Hatena web services web crawler bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Google Web Preview", + "addition_date": "2018/05/31", + "instances": [ + "Mozilla/5.0 (Linux; U; Android 2.3.4; generic) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Version/4.0 Mobile Safari/537.36", + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; Google Web Preview) Chrome/27.0.1453 Safari/537.36" + ], + "description": "Google web preview web crawler bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "MauiBot", + "addition_date": "2018/06/06", + "instances": [ + "MauiBot (crawler.feedback+wc@gmail.com)" + ], + "description": "Maui web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "AlphaBot", + "addition_date": "2018/05/27", + "instances": [ + "Mozilla/5.0 (compatible; AlphaBot/3.2; +http://alphaseobot.com/bot.html)" + ], + "url": "http://alphaseobot.com/bot.html", + "description": "AlphaBot SEO web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "SBL-BOT", + "addition_date": "2018/06/06", + "instances": [ + "SBL-BOT (http://sbl.net)" + ], + "url": "http://sbl.net", + "description": "Bot of SoftByte BlackWidow", + "tags": [ + "scanner" + ] + }, + { + "pattern": "IAS crawler", + "addition_date": "2018/06/06", + "instances": [ + "IAS crawler (ias_crawler; http://integralads.com/site-indexing-policy/)" + ], + "url": "http://integralads.com/site-indexing-policy/", + "description": "Bot of Integral Ad Science, Inc.", + "tags": [ + "advertising" + ] + }, + { + "pattern": "adscanner", + "addition_date": "2018/06/24", + "instances": [ + "Mozilla/5.0 (compatible; adscanner/)" + ], + "description": "AdScanner web crawler for ad analysis", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Netvibes", + "addition_date": "2018/06/24", + "instances": [ + "Netvibes (crawler/bot; http://www.netvibes.com", + "Netvibes (crawler; http://www.netvibes.com)" + ], + "url": "http://www.netvibes.com", + "description": "Netvibes web crawler for content aggregation", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "acapbot", + "addition_date": "2018/06/27", + "instances": [ + "Mozilla/5.0 (compatible;acapbot/0.1;treat like Googlebot)", + "Mozilla/5.0 (compatible;acapbot/0.1.;treat like Googlebot)" + ], + "description": "ACAP web crawler for content analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Baidu-YunGuanCe", + "addition_date": "2018/06/27", + "instances": [ + "Baidu-YunGuanCe-Bot(ce.baidu.com)", + "Baidu-YunGuanCe-SLABot(ce.baidu.com)", + "Baidu-YunGuanCe-ScanBot(ce.baidu.com)", + "Baidu-YunGuanCe-PerfBot(ce.baidu.com)", + "Baidu-YunGuanCe-VSBot(ce.baidu.com)" + ], + "url": "https://ce.baidu.com/topic/topic20150908", + "description": "Baidu Cloud Watch", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "bitlybot", + "addition_date": "2018/06/27", + "instances": [ + "bitlybot/3.0 (+http://bit.ly/)", + "bitlybot/2.0", + "bitlybot" + ], + "url": "http://bit.ly/", + "description": "Bit.ly web crawler for link tracking", + "tags": [ + "seo" + ] + }, + { + "pattern": "blogmuraBot", + "addition_date": "2018/06/27", + "instances": [ + "blogmuraBot (+http://www.blogmura.com)" + ], + "url": "http://www.blogmura.com", + "description": "A blog ranking site which links to blogs on just about every theme possible.", + "tags": [ + "seo" + ] + }, + { + "pattern": "Bot\\.AraTurka\\.com", + "addition_date": "2018/06/27", + "instances": [ + "Bot.AraTurka.com/0.0.1" + ], + "url": "http://www.araturka.com", + "description": "AraTurka web crawler for content discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "bot-pge\\.chlooe\\.com", + "addition_date": "2018/06/27", + "instances": [ + "bot-pge.chlooe.com/1.0.0 (+http://www.chlooe.com/)" + ], + "description": "Chlooe web crawler for content analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "BoxcarBot", + "addition_date": "2018/06/27", + "instances": [ + "Mozilla/5.0 (compatible; BoxcarBot/1.1; +awesome@boxcar.io)" + ], + "url": "https://boxcar.io/", + "description": "Boxcar web crawler for notifications", + "tags": [ + "seo" + ] + }, + { + "pattern": "BTWebClient", + "addition_date": "2018/06/27", + "instances": [ + "BTWebClient/180B(9704)" + ], + "url": "http://www.utorrent.com/", + "description": "µTorrent BitTorrent Client", + "tags": [ + "http-library" + ] + }, + { + "pattern": "ContextAd Bot", + "addition_date": "2018/06/27", + "instances": [ + "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0;.NET CLR 1.0.3705; ContextAd Bot 1.0)", + "ContextAd Bot 1.0" + ], + "description": "ContextAd web crawler for advertising", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Digincore bot", + "addition_date": "2018/06/27", + "instances": [ + "Mozilla/5.0 (compatible; Digincore bot; https://www.digincore.com/crawler.html for rules and instructions.)" + ], + "url": "http://www.digincore.com/crawler.html", + "description": "Digincore web crawler for content analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Disqus", + "addition_date": "2018/06/27", + "instances": [ + "Disqus/1.0" + ], + "url": "https://disqus.com/", + "description": "validate and quality check pages.", + "tags": [ + "seo" + ] + }, + { + "pattern": "Feedly", + "addition_date": "2018/06/27", + "instances": [ + "Feedly/1.0 (+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)", + "FeedlyBot/1.0 (http://feedly.com)" + ], + "url": "https://www.feedly.com/fetcher.html", + "description": "Feedly Fetcher is how Feedly grabs RSS or Atom feeds when users choose to add them to their Feedly or any of the other applications built on top of the feedly cloud.", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Fetch\\/", + "addition_date": "2018/06/27", + "instances": [ + "Fetch/2.0a (CMS Detection/Web/SEO analysis tool, see http://guess.scritch.org)" + ], + "description": "Fetch web crawler for CMS detection", + "tags": [ + "seo" + ] + }, + { + "pattern": "Fever", + "addition_date": "2018/06/27", + "instances": [ + "Fever/1.38 (Feed Parser; http://feedafever.com; Allow like Gecko)" + ], + "url": "http://feedafever.com", + "description": "Fever feed reader web crawler bot", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Flamingo_SearchEngine", + "addition_date": "2018/06/27", + "instances": [ + "Flamingo_SearchEngine (+http://www.flamingosearch.com/bot)" + ], + "description": "Flamingo search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "FlipboardProxy", + "addition_date": "2018/06/27", + "instances": [ + "Mozilla/5.0 (compatible; FlipboardProxy/1.1; +http://flipboard.com/browserproxy)", + "Mozilla/5.0 (compatible; FlipboardProxy/1.2; +http://flipboard.com/browserproxy)", + "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)", + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)", + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:49.0) Gecko/20100101 Firefox/49.0 (FlipboardProxy/1.2; +http://flipboard.com/browserproxy)" + ], + "url": "https://about.flipboard.com/browserproxy/", + "description": "a proxy service to fetch, validate, and prepare certain elements of websites for presentation through the Flipboard Application", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "g2reader-bot", + "addition_date": "2018/06/27", + "instances": [ + "g2reader-bot/1.0 (+http://www.g2reader.com/)" + ], + "url": "http://www.g2reader.com/", + "description": "G2Reader web crawler for content discovery", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "G2 Web Services", + "addition_date": "2019/03/01", + "instances": [ + "G2 Web Services/1.0 (built with StormCrawler Archetype 1.8; https://www.g2webservices.com/; developers@g2llc.com)" + ], + "url": "https://www.g2webservices.com/", + "description": "G2 web services web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "imrbot", + "addition_date": "2018/06/27", + "instances": [ + "Mozilla/5.0 (compatible; imrbot/1.10.8 +http://www.mignify.com)" + ], + "url": "http://www.mignify.com", + "description": "Mignify imrbot web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "K7MLWCBot", + "addition_date": "2018/06/27", + "instances": [ + "K7MLWCBot/1.0 (+http://www.k7computing.com)" + ], + "url": "http://www.k7computing.com", + "description": "Virus scanner", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Kemvibot", + "addition_date": "2018/06/27", + "instances": [ + "Kemvibot/1.0 (http://kemvi.com, marco@kemvi.com)" + ], + "url": "http://kemvi.com", + "description": "Kemvi web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "Landau-Media-Spider", + "addition_date": "2018/06/27", + "instances": [ + "Landau-Media-Spider/1.0(http://bots.landaumedia.de/bot.html)" + ], + "url": "http://bots.landaumedia.de/bot.html", + "description": "Landau Media web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "linkapediabot", + "addition_date": "2018/06/27", + "instances": [ + "linkapediabot (+http://www.linkapedia.com)" + ], + "url": "http://www.linkapedia.com", + "description": "Linkapedia web crawler for link discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "vkShare", + "addition_date": "2018/07/02", + "instances": [ + "Mozilla/5.0 (compatible; vkShare; +http://vk.com/dev/Share)" + ], + "url": "http://vk.com/dev/Share", + "description": "VK Share web crawler for social sharing", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Siteimprove\\.com", + "addition_date": "2018/06/22", + "instances": [ + "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) LinkCheck by Siteimprove.com", + "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) Match by Siteimprove.com", + "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl by Siteimprove.com", + "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.0) LinkCheck by Siteimprove.com" + ], + "description": "Siteimprove web crawler for site analysis", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "BLEXBot\\/", + "addition_date": "2018/07/07", + "instances": [ + "Mozilla/5.0 (compatible; BLEXBot/1.0; +http://webmeup-crawler.com/)" + ], + "url": "http://webmeup-crawler.com", + "description": "WebMeUp BLEX web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "DareBoost", + "addition_date": "2018/07/07", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.75 Safari/537.36 DareBoost" + ], + "url": "https://www.dareboost.com/", + "description": "Bot to test, Analyze and Optimize website", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "ZuperlistBot\\/", + "addition_date": "2018/07/07", + "instances": [ + "Mozilla/5.0 (compatible; ZuperlistBot/1.0)" + ], + "description": "Zuperlist web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "Miniflux\\/", + "addition_date": "2018/07/07", + "instances": [ + "Mozilla/5.0 (compatible; Miniflux/2.0.x-dev; +https://miniflux.net)", + "Mozilla/5.0 (compatible; Miniflux/2.0.3; +https://miniflux.net)", + "Mozilla/5.0 (compatible; Miniflux/2.0.7; +https://miniflux.net)", + "Mozilla/5.0 (compatible; Miniflux/2.0.10; +https://miniflux.net)", + "Mozilla/5.0 (compatibl$; Miniflux/2.0.x-dev; +https://miniflux.app)", + "Mozilla/5.0 (compatible; Miniflux/2.0.11; +https://miniflux.app)", + "Mozilla/5.0 (compatible; Miniflux/2.0.12; +https://miniflux.app)", + "Mozilla/5.0 (compatible; Miniflux/ae1dc1a; +https://miniflux.app)", + "Mozilla/5.0 (compatible; Miniflux/3b6e44c; +https://miniflux.app)" + ], + "url": "https://miniflux.net", + "description": "Miniflux is a minimalist and opinionated feed reader.", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Feedspot", + "addition_date": "2018/07/07", + "instances": [ + "Mozilla/5.0 (compatible; Feedspotbot/1.0; +http://www.feedspot.com/fs/bot)", + "Mozilla/5.0 (compatible; Feedspot/1.0 (+https://www.feedspot.com/fs/fetcher; like FeedFetcher-Google)" + ], + "url": "http://www.feedspot.com/fs/bot", + "description": "Feedspot web crawler for feed discovery", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Diffbot\\/", + "addition_date": "2018/07/07", + "instances": [ + "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.2) Gecko/20090729 Firefox/3.5.2 (.NET CLR 3.5.30729; Diffbot/0.1; +http://www.diffbot.com)" + ], + "url": "http://www.diffbot.com", + "description": "Diffbot web crawler for content extraction", + "tags": [ + "seo" + ] + }, + { + "pattern": "SEOkicks", + "addition_date": "2018/08/22", + "instances": [ + "Mozilla/5.0 (compatible; SEOkicks; +https://www.seokicks.de/robot.html)" + ], + "url": "https://www.seokicks.de/robot.html", + "description": "SEOkicks web crawler for SEO analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "tracemyfile", + "addition_date": "2018/08/23", + "instances": [ + "Mozilla/5.0 (compatible; tracemyfile/1.0; +bot@tracemyfile.com)" + ], + "description": "TraceMyFile web crawler for file tracking", + "tags": [ + "seo" + ] + }, + { + "pattern": "Nimbostratus-Bot", + "addition_date": "2018/08/29", + "instances": [ + "Mozilla/5.0 (compatible; Nimbostratus-Bot/v1.3.2; http://cloudsystemnetworks.com)" + ], + "description": "Nimbostratus web crawler for cloud analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "zgrab", + "addition_date": "2018/08/30", + "instances": [ + "Mozilla/5.0 zgrab/0.x" + ], + "url": "https://github.com/zmap/zgrab2", + "description": "Zgrab web crawler for security scanning", + "tags": [ + "scanner" + ] + }, + { + "pattern": "PR-CY\\.RU", + "addition_date": "2018/08/30", + "instances": [ + "Mozilla/5.0 (compatible; PR-CY.RU; + https://a.pr-cy.ru)" + ], + "url": "https://a.pr-cy.ru/", + "description": "PR-CY web crawler for SEO analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "AdsTxtCrawler", + "addition_date": "2018/08/30", + "instances": [ + "AdsTxtCrawler/1.0" + ], + "description": "AdsTxt web crawler for ads.txt validation", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Datafeedwatch", + "addition_date": "2018/09/05", + "instances": [ + "Datafeedwatch/2.1.x" + ], + "url": "https://www.datafeedwatch.com/", + "description": "Datafeedwatch web crawler for feed analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Zabbix", + "addition_date": "2018/09/05", + "instances": [ + "Zabbix" + ], + "url": "https://www.zabbix.com/documentation/3.4/manual/web_monitoring", + "description": "Zabbix web crawler for monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "TangibleeBot", + "addition_date": "2018/09/05", + "instances": [ + "TangibleeBot/1.0.0.0 (http://tangiblee.com/bot)" + ], + "url": "http://tangiblee.com/bot", + "description": "Tangiblee web crawler for visual search", + "tags": [ + "seo" + ] + }, + { + "pattern": "google-xrawler", + "addition_date": "2018/09/05", + "instances": [ + "google-xrawler" + ], + "url": "https://webmasters.stackexchange.com/questions/105560/what-is-the-google-xrawler-user-agent-used-for", + "description": "Google xrawler web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "axios", + "addition_date": "2018/09/06", + "instances": [ + "axios/0.18.0", + "axios/0.19.0" + ], + "url": "https://github.com/axios/axios", + "description": "Axios HTTP client library for requests", + "tags": [ + "http-library" + ] + }, + { + "pattern": "Amazon CloudFront", + "addition_date": "2018/09/07", + "instances": [ + "Amazon CloudFront" + ], + "url": "https://aws.amazon.com/cloudfront/", + "description": "Amazon CloudFront web crawler bot", + "tags": [ + "http-library" + ] + }, + { + "pattern": "Pulsepoint ", + "addition_date": "2018/09/24", + "instances": [ + "Pulsepoint XT3 web scraper" + ], + "description": "Pulsepoint web crawler for content discovery", + "tags": [ + "advertising" + ] + }, + { + "pattern": "CloudFlare-AlwaysOnline", + "addition_date": "2018/09/27", + "instances": [ + "Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online) AppleWebKit/534.34", + "Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +https://www.cloudflare.com/always-online) AppleWebKit/534.34" + ], + "url": "https://www.cloudflare.com/always-online/", + "description": "CloudFlare always online web crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Cloudflare-Healthchecks", + "addition_date": "2024/12/17", + "instances": [ + "Mozilla/5.0 (compatible; Cloudflare-Healthchecks/1.0; +https://www.cloudflare.com/; healthcheck-id: AAAAAAAAAAAAAAAA)" + ], + "url": "https://developers.cloudflare.com/health-checks/", + "description": "CloudFlare health checks web crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Cloudflare-Traffic-Manager", + "addition_date": "2024/12/17", + "instances": [ + "Mozilla/5.0 (compatible; Cloudflare-Traffic-Manager/1.0; +https://www.cloudflare.com/traffic-manager/; pool-id: AAAAAAAAAAAAAAAA)" + ], + "url": "https://developers.cloudflare.com/load-balancing/monitors/", + "description": "CloudFlare traffic manager web crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "CloudFlare-Prefetch", + "addition_date": "2024/12/17", + "instances": [ + "Mozilla/5.0 (compatible; CloudFlare-Prefetch/0.1; +http://www.cloudflare.com/)" + ], + "url": "https://developers.cloudflare.com/speed/optimization/content/prefetch-urls/", + "description": "CloudFlare prefetch web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "Cloudflare-SSLDetector", + "addition_date": "2024/12/17", + "instances": [ + "Cloudflare-SSLDetector" + ], + "url": "https://developers.cloudflare.com/ssl/origin-configuration/ssl-tls-recommender/", + "description": "CloudFlare SSL detector web crawler", + "tags": [ + "scanner" + ] + }, + { + "pattern": "https:\\/\\/developers\\.cloudflare\\.com\\/security-center\\/", + "addition_date": "2024/12/17", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36 (compatible; +https://developers.cloudflare.com/security-center/)" + ], + "url": "https://developers.cloudflare.com/ssl/origin-configuration/ssl-tls-recommender/", + "description": "CloudFlare security center web crawler", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Google-Structured-Data-Testing-Tool", + "addition_date": "2018/10/02", + "instances": [ + "Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +https://search.google.com/structured-data/testing-tool)", + "Mozilla/5.0 (compatible; Google-Structured-Data-Testing-Tool +http://developers.google.com/structured-data/testing-tool/)" + ], + "url": "https://search.google.com/structured-data/testing-tool", + "description": "Google structured data testing web crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "WordupInfoSearch", + "addition_date": "2018/10/07", + "instances": [ + "WordupInfoSearch/1.0" + ], + "description": "Wordup info search web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "WebDataStats", + "addition_date": "2018/10/08", + "instances": [ + "Mozilla/5.0 (compatible; WebDataStats/1.0 ; +https://webdatastats.com/policy.html)" + ], + "url": "https://webdatastats.com/", + "description": "WebDataStats web crawler for data analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "HttpUrlConnection", + "addition_date": "2018/10/08", + "instances": [ + "Jersey/2.25.1 (HttpUrlConnection 1.8.0_141)" + ], + "description": "Java HttpUrlConnection HTTP client", + "tags": [ + "http-library" + ] + }, + { + "pattern": "ZoomBot", + "addition_date": "2018/10/10", + "instances": [ + "ZoomBot (Linkbot 1.0 http://suite.seozoom.it/bot.html)" + ], + "url": "http://suite.seozoom.it/bot.html", + "description": "SEOZoom web crawler for SEO analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "VelenPublicWebCrawler", + "addition_date": "2018/10/09", + "url": "https://velen.io/", + "instances": [ + "VelenPublicWebCrawler (velen.io)" + ], + "description": "Velen web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "MoodleBot", + "addition_date": "2018/10/10", + "instances": [ + "MoodleBot/1.0" + ], + "description": "Moodle web crawler for learning platforms", + "tags": [ + "seo" + ] + }, + { + "pattern": "jpg-newsbot", + "addition_date": "2018/10/10", + "instances": [ + "jpg-newsbot/2.0; (+https://vipnytt.no/bots/)" + ], + "url": "https://vipnytt.no/bots/", + "description": "JPG news web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "outbrain", + "addition_date": "2018/10/14", + "instances": [ + "Mozilla/5.0 (Java) outbrain" + ], + "url": "https://www.outbrain.com/help/advertisers/invalid-url/", + "description": "Outbrain web crawler for content discovery", + "tags": [ + "advertising" + ] + }, + { + "pattern": "W3C_Validator", + "addition_date": "2018/10/14", + "instances": [ + "W3C_Validator/1.3" + ], + "url": "https://validator.w3.org/services", + "description": "W3C HTML validator web crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Validator\\.nu", + "addition_date": "2018/10/14", + "instances": [ + "Validator.nu/LV" + ], + "url": "https://validator.w3.org/services", + "description": "Validator.nu HTML validator web crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "W3C-checklink", + "addition_date": "2018/10/14", + "depends_on": [ + "libwww-perl" + ], + "instances": [ + "W3C-checklink/2.90 libwww-perl/5.64", + "W3C-checklink/3.6.2.3 libwww-perl/5.64", + "W3C-checklink/4.2 [4.20] libwww-perl/5.803", + "W3C-checklink/4.2.1 [4.21] libwww-perl/5.803", + "W3C-checklink/4.3 [4.42] libwww-perl/5.805", + "W3C-checklink/4.3 [4.42] libwww-perl/5.808", + "W3C-checklink/4.3 [4.42] libwww-perl/5.820", + "W3C-checklink/4.5 [4.154] libwww-perl/5.823", + "W3C-checklink/4.5 [4.160] libwww-perl/5.823" + ], + "url": "https://validator.w3.org/services", + "description": "W3C checklink web crawler for link validation", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "W3C-mobileOK", + "addition_date": "2018/10/14", + "instances": [ + "W3C-mobileOK/DDC-1.0" + ], + "url": "https://validator.w3.org/services", + "description": "W3C mobile OK web crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "W3C_I18n-Checker", + "addition_date": "2018/10/14", + "instances": [ + "W3C_I18n-Checker/1.0" + ], + "url": "https://validator.w3.org/services", + "description": "W3C internationalization web crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "FeedValidator", + "addition_date": "2018/10/14", + "instances": [ + "FeedValidator/1.3" + ], + "url": "https://validator.w3.org/services", + "description": "Feed validator web crawler for RSS validation", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "W3C_CSS_Validator", + "addition_date": "2018/10/14", + "instances": [ + "Jigsaw/2.3.0 W3C_CSS_Validator_JFouffa/2.0" + ], + "url": "https://validator.w3.org/services", + "description": "W3C CSS validator web crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "W3C_Unicorn", + "addition_date": "2018/10/14", + "instances": [ + "W3C_Unicorn/1.0" + ], + "url": "https://validator.w3.org/services", + "description": "W3C Unicorn validator web crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Google-PhysicalWeb", + "addition_date": "2018/10/21", + "instances": [ + "Mozilla/5.0 (Google-PhysicalWeb)" + ], + "description": "Google physical web web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Blackboard", + "addition_date": "2018/10/28", + "instances": [ + "Blackboard Safeassign" + ], + "url": "https://help.blackboard.com/Learn/Administrator/Hosting/Tools_Management/SafeAssign", + "description": "Blackboard SafeAssign web crawler bot", + "tags": [ + "academic" + ] + }, + { + "pattern": "ICBot\\/", + "addition_date": "2018/10/23", + "instances": [ + "Mozilla/5.0 (compatible; ICBot/0.1; +https://ideasandcode.xyz" + ], + "url": "https://ideasandcode.xyz", + "description": "Ideas and Code web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "BazQux", + "addition_date": "2018/10/23", + "instances": [ + "Mozilla/5.0 (compatible; BazQux/2.4; +https://bazqux.com/fetcher; 1 subscribers)" + ], + "url": "https://bazqux.com/fetcher", + "description": "BazQux RSS reader web crawler bot", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Twingly", + "addition_date": "2018/10/23", + "instances": [ + "Mozilla/5.0 (compatible; Twingly Recon; twingly.com)" + ], + "url": "https://twingly.com", + "description": "Twingly blog search web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "Rivva", + "addition_date": "2018/10/23", + "instances": [ + "Mozilla/5.0 (compatible; Rivva; http://rivva.de)" + ], + "url": "http://rivva.de", + "description": "Rivva blog search web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "Experibot", + "addition_date": "2018/11/03", + "instances": [ + "Experibot-v2 http://goo.gl/ZAr8wX", + "Experibot-v3 http://goo.gl/ZAr8wX" + ], + "url": "https://amirkr.wixsite.com/experibot", + "description": "Experibot web crawler for testing", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "awesomecrawler", + "addition_date": "2018/11/24", + "instances": [ + "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.5 Safari/537.22 +awesomecrawler" + ], + "description": "Awesome web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "Dataprovider\\.com", + "addition_date": "2018/11/24", + "instances": [ + "Mozilla/5.0 (compatible; Dataprovider.com)" + ], + "url": "https://www.dataprovider.com/", + "description": "Dataprovider web crawler for data collection", + "tags": [ + "seo" + ] + }, + { + "pattern": "GroupHigh\\/", + "addition_date": "2018/11/24", + "instances": [ + "Mozilla/5.0 (compatible; GroupHigh/1.0; +http://www.grouphigh.com/" + ], + "url": "http://www.grouphigh.com/", + "description": "GroupHigh web crawler for influencer marketing", + "tags": [ + "seo" + ] + }, + { + "pattern": "theoldreader\\.com", + "addition_date": "2018/12/02", + "instances": [ + "Mozilla/5.0 (compatible; theoldreader.com)" + ], + "url": "https://www.theoldreader.com/", + "description": "The Old Reader web crawler bot", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "AnyEvent", + "addition_date": "2018/12/07", + "instances": [ + "Mozilla/5.0 (compatible; U; AnyEvent-HTTP/2.24; +http://software.schmorp.de/pkg/AnyEvent)" + ], + "url": "http://software.schmorp.de/pkg/AnyEvent.html", + "description": "AnyEvent Perl HTTP web crawler library", + "tags": [ + "http-library" + ] + }, + { + "pattern": "Uptimebot\\.org", + "addition_date": "2019/01/17", + "instances": [ + "Uptimebot.org - Free website monitoring" + ], + "url": "http://uptimebot.org/", + "description": "Uptimebot website monitoring web crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Nmap Scripting Engine", + "addition_date": "2019/02/04", + "instances": [ + "Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)" + ], + "url": "https://nmap.org/book/nse.html", + "description": "Nmap NSE web crawler for security scanning", + "tags": [ + "scanner" + ] + }, + { + "pattern": "2ip\\.ru", + "addition_date": "2019/02/12", + "instances": [ + "2ip.ru CMS Detector (https://2ip.ru/cms/)" + ], + "url": "https://2ip.ru/cms/", + "description": "2IP CMS detector web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "Clickagy", + "addition_date": "2019/02/19", + "instances": [ + "Clickagy Intelligence Bot v2" + ], + "url": "https://www.clickagy.com", + "description": "Clickagy intelligence web crawler bot", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Caliperbot", + "addition_date": "2019/03/02", + "instances": [ + "Caliperbot/1.0 (+http://www.conductor.com/caliperbot)" + ], + "url": "http://www.conductor.com/caliperbot", + "description": "Conductor Caliperbot web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "MBCrawler", + "addition_date": "2019/03/02", + "instances": [ + "MBCrawler/1.0 (https://monitorbacklinks.com)" + ], + "url": "https://monitorbacklinks.com", + "description": "Monitor Backlinks web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "online-webceo-bot", + "addition_date": "2019/03/02", + "instances": [ + "Mozilla/5.0 (compatible; online-webceo-bot/1.0; +http://online.webceo.com)" + ], + "url": "http://online.webceo.com", + "description": "Online WebCEO web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "B2B Bot", + "addition_date": "2019/03/02", + "instances": [ + "B2B Bot" + ], + "description": "B2B web crawler for business discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "AddSearchBot", + "addition_date": "2019/03/02", + "instances": [ + "Mozilla/5.0 (compatible; AddSearchBot/0.9; +http://www.addsearch.com/bot; info@addsearch.com)" + ], + "url": "http://www.addsearch.com/bot", + "description": "AddSearch web crawler for site search", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Google Favicon", + "addition_date": "2019/03/14", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.75 Safari/537.36 Google Favicon" + ], + "description": "Google favicon web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "HubSpot", + "addition_date": "2019/04/15", + "instances": [ + "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36 HubSpot Webcrawler - web-crawlers@hubspot.com", + "Mozilla/5.0 (X11; Linux x86_64; HubSpot Single Page link check; web-crawlers+links@hubspot.com) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36", + "Mozilla/5.0 (compatible; HubSpot Crawler; web-crawlers@hubspot.com)", + "HubSpot Connect 2.0 (http://dev.hubspot.com/) - BizOpsCompanies-Tq2-BizCoDomainValidationAudit" + ], + "description": "HubSpot web crawler for marketing automation", + "tags": [ + "seo" + ] + }, + { + "pattern": "Chrome-Lighthouse", + "addition_date": "2019/03/15", + "instances": [ + "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5 Build/MRA58N) AppleWebKit/537.36(KHTML, like Gecko) Chrome/69.0.3464.0 Mobile Safari/537.36 Chrome-Lighthouse", + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36(KHTML, like Gecko) Chrome/69.0.3464.0 Safari/537.36 Chrome-Lighthouse", + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3694.0 Safari/537.36 Chrome-Lighthouse", + "Mozilla/5.0 (Linux; Android 7.0; Moto G (4)) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4695.0 Mobile Safari/537.36 Chrome-Lighthouse", + "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3694.0 Mobile Safari/537.36 Chrome-Lighthouse" + ], + "url": "https://developers.google.com/speed/pagespeed/insights", + "description": "Chrome Lighthouse web crawler for audits", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "HeadlessChrome", + "url": "https://developers.google.com/web/updates/2017/04/headless-chrome", + "addition_date": "2019/06/17", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/74.0.3729.169 Safari/537.36", + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/69.0.3494.0 Safari/537.36", + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/76.0.3803.0 Safari/537.36" + ], + "description": "Headless Chrome web crawler for testing", + "tags": [ + "browser-automation" + ] + }, + { + "pattern": "CheckMarkNetwork\\/", + "addition_date": "2019/06/30", + "instances": [ + "CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)" + ], + "url": "https://www.checkmarknetwork.com/", + "description": "CheckMark Network web crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "www\\.uptime\\.com", + "addition_date": "2019/07/21", + "instances": [ + "Mozilla/5.0 (compatible; Uptimebot/1.0; +http://www.uptime.com/uptimebot)" + ], + "url": "http://www.uptime.com/uptimebot", + "description": "Uptime.com website monitoring web crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Streamline3Bot\\/", + "addition_date": "2019/07/21", + "instances": [ + "Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) Streamline3Bot/1.0", + "Mozilla/5.0 (Windows NT 6.1; Win64; x64; +https://www.ubtsupport.com/legal/Streamline3Bot.php) Streamline3Bot/1.0" + ], + "url": "https://www.ubtsupport.com/legal/Streamline3Bot.php", + "description": "Streamline3 web crawler for monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "serpstatbot\\/", + "addition_date": "2019/07/25", + "instances": [ + "serpstatbot/1.0 (advanced backlink tracking bot; http://serpstatbot.com/; abuse@serpstatbot.com)", + "serpstatbot/1.0 (advanced backlink tracking bot; curl/7.58.0; http://serpstatbot.com/; abuse@serpstatbot.com)" + ], + "url": "http://serpstatbot.com", + "description": "Serpstat web crawler for backlink tracking", + "tags": [ + "seo" + ] + }, + { + "pattern": "MixnodeCache\\/", + "addition_date": "2019/08/04", + "instances": [ + "MixnodeCache/1.8(+https://cache.mixnode.com/)" + ], + "url": "https://cache.mixnode.com/", + "description": "Mixnode cache web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "^curl", + "addition_date": "2019/08/15", + "instances": [ + "curl", + "curl/7.29.0", + "curl/7.47.0", + "curl/7.54.0", + "curl/7.55.1", + "curl/7.64.0", + "curl/7.64.1", + "curl/7.65.3" + ], + "url": "https://curl.haxx.se/", + "description": "cURL command-line tool for HTTP requests", + "tags": [ + "http-library" + ] + }, + { + "pattern": "SimpleScraper", + "addition_date": "2019/08/16", + "instances": [ + "Mozilla/5.0 (compatible; SimpleScraper)" + ], + "url": "https://github.com/ramonkcom/simple-scraper/", + "description": "Simple Scraper web scraping tool", + "tags": [ + "seo" + ] + }, + { + "pattern": "RSSingBot", + "addition_date": "2019/09/15", + "instances": [ + "RSSingBot (http://www.rssing.com)" + ], + "url": "http://www.rssing.com", + "description": "RSSing RSS reader web crawler bot", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Jooblebot", + "addition_date": "2019/09/25", + "instances": [ + "Mozilla/5.0 (compatible; Jooblebot/2.0; Windows NT 6.1; WOW64; +http://jooble.org/jooble-bot) AppleWebKit/537.36 (KHTML, like Gecko) Safari/537.36" + ], + "url": "http://jooble.org/jooble-bot", + "description": "Jooble job search web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "fedoraplanet", + "addition_date": "2019/09/28", + "instances": [ + "venus/fedoraplanet" + ], + "url": "http://fedoraplanet.org/", + "description": "Fedora Planet web crawler for aggregation", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Friendica", + "addition_date": "2019/09/28", + "instances": [ + "Friendica 'The Tazmans Flax-lily' 2019.01-1293; https://hoyer.xyz" + ], + "url": "https://hoyer.xyz", + "description": "Friendica social network web crawler bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "NextCloud", + "addition_date": "2019/09/30", + "instances": [ + "NextCloud-News/1.0" + ], + "url": "https://nextcloud.com/", + "description": "NextCloud news reader web crawler bot", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Tiny Tiny RSS", + "addition_date": "2019/10/04", + "instances": [ + "Tiny Tiny RSS/1.15.3 (http://tt-rss.org/)", + "Tiny Tiny RSS/17.12 (a2d1fa5) (http://tt-rss.org/)", + "Tiny Tiny RSS/19.2 (b68db2d) (http://tt-rss.org/)", + "Tiny Tiny RSS/19.8 (http://tt-rss.org/)" + ], + "url": "http://tt-rss.org/", + "description": "Tiny Tiny RSS reader web crawler bot", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "RegionStuttgartBot", + "addition_date": "2019/10/17", + "instances": [ + "Mozilla/5.0 (compatible; RegionStuttgartBot/1.0; +http://it.region-stuttgart.de/competenzatlas/unternehmen-suchen/)" + ], + "url": "http://it.region-stuttgart.de/competenzatlas/unternehmen-suchen/", + "description": "Region Stuttgart web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "Bytespider", + "addition_date": "2019/11/11", + "instances": [ + "Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.3754.1902 Mobile Safari/537.36; Bytespider", + "Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.4454.1745 Mobile Safari/537.36; Bytespider", + "Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.7597.1164 Mobile Safari/537.36; Bytespider;bytespider@bytedance.com", + "Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2988.1545 Mobile Safari/537.36; Bytespider", + "Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.4141.1682 Mobile Safari/537.36; Bytespider", + "Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.3478.1649 Mobile Safari/537.36; Bytespider", + "Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.5267.1259 Mobile Safari/537.36; Bytespider", + "Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.7990.1979 Mobile Safari/537.36; Bytespider", + "Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.2268.1523 Mobile Safari/537.36; Bytespider", + "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2576.1836 Mobile Safari/537.36; Bytespider", + "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.9681.1227 Mobile Safari/537.36; Bytespider", + "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.6023.1635 Mobile Safari/537.36; Bytespider", + "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.4944.1981 Mobile Safari/537.36; Bytespider", + "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.3613.1739 Mobile Safari/537.36; Bytespider", + "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.4022.1033 Mobile Safari/537.36; Bytespider", + "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.3248.1547 Mobile Safari/537.36; Bytespider", + "Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.5527.1507 Mobile Safari/537.36; Bytespider", + "Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.5216.1326 Mobile Safari/537.36; Bytespider", + "Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.9038.1080 Mobile Safari/537.36; Bytespider" + ], + "url": "https://stackoverflow.com/questions/57908900/what-is-the-bytespider-user-agent", + "description": "ByteDance Bytespider web crawler bot", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "Datanyze", + "addition_date": "2019/11/17", + "instances": [ + "Mozilla/5.0 (X11; Datanyze; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/65.0.3325.181 Safari/537.36" + ], + "url": "https://www.datanyze.com/dnyzbot/", + "description": "Datanyze web crawler for technology detection", + "tags": [ + "seo" + ] + }, + { + "pattern": "Google-Site-Verification", + "addition_date": "2019/12/11", + "instances": [ + "Mozilla/5.0 (compatible; Google-Site-Verification/1.0)" + ], + "url": "https://support.google.com/webmasters/answer/9008080", + "description": "Google site verification web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "TrendsmapResolver", + "addition_date": "2020/02/24", + "instances": [ + "Mozilla/5.0 (compatible; TrendsmapResolver/0.1)" + ], + "url": "https://www.trendsmap.com/", + "description": "Trendsmap web crawler for trend analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "tweetedtimes", + "addition_date": "2020/02/24", + "instances": [ + "Mozilla/5.0 (compatible; +http://tweetedtimes.com)" + ], + "url": "https://tweetedtimes.com/", + "description": "Tweeted Times web crawler for news", + "tags": [ + "seo" + ] + }, + { + "pattern": "NTENTbot", + "addition_date": "2020/02/24", + "instances": [ + "Mozilla/5.0 (compatible; NTENTbot; +http://www.ntent.com/ntentbot)" + ], + "url": "https://ntent.com/ntentbot/", + "description": "NTENT web crawler for search results", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Gwene", + "addition_date": "2020/02/24", + "instances": [ + "Gwene/1.0 (The gwene.org rss-to-news gateway) Googlebot" + ], + "url": "https://gwene.org", + "description": "Gwene RSS to news gateway web crawler", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "SimplePie", + "addition_date": "2020/02/24", + "instances": [ + "SimplePie/1.3-dev (Feed Parser; http://simplepie.org; Allow like Gecko)" + ], + "url": "http://simplepie.org", + "description": "SimplePie PHP feed parser web crawler", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "SearchAtlas", + "addition_date": "2020/03/02", + "instances": [ + "SearchAtlas.com SEO Crawler" + ], + "url": "http://SearchAtlas.com", + "description": "SearchAtlas SEO web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Superfeedr", + "addition_date": "2020/03/02", + "instances": [ + "Superfeedr bot/2.0 http://superfeedr.com - Make your feeds realtime: get in touch - feed-id:1162088860" + ], + "url": "http://superfeedr.com", + "description": "Superfeedr feed reader web crawler bot", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "feedbot", + "addition_date": "2020/03/02", + "instances": [ + "wp.com feedbot/1.0 (+https://wp.com)" + ], + "url": "http://wp.com", + "description": "WordPress.com feed web crawler bot", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "UT-Dorkbot", + "addition_date": "2020/03/02", + "instances": [ + "UT-Dorkbot/1.0" + ], + "url": "https://security.utexas.edu/dorkbot", + "description": "University of Texas security web crawler", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Amazonbot", + "addition_date": "2020/03/02", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)" + ], + "url": "https://developer.amazon.com/support/amazonbot", + "description": "Amazon web crawler for product discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "AmazonProductDiscovery", + "addition_date": "2025/12/22", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 (compatible; AmazonProductDiscovery/1.0; https://vendorcentral.amazon.com/support/amazonproductbot)", + "Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0 (compatible; AmazonProductDiscovery/1.0; https://vendorcentral.amazon.com/support/amazonproductbot)" + ], + "url": "https://vendorcentral.amazon.com/support/amazonproductbot", + "description": "Amazon product discovery web crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "AmazonSellerInitiatedListing", + "addition_date": "2025/12/22", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 (compatible; AmazonSellerInitiatedListing/1.0; https://vendorcentral.amazon.com/support/amazonproductbot)" + ], + "url": "https://vendorcentral.amazon.com/support/amazonproductbot", + "description": "Amazon seller listing web crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "SerendeputyBot", + "addition_date": "2020/03/02", + "instances": [ + "SerendeputyBot/0.8.6 (http://serendeputy.com/about/serendeputy-bot)" + ], + "url": "http://serendeputy.com/about/serendeputy-bot", + "description": "Serendeputy web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "Eyeotabot", + "addition_date": "2020/03/02", + "instances": [ + "Mozilla/5.0 (compatible; Eyeotabot/1.0; +http://www.eyeota.com)" + ], + "url": "http://www.eyeota.com", + "description": "Eyeota web crawler for audience data", + "tags": [ + "advertising" + ] + }, + { + "pattern": "officestorebot", + "addition_date": "2020/03/02", + "instances": [ + "Mozilla/5.0 (compatible; officestorebot/1.0; +https://aka.ms/officestorebot)" + ], + "url": "https://aka.ms/officestorebot", + "description": "Microsoft Office Store web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Neticle Crawler", + "addition_date": "2020/03/02", + "instances": [ + "Neticle Crawler v1.0 ( https://neticle.com/bot/en/ )" + ], + "url": "https://neticle.com/bot/en/", + "description": "Neticle web crawler for content analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "SurdotlyBot", + "addition_date": "2020/03/02", + "instances": [ + "Mozilla/5.0 (compatible; SurdotlyBot/1.0; +http://sur.ly/bot.html; Linux; Android 4; iPhone; CPU iPhone OS 6_0_1 like Mac OS X)" + ], + "url": "http://sur.ly/bot.html", + "description": "Surly web crawler for link shortening", + "tags": [ + "seo" + ] + }, + { + "pattern": "LinkisBot", + "addition_date": "2020/03/02", + "instances": [ + "Mozilla/5.0 (compatible; LinkisBot/1.0; bot@linkis.com) (iPhone; CPU iPhone OS 8_4_1 like Mac OS X) Mobile/12H321" + ], + "description": "Linkis web crawler for link sharing", + "tags": [ + "seo" + ] + }, + { + "pattern": "AwarioSmartBot", + "addition_date": "2020/03/02", + "instances": [ + "AwarioSmartBot/1.0 (+https://awario.com/bots.html; bots@awario.com)" + ], + "url": "https://awario.com/bots.html", + "description": "Awario smart web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "AwarioRssBot", + "addition_date": "2020/03/02", + "instances": [ + "AwarioRssBot/1.0 (+https://awario.com/bots.html; bots@awario.com)" + ], + "url": "https://awario.com/bots.html", + "description": "Awario RSS feed web crawler bot", + "tags": [ + "feed-reader", + "seo" + ] + }, + { + "pattern": "RyteBot", + "addition_date": "2020/03/02", + "instances": [ + "RyteBot/1.0.0 (+https://bot.ryte.com/)" + ], + "url": "https://bot.ryte.com/", + "description": "Ryte web crawler for site analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "FreeWebMonitoring SiteChecker", + "addition_date": "2020/03/02", + "instances": [ + "FreeWebMonitoring SiteChecker/0.2 (+https://www.freewebmonitoring.com/bot.html)" + ], + "url": "https://www.freewebmonitoring.com/bot.html", + "description": "FreeWebMonitoring site checker web crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "AspiegelBot", + "addition_date": "2020/03/16", + "instances": [ + "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; AspiegelBot)" + ], + "url": "https://aspiegel.com", + "description": "Aspiegel web crawler for content discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "NAVER Blog Rssbot", + "addition_date": "2020/03/16", + "instances": [ + "NAVER Blog Rssbot" + ], + "url": "http://www.naver.com", + "description": "Naver blog RSS web crawler bot", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "zenback bot", + "addition_date": "2020/03/16", + "instances": [ + "Mozilla/5.0 (compatible; zenback bot; powered by logly +http://corp.logly.co.jp/)" + ], + "url": "http://corp.logly.co.jp/", + "description": "Zenback web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "SentiBot", + "addition_date": "2020/03/16", + "instances": [ + "SentiBot www.sentibot.eu (compatible with Googlebot)" + ], + "url": "https://sites.google.com/senti1.com/sentibot-eu/home", + "description": "SentiBot web crawler for sentiment analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Domains Project\\/", + "addition_date": "2020/03/16", + "instances": [ + "Mozilla/5.0 (compatible; Domains Project/1.0.3; +https://github.com/tb0hdan/domains)" + ], + "url": "https://github.com/tb0hdan/domains", + "description": "Domains Project web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "Pandalytics", + "addition_date": "2020/03/16", + "instances": [ + "Pandalytics/1.0 (https://domainsbot.com/pandalytics/)" + ], + "url": "https://domainsbot.com/pandalytics/", + "description": "Pandalytics web crawler for domain analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "VKRobot", + "addition_date": "2020/03/16", + "instances": [ + "Mozilla/5.0 (compatible; VKRobot/1.0)" + ], + "description": "VK social network web crawler bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "bidswitchbot", + "addition_date": "2020/03/16", + "instances": [ + "bidswitchbot/1.0" + ], + "url": "https://www.bidswitch.com/about-us/", + "description": "BidSwitch web crawler for advertising", + "tags": [ + "advertising" + ] + }, + { + "pattern": "tigerbot", + "addition_date": "2020/03/16", + "instances": [ + "tigerbot" + ], + "description": "Tiger web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "NIXStatsbot", + "addition_date": "2020/03/16", + "instances": [ + "Mozilla/5.0 (compatible; NIXStatsbot/1.1; +http://www.nixstats.com/bot.html)" + ], + "url": "http://www.nixstats.com/bot.html", + "description": "NIXStats web crawler for monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Atom Feed Robot", + "addition_date": "2020/03/16", + "instances": [ + "RSSMicro.com RSS/Atom Feed Robot" + ], + "url": "https://rssmicro.com", + "description": "RSSMicro atom feed web crawler bot", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "[Cc]urebot", + "addition_date": "2020/03/16", + "instances": [ + "Curebot/1.0", + "curebot-feed-fetcher" + ], + "description": "Cure web crawler for content discovery", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "PagePeeker\\/", + "addition_date": "2020/03/16", + "instances": [ + "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36 (compatible; PagePeeker/3.0; +https://pagepeeker.com/robots/)" + ], + "url": "https://pagepeeker.com/robots/", + "description": "PagePeeker web crawler for screenshots", + "tags": [ + "seo" + ] + }, + { + "pattern": "Vigil\\/", + "addition_date": "2020/03/16", + "instances": [ + "Mozilla/5.0 (compatible; Vigil/1.0; +http://vigil-app.com/bot.html)" + ], + "url": "http://vigil-app.com/bot.html", + "description": "Vigil web crawler for monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "rssbot\\/", + "addition_date": "2020/03/16", + "instances": [ + "rssbot/1.4.3 (+https://t.me/RustRssBot)" + ], + "url": "https://github.com/iovxw/rssbot", + "description": "RSS bot web crawler for feeds", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "startmebot\\/", + "addition_date": "2020/03/16", + "instances": [ + "Mozilla/5.0 (compatible; startmebot/1.0; +https://start.me/bot)" + ], + "url": "https://start.me/bot", + "description": "Start.me web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "JobboerseBot", + "addition_date": "2020/03/16", + "instances": [ + "Mozilla/5.0 (X11; U; Linux Core i7-4980HQ; de; rv:32.0; compatible; JobboerseBot; http://www.jobboerse.com/bot.htm) Gecko/20100101 Firefox/38.0" + ], + "url": "http://www.jobboerse.com/bot.htm", + "description": "Jobboerse web crawler for job discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "seewithkids", + "addition_date": "2020/03/16", + "instances": [ + "http://seewithkids.com/bot" + ], + "url": "http://seewithkids.com/bot", + "description": "SeeWithKids web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "NINJA bot", + "addition_date": "2020/03/16", + "instances": [ + "NINJA bot" + ], + "description": "NINJA web crawler for content discovery", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Cutbot", + "addition_date": "2020/03/16", + "instances": [ + "Cutbot; 1.5; http://cutbot.net/" + ], + "url": "http://cutbot.net/", + "description": "Cutbot web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "BublupBot", + "addition_date": "2020/03/16", + "instances": [ + "BublupBot (+https://www.bublup.com/bublup-bot.html)" + ], + "url": "https://www.bublup.com/bublup-bot.html", + "description": "Bublup web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "BrandONbot", + "addition_date": "2020/03/16", + "instances": [ + "BrandONbot (http://brandonmedia.net)" + ], + "url": "http://brandonmedia.net", + "description": "BrandON web crawler for brand monitoring", + "tags": [ + "seo" + ] + }, + { + "pattern": "RidderBot", + "addition_date": "2020/03/16", + "instances": [ + "Mozilla/5.0 (compatible; RidderBot/1.0; bot@ridder.co)", + "Mozilla/5.0 (compatible; RidderBot/1.0; bot@ridder.co) (iPhone; CPU iPhone OS 8_4_1 like Mac OS X) Mobile/12H321" + ], + "url": "https://ridder.co/", + "description": "Ridder web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "Taboolabot", + "addition_date": "2020/03/16", + "instances": [ + "Mozilla/5.0 (compatible; Taboolabot/3.7; +http://www.taboola.com)" + ], + "url": "http://www.taboola.com", + "description": "Taboola web crawler for content discovery", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Dubbotbot", + "addition_date": "2020/03/16", + "instances": [ + "Mozilla/5.0 (compatible; Dubbotbot/0.2; +http://dubbot.com)" + ], + "url": "http://dubbot.com", + "description": "Dubbot web crawler for content discovery", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "FindITAnswersbot", + "addition_date": "2020/03/16", + "instances": [ + "Mozilla/5.0 (compatible;FindITAnswersbot/1.0;+http://search.it-influentials.com/bot.htm)" + ], + "url": "http://search.it-influentials.com/bot.htm", + "description": "FindITAnswers web crawler for discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "infoobot", + "addition_date": "2020/03/16", + "instances": [ + "infoobot/0.1 (https://www.infoo.nl/bot.html)" + ], + "url": "https://www.infoo.nl/bot.html", + "description": "Infoo web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "Refindbot", + "addition_date": "2020/03/16", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36 (Refindbot/1.0)" + ], + "url": "https://refind.com/about", + "description": "Refind web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "BlogTraffic\\/\\d\\.\\d+ Feed-Fetcher", + "addition_date": "2020/03/16", + "instances": [ + "Mozilla/5.0 (compatible; BlogTraffic/1.4 Feed-Fetcher; +http://www.blogtraffic.de/rss-bot.html)" + ], + "url": "http://www.blogtraffic.de/rss-bot.html", + "description": "BlogTraffic feed fetcher web crawler", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "SeobilityBot", + "addition_date": "2020/03/16", + "instances": [ + "SeobilityBot (SEO Tool; https://www.seobility.net/sites/bot.html)" + ], + "url": "https://www.seobility.net/sites/bot.html", + "description": "Seobility web crawler for SEO analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Cincraw", + "addition_date": "2020/03/16", + "instances": [ + "Mozilla/5.0 (compatible; Cincraw/1.0; +http://cincrawdata.net/bot/)" + ], + "url": "http://cincrawdata.net/bot/", + "description": "Cincraw web crawler for data collection", + "tags": [ + "seo" + ] + }, + { + "pattern": "Dragonbot", + "addition_date": "2020/03/16", + "instances": [ + "Mozilla/5.0 (Windows NT 6.1; rv:34.0) Gecko/20100101 Firefox/34.0; Dragonbot; http://www.dragonmetrics.com" + ], + "url": "http://www.dragonmetrics.com", + "description": "DragonMetrics web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "VoluumDSP-content-bot", + "addition_date": "2020/03/16", + "instances": [ + "Mozilla/5.0 (compatible; VoluumDSP-content-bot/2.0; +dsp-dev@codewise.com)" + ], + "url": "https://codewise.com", + "description": "Voluum DSP web crawler for advertising", + "tags": [ + "advertising" + ] + }, + { + "pattern": "FreshRSS", + "addition_date": "2020/03/16", + "instances": [ + "FreshRSS/1.11.2 (Linux; https://freshrss.org) like Googlebot" + ], + "url": "https://freshrss.org", + "description": "FreshRSS feed reader web crawler bot", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "BitBot", + "addition_date": "2020/03/16", + "instances": [ + "Mozilla/5.0 (compatible; BitBot/v1.19.0; +https://bitbot.dev)" + ], + "url": "https://bitbot.dev", + "description": "BitBot web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "^PHP-Curl-Class", + "addition_date": "2020/12/10", + "instances": [ + "PHP-Curl-Class/4.13.0 (+https://github.com/php-curl-class/php-curl-class) PHP/7.2.24 curl/7.61.1", + "PHP-Curl-Class/4.13.0 (+https://github.com/php-curl-class/php-curl-class) PHP/7.3.19 curl/7.66.0", + "PHP-Curl-Class/4.13.0 (+https://github.com/php-curl-class/php-curl-class) PHP/7.3.23 curl/7.66.0", + "PHP-Curl-Class/4.13.0 (+https://github.com/php-curl-class/php-curl-class) PHP/7.4.7 curl/7.69.1", + "PHP-Curl-Class/4.13.0 (+https://github.com/php-curl-class/php-curl-class) PHP/7.4.9 curl/7.69.1", + "PHP-Curl-Class/4.13.0 (+https://github.com/php-curl-class/php-curl-class) PHP/7.4.10 curl/7.69.1", + "PHP-Curl-Class/4.13.0 (+https://github.com/php-curl-class/php-curl-class) PHP/7.4.11 curl/7.69.1" + ], + "url": "https://github.com/php-curl-class/php-curl-class", + "description": "PHP Curl Class HTTP client library", + "tags": [ + "http-library" + ] + }, + { + "pattern": "Google-Certificates-Bridge", + "addition_date": "2020/12/23", + "instances": [ + "Google-Certificates-Bridge" + ], + "description": "Google certificates bridge web crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "centurybot", + "addition_date": "2022/04/26", + "instances": [ + "Mozilla/5.0 (compatible; Go-http-client/1.1; +centurybot9@gmail.com)" + ], + "description": "Century web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "Viber", + "addition_date": "2021/04/27", + "instances": [ + "Viber" + ], + "url": "https://www.viber.com/", + "description": "Viber messaging web crawler bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "e\\.ventures Investment Crawler", + "addition_date": "2021/06/05", + "url": "https://www.eventures.vc/", + "instances": [ + "e.ventures Investment Crawler (eventures.vc)" + ], + "description": "E.ventures investment web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "evc-batch", + "addition_date": "2021/06/07", + "url": "https://www.eventures.vc/", + "instances": [ + "Mozilla/5.0 (compatible; evc-batch/2.0)" + ], + "description": "E.ventures batch web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "PetalBot", + "addition_date": "2021/06/07", + "instances": [ + "Mozilla/5.0 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot)", + "Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)" + ], + "url": "https://webmaster.petalsearch.com/site/petalbot", + "description": "Petal search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "virustotal", + "addition_date": "2021/09/22", + "instances": [ + "Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US) AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)", + "AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)" + ], + "url": "https://www.virustotal.com/gui/home/url", + "description": "VirusTotal web crawler for security", + "tags": [ + "scanner" + ] + }, + { + "pattern": "(^| )PTST\\/", + "addition_date": "2021/12/05", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36 PTST/211202.211915", + "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0 PTST/211202.211915" + ], + "url": "https://www.webpagetest.org", + "description": "WebPageTest web crawler for testing", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "minicrawler", + "addition_date": "2022/01/12", + "instances": [ + "Testomatobot/1.0 (Linux x86_64; +https://www.testomato.com/testomatobot) minicrawler/5.2.2" + ], + "url": "https://www.testomato.com/bot", + "description": "Testomato web crawler for testing", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Cookiebot", + "addition_date": "2022/01/23", + "url": "https://www.cookiebot.com/", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Cookiebot/1.0; +http://cookiebot.com/) Chrome/97.0.4692.71 Safari/537.36" + ], + "description": "Cookiebot web crawler for cookie scanning", + "tags": [ + "scanner" + ] + }, + { + "pattern": "trovitBot", + "addition_date": "2022/06/08", + "url": "http://www.trovit.com/bot.html", + "instances": [ + "Mozilla/5.0 (compatible; trovitBot 1.0; +http://www.trovit.com/bot.html)" + ], + "description": "Trovit web crawler for content discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "seostar\\.co", + "addition_date": "2022/08/04", + "url": "https://seostar.co/robot/", + "instances": [ + "Mozilla/5.0 (compatible; Adsbot/3.1; +https://seostar.co/robot/)" + ], + "description": "SEOstar web crawler for SEO analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "IonCrawl", + "addition_date": "2022/08/04", + "url": "https://www.ionos.de/terms-gtc/faq-crawler-en", + "instances": [ + "IonCrawl (https://www.ionos.de/terms-gtc/faq-crawler-en/)" + ], + "description": "IONOS IonCrawl web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "Uptime-Kuma", + "addition_date": "2022/10/17", + "url": "https://uptime.kuma.pet/", + "instances": [ + "Uptime-Kuma/1.23.16", + "Uptime-Kuma/1.23.15", + "Uptime-Kuma/1.23.14", + "Uptime-Kuma/1.23.13", + "Uptime-Kuma/1.23.12", + "Uptime-Kuma/1.23.11", + "Uptime-Kuma/1.23.10", + "Uptime-Kuma/1.18.0" + ], + "description": "Uptime Kuma web crawler for monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Seekport", + "addition_date": "2022/10/17", + "url": "https://bot.seekport.com", + "instances": [ + "Mozilla/5.0 (compatible; SeekportBot; +https://bot.seekport.com)", + "Mozilla/5.0 (compatible; Seekport Crawler; http://seekport.com/)" + ], + "description": "Seekport web crawler for content discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "FreshpingBot", + "addition_date": "2022/10/17", + "url": "https://www.freshworks.com/website-monitoring/", + "instances": [ + "FreshpingBot/1.0 (+https://freshping.io/)" + ], + "description": "Freshping web crawler for monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Feedbin", + "addition_date": "2022/11/05", + "url": "https://feedbin.com/", + "instances": [ + "Feedbin feed-id:2005098 - 2 subscribers" + ], + "description": "Feedbin feed reader web crawler bot", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "CriteoBot", + "addition_date": "2022/11/13", + "url": "https://www.criteo.com/", + "instances": [ + "CriteoBot/0.1 (+https://www.criteo.com/criteo-crawler/)" + ], + "description": "Criteo web crawler for advertising", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Snap URL Preview Service", + "addition_date": "2022/11/13", + "url": "https://snap.com/", + "instances": [ + "Snap URL Preview Service; bot; snapchat; https://developers.snap.com/robots" + ], + "description": "Snapchat URL preview web crawler", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Better Uptime Bot", + "addition_date": "2022/11/13", + "url": "https://betteruptime.com/", + "instances": [ + "Better Uptime Bot Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36" + ], + "description": "Better Uptime web crawler for monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "RuxitSynthetic", + "addition_date": "2023/02/16", + "url": "https://www.dynatrace.com/support/help/platform-modules/digital-experience/synthetic-monitoring/browser-monitors/configure-browser-monitors#expand--default-user-agent", + "instances": [ + "RuxitSynthetic/1.0" + ], + "description": "Dynatrace Ruxit synthetic web crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Google-Read-Aloud", + "addition_date": "2023/02/16", + "url": "https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.118 Safari/537.36 (compatible; Google-Read-Aloud; +https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers)", + "Mozilla/5.0 (Linux; Android 7.0; SM-G930V Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36 (compatible; Google-Read-Aloud; +https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers)" + ], + "description": "Google read aloud web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Valve\\/Steam", + "addition_date": "2023/05/24", + "instances": [ + "Valve/Steam HTTP Client 1.0 (SteamChatURLLookup)" + ], + "description": "Steam web crawler for link previews", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "OdklBot\\/", + "addition_date": "2023/05/24", + "instances": [ + "OdklBot/1.0 (share@odnoklassniki.ru)", + "Mozilla/5.0 (compatible; OdklBot/1.0 like Linux; klass@odnoklassniki.ru)" + ], + "url": "https://odnoklassniki.ru/", + "description": "Odnoklassniki web crawler for sharing", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "GPTBot", + "addition_date": "2023/08/09", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)" + ], + "url": "https://platform.openai.com/docs/gptbot", + "description": "OpenAI GPT web crawler bot", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "ChatGPT-User", + "addition_date": "2024/04/19", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot" + ], + "url": "https://openai.com/bot", + "description": "ChatGPT user web crawler bot", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "OAI-SearchBot", + "addition_date": "2024/09/24", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" + ], + "url": "https://platform.openai.com/docs/bots", + "description": "OpenAI search web crawler bot", + "tags": [ + "ai-crawler", + "search-engine" + ] + }, + { + "pattern": "YandexRenderResourcesBot\\/", + "addition_date": "2023/08/16", + "instances": [ + "Mozilla/5.0 (compatible; YandexRenderResourcesBot/1.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0" + ], + "url": "http://yandex.com/bots", + "description": "Yandex render resources web crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "LightspeedSystemsCrawler", + "addition_date": "2023/08/16", + "instances": [ + "LightspeedSystemsCrawler Mozilla/5.0 (Windows; U; MSIE 9.0; Windows NT 9.0; en-US" + ], + "description": "Lightspeed Systems web crawler bot", + "tags": [ + "scanner" + ] + }, + { + "pattern": "ev-crawler\\/", + "addition_date": "2023/08/16", + "instances": [ + "Mozilla/5.0 (compatible; ev-crawler/1.0; +https://headline.com/legal/crawler)" + ], + "url": "https://headline.com/legal/crawler", + "description": "Headline web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "BitSightBot\\/", + "addition_date": "2023/08/16", + "instances": [ + "Mozilla/5.0 (compatible; BitSightBot/1.0)" + ], + "url": "https://www.bitsight.com", + "description": "BitSight web crawler for security", + "tags": [ + "scanner" + ] + }, + { + "pattern": "woorankreview\\/", + "addition_date": "2023/08/16", + "instances": [ + "Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit/604.1.38 (KHTML, like Gecko) Version/11.0 Mobile/15A372 Safari/604.1 (compatible; woorankreview/2.0; +https://www.woorank.com/)", + "Mozilla/5.0 (compatible; woorankreview/2.0; +https://www.woorank.com/)" + ], + "url": "https://www.woorank.com/", + "description": "WooRank web crawler for SEO analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Google-Safety", + "addition_date": "2023/08/17", + "instances": [ + "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.5735.179 Mobile Safari/537.36 (compatible; Google-Safety; +http://www.google.com/bot.html)", + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.5735.179 Safari/537.36 (compatible; Google-Safety; +http://www.google.com/bot.html)", + "Google-Safety" + ], + "url": "https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers", + "description": "Google safety web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "AwarioBot", + "addition_date": "2023/08/23", + "instances": [ + "Mozilla/5.0 (compatible; AwarioBot/1.0; +https://awario.com/bots.html)" + ], + "url": "https://awario.com/bots.html", + "description": "Awario web crawler for monitoring", + "tags": [ + "seo" + ] + }, + { + "pattern": "DataForSeoBot", + "addition_date": "2023/08/23", + "instances": [ + "Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot)" + ], + "url": "https://dataforseo.com/dataforseo-bot", + "description": "DataForSEO web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Linespider", + "addition_date": "2023/08/24", + "instances": [ + "Mozilla/5.0 (compatible; Linespider/1.1; +https://lin.ee/4dwXkTH)", + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Linespider/1.1; +https://lin.ee/4dwXkTH) Chrome/W.X.Y.Z Safari/537.36" + ], + "url": "https://help2.line.me/linesearchbot/web/?contentId=50006055&lang=en", + "description": "LINE Linespider web crawler bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "WellKnownBot", + "addition_date": "2023/08/29", + "instances": [ + "Mozilla/5.0 (compatible; WellKnownBot/0.1; +https://well-known.dev/about/#bot)" + ], + "url": "https://well-known.dev/about/#bot)", + "description": "WellKnown web crawler for discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "A Patent Crawler", + "addition_date": "2023/08/29", + "instances": [ + "E. Orliac, G. Fourestey/2.3 (A Patent Crawler; http://scitas.epfl.ch/; etienne.orliac@epfl.ch, gilles.fourestey@epfl.ch)" + ], + "url": "http://scitas.epfl.ch/", + "description": "EPFL patent web crawler bot", + "tags": [ + "academic" + ] + }, + { + "pattern": "StractBot", + "addition_date": "2023/09/06", + "instances": [ + "Mozilla/5.0 (compatible; StractBot/0.1; open source search engine; +https://trystract.com/webmasters)" + ], + "url": "https://trystract.com/webmasters", + "description": "Stract search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "search\\.marginalia\\.nu", + "addition_date": "2023/09/08", + "instances": [ + "search.marginalia.nu" + ], + "url": "https://search.marginalia.nu", + "description": "Marginalia search web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "YouBot", + "addition_date": "2023/09/08", + "instances": [ + "YouBot (+http://www.you.com)" + ], + "url": "https://you.com/", + "description": "You.com search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Nicecrawler", + "addition_date": "2023/09/08", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Nicecrawler/1.1; +http://www.nicecrawler.com/) Chrome/90.0.4430.97 Safari/537.36" + ], + "url": "http://www.nicecrawler.com/", + "description": "Nicecrawler web crawler for discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Neevabot", + "addition_date": "2023/09/08", + "instances": [ + "Mozilla/5.0 (compatible; Neevabot/1.0; +https://neeva.com/neevabot)" + ], + "url": "https://neeva.com/neevabot", + "description": "Neeva search engine web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "BrightEdge Crawler", + "addition_date": "2023/09/08", + "instances": [ + "BrightEdge Crawler/1.0 (crawler@brightedge.com)" + ], + "url": "https://www.brightedge.com/", + "description": "BrightEdge web crawler for SEO", + "tags": [ + "seo" + ] + }, + { + "pattern": "SiteCheckerBotCrawler", + "addition_date": "2023/09/08", + "instances": [ + "SiteCheckerBotCrawler/1.0 (+http://sitechecker.pro)" + ], + "url": "http://sitechecker.pro", + "description": "SiteChecker web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "TombaPublicWebCrawler", + "addition_date": "2023/09/08", + "instances": [ + "Mozilla/5.0 (compatible; TombaPublicWebCrawler/1.0; +https://tombascraper.com)" + ], + "url": "https://tombascraper.com", + "description": "Tomba web crawler for email discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "CrawlyProjectCrawler", + "addition_date": "2023/09/08", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/115.0.0.0 Safari/537.36 (compatible; CrawlyProjectCrawler/0.1.3; crawlyproject@digitaldragon.dev +https://crawlyproject.digitaldragon.dev/)" + ], + "url": "https://crawlyproject.digitaldragon.dev/", + "description": "Crawly Project web crawler framework", + "tags": [ + "seo" + ] + }, + { + "pattern": "KomodiaBot", + "addition_date": "2023/09/08", + "instances": [ + "Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://www.komodia.com/newwiki/index.php/URL_server_crawler) KomodiaBot/1.0" + ], + "url": "http://www.komodia.com/newwiki/index.php/URL_server_crawler", + "description": "Komodia web crawler for classification", + "tags": [ + "seo" + ] + }, + { + "pattern": "KStandBot", + "addition_date": "2023/09/08", + "instances": [ + "Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://url-classification.io/wiki/index.php?title=URL_server_crawler) KStandBot/1.0" + ], + "url": "http://url-classification.io", + "description": "KStand web crawler for classification", + "tags": [ + "seo" + ] + }, + { + "pattern": "CISPA Webcrawler", + "addition_date": "2023/09/08", + "instances": [ + "CISPA Webcrawler (https://vuln-notify-checker.cispa.saarland)" + ], + "url": "https://vuln-notify-checker.cispa.saarland", + "description": "CISPA web crawler for vulnerability", + "tags": [ + "academic" + ] + }, + { + "pattern": "MTRobot", + "addition_date": "2023/09/08", + "instances": [ + "MTRobot/0.2 (Metrics Tools Analytics Crawler; https://metrics-tools.de/robot.html; crawler@metrics-tools.de)" + ], + "url": "https://metrics-tools.de/robot.html", + "description": "Metrics Tools web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "hyscore\\.io", + "addition_date": "2023/09/08", + "instances": [ + "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1. 4 (compatible; HyScore/1.0; +https://hyscore.io/crawler/)" + ], + "url": "https://hyscore.io/crawler/", + "description": "HyScore web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "AlexandriaOrgBot", + "addition_date": "2023/09/08", + "instances": [ + "Mozilla/5.0 (Linux) (compatible; AlexandriaOrgBot/1.0; +https://www.alexandria.org/bot.html)" + ], + "url": "https://www.alexandria.org/bot.html", + "description": "Alexandria web crawler for discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "2ip bot", + "addition_date": "2023/09/08", + "instances": [ + "2ip bot/1.1 (+http://2ip.io)" + ], + "url": "http://2ip.io", + "description": "2IP web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Yellowbrandprotectionbot", + "addition_date": "2023/09/08", + "instances": [ + "Mozilla/5.0 (compatible; Yellowbrandprotectionbot/1.0; +https://www.yellowbp.com/bot.html)" + ], + "url": "https://www.yellowbp.com/bot.html", + "description": "Yellow brand protection web crawler", + "tags": [ + "seo" + ] + }, + { + "pattern": "SEOlizer", + "addition_date": "2023/09/08", + "instances": [ + "SEOlizer/1.1 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13 (+https://www.seolizer.de/bot.html)" + ], + "url": "https://www.seolizer.de/bot.html", + "description": "SEOlizer web crawler for SEO analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "vuhuvBot", + "addition_date": "2023/09/08", + "instances": [ + "Mozilla/5.0 (compatible; vuhuvBot/1.0; +http://vuhuv.com/bot.html)" + ], + "url": "http://vuhuv.com/bot.html", + "description": "Vuhuv web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "INETDEX-BOT", + "addition_date": "2023/09/08", + "instances": [ + "INETDEX-BOT/1.5 (Mozilla/5.0; https://inetdex.com/bot.html)" + ], + "url": "https://inetdex.com/bot.html", + "description": "INETDEX web crawler for indexing", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Synapse", + "addition_date": "2023/09/08", + "instances": [ + "Synapse (bot; +https://github.com/matrix-org/synapse)" + ], + "url": "https://github.com/matrix-org/synapse", + "description": "Matrix Synapse web crawler bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "t3versionsBot", + "addition_date": "2023/09/08", + "instances": [ + "Mozilla/5.0 (compatible; t3versionsBot/1.0; +https://www.t3versions.com/bot)" + ], + "url": "https://www.t3versions.com/bot", + "description": "T3versions web crawler for discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "deepnoc", + "addition_date": "2023/09/08", + "instances": [ + "deepnoc - https://deepnoc.com/bot" + ], + "url": "https://deepnoc.com/bot", + "description": "DeepNOC web crawler for monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Cocolyzebot", + "addition_date": "2023/09/08", + "instances": [ + "Mozilla/5.0 (compatible; Cocolyzebot/1.0; https://cocolyze.com/bot)" + ], + "url": "https://cocolyze.com/bot", + "description": "Cocolyze web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "hypestat", + "addition_date": "2023/09/08", + "instances": [ + "Mozilla/5.0 (compatible; hypestat/1.0; +https://hypestat.com/bot)" + ], + "url": "https://hypestat.com/bot", + "description": "Hypestat web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "ReverseEngineeringBot", + "addition_date": "2023/09/08", + "instances": [ + "Mozilla/5.0 (compatible; ReverseEngineeringBot/0.1; +https://torus.company/bot.html)" + ], + "url": "https://torus.company/bot.html", + "description": "Torus reverse engineering web crawler", + "tags": [ + "scanner" + ] + }, + { + "pattern": "sempi\\.tech", + "addition_date": "2023/09/08", + "instances": [ + "Mozilla/5.0 (compatible; Semanticbot/1.0; +http://sempi.tech/bot.html)" + ], + "url": "http://sempi.tech/bot.html", + "description": "Sempi web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "Iframely", + "addition_date": "2023/09/08", + "instances": [ + "Iframely/1.3.1 (+https://iframely.com/docs/about) Atlassian" + ], + "url": "https://iframely.com/docs/about", + "description": "Iframely web crawler for embeds", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "MetaInspector", + "addition_date": "2023/09/08", + "instances": [ + "MetaInspector/5.6.0 (+https://github.com/jaimeiniesta/metainspector)" + ], + "url": "https://github.com/jaimeiniesta/metainspector", + "description": "MetaInspector web crawler for metadata", + "tags": [ + "seo" + ] + }, + { + "pattern": "node-fetch", + "addition_date": "2023/09/08", + "instances": [ + "node-fetch/1.0 (+https://github.com/bitinn/node-fetch)" + ], + "url": "https://github.com/bitinn/node-fetch", + "description": "Node-fetch HTTP client library", + "tags": [ + "http-library" + ] + }, + { + "pattern": "l9explore", + "addition_date": "2023/09/08", + "instances": [ + "l9explore/1.2.2", + "lkxscan/v0.1.0 (+https://leakix.net) l9explore/v1.0.0 (+https://github.com/LeakIX/l9explore)" + ], + "url": "https://github.com/LeakIX/l9explore", + "description": "L9explore web crawler for discovery", + "tags": [ + "scanner" + ] + }, + { + "pattern": "python-opengraph", + "addition_date": "2023/09/08", + "instances": [ + "python-opengraph-jaywink/0.2.0 (+https://github.com/jaywink/python-opengraph)" + ], + "url": "https://github.com/jaywink/python-opengraph", + "description": "Python OpenGraph web crawler bot", + "tags": [ + "http-library" + ] + }, + { + "pattern": "OpenGraphCheck", + "addition_date": "2023/09/08", + "instances": [ + "OpenGraphCheck/2.1 (+https://opengraphcheck.com)" + ], + "url": "https://opengraphcheck.com", + "description": "OpenGraphCheck web crawler for metadata", + "tags": [ + "seo" + ] + }, + { + "pattern": "developers\\.google\\.com\\/\\+\\/web\\/snippet", + "addition_date": "2023/09/08", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 Google-PageRenderer Google (+https://developers.google.com/+/web/snippet/)", + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36 Google (+https://developers.google.com/+/web/snippet/" + ], + "url": "https://developers.google.com/+/web/snippet", + "description": "Google snippet web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "SenutoBot", + "addition_date": "2023/09/08", + "instances": [ + "SenutoBot/1.0 (compatible; SenutoBot/1.0; +https://www.senuto.com/)" + ], + "url": "https://www.senuto.com", + "description": "Senuto web crawler for SEO analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "MaCoCu", + "addition_date": "2023/09/08", + "instances": [ + "Mozilla/5.0 (compatible; MaCoCu; +https://www.clarin.si/info/macocu-massive-collection-and-curation-of-monolingual-and-bilingual-data/)" + ], + "url": "https://www.clarin.si/info/macocu-massive-collection-and-curation-of-monolingual-and-bilingual-data", + "description": "MaCoCu web crawler for language research", + "tags": [ + "academic" + ] + }, + { + "pattern": "NewsBlur", + "addition_date": "2023/09/08", + "instances": [ + "NewsBlur Feed Fetcher - 1 subscriber - http://www.newsblur.com/site/0000000/webpage (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Safari/605.1.15)" + ], + "url": "http://www.newsblur.com", + "description": "NewsBlur feed reader web crawler bot", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "inoreader", + "addition_date": "2023/09/08", + "instances": [ + "Mozilla/5.0 (compatible; inoreader.com; 1 subscribers)" + ], + "url": "http://inoreader.com", + "description": "Inoreader feed reader web crawler bot", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "NetSystemsResearch", + "addition_date": "2023/09/08", + "instances": [ + "NetSystemsResearch studies the availability of various services across the internet. Our website is netsystemsresearch.com" + ], + "url": "http://netsystemsresearch.com", + "description": "NetSystemsResearch web crawler bot", + "tags": [ + "scanner" + ] + }, + { + "pattern": "PageThing", + "addition_date": "2023/09/08", + "instances": [ + "PageThing http://pagething.com curl www" + ], + "url": "http://pagething.com", + "description": "PageThing web crawler for content discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "WordPress\\/", + "addition_date": "2023/10/24", + "instances": [ + "WordPress/X.X.X; https://example.com" + ], + "url": "https://wordpress.org", + "description": "WordPress web crawler for site discovery", + "tags": [ + "seo" + ] + }, + { + "pattern": "PhxBot", + "addition_date": "2024/01/06", + "instances": [ + "PhxBot/0.1 (phxbot@protonmail.com)" + ], + "description": "Phoenix web crawler for content discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "ImagesiftBot", + "addition_date": "2024/01/06", + "instances": [ + "Mozilla/5.0 (compatible; ImagesiftBot; +imagesift.com)" + ], + "url": "https://imagesift.com/about", + "description": "Imagesift bot for image search and indexing", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Expanse", + "addition_date": "2024/02/01", + "instances": [ + "Expanse, a Palo Alto Networks company, searches across the global IPv4 space multiple times per day to identify customers' presences on the Internet. If you would like to be excluded from our scans, please send IP addresses/domains to: scaninfo@paloaltonetworks.com" + ], + "url": "https://www.paloaltonetworks.com/cortex/cortex-xpanse", + "description": "Palo Alto Networks Expanse bot for internet asset discovery", + "tags": [ + "scanner" + ] + }, + { + "pattern": "InternetMeasurement", + "addition_date": "2024/02/01", + "instances": [ + "Mozilla/5.0 (compatible; InternetMeasurement/1.0; +https://internet-measurement.com/)" + ], + "url": "https://internet-measurement.com", + "description": "Internet Measurement bot for network research and analysis", + "tags": [ + "academic" + ] + }, + { + "pattern": "^BW\\/", + "addition_date": "2024/02/08", + "instances": [ + "BW/1.1; bit.ly/3eZNDnO", + "BW/1.1; rb.gy/oupwis" + ], + "url": "https://builtwith.com/biup", + "description": "BuiltWith web crawler for technology detection", + "tags": [ + "seo" + ] + }, + { + "pattern": "GeedoBot", + "addition_date": "2024/02/11", + "instances": [ + "Mozilla/5.0 (compatible; GeedoBot; +http://www.geedo.com/bot.html)" + ], + "url": "http://www.geedo.com", + "description": "Geedo web crawler for content discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Audisto Crawler", + "addition_date": "2024/03/14", + "instances": [ + "Audisto Crawler (mobile; +https://audisto.com/bot)", + "Audisto Crawler (desktop; +https://audisto.com/bot)", + "Audisto Crawler (mobile; essential; +https://audisto.com/bot)", + "Audisto Crawler (desktop; essential; +https://audisto.com/bot)" + ], + "url": "https://audisto.com/help/crawler/bot/", + "description": "Audisto SEO web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "PerplexityBot\\/", + "addition_date": "2024/03/14", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" + ], + "url": "https://docs.perplexity.ai/docs/perplexitybot", + "description": "Perplexity AI web crawler for search", + "tags": [ + "ai-crawler", + "search-engine" + ] + }, + { + "pattern": "[cC]laude[bB]ot", + "addition_date": "2024/04/19", + "instances": [ + "claudebot", + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" + ], + "url": "https://www.anthropic.com/", + "description": "Anthropic Claude web crawler bot", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "Monsidobot", + "addition_date": "2024/05/14", + "instances": [ + "Mozilla/5.0 (compatible; Monsidobot/2.2; +http://monsido.com/bot.html; info@monsido.com)" + ], + "url": "http://monsido.com/bot.html", + "description": "Monsido web crawler for website monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "GroupMeBot", + "addition_date": "2024/05/19", + "instances": [ + "GroupMeBot/1.0" + ], + "url": "https://groupme.com/", + "description": "GroupMe web crawler for messaging", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Vercelbot", + "addition_date": "2024/08/30", + "instances": [ + "Vercelbot (+https://vercel.com)" + ], + "url": "https://github.com/vercel/vercel/discussions/5095#discussioncomment-58705", + "description": "Vercel web crawler for deployment", + "tags": [ + "seo" + ] + }, + { + "pattern": "vercel-screenshot", + "addition_date": "2024/08/30", + "instances": [], + "description": "Vercel screenshot web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "facebookcatalog\\/", + "addition_date": "2024/10/03", + "instances": [ + "facebookcatalog/1.0" + ], + "url": "https://developers.facebook.com/docs/sharing/webmasters/web-crawlers", + "description": "Facebook catalog web crawler bot", + "tags": [ + "advertising" + ] + }, + { + "pattern": "meta-externalads\\/", + "addition_date": "2025/08/08", + "instances": [ + "meta-externalads/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)", + "meta-externalads/1.1" + ], + "url": "https://developers.facebook.com/docs/sharing/webmasters/web-crawlers", + "description": "Meta external ads web crawler bot", + "tags": [ + "advertising" + ] + }, + { + "pattern": "meta-externalagent\\/", + "addition_date": "2024/10/03", + "instances": [ + "meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)", + "meta-externalagent/1.1" + ], + "url": "https://developers.facebook.com/docs/sharing/webmasters/web-crawlers", + "description": "Meta external agent web crawler bot", + "tags": [ + "ai-crawler", + "social-preview" + ] + }, + { + "pattern": "meta-externalfetcher\\/", + "addition_date": "2024/10/03", + "instances": [ + "meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)", + "meta-externalfetcher/1.1" + ], + "url": "https://developers.facebook.com/docs/sharing/webmasters/web-crawlers", + "description": "Meta external fetcher web crawler bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "AcademicBotRTU", + "addition_date": "2024/10/17", + "instances": [ + "AcademicBotRTU (https://academicbot.rtu.lv; mailto:caps@rtu.lv)" + ], + "url": "https://academicbot.rtu.lv", + "description": "Academic Bot RTU web crawler bot", + "tags": [ + "academic" + ] + }, + { + "pattern": "KeybaseBot", + "addition_date": "2024/10/21", + "url": "https://book.keybase.io/docs/chat/link-previews", + "instances": [ + "Mozilla/5.0 (compatible; KeybaseBot; +https://keybase.io)" + ], + "description": "Keybase web crawler for link previews", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Lemmy", + "addition_date": "2025/02/11", + "instances": [ + "Lemmy/0.19.8; +https://leminal.space" + ], + "url": "https://leminal.space", + "description": "Lemmy social network web crawler bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "CookieHubScan", + "addition_date": "2024/11/29", + "url": "https://www.cookiehub.com/", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 CookieHubScan/3.0" + ], + "description": "CookieHub web crawler for cookie scanning", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Hydrozen\\.io", + "addition_date": "2025/02/02", + "instances": [ + "Hydrozen.io/1.0" + ], + "url": "https://docs.hydrozen.io/overview/misc/user-agent-and-ip-list", + "description": "Hydrozen web crawler for monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "HTTP Banner Detection", + "addition_date": "2025/02/10", + "instances": [ + "HTTP Banner Detection (https://security.ipip.net)" + ], + "url": "https://security.ipip.net", + "description": "IPIP HTTP banner detection web crawler", + "tags": [ + "scanner" + ] + }, + { + "pattern": "SummalyBot", + "addition_date": "2025/02/10", + "instances": [ + "SummalyBot/5.1.0" + ], + "url": "https://github.com/misskey-dev/summaly", + "description": "Summaly web crawler for content summarization", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "MicrosoftPreview\\/", + "addition_date": "2025/02/11", + "url": "https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0", + "instances": [ + "MicrosoftPreview/2.0; +https://aka.ms/MicrosoftPreview" + ], + "description": "Microsoft preview web crawler bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "GeedoProductSearch", + "addition_date": "2025/03/15", + "url": "http://www.geedo.com/product-search.html", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GeedoProductSearch; +http://www.geedo.com/product-search.html) Chrome/79.0.3945.88 Safari/537.36" + ], + "description": "Geedo product search web crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "TikTokSpider", + "addition_date": "2025/03/16", + "instances": [ + "Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; TikTokSpider; ttspider-feedback@tiktok.com)" + ], + "description": "TikTok web crawler for content discovery", + "tags": [ + "ai-crawler", + "social-preview" + ] + }, + { + "pattern": "OnCrawl\\/", + "addition_date": "2025/03/27", + "url": "http://www.oncrawl.com", + "instances": [ + "Mozilla/5.0 (compatible; OnCrawl/1.0; +http://www.oncrawl.com)", + "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; OnCrawl/1.0; +http://www.oncrawl.com)" + ], + "description": "OnCrawl SEO web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "sindresorhus\\/got", + "addition_date": "2025/04/22", + "url": "https://github.com/sindresorhus/got", + "instances": [ + "got (https://github.com/sindresorhus/got)" + ], + "description": "Got HTTP client library for requests", + "tags": [ + "http-library" + ] + }, + { + "pattern": "CensysInspect\\/", + "addition_date": "2025/04/22", + "url": "https://about.censys.io", + "instances": [ + "Mozilla/5.0 (compatible; CensysInspect/1.1; +https://about.censys.io/)" + ], + "description": "Censys web crawler for security scanning", + "tags": [ + "scanner" + ] + }, + { + "pattern": "SBIntuitionsBot\\/", + "addition_date": "2025/04/23", + "url": "https://www.sbintuitions.co.jp/bot/", + "instances": [ + "Mozilla/5.0 (compatible; SBIntuitionsBot/0.1; +https://www.sbintuitions.co.jp/bot/)" + ], + "description": "SB Intuitions web crawler bot", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "sitebulb", + "addition_date": "2025/04/30", + "url": "https://sitebulb.com/", + "instances": [ + "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6439.0 Mobile Safari/537.36 +https://sitebulb.com" + ], + "description": "Sitebulb SEO web crawler for analysis", + "tags": [ + "seo" + ] + }, + { + "pattern": "YextBot\\/", + "addition_date": "2025/08/08", + "url": "https://hitchhikers.yext.com/modules/kg140-yext-site-crawler/01-create-a-crawler/", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/87.0.4280.88 YextBot/Java Safari/537.36" + ], + "description": "Yext web crawler for site crawling", + "tags": [ + "seo" + ] + }, + { + "pattern": "DatadogSynthetics", + "addition_date": "2025/08/19", + "url": "https://docs.datadoghq.com/synthetics/", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.7204.168 Safari/537.36 DatadogSynthetics" + ], + "description": "Datadog synthetics web crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Google-Ads-Conversions", + "addition_date": "2025/09/10", + "url": "https://developers.google.com/google-ads/api/docs/conversions/upload-online", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 Chrome/139.0.7258.127 Safari/537.36 Google-Ads-Conversions" + ], + "description": "Google Ads conversions web crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "ObservePoint", + "addition_date": "2025/12/23", + "url": "https://help.observepoint.com/en/articles/9101465-allow-exclude-observepoint-traffic#h_2a8176c9b9", + "instances": [], + "description": "ObservePoint web crawler for monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Checkly", + "addition_date": "2026/02/11", + "url": "https://www.checklyhq.com/docs/", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.6998.35 Safari/537.36 (Checkly, https://www.checklyhq.com)" + ], + "description": "Checkly web crawler for monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "ALittle Client", + "addition_date": "2026/04/07", + "url": "https://udger.com/resources/ua-list/bot-detail?bot=ALittle+Client", + "instances": [ + "ALittle Client" + ], + "description": "ALittle web crawler for content discovery", + "tags": [ + "http-library" + ] + }, + { + "pattern": "AliyunSecBot", + "addition_date": "2026/04/07", + "url": "https://service.alibaba.com", + "instances": [ + "AliyunSecBot/Aliyun AliyunSecBot@service.alibaba.com" + ], + "description": "Alibaba Aliyun security web crawler bot", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Claude-Web", + "addition_date": "2026/04/07", + "url": "https://anthropic.com", + "instances": [ + "Claude-Web/1.0 (web crawler; +https://www.anthropic.com/; bots@anthropic.com)" + ], + "description": "Anthropic Claude web crawler bot", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "anthropic-ai", + "addition_date": "2026/04/07", + "url": "https://anthropic.com", + "instances": [ + "anthropic-ai" + ], + "description": "Anthropic AI web crawler bot", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "Claude-User", + "addition_date": "2026/04/07", + "url": "https://useragents.io/uas/mozilla-5-0-applewebkit-537-36-khtml-like-gecko-compatible-claudebot-1-0-supportanthropic-com_954fa13a8e1e46d8267fb56e2d48100e", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)", + "Claude-User (claude-code/2.1.86; +https://support.anthropic.com/)" + ], + "description": "Anthropic Claude user web crawler bot", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "Claude-SearchBot", + "addition_date": "2026/04/07", + "url": "https://useragents.io/uas/mozilla-5-0-applewebkit-537-36-khtml-like-gecko-compatible-claudebot-1-0-supportanthropic-com_954fa13a8e1e46d8267fb56e2d48100e", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com)" + ], + "description": "Anthropic Claude search web crawler bot", + "tags": [ + "ai-crawler", + "search-engine" + ] + }, + { + "pattern": "Google-Extended", + "addition_date": "2026/04/07", + "url": "https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers", + "instances": [ + "Mozilla/5.0 (compatible; Google-Extended/1.0; +http://www.google.com/bot.html)" + ], + "description": "Google extended web crawler bot", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "cohere-ai", + "addition_date": "2026/04/07", + "url": "https://cohere.com", + "instances": [ + "cohere-ai" + ], + "description": "Cohere AI web crawler bot", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "Timpibot", + "addition_date": "2026/04/07", + "url": "https://timpi.io", + "instances": [ + "Timpibot/0.9 (+http://www.timpi.io)", + "Mozilla/5.0 (compatible; Timpibot/0.8; +http://www.timpi.io)", + "Mozilla/5.0 (compatible; Timpibot/0.9; +http://www.timpi.io)" + ], + "description": "Timpi web crawler for content discovery", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "SERankingBacklinksBot", + "addition_date": "2026/04/07", + "url": "https://seranking.com/backlinks-crawler", + "instances": [ + "Mozilla/5.0 (compatible; SERankingBacklinksBot/1.0; +https://seranking.com/backlinks-crawler)" + ], + "description": "SEranking backlinks web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "CMSChecker", + "addition_date": "2026/04/07", + "instances": [ + "Mozilla/5.0 (compatible; CMSChecker/1.0; +https://cmschecker.net)" + ], + "description": "CMS Checker web crawler for detection", + "tags": [ + "seo" + ] + }, + { + "pattern": "Wayback", + "addition_date": "2026/04/07", + "url": "https://archive.org", + "instances": [ + "Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)" + ], + "description": "Internet Archive Wayback web crawler bot", + "tags": [ + "archiver" + ] + }, + { + "pattern": "Playwright", + "addition_date": "2026/04/07", + "url": "https://playwright.dev", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Playwright/1.40.0" + ], + "description": "Playwright browser automation web crawler", + "tags": [ + "browser-automation" + ] + }, + { + "pattern": "Puppeteer", + "addition_date": "2026/04/07", + "url": "https://pptr.dev", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/120.0.0.0 Safari/537.36 Puppeteer" + ], + "description": "Puppeteer browser automation web crawler", + "tags": [ + "browser-automation" + ] + }, + { + "pattern": "Selenium", + "addition_date": "2026/04/07", + "url": "https://www.selenium.dev", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36; Selenium" + ], + "description": "Selenium browser automation web crawler", + "tags": [ + "browser-automation" + ] + }, + { + "pattern": "Nikto", + "addition_date": "2026/04/07", + "url": "https://cirt.net/Nikto2", + "instances": [ + "Mozilla/5.00 (Nikto/2.1.5) (Evasions:None) (Test:Port Check)", + "Mozilla/5.0 (X11; Linux x86_64) Nikto/2.5.0 (Evasions:None) (Test:Port Check)" + ], + "description": "Nikto web server security scanner bot", + "tags": [ + "scanner" + ] + }, + { + "pattern": "sqlmap", + "addition_date": "2026/04/07", + "url": "https://sqlmap.org", + "instances": [ + "sqlmap/1.7.8#stable (https://sqlmap.org)" + ], + "description": "SQLMap SQL injection testing web crawler", + "tags": [ + "scanner" + ] + }, + { + "pattern": "ZmEu", + "addition_date": "2026/04/07", + "url": "https://en.wikipedia.org/wiki/ZmEu_(vulnerability_scanner)", + "instances": [ + "ZmEu" + ], + "description": "ZmEu vulnerability scanner web crawler", + "tags": [ + "scanner" + ] + }, + { + "pattern": "masscan", + "addition_date": "2026/04/07", + "url": "https://github.com/robertdavidgraham/masscan", + "instances": [ + "masscan/1.0 (https://github.com/robertdavidgraham/masscan)" + ], + "description": "Masscan network scanner web crawler bot", + "tags": [ + "scanner" + ] + }, + { + "pattern": "WPScan", + "addition_date": "2026/04/07", + "url": "https://wpscan.com", + "instances": [ + "WPScan v3.8.22 (https://wpscan.com/wordpress-security-scanner)", + "Mozilla/5.0 (compatible; WPScan; +https://wpscan.com/wordpress-security-scanner)" + ], + "description": "WPScan WordPress security scanner bot", + "tags": [ + "scanner" + ] + }, + { + "pattern": "[aA]cunetix", + "addition_date": "2026/04/07", + "url": "https://www.acunetix.com", + "instances": [ + "Mozilla/5.0 (Windows NT 6.1; WOW64) acunetix-product/wvs (Acunetix Web Vulnerability Scanner - Free Edition)", + "acunetix-product/wvs" + ], + "description": "Acunetix web vulnerability scanner bot", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Nessus", + "addition_date": "2026/04/07", + "url": "https://www.tenable.com/products/nessus", + "instances": [ + "Mozilla/5.0 (compatible; Nessus; http://www.nessus.org)" + ], + "description": "Nessus vulnerability scanner web crawler", + "tags": [ + "scanner" + ] + }, + { + "pattern": "[dD]ir[Bb]uster", + "addition_date": "2026/04/07", + "url": "https://github.com/KajanM/DirBuster", + "instances": [ + "DirBuster-1.0-RC1 (http://www.owasp.org/index.php/Category:OWASP_DirBuster_Project)" + ], + "description": "DirBuster directory scanner web crawler", + "tags": [ + "scanner" + ] + }, + { + "pattern": "StatusCake", + "addition_date": "2026/04/07", + "url": "https://www.statuscake.com", + "instances": [ + "StatusCake_Uptime_Checker/1.0" + ], + "description": "StatusCake uptime monitoring web crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "colly", + "addition_date": "2026/04/07", + "url": "https://go-colly.org", + "instances": [ + "colly - https://github.com/gocolly/colly", + "colly/2.1.0" + ], + "description": "Colly Go web crawler framework", + "tags": [ + "http-library" + ] + }, + { + "pattern": "[mM]echanize", + "addition_date": "2026/04/07", + "url": "https://github.com/sparklemotion/mechanize", + "instances": [ + "Mechanize/2.9.1 Ruby/3.1.2 (http://github.com/sparklemotion/mechanize/)" + ], + "description": "Mechanize Ruby web crawler library", + "tags": [ + "http-library" + ] + }, + { + "pattern": "air\\.ai\\/scanning", + "addition_date": "2026/04/07", + "instances": [ + "air.ai/scanning Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/126.0.0.0 Safari/537.36" + ], + "description": "Air.ai web crawler for scanning", + "tags": [ + "scanner" + ] + }, + { + "pattern": "asnriskscorer", + "addition_date": "2026/04/07", + "instances": [ + "asnriskscorer/1.0" + ], + "description": "ASN Risk Scorer web crawler bot", + "tags": [ + "scanner" + ] + }, + { + "pattern": "OICrawler", + "addition_date": "2026/04/07", + "url": "https://openindex.ai", + "instances": [ + "OICrawler/Nutch https://openindex.ai" + ], + "description": "OpenIndex web crawler for indexing", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "l9scan", + "addition_date": "2026/04/07", + "url": "https://github.com/LeakIX/l9scan", + "instances": [ + "Mozilla/5.0 (l9scan/2.0; +https://github.com/LeakIX/l9scan)" + ], + "description": "L9scan web crawler for scanning", + "tags": [ + "scanner" + ] + }, + { + "pattern": "SlaccaleBot", + "addition_date": "2026/04/07", + "instances": [ + "SlaccaleBot" + ], + "description": "Slaccale web crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "CustomAsyncHttpClient", + "addition_date": "2026/04/07", + "instances": [ + "CustomAsyncHttpClient" + ], + "description": "Custom async HTTP client web crawler", + "tags": [ + "http-library" + ] + }, + { + "pattern": "^HTTPie\\/", + "addition_date": "2026/04/07", + "url": "https://httpie.io", + "instances": [ + "HTTPie/3.2.2" + ], + "description": "HTTPie command-line HTTP client tool", + "tags": [ + "http-library" + ] + }, + { + "pattern": "Gemini-Deep-Research", + "addition_date": "2026/04/07", + "url": "https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Gemini-Deep-Research; +https://gemini.google/overview/deep-research/) Chrome/135.0.0.0 Safari/537.36" + ], + "description": "Google Gemini deep research web crawler", + "tags": [ + "ai-crawler", + "search-engine" + ] + }, + { + "pattern": "Perplexity-User", + "addition_date": "2026/04/07", + "url": "https://docs.perplexity.ai/guides/bots", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)" + ], + "description": "Perplexity user web crawler bot", + "tags": [ + "ai-crawler", + "search-engine" + ] + }, + { + "pattern": "PerplexityUser", + "addition_date": "2026/04/07", + "url": "https://perplexity.ai", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityUser/1.0; +https://perplexity.ai)" + ], + "description": "Perplexity AI web crawler bot", + "tags": [ + "ai-crawler", + "search-engine" + ] + }, + { + "pattern": "meta-webindexer", + "addition_date": "2026/04/07", + "url": "https://developers.facebook.com/docs/sharing/webmasters/web-crawlers#meta-webindexer", + "instances": [ + "meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)" + ], + "description": "Meta web indexer bot for Facebook and Instagram content crawling", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "DuckAssistBot", + "addition_date": "2026/04/07", + "url": "https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot", + "instances": [ + "DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)" + ], + "description": "DuckDuckGo assistant bot for web crawling and search", + "tags": [ + "search-engine", + "ai-crawler" + ] + }, + { + "pattern": "MistralAI-User", + "addition_date": "2026/04/07", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)" + ], + "description": "Mistral AI web crawler bot for content indexing", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "webzio", + "addition_date": "2026/04/07", + "url": "https://webz.io/blog/company/from-omgilibot-to-the-webzbot-duo-a-powerful-leap-for-ethical-and-comprehensive-data-collection/#", + "instances": [ + "webzio (+https://webz.io/bot.html)" + ], + "description": "Webz.io web crawler bot for ethical data collection", + "tags": [ + "seo" + ] + }, + { + "pattern": "newsai\\/", + "addition_date": "2026/04/14", + "url": "https://knownagents.com/agents/newsai", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 newsai/1.0 Safari/537.36" + ], + "description": "NewsAI crawler for news content aggregation and indexing", + "tags": [ + "ai-crawler", + "feed-reader" + ] + }, + { + "pattern": "^ArenaUnfurlBot", + "url": "https://arena.ai/", + "instances": [ + "ArenaUnfurlBot/1.0 (Link preview bot for AI model comparison platform; +https://arena.ai) Mozilla/5.0 (compatible)" + ], + "addition_date": "2026/04/26", + "description": "Arena.ai link preview generator", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "A360-Search", + "url": "https://area360.uk/", + "instances": [ + "Mozilla/5.0 (compatible; A360-Search; +https://area360.uk/)" + ], + "addition_date": "2026/04/26", + "description": "Area360 real estate data crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "AASA-Bot", + "url": "https://developer.apple.com/documentation/xcode/allowing-apps-and-websites-to-link-to-your-content", + "instances": [ + "AASA-Bot/1.0.0" + ], + "addition_date": "2026/04/26", + "description": "Apple's Universal Links file fetcher", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "AccessStatus", + "url": "https://accesslink.fr/page/a-propos-de-accessstatus/", + "instances": [ + "Mozilla/5.0 (compatible; AccessStatus/1.0; +https://www.accesslink.fr/page/a-propos-de-accessstatus/)" + ], + "addition_date": "2026/04/26", + "description": "HTTP status code monitoring bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Acquia optimize", + "url": "https://knownagents.com/agents/acquia-optimize-monsido", + "instances": [ + "Acquia optimize (Monsido)" + ], + "addition_date": "2026/04/26", + "description": "Acquia website optimization monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "ActiveComply", + "url": "https://knownagents.com/agents/activecomply-bot", + "instances": [ + "Mozilla/5.0 (compatible; ActiveComply; +https://www.activecomply.com/)", + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.7339.16 Safari/537.36 (compatible; ActiveComply/2.0; +https://app.activecomply.com/bot)" + ], + "addition_date": "2026/04/26", + "description": "ActiveComply compliance monitoring bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "AdkernelTopicCrawler", + "url": "http://adkernel.com/robot/", + "instances": [ + "Mozilla/5.0 (compatible; AdkernelTopicCrawler/1.0; +http://adkernel.com/robot/)" + ], + "addition_date": "2026/04/26", + "description": "Adkernel ad tech solutions crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "AlertSite", + "url": "https://smartbear.com/product/alertsite/", + "instances": [ + "Mozilla/5.0 (compatible; AlertSite; +https://www.alertsite.com/)" + ], + "addition_date": "2026/04/26", + "description": "SmartBear's synthetic monitoring service", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "AllAfrica", + "url": "https://allafrica.com/misc/info/about/", + "instances": [ + "Mozilla/5.0 (compatible; AllAfrica; +https://allafrica.com/)", + "Mozilla/5.0 (compatible; AllAfrica NewsBot; +https://allafrica.com/)" + ], + "addition_date": "2026/04/26", + "description": "AllAfrica news aggregation crawler", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Amazing-SearchBot", + "url": "https://amazing.com/bot.html", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazing-SearchBot/0.1; +https://amazing.com/bot.html) Chrome/119.0.6045.214 Safari/537.36" + ], + "addition_date": "2026/04/26", + "description": "Amazing e-commerce platform search crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Amazon-Bedrock-AgentCore-Browser", + "url": "https://docs.aws.amazon.com/bedrock-agentcore/", + "instances": [ + "Mozilla/5.0 (compatible; Amazon-Bedrock-AgentCore-Browser; +https://aws.amazon.com/bedrock/)" + ], + "addition_date": "2026/04/26", + "description": "AWS cloud browser for AI agents", + "tags": [ + "browser-automation" + ] + }, + { + "pattern": "AmazonBuyForMe", + "url": "https://buyforme.amazon/", + "instances": [ + "Mozilla/5.0 (compatible; AmazonBuyForMe; +https://www.amazon.com/)" + ], + "addition_date": "2026/04/26", + "description": "Amazon bot for Buy For Me service purchases", + "tags": [ + "browser-automation" + ] + }, + { + "pattern": "Amzn-SearchBot", + "url": "https://developer.amazon.com/amazonbot", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/0.1) Chrome/119.0.6045.214 Safari/537.36" + ], + "addition_date": "2026/04/26", + "description": "Amazon's AI search indexer for Alexa", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "Amzn-User", + "url": "https://developer.amazon.com/amazonbot", + "instances": [ + "Mozilla/5.0 (compatible; Amzn-User; +https://developer.amazon.com/)" + ], + "addition_date": "2026/04/26", + "description": "Amazon AI assistant for Alexa queries", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "Anchor Browser", + "url": "https://knownagents.com/agents/anchor-browser", + "instances": [ + "Mozilla/5.0 (compatible; Anchor Browser; +https://anchorbrowser.io/)" + ], + "addition_date": "2026/04/26", + "description": "Anchor's cloud-hosted browser for AI agents", + "tags": [ + "browser-automation" + ] + }, + { + "pattern": "Anomura", + "url": "https://docs.direqt-search.com/direqt-bots/direqt-crawlers-and-user-agents", + "instances": [ + "Anomura/1.2 (+https://www.direqt.ai)" + ], + "addition_date": "2026/04/26", + "description": "Direqt's AI search indexer", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "AP3A\\.240617\\.008", + "url": "https://knownagents.com/agents/008", + "instances": [ + "Mozilla/5.0 (Linux; Android 15; CPH2557 Build/AP3A.240617.008; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/142.0.7444.142 Mobile Safari/537.36 Instagram 406.0.0.58.159 Android (35/15; 480dpi; 1080x2400; OPPO; CPH2557; OP573DL1; mt6833; en_MY; 822918295; IABMV/1) NV/1" + ], + "addition_date": "2026/04/26", + "description": "80legs web scraping service", + "tags": [ + "scanner" + ] + }, + { + "pattern": "ApifyBot", + "url": "https://knownagents.com/agents/apifybot", + "instances": [ + "Mozilla/5.0 (compatible; ApifyBot/1.0)" + ], + "addition_date": "2026/04/26", + "description": "Apify's web scraping and data extraction", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "ApifyWebsiteContentCrawler", + "url": "https://apify.com/apify/website-content-crawler", + "instances": [ + "ApifyWebsiteContentCrawler/1.0 (+https://apify.com/apify/website-content-crawler)" + ], + "addition_date": "2026/04/26", + "description": "Apify's full website content extractor", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "Archive-It", + "url": "http://archive-it.org/files/site-owners-special.html", + "instances": [ + "UM-Bentley-Archive-It", + "Mozilla/5.0 (compatible; special_archiver; Archive-It; +http://archive-it.org/files/site-owners-special.html)" + ], + "addition_date": "2026/04/26", + "description": "Internet Archive's web preservation crawler", + "tags": [ + "archiver" + ] + }, + { + "pattern": "artemis web reader", + "url": "https://artemis.jamesg.blog/bot", + "instances": [ + "artemis web reader/1.0 - https://artemis.jamesg.blog/bot" + ], + "addition_date": "2026/04/26", + "description": "Artemis personal web reader", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "atlassian-bot", + "url": "https://support.atlassian.com/organization-administration/docs/connect-custom-website-to-rovo/", + "instances": [ + "Mozilla/5.0 (compatible; atlassian-bot; +https://www.atlassian.com/)" + ], + "addition_date": "2026/04/26", + "description": "Atlassian Rovo's AI search crawler", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "Attracta", + "url": "https://attracta.com/", + "instances": [ + "Mozilla/5.0 (compatible; Attracta)" + ], + "addition_date": "2026/04/26", + "description": "Attracta SEO optimization crawler", + "tags": [ + "seo" + ] + }, + { + "pattern": "AudigentAdBot", + "url": "http://audigent.com/bot.html", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; AudigentAdBot; +http://www.audigent.com/bot.html) Chrome/W.X.Y.Z Safari/537.36" + ], + "addition_date": "2026/04/26", + "description": "Audigent targeted advertising crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Authory", + "url": "https://authory.com/about", + "instances": [ + "Mozilla/5.0 (compatible; Authory/1.0; +https://authory.com/)" + ], + "addition_date": "2026/04/26", + "description": "Automated content archiving for journalists", + "tags": [ + "archiver" + ] + }, + { + "pattern": "Automaton|Newsify Feed Fetcher", + "url": "https://knownagents.com/agents/automaton", + "instances": [ + "Newsify Feed Fetcher - 3 subscribers - http://www.newsify.co/site/899429/automaton-twocanoes-software (Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_1) AppleWebKit/534.48.3 (KHTML, like Gecko) Version/5.1 Safari/534.48.3)" + ], + "addition_date": "2026/04/26", + "description": "Automaton feed fetcher monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "AwarioRendererBot", + "url": "https://awario.com/help/", + "instances": [ + "AwarioRendererBot/1.0 (+https://awario.com/bots.html; bots@awario.com)" + ], + "addition_date": "2026/04/26", + "description": "Awario social media monitoring crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "AzureAI-SearchBot", + "url": "https://azure.microsoft.com/en-us/products/ai-services", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; AzureAI-SearchBot/1.0;" + ], + "addition_date": "2026/04/26", + "description": "Microsoft's Azure AI search indexer", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "BestChange", + "url": "https://bestchange.com/", + "instances": [ + "Mozilla/5.0 (compatible; BestChange; +https://bestchange.com/)", + "Mozilla/5.0 (compatible; BestChange Bot; +https://bestchange.com/)" + ], + "addition_date": "2026/04/26", + "description": "BestChange exchange rate monitoring crawler", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "bigsur\\.ai", + "url": "https://bigsur.ai/", + "instances": [ + "bigsur.ai" + ], + "addition_date": "2026/04/26", + "description": "Big Sur AI's web agent crawler", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "bl\\.uk_lddc_bot", + "url": "https://bl.uk/legal-deposit-web-archiving", + "instances": [ + "bl.uk_lddc_bot/3.4.0-20220727 (+https://www.bl.uk/legal-deposit-web-archiving)" + ], + "addition_date": "2026/04/26", + "description": "British Library's legal deposit archiver", + "tags": [ + "archiver" + ] + }, + { + "pattern": "BlingERP", + "url": "https://bling.com.br/", + "instances": [ + "Mozilla/5.0 (compatible; BlingERP; +https://www.bling.com.br/)" + ], + "addition_date": "2026/04/26", + "description": "Bling ERP data synchronization crawler", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Blockaid", + "url": "https://knownagents.com/agents/blockaid", + "instances": [ + "Mozilla/5.0 (compatible; Blockaid; +https://www.blockaid.io/)" + ], + "addition_date": "2026/04/26", + "description": "Blockaid security monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Bloglines", + "url": "http://bloglines.com/", + "instances": [ + "Bloglines/3.1 (http://www.bloglines.com)" + ], + "addition_date": "2026/04/26", + "description": "Bloglines RSS feed reader", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "BlogVault", + "url": "https://blogvault.net/", + "instances": [ + "BlogVault/1.0 (+https://blogvault.net)" + ], + "addition_date": "2026/04/26", + "description": "WordPress backup and monitoring service", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "bluesky-domain-status-classifier", + "url": "https://blueskyweb.xyz/", + "instances": [ + "bluesky-domain-status-classifier/1.0" + ], + "addition_date": "2026/04/26", + "description": "Bluesky social network link preview bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Bluesky\\/", + "url": "https://knownagents.com/agents/bluesky-link-preview-service", + "instances": [ + "Bluesky/" + ], + "addition_date": "2026/04/26", + "description": "Bluesky link preview service", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "bne\\.es_bot", + "url": "https://bne.es/es/colecciones/archivo-web-espanola/aviso-webmasters", + "instances": [ + "Mozilla/5.0 (compatible; bne.es_bot; https://www.bne.es/es/colecciones/archivo-web-espanola/aviso-webmasters) Firefox/129.0.1" + ], + "addition_date": "2026/04/26", + "description": "Spanish National Library web archiver", + "tags": [ + "archiver" + ] + }, + { + "pattern": "Brightbot", + "url": "https://brightdata.com/brightbot", + "instances": [ + "Brightbot 1.0" + ], + "addition_date": "2026/04/26", + "description": "Bright Data's AI-ready data collector", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "BrowserBot-Observer", + "url": "https://obsrvr.net/about", + "instances": [ + "BrowserBot-Observer (+https://siteobserver.co)" + ], + "addition_date": "2026/04/26", + "description": "Observer third-party risk security scanner", + "tags": [ + "scanner" + ] + }, + { + "pattern": "BufferLinkPreviewBot", + "url": "https://scraper.buffer.com/about/bots/link-preview-bot", + "instances": [ + "BufferLinkPreviewBot/1.0 (+https://scraper.buffer.com/about/bots/link-preview-bot)" + ], + "addition_date": "2026/04/26", + "description": "Buffer social media link preview bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Bugsnag", + "url": "https://knownagents.com/agents/bugsnag-script-fetcher", + "instances": [ + "Mozilla/5.0 (compatible; Bugsnag; +https://www.bugsnag.com/)", + "Bugsnag script fetcher (support@bugsnag.com)" + ], + "addition_date": "2026/04/26", + "description": "Bugsnag JavaScript source fetcher", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Buttondown", + "url": "https://buttondown.email/features", + "instances": [ + "Buttondown RSS-Feed-Parser/1.0 (https://buttondown.com)" + ], + "addition_date": "2026/04/26", + "description": "Buttondown RSS to email converter", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "CapitalOneBot", + "url": "https://developer.capitalone.com/", + "instances": [ + "Mozilla/5.0 (compatible; CapitalOneBot; +https://www.capitalone.com/)" + ], + "addition_date": "2026/04/26", + "description": "Capital One dealer website crawler", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "CertChief", + "url": "https://cert.chief.app/", + "instances": [ + "Mozilla/5.0 (compatible; CertChief; +https://www.certchief.com/)" + ], + "addition_date": "2026/04/26", + "description": "CertChief SSL certificate monitoring bot", + "tags": [ + "scanner" + ] + }, + { + "pattern": "channable", + "url": "https://channable.com/", + "instances": [ + "Mozilla/5.0 (compatible; channable; +https://www.channable.com/)" + ], + "addition_date": "2026/04/26", + "description": "Channable e-commerce marketing automation crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Channel3Bot", + "url": "https://trychannel3.com/channel3bot", + "instances": [ + "Mozilla/5.0 (compatible; Channel3Bot/1.0; +https://trychannel3.com/channel3bot)" + ], + "addition_date": "2026/04/26", + "description": "Channel3's universal product catalog indexer", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "Chirp|gotosocial", + "url": "http://binarycanary.com/", + "instances": [ + "gotosocial/0.19.2+git-90851fc (+https://chirp.zadzmo.org)" + ], + "addition_date": "2026/04/26", + "description": "Binary Canary uptime monitoring bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "ClickUpLinkUnfurler", + "url": "https://clickup.com/", + "instances": [ + "ClickUpLinkUnfurler/1.0 (+https://clickup.com)" + ], + "addition_date": "2026/04/26", + "description": "ClickUp link preview generator", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Cloudflare-AutoRAG", + "url": "https://developers.cloudflare.com/autorag", + "instances": [ + "Cloudflare-AutoRAG (https://developers.cloudflare.com/autorag; autorag@cloudflare.com)" + ], + "addition_date": "2026/04/26", + "description": "Cloudflare's RAG service indexer", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "Cloudflare-Custom-Hostname-Verification", + "url": "https://knownagents.com/agents/cloudflare-custom-hostname-verification", + "instances": [ + "Mozilla/5.0 (compatible; Cloudflare-Custom-Hostname-Verification; +https://www.cloudflare.com/)" + ], + "addition_date": "2026/04/26", + "description": "Cloudflare hostname verification crawler", + "tags": [ + "seo" + ] + }, + { + "pattern": "Cloudflare-Stream-Webhook", + "url": "https://knownagents.com/agents/cloudflare-stream-webhook", + "instances": [ + "Mozilla/5.0 (compatible; Cloudflare-Stream-Webhook; +https://www.cloudflare.com/)" + ], + "addition_date": "2026/04/26", + "description": "Cloudflare Stream webhook crawler", + "tags": [ + "seo" + ] + }, + { + "pattern": "CloudflareRadarURLScanner", + "url": "https://knownagents.com/agents/cloudflare-radar-url-scanner", + "instances": [ + "Mozilla/5.0 (compatible; CloudflareRadarURLScanner; +https://radar.cloudflare.com/)" + ], + "addition_date": "2026/04/26", + "description": "Cloudflare URL security scanner", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Cloudtrellis", + "url": "https://cloudtrellis.com/", + "instances": [ + "Mozilla/5.0 (compatible; Cloudtrellis; +https://www.cloudtrellis.com/)" + ], + "addition_date": "2026/04/26", + "description": "Cloudtrellis website audit crawler", + "tags": [ + "seo" + ] + }, + { + "pattern": "[cC]ludo", + "url": "https://knownagents.com/agents/cludo", + "instances": [ + "Mozilla/5.0 (compatible; Cludo; +https://www.cludo.com/)", + "Mozilla/5.0 (Windows NT 10.0; cludo.com bot) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" + ], + "addition_date": "2026/04/26", + "description": "Cludo site search monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Code\\/1\\.", + "url": "https://github.com/features/copilot", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Code/1.115.0 Chrome/142.0.7444.265 Electron/39.8.5 Safari/537.36" + ], + "addition_date": "2026/04/26", + "description": "GitHub Copilot AI coding agent", + "tags": [ + "browser-automation" + ] + }, + { + "pattern": "Collapsify", + "url": "https://developers.cloudflare.com/", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10; rv:60.0) Gecko/20100101 Firefox/60.0 Collapsify" + ], + "addition_date": "2026/04/26", + "description": "Cloudflare error page caching service", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "ContextualBot[\\s\\S]*outcomes\\.net", + "url": "http://outcomes.net/", + "instances": [ + "Mozilla/5.0 (compatible; ContextualBot/1.0; +http://outcomes.net)" + ], + "addition_date": "2026/04/26", + "description": "Mobian contextual intelligence crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Convermax", + "url": "https://docs.convermax.com/", + "instances": [ + "Convermax/1.0 (+https://docs.convermax.com/indexer)" + ], + "addition_date": "2026/04/26", + "description": "Convermax auto parts search indexer", + "tags": [ + "seo" + ] + }, + { + "pattern": "cookie-maestro", + "url": "https://cookiemaestro.com/documentatie/limit-cookie-maestro-using-robots-txt", + "instances": [ + "Mozilla/5.0 (compatible; cookie-maestro; +https://www.cookiemaestro.com/)" + ], + "addition_date": "2026/04/26", + "description": "Cookie Maestro GDPR compliance scanner", + "tags": [ + "scanner" + ] + }, + { + "pattern": "CookieHubVerify", + "url": "https://cookiehub.com/", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 CookieHubVerify/3.0" + ], + "addition_date": "2026/04/26", + "description": "CookieHub consent banner verification scanner", + "tags": [ + "scanner" + ] + }, + { + "pattern": "CookieYesbot", + "url": "http://cookieyes.com/documentation/cookieyesbot", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; CookieYesbot/1.0; +http://www.cookieyes.com/documentation/cookieyesbot) Chrome/131.0.6778.0 Safari/537.36" + ], + "addition_date": "2026/04/26", + "description": "CookieYes consent compliance scanner", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Crazy Egg", + "url": "https://knownagents.com/agents/crazy-egg", + "instances": [ + "Mozilla/5.0 (compatible; Crazy Egg; +https://www.crazyegg.com/)" + ], + "addition_date": "2026/04/26", + "description": "Crazy Egg analytics monitoring bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Current[\\s\\S]*RSS Reader", + "url": "https://currentreader.app/", + "instances": [ + "Mozilla/5.0 (compatible; Current/1.0; +https://currentreader.app; RSS Reader)" + ], + "addition_date": "2026/04/26", + "description": "Current RSS reader application", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "cypex\\.ai\\/scanning", + "url": "https://cypex.ai/", + "instances": [ + "cypex.ai/scanning Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/126.0.0.0 Safari/537.36" + ], + "addition_date": "2026/04/26", + "description": "Cypex security vulnerability scanner", + "tags": [ + "scanner" + ] + }, + { + "pattern": "DeepCrawl", + "url": "https://lumar.io/spdr/", + "instances": [ + "Mozilla/5.0 (compatible; MentalHealthLeadBot/DeepCrawl/1.0; +https://aiinstall.co)" + ], + "addition_date": "2026/04/26", + "description": "Lumar enterprise SEO platform crawler", + "tags": [ + "seo" + ] + }, + { + "pattern": "DigiCert DCV", + "url": "https://digicert.com/", + "instances": [ + "DigiCert DCV/1.1" + ], + "addition_date": "2026/04/26", + "description": "DigiCert domain validation scanner", + "tags": [ + "scanner" + ] + }, + { + "pattern": "dlvr\\.it", + "url": "http://dlvr.it/", + "instances": [ + "dlvr.it/1.0 (+http://dlvr.it/fetcher)" + ], + "addition_date": "2026/04/26", + "description": "dlvr.it social media automation service", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Dotcom-Monitor", + "url": "https://knownagents.com/agents/doctom-monitor", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36 (Dotcom-Monitor)" + ], + "addition_date": "2026/04/26", + "description": "Doctom website monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "DrataAutopilot", + "url": "https://knownagents.com/agents/drata-autopilot", + "instances": [ + "Mozilla/5.0 (compatible; DrataAutopilot; +https://drata.com/)" + ], + "addition_date": "2026/04/26", + "description": "Drata compliance monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "DreamHost Data Team", + "url": "http://dreamhost.com/support/", + "instances": [ + "DreamHost Data Team (+http://www.dreamhost.com/support/)" + ], + "addition_date": "2026/04/26", + "description": "DreamHost hosting monitoring bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "ds9", + "url": "https://data.dss.sps.copyright.com/docs/user_agent.html", + "instances": [ + "ds9 2.004.ec2" + ], + "addition_date": "2026/04/26", + "description": "Copyright Clearance Center licensing crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": " DVbot", + "url": "http://doubleverify.com/", + "instances": [ + "Mozilla/5.0 (Windows NT 6.3;compatible; DVbot/1.0; +http://www.doubleverify.com)" + ], + "addition_date": "2026/04/26", + "description": "DoubleVerify ad verification crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "EcoVadisSustainabilityBot", + "url": "https://ecovadis.com/", + "instances": [ + "Mozilla/5.0 (compatible; EcoVadisSustainabilityBot/1.0; +https://www.ecovadis.com; Crawling on behalf of one of your business partners.)" + ], + "addition_date": "2026/04/26", + "description": "EcoVadis sustainability ratings crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "elmah\\.io Uptime Monitoring", + "url": "https://knownagents.com/agents/elmah-io-uptime-monitoring", + "instances": [ + "elmah.io Uptime Monitoring" + ], + "addition_date": "2026/04/26", + "description": "elmah.io uptime monitoring service", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "EvernoteRichLinkBot", + "url": "https://evernote.com/", + "instances": [ + "EvernoteRichLinkBot/1.0 (+https://evernote.com)" + ], + "addition_date": "2026/04/26", + "description": "Evernote rich link preview bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "EzLynx", + "url": "http://ezoic.com/bot.html", + "instances": [ + "https://www.ezoic.com/bot/ Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Mobile Safari/537.36 (compatible; EzLynx/0.1; +http://www.ezoic.com/bot.html)" + ], + "addition_date": "2026/04/26", + "description": "Ezoic website optimization crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "EzoicBot", + "url": "https://ezoic.com/bot/", + "instances": [ + "Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Mobile Safari/537.36 (compatible; EzoicBot-UptimeOwl; +http://www.ezoic.com/bot)", + "Mozilla/5.0 (Linux; Android 8.0; Pixel 2 Build/OPD3.170816.012) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Mobile Safari/537.36 (compatible; EzoicBot-Nicheiq; +http://www.ezoic.com/bot)" + ], + "addition_date": "2026/04/26", + "description": "Ezoic ad monetization platform crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "FacebookBot", + "url": "https://developers.facebook.com/docs/sharing/bot/", + "instances": [ + "Mozilla/5.0 (compatible; FacebookBot/1.0; +https://developers.facebook.com/docs/sharing/webmasters/facebookbot/)" + ], + "addition_date": "2026/04/26", + "description": "Meta's AI speech recognition training crawler", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "FastDAST", + "url": "https://knownagents.com/agents/black-duck-fast-dynamic", + "instances": [ + "FastDAST", + "Mozilla/5.0 (compatible; FastDAST; +https://www.blackduck.com/)" + ], + "addition_date": "2026/04/26", + "description": "Black Duck security testing bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Feeder \\/", + "url": "https://knownagents.com/agents/feeder", + "instances": [ + "Feeder / 1.10.8(88)" + ], + "addition_date": "2026/04/26", + "description": "Feeder RSS reader app", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "FeedFlow", + "url": "https://feedflow.dev/", + "instances": [ + "FeedFlow/1.0" + ], + "addition_date": "2026/04/26", + "description": "FeedFlow RSS reader application", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "FindFiles\\.net", + "url": "https://findfiles.net/bot", + "instances": [ + "FindFiles.net-FaviconFetcher/1.0 (+https://findfiles.net/bot)" + ], + "addition_date": "2026/04/26", + "description": "FindFiles.net search engine crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "FirecrawlAgent", + "url": "https://firecrawl.dev/", + "instances": [ + "Mozilla/5.0 (compatible; FirecrawlAgent; +https://firecrawl.dev/)" + ], + "addition_date": "2026/04/26", + "description": "Firecrawl's LLM data extraction crawler", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "FyndSearchEngine-Crawler", + "url": "https://fynd.bot/", + "instances": [ + "FyndSearchEngine-Crawler (by fynd.bot; https://fynd.bot)" + ], + "addition_date": "2026/04/26", + "description": "Fynd search engine indexing crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "FyndSearchEngine-ReCrawler", + "url": "https://fynd.bot/", + "instances": [ + "FyndSearchEngine-ReCrawler (by fynd.bot; https://fynd.bot)" + ], + "addition_date": "2026/04/26", + "description": "Fynd search engine re-indexing crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Goodreads", + "url": "https://goodreads.com/", + "instances": [ + "Mozilla/5.0 (compatible; Goodreads; +https://www.goodreads.com)" + ], + "addition_date": "2026/04/26", + "description": "Goodreads book preview fetcher", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Google Trust Services", + "url": "https://knownagents.com/agents/google-trust-services-dcv-check", + "instances": [ + "Google Trust Services (DCV Check)" + ], + "addition_date": "2026/04/26", + "description": "Google SSL certificate validation", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Google-Agent", + "url": "https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent)" + ], + "addition_date": "2026/04/26", + "description": "Google's user-triggered agent for web navigation", + "tags": [ + "browser-automation" + ] + }, + { + "pattern": "Google-Gemini-CLI", + "url": "https://geminicli.com/", + "instances": [ + "Mozilla/5.0 (compatible; Google-Gemini-CLI/1.0; +https://github.com/google-gemini/gemini-cli)" + ], + "addition_date": "2026/04/26", + "description": "Google's AI coding agent for terminal", + "tags": [ + "browser-automation" + ] + }, + { + "pattern": "Google-NotebookLM", + "url": "https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers", + "instances": [ + "Mozilla/5.0 (compatible; Google-NotebookLM; +https://notebooklm.google.com/)" + ], + "addition_date": "2026/04/26", + "description": "Google's AI research and note-taking assistant", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "GoogleAgent-Mariner", + "url": "https://deepmind.google/technologies/project-mariner/", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleAgent-Mariner) Chrome/142.0.7444.162 Safari/537.36" + ], + "addition_date": "2026/04/26", + "description": "Google's AI agent for browser-based tasks", + "tags": [ + "browser-automation" + ] + }, + { + "pattern": "Greppr Web Crawler", + "url": "https://greppr.org/", + "instances": [ + "Mozilla/5.0 (compatible; Greppr Web Crawler/0.12.0 (https://greppr.org/))" + ], + "addition_date": "2026/04/26", + "description": "Greppr unfiltered search engine crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Hardenize", + "url": "https://hardenize.com/", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Hardenize" + ], + "addition_date": "2026/04/26", + "description": "Hardenize security configuration scanner", + "tags": [ + "scanner" + ] + }, + { + "pattern": "HoneybadgerBot", + "url": "https://knownagents.com/agents/honeybadgerbot", + "instances": [ + "Mozilla/5.0 (compatible; HoneybadgerBot; +https://www.honeybadger.io/)" + ], + "addition_date": "2026/04/26", + "description": "Honeybadger error monitoring bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "IbouBot", + "url": "https://ibou.io/iboubot.html", + "instances": [ + "Mozilla/5.0 (compatible; IbouBot/1.0; +bot@ibou.io; +https://ibou.io/iboubot.html)" + ], + "addition_date": "2026/04/26", + "description": "Ibou web graph search crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "imageSpider", + "url": "https://knownagents.com/agents/imagespider", + "instances": [ + "Mozilla/5.0 (compatible; imageSpider; +https://www.bytedance.com/)" + ], + "addition_date": "2026/04/26", + "description": "ByteDance's image collection crawler", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "Innologica", + "url": "https://knownagents.com/agents/innologica", + "instances": [ + "Mozilla/5.0 (compatible; Innologica; +https://www.innologica.com/)" + ], + "addition_date": "2026/04/26", + "description": "Innologica content fetcher", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "kagi-fetcher", + "url": "https://help.kagi.com/kagi/ai/kagi-ai.html", + "instances": [ + "kagi-fetcher/1.0" + ], + "addition_date": "2026/04/26", + "description": "Kagi AI assistant for web content", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "Kangaroo Bot", + "url": "https://kangaroollm.com.au/kangaroo-bot/", + "instances": [ + "Mozilla/5.0 (compatible; Kangaroo Bot/1.0; +http://www.kangaroo.com)" + ], + "addition_date": "2026/04/26", + "description": "Australian AI model training crawler", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "Known Agent", + "url": "https://knownagents.com/", + "instances": [ + "Mozilla/5.0 (compatible; Known Agent/1.0; +https://knownagents.com)", + "Mozilla/5.0 (compatible; Known Agents Browser/1.0; +https://knownagents.com)" + ], + "addition_date": "2026/04/26", + "description": "Known Agents test and development bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "KrawlerBot", + "url": "https://krawler.app/robot", + "instances": [ + "Mozilla/5.0 (compatible; KrawlerBot; +https://www.krawler.com/)" + ], + "addition_date": "2026/04/26", + "description": "Krawler web scraping service bot", + "tags": [ + "scanner" + ] + }, + { + "pattern": "laion-huggingface-processor", + "url": "https://knownagents.com/agents/laion-huggingface-processor", + "instances": [ + "Mozilla/5.0 (compatible; laion-huggingface-processor; +https://laion.ai/)" + ], + "addition_date": "2026/04/26", + "description": "LAION's image dataset builder for AI", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "LinkCheckerBot", + "url": "https://knownagents.com/agents/linkchecker-bot", + "instances": [ + "Mozilla/5.0 (compatible; LinkCheckerBot/2.0; +https://linkchecker.pro/robot/)" + ], + "addition_date": "2026/04/26", + "description": "LinkChecker broken link bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "LinkupBot", + "url": "https://linkup.so/bot", + "instances": [ + "LinkupBot/1.0 (LinkupBot for web indexing; https://linkup.so/bot; bot@linkup.so)" + ], + "addition_date": "2026/04/26", + "description": "Linkup's enterprise AI search crawler", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "LMArenaUnfurlBot", + "url": "https://lmarena.ai/", + "instances": [ + "LMArenaUnfurlBot/1.0 (Link preview bot for AI model comparison platform; +https://lmarena.ai) Mozilla/5.0 (compatible)" + ], + "addition_date": "2026/04/26", + "description": "LM Arena link preview bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "lyonl-asset-proxy", + "url": "https://lyonl.com/crawler", + "instances": [ + "lyonl-asset-proxy/1.0 (+https://lyonl.com/crawler)" + ], + "addition_date": "2026/04/26", + "description": "Lyonl search engine asset proxy", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "lyonl-crawler", + "url": "https://lyonl.com/crawler", + "instances": [ + "lyonl-crawler/0.1 (+https://lyonl.com/crawler; crawler@lyonl.com)" + ], + "addition_date": "2026/04/26", + "description": "Lyonl open web search crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "MagiBot", + "url": "https://magi.com/bots", + "instances": [ + "Mozilla/5.0 (compatible; MagiBot; +https://www.magi.com/)" + ], + "addition_date": "2026/04/26", + "description": "Peak Labs information extraction crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "MagpieRSS", + "url": "http://magpierss.sf.net/", + "instances": [ + "MagpieRSS/0.7 ( http://magpierss.sf.net)" + ], + "addition_date": "2026/04/26", + "description": "MagpieRSS PHP feed parser", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "mail\\.ru", + "url": "https://knownagents.com/agents/mailrubot", + "instances": [ + "Mozilla/5.0 (Windows NT 6.1; WOW64; https://top.mail.ru/passremind) AppleWebKit/537.22 (KHTML, like Gecko)" + ], + "addition_date": "2026/04/26", + "description": "Mail.ru content fetcher", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "MailChimp", + "url": "http://mailchimp.com/", + "instances": [ + "Mozilla/5.0 (compatible; MailChimp; +https://mailchimp.com/)" + ], + "addition_date": "2026/04/26", + "description": "Mailchimp email campaign content fetcher", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Manus-User", + "url": "https://knownagents.com/agents/manus-user", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36; Manus-User/1.0" + ], + "addition_date": "2026/04/26", + "description": "Butterfly Effect's AI agent for web tasks", + "tags": [ + "browser-automation" + ] + }, + { + "pattern": "McontextualBot", + "url": "http://mcontextual.net/mcontextual-bot", + "instances": [ + "Mozilla/5.0 (compatible; McontextualBot/0.1; +http://mcontextual.net/mcontextual-bot)" + ], + "addition_date": "2026/04/26", + "description": "MContextual advertising targeting crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Mediumbot-MetaTagFetcher", + "url": "https://medium.com/", + "instances": [ + "Mediumbot-MetaTagFetcher/0.3 (+https://medium.com/)" + ], + "addition_date": "2026/04/26", + "description": "Medium link preview meta tag fetcher", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "MetaIAB Facebook", + "url": "https://knownagents.com/agents/facebook", + "instances": [ + "Mozilla/5.0 (Linux; Android 16; Pixel 10 Pro XL Build/CP1A.260305.018; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/146.0.7680.174 Mobile Safari/537.36 MetaIAB Facebook" + ], + "addition_date": "2026/04/26", + "description": "Facebook link preview fetcher", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "MixrankBot", + "url": "https://knownagents.com/agents/mixrankbot", + "instances": [ + "Mozilla/5.0 (compatible; MixrankBot; crawler@mixrank.com)" + ], + "addition_date": "2026/04/26", + "description": "MixRank business intelligence crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "ModernizeBot", + "url": "https://modernizeyourwebsite.com/bot", + "instances": [ + "ModernizeBot/0.1 (+https://modernizeyourwebsite.com/bot)" + ], + "addition_date": "2026/04/26", + "description": "ModernizeYourWebsite analysis crawler", + "tags": [ + "seo" + ] + }, + { + "pattern": "MontasticMonitor", + "url": "https://knownagents.com/agents/montasticmonitor", + "instances": [ + "Mozilla/5.0 (compatible; MontasticMonitor; +https://montastic.com/)" + ], + "addition_date": "2026/04/26", + "description": "Montastic uptime monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "NanoInteractive", + "url": "https://nanointeractive.com/crawler/", + "instances": [ + "Mozilla/5.0 (iPhone; CPU iPhone OS 16_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.1 Mobile/15E148 Safari/604.1 (compatible; NanoInteractive/1.0; +https://nanointeractive.com/crawler/)" + ], + "addition_date": "2026/04/26", + "description": "Nano Interactive ad targeting crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "NestDaddybot", + "url": "https://nestdaddy.com/bot", + "instances": [ + "NestDaddybot/1.0.0 (+https://nestdaddy.com/bot)" + ], + "addition_date": "2026/04/26", + "description": "NestDaddy search engine crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Netcraft SSL Server Survey", + "url": "https://knownagents.com/agents/netcraft-ssl-server-survey", + "instances": [ + "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Netcraft SSL Server Survey - contact info@netcraft.com)" + ], + "addition_date": "2026/04/26", + "description": "Netcraft SSL certificate intelligence scanner", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Netcraft Web Server Survey", + "url": "https://netcraft.com/blog/june-2025-web-server-survey", + "instances": [ + "Mozilla/4.0 (compatible; Netcraft Web Server Survey)" + ], + "addition_date": "2026/04/26", + "description": "Netcraft web server survey crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "NetSeer crawler", + "url": "http://netseer.com/crawler.html", + "instances": [ + "Mozilla/5.0 (compatible; NetSeer crawler/2.0; +http://www.netseer.com/crawler.html; crawler@netseer.com)" + ], + "addition_date": "2026/04/26", + "description": "NetSeer intelligence gathering crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Netumo|netumo", + "url": "https://docs.netumo.com/", + "instances": [ + "Netumo/0.2 (+https://www.netumo.app/)" + ], + "addition_date": "2026/04/26", + "description": "Netumo uptime monitoring bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "NewRelicSynthetics", + "url": "https://knownagents.com/agents/new-relic", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.60 Safari/537.36 NewRelicSynthetics/1.0" + ], + "addition_date": "2026/04/26", + "description": "New Relic synthetic monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "NewsRoom\\.BI", + "url": "http://newsroom.bi/bot.html", + "instances": [ + "Mozilla/5.0 (compatible; NewsRoom.BI/0.1; +http://www.newsroom.bi/bot.html)" + ], + "addition_date": "2026/04/26", + "description": "Marfeel publisher analytics crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Nitro-", + "url": "https://knownagents.com/agents/nitro", + "instances": [ + "Mozilla/5.0 (compatible; Nitro-; +https://www.nitropack.io/)", + "Mozilla/5.0 (Linux; Android 15; GEC77) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Mobile Safari/537.36 Nitro-Optimizer-Agent" + ], + "addition_date": "2026/04/26", + "description": "NitroPack speed optimization crawler", + "tags": [ + "seo" + ] + }, + { + "pattern": "NitroBot", + "url": "https://knownagents.com/agents/nitrobot", + "instances": [ + "Mozilla/5.0 (compatible; NitroBot; +https://www.nitropack.io/)" + ], + "addition_date": "2026/04/26", + "description": "NitroPack SEO optimization bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "Noibu", + "url": "https://noibu.com/", + "instances": [ + "Mozilla/5.0 (compatible; Noibu; +https://www.noibu.com/)" + ], + "addition_date": "2026/04/26", + "description": "Noibu ecommerce error monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "NostoCrawlerBot", + "url": "http://my.nosto.com/tagging", + "instances": [ + "Mozilla/5.0 (compatible; NostoCrawlerBot/1.0; +http://my.nosto.com/tagging)" + ], + "addition_date": "2026/04/26", + "description": "Nosto ecommerce personalization crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "OneTrust", + "url": "https://knownagents.com/agents/onetrust-cmp-scanner", + "instances": [ + "OneTrust" + ], + "addition_date": "2026/04/26", + "description": "OneTrust consent management scanner", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "opencode-smartfetch", + "url": "https://opencode.ai/", + "instances": [ + "opencode-smartfetch/1.0" + ], + "addition_date": "2026/04/26", + "description": "Open-source AI coding agent", + "tags": [ + "browser-automation" + ] + }, + { + "pattern": ";Owler", + "url": "https://knownagents.com/agents/owler", + "instances": [ + "robot;Owler;sthiyagarajan@owler-inc.com" + ], + "addition_date": "2026/04/26", + "description": "Owler competitive intelligence crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "ParselySharesBot", + "url": "https://docs.parse.ly/", + "instances": [ + "ParselySharesBot (+http://parsely.com)" + ], + "addition_date": "2026/04/26", + "description": "Parse.ly content analytics crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "PhindBot", + "url": "https://knownagents.com/agents/phindbot", + "instances": [ + "Mozilla/5.0 (compatible; PhindBot; +https://www.phind.com/)" + ], + "addition_date": "2026/04/26", + "description": "AI-powered developer answer engine crawler", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "PodchaserParser", + "url": "https://podchaser.com/", + "instances": [ + "PodchaserParser/2.0 (https://podchaser.com)" + ], + "addition_date": "2026/04/26", + "description": "Podchaser podcast database crawler", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Podimo", + "url": "https://podimo.com/", + "instances": [ + "Podimo/1.0 (+https://podimo.com/)" + ], + "addition_date": "2026/04/26", + "description": "Podimo podcast streaming platform bot", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Poggio-Citations", + "url": "https://docs.poggio.io/api/robots", + "instances": [ + "Poggio-Citations 1.0 (+https://docs.poggio.io/api/robots)" + ], + "addition_date": "2026/04/26", + "description": "Poggio's AI sales enablement data collector", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "productsup\\.io\\/crawler", + "url": "https://help.productsup.com/en/29437-29446-import-data-by-crawling-your-website.html", + "instances": [ + "productsup.io/crawler" + ], + "addition_date": "2026/04/26", + "description": "Productsup product feed management crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "qcbot", + "url": "http://quic.cloud/bot.html", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; qcbot/1.0; +http://quic.cloud/bot.html) Chrome/112.0.0.0 Safari/537.36" + ], + "addition_date": "2026/04/26", + "description": "QUIC.cloud optimization service crawler", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Qualys", + "url": "https://knownagents.com/agents/qualys", + "instances": [ + "Mozilla/5.0 (compatible; Qualys; +https://www.qualys.com/)" + ], + "addition_date": "2026/04/26", + "description": "Qualys security scanning", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Quora-Bot", + "url": "http://quora.com/", + "instances": [ + "Quora-Bot/1.0 (http://www.quora.com)" + ], + "addition_date": "2026/04/26", + "description": "Quora link preview crawler", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Qwantbot", + "url": "https://help.qwant.com/", + "instances": [ + "Mozilla/5.0 (compatible; Qwantbot/1.0_800166; +https://help.qwant.com/bot/)", + "Mozilla/5.0 (compatible; Qwantbot/1.0_12345; +https://help.qwant.com/bot/)", + "Mozilla/5.0 (compatible; Qwantbot-news/2.0; +https://help.qwant.com/bot/)" + ], + "addition_date": "2026/04/26", + "description": "Qwant privacy-focused search crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Qwarrybot", + "url": "http://qwarry.com/bot.html", + "instances": [ + "Mozilla/5.0 (compatible; Qwarrybot/2.0; +http://www.qwarry.com/bot.html)" + ], + "addition_date": "2026/04/26", + "description": "Qwarry contextual advertising crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "RSiteAuditor", + "url": "https://dataforseo.com/apis/on-page-api", + "instances": [ + "Mozilla/5.0 (compatible; RSiteAuditor)" + ], + "addition_date": "2026/04/26", + "description": "DataForSEO on-page audit crawler", + "tags": [ + "seo" + ] + }, + { + "pattern": "RSS\\.Social", + "url": "https://rss.social/bot", + "instances": [ + "RSS.Social/1.0 +https://rss.social/bot" + ], + "addition_date": "2026/04/26", + "description": "RSS.Social feed automation bot", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Salesforce\\.com", + "url": "https://knownagents.com/agents/sfdc-callout", + "instances": [ + "Salesforce.com" + ], + "addition_date": "2026/04/26", + "description": "Salesforce HTTP callout user agent", + "tags": [ + "http-library" + ] + }, + { + "pattern": "Scope3", + "url": "https://docs.scope3.com/docs/scope3-crawler", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36 Scope3/2.0 (scope3.com)" + ], + "addition_date": "2026/04/26", + "description": "Scope3 content classification and brand safety", + "tags": [ + "advertising" + ] + }, + { + "pattern": "scraping@nytimes\\.com", + "url": "https://github.com/nytimes", + "instances": [ + "scraping@nytimes.com" + ], + "addition_date": "2026/04/26", + "description": "NYTimes newsroom data collection bot", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Scrubby", + "url": "http://scrubtheweb.com/", + "instances": [ + "Scrubby/2.2 (http://www.scrubtheweb.com/)" + ], + "addition_date": "2026/04/26", + "description": "Scrub the Web business directory crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Scrunchbot", + "url": "https://scrunchai.com/bots", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Scrunchbot/1.0; +https://scrunchai.com/bots)" + ], + "addition_date": "2026/04/26", + "description": "Scrunch AI brand visibility tracker", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "seo4ajax\\.com", + "url": "https://seo4ajax.com/", + "instances": [ + "seo4ajax.com" + ], + "addition_date": "2026/04/26", + "description": "seo4ajax SPA pre-rendering crawler", + "tags": [ + "seo" + ] + }, + { + "pattern": "SequelWP", + "url": "https://sequelwp.com/", + "instances": [ + "Mozilla/5.0 (compatible; SequelWP; +https://sequelwp.com/)" + ], + "addition_date": "2026/04/26", + "description": "SequelWP hosting uptime monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "ServerDensity", + "url": "https://knownagents.com/agents/server-density", + "instances": [ + "Mozilla/5.0 (compatible; ServerDensity; +https://www.serverdensity.com/)" + ], + "addition_date": "2026/04/26", + "description": "Server Density infrastructure monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "ShapBot", + "url": "https://docs.parallel.ai/resources/crawler", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ShapBot/0.1.0" + ], + "addition_date": "2026/04/26", + "description": "Parallel's AI agent web context provider", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "ShortPixel", + "url": "https://shortpixel.com/", + "instances": [ + "ShortPixel/1.0 (+https://shortpixel.com)" + ], + "addition_date": "2026/04/26", + "description": "ShortPixel image optimization service", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Silktide", + "url": "https://silktide.com/", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36 Silktide" + ], + "addition_date": "2026/04/26", + "description": "Silktide website quality monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "SiteLock", + "url": "https://knownagents.com/agents/sitelock", + "instances": [ + "SiteLock (AccMan Connector)/1.0" + ], + "addition_date": "2026/04/26", + "description": "SiteLock security monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "SmarshBot", + "url": "https://smarsh.com/platform/compliance-management/web-archive", + "instances": [ + "SmarshBot/1.0" + ], + "addition_date": "2026/04/26", + "description": "Smarsh's compliance archiving bot", + "tags": [ + "archiver" + ] + }, + { + "pattern": "SMTnetPMBot", + "url": "https://smtnet.com/", + "instances": [ + "SMTnetPMBot/" + ], + "addition_date": "2026/04/26", + "description": "SMTnet electronics manufacturing search crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Software-Security-Research", + "url": "https://reverse-proxies-measurements.softsec.ruhr-uni-bochum.de/", + "instances": [ + "Mozilla/5.0 (compatible; Software-Security-Research; +https://www.ruhr-uni-bochum.de/)" + ], + "addition_date": "2026/04/26", + "description": "Ruhr University CDN security research crawler", + "tags": [ + "scanner", + "academic" + ] + }, + { + "pattern": "SottopopNone", + "url": "https://upcontent.com/robots", + "instances": [ + "SottopopNone/1.0(+https://upcontent.com/robots)" + ], + "addition_date": "2026/04/26", + "description": "UpContent content curation crawler", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Spider[\\s\\S]*spider\\.com", + "url": "https://www.spider.com/solutions/web-crawler", + "instances": [ + "Mozilla/5.0 (compatible; Spider; +https://www.spider.com/)" + ], + "addition_date": "2026/04/26", + "description": "AI project web data crawler", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "Splunk", + "url": "https://knownagents.com/agents/splunk", + "instances": [ + "Mozilla/5.0 (compatible; Splunk/1.0.0; https://splunk.com)" + ], + "addition_date": "2026/04/26", + "description": "Splunk monitoring and analytics", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "StatusNestBacklinkSpider", + "url": "https://statusnest.com/bot", + "instances": [ + "Mozilla/5.0 (compatible; StatusNestBacklinkSpider; +https://statusnest.com/)" + ], + "addition_date": "2026/04/26", + "description": "StatusNest backlink tracking crawler", + "tags": [ + "seo" + ] + }, + { + "pattern": "stepstoneCrawlBot", + "url": "https://thestepstonegroup.com/crawler/", + "instances": [ + "Mozilla/5.0 (compatible; stepstoneCrawlBot; +https://www.thestepstonegroup.com/crawler/)" + ], + "addition_date": "2026/04/26", + "description": "Stepstone Group job search crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "TavilyBot", + "url": "https://knownagents.com/agents/tavilybot", + "instances": [ + "Mozilla/5.0 (compatible; TavilyBot; +https://tavily.com/)" + ], + "addition_date": "2026/04/26", + "description": "Tavily's real-time AI agent data crawler", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "ThousandEyes", + "url": "https://knownagents.com/agents/thousand-eyes-cloud-agent", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36 (ThousandEyes Agent)" + ], + "addition_date": "2026/04/26", + "description": "ThousandEyes network monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Trae\\/", + "url": "https://trae.ai/", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Trae/1.107.1 Chrome/142.0.7444.235 Electron/39.2.7 Safari/537.36" + ], + "addition_date": "2026/04/26", + "description": "ByteDance's AI coding agent", + "tags": [ + "browser-automation" + ] + }, + { + "pattern": "TwinAgent", + "url": "https://twin.so/", + "instances": [ + "Mozilla/5.0 (compatible; TwinAgent; +https://www.twinagent.com/)" + ], + "addition_date": "2026/04/26", + "description": "Twin's automated worker for workflow execution", + "tags": [ + "browser-automation" + ] + }, + { + "pattern": "uipbot", + "url": "https://knownagents.com/agents/uipbot", + "instances": [ + "uipbot/1.0 (uipbot@semasio.com)" + ], + "addition_date": "2026/04/26", + "description": "Semasio semantic targeting crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "um-FC", + "url": "https://ubermetrics-technologies.com/", + "instances": [ + "Mozilla/5.0 (compatible; um-FC/1.0; https://www.ubermetrics-technologies.com/; Windows NT 6.1; WOW64; rv:125.0) Gecko/20100101 Firefox/125.1" + ], + "addition_date": "2026/04/26", + "description": "Ubermetrics media monitoring crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "um-IC", + "url": "https://ubermetrics-technologies.com/", + "instances": [ + "Mozilla/5.0 (compatible; um-IC/1.0; https://www.ubermetrics-technologies.com/; Windows NT 6.1; WOW64; rv:125.0) Gecko/20100101 Firefox/125.1" + ], + "addition_date": "2026/04/26", + "description": "Ubermetrics communications intelligence crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "UptimeStatistics", + "url": "https://knownagents.com/agents/uptimestatistics", + "instances": [ + "Mozilla/5.0 (compatible; UptimeStatistics; +https://www.uptimestatistics.com/)" + ], + "addition_date": "2026/04/26", + "description": "UptimeStatistics monitoring service", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Verispider", + "url": "http://projecthoneypot.org/", + "instances": [ + "Mozilla/5.0 (compatible; Verispider; +https://www.projecthoneypot.org/)" + ], + "addition_date": "2026/04/26", + "description": "Verispider spam and security scanner", + "tags": [ + "scanner" + ] + }, + { + "pattern": "visionheight\\.com\\/scan", + "url": "https://knownagents.com/agents/visionheight-comscan", + "instances": [ + "visionheight.com/scan Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/126.0.0.0 Safari/537.36" + ], + "addition_date": "2026/04/26", + "description": "Vision Height security scanner", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Watchbot monitoring robot", + "url": "https://watchbot.fflow.net/", + "instances": [ + "Watchbot monitoring robot (https://watchbot.fflow.net)" + ], + "addition_date": "2026/04/26", + "description": "Watchbot SSL and uptime monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Watchful", + "url": "https://knownagents.com/agents/watchful", + "instances": [ + "Mozilla/5.0 (compatible; Watchful; +https://www.watchful.net/)" + ], + "addition_date": "2026/04/26", + "description": "Watchful website monitoring", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "weborama-fetcher", + "url": "http://weborama.com/", + "instances": [ + "weborama-fetcher (+http://www.weborama.com)" + ], + "addition_date": "2026/04/26", + "description": "Weborama advertising optimization crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "webspidermount", + "url": "https://webspidermount.com/features/", + "instances": [ + "Mozilla/5.0 (compatible; webspidermount; +https://www.webspidermount.com/)" + ], + "addition_date": "2026/04/26", + "description": "Webspidermount job listing extraction service", + "tags": [ + "advertising" + ] + }, + { + "pattern": "WepchSearchEngine", + "url": "https://wepch.com/search-engine", + "instances": [ + "Mozilla/5.0 (compatible; WepchSearchEngine; +https://www.wepch.com/)" + ], + "addition_date": "2026/04/26", + "description": "Wepch search engine development crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "wknd-bot", + "url": "https://developer.wunderkind.co/docs/server-side-tracking-implementation", + "instances": [ + "wknd-bot" + ], + "addition_date": "2026/04/26", + "description": "Wunderkind marketing automation tracker", + "tags": [ + "advertising" + ] + }, + { + "pattern": "WPMU DEV Hub", + "url": "https://wpmudev.com/", + "instances": [ + "Mozilla/5.0 (compatible; WPMU DEV Hub/2.0; +https://wpmudev.com)" + ], + "addition_date": "2026/04/26", + "description": "WPMU DEV WordPress management bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "WTotem", + "url": "https://knownagents.com/agents/wtotem", + "instances": [ + "Mozilla/5.0 (compatible; WTotem; +https://www.webtotem.com/)" + ], + "addition_date": "2026/04/26", + "description": "WTotem malware detection scanner", + "tags": [ + "scanner" + ] + }, + { + "pattern": "XoviOnpageCrawler", + "url": "http://xovi.de/", + "instances": [ + "Mozilla/5.0 (compatible; XoviOnpageCrawler; +http://www.xovi.de/)" + ], + "addition_date": "2026/04/26", + "description": "XOVI SEO suite on-page crawler", + "tags": [ + "seo" + ] + }, + { + "pattern": "yelpspider", + "url": "https://yelp.com/", + "instances": [ + "Mozilla/5.0 compatible; yelpspider/yelpspider-1.0 (Crawlerbot run by Yelp Inc; yelpbot at yelp dot com)" + ], + "addition_date": "2026/04/26", + "description": "Yelp business information crawler", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "ZanistaBot", + "url": "https://zanista.ai/crawler-info", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ZanistaBot/1.0; +https://zanista.ai/crawler-info) Chrome/W.X.Y.Z Safari/537.36" + ], + "addition_date": "2026/04/26", + "description": "Zanista's AI search crawler", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "ZoomInfo-", + "url": "https://knownagents.com/agents/zoominfo", + "instances": [ + "Mozilla/5.0 (compatible; ZoomInfo-DataEnrichment/1.0; +https://www.zoominfo.com)" + ], + "addition_date": "2026/04/26", + "description": "ZoomInfo business intelligence search crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "7Siters", + "url": "https://7ooo.ru/siters/", + "instances": [ + "7Siters/1.2 ( https://7ooo.ru/siters/ )" + ], + "addition_date": "2026/04/26", + "description": "Russian web crawler for site indexing and analysis", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Accessible Web Bot", + "url": "https://accessibleweb.com/bot/", + "instances": [ + "Accessible Web Bot" + ], + "addition_date": "2026/04/26", + "description": "Monitors websites for WCAG accessibility compliance violations", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "AtVowBot", + "url": "https://brandeem.com/", + "instances": [ + "AtVowBot" + ], + "addition_date": "2026/04/26", + "description": "Brand monitoring and reputation management web crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Bibliotheque Nacional de France Crawler", + "url": "https://www.bnf.fr/en/web-legal-deposit", + "instances": [ + "Bibliotheque Nacional de France Crawler" + ], + "addition_date": "2026/04/26", + "description": "French National Library web archiving and preservation bot", + "tags": [ + "archiver" + ] + }, + { + "pattern": "Bling ERP", + "url": "https://www.bling.com.br/", + "instances": [ + "Bling ERP" + ], + "addition_date": "2026/04/26", + "description": "Brazilian ERP system data synchronization and integration crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "CDSCbot", + "url": "https://wiki.communitydata.science/CommunityData:Fediverse_research", + "instances": [ + "CDSCbot/2024.08.08 https://wiki.communitydata.science/CommunityData:Fediverse_research" + ], + "addition_date": "2026/04/26", + "description": "Academic research crawler for Fediverse decentralized social networks", + "tags": [ + "academic" + ] + }, + { + "pattern": "Critical CSS Bot", + "url": "https://criticalcss.com/", + "instances": [ + "Critical CSS Bot" + ], + "addition_date": "2026/04/26", + "description": "Extracts critical CSS for web performance optimization tools", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "CybaaBot", + "url": "https://cybaa.io/bot-policy", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36 CybaaBot" + ], + "addition_date": "2026/04/26", + "description": "Cybaa advertising and marketing analytics web crawler bot", + "tags": [ + "advertising" + ] + }, + { + "pattern": "CyberFindCrawler", + "url": "https://cyberfind.net/bot.html", + "instances": [ + "CyberFindCrawler" + ], + "addition_date": "2026/04/26", + "description": "Web search engine crawler for content discovery indexing", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Dark Visitor", + "url": "https://darkvisitors.com/", + "instances": [ + "Mozilla/5.0 (compatible; Dark Visitor/1.0; +https://darkvisitors.com)" + ], + "addition_date": "2026/04/26", + "description": "Bot analytics and AI agent tracking service crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Determ", + "url": "https://www.determ.com/", + "instances": [ + "Determ" + ], + "addition_date": "2026/04/26", + "description": "Media monitoring and brand reputation tracking web crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "DNSScanner", + "url": "https://rapef.info/_contacts/", + "instances": [ + "DNSScanner" + ], + "addition_date": "2026/04/26", + "description": "DNS and network infrastructure security scanning crawler bot", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Drupalbot", + "url": "https://www.drupal.org/", + "instances": [ + "Mozilla/5.0 (compatible; Drupalbot; +https://www.drupal.org)" + ], + "addition_date": "2026/04/26", + "description": "Drupal CMS community website crawler and content indexer", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "eMoney Advisor", + "url": "https://emoneyadvisor.com/", + "instances": [ + "eMoney Advisor" + ], + "addition_date": "2026/04/26", + "description": "Financial planning software data integration and sync crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "everyfeed-spider", + "url": "http://everyfeed.com/", + "instances": [ + "everyfeed-spider/2.0 (http://www.everyfeed.com)" + ], + "addition_date": "2026/04/26", + "description": "RSS and Atom feed aggregation and discovery crawler", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "ExteContextCrawl", + "url": "http://crawl001.exte.ai/", + "instances": [ + "Mozilla/5.0 (compatible; ExteContextCrawl/1.0; +http://crawl001.exte.ai)" + ], + "addition_date": "2026/04/26", + "description": "AI context extraction and content understanding web crawler", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "FediDB", + "url": "https://fedidb.org/crawler.html", + "instances": [ + "FediDB/0.5.0; +https://fedidb.org/crawler.html" + ], + "addition_date": "2026/04/26", + "description": "Fediverse instance database and statistics collection crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "FediIndex", + "url": "https://fedi.wrm.sr/about", + "instances": [ + "FediIndex/1.0 (+https://fedi.wrm.sr/about)" + ], + "addition_date": "2026/04/26", + "description": "Fediverse server indexing and discovery service web crawler", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "FediList Agent", + "url": "https://fedilist.com/", + "instances": [ + "FediList Agent/3 (https://fedilist.com/)" + ], + "addition_date": "2026/04/26", + "description": "Fediverse instance directory and listing service crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Fedineko", + "url": "https://fedineko.org/about", + "instances": [ + "Fedineko (crabo/0.3.1; +https://fedineko.org/about)" + ], + "addition_date": "2026/04/26", + "description": "Fediverse network mapping and analysis research crawler tool", + "tags": [ + "academic" + ] + }, + { + "pattern": "FedReporter Bot for FFIEC", + "url": "https://www.fedreporter.com/", + "instances": [ + "FedReporter Bot for FFIEC" + ], + "addition_date": "2026/04/26", + "description": "Federal financial institution regulatory compliance data crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Feedsearch Bot", + "url": "https://feedearch.dev/", + "instances": [ + "Mozilla/5.0 (Compatible; Feedsearch Bot; +https://feedearch.dev)" + ], + "addition_date": "2026/04/26", + "description": "RSS and feed discovery search engine crawler bot", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Feedsearch-Crawler", + "url": "https://pypi.org/project/feedsearch-crawler", + "instances": [ + "Mozilla/5.0 (compatible; Feedsearch-Crawler; +https://pypi.org/project/feedsearch-crawler)" + ], + "addition_date": "2026/04/26", + "description": "Python library for RSS feed detection and extraction", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "fiperbot", + "url": "https://fiper.net/", + "instances": [ + "Mozilla/5.0 (compatible; fiperbot/0.1 +https://www.fiper.net/bot.html)" + ], + "addition_date": "2026/04/26", + "description": "Web content indexing and search engine crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "FleebsBot", + "url": "https://fleebs.com/bot", + "instances": [ + "Mozilla/5.0 (compatible; FleebsBot/1~w; +https://fleebs.com/bot)" + ], + "addition_date": "2026/04/26", + "description": "Web content aggregation and discovery service crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Fluid", + "url": "http://leak.info/bot.html", + "instances": [ + "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/528.16 (KHTML, like Gecko) Fluid/0.9.6 Safari/528.16" + ], + "addition_date": "2026/04/26", + "description": "Site-specific browser application for web content access", + "tags": [ + "browser-automation" + ] + }, + { + "pattern": "Flyriverbot", + "url": "https://flyriver.com/crawler", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Flyriverbot/1.1 (+https://www.flyriver.com/; AI Content Source Check)" + ], + "addition_date": "2026/04/26", + "description": "AI content source verification and attribution checking crawler", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "Freshbot", + "url": "http://webagent.wise-guys.nl/", + "instances": [ + "Mozilla/5.0 (X11; Linux i686; rv:6.0) Gecko/20100101 Firefox/6.0 (Freshbot/1.0/EU; http://webagent.wise-guys.nl/)" + ], + "addition_date": "2026/04/26", + "description": "European web content monitoring and freshness checking crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Gaisbot", + "url": "http://gais.cs.ccu.edu.tw/robot.php", + "instances": [ + "Gaisbot/3.0 (robot@gais.cs.ccu.edu.tw; http://gais.cs.ccu.edu.tw/robot.php)" + ], + "addition_date": "2026/04/26", + "description": "Academic research web crawler from Taiwan university project", + "tags": [ + "academic" + ] + }, + { + "pattern": "GenomeCrawlerd", + "url": "https://nokia.com/genomecrawler", + "instances": [ + "Mozilla/5.0 (compatible; GenomeCrawlerd/1.0; +https://www.nokia.com/genomecrawler)" + ], + "addition_date": "2026/04/26", + "description": "Nokia web content analysis and data collection crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "HaloBot", + "url": "https://haloscan.com/", + "instances": [ + "Mozilla/5.0 (compatible; HaloBot/1.0)" + ], + "addition_date": "2026/04/26", + "description": "Haloscan comment system integration and content fetching bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "IRLbot", + "url": "http://irl.cs.tamu.edu/crawler", + "instances": [ + "IRLbot/3.0 (compatible; MSIE 6.0; http://irl.cs.tamu.edu/crawler)" + ], + "addition_date": "2026/04/26", + "description": "Texas A&M University academic research web crawler bot", + "tags": [ + "academic" + ] + }, + { + "pattern": "kaikki\\.org-digital-archive", + "url": "https://kaikki.org/", + "instances": [ + "kaikki.org-digital-archive" + ], + "addition_date": "2026/04/26", + "description": "Digital archiving and preservation service web crawler bot", + "tags": [ + "archiver" + ] + }, + { + "pattern": "kb\\.dk_bot", + "url": "https://www.kb.dk/en/", + "instances": [ + "kb.dk_bot" + ], + "addition_date": "2026/04/26", + "description": "Royal Danish Library web archiving and preservation crawler", + "tags": [ + "archiver" + ] + }, + { + "pattern": "Library Of Congress Web Archiving", + "url": "https://www.loc.gov/programs/web-archiving/", + "instances": [ + "Library Of Congress Web Archiving" + ], + "addition_date": "2026/04/26", + "description": "US Library of Congress web preservation archiving crawler", + "tags": [ + "archiver" + ] + }, + { + "pattern": "MagnetmeBot", + "url": "https://magnet.me/", + "instances": [ + "MagnetmeBot" + ], + "addition_date": "2026/04/26", + "description": "Job board and recruitment platform web scraping crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "MatchorySearch", + "url": "https://matchory.com/", + "instances": [ + "Mozilla/5.0 (compatible; MatchorySearch/1.3; +https://www.matchory.com)" + ], + "addition_date": "2026/04/26", + "description": "Search engine and content discovery indexing crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Minoru's Fediverse Crawler", + "url": "https://nodes.fediverse.party/", + "instances": [ + "Minoru's Fediverse Crawler (+https://nodes.fediverse.party)" + ], + "addition_date": "2026/04/26", + "description": "Fediverse network node discovery and mapping crawler tool", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "MirrorWebCrawler", + "url": "https://www.mirrorweb.com/", + "instances": [ + "MirrorWebCrawler" + ], + "addition_date": "2026/04/26", + "description": "Enterprise web archiving and compliance preservation crawler bot", + "tags": [ + "archiver" + ] + }, + { + "pattern": "mithril-crawler", + "url": "https://498-search-engine.github.io/website/", + "instances": [ + "mithril-crawler" + ], + "addition_date": "2026/04/26", + "description": "Academic search engine project web indexing crawler bot", + "tags": [ + "academic" + ] + }, + { + "pattern": "ModatScanner", + "url": "https://modat.io/", + "instances": [ + "Mozilla/5.0 (compatible; ModatScanner/1.2; +https://modat.io/)" + ], + "addition_date": "2026/04/26", + "description": "Website monitoring and change detection service crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "NapBot", + "url": "http://napbot.com/", + "instances": [ + "NapBot" + ], + "addition_date": "2026/04/26", + "description": "Web content indexing and search engine crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "New York Times Newsgathering", + "url": "https://www.nytimes.com/", + "instances": [ + "New York Times Newsgathering" + ], + "addition_date": "2026/04/26", + "description": "News media content aggregation and research crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "NLUX_IAHarvester", + "url": "http://crawl.bnl.lu/", + "instances": [ + "Mozilla/5.0 (compatible; NLUX_IAHarvester/3.4.0; +http://crawl.bnl.lu/)" + ], + "addition_date": "2026/04/26", + "description": "Luxembourg National Library web archiving and harvesting crawler", + "tags": [ + "archiver" + ] + }, + { + "pattern": "NoahBot", + "url": "https://noahwire.com/bot-info", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; NoahBot/1.0; +https://noahwire.com/bot-info)" + ], + "addition_date": "2026/04/26", + "description": "News aggregation and content discovery service crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "PlagAwareBot", + "url": "https://plagaware.com/bot", + "instances": [ + "Mozilla/5.0 (compatible; PlagAwareBot/3.2; +https://www.plagaware.com/bot)" + ], + "addition_date": "2026/04/26", + "description": "Plagiarism detection and content originality checking crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Rakuten Image extraction bot", + "url": "https://www.rakuten.com/", + "instances": [ + "Rakuten Image extraction bot" + ], + "addition_date": "2026/04/26", + "description": "E-commerce product image extraction and indexing crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "ResearchBot", + "url": "https://kaust.edu.sa/bot", + "instances": [ + "StateOfPlay-ResearchBot/1.0" + ], + "addition_date": "2026/04/26", + "description": "Academic research data collection and analysis crawler bot", + "tags": [ + "academic" + ] + }, + { + "pattern": "rss-is-dead\\.lol web bot", + "url": "https://rss-is-dead.lol/", + "instances": [ + "Mozilla/5.0 (compatible; rss-is-dead.lol web bot; +https://rss-is-dead.lol)" + ], + "addition_date": "2026/04/26", + "description": "RSS feed monitoring and availability checking service crawler", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "seoLyt", + "url": "https://seolyt.com/", + "instances": [ + "Mozilla/5.0 (compatible; seoLyt/1.0; +https://seolyt.com)" + ], + "addition_date": "2026/04/26", + "description": "SEO analysis and website optimization tool crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "SirdataBot", + "url": "https://semantic-api.docs.sirdata.net/contextual-api/contextual-api/introduction", + "instances": [ + "SirdataBot (+https://semantic-api.docs.sirdata.net/contextual-api/contextual-api/introduction)" + ], + "addition_date": "2026/04/26", + "description": "Contextual advertising and semantic content analysis crawler bot", + "tags": [ + "advertising" + ] + }, + { + "pattern": "SitesOverPagesBot", + "url": "https://sitesoverpages.com/bot", + "instances": [ + "SitesOverPagesBot" + ], + "addition_date": "2026/04/26", + "description": "Website structure analysis and sitemap generation crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "SleepBot", + "url": "http://sleepbot.com/", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; SleepBot/1.0; +http://sleepbot.com/) Chrome/131.0.0.0 Safari/537.36" + ], + "addition_date": "2026/04/26", + "description": "Web content monitoring and tracking service crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Sosospider", + "url": "http://help.soso.com/webspider.htm", + "instances": [ + "Sosospider" + ], + "addition_date": "2026/04/26", + "description": "Chinese search engine web indexing and crawling bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Termly", + "url": "https://termly.io/", + "instances": [ + "Termly" + ], + "addition_date": "2026/04/26", + "description": "Privacy policy and compliance scanning service crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "TLS tester", + "url": "https://testssl.sh/dev/", + "instances": [ + "TLS tester from https://testssl.sh/" + ], + "addition_date": "2026/04/26", + "description": "SSL/TLS security testing and vulnerability scanning crawler bot", + "tags": [ + "scanner" + ] + }, + { + "pattern": "trafilatura", + "url": "https://github.com/adbar/trafilatura", + "instances": [ + "trafilatura/2.0.0 (+https://github.com/adbar/trafilatura)" + ], + "addition_date": "2026/04/26", + "description": "Python library for web content extraction and scraping", + "tags": [ + "http-library" + ] + }, + { + "pattern": "UrlBeeBot", + "url": "https://urlbee.com/", + "instances": [ + "UrlBeeBot" + ], + "addition_date": "2026/04/26", + "description": "URL analysis and threat detection security crawler bot", + "tags": [ + "scanner" + ] + }, + { + "pattern": "videootv Bot", + "url": "https://www.digitalgreen.org/", + "instances": [ + "videootv Bot" + ], + "addition_date": "2026/04/26", + "description": "Video content aggregation and educational media crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "vmcrawl", + "url": "https://docs.vmst.io/vmcrawl", + "instances": [ + "vmcrawl/0.17.7 (https://vmcrawl.com)", + "vmcrawl/0.x (https://docs.vmst.io/vmcrawl)" + ], + "addition_date": "2026/04/26", + "description": "Fediverse instance monitoring and discovery service crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "WadooBot", + "url": "https://wadoo.net/wadoobot/", + "instances": [ + "WadooBot" + ], + "addition_date": "2026/04/26", + "description": "Web content indexing and search engine crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Website-info\\.net-Robot", + "url": "https://website-info.net/robot", + "instances": [ + "Mozilla/5.0 (compatible; Website-info.net-Robot; https://website-info.net/robot)" + ], + "addition_date": "2026/04/26", + "description": "Website information and analytics data collection crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "WebZIP", + "url": "http://spidersoft.com/", + "instances": [ + "WebZIP/3.5 (http://www.spidersoft.com)" + ], + "addition_date": "2026/04/26", + "description": "Website downloading and offline browsing tool crawler bot", + "tags": [ + "archiver" + ] + }, + { + "pattern": "WikiDo", + "url": "http://wikido.com/", + "instances": [ + "WikiDo/1.1 (http://wikido.com; crawler@wikido.com)" + ], + "addition_date": "2026/04/26", + "description": "Wiki content aggregation and search engine crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "WOVN Crawler", + "url": "https://wovn.io/", + "instances": [ + "WOVN Crawler" + ], + "addition_date": "2026/04/26", + "description": "Website translation and localization service crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "YoudaoBot", + "url": "http://youdao.com/help/webmaster/spider/", + "instances": [ + "YoudaoBot" + ], + "addition_date": "2026/04/26", + "description": "Chinese search engine and translation service crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "ZyBorg", + "url": "http://wisenutbot.com/", + "instances": [ + "Mozilla/4.0 compatible ZyBorg/1.0 (wn-14.zyborg@looksmart.net; http://www.WISEnutbot.com)" + ], + "addition_date": "2026/04/26", + "description": "WISEnut search engine web indexing and crawling bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Aranet-SearchBot", + "url": "https://aranet.ai/bot", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Aranet-SearchBot/1.0; +https://aranet.ai/bot) Chrome/131.0.0.0 Safari/537.36" + ], + "addition_date": "2026/04/26", + "description": "AI-powered search engine and content indexing crawler bot", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "crawl4ai", + "url": "https://github.com/unclecode/crawl4ai", + "instances": [ + "crawl4ai-adapter/1.0" + ], + "addition_date": "2026/04/26", + "description": "Open-source LLM-friendly web scraping and extraction library", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "DeepSeekBot", + "url": "http://deepseek.com/bot", + "instances": [ + "Mozilla/5.0 (compatible; DeepSeekBot/1.0; +https://www.deepseek.com/bot)" + ], + "addition_date": "2026/04/26", + "description": "DeepSeek AI model training and data collection crawler", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "iaskspider", + "url": "https://www.iask.com/", + "instances": [ + "iaskspider/2.0" + ], + "addition_date": "2026/04/26", + "description": "Chinese AI search engine and question answering crawler", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "KunatoCrawler", + "url": "http://kunato.ai/bot.html", + "instances": [ + "Mozilla/5.0 (compatible; KunatoCrawler/1.0; +http://kunato.ai/bot.html)" + ], + "addition_date": "2026/04/26", + "description": "AI training data collection and web scraping crawler", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "TerraCotta", + "url": "https://github.com/CeramicTeam/CeramicTerracotta", + "instances": [ + "TerraCotta https://github.com/CeramicTeam/CeramicTerracotta" + ], + "addition_date": "2026/04/26", + "description": "Ceramic Network decentralized data indexing and crawling bot", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "ABEvalBot", + "url": "https://knownagents.com/agents/abevalbot", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ABEvalBot/0.1) Version/11.1.2 Safari/605.1.15" + ], + "addition_date": "2026/04/26", + "description": "A/B testing evaluation and website analysis crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "blekkobot", + "url": "https://knownagents.com/agents/blekkobot", + "instances": [ + "blekkobot" + ], + "addition_date": "2026/04/26", + "description": "Blekko search engine web indexing and crawling bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "br-crawler", + "url": "https://knownagents.com/agents/br-crawler", + "instances": [ + "br-crawler/0.5" + ], + "addition_date": "2026/04/26", + "description": "Brazilian web content indexing and discovery crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "BuddyBot", + "url": "https://knownagents.com/agents/buddybot", + "instances": [ + "BuddyBot" + ], + "addition_date": "2026/04/26", + "description": "Social networking and content aggregation service crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "CapterraBot", + "url": "https://www.capterra.com", + "instances": [ + "Mozilla/5.0 (compatible; CapterraBot/1.0; +https://www.capterra.com)" + ], + "addition_date": "2026/04/26", + "description": "Capterra software review platform data collection crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "carbon-umbrella-bot", + "url": "https://knownagents.com/agents/carbon-umbrella-bot", + "instances": [ + "carbon-umbrella-bot/1.0" + ], + "addition_date": "2026/04/26", + "description": "Carbon footprint tracking and environmental monitoring crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "caveman-hunter", + "url": "https://fedi.buzz/", + "instances": [ + "caveman-hunter/0.0.0 (+https://fedi.buzz/)" + ], + "addition_date": "2026/04/26", + "description": "Fediverse content discovery and social network crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Centro Ads\\.txt Crawler", + "url": "https://knownagents.com/agents/centro-ads-txt-crawler", + "instances": [ + "Centro Ads.txt Crawler/1.0" + ], + "addition_date": "2026/04/26", + "description": "Advertising transparency ads.txt file verification crawler bot", + "tags": [ + "advertising" + ] + }, + { + "pattern": "WISEbot", + "url": "http://www.cision.com", + "instances": [ + "Mozilla/5.0 (compatible; WISEbot/1.0; +http://www.cision.com)" + ], + "addition_date": "2026/04/26", + "description": "Cision media monitoring and PR analytics crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "CodaBot", + "url": "https://coda.io/", + "instances": [ + "CodaBot/1.0" + ], + "addition_date": "2026/04/26", + "description": "Coda document collaboration platform link preview crawler bot", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Corporama matcher", + "url": "https://corporama.fr/", + "instances": [ + "Corporama matcher 1.2" + ], + "addition_date": "2026/04/26", + "description": "Corporate data matching and business intelligence crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "CyotekWebCopy", + "url": "https://www.cyotek.com/cyotek-webcopy", + "instances": [ + "CyotekWebCopy/1.9 CyotekHTTP/6.4" + ], + "addition_date": "2026/04/26", + "description": "Website copying and offline browsing tool crawler bot", + "tags": [ + "archiver" + ] + }, + { + "pattern": "Datadog Agent", + "url": "https://www.datadoghq.com/", + "instances": [ + "Datadog Agent/7.69.2" + ], + "addition_date": "2026/04/26", + "description": "Datadog infrastructure monitoring and synthetic testing agent bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Dazzle BlueSky Bot", + "url": "https://knownagents.com/agents/dazzle-bluesky-bot", + "instances": [ + "Dazzle BlueSky Bot/1.1" + ], + "addition_date": "2026/04/26", + "description": "BlueSky social network link preview and content crawler", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "DominicBot", + "url": "https://vanylla.org/bot", + "instances": [ + "DominicBot/1.0 (+https://vanylla.org/bot)" + ], + "addition_date": "2026/04/26", + "description": "Vanylla social network content indexing and discovery crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Dow Jones Searchbot", + "url": "https://www.dowjones.com/", + "instances": [ + "Mozilla/5.0 (compatible; Dow Jones Searchbot)" + ], + "addition_date": "2026/04/26", + "description": "Dow Jones news and financial content aggregation crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Download Ninja", + "url": "https://knownagents.com/agents/download-ninja", + "instances": [ + "Download Ninja/4.0" + ], + "addition_date": "2026/04/26", + "description": "File download manager and web content fetching tool", + "tags": [ + "http-library" + ] + }, + { + "pattern": "EmailWolf", + "url": "https://knownagents.com/agents/emailwolf", + "instances": [ + "EmailWolf 1.00" + ], + "addition_date": "2026/04/26", + "description": "Email address extraction and contact scraping crawler bot", + "tags": [ + "scanner" + ] + }, + { + "pattern": "fedistatsCrawler", + "url": "https://knownagents.com/agents/fedistatscrawler", + "instances": [ + "fedistatsCrawler/1.0" + ], + "addition_date": "2026/04/26", + "description": "Fediverse statistics collection and network analysis crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "GoParserBot", + "url": "https://knownagents.com/agents/goparserbot", + "instances": [ + "Mozilla/5.0 (compatible; GoParserBot/1.0)" + ], + "addition_date": "2026/04/26", + "description": "Go-based web content parsing and extraction crawler bot", + "tags": [ + "http-library" + ] + }, + { + "pattern": "gsa-crawler", + "url": "https://knownagents.com/agents/gsa-crawler", + "instances": [ + "gsa-crawler (Enterprise; T3-LNMUGC6JBKQNE; thomas@brainfuck.space)" + ], + "addition_date": "2026/04/26", + "description": "Google Search Appliance enterprise search indexing crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "HanaleiBot", + "url": "https://knownagents.com/agents/hanaleibot", + "instances": [ + "HanaleiBot runid=beta-stage-integration-test" + ], + "addition_date": "2026/04/26", + "description": "Testing and integration verification web crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "NicheIndex", + "url": "https://nicheindex.co", + "instances": [ + "NicheIndex/1.0 RSS Harvest (+https://nicheindex.co)" + ], + "addition_date": "2026/04/26", + "description": "Niche market RSS feed harvesting and indexing crawler", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "HeadOnlyScraper", + "url": "https://knownagents.com/agents/headonlyscraper", + "instances": [ + "Mozilla/5.0 (HeadOnlyScraper)", + "Mozilla/5.0 (HeadOnlyScraperV2)" + ], + "addition_date": "2026/04/26", + "description": "HTTP HEAD request metadata collection and validation crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "HenkBot", + "url": "https://valyu.ai/crawler", + "instances": [ + "Mozilla/5.0 (compatible; HenkBot/1.0; +https://valyu.ai/crawler)" + ], + "addition_date": "2026/04/26", + "description": "Valyu AI content analysis and data collection crawler", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "Impact\\.com Agent", + "url": "https://impact.com/", + "instances": [ + "Mozilla/5.0 (compatible;Impact.com Agent) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36" + ], + "addition_date": "2026/04/26", + "description": "Impact.com affiliate marketing and partnership tracking crawler bot", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Keydrop\\.io", + "url": "https://onlyscans.com/about", + "instances": [ + "Mozilla/5.0; Keydrop.io/1.0(onlyscans.com/about);" + ], + "addition_date": "2026/04/26", + "description": "Keydrop gaming marketplace security scanning and monitoring crawler", + "tags": [ + "scanner" + ] + }, + { + "pattern": "larbin", + "url": "http://larbin.sourceforge.net/", + "instances": [ + "larbin_2.6.2 (larbin@correa.org)" + ], + "addition_date": "2026/04/26", + "description": "Open-source web crawler for large-scale indexing projects", + "tags": [ + "http-library" + ] + }, + { + "pattern": "SENTINEL-LinkCheck", + "url": "https://sentinel.oblivionzone.com/bot", + "instances": [ + "SENTINEL-LinkCheck/1.0 (+https://sentinel.oblivionzone.com/bot)" + ], + "addition_date": "2026/04/26", + "description": "Sentinel link validation and broken link detection crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "linko", + "url": "https://linko.app/crawler", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36 https://linko.app/crawler" + ], + "addition_date": "2026/04/26", + "description": "Linko link management and URL tracking service crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "LinkpadBot", + "url": "https://linkpad.org/robot/", + "instances": [ + "Mozilla/5.0 (compatible; LinkpadBot/2.4; +https://linkpad.org/robot/)" + ], + "addition_date": "2026/04/26", + "description": "Linkpad bookmark management and link discovery crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "lwp-trivial", + "url": "https://metacpan.org/pod/LWP", + "instances": [ + "lwp-trivial/1.35" + ], + "addition_date": "2026/04/26", + "description": "Perl LWP library simple HTTP client and fetcher", + "tags": [ + "http-library" + ] + }, + { + "pattern": "Magus Bot", + "url": "https://knownagents.com/agents/magus-bot", + "instances": [ + "Magus Bot 1.0" + ], + "addition_date": "2026/04/26", + "description": "Web content analysis and data extraction crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "NaverBot", + "url": "https://www.naver.com/", + "instances": [ + "Mozilla/5.0 (compatible; NaverBot/1.0; nhnbot@naver.com)" + ], + "addition_date": "2026/04/26", + "description": "Naver Korean search engine web indexing crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "loopimprovements\\.com", + "url": "http://loopimprovements.com/robot.html", + "instances": [ + "NetResearchServer/4.0(loopimprovements.com/robot.html)" + ], + "addition_date": "2026/04/26", + "description": "Network research and web data collection crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "OpenTheBoxBot", + "url": "https://knownagents.com/agents/opentheboxbot", + "instances": [ + "OpenTheBoxBot/1.0" + ], + "addition_date": "2026/04/26", + "description": "Content discovery and web indexing service crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "OWLer-W", + "url": "https://openwebsearch.eu/", + "instances": [ + "OWLer-W (owler@ows.eu)" + ], + "addition_date": "2026/04/26", + "description": "Business intelligence and company data collection crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "peer39_crawler", + "url": "https://www.peer39.com/", + "instances": [ + "peer39_crawler/1.0" + ], + "addition_date": "2026/04/26", + "description": "Peer39 contextual advertising and brand safety crawler bot", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Pixalate\\.com", + "url": "https://www.pixalate.com/", + "instances": [ + "Pixalate.com/1.0" + ], + "addition_date": "2026/04/26", + "description": "Pixalate ad fraud detection and brand safety crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Poduptime", + "url": "https://fediverse.observer", + "instances": [ + "Mozilla/5.0 (compatible; Poduptime; +fediverse.observer)" + ], + "addition_date": "2026/04/26", + "description": "Fediverse pod uptime monitoring and availability checker crawler", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Pomothy-Bot", + "url": "https://knownagents.com/agents/pomothy-bot", + "instances": [ + "Mozilla/5.0 (compatible; Pomothy-Bot/1.0)" + ], + "addition_date": "2026/04/26", + "description": "Web content monitoring and change detection crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "PulsePoint-Crawler", + "url": "https://www.pulsepoint.com/", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko; compatible; PulsePoint-Crawler) Chrome/120.0.0.0 Safari/537.36" + ], + "addition_date": "2026/04/26", + "description": "PulsePoint programmatic advertising and ad verification crawler bot", + "tags": [ + "advertising" + ] + }, + { + "pattern": "rawweb-bot", + "url": "https://knownagents.com/agents/rawweb-bot", + "instances": [ + "rawweb-bot/1.0" + ], + "addition_date": "2026/04/26", + "description": "Raw web content extraction and data scraping crawler", + "tags": [ + "http-library" + ] + }, + { + "pattern": "semantic-visions", + "url": "https://semantic-visions.com/", + "instances": [ + "Mozilla/5.0 (Linux; CentOS; compatible; semantic-visions-discovery; HTTPClient 4.5)", + "Mozilla/5.0 (X11; compatible; semantic-visions.com crawler; HTTPClient 4.5)" + ], + "addition_date": "2026/04/26", + "description": "Semantic content analysis and AI training data crawler", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "Sindup", + "url": "https://www.sindup.com/", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 Sindup/1.0.0" + ], + "addition_date": "2026/04/26", + "description": "Sindup competitive intelligence and market monitoring crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "SiteSucker", + "url": "https://ricks-apps.com/osx/sitesucker/", + "instances": [ + "SiteSucker for macOS/6.1.5" + ], + "addition_date": "2026/04/26", + "description": "macOS website downloading and offline browsing tool crawler", + "tags": [ + "archiver" + ] + }, + { + "pattern": "SpringserveBot", + "url": "https://www.springserve.com/", + "instances": [ + "SpringserveBot/1.0" + ], + "addition_date": "2026/04/26", + "description": "Springserve ad server and video advertising platform crawler", + "tags": [ + "advertising" + ] + }, + { + "pattern": "SQWatcher", + "url": "http://sqcompliance.com/sqwatcher.html", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 SQWatcher/202312 (sqcompliance.com/sqwatcher.html)" + ], + "addition_date": "2026/04/26", + "description": "Software quality compliance monitoring and testing crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Supabase Paired Crawler", + "url": "https://supabase.com/", + "instances": [ + "Mozilla/5.0 (Supabase Paired Crawler HTTP/2)" + ], + "addition_date": "2026/04/26", + "description": "Supabase database platform link preview and metadata crawler", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "sv-watchagent", + "url": "https://semantic-visions.com/", + "instances": [ + "Mozilla/5.0 (watchOS 6.2; Watch; compatible; sv-watchagent; HTTPClient 4.5)" + ], + "addition_date": "2026/04/26", + "description": "Semantic Visions content monitoring and analysis crawler agent", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Swiftbot", + "url": "http://swiftype.com/swiftbot", + "instances": [ + "Mozilla/5.0 (compatible; Swiftbot/1.0; UID/649dce5512ab734096a50277; +http://swiftype.com/swiftbot)" + ], + "addition_date": "2026/04/26", + "description": "Swiftype site search engine indexing and crawling bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "SynthesiBot", + "url": "https://knownagents.com/agents/synthesibot", + "instances": [ + "Mozilla/5.0 (compatible; SynthesiBot/1.0)" + ], + "addition_date": "2026/04/26", + "description": "Content synthesis and data aggregation crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "TaraGroup Intelligent Bot", + "url": "https://knownagents.com/agents/taragroup-intelligent-bot", + "instances": [ + "TaraGroup Intelligent Bot V1" + ], + "addition_date": "2026/04/26", + "description": "TaraGroup AI-powered web intelligence and analysis crawler", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "Thinkbot", + "url": "https://boston.conman.org/2025/08/21.1", + "instances": [ + "Mozilla/5.0 (compatible; Thinkbot/0.5.8; +In_the_test_phase,_if_the_Thinkbot_brings_you_trouble,_please_block_its_IP_address._Thank_you.)" + ], + "addition_date": "2026/04/26", + "description": "Experimental AI thinking and reasoning web crawler bot", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "TSMbot", + "url": "https://knownagents.com/agents/tsmbot", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7; TSMbot) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36" + ], + "addition_date": "2026/04/26", + "description": "TSM brand monitoring and content tracking crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "TSM-turingos", + "url": "https://knownagents.com/agents/turingos", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7; TSM-turingos-1253296984) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36" + ], + "addition_date": "2026/04/26", + "description": "TuringOS AI-powered content analysis and monitoring crawler", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "UGAResearchAgent", + "url": "https://nislabuga-scan.uga.edu/", + "instances": [ + "Mozilla/5.0 (compatible; UGAResearchAgent/1.0; Please visit: NISLabUGA.github.io)" + ], + "addition_date": "2026/04/26", + "description": "University of Georgia academic network research crawler bot", + "tags": [ + "academic" + ] + }, + { + "pattern": "UrlSuMa\\.de crawler", + "url": "https://urlsuma.de/", + "instances": [ + "Mozilla/5.0 (compatible; UrlSuMa.de crawler)" + ], + "addition_date": "2026/04/26", + "description": "URL summarization and content preview generation crawler bot", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "WanscannerBot", + "url": "https://abuse.pend.re", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64; WanscannerBot/1.2; +https://abuse.pend.re) Gecko/20100101 Firefox/10.0" + ], + "addition_date": "2026/04/26", + "description": "Network security scanning and vulnerability detection crawler bot", + "tags": [ + "scanner" + ] + }, + { + "pattern": "WebCapture", + "url": "https://knownagents.com/agents/webcapture-2-0", + "instances": [ + "Mozilla/3.0 (compatible; WebCapture 2.0; Auto; Windows)" + ], + "addition_date": "2026/04/26", + "description": "Website screenshot and page capture archiving tool crawler", + "tags": [ + "archiver" + ] + }, + { + "pattern": "WebCopier", + "url": "http://www.maximumsoft.com/", + "instances": [ + "WebCopier v4.6" + ], + "addition_date": "2026/04/26", + "description": "Website copying and offline browsing tool crawler bot", + "tags": [ + "archiver" + ] + }, + { + "pattern": "cognitiveseo\\.com", + "url": "http://cognitiveseo.com/bot.html", + "instances": [ + "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.10) Gecko/20050716 Thunderbird/1.0.6 - WebCrawler http://cognitiveseo.com/bot.html" + ], + "addition_date": "2026/04/26", + "description": "CognitiveSEO backlink analysis and SEO research crawler bot", + "tags": [ + "seo" + ] + }, + { + "pattern": "Xing Bot", + "url": "https://www.xing.com/", + "instances": [ + "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:15.0) Xing Bot" + ], + "addition_date": "2026/04/26", + "description": "Xing professional network link preview and content crawler", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "XML Sitemaps Generator", + "url": "http://www.xml-sitemaps.com", + "instances": [ + "Mozilla/5.0 (compatible; XML Sitemaps Generator; http://www.xml-sitemaps.com) Gecko XML-Sitemaps/1.0" + ], + "addition_date": "2026/04/26", + "description": "XML sitemap generation and website structure mapping crawler", + "tags": [ + "seo" + ] + }, + { + "pattern": "YandoriRSSBot", + "url": "https://knownagents.com/agents/yandorirssbot", + "instances": [ + "YandoriRSSBot/3.0 (Go)" + ], + "addition_date": "2026/04/26", + "description": "RSS feed aggregation and content syndication crawler bot", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Zealbot", + "url": "https://knownagents.com/agents/zealbot", + "instances": [ + "Mozilla/4.0 (compatible; Zealbot 1.0)" + ], + "addition_date": "2026/04/26", + "description": "Web content indexing and search engine crawler bot", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "008\\/", + "url": "https://datadome.co/bots/008-2/", + "instances": [ + "008/0.83 (http://www.advancedwebranking.com)" + ], + "description": "SEO monitoring bot tracks keyword rankings across regions", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "monitoring360bot\\/", + "url": "https://app.360monitoring.com/bot.html", + "instances": [ + "Mozilla/5.0 (compatible; monitoring360bot/1.1; +https://app.360monitoring.com/bot.html)" + ], + "description": "Monitors website uptime and performance at intervals", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "AdagioBot", + "url": "https://datadome.co/bots/adagio-digital/", + "instances": [ + "AdagioBot" + ], + "description": "SEO analysis bot for performance and optimization", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "adbeat\\.com", + "url": "https://www.adbeat.com/operation_policy", + "instances": [ + "Mozilla/5.0 (iPad; CPU OS 15_3_1 like Mac OS X) adbeat.com/policy AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.3.1 Mobile/15E148 Safari/604.1" + ], + "description": "Competitive intelligence bot for digital ad tracking", + "addition_date": "2026/04/17", + "tags": [ + "advertising" + ] + }, + { + "pattern": "AdminLabs", + "url": "https://datadome.co/bots/adminlabs/", + "instances": [ + "AdminLabs" + ], + "description": "Web scraping tool for data collection and analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "advanced_crawler", + "url": "https://datadome.co/bots/advanced-crawler/", + "instances": [ + "advanced_crawler (+http://example.com)" + ], + "description": "Sophisticated crawler with dynamic content rendering", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Adventurer", + "url": "https://datadome.co/bots/adventurer/", + "instances": [ + "Adventurer" + ], + "description": "Web crawler for indexing content and analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "AGAKIDSBOT", + "url": "https://agakids.ru/project/", + "instances": [ + "Mozilla/5.0 (compatible; Linux; AGAKIDSBOT/3.1 +agakids.ru/project) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" + ], + "description": "Web crawler for indexing and data gathering", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "AgencyAnalyticsBot", + "url": "https://agencyanalytics.com/features/seo-site-audit", + "instances": [ + "AgencyAnalyticsBot" + ], + "description": "SEO audit bot for performance and backlink analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "AI2Bot", + "url": "https://datadome.co/bots/ai2bot/", + "instances": [ + "AI2Bot" + ], + "description": "Academic research crawler for NLP dataset creation", + "addition_date": "2026/04/17", + "tags": [ + "academic", + "ai-crawler" + ] + }, + { + "pattern": "AkismetBot", + "url": "https://akismet.com/development/api/", + "instances": [ + "AkismetBot" + ], + "description": "Spam filtering service for WordPress and CMS", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "alexa site audit", + "url": "https://www.alexa.com/help/webmasters", + "instances": [ + "Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; )" + ], + "description": "Web crawler for SEO data aggregation and analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Algolia Crawler", + "url": "https://www.algolia.com/doc/", + "instances": [ + "Algolia Crawler" + ], + "description": "Hosted search engine and site search crawler", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "alienfarm", + "url": "https://datadome.co/bots/alienfarm/", + "instances": [ + "alienfarm" + ], + "description": "Web crawler for data aggregation and intelligence", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "allOrigins", + "url": "https://allorigins.win/", + "instances": [ + "Mozilla/5.0 (compatible; allOrigins/3; +http://allorigins.win/)" + ], + "description": "Web scraping proxy tool bypassing CORS restrictions", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "AmazonAdBot", + "url": "https://advertising.amazon.com/resources/", + "instances": [ + "AmazonAdBot" + ], + "description": "Ad verification bot for delivery and impressions", + "addition_date": "2026/04/17", + "tags": [ + "advertising" + ] + }, + { + "pattern": "KendraBot", + "url": "https://docs.aws.amazon.com/kendra/latest/dg/what-is-kendra.html", + "instances": [ + "KendraBot" + ], + "description": "AI-powered enterprise search crawler for documents", + "addition_date": "2026/04/17", + "tags": [ + "search-engine", + "ai-crawler" + ] + }, + { + "pattern": "AppSiteAssociation", + "url": "https://developer.apple.com/documentation/applications/allowing-app-linking-to-your-website", + "instances": [ + "AppSiteAssociation" + ], + "description": "Verifies app connection to website domains", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Aragog\\/", + "url": "https://wordads.co/", + "instances": [ + "Aragog/1.0 (+https://wordads.co)" + ], + "description": "Web crawler for indexing and data collection", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Aranea", + "url": "http://unesco.uniba.sk/guest/", + "instances": [ + "Aranea Web-Crawled Corpora Project (+http://unesco.uniba.sk/guest (Hebrew 2024 Spring Crawl))" + ], + "description": "Automated web crawler for content aggregation", + "addition_date": "2026/04/17", + "tags": [ + "academic" + ] + }, + { + "pattern": "ArchiveBox", + "url": "https://archivebox.io/", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36 ArchiveBox/0.7.2 (+https://github.com/ArchiveBox/ArchiveBox/)" + ], + "description": "Web archiving tool for long-term content preservation", + "addition_date": "2026/04/17", + "tags": [ + "archiver" + ] + }, + { + "pattern": "ArquivoBot", + "url": "https://arquivo.pt/about", + "instances": [ + "ArquivoBot" + ], + "description": "Preserves Portuguese websites historical content", + "addition_date": "2026/04/17", + "tags": [ + "archiver" + ] + }, + { + "pattern": "Arquivo-web-crawler", + "url": "https://arquivo.pt/robot", + "instances": [ + "Arquivo-web-crawler (compatible; +https://arquivo.pt/robot)" + ], + "description": "Portuguese web archive for digital heritage", + "addition_date": "2026/04/17", + "tags": [ + "archiver" + ] + }, + { + "pattern": "ArtemisBot", + "url": "https://datadome.co/bots/artemis-web-reader/", + "instances": [ + "ArtemisBot" + ], + "description": "Vulnerability scanning and cybersecurity research bot", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Asana\\/", + "url": "https://datadome.co/bots/asana-crawler/", + "instances": [ + "Asana/1.4.0 WebsiteMetadataRetriever" + ], + "description": "Work management platform metadata retriever bot", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "AudistoBot", + "url": "https://audisto.com/webcrawler/", + "instances": [ + "AudistoBot" + ], + "description": "Technical SEO and website audit bot", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Autoconfig Test from USTC", + "url": "https://datadome.co/bots/autoconfig-test-from-ustc/", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.93 Safari/537.36 (Autoconfig Test from USTC, Quit please contact: mailservertest2023@gmail.com)" + ], + "description": "USTC university research crawler for web analysis", + "addition_date": "2026/04/17", + "tags": [ + "academic" + ] + }, + { + "pattern": "tracking-quality-spider", + "url": "https://datadome.co/bots/awin-com-crawler/", + "instances": [ + "Mozilla/5.0 (compatible; tracking-quality-spider/0.1; https://www.awin.com)" + ], + "description": "Affiliate marketing network tracking and verification", + "addition_date": "2026/04/17", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Bad Neighborhood Header Detector", + "url": "https://datadome.co/bots/bad-neighborhood/", + "instances": [ + "Bad Neighborhood Header Detector (http://www.bad-neighborhood.com/header_detector.php)" + ], + "description": "Website security risk analysis for link quality", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "BaiduAdsBot", + "url": "https://datadome.co/bots/baidu-ads-server-proxy/", + "instances": [ + "BaiduAdsBot" + ], + "description": "Baidu ad verification and tracking bot", + "addition_date": "2026/04/17", + "tags": [ + "advertising" + ] + }, + { + "pattern": "BDBot\\/", + "url": "https://datadome.co/bots/bdbot/", + "instances": [ + "BDBot/1.0" + ], + "description": "Web crawler for data aggregation and analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "BeeperBot", + "url": "https://www.beeper.com/", + "instances": [ + "BeeperBot/0 Matrix-Media-Repo/1" + ], + "description": "Web crawler for aggregation and business intelligence", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "BetterUptimeBot", + "url": "https://betteruptime.com/docs", + "instances": [ + "BetterUptimeBot" + ], + "description": "Website uptime and performance monitoring service", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "BnFBot", + "url": "https://www.bnf.fr/en/web-services", + "instances": [ + "BnFBot" + ], + "description": "French national library archiving cultural content", + "addition_date": "2026/04/17", + "tags": [ + "archiver" + ] + }, + { + "pattern": "BigUpDataBot", + "url": "https://datadome.co/bots/bigupdata/", + "instances": [ + "BigUpDataBot" + ], + "description": "Analytics crawler for SEO and metric tracking", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "BinaryCanary", + "url": "https://www.binarycanary.com/monitoring/", + "instances": [ + "BinaryCanary/1.0 (+https://www.binarycanary.com)" + ], + "description": "Uptime monitoring for servers and services", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Bitbucket-Webhooks", + "url": "https://support.atlassian.com/bitbucket-cloud", + "instances": [ + "Bitbucket-Webhooks/2.0" + ], + "description": "Webhook notifications for CI/CD pipeline automation", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "bl\\.uk_ldfc_bot", + "url": "https://www.bl.uk/legal-deposit/web-archiving", + "instances": [ + "bl.uk_ldfc_bot/3.4.0-20220727 (+https://www.bl.uk/legal-deposit/web-archiving)" + ], + "description": "British Library web archiver for content preservation", + "addition_date": "2026/04/17", + "tags": [ + "archiver" + ] + }, + { + "pattern": "BlackDuck-FD", + "url": "https://www.synopsys.com/software-integrity/security-testing/dynamic-analysis.html", + "instances": [ + "BlackDuck-FD" + ], + "description": "Vulnerability scanner for application security testing", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Blogtrottr", + "url": "https://datadome.co/bots/blogtrottr/", + "instances": [ + "Blogtrottr/2.0" + ], + "description": "Email subscription service for blog and feed updates", + "addition_date": "2026/04/17", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "BlueskyPreviewBot", + "url": "https://docs.bsky.app", + "instances": [ + "BlueskyPreviewBot/1.0" + ], + "description": "Bluesky social platform link preview fetcher", + "addition_date": "2026/04/17", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "BoardGamePricesBot", + "url": "https://datadome.co/bots/boardgameprices/", + "instances": [ + "Likely BoardGamePricesBot or similar (not officially published)" + ], + "description": "Board game price comparison and availability tracker", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "BotPoke", + "url": "https://datadome.co/bots/botpoke/", + "instances": [ + "BotPoke" + ], + "description": "Web crawler for data aggregation and analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "BDFetch", + "url": "http://www.brandprotect.com/", + "instances": [ + "BDFetch" + ], + "description": "Brand protection bot for unauthorized use detection", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Brandwatch", + "url": "https://www.brandwatch.com/legal/crawlers/", + "instances": [ + "Brandwatch/1.0 (varies)" + ], + "description": "Social listening and sentiment analysis platform", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "BraveBot", + "url": "https://search.brave.com/help/web-discovery-project", + "instances": [ + "BraveBot / Mozilla/5.0 (compatible; BraveBot/1.0)" + ], + "description": "Privacy-focused search engine indexing crawler", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "brokenlinkcheck\\.com", + "url": "https://datadome.co/bots/brokenlinkcheck-com/", + "instances": [ + "Mozilla/5.0 (Windows NT 5.1) brokenlinkcheck.com/1.2" + ], + "description": "Broken link detection and website integrity checker", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "BW\\/", + "url": "https://builtwith.com/biup", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BW/1.2; rb.gy/oupwis) Chrome/124.0.0.0 Safari/537.36" + ], + "description": "Technology profiling crawler for web analytics", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Bushbaby", + "url": "https://datadome.co/bots/bushbaby/", + "instances": [ + "Bushbaby (reported in logs)" + ], + "description": "Content aggregation and market intelligence crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Butterfly", + "url": "http://labs.topsy.com/butterfly/", + "instances": [ + "Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8" + ], + "description": "Experimental data collection crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "rss-parser", + "url": "https://buttondown.email/about", + "instances": [ + "rss-parser / Buttondown" + ], + "description": "Newsletter RSS feed parser and aggregator", + "addition_date": "2026/04/17", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "CaliberBot", + "url": "https://www.calibermind.com/platform", + "instances": [ + "CaliberBot (varies)" + ], + "description": "Market research and analytics crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "CapitalOneShopping", + "url": "https://datadome.co/bots/capital-one-shopping-bot/", + "instances": [ + "CapitalOneShopping/1.0 (example)" + ], + "description": "E-commerce price comparison and shopping bot", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Catchpoint", + "url": "ttps://catchpoint.com/bots", + "instances": [ + "Catchpoint" + ], + "description": "Digital experience and performance monitoring bot", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "centuryb\\.o\\.t9", + "url": "https://datadome.co/bots/centurybot9/", + "instances": [ + "Mozilla/5.0 (compatible; +centuryb.o.t9[at]gmail.com)" + ], + "description": "General web crawler for data aggregation", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "CERT PL", + "url": "https://cert.pl/skanowanie", + "instances": [ + "Mozilla/5.0 (compatible; Artemis; CERT PL; +https://cert.pl/skanowanie)" + ], + "description": "Polish cybersecurity vulnerability and threat scanner", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "certytags", + "url": "https://certybot.certytags.com/", + "instances": [ + "Mozilla/5.0 (compatible; certytags/1.0; +https://certybot.certytags.com)" + ], + "description": "Web content indexing and verification crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "ChargeBeeBot", + "url": "https://chargebee.com/resources", + "instances": [ + "ChargeBeeBot" + ], + "description": "Subscription billing verification and integration bot", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Charlotte", + "url": "https://datadome.co/bots/charlotte-bot/", + "instances": [ + "Commonly seen variations include: Mozilla/5.0 (X11; U; Linux i686 (x86_64); en-US; rv:1.8.1.11) Gecko/20080109 (Charlotte/0.9t; http://www.searchme.com/support/)" + ], + "description": "Financial institution and robot-friendly crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "ChatGLM-Spider", + "url": "https://chatglm.cn/", + "instances": [ + "Mozilla/5.0 (compatible; ChatGLM-Spider/1.0; +https://chatglm.cn/)" + ], + "description": "AI model training data collection crawler", + "addition_date": "2026/04/17", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "Chatwork LinkPreview", + "url": "https://www.chatwork.com/", + "instances": [ + "Chatwork LinkPreview v1" + ], + "description": "Collaboration platform URL preview fetcher", + "addition_date": "2026/04/17", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "CheckHost", + "url": "https://datadome.co/bots/check-host/", + "instances": [ + "CheckHost (https://check-host.net/)" + ], + "description": "Website uptime and performance monitoring bot", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Goodzer", + "url": "https://discord.com/discovery/applications/1065250549408223252", + "instances": [ + "Mozilla/5.0 (compatible; Goodzer/1.0) and Mozilla/5.0 (compatible; Goodzer/2.0; crawler@goodzer.com)" + ], + "description": "Discord bot enhancement and server functionality crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Chrome Privacy Preserving Prefetch Proxy", + "url": "https://datadome.co/bots/chrome-privacypreserving-prefetch-proxy/", + "instances": [ + "Chrome Privacy Preserving Prefetch Proxy" + ], + "description": "Chrome browser resource prefetching with privacy protection", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "CirrusExplorer", + "url": "https://cseu.ro/explorer.php", + "instances": [ + "CirrusExplorer/2 (https://cseu.ro/explorer.php)" + ], + "description": "Web crawler for indexing and competitive analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "CLASSLA-web", + "url": "https://www.clarin.si/info/classla-web-crawler/", + "instances": [ + "Mozilla/5.0 (compatible; CLASSLA-web; +https://www.clarin.si/info/classla-web-crawler/)" + ], + "description": "Linguistic research data crawler for NLP", + "addition_date": "2026/04/17", + "tags": [ + "academic" + ] + }, + { + "pattern": "Clearscopebot", + "url": "https://datadome.co/bots/clearscope-clearscopebot/", + "instances": [ + "UA Clearscopebot or similar" + ], + "description": "Content optimization and keyword performance analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "WorldBot", + "url": "https://datadome.co/bots/clickagy-intelligence-bot/", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; WorldBot/0.1; +https://worldlabs.ai)" + ], + "description": "Behavioral analytics and audience segmentation crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Cloudflare-Validator", + "url": "https://datadome.co/bots/cloudflare-crawler/", + "instances": [ + "Cloudflare-Validator/1.0" + ], + "description": "Cloudflare security and performance validation", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "cloudflare-csup", + "url": "https://datadome.co/bots/cloudflare-csup/", + "instances": [ + "cloudflare-csup (Cloudflare Radar)" + ], + "description": "Cache status updates and performance monitoring", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Cloudflare-Custom-Error-Page-Crawler", + "url": "https://developers.cloudflare.com", + "instances": [ + "Cloudflare-Custom-Error-Page-Crawler" + ], + "description": "Custom error page validation and testing", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Cloudflare-Radar-Scanner", + "url": "https://radar.cloudflare.com", + "instances": [ + "Cloudflare-Radar-Scanner" + ], + "description": "URL scanning for traffic and threat intelligence", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Cloudflare-SpeedTest", + "url": "https://www.cloudflare.com/speedtest", + "instances": [ + "Cloudflare-SpeedTest" + ], + "description": "Website performance and latency testing bot", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Cloudflare-Stream-Hook", + "url": "https://developers.cloudflare.com/stream/webhooks/", + "instances": [ + "Cloudflare-Stream-Hook" + ], + "description": "Video lifecycle event notifications and management", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "cognitiveSEO Bot", + "url": "https://cognitiveseo.com/bot", + "instances": [ + "cognitiveSEO Bot" + ], + "description": "SEO analytics and digital marketing crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "cohere-training-data-crawler", + "url": "https://datadome.co/bots/cohere-training-data-crawler/", + "instances": [ + "cohere-training-data-crawler (+crawler@cohere.ai)" + ], + "description": "LLM training data collection for AI models", + "addition_date": "2026/04/17", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "CommaFeed", + "url": "https://datadome.co/bots/commafeed/", + "instances": [ + "CommaFeed/4.4.0 (https://github.com/Athou/commafeed)" + ], + "description": "RSS feed aggregator and reader bot", + "addition_date": "2026/04/17", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "researchscan\\.comsys\\.rwth-aachen\\.de", + "url": "http://researchscan.comsys.rwth-aachen.de/", + "instances": [ + "Mozilla/5.0 researchscan.comsys.rwth-aachen.de" + ], + "description": "Academic network measurement and analysis", + "addition_date": "2026/04/17", + "tags": [ + "academic" + ] + }, + { + "pattern": "contentkingapp", + "url": "https://whatis.contentkingapp.com/", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36 (+https://whatis.contentkingapp.com)" + ], + "description": "Real-time SEO auditing and monitoring tool", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "CookieHub Bot", + "url": "https://www.cookiehub.com/docs", + "instances": [ + "CookieHub Bot" + ], + "description": "Cookie consent and privacy compliance auditor", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Cotoyogi", + "url": "https://ds.rois.ac.jp/center8/crawler/", + "instances": [ + "Mozilla/5.0 (compatible; Cotoyogi/4.0; +https://ds.rois.ac.jp/center8/crawler/)" + ], + "description": "Web crawler for content and data aggregation", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Coveobot", + "url": "https://platform.cloud.coveo.com/", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) (compatible; Coveobot/2.0;+http://www.coveo.com/bot.html)" + ], + "description": "Enterprise search and content indexing platform", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Crawlson", + "url": "https://www.crawlson.com/", + "instances": [ + "Mozilla/5.0 (compatible; Crawlson/1.0; +https://www.crawlson.com/domain)" + ], + "description": "Web crawler for auditing and monitoring", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "RepoLookoutBot", + "url": "https://www.repo-lookout.org/", + "instances": [ + "RepoLookoutBot/v1.1.0-297-gcf436d3 (abuse reports to abuse@repo-lookout.org)" + ], + "description": "Repository and web content lookup crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Criticalcss\\.com", + "url": "https://criticalcss.com/", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.0 Safari/537.36 Criticalcss.com/2.0.0" + ], + "description": "Critical CSS extraction for performance optimization", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "cron-job\\.org", + "url": "https://cron-job.org/en/", + "instances": [ + "cron-job.org monitor" + ], + "description": "Automated website monitoring and uptime checking", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "DnBCrawler", + "url": "https://datadome.co/bots/dnbcrawler/", + "instances": [ + "DnBCrawler-Analytics" + ], + "description": "Business data analytics crawler gathering company information insights", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "DMBrowser", + "url": "https://www.dotcom-monitor.com/wiki/knowledge-base-main/", + "instances": [ + "DMBrowser/" + ], + "description": "Synthetic monitoring browser checking website performance and availability", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "DomCopBot", + "url": "https://www.domcop.com/bot", + "instances": [ + "DomCopBot (https://www.domcop.com/bot)" + ], + "description": "Domain research crawler analyzing website metrics and SEO data", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "downnotifier\\.com", + "url": "https://datadome.co/bots/downnotifier-com-monitoring/", + "instances": [ + "downnotifier.com monitoring" + ], + "description": "Uptime monitoring service alerting on website downtime events", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "DowntimeDetector\\/", + "url": "https://datadome.co/bots/downtimedetector/", + "instances": [ + "DowntimeDetector/4.0 (+https://downforeveryoneorjustme.com)" + ], + "description": "Availability monitor detecting and reporting website downtime incidents", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Dlc\\/", + "url": "https://www.drlinkcheck.com/", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36 Dlc/2.0.1" + ], + "description": "Link checking crawler auditing broken URLs across websites", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Dratabot", + "url": "https://dratabot.com", + "instances": [ + "Dratabot (+https://dratabot.com)" + ], + "description": "Compliance automation crawler auditing security controls and documentation", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "EasyBib AutoCite", + "url": "http://www.easybib.com/", + "instances": [ + "EasyBib AutoCite (http://autocite-info.citation-api.com/)" + ], + "description": "Citation generator fetching metadata for bibliography and reference creation", + "addition_date": "2026/04/17", + "tags": [ + "academic" + ] + }, + { + "pattern": "easybill-ImportManager", + "url": "https://www.easybill.de/api/", + "instances": [ + "easybill-ImportManager/ShopwareClient" + ], + "description": "Billing platform crawler importing product data from ecommerce stores", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "EasyCron\\/", + "url": "https://www.easycron.com", + "instances": [ + "EasyCron/1.0 (https://www.easycron.com/)" + ], + "description": "Cron job service executing scheduled HTTP requests and monitoring endpoints", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "easyDNS Monitoring", + "url": "http://easyurl.net/monitoring", + "instances": [ + "easyDNS Monitoring ( http://easyurl.net/monitoring )" + ], + "description": "DNS monitoring service verifying domain resolution and availability", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "EchoboxBot\\/", + "url": "https://www.echobox.com/", + "instances": [ + "Mozilla/5.0 (compatible; EchoboxBot/1.0; hash/w4mwnpbXf3MFAbxOkJRw; +http://www.echobox.com)" + ], + "description": "Social media automation crawler fetching content for publishing optimization", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Cronless", + "url": "https://datadome.co/bots/cronless/", + "instances": [ + "Cronless/2.0 (+http://cronless.com)" + ], + "description": "Dynamic task execution crawler without fixed scheduling", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "crusty\\/", + "url": "https://github.com/let4be/crusty", + "instances": [ + "crusty/0.12.0" + ], + "description": "Undocumented web crawler of unknown origin", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "csirt\\.cz", + "url": "https://csirt.cz/cs/dns-crawler", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36 1.6.4 (+https://csirt.cz/cs/dns-crawler)" + ], + "description": "Czech cybersecurity team vulnerability scanning bot", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "CXK_Bot", + "url": "https://datadome.co/bots/cxk_bot/", + "instances": [ + "CXK/5.0 (linux; 43A1AAD0; ARMv9) assistenC5/v6.5.6 (KHTML, lcxk) CXK_Bot/2.3.1" + ], + "description": "Web content indexing and SEO analysis crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "daumoa", + "url": "http://cs.daum.net/faq/15/4118.html?faqId=28966", + "instances": [ + "daumoa,damoa,daum,daumos,duamoa,duam,duamo" + ], + "description": "South Korean Daum search engine indexing bot", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "DaspeedBot", + "url": "https://datadome.co/bots/dawap-bot/", + "instances": [ + "DaspeedBot/1.0 (+https://daspeed.io/bot-info)" + ], + "description": "Web content indexing and analytics crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Dead Link Checker", + "url": "http://www.dead-link-checker.com/", + "instances": [ + "Mozilla/5.0 (compatible; Dead Link Checker; http://www.dead-link-checker.com/)" + ], + "description": "Broken link detection and website maintenance bot", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Deskyobot", + "url": "https://www.deskyo.com/bot", + "instances": [ + "Deskyobot/1.0 (+https://www.deskyo.com/bot)" + ], + "description": "Website content indexing and SEO analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Detectify", + "url": "https://detectify.com/what-is-detectify", + "instances": [ + "Mozilla/5.0 (compatible; Detectify) +https://detectify.com/bot/" + ], + "description": "Vulnerability and attack surface scanning bot", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Devin", + "url": "https://docs.devin.ai/get-started/devin-intro", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36; Devin/1.0; +https://devin.ai" + ], + "description": "AI-driven browser automation and content extraction", + "addition_date": "2026/04/17", + "tags": [ + "browser-automation", + "ai-crawler" + ] + }, + { + "pattern": "DF Bot", + "url": "https://datadome.co/bots/df-bot/", + "instances": [ + "DF Bot 1.0" + ], + "description": "Web content indexing and competitive analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "DingTalkBot-LinkService", + "url": "https://www.dingtalk.com/", + "instances": [ + "DingTalkBot-LinkService/1.0 (+https://open-doc.dingtalk.com/microapp/faquestions/ftpfeu)" + ], + "description": "Alibaba DingTalk link preview and metadata fetcher", + "addition_date": "2026/04/17", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Discourse Forum Onebox", + "url": "https://discourse.org/", + "instances": [ + "Discourse Forum Onebox v3.3.0.beta1-dev" + ], + "description": "Forum link preview and embed content fetcher", + "addition_date": "2026/04/17", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Dmbot", + "url": "https://datadome.co/bots/dmbot/", + "instances": [ + "Mozilla/5.0 (compatible; Dmbot/1.1)" + ], + "description": "Web crawler for data aggregation and SEO", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "SustainabilityCrawler", + "url": "https://datadome.co/bots/ecovadis-bot/", + "instances": [ + "ecotrek GmbH - SustainabilityCrawler (bot@ecotrek.tech)" + ], + "description": "ESG and sustainability rating data collection", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "edansbot", + "url": "https://datadome.co/bots/edansbot/", + "instances": [ + "Mozilla/5.0 edansbot (Windows NT 10.0; Win64; x64; rv:90.0) Gecko/20100101 Firefox/118.0" + ], + "description": "Web content indexing and analytics crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "EdgeWatch", + "url": "https://about.edgewatch.com/", + "instances": [ + "Mozilla/5.0 (compatible; EdgeWatch/1.1; https://about.edgewatch.com/)" + ], + "description": "Performance monitoring and security assessment bot", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Do Not Track Verifier", + "url": "https://datadome.co/bots/eff-crawler/", + "instances": [ + "Electronic Frontier Foundation's Do Not Track Verifier (for questions or concerns email dnt-policy@eff.org)" + ], + "description": "Privacy practices and security protocol analyzer", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "elmahio-uptimebot", + "url": "https://elmah.io", + "instances": [ + "elmahio-uptimebot/2.0" + ], + "description": "Error logging uptime and availability monitoring", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "eMoneyBot", + "url": "https://emoneyadvisor.com", + "instances": [ + "eMoneyBot/1.0; +https://emoneyadvisor.com/DataAggregationNotice/" + ], + "description": "Financial account aggregation and data scraping", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "EpivozCrawler", + "url": "https://www.techmeme.com", + "instances": [ + "EpivozCrawler/1.7" + ], + "description": "Content scraping and indexing crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "eRepublik\\.tools", + "url": "https://erepublik.tools", + "instances": [ + "eRepublik.tools - Multithreaded Crawler - " + ], + "description": "MMO game data and analytics crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "EvoUptimeBot", + "url": "https://www.evo.agency", + "instances": [ + "EvoUptimeBot/1.0" + ], + "description": "Uptime and performance monitoring service", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "ExodusMovement", + "url": "https://www.exodus.io", + "instances": [ + "ExodusMovement/1.0 GlobalCoinHeight/1.0" + ], + "description": "Web scraping and data extraction bot", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Ezgif", + "url": "https://ezgif.com/about", + "instances": [ + "Mozilla/5.0 (compatible; Ezgif/69.420; +https://ezgif.com/about)" + ], + "description": "GIF and image fetching and editing bot", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "factset_spyderbot", + "url": "https://www.factset.com/", + "instances": [ + "Mozilla/5.0(windows NT 10.0; Win64; x64) AppleWebkit/537.36(KHTML, like Gecko) Chrome/90.0.4430.72 Safari/537.36 factset_spyderbot" + ], + "description": "Financial data and regulatory filing crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "FastmailUA", + "url": "https://www.fastmail.com/policies/bots/", + "instances": [ + "FastmailUA/1.0" + ], + "description": "Email link preview and image proxy fetcher", + "addition_date": "2026/04/17", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "FDL Stats Bot", + "url": "https://ftwentertainment.com", + "instances": [ + "FDL Stats Bot" + ], + "description": "Firebase dynamic link analytics and validation", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Fedicabot", + "url": "https://fedica.com/info/fedicabot", + "instances": [ + "Fedicabot / FedicaApp" + ], + "description": "Social media analytics and link preview bot", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "FedReporterDataBot", + "url": "https://fedreporter.net/FedReporterBotDocumentation/Readme.txt", + "instances": [ + "FedReporterDataBot/1.0 (+https://fedreporter.net/FedReporterBotDocumentation;)" + ], + "description": "Financial and regulatory filing data aggregator", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Feed Image Audit", + "url": "https://image-validator.com/", + "instances": [ + "Feed Image Audit -- App" + ], + "description": "Image validation and compliance auditing", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "FeedBurner", + "url": "http://www.feedburner.com/", + "instances": [ + "FeedBurner/1.0 (http://www.FeedBurner.com)" + ], + "description": "RSS feed distribution and subscriber management", + "addition_date": "2026/04/17", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "feeder\\.co", + "url": "https://feeder.co/crawler", + "instances": [ + "Mozilla/5.0 (feeder.co; Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36" + ], + "description": "RSS feed aggregation and monitoring service", + "addition_date": "2026/04/17", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Feedpresso Content Index Bot", + "url": "https://datadome.co/bots/feedpresso-crawler/", + "instances": [ + "Feedpresso Content Index Bot" + ], + "description": "News aggregation platform content indexer", + "addition_date": "2026/04/17", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Feedwind", + "url": "http://feed.mikle.com/support/description/", + "instances": [ + "Mozilla/5.0 (compatible; Feedwind/3.0; +http://feed.mikle.com/support/description/)" + ], + "description": "Dynamic content aggregation widget fetcher", + "addition_date": "2026/04/17", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "fidget-spinner-bot", + "url": "https://datadome.co/bots/fidget-spinner-bot/", + "instances": [ + "fidget-spinner-bot" + ], + "description": "Content indexing and SEO analysis crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "FirmoGraph", + "url": "https://datadome.co/bots/firmograph/", + "instances": [ + "FirmoGraph (+https://firmograph.io)" + ], + "description": "Business data collection and analytics crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "FlipboardRSS", + "url": "http://flipboard.com/browserproxy", + "instances": [ + "Mozilla/5.0 (compatible; FlipboardRSS/1.2; +http://flipboard.com/browserproxy)" + ], + "description": "Social magazine content aggregation and curation", + "addition_date": "2026/04/17", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Foregenix", + "url": "http://www.foregenix.com/scan", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko, Foregenix) Chrome/91.0.4472.77 Safari/537.36" + ], + "description": "Cybersecurity vulnerability and weakness scanner", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Freespoke\\/", + "url": "https://docs.freespoke.com/search/bot/", + "instances": [ + "Mozilla/5.0 (compatible; Freespoke/2.0; +https://crawler.freespoke.com)" + ], + "description": "Privacy-focused search engine indexing web content for results", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Friendly testing bot", + "url": "https://datadome.co/bots/friendly-testing-bot/", + "instances": [ + "Friendly testing bot" + ], + "description": "Testing crawler verifying website functionality and response behavior", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "friendly-spider", + "url": "https://datadome.co/bots/friendly-spider/", + "instances": [ + "friendly-spider" + ], + "description": "Web crawler indexing content for search and data aggregation", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "FriendlyCrawler\\/", + "url": "https://datadome.co/bots/friendlycrawler/", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; FriendlyCrawler/1.0) Chrome/120.0.6099.216 Safari/605.1.15" + ], + "description": "General web crawler indexing content for data collection purposes", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "FullStoryBot\\/", + "url": "https://help.fullstory.com/spp-ref/343521-what-is-the-fullstorybot", + "instances": [ + "FullStoryBot/1.0" + ], + "description": "Analytics platform crawler collecting session and behavioral data", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Funnelback", + "url": "https://docs.squiz.net/funnelback/docs/latest/", + "instances": [ + "Funnelback" + ], + "description": "Enterprise search crawler indexing content for internal search platforms", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "FuseonBot\\/", + "url": "https://datadome.co/bots/fuseonbot/", + "instances": [ + "FuseonBot/1.1 (+http://linkaffinity.io)" + ], + "description": "Link affinity crawler indexing web content for SEO analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Gabanzabot\\/", + "url": "https://datadome.co/bots/gabanzabot/", + "instances": [ + "Gabanzabot/1.1 (Gabanza Search Engine; https://www.gabanza.com)" + ], + "description": "Search engine crawler indexing websites for Gabanza search results", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "gdnplus\\.com", + "url": "https://datadome.co/bots/gdnp-crawler/", + "instances": [ + "https://gdnplus.com:Gather Analyze Provide." + ], + "description": "Data gathering crawler analyzing and providing web content insights", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "getthit\\.com", + "url": "https://www.getthit.com/bot", + "instances": [ + "www.GettHIT.com | Free Traffic Exchange Bot | If you are seeing this, then your website has been listed in our traffic exchange service. | Visit Us : https://www.getthit.com/bot | Macintosh; Intel Mac OS X 10_7_5 (compatible; getthit.com/3.1;)" + ], + "description": "Traffic exchange service bot generating visits to listed websites", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "GG PeekBot", + "url": "https://www.gg.pl/", + "instances": [ + "GG PeekBot 2.0 ( https://www.gg.pl/ https://www.gg.pl/info/praca/ )" + ], + "description": "Polish messaging platform crawler generating link previews for users", + "addition_date": "2026/04/17", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Ghost Inspector", + "url": "https://ghostinspector.com/", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0 Safari/537.36 Ghost Inspector (63c85ddbde52d6697f57c623)" + ], + "description": "Browser testing service automating UI tests and monitoring website changes", + "addition_date": "2026/04/17", + "tags": [ + "browser-automation" + ] + }, + { + "pattern": "github-camo", + "url": "https://github.com/atmos/camo", + "instances": [ + "github-camo (c006e452)" + ], + "description": "GitHub image proxy fetching external images for secure rendering", + "addition_date": "2026/04/17", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "GlobalWebSearch", + "url": "https://datadome.co/bots/globalwebsearch/", + "instances": [ + "GlobalWebSearchx" + ], + "description": "Web crawler indexing content for global search engine results", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Golfe\\/", + "url": "https://datadome.co/bots/6hphjxgx/", + "instances": [ + "Mozilla/7.0 (compatible; Golfe/1.1; +http://www.goo-olfe.ae/bot.html)" + ], + "description": "UAE-based search engine crawler indexing web content for search", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Google-Apps-Script", + "url": "https://script.google.com/", + "instances": [ + "Mozilla/5.0 (compatible; Google-Apps-Script; beanserver; +https://script.google.com; id: UAEmdDd_XLoqpGxtGfu5uvUaQlW77VLYz-w)" + ], + "description": "Google scripting platform fetching URLs for automation workflows", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "GoogleStackdriverMonitoring", + "url": "https://cloud.google.com/monitoring", + "instances": [ + "GoogleStackdriverMonitoring-UptimeChecks(https://cloud.google.com/monitoring)" + ], + "description": "Google Cloud monitoring service performing uptime checks on endpoints", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "GoogleAssociationService\\/", + "url": "https://developers.google.com/identity/credential-sharing/digital-asset-links#:~:text=then%20act%20upon.-,Overview,as%20location%2C%20with%20website%20B.", + "instances": [ + "GoogleAssociationService/" + ], + "description": "Google service verifying digital asset links between apps and websites", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "GoogleImageProxy", + "url": "https://support.google.com/webmasters/answer/1061943?hl=en", + "instances": [ + "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)" + ], + "description": "Google image proxy fetching and caching images for Gmail display", + "addition_date": "2026/04/17", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "GoogleProducer", + "url": "https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers#googleproducer", + "instances": [ + "GoogleProducer; (+https://developers.google.com/search/docs/crawling-indexing/google-producer)" + ], + "description": "Google user-triggered fetcher retrieving content for Google products", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Googlebot-IA\\/", + "url": "https://scholar.google.com/intl/en/scholar/libraries.html", + "instances": [ + "Googlebot-IA/2.1" + ], + "description": "Google Scholar crawler indexing academic papers and library content", + "addition_date": "2026/04/17", + "tags": [ + "search-engine", + "academic" + ] + }, + { + "pattern": "Google-Trust-Services\\/", + "url": "https://pki.goog/", + "instances": [ + "Mozilla/5.0 (compatible; Google-Trust-Services/2.0; http://pki.goog/)" + ], + "description": "Google PKI service validating SSL certificates and checking revocation status", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Google-Area120", + "url": "https://area120.google.com/", + "instances": [ + "Google-Area120-PrivacyPolicyFetcher" + ], + "description": "Google Area 120 experimental tool fetching privacy policies from websites", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Google-CloudVertexBot", + "url": "https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers#google-cloudvertexbot", + "instances": [ + "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-CloudVertexBot; +https://cloud.google.com/enterprise-search)", + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-CloudVertexBot; +https://cloud.google.com/enterprise-search) Chrome/W.X.Y.Z Safari/537.36" + ], + "description": "Google Vertex AI crawler indexing content for enterprise search solutions", + "addition_date": "2026/04/17", + "tags": [ + "search-engine", + "ai-crawler" + ] + }, + { + "pattern": "GoogleAssociationService$", + "url": "https://developers.google.com/digital-asset-links", + "instances": [ + "GoogleAssociationService" + ], + "description": "Google service verifying digital asset link associations between platforms", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "GoogleDocs", + "url": "https://docs.google.com/", + "instances": [ + "Mozilla/5.0 (compatible; GoogleDocs; apps-spreadsheets; +http://docs.google.com)" + ], + "description": "Google Docs crawler fetching web content for spreadsheet import functions", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "GoPay", + "url": "https://doc.gopay.com/", + "instances": [ + "GoPay" + ], + "description": "Payment gateway service verifying merchant websites and payment endpoints", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "GotSiteMonitor\\.com", + "url": "https://datadome.co/bots/gotsitemonitor/", + "instances": [ + "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.64 Safari/537.11 GotSiteMonitor.com" + ], + "description": "Website monitoring service tracking uptime and alerting on downtime", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "synthetic-monitoring-agent\\/", + "url": "https://grafana.com/", + "instances": [ + "synthetic-monitoring-agent/v0.20.1-0-g66af84ad (linux amd64; 66af84ad6a8755895d4b69606281ca7354c1589a; 2024-02-12 16:50:07+00:00; +https://github.com/grafana/synthetic-monitoring-agent)" + ], + "description": "Grafana synthetic monitoring agent probing endpoints for availability", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Grammarly\\/", + "url": "https://datadome.co/bots/grammarly/", + "instances": [ + "Grammarly/1.0 (http://www.grammarly.com)" + ], + "description": "Writing assistant crawler fetching web content for grammar analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "gregcrawler", + "url": "https://datadome.co/bots/gregcrawler/", + "instances": [ + "gregcrawler" + ], + "description": "Web crawler indexing content for data aggregation and analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "GroovinaAdsbot\\/", + "url": "https://www.groovinads.com/", + "instances": [ + "Mozilla/5.0 (compatible; GroovinaAdsbot/1.0; +https://www.groovinads.com/en/#bot)" + ], + "description": "Advertising platform crawler indexing products and pricing for campaigns", + "addition_date": "2026/04/17", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Grover\\/", + "url": "https://datadome.co/bots/grover-bot/", + "instances": [ + "Grover/Grover-1.18 (Web Crawler)" + ], + "description": "Anonymous web crawler likely used for research or content scraping", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "GTmetrix", + "url": "https://gtmetrix.com/", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36 GTmetrix" + ], + "description": "Performance testing tool analyzing page speed and optimization recommendations", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "GuestpostsBot\\/", + "url": "https://guestposts.com.br/", + "instances": [ + "Mozilla/5.0 (compatible; GuestpostsBot/2.0; +https://guestposts.com.br/blog/robot/)" + ], + "description": "Guest posting platform crawler discovering and indexing blog opportunities", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Gulper Web Bot", + "url": "https://datadome.co/bots/gulperbot/", + "instances": [ + "Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)" + ], + "description": "Academic web crawler collecting data for research and link analysis", + "addition_date": "2026/04/17", + "tags": [ + "academic" + ] + }, + { + "pattern": "Verity\\/", + "url": "https://gumgum.com/verity", + "instances": [ + "Verity/1.1 (https://gumgum.com/verity; verity-support@gumgum.com)" + ], + "description": "Contextual advertising crawler analyzing page content for brand safety", + "addition_date": "2026/04/17", + "tags": [ + "advertising" + ] + }, + { + "pattern": "HappyWing", + "url": "https://datadome.co/bots/happywing/", + "instances": [ + "HappyWing" + ], + "description": "Web crawler indexing content for data aggregation and analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "harsilbot\\/", + "url": "http://www.harsil.com/bot", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36 (compatible; harsilbot/1.1; +http://www.harsil.com/bot)" + ], + "description": "Web crawler indexing and collecting data for SEO analysis purposes", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "HawaiiBot", + "url": "https://datadome.co/bots/hawaiibot/", + "instances": [ + "HawaiiBot" + ], + "description": "Web crawler gathering content for data aggregation and indexing", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "hCardValidator", + "url": "http://hcard.geekhood.net/", + "instances": [ + "hCardValidator/1 PHP/5 (http://hcard.geekhood.net)" + ], + "description": "Microformat validator checking hCard contact data on web pages", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Hello World", + "url": "https://datadome.co/bots/hello-world/", + "instances": [ + "Hello World" + ], + "description": "Test crawler sending generic requests to verify web server responses", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "HelloworkJobPostingBot\\/", + "url": "https://www.hellowork-group.com/en/", + "instances": [ + "HelloworkJobPostingBot/1.0" + ], + "description": "Job board crawler indexing employment listings and posting data", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "HetrixTools", + "url": "https://hetrixtools.com/uptime-monitoring-bot.html", + "instances": [ + "HetrixTools Uptime Monitoring Bot. https://hetrix.tools/uptime-monitoring-bot.html" + ], + "description": "Uptime monitoring bot checking server availability and alerting downtime", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "HIFIBot\\/", + "url": "https://hi.fi/", + "instances": [ + "Mozilla/5.0 (compatible; HIFIBot/1; +https://hi.fi)" + ], + "description": "Web crawler indexing content for search and information aggregation", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Hlidam\\.to robot", + "url": "https://hlidam.to/", + "instances": [ + "Hlidam.to robot" + ], + "description": "Czech job listing crawler indexing employment offers and postings", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Honeybadger Uptime Check", + "url": "https://www.honeybadger.io/", + "instances": [ + "Honeybadger Uptime Check" + ], + "description": "Error monitoring service checking website availability and health", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "HostTracker\\/", + "url": "https://datadome.co/bots/hosttracker/", + "instances": [ + "Mozilla/5.0 (compatible;HostTracker/2.0;+http://www.host-tracker.com/)" + ], + "description": "Website monitoring service checking availability from multiple global locations", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Hotjar", + "url": "https://www.hotjar.com/", + "instances": [ + "Mozilla/5.0 (iPhone; CPU iPhone OS 11_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Hotjar Version/11.0 Mobile/15E148 Safari/604.1" + ], + "description": "Analytics platform crawler collecting heatmap and behavior tracking data", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "hstspreload-bot", + "url": "https://hstspreload.org/", + "instances": [ + "hstspreload-bot" + ], + "description": "HSTS preload list checker verifying HTTPS enforcement across websites", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Huckabot\\/", + "url": "https://huckabuy.com/", + "instances": [ + "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; Huckabot/0.0; +https://huckabuy.com/)" + ], + "description": "E-commerce SEO crawler analyzing product pages for optimization insights", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Hype Machine\\/", + "url": "https://hypem.com/latest", + "instances": [ + "Hype Machine/4.0 hypem.com (anthony@hypem.com)" + ], + "description": "Music blog aggregator crawler indexing tracks and artist content", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Web Screen Service By hyperhost", + "url": "https://datadome.co/bots/hyperhost-ua-crawler/", + "instances": [ + "Web Screen Service By hyperhost.ua" + ], + "description": "Web screenshot service capturing page visuals for hosting platform", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "AdsBot-IAB", + "url": "https://iabtechlab.com/ads-txt/", + "instances": [ + "AdsBot-IAB" + ], + "description": "IAB technology crawler verifying ads.txt files for programmatic advertising", + "addition_date": "2026/04/17", + "tags": [ + "advertising" + ] + }, + { + "pattern": "iAskBot\\/", + "url": "https://datadome.co/bots/iaskspider/", + "instances": [ + "Mozilla/5.0 AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; iAskBot/1.0; +https://iask.ai/) Chrome/120.0.6099.119 Safari/605.1.15" + ], + "description": "AI search engine crawler indexing web content for question answering", + "addition_date": "2026/04/17", + "tags": [ + "search-engine", + "ai-crawler" + ] + }, + { + "pattern": "IBM Crawler", + "url": "https://datadome.co/bots/ibm-crawler/", + "instances": [ + "IBM Crawler" + ], + "description": "IBM enterprise crawler indexing content for Watson and cloud services", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "IFTTT\\/", + "url": "https://ifttt.com/feed/details", + "instances": [ + "IFTTT/1.0 (https://ifttt.com/support)" + ], + "description": "Automation platform crawler fetching feeds for conditional workflow triggers", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "ImageFetcher\\/", + "url": "http://wsrv.nl/", + "instances": [ + "Mozilla/5.0 (compatible; ImageFetcher/9.0; +http://wsrv.nl/)" + ], + "description": "Image proxy service fetching and caching images for web applications", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "ImageMind", + "url": "https://datadome.co/bots/imagemind/", + "instances": [ + "ImageMind" + ], + "description": "Image analysis crawler collecting visual data for AI training purposes", + "addition_date": "2026/04/17", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "img2dataset", + "url": "https://github.com/rom1504/img2dataset", + "instances": [ + "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0 (compatible; img2dataset; +https://github.com/rom1504/img2dataset)" + ], + "description": "Open source tool downloading images for machine learning dataset creation", + "addition_date": "2026/04/17", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "imgproxy\\/", + "url": "https://imgproxy.net/", + "instances": [ + "imgproxy/3.21.0" + ], + "description": "Image processing proxy resizing and optimizing images for web delivery", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "impendoom-bot\\/", + "url": "https://impendoom.com/", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; impendoom-bot/0.1.0; +https://impendoom.com/) Safari/537.36" + ], + "description": "Web crawler indexing content for competitive intelligence and data analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "IndeedJobBot", + "url": "https://www.indeed.com/about/indeed-crawlers", + "instances": [ + "IndeedJobBot" + ], + "description": "Job search engine crawler indexing employment listings and company pages", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Innguma\\/", + "url": "https://factory.innguma.com/fetcher/", + "instances": [ + "Innguma/1.0 (+https://factory.innguma.com/fetcher/)" + ], + "description": "Content fetcher crawler gathering web data for information extraction", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Instapaper\\/", + "url": "https://www.instapaper.com/publishers", + "instances": [ + "Instapaper/4.0Instaparser/1.0" + ], + "description": "Read-later service crawler fetching articles for offline reading storage", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Integromat\\/", + "url": "https://developers.make.com/api-documentation", + "instances": [ + "Integromat/production" + ], + "description": "Automation platform crawler fetching web data for workflow integration", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "intelx\\.io_bot", + "url": "https://intelx.io/", + "instances": [ + "Mozilla/5.0 (compatible; intelx.io_bot +https://intelx.io)" + ], + "description": "Intelligence search engine crawler indexing leaks and dark web content", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "internetVista monitor", + "url": "https://datadome.co/bots/internetvista-monitor/", + "instances": [ + "internetVista monitor (Mozilla compatible)" + ], + "description": "Website monitoring service checking availability performance and SSL status", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Irokez\\.cz monitoring", + "url": "https://datadome.co/bots/irokez-cz-monitoring/", + "instances": [ + "Irokez.cz monitoring v1.2 - (http://www.irokez.cz, Irokez.cz, crawl)" + ], + "description": "Czech web monitoring service tracking availability and performance metrics", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "IsDownBot\\/", + "url": "https://help.isdown.app/custom-monitors/isdownbot", + "instances": [ + "Mozilla/5.0 (compatible; IsDownBot/1.0;)" + ], + "description": "Downtime monitoring bot checking website and service availability status", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "ISSCyberRiskCrawler\\/", + "url": "https://datadome.co/bots/isscyberriskcrawler/", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 13_0_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36 ISSCyberRiskCrawler/1.1.4" + ], + "description": "Cybersecurity risk crawler scanning websites for vulnerabilities and exposures", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "iubenda-radar\\/", + "url": "https://www.iubenda.com/", + "instances": [ + "iubenda-radar/2.25.2" + ], + "description": "Compliance platform crawler auditing privacy policies and cookie consent", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "UptimeBot\\/", + "url": "https://jaggedpixel.co/", + "instances": [ + "Mozilla/5.0 (compatible; UptimeBot/1.0; +https://www.getuptime.co)" + ], + "description": "Uptime monitoring service verifying website availability and response latency", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "jetmon\\/", + "url": "https://automattic.com/", + "instances": [ + "jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)" + ], + "description": "WordPress Jetpack site uptime monitor and health checking service", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "jobswithgptcom-bot", + "url": "https://jobswithgpt.com/bot.html", + "instances": [ + "jobswithgptcom-bot" + ], + "description": "Job portal crawler indexing career pages and employment listings", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Jumio", + "url": "https://github.com/Jumio/implementation-guides/blob/master/netverify/callback.md", + "instances": [ + "Jumio" + ], + "description": "Identity and KYC verification service web content analyzer", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Kagibot\\/", + "url": "https://kagi.com/", + "instances": [ + "Mozilla/5.0 (compatible; Kagibot/1.0; +https://kagi.com/bot)" + ], + "description": "Privacy-focused search engine crawler indexing content for rankings", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "KangarooBot\\/", + "url": "https://datadome.co/bots/kangaroo-bot/", + "instances": [ + "Mozilla/5.0 (compatible; KangarooBot/1.0; +https://kangaroobot.com)" + ], + "description": "Data scraping and content aggregation crawler mimicking human behavior", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "KargoBot-Artemis", + "url": "https://www.kargo.com/", + "instances": [ + "Mozilla/5.0 (iPhone; CPU iPhone OS 13_2_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.3 Mobile/15E148 Safari/604.1 (compatible; KargoBot-Artemis-Mobile; +https://www.kargo.com/kargobot-artemis)" + ], + "description": "Ad-tech crawler analyzing pages for contextual targeting and brand safety", + "addition_date": "2026/04/17", + "tags": [ + "advertising" + ] + }, + { + "pattern": "kazbtbot\\/", + "url": "http://kazbt.com/", + "instances": [ + "Mozilla/5.0 (compatible; kazbtbot/0.1; +http://kazbt.com/)" + ], + "description": "Web crawler for content indexing and data aggregation purposes", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "keycdn-tools\\/", + "url": "https://tools.keycdn.com/", + "instances": [ + "keycdn-tools/speed" + ], + "description": "CDN performance monitoring measuring website load times and metrics", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "keys-so-bot", + "url": "https://datadome.co/bots/keys-so-bot/", + "instances": [ + "Mozilla/5.0 (keys-so-bot)" + ], + "description": "Automated crawler for indexing and data collection purposes", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "kinsta-bot", + "url": "https://kinsta.com/", + "instances": [ + "kinsta-bot" + ], + "description": "Hosting platform crawler for cache warming and SEO audit checks", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Klaviyo\\/", + "url": "https://www.klaviyo.com/", + "instances": [ + "Klaviyo/1.0" + ], + "description": "Marketing automation bot fetching product data for email campaigns", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Kukei\\.eu-Bot\\/", + "url": "https://kukei.eu/", + "instances": [ + "Mozilla/5.0 (compatible; Kukei.eu-Bot/0.2; +https://kukei.eu)" + ], + "description": "Web crawler indexing and analyzing page content for search purposes", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "LAC_IAHarvester", + "url": "https://library-archives.canada.ca/eng/services/government-canada/web-social-media-preservation-program/Pages/web-archive.aspx", + "instances": [ + "Mozilla/5.0 (compatible; LAC_IAHarvester/3.3.0; +https://library-archives.canada.ca/eng/services/government-canada/web-social-media-preservation-program/Pages/web-archive.aspx)" + ], + "description": "Canadian library web archiving bot preserving government websites", + "addition_date": "2026/04/17", + "tags": [ + "archiver" + ] + }, + { + "pattern": "LastModBot\\/", + "url": "https://last-modified.com/", + "instances": [ + "Mozilla/5.0 (compatible; LastModBot/0.1; +https://last-modified.com/en)" + ], + "description": "Web crawler detecting page freshness by analyzing last modified headers", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "LegalMonster", + "url": "https://www.legalmonster.com/", + "instances": [ + "LegalMonster" + ], + "description": "Compliance monitoring scanner detecting cookies and tracking consent", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Let's Encrypt", + "url": "https://letsencrypt.org/", + "instances": [ + "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" + ], + "description": "SSL certificate validation bot verifying domain control for TLS certs", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Level9SearchBot\\/", + "url": "https://level9.com/", + "instances": [ + "Mozilla/5.0 (compatible; Level9SearchBot/2-1.5)" + ], + "description": "Search and OSINT crawler indexing public content for search purposes", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "loc\\.gov\\/programs\\/web-archiving", + "url": "https://www.loc.gov/programs/web-archiving/for-site-owners", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36 (+https://www.loc.gov/programs/web-archiving/for-site-owners/)" + ], + "description": "Government web archiver preserving cultural content for long-term access", + "addition_date": "2026/04/17", + "tags": [ + "archiver" + ] + }, + { + "pattern": "LinerBot\\/", + "url": "https://docs.getliner.com/docs/linerbot", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; LinerBot/1.0; +https://docs.getliner.com/docs/linerbot)" + ], + "description": "AI workspace crawler indexing web content for search and discovery", + "addition_date": "2026/04/17", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "LinkTiger", + "url": "https://linktiger.com/", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.87 Safari/537.36 LinkTiger" + ], + "description": "Broken link detection crawler auditing website link integrity", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "LinkAce\\/", + "url": "https://www.linkace.org/", + "instances": [ + "LinkAce/1 (https://github.com/Kovah/LinkAce)" + ], + "description": "Self-hosted bookmark archive crawler preserving web content", + "addition_date": "2026/04/17", + "tags": [ + "archiver" + ] + }, + { + "pattern": "LinksIndexerBot\\/", + "url": "https://linksindexer.com/bot", + "instances": [ + "Mozilla/5.0 (compatible; LinksIndexerBot/1.0; +http://linksindexer.com/bot)" + ], + "description": "SEO crawler indexing backlinks and submitting URLs for search engines", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "LinkWalker\\/", + "url": "https://datadome.co/bots/linkwalker/", + "instances": [ + "Mozilla/5.0 (compatible; LinkWalker/2.0; +http://www.seventwentyfour.com)" + ], + "description": "Automated web crawler for content and data collection purposes", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "LogicMonitor", + "url": "https://www.logicmonitor.com/support/about-logicmonitor/overview/logicmonitor-public-ip-addresses-dns-names", + "instances": [ + "LogicMonitor SiteMonitor/1.0" + ], + "description": "Synthetic monitoring bot checking website and API availability", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "LoomlyBot", + "url": "https://www.loomly.com/", + "instances": [ + "LoomlyBot" + ], + "description": "Social media management metadata fetcher for link previews", + "addition_date": "2026/04/17", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Macrobondbot", + "url": "https://www.macrobond.com/", + "instances": [ + "Mozilla/5.0 (compatible; Macrobondbot +http://redir.macrobond.com/go/bot/)" + ], + "description": "Financial data crawler gathering macroeconomic time-series data", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "MADBbot\\/", + "url": "https://madb.zapto.org/bot.html", + "instances": [ + "MADBbot/0.1 (Gathering webpages for data analytics; https://madb.zapto.org/bot.html; ma-db-crawl@googlegroups.com)" + ], + "description": "Web crawler gathering data for analytics and research purposes", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Magellan", + "url": "https://datadome.co/bots/magellan/", + "instances": [ + "Magellan" + ], + "description": "Web content indexing and competitive analysis crawler tool", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "magicsearchdev\\/", + "url": "https://datadome.co/bots/magicsearchdev/", + "instances": [ + "magicsearchdev/1.0" + ], + "description": "Web crawler for content indexing and data aggregation purposes", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Magnet\\.me-web\\/", + "url": "https://magnet.me/bot.html", + "instances": [ + "Magnet.me-web/1.0 (+https://magnet.me/bot.html)" + ], + "description": "Graduate recruitment platform crawling job postings and employer data", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "MainWP\\/", + "url": "https://www.dshost.com.au/", + "instances": [ + "Mozilla/5.0 (compatible; MainWP/4.2.7.1; +http://mainwp.com)" + ], + "description": "WordPress management platform crawler auditing child site health", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Make\\/", + "url": "https://www.make.com/en/", + "instances": [ + "Make/production" + ], + "description": "Automation platform crawler validating endpoints and collecting metadata", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "ManageWP", + "url": "https://managewp.com/", + "instances": [ + "ManageWP" + ], + "description": "WordPress dashboard scanner verifying site availability and health", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "MarketGoo\\/", + "url": "https://www.marketgoo.com/", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/600.1.17 (KHTML, like Gecko) Version/7.1 Safari/537.85.10 MarketGoo/2.1" + ], + "description": "SEO audit spider evaluating technical search engine optimization signals", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "MarketingMiner", + "url": "https://datadome.co/bots/marketingminer-bot/", + "instances": [ + "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36(compatible; MarketingMiner v2.0; +https://www.marketingminer.com)" + ], + "description": "Digital marketing analytics crawler gathering SEO and performance data", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "dbot\\)", + "url": "https://www.marsflag.com/ja/marsfinder/", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) (dbot)" + ], + "description": "Link preview and content discovery metadata harvester bot", + "addition_date": "2026/04/17", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Mattermost-Bot\\/", + "url": "https://datadome.co/bots/mattermost-bot/", + "instances": [ + "Mattermost-Bot/1.1" + ], + "description": "Team communication platform fetching URLs for preview and automation", + "addition_date": "2026/04/17", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Mavifinds", + "url": "https://brandimi.com/mavifinds-bot/", + "instances": [ + "Mavifinds Bot" + ], + "description": "Shopping discovery crawler indexing product prices and availability", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "MB-LinkChecker", + "url": "https://www.marcobeierer.com/tools", + "instances": [ + "MB-LinkChecker" + ], + "description": "Website crawler for structure analysis and metadata collection", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "MedialogiaBot", + "url": "https://www.mlg.ru/", + "instances": [ + "Mozilla/5.0 (compatible; MedialogiaBot; +http://home.prod.mlg.ru/bots.txt)" + ], + "description": "Media monitoring crawler tracking news and brand mentions online", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "MediaMonitoringBot\\/", + "url": "https://mediamonitoringbot.com/", + "instances": [ + "MediaMonitoringBot/1.1 (+https://mediamonitoringbot.com/crawler; crawler@mediamonitoringbot.com)" + ], + "description": "OSINT crawler harvesting mentions from news blogs and forums", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "MediavineMetadataParser\\/", + "url": "https://radar.cloudflare.com/bots/directory/mediavine-metadata-parser/mediavine.com", + "instances": [ + "MediavineMetadataParser/7.7.3" + ], + "description": "Ad network metadata extractor gathering schema and preview data", + "addition_date": "2026/04/17", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Pywikibot\\/", + "url": "https://www.mediawiki.org/wiki/Manual:Pywikibot", + "instances": [ + "wlc3 Pywikibot/9.0.0.dev0 (g18371) requests/2.31.0 Python/3.10.12.final.0" + ], + "description": "MediaWiki crawler indexing wiki content for search and analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "CentComBot\\/", + "url": "https://centcom.melonmesa.com/", + "instances": [ + "Mozilla/5.0 (compatible; CentComBot/" + ], + "description": "Content indexing and dataset building crawler tool", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "MergadoBot", + "url": "https://datadome.co/bots/mergadobot/", + "instances": [ + "MergadoBot (+http://mergado.cz)" + ], + "description": "E-commerce product data feed optimization and aggregation crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Meta-ExternalHit\\/", + "url": "https://datadome.co/bots/meta-externalagent/", + "instances": [ + "Mozilla/5.0 (compatible; Meta-ExternalHit/1.1; +http://www.facebook.com/externalhit_uatext.php)" + ], + "description": "Meta platform bot fetching link previews and metadata for sharing", + "addition_date": "2026/04/17", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Metorik", + "url": "https://metorik.com/", + "instances": [ + "Metorik API Client/2.0.1" + ], + "description": "E-commerce analytics platform crawler for WooCommerce and Shopify stores", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "MgidBot", + "url": "https://www.mgid.com", + "instances": [ + "MgidBot 1.0" + ], + "description": "Native advertising platform crawler analyzing pages for targeting", + "addition_date": "2026/04/17", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Miniature\\.io\\/", + "url": "https://miniature.io/", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) Miniature.io/6.0 AppleWebKit/537.36 (KHTML, like Gecko) QtWebEngine/5.12.4 Chrome/69.0.3497.128 Safari/537.36" + ], + "description": "Web page archiving and thumbnail generation bot tool", + "addition_date": "2026/04/17", + "tags": [ + "archiver" + ] + }, + { + "pattern": "mirrorweb\\.com", + "url": "https://www.mirrorweb.com/", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36 +https://www.mirrorweb.com" + ], + "description": "Web archiving platform crawler capturing timestamped website snapshots", + "addition_date": "2026/04/17", + "tags": [ + "archiver" + ] + }, + { + "pattern": "MissinglettrBot\\/", + "url": "https://missinglettr.com/", + "instances": [ + "Mozilla/5.0 (compatible; MissinglettrBot/2.0; +http://missinglettr.com/bot/)" + ], + "description": "Social media automation platform crawler for blog and feed indexing", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "crawler_eb_germany", + "url": "https://datadome.co/bots/crawler_eb_germany/", + "instances": [ + "crawler_eb_germany_2.0" + ], + "description": "General web crawler for indexing and data collection purposes", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "ModularConnector\\/", + "url": "https://uniqoders.com/", + "instances": [ + "ModularConnector/1.0 (Linux)" + ], + "description": "Distributed modular crawler framework for content processing", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Mollie HTTP client", + "url": "https://www.mollie.com/", + "instances": [ + "Mollie HTTP client/1.0" + ], + "description": "Payment service provider verifying merchant sites and compliance", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Monibot", + "url": "https://datadome.co/bots/monibot/", + "instances": [ + "Monibot" + ], + "description": "Web crawler for competitive analysis and price monitoring", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "monitis -", + "url": "https://www.monitis.com/", + "instances": [ + "Mozilla/5.0 (compatible; monitis - premium monitoring service; http://www.monitis.com)" + ], + "description": "Synthetic monitoring bot checking website availability and performance", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "MonitoRSS\\/", + "url": "https://monitorss.xyz/", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0 MonitoRSS/1.0" + ], + "description": "Discord bot crawler fetching RSS feeds for channel notifications", + "addition_date": "2026/04/17", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "MonSpark\\/", + "url": "https://monspark.com/", + "instances": [ + "Mozilla/5.0+(compatible; MonSpark/1.0; http://www.monspark.com/)" + ], + "description": "Website monitoring service bot checking uptime and performance", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "montastic-monitor", + "url": "https://montastic.com/", + "instances": [ + "montastic-monitor www.montastic.com" + ], + "description": "Uptime monitoring bot tracking website availability and latency", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "MonTools\\.com", + "url": "https://datadome.co/bots/montools/", + "instances": [ + "Mozilla/5.0 (compatible; MonTools.com)" + ], + "description": "Web crawler for scanning and indexing website pages", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "MotoMinerBot\\/", + "url": "https://motominer.com/Bot", + "instances": [ + "Mozilla/5.0 (compatible; MotoMinerBot/1.0; +https://motominer.com/Bot)" + ], + "description": "Automotive marketplace crawler indexing vehicle listings and prices", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "MRGbot\\/", + "url": "https://www.mrg.ro/", + "instances": [ + "Mozilla/5.0 (compatible; MRGbot/1.0; +https://www.mrg.ro/bot.html)" + ], + "description": "Security testing lab crawler for anti-fraud control evaluation", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "MxToolbox", + "url": "https://datadome.co/bots/mxtoolbox-bot/", + "instances": [ + "Mozilla/5.0+(compatible; MxToolbox/Beta7; http://www.mxtoolbox.com/)" + ], + "description": "Email delivery and reputation monitoring for organizations", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "my-tiny-bot", + "url": "https://datadome.co/bots/my-tiny-bot/", + "instances": [ + "my-tiny-bot" + ], + "description": "Lightweight web crawler for content indexing and analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "MyBot\\/", + "url": "https://datadome.co/bots/mybot/", + "instances": [ + "Mozilla/5.0 (compatible; MyBot/1.0)" + ], + "description": "Web crawler for indexing and data aggregation purposes", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "nbertaupete95", + "url": "https://datadome.co/bots/nbertaupete95/", + "instances": [ + "Mozilla/5.0/Firefox/42.0 - nbertaupete95(at)gmail.com" + ], + "description": "Web crawler for SEO and competitive analysis purposes", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "NetAPI", + "url": "https://datadome.co/bots/netapi/", + "instances": [ + "NetAPI v1" + ], + "description": "Automated crawler interacting with websites for data collection", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "NetpeakCheckerBot\\/", + "url": "https://netpeaksoftware.com/", + "instances": [ + "Mozilla/5.0 (compatible; NetpeakCheckerBot/3.6; +https://netpeaksoftware.com/checker)" + ], + "description": "SEO analysis crawler identifying technical issues and metrics", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "NetShelter ContentScan", + "url": "https://datadome.co/bots/netshelter-contentscan/", + "instances": [ + "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3 (NetShelter ContentScan)" + ], + "description": "Media network crawler categorizing content for ad targeting", + "addition_date": "2026/04/17", + "tags": [ + "advertising" + ] + }, + { + "pattern": "NETVIGIE", + "url": "https://netvigie.com/", + "instances": [ + "NETVIGIE" + ], + "description": "European performance and uptime monitoring synthetic crawler", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "NewRelicbot\\/", + "url": "https://newrelic.com/", + "instances": [ + "Mozilla/5.0 (compatible; NewRelicbot/2.1; +http://www.newrelic.com)" + ], + "description": "Application monitoring bot for uptime and performance checking", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "nyt_scraping", + "url": "https://int.nyt.com/assets/scraping.json", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 nyt_scraping/scraping@nytimes.com" + ], + "description": "News organization newsgathering crawler for reporting support", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "NewsNow\\/", + "url": "https://www.newsnow.co.uk/h/", + "instances": [ + "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT) NewsNow/1.0" + ], + "description": "UK news aggregation platform crawler for content discovery", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "NLNZ_IAHarvester", + "url": "https://natlib.govt.nz/publishers-and-authors/web-harvesting", + "instances": [ + "Mozilla/5.0 (compatible; NLNZ_IAHarvester2024/3.3.0; +https://natlib.govt.nz/publishers-and-authors/web-harvesting/domain-harvest)" + ], + "description": "New Zealand library digital preservation and archiving bot", + "addition_date": "2026/04/17", + "tags": [ + "archiver" + ] + }, + { + "pattern": "NodePing", + "url": "https://datadome.co/bots/nodeping/", + "instances": [ + "NodePing" + ], + "description": "Website monitoring service performing uptime checks globally", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "nomore404\\.com robot", + "url": "https://datadome.co/bots/nomore404-com-robot/", + "instances": [ + "Mozilla/5.0 (nomore404.com robot/1.1; +https://nomore404.com/)" + ], + "description": "Broken link detection crawler identifying HTTP 404 errors", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "noorobot", + "url": "https://noordigital.com/", + "instances": [ + "noorobot" + ], + "description": "Web scraper performing high-volume programmatic content collection", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Nooshub\\/", + "url": "https://www.nooshub.com/", + "instances": [ + "Mozilla/5.0 (compatible; Nooshub/1.0; +https://www.nooshub.com/statics/bots)" + ], + "description": "AI feed platform crawler indexing content and RSS feeds", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Notabot", + "url": "https://corp.helpfeel.com/ja/home", + "instances": [ + "Notabot" + ], + "description": "Content aggregation and metadata extraction crawler bot", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Novaact\\/", + "url": "https://datadome.co/bots/novaact/", + "instances": [ + "Mozilla/5.0 (compatible; Novaact/1.0; +https://www.novaact.com)" + ], + "description": "Web crawler for data aggregation and analysis purposes", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Novellum", + "url": "https://crawl.corp.novellum.ai/docs", + "instances": [ + "Novellum" + ], + "description": "AI data collection crawler for model training and enrichment", + "addition_date": "2026/04/17", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "NsToolsBot\\/", + "url": "https://ns.tools/", + "instances": [ + "NsToolsBot/analyse" + ], + "description": "Network and security analysis tool crawler gathering server data", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "nvdorz", + "url": "https://datadome.co/bots/nvdorz/", + "instances": [ + "nvdorz" + ], + "description": "Web crawler for content indexing and analysis purposes", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Odin;", + "url": "https://docs.getodin.com/", + "instances": [ + "Mozilla/5.0 (compatible; Odin; https://docs.getodin.com/)" + ], + "description": "Automated crawler for data collection and indexing purposes", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Offline Explorer", + "url": "https://datadome.co/bots/offline-explorer/", + "instances": [ + "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Offline Explorer; MSIECrawler)" + ], + "description": "Website downloader tool for offline archival and content analysis", + "addition_date": "2026/04/17", + "tags": [ + "archiver" + ] + }, + { + "pattern": "OhDear\\/", + "url": "https://ohdear.app/docs/faq/what-is-the-oh-dear-checker", + "instances": [ + "Mozilla/5.0 (compatible; OhDear/1.1; +https://ohdear.app/checkerOhDear.app (+https://ohdear.app/docs/checks/uptime)" + ], + "description": "Website monitoring bot scanning uptime and TLS certificate validity", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Omnisend\\/", + "url": "https://bots.omnisend.io/cf.txt", + "instances": [ + "Omnisend/1.0" + ], + "description": "Ecommerce marketing automation crawler fetching campaign URLs metadata", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Online Domain Tools", + "url": "https://datadome.co/bots/online-domain-tools/", + "instances": [ + "Mozilla/5.0 (compatible; Online Domain Tools - Server Monitor/1.0; +http://server-monitoring.online-domain-tools.com)" + ], + "description": "SEO analysis crawler identifying technical issues and metrics", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "WebCEO Online\\/", + "url": "https://datadome.co/bots/online-webceo-bot/", + "instances": [ + "Mozilla/5.0 (compatible; WebCEO Online/1.0; +http://online.webceo.com)" + ], + "description": "SEO platform crawler analyzing keywords and backlinks", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "OnlineOrNot\\.com_bot", + "url": "https://onlineornot.com", + "instances": [ + "OnlineOrNot.com_bot_1.0_(https://onlineornot.com)" + ], + "description": "Web crawler for systematic content browsing and indexing", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "OpenGraph\\.io\\/", + "url": "https://www.opengraph.io/", + "instances": [ + "Mozilla/5.0 (compatible; OpenGraph.io/1.1; +https://opengraph.io/ AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36" + ], + "description": "Open Graph metadata extractor for link preview generation", + "addition_date": "2026/04/17", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "OpenRSS", + "url": "https://openrss.org/", + "instances": [ + "OpenRSS" + ], + "description": "Feed discovery crawler converting websites into RSS feeds", + "addition_date": "2026/04/17", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "OpenVAS", + "url": "https://openvas.org/", + "instances": [ + "Mozilla/5.0 [en] (X11, U; OpenVAS)" + ], + "description": "Vulnerability scanner crawler identifying security weaknesses", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Owler \\(ows\\.eu", + "url": "https://ows.eu/owler", + "instances": [ + "Owler (ows.eu/owler)" + ], + "description": "European open search crawler for decentralized indexing", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Operator\\/", + "url": "https://datadome.co/bots/operator/", + "instances": [ + "Mozilla/5.0 (compatible; Operator/1.0; +https://www.example.com/bot.html)" + ], + "description": "Automated crawler performing repetitive internet tasks", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Orbbot\\/", + "url": "https://datadome.co/bots/orbbot/", + "instances": [ + "Mozilla/5.0 (compatible; Orbbot/1.1;)" + ], + "description": "Web crawler indexing content for SEO and analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "zebra-v2-bot", + "url": "https://datadome.co/bots/zebra-v2-bot/", + "instances": [ + "zebra-v2-bot" + ], + "description": "Web crawler for indexing and data collection purposes", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Orlo-LinkPreview\\/", + "url": "https://orlo.tech/", + "instances": [ + "Orlo-LinkPreview/1.0" + ], + "description": "Social platform link preview and metadata fetcher", + "addition_date": "2026/04/17", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "Cozi-iCalendar-FeedReader", + "url": "https://ourfamilywizard.com/", + "instances": [ + "Cozi-iCalendar-FeedReader" + ], + "description": "Co-parenting platform crawler fetching shared content metadata", + "addition_date": "2026/04/17", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "OutsellURLValidator", + "url": "https://www.outsell.com/", + "instances": [ + "OutsellURLValidator" + ], + "description": "Digital marketing crawler validating campaign link health", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Overcast\\/", + "url": "https://overcast.fm/", + "instances": [ + "Overcast/1.0 Podcast Sync (123 subscribers; feed-id=456789; +http://overcast.fm/)" + ], + "description": "Podcast platform crawler discovering and refreshing feeds", + "addition_date": "2026/04/17", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "PRTGCloudBot\\/", + "url": "https://www.paessler.com/", + "instances": [ + "Mozilla/5.0 (compatible; PRTGCloudBot/1.0; +http://www.paessler.com/prtgcloudbot; for_[edf7e62f223b268942b7efa36b6be1e305fcdadb])" + ], + "description": "Paessler cloud monitoring bot checking uptime and health", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Pagespeed\\/", + "url": "http://www.pagespeed.de/", + "instances": [ + "Mozilla/5.0 (compatible; Pagespeed/1.1 Fetcher; +http://www.pagespeed.de)" + ], + "description": "Performance analysis crawler assessing web page metrics", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "PanguBot", + "url": "https://datadome.co/bots/pangubot/", + "instances": [ + "Mozilla/5.0 (compatible; PanguBot +https://www.pangutech.com)" + ], + "description": "Chinese search engine crawler indexing websites", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Panopta", + "url": "https://www.panopta.com/", + "instances": [ + "Panopta v1.1" + ], + "description": "Synthetic monitoring bot verifying uptime and transactions", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Paqlebot\\/", + "url": "http://www.paqle.dk/about/paqlebot", + "instances": [ + "Mozilla/5.0 (compatible; Paqlebot/2.0; +http://www.paqle.dk/about/paqlebot)" + ], + "description": "Web crawler gathering data for analytics and research", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "parse\\.ly scraper\\/", + "url": "https://www.parse.ly/", + "instances": [ + "Mozilla/5.0 (compatible; parse.ly scraper/0.16; +http://parsely.com)" + ], + "description": "Content analytics platform crawler for metadata extraction", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "PayPal\\/", + "url": "https://www.paypal.com/ipn", + "instances": [ + "PayPal/AUHR-214.0-51787073PayPal IPN ( https://www.paypal.com/ipn )" + ], + "description": "Payment processor bot validating endpoints and compliance", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "PDF24 URL To PDF", + "url": "https://tools.pdf24.org/webpage-to-pdf", + "instances": [ + "PDF24 URL To PDF" + ], + "description": "PDF conversion crawler capturing web pages as documents", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "PingAdmin\\.Ru\\/", + "url": "http://ping-admin.ru/", + "instances": [ + "Mozilla/5.0 (compatible; PingAdmin.Ru/1.2; +http://pingadmin.ru/free_test/)" + ], + "description": "Uptime monitoring service bot checking website availability", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "pingping\\.io\\/", + "url": "https://pingping.io/", + "instances": [ + "pingping.io/1.0" + ], + "description": "Lightweight monitoring bot validating URL availability", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "PlayStore-Google", + "url": "https://support.google.com/webmasters/answer/1061943", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.6478.182 Safari/537.36 PlayStore-Google" + ], + "description": "Google Play Store crawler indexing apps and content", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Plesk screenshot bot", + "url": "https://www.plesk.com/", + "instances": [ + "Plesk screenshot bot https://support.plesk.com/hc/en-us/articles/10301006946066" + ], + "description": "Hosting platform management crawler for website auditing", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "PocketCasts\\/", + "url": "https://support.pocketcasts.com/knowledge-base/pocket-casts-feed-parser/", + "instances": [ + "PocketCasts/1.0 (Pocket Casts Feed Parser; +http://pocketcasts.com/)" + ], + "description": "Podcast platform crawler parsing feeds and episodes", + "addition_date": "2026/04/17", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Potions\\/", + "url": "https://get-potions.com/", + "instances": [ + "Potions/1.0.0" + ], + "description": "E-commerce personalization crawler ingesting catalogs", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "PressEngineBot", + "url": "https://www.pressengine.net/", + "instances": [ + "PressEngineBot (+http://pressengine.net/crawl-policy)" + ], + "description": "Press release aggregation crawler for content syndication", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "PricedroneShoppingBot\\/", + "url": "http://pricedrone.com/robot/", + "instances": [ + "Mozilla/5.0 (compatible; PricedroneShoppingBot/1.0; +http://pricedrone.com/robot/)" + ], + "description": "Price comparison crawler aggregating product pricing", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "PriEcoBot\\/", + "url": "https://datadome.co/bots/priecobot/", + "instances": [ + "Mozilla/5.0 (compatible; PriEcoBot/1.0; +https://prieco.net)" + ], + "description": "Web crawler for content aggregation and analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "PrintFriendly\\.com", + "url": "https://www.printfriendly.com/", + "instances": [ + "Mozilla/5.0 (X11; Linux i586; rv:31.0) Gecko/20100101 Firefox/31.0 PrintFriendly.com" + ], + "description": "Print optimization crawler converting pages to printable format", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Pro-Sitemaps\\/", + "url": "https://pro-sitemaps.com/", + "instances": [ + "Mozilla/5.0 (compatible; Pro Sitemaps Generator; pro-sitemaps.com) Gecko Pro-Sitemaps/1.0" + ], + "description": "Sitemap generation crawler discovering and indexing URLs", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "ProbelySPDR\\/", + "url": "https://probely.com/sos", + "instances": [ + "Mozilla/5.0 (compatible; +https://probely.com/sos) ProbelySPDR/0.1.0" + ], + "description": "Security testing crawler identifying web vulnerabilities", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "ProjectShield-UrlCheck", + "url": "https://projectshield.withgoogle.com/", + "instances": [ + "Mozilla/5.0 (compatible; ProjectShield-UrlCheck; +http://g.co/projectshield)" + ], + "description": "Google DDoS protection bot validating customer origins", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Blackbox Exporter\\/", + "url": "https://github.com/prometheus/blackbox_exporter", + "instances": [ + "Blackbox Exporter/0.20.0-rc.0" + ], + "description": "Prometheus monitoring crawler checking endpoint health", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Protopage\\/", + "url": "https://datadome.co/bots/protopage/", + "instances": [ + "Protopage/3.0 (http://www.protopage.com)" + ], + "description": "Start page and RSS reader crawler aggregating content", + "addition_date": "2026/04/17", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "PS_Daily\\/", + "url": "https://datadome.co/bots/ps-daily/", + "instances": [ + "PS_Daily/1.0 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36" + ], + "description": "Web crawler for data collection and analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "pulsetic\\.com", + "url": "https://pulsetic.com/", + "instances": [ + "pulsetic.com (+https://pulsetic.com)" + ], + "description": "Uptime monitoring bot tracking website availability", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "PWABuilderHttpAgent", + "url": "https://www.pwabuilder.com/", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36 Edg/96.0.1054.57 PWABuilderHttpAgent" + ], + "description": "Microsoft PWA readiness evaluation crawler tool", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "QualifiedBot\\/", + "url": "https://www.qualified.com/legal/qualified-crawler-user-agent", + "instances": [ + "QualifiedBot/1.0", + "Mozilla/5.0 (compatible; QualifiedBot/1.0; +https://www.qualified.com/legal/qualified-crawler-user-agent)" + ], + "description": "Conversational sales platform crawler inventorying site URLs", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Quantcastbot\\/", + "url": "http://www.quantcast.com/bot", + "instances": [ + "Quantcastbot/1.0 (+http://www.quantcast.com/bot)" + ], + "description": "Audience analytics crawler for ad measurement and targeting", + "addition_date": "2026/04/17", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Rackspace Monitoring\\/", + "url": "https://support.rackspace.com/how-to/about-the-rackspace-monitoring-agent/", + "instances": [ + "Rackspace Monitoring/1.1 (https://monitoring.api.rackspacecloud.com)" + ], + "description": "Cloud infrastructure monitoring bot checking endpoint availability", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "rakutenusabot-image\\/", + "url": "https://product-image.ebates.com/item-gsp/rakutenusabot.html", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; rakutenusabot-image/1.0) Chrome/114.0.0.0 Safari/537.36" + ], + "description": "E-commerce platform crawler extracting product images", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "top100\\.rambler\\.ru crawler", + "url": "https://datadome.co/bots/rambler-bot/", + "instances": [ + "Mozilla/5.0 (compatible; top100.rambler.ru crawler)" + ], + "description": "Russian search engine crawler indexing web content", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "RankurBot\\/", + "url": "http://rankur.com/technology.html", + "instances": [ + "RankurBot/3.3 (+http://rankur.com)" + ], + "description": "Social media and reputation monitoring crawler collecting data", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "RavenCrawler\\/", + "url": "https://datadome.co/bots/ravencrawler/", + "instances": [ + "Mozilla/5.0 (compatible; RavenCrawler/2.0; +https://raventools.com/seo-website-auditor/)" + ], + "description": "SEO auditing crawler analyzing website optimization signals", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Readable\\/", + "url": "https://readable.com/", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 Readable/1.1.4" + ], + "description": "Content analysis crawler computing readability and metrics", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Recurly Webhooks\\/", + "url": "https://docs.recurly.com/docs/webhooks", + "instances": [ + "Recurly Webhooks/2.0 (+https://docs.recurly.com/push-notifications)" + ], + "description": "Billing platform webhook validation and monitoring bot", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "RED\\/", + "url": "https://redbot.org/", + "instances": [ + "RED/2.0.14 (https://redbot.org/)" + ], + "description": "HTTP response testing and validation analyzer", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Reelevant\\/", + "url": "https://reelevant.com/", + "instances": [ + "Reelevant/1.0" + ], + "description": "Personalization platform fetching URLs for email and content", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "remove\\.bg\\/", + "url": "https://www.remove.bg/", + "instances": [ + "remove.bg/1.0 background removal website" + ], + "description": "Image editing service crawler accessing and processing images", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Retool\\/", + "url": "https://docs.tryretool.com/docs/apis", + "instances": [ + "Retool/2.0 (+https://docs.tryretool.com/docs/apis)" + ], + "description": "Internal tools platform fetching page metadata and content", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "RetroListeCOM\\/", + "url": "https://retroliste.com/", + "instances": [ + "Mozilla/5.0 (compatible; RetroListeCOM/1.0)" + ], + "description": "Directory and price comparison crawler indexing listings", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "RevvimGort\\/", + "url": "https://revvim.com", + "instances": [ + "Chrome/54.0.2840.71 (compatible; RevvimGort/5.0; +http://www.revvim.com; webmaster@revvim.com)" + ], + "description": "SEO optimization crawler detecting performance issues", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "reward-gateway", + "url": "https://www.rewardgateway.co", + "instances": [ + "reward-gateway" + ], + "description": "Employee engagement platform crawler validating offers", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Riddler \\(http:\\/\\/riddler\\.io", + "url": "http://riddler.io/about", + "instances": [ + "Riddler (http://riddler.io/about)" + ], + "description": "Internet scanning crawler gathering domain and IP metadata", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "RobotsChecker\\/", + "url": "http://www.blocked.org.uk/", + "instances": [ + "RobotsChecker/0.6 (+http://www.blocked.org.uk)" + ], + "description": "Website health monitoring and content indexing crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "RSSAPI\\/", + "url": "https://rssapi.net", + "instances": [ + "RSSAPI/2.0 (+https://rssapi.net/)" + ], + "description": "RSS and Atom feed crawler for content aggregation", + "addition_date": "2026/04/17", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "rss2tg", + "url": "https://rss2tg.duck.consulting", + "instances": [ + "Mozilla/5.0 (compatible; rss2tg bot; +http://komar.in/en/rss2tg_crawler)" + ], + "description": "RSS to Telegram converter fetching feed content", + "addition_date": "2026/04/17", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "RssReaderBot", + "url": "https://datadome.co/bots/rssreaderbot/", + "instances": [ + "RssReaderBot" + ], + "description": "RSS feed aggregator indexing content updates", + "addition_date": "2026/04/17", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "s4a-probe-bot\\/", + "url": "https://www.seo4ajax.com/webscraper/", + "instances": [ + "Mozilla/5.0 AppleWebKit (compatible; s4a-probe-bot/1.0; Fake-Googlebot; +https://www.seo4ajax.com/webscraper)" + ], + "description": "Web scraping and SEO analysis crawler tool", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "SFDC-Callout\\/", + "url": "https://help.salesforce.com/articleView?id=000321501&type=1&mode=1", + "instances": [ + "SFDC-Callout/49.0" + ], + "description": "Salesforce platform crawler fetching web resources", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "page-preview-tool", + "url": "https://www.salesviewer.com/en/", + "instances": [ + "page-preview-tool Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36" + ], + "description": "B2B intelligence platform identifying visiting companies", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "SandobaCrawler\\/", + "url": "https://www.sandoba.com/en/crawler/", + "instances": [ + "mozilla/5.0 (compatible; SandobaCrawler/1.0; +https://www.sandoba.com/en/crawler/)" + ], + "description": "Web crawler for data aggregation and indexing", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Sansec Security Monitor\\/", + "url": "https://sansec.io/monitor", + "instances": [ + "Mozilla/5.0 (compatible; Sansec Security Monitor/1.0; +https://sansec.io/monitor)" + ], + "description": "E-commerce malware detection scanner for skimmers", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "GIFTEDVISITOR SCAN", + "url": "https://datadome.co/bots/scan/", + "instances": [ + "GIFTEDVISITOR SCAN" + ], + "description": "Web crawler for content indexing and analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Schema-Markup-Validator", + "url": "https://validator.schema.org/", + "instances": [ + "Mozilla/5.0 (compatible; Schema-Markup-Validator; +https://validator.schema.org/)" + ], + "description": "Structured data validation crawler for schema compliance", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Scoop\\.it\\/", + "url": "https://datadome.co/bots/scoop-it/", + "instances": [ + "Mozilla/5.0 (compatible; Scoop.it/1.0; +http://www.scoop.it/bot)" + ], + "description": "Content curation platform crawler discovering relevant content", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "ScourRSSBot\\/", + "url": "https://scour.ing/bot", + "instances": [ + "ScourRSSBot/1.0 (+https://scour.ing/bot)" + ], + "description": "RSS feed discovery and monitoring crawler bot", + "addition_date": "2026/04/17", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "ScrapeheroBot\\/", + "url": "https://scrapehero.de/", + "instances": [ + "Mozilla/5.0 (compatible; ScrapeheroBot/1.0; +https://scrapehero.de/)" + ], + "description": "Web scraping service crawler for data extraction", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "screeenly-bot", + "url": "https://3.screeenly.com/ua", + "instances": [ + "screeenly-bot 2.0" + ], + "description": "Screenshot capture service crawler taking website snapshots", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "SEBot-WA", + "url": "https://help.seranking.com/en/project-tools/website-audit/overview", + "instances": [ + "SEBot-WA" + ], + "description": "SEO audit crawler performing technical analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Searcherweb", + "url": "https://datadome.co/bots/searcherweb/", + "instances": [ + "Searcherweb" + ], + "description": "Web crawler indexing content for search engines", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Searcherxweb", + "url": "https://datadome.co/bots/searcherxweb/", + "instances": [ + "Searcherxweb" + ], + "description": "Web crawler for indexing and data collection", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "SearchExpress", + "url": "https://datadome.co/bots/searchexpress/", + "instances": [ + "SearchExpress" + ], + "description": "Search engine crawler gathering website information", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "SecurityHeaders", + "url": "https://securityheaders.com/", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36 SecurityHeaders" + ], + "description": "HTTP security headers analyzer and evaluator", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "semaltbot\\/", + "url": "https://datadome.co/bots/semaltbot/", + "instances": [ + "semaltbot/0.1 (+http://semalt.net)" + ], + "description": "SEO analytics crawler collecting ranking and performance data", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "SendGrid Event API", + "url": "https://sendgrid.com/docs/for-developers/tracking-events/event/", + "instances": [ + "SendGrid Event API" + ], + "description": "Email service crawler scanning links in messages", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "SentryUptimeBot\\/", + "url": "https://docs.sentry.io/product/alerts/uptime-monitoring/troubleshooting/#verify-firewall-configuration", + "instances": [ + "SentryUptimeBot/1.0 (+http://docs.sentry.io/product/alerts/uptime-monitoring/)" + ], + "description": "Application monitoring bot verifying URL availability", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "seo-audit-check-bot\\/", + "url": "https://www.webceo.com/webceo-bots.htm", + "instances": [ + "Mozilla/5.0 (compatible; seo-audit-check-bot/1.0)" + ], + "description": "SEO auditing crawler evaluating optimization signals", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "s4a\\/", + "url": "https://www.seo4ajax.com/", + "instances": [ + "Mozilla/5.0 AppleWebKit (compatible; s4a/1.0; +https://www.seo4ajax.com/webscraper)" + ], + "description": "JavaScript rendering and SEO prerendering service", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "ClarityBot\\/", + "url": "https://www.seoclarity.net/bot.html", + "instances": [ + "Mozilla/5.0 (compatible; ClarityBot/9.0; +https://www.seoclarity.net/bot.html)" + ], + "description": "SEO analytics crawler providing optimization insights", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "SeoSiteCheckup", + "url": "https://datadome.co/bots/seositecheckup/", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36 SeoSiteCheckup (https://seositecheckup.com)" + ], + "description": "SEO site analysis crawler evaluating search performance", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "SeoulBot", + "url": "https://datadome.co/bots/seoulbot/", + "instances": [ + "SeoulBot" + ], + "description": "Web crawler for SEO and competitive analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "SERPtimizerBot", + "url": "http://serptimizer.com/serptimizer-bot", + "instances": [ + "Mozilla/5.0 (compatible; SERPtimizerBot; +http://serptimizer.com/serptimizer-bot)" + ], + "description": "SEO analysis crawler gathering data for optimization", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Server Density Service Monitoring", + "url": "https://www.stackpath.com/", + "instances": [ + "Server Density Service Monitoring v2Server Density Agent" + ], + "description": "Cloud monitoring bot checking uptime and performance", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "ServerHunterSpider\\/", + "url": "https://www.serverhunter.com/", + "instances": [ + "Mozilla/5.0 (compatible; ServerHunterSpider/1.1; +https://www.serverhunter.com/spider/)" + ], + "description": "Hosting and VPS price comparison crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "SeznamHomepageCrawler\\/", + "url": "http://napoveda.seznam.cz/en/seznambot-intro/", + "instances": [ + "SeznamHomepageCrawler/v1.0.6" + ], + "description": "Czech search engine crawler indexing web pages", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Shopify-Captain-Hook", + "url": "https://shopify.dev/docs/apps/build/webhooks", + "instances": [ + "Shopify-Captain-Hook" + ], + "description": "E-commerce platform crawler for webhook verification", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Shortwave Image Fetcher", + "url": "https://www.shortwave.com/", + "instances": [ + "Shortwave Image Fetcher" + ], + "description": "Email client image proxy and link preview fetcher", + "addition_date": "2026/04/17", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "linkReader\\/", + "url": "https://sider.ai/", + "instances": [ + "linkReader/1.0 (+https://gochitchat.ai)" + ], + "description": "AI assistant crawler for content indexing", + "addition_date": "2026/04/17", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "Sidetrade indexer bot", + "url": "https://datadome.co/bots/sidetrade-crawler/", + "instances": [ + "Sidetrade indexer bot" + ], + "description": "Financial analytics crawler gathering business data", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "^Silk\\/", + "url": "https://www.useragentstring.com/pages/silk/", + "instances": [ + "Silk/1.0 Also seen as: silk/1.0 (+http://www.slider.com/silk.htm)/3.7" + ], + "description": "DMOZ directory crawler indexing web content", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "SinceraSyntheticUser\\/", + "url": "https://datadome.co/bots/sincera-bot/", + "instances": [ + "Mozilla/5.0 (compatible; SinceraSyntheticUser/1.0; +http://app.sincera.io/bots)" + ], + "description": "Web data aggregation and analytics crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Optimizer\\)", + "url": "https://www.sistrix.com/faq/uptime", + "instances": [ + "Mozilla/5.0 (compatible; Optimizer)" + ], + "description": "SEO analysis and website optimization crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Site24x7", + "url": "https://www.site24x7.com/", + "instances": [ + "Site24x7" + ], + "description": "Cloud monitoring service checking website health", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "SiteAuditBot\\/", + "url": "https://datadome.co/bots/siteauditbot/", + "instances": [ + "Mozilla/5.0 (compatible; SiteAuditBot/1.1; +https://siteauditbot.com)" + ], + "description": "Website SEO auditing and link checking crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "SiteCheck-sitecrawl", + "url": "https://support.siteimprove.com/hc/en-gb/articles/206345523-What-IP-addresses-and-user-agents-are-used-by-Siteimprove-", + "instances": [ + "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0) SiteCheck-sitecrawl" + ], + "description": "Digital governance platform auditing web quality", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "SiteScoreBot", + "url": "https://datadome.co/bots/sitescorebot/", + "instances": [ + "SiteScoreBot v20210315 - https://sitescore.ai" + ], + "description": "Website performance and security evaluation crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "SiteSearch360\\/", + "url": "https://www.sitesearch360.com/", + "instances": [ + "Mozilla/5.0 (compatible; SiteSearch360/1.0; +https://sitesearch360.com/)" + ], + "description": "On-site search platform crawler building indices", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "SiteUptime\\.com", + "url": "https://datadome.co/bots/siteuptime-com/", + "instances": [ + "SiteUptime.com" + ], + "description": "Uptime monitoring service checking website availability", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Konturbot\\/", + "url": "https://datadome.co/bots/skb-kontur-bot/", + "instances": [ + "Mozilla/5.0 (compatible; Konturbot/1.2; +http://kontur.ru; cargo@kontur.ru)" + ], + "description": "Russian business intelligence and analytics crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "SkroutzBot", + "url": "https://www.skroutz.gr/", + "instances": [ + "SkroutzBot v1.0" + ], + "description": "Price comparison platform crawler aggregating products", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "SkyworkSpider", + "url": "https://datadome.co/bots/skyworkspider/", + "instances": [ + "SkyworkSpider" + ], + "description": "Web crawler for content indexing and analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "SlickBot\\/", + "url": "https://www.slickstream.com/", + "instances": [ + "Mozilla/5.0 (compatible; SlickBot/1.0; +http://slickstream.com)" + ], + "description": "Content recommendation and search indexing crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "SmartologyBot\\/", + "url": "https://smartology.net/smartologybot/", + "instances": [ + "SmartologyBot/1.0 (+http://www.smartology.net/smartologybot)" + ], + "description": "Contextual advertising crawler for brand safety", + "addition_date": "2026/04/17", + "tags": [ + "advertising" + ] + }, + { + "pattern": "SnapURLPreview\\/", + "url": "https://business.snapchat.com/legal/snapchat-automated-crawler", + "instances": [ + "Mozilla/5.0 (compatible; SnapURLPreview/1.0; +https://www.snapchat.com)" + ], + "description": "Social platform URL preview and metadata fetcher", + "addition_date": "2026/04/17", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "SnapchatAds\\/", + "url": "https://businesshelp.snapchat.com/s/article/adsbot-crawler", + "instances": [ + "Mozilla/5.0 (compatible; SnapchatAds/1.0; +https://businesshelp.snapchat.com/s/article/adsbot-crawler)" + ], + "description": "Advertising platform crawler for ad verification", + "addition_date": "2026/04/17", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Snipcart\\/", + "url": "https://snipcart.com/", + "instances": [ + "Snipcart/1.0" + ], + "description": "E-commerce shopping cart crawler validating products", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "solarwinds\\/", + "url": "https://documentation.solarwinds.com/en/success_center/observability/content/get-started/dem_getting_started_guide.htm", + "instances": [ + "solarwinds/1.0 ( www.solarwinds.com/solarwinds-observability)" + ], + "description": "Observability platform monitoring website performance", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Sora POS\\/", + "url": "https://www.sora-caisse.com/sora.pdf", + "instances": [ + "Sora POS/1.0 (Sora Websoft)" + ], + "description": "Point-of-sale system crawler for transaction data", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "SparkShipping", + "url": "https://www.sparkshipping.com/", + "instances": [ + "SparkShipping" + ], + "description": "E-commerce data crawler normalizing product catalogs", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "SparkPost", + "url": "https://www.sparkpost.com/", + "instances": [ + "SparkPost" + ], + "description": "Email service crawler validating links and tracking", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Spawning-AI", + "url": "https://datadome.co/bots/spawning-bot/", + "instances": [ + "Spawning-AI" + ], + "description": "Web crawler indexing content for AI models", + "addition_date": "2026/04/17", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "IDG\\/EU", + "url": "https://datadome.co/bots/spaziodati-bot/", + "instances": [ + "IDG/EU (http://spaziodati.eu/)" + ], + "description": "Semantic data and business intelligence crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Specificfeeds", + "url": "https://follow.it/", + "instances": [ + "FeedBurner/1.0 (http://www.FeedBurner.com) Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36 Specificfeeds" + ], + "description": "Content aggregation crawler for personalized feeds", + "addition_date": "2026/04/17", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "Spectate\\/", + "url": "https://docs.spectate.net/faq/uptime-monitor-bot", + "instances": [ + "Spectate/1.0 (+https://docs.spectate.net/faq/uptime-monitor-bot)" + ], + "description": "Content monitoring and uptime checking crawler", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "SpiderLing", + "url": "http://nlp.fi.muni.cz/projects/biwec/", + "instances": [ + "Mozilla/5.0 (compatible; SpiderLing (a SPIDER for LINGustic research); +http://nlp.fi.muni.cz/projects/biwec/)" + ], + "description": "Linguistic research crawler building language corpora", + "addition_date": "2026/04/17", + "tags": [ + "academic" + ] + }, + { + "pattern": "splash Version\\/", + "url": "https://www.zyte.com/splash/", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/602.1 (KHTML, like Gecko) splash Version/10.0 Safari/602.1" + ], + "description": "Headless browser for web scraping and testing", + "addition_date": "2026/04/17", + "tags": [ + "browser-automation" + ] + }, + { + "pattern": "Rigor\\)", + "url": "https://www.splunk.com/", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64; Rigor) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36" + ], + "description": "Splunk synthetic monitoring for performance checks", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "TwinWaveScanner", + "url": "https://www.splunk.com/en_us/products/attack-analyzer.html", + "instances": [ + "TwinWaveScanner" + ], + "description": "Security threat analysis and phishing detection bot", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "SSL Labs \\(https:\\/\\/www\\.ssllabs\\.com", + "url": "https://www.ssllabs.com/", + "instances": [ + "SSL Labs (https://www.ssllabs.com/about/assessment.html)" + ], + "description": "TLS certificate and encryption configuration analyzer", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "SSSSBot\\/", + "url": "https://datadome.co/bots/ssssbot/", + "instances": [ + "Mozilla/5.0 (compatible; SSSSBot/0.0.1; +http://s-ans.xyz:80/)" + ], + "description": "Web crawler for content indexing and analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Stape\\/", + "url": "https://stape.io/helpdesk/documentation/stape-scanner", + "instances": [ + "Stape/1.0.0Stape" + ], + "description": "Server-side tagging platform tag verification bot", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "StartpagePrivateImageProxy\\/", + "url": "https://datadome.co/bots/startpage-bot/", + "instances": [ + "StartpagePrivateImageProxy/2.0 (https://www.startpage.com/; support@startpage.com) requests/2.25.1" + ], + "description": "Privacy-focused search engine crawler indexing pages", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Statabot\\/", + "url": "https://www.stata.com/support/statabot", + "instances": [ + "Statabot/1.0 https://www.stata.com/support/statabot" + ], + "description": "Analytics and market intelligence web crawler", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "StatistikAustria\\/", + "url": "https://www.statistik.at/ueber-uns/innovationen-und-experimentelle-statistik/einsatz-von-kassenscannerdaten-und-webscraping-in-der-preisstatistik", + "instances": [ + "Mozilla/5.0 (compatible; StatistikAustria/1.0; +http://www.statistik.gv.at)" + ], + "description": "National statistics office crawler for price data", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "StatsDroneBot", + "url": "https://statsdrone.com/statsdrone-bot-documentation/", + "instances": [ + "StatsDroneBot (https://statsdrone.com/statsdrone-bot-documentation/) Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36" + ], + "description": "Affiliate marketing link validation and monitoring bot", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Stripe\\/", + "url": "https://stripe.com/docs/webhooks", + "instances": [ + "Stripe/1.0 (+https://stripe.com/docs/webhooks)" + ], + "description": "Payment processor crawler verifying merchant sites and collecting metadata signals", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Sucuri", + "url": "https://blog.sucuri.net/2012/10/ask-sucuri-how-does-sitecheck-work.html", + "instances": [ + "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6 MSIE 7.0; Sucuri Integrity Monitor/2.4" + ], + "description": "Security scanner identifying malware blacklist status and checking plugin vulnerability compliance", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Svix-Webhooks\\/", + "url": "https://docs.svix.com/receiving/source-ips", + "instances": [ + "Svix-Webhooks/1.65.0 (sender-9YMgn; +https://www.svix.com/http-sender/)Webhooks/1.65.0 (sender-9YMgn)" + ], + "description": "Webhooks infrastructure provider crawler cataloging vendor sites and delivery practices", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "SwifteqLinkChecker", + "url": "https://www.swifteq.com/", + "instances": [ + "SwifteqLinkChecker" + ], + "description": "Web crawler verifying link availability status redirects and basic page content validation", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Swisscows", + "url": "https://swisscows.com/", + "instances": [ + "Swisscows Favicons" + ], + "description": "Privacy-focused search engine crawler indexing web pages without storing personal data", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "Datadog Synthetic", + "url": "https://docs.datadoghq.com/synthetics/", + "instances": [ + "Datadog Synthetic" + ], + "description": "Synthetic monitoring bot testing website APIs and collecting uptime performance data", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "TactiScout\\/", + "url": "http://find-it.world/TempCrawl/Crawltheque.php", + "instances": [ + "TactiScout/recruit (+http://find-it.world/TempCrawl/Crawltheque.php)" + ], + "description": "Data collection crawler gathering competitive intelligence and monitoring market trends analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "tchelebi\\/", + "url": "https://tchelebi.io/", + "instances": [ + "Mozilla/5.0 (compatible; tchelebi/1.0; +http://tchelebi.io)" + ], + "description": "Web crawling tool indexing content for SEO data aggregation and competitive analysis purposes", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "bitdiscovery", + "url": "https://www.tenable.com/products/tenable-asm", + "instances": [ + "bitdiscovery" + ], + "description": "Security vulnerability scanner identifying weaknesses and compliance issues in web infrastructure", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Test Certificate Info", + "url": "https://datadome.co/bots/test-certificate-info/", + "instances": [ + "Test Certificate Info" + ], + "description": "SSL TLS certificate analyzer assessing website security encryption and expiration status", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Testcrawler", + "url": "https://datadome.co/bots/testcrawler/", + "instances": [ + "Testcrawler" + ], + "description": "Testing crawler evaluating website performance security vulnerabilities and SEO optimization", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "test-bot", + "url": "https://datadome.co/bots/test-bot/", + "instances": [ + "test-bot" + ], + "description": "Automated testing script monitoring uptime and gathering performance analysis data", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "TestLocally\\/", + "url": "https://testlocal.ly/", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.6422.26 Safari/537.36 TestLocally/1.0" + ], + "description": "Development testing bot simulating user interactions and verifying website functionality behavior", + "addition_date": "2026/04/17", + "tags": [ + "browser-automation" + ] + }, + { + "pattern": "TestURI", + "url": "https://datadome.co/bots/testuri-crawler/", + "instances": [ + "Mozilla/5.0 (compatible; TestURI; +http://testuri.org/)" + ], + "description": "Web crawler testing performance security compliance and assessing website configuration", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "TextRazor", + "url": "https://datadome.co/bots/textrazor-crawler/", + "instances": [ + "TextRazor Downloader (https://www.textrazor.com)" + ], + "description": "NLP service crawler collecting web data for text analysis entity extraction processing", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "The Knowledge AI", + "url": "https://datadome.co/bots/the-knowledge-ai/", + "instances": [ + "The Knowledge AI" + ], + "description": "AI data collection crawler indexing content and enhancing machine learning models", + "addition_date": "2026/04/17", + "tags": [ + "ai-crawler" + ] + }, + { + "pattern": "TheInternetSearchx", + "url": "https://datadome.co/bots/theinternetsearch/", + "instances": [ + "TheInternetSearchx" + ], + "description": "Search engine crawler indexing websites and improving search result database coverage", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "thesis-research-bot", + "url": "https://datadome.co/bots/thesis-research-bot/", + "instances": [ + "thesis-research-bot" + ], + "description": "Academic research crawler collecting web data trends and analyzing website structures systematically", + "addition_date": "2026/04/17", + "tags": [ + "academic" + ] + }, + { + "pattern": "Trellis-Services", + "url": "https://www.mediavine.com/", + "instances": [ + "Trellis-Services" + ], + "description": "Content scraper and fetching service crawler for programmatic large-scale web page retrieval", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "trentwil\\.es", + "url": "https://trentwil.es/domains.html", + "instances": [ + "domain research project (+https://trentwil.es/domains.html)" + ], + "description": "Web crawler indexing content for data aggregation SEO analysis and competitive intelligence", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Trustly\\/", + "url": "https://www.trustly.net/", + "instances": [ + "Trustly/1" + ], + "description": "Open banking crawler automating user sessions for account data and payment initiation", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "TTD-Content", + "url": "https://www.thetradedesk.com/us/ttd-content", + "instances": [ + "Mozilla/5.0 (compatible; TTD-Content; +https://www.thetradedesk.com/general/ttd-content)" + ], + "description": "Ad network content crawler analyzing pages for contextual advertising and brand safety", + "addition_date": "2026/04/17", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Tweakers", + "url": "https://tweakers.net/", + "instances": [ + "Tweakers Image Proxy https://tweakers.net" + ], + "description": "Dutch technology website crawler aggregating product reviews and IT content data", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "TwilioProxy\\/", + "url": "https://www.upday.com/", + "instances": [ + "TwilioProxy/1.1" + ], + "description": "Communication platform fetcher retrieving media links and previews for messaging services", + "addition_date": "2026/04/17", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "UASlinkChecker\\/", + "url": "https://udger.com/support/UASlinkChecker", + "instances": [ + "Mozilla/5.0 (compatible; UASlinkChecker/2.1; +https://udger.com/support/UASlinkChecker)" + ], + "description": "Link checking crawler validating URL availability and HTTP status across websites", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "hgfAlphaXCrawl\\/", + "url": "https://datadome.co/bots/uni-passau_bot/", + "instances": [ + "hgfAlphaXCrawl/1.0 (+https://www.fim.uni-passau.de/data-science/forschung/open-search)" + ], + "description": "University research crawler collecting data for academic projects and information retrieval", + "addition_date": "2026/04/17", + "tags": [ + "academic" + ] + }, + { + "pattern": "Unshorten\\.It\\!", + "url": "https://unshorten.it/", + "instances": [ + "Unshorten.It!/1.0 (https://unshorten.it/)" + ], + "description": "URL expansion crawler revealing shortened link destinations for cybersecurity threat intelligence", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "updown\\.io", + "url": "https://updown.io", + "instances": [ + "updown.io daemon 2.11" + ], + "description": "Website monitoring crawler checking availability and performance metrics from locations", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Uptime\\/", + "url": "https://uptime.com/", + "instances": [ + "Mozilla/5.0 (compatible; Uptime/1.0; http://uptime.com)" + ], + "description": "Synthetic monitoring bot verifying site availability and transaction health globally", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "uptimedoctor", + "url": "http://uptimestatistics.com/", + "instances": [ + "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36 Edge/17.17134 www.uptimedoctor.com (username slowmail)" + ], + "description": "Uptime monitoring probe verifying website availability and SSL certificate health", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Uptimia", + "url": "https://datadome.co/bots/uptimia/", + "instances": [ + "Mozilla/5.0 (compatible; Uptimia; www.uptimia.com)" + ], + "description": "Platform crawler monitoring website performance uptime and simulating user interactions", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "uptrends", + "url": "https://www.uptrends.com/", + "instances": [ + "uptrends" + ], + "description": "Synthetic monitoring agent testing transaction availability and performance from locations", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Urlcheckr\\/", + "url": "https://www.urlcheckr.com/", + "instances": [ + "Urlcheckr/2.0" + ], + "description": "Dead link checker scanner identifying broken hyperlinks and inaccessible web pages", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "URLSuMaBot", + "url": "https://www.urlsuma.de/bot.aspx", + "instances": [ + "Mozilla / 5.0(Windows NT 10.0; Win64; x64) AppleWebKit / 537.36(KHTML, like Gecko; compatible; URLSuMaBot / 1.0; +https://www.urlsuma.de/bot.aspx) Chrome / 70.0.3538.77 Safari / 537.36" + ], + "description": "Web crawler systematically indexing and collecting data from websites for analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "useeBookChecker\\/", + "url": "https://datadome.co/bots/useebookchecker/", + "instances": [ + "Mozilla/5.0 (compatible; useeBookChecker/0.2; +http://usee.pl/useeBookChecker.html)" + ], + "description": "Ebook metadata scanner identifying and cataloging digital publication availability data", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Vagabondo\\/", + "url": "https://datadome.co/bots/vagabondo-bot/", + "instances": [ + "Mozilla/4.0 (compatible; Vagabondo/4.0Beta; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.nl/)", + "Mozilla/4.0 (compatible; Vagabondo/2.2; webcrawler at wise-guys dot nl)", + "Mozilla/5.0 (compatible; Vagabondo/2.1; webcrawler at wise-guys dot nl)", + "Mozilla/3.0 (Vagabondo/2.0 MT; webcrawler@NOSPAMexperimental.net; http://aanmelden.ilse.nl/?aanmeld_mode=webhints)" + ], + "description": "Legacy Dutch web crawler indexing mobile pages for regional search and content", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "VaultPress", + "url": "https://vaultpress.com/", + "instances": [ + "VaultPress" + ], + "description": "WordPress backup and security scanner performing malware scanning and integrity", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "videootvBot", + "url": "https://videoo.tv/", + "instances": [ + "Mozilla/5.0 (compatible; videootvBot; +https://www.videoo.tv)" + ], + "description": "Video aggregator crawler discovering indexing video content and stream endpoints", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "VsuSearchSpider\\/", + "url": "https://datadome.co/bots/vsusearchspider/", + "instances": [ + "VsuSearchSpider/1.0" + ], + "description": "Web crawler indexing and gathering information from websites for search purposes", + "addition_date": "2026/04/17", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "vu-server-health-scanner\\/", + "url": "http://130.37.198.75/index.html", + "instances": [ + "Mozilla/5.0 (compatible;vu-server-health-scanner/1.0;https://130.37.198.75/index.html)" + ], + "description": "Academic crawler indexing web content for research data collection and analysis", + "addition_date": "2026/04/17", + "tags": [ + "academic" + ] + }, + { + "pattern": "WARDBot\\/", + "url": "https://ward.ai/robot", + "instances": [ + "Mozilla/5.0 (compatible; WARDBot/1.0; http://ward.ai/robot)" + ], + "description": "Web automation crawler extracting website data for reconnaissance and analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "WebsiteOps", + "url": "https://watchful.net/faqs/technical-support/how-do-i-whitelist-the-watchful-ip-address", + "instances": [ + "WebsiteOps (hello@websiteops.io)" + ], + "description": "Web crawling client monitoring uptime changes and analyzing website data", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "WatchMouse", + "url": "https://asm.saas.broadcom.com/", + "instances": [ + "WatchMouse (http://watchmouse.com/ ; HQ)" + ], + "description": "Web performance monitoring tool assessing website availability from global testing locations", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Web Measure\\/", + "url": "https://webresearch.eecs.umich.edu/overview-of-web-measurements/", + "instances": [ + "Web Measure/1.0 (https://webresearch.eecs.umich.edu/overview-of-web-measurements/) Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36" + ], + "description": "Web analytics crawler gathering data for performance optimization and competitive analysis", + "addition_date": "2026/04/17", + "tags": [ + "academic" + ] + }, + { + "pattern": "Webflow", + "url": "https://datadome.co/bots/webflow-bot/", + "instances": [ + "Webflow 1.0 - Site Screenshot" + ], + "description": "Platform crawler rendering and indexing Webflow hosted websites for optimization", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "webgains-bot", + "url": "https://datadome.co/bots/webgains-bot/", + "instances": [ + "webgains-bot (https://www.webgains.com/public/en/)" + ], + "description": "Affiliate marketing crawler extracting product listings and pricing information", + "addition_date": "2026/04/17", + "tags": [ + "advertising" + ] + }, + { + "pattern": "webprosbot\\/", + "url": "https://datadome.co/bots/webprosbot/", + "instances": [ + "webprosbot/2.0 (+mailto:abuse-6337@webpros.com)" + ], + "description": "SEO crawler analyzing websites for optimization content and competitive research", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "websitepulse", + "url": "https://www.websitepulse.com/kb/websitepulse", + "instances": [ + "websitepulse checker/3.0 (compatible; MSIE 5.5; Netscape 4.75; Linux)" + ], + "description": "Synthetic monitoring agent measuring availability performance and content integrity", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "WebSniffer\\/", + "url": "http://websniffer.com/", + "instances": [ + "WebSniffer/1.1 (+http://websniffer.com/)" + ], + "description": "Web analyzer retrieving HTTP headers and HTML content for diagnostic purposes", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "WSM\\/", + "url": "https://webspidermount.com/", + "instances": [ + "Mozilla/5.0 (compatible; WSM/2.0; +https://webspidermount.com/)" + ], + "description": "Web scraper crawler for content collection and data aggregation purposes", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "WebwikiBot\\/", + "url": "https://www.webwiki.com/", + "instances": [ + "Mozilla/5.0 (compatible; WebwikiBot/2.1; +https://www.webwiki.com)" + ], + "description": "Directory crawler collecting website metadata and content for comprehensive indexing", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "WEDOS OnLine", + "url": "https://www.wedos.online", + "instances": [ + "WEDOS OnLine monitoring; https://www.wedos.online/" + ], + "description": "Website monitoring service checking availability DNS and SSL certificate status", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "WhatsMyIP\\.org", + "url": "http://whatsmyip.org/ua", + "instances": [ + "Mozilla/5.0 (WhatsMyIP.org Text_to_Code_Ratio_Tool) http://whatsmyip.org/ua" + ], + "description": "IP address analysis tool gathering network configuration and geolocation data", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "WhatWeb\\/", + "url": "https://www.whatweb.net/", + "instances": [ + "WhatWeb/0.5.5" + ], + "description": "Web technology scanner identifying frameworks servers and CMS configurations", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Wheregoes\\.com", + "url": "https://wheregoes.com/", + "instances": [ + "Wheregoes.com Redirect Checker/1.0" + ], + "description": "URL redirect analyzer mapping redirect chains and HTTP forwarding paths", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "wheresitup\\.com\\/", + "url": "https://wheresitup.com/", + "instances": [ + "wheresitup.com/1.1" + ], + "description": "Availability monitor crawler tracking website uptime and performance metrics", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "Citoid", + "url": "https://www.mediawiki.org/wiki/Citoid", + "instances": [ + "Citoid (Wikimedia tool; learn more at https://www.mediawiki.org/wiki/Citoid)" + ], + "description": "Wikipedia citation generator extracting metadata for automatic reference creation", + "addition_date": "2026/04/17", + "tags": [ + "academic" + ] + }, + { + "pattern": "WireReaderBot\\/", + "url": "https://datadome.co/bots/wirereaderbot/", + "instances": [ + "Mozilla/5.0 (compatible; WireReaderBot/1.0; +https://wirereader.app)" + ], + "description": "Web crawler indexing content for data aggregation and competitive analysis", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "ZoteroTranslationServer\\/WMF", + "url": "https://wikitech.wikimedia.org/wiki/Zotero", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/110.0.0.0 Safari/537.36 ZoteroTranslationServer/WMF (mailto:noc@wikimedia.org)" + ], + "description": "Citation server extracting metadata from URLs for Wikipedia reference generation", + "addition_date": "2026/04/17", + "tags": [ + "academic" + ] + }, + { + "pattern": "wmtips\\.com\\/", + "url": "http://www.wmtips.com/tools/", + "instances": [ + "Mozilla/5.0 (compatible; wmtips.com/2.0; +http://www.wmtips.com/tools/)" + ], + "description": "SEO analysis crawler scanning websites for optimization issues and metrics", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "WordCountBot\\/", + "url": "https://weglot.com/", + "instances": [ + "Mozilla/5.0 (compatible; WordCountBot/0.1;)" + ], + "description": "Content analysis tool computing word counts keyword density and readability", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Wordup-1", + "url": "https://datadome.co/bots/wordup-1/", + "instances": [ + "Wordup-1" + ], + "description": "Web crawler indexing and analyzing content for search and data analytics", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "workona-favicon-service\\/", + "url": "https://datadome.co/bots/workona-bot/", + "instances": [ + "workona-favicon-service/1.0.0" + ], + "description": "Productivity platform crawler organizing web content and managing bookmarks", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Indy Library", + "url": "https://secure.ogone.com/", + "instances": [ + "Mozilla/3.0 (compatible; Indy Library)" + ], + "description": "Payment processor crawler verifying merchant sites and payment flow transactions", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "WJHRO\\/", + "url": "https://docs.worldpay.com/apis", + "instances": [ + "WJHRO/1.0 (WorldPay Java HTTP Request Object)" + ], + "description": "Payment platform crawler testing integrations and verifying checkout flow functionality", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "WormlyBot", + "url": "https://www.wormly.com/help/server-monitoring/website", + "instances": [ + "Mozilla/5.0 (compatible; WormlyBot; +http://wormly.com)" + ], + "description": "Synthetic monitoring crawler testing site APIs and measuring uptime performance", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "WovnCrawler\\/", + "url": "https://support.wovn.io/hc/ja/articles/360043165091", + "instances": [ + "WovnCrawler/1.0" + ], + "description": "Localization platform crawler extracting content and generating translated versions", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "wowLink Crawler\\/", + "url": "https://datadome.co/bots/wowlink-crawler/", + "instances": [ + "wowLink Crawler/1.0" + ], + "description": "Link management crawler indexing and analyzing web content for optimization", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "WP Time Capsule", + "url": "https://docs.wptimecapsule.com/", + "instances": [ + "WP Time Capsule API/1.0 ( https://cron.wptimecapsule.com/; https://service.wptimecapsule.com/ )" + ], + "description": "WordPress backup service crawler detecting changes and performing incremental backups", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "WPUmbrella", + "url": "https://wp-umbrella.com/bot", + "instances": [ + "WPUmbrella" + ], + "description": "WordPress monitor scanner assessing availability and tracking plugin updates", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "wpbot\\/", + "url": "https://datadome.co/bots/wpbot/", + "instances": [ + "Mozilla/5.0 (compatible; wpbot/1.1; +https://forms.gle/ajBaxygz9jSR8p8G9)" + ], + "description": "Web crawler indexing websites and analyzing content for search optimization", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "WPMU DEV Broken Link Checker", + "url": "https://wpmudev.com/docs/hub-2-0/broken-link-checker-2/", + "instances": [ + "WPMU DEV Broken Link Checker Spider" + ], + "description": "Link auditor identifying dead links bad anchors and HTTP errors for maintenance", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "WPMUDEV Uptime Monitor", + "url": "https://wpmudev.com/monitor/", + "instances": [ + "WPMUDEV Uptime Monitor 5.0 (https://wpmudev.com)" + ], + "description": "HTTP probe checking website availability status codes and SSL certificate health", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "WPSec\\/", + "url": "https://wpsec.com/", + "instances": [ + "Mozilla/5.0 (compatible; WPSec/1.3; +https://wpsec.com)" + ], + "description": "WordPress security scanner identifying vulnerabilities outdated plugins and misconfigurations", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "WRTNBot", + "url": "https://datadome.co/bots/wrtnbot/", + "instances": [ + "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; WRTNBot; +https://wrtn.ai/wrtnbot)" + ], + "description": "Web crawler indexing content for data aggregation and market analysis purposes", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "abuse\\.xmco\\.fr", + "url": "https://abuse.xmco.fr/", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0 abuse.xmco.fr" + ], + "description": "Cybersecurity crawler scanning websites for vulnerabilities and threat analysis signals", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "XY-Archive-Compliance", + "url": "https://xy-archive.helpscoutdocs.com/article/61-does-xy-archive-have-a-dedicated-ip-address", + "instances": [ + "Mozilla/5.0 (compatible; XY-Archive-Compliance-Crawler; +https://archive.xyplanningnetwork.com/)", + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 (compatible; XY-Archive-Compliance-Archiver; +https://archive.xyplanningnetwork.com/)" + ], + "description": "Archiving compliance crawler collecting content for legal retention and eDiscovery", + "addition_date": "2026/04/17", + "tags": [ + "archiver" + ] + }, + { + "pattern": "Yahoo Ad monitoring", + "url": "https://developer.yahoo.com/api/", + "instances": [ + "Mozilla/5.0 (compatible; Yahoo Ad monitoring; https://help.yahoo.com/kb/yahoo-ad-monitoring-SLN24857.html)" + ], + "description": "Ad ecosystem crawler auditing quality safety and policy compliance across inventory", + "addition_date": "2026/04/17", + "tags": [ + "advertising" + ] + }, + { + "pattern": "YahooMailProxy", + "url": "https://help.yahoo.com/kb/yahoo-mail-proxy-SLN28749.html", + "instances": [ + "YahooMailProxy; https://help.yahoo.com/kb/yahoo-mail-proxy-SLN28749.html" + ], + "description": "Email proxy crawler fetching URLs and resources for security scanning and rendering", + "addition_date": "2026/04/17", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "YahooCacheSystem", + "url": "https://developer.yahoo.com/oauth2/guide/", + "instances": [ + "YahooCacheSystem; YahooWebServiceClient" + ], + "description": "Content fetcher crawler caching web pages and thumbnails for Yahoo properties", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "YLT Chrome", + "url": "http://yellowlab.tools/", + "instances": [ + "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) YLT Chrome/85.0.4183.121 Safari/537.36" + ], + "description": "Performance analyzer evaluating front-end quality and optimization recommendations", + "addition_date": "2026/04/17", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "YokoyGroupAG\\/", + "url": "https://yokoy.io/", + "instances": [ + "YokoyGroupAG/1.0" + ], + "description": "Expense management platform webhook client delivering events and verifying endpoints", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Yuuperbot", + "url": "https://datadome.co/bots/yuuperbot/", + "instances": [ + "Yuuperbot" + ], + "description": "Web crawler indexing content for data aggregation market analysis and intelligence", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Zapier", + "url": "https://zapier.com/", + "instances": [ + "Zapier" + ], + "description": "Automation client fetching web pages metadata and content for workflow integration", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Zendesk Webhook", + "url": "https://support.zendesk.com/", + "instances": [ + "Zendesk Webhook" + ], + "description": "Support platform service validating webhook endpoints and fetching URL metadata", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Zombiebot\\/", + "url": "http://www.zombiedomain.net/robot/", + "instances": [ + "Mozilla/5.0 (Windows NT 6.3; Trident/7.0; rv 11.0) like Gecko (compatible; Zombiebot/2.1; +http://www.zombiedomain.net/robot/)" + ], + "description": "Web crawler designed to index content for SEO analysis and competitive intelligence", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "zzhbot", + "url": "https://datadome.co/bots/zzhbot/", + "instances": [ + "zzhbot" + ], + "description": "Web crawler indexing and analyzing content for search engine and data aggregation", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Penthouse Critical Path CSS Generator", + "url": "https://criticalcss.com/", + "instances": [ + "Penthouse Critical Path CSS Generator" + ], + "description": "CSS above-the-fold extractor generating critical path stylesheets", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Google-AdWords-Express", + "url": "https://developers.google.com/search/docs/crawling-indexing/google-user-triggered-fetchers#googleproducer", + "instances": [ + "Google-AdWords-Express" + ], + "description": "Google Ads Express fetching advertiser landing pages for review", + "addition_date": "2026/04/17", + "tags": [ + "advertising" + ] + }, + { + "pattern": "Notion\\/", + "url": "https://www.iubenda.com/", + "instances": [ + "Notion/1.0 (https://notion.so; team@makenotion.com)" + ], + "description": "Notion workspace crawler fetching URLs for link previews", + "addition_date": "2026/04/17", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "SSL Labs$", + "url": "https://www.qualys.com/apps/pci-compliance/", + "instances": [ + "SSL Labs" + ], + "description": "Qualys SSL Labs scanner testing TLS configuration and certificate grades", + "addition_date": "2026/04/17", + "tags": [ + "scanner" + ] + }, + { + "pattern": "Skroutz ImageBot", + "url": "https://www.skroutz.gr/", + "instances": [ + "Skroutz ImageBot v1" + ], + "description": "Greek price comparison platform crawler fetching product images", + "addition_date": "2026/04/17", + "tags": [ + "seo" + ] + }, + { + "pattern": "Tumblr\\/", + "url": "https://automattic.com/", + "instances": [ + "Tumblr/14.0.835.186" + ], + "description": "Tumblr social blogging platform crawler fetching link previews", + "addition_date": "2026/04/17", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "upday\\/", + "url": "https://www.upday.com/", + "instances": [ + "Mozilla/5.0 (compatible; upday/1.0; +upday)" + ], + "description": "Samsung news aggregator crawler fetching articles for daily briefings", + "addition_date": "2026/04/17", + "tags": [ + "feed-reader" + ] + }, + { + "pattern": "watchTowr", + "addition_date": "2026/04/23", + "url": "https://watchtowr.com", + "instances": [ + "Mozilla/5.0 (watchTowr; Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0" + ], + "description": "watchTowr is an external attack surface management tool that scans assets and generates findings against those assets", + "tags": [ + "scanner" + ] + }, + { + "pattern": "PRTG Network Monitor", + "addition_date": "2026/05/18", + "url": "https://www.paessler.com/manuals/prtg/http_transaction_sensor", + "instances": [ + "Mozilla/5.0 (compatible; PRTG Network Monitor (www.paessler.com ); Windows)" + ], + "description": "PRTG HTTP Transaction Sensor", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "GeedoShopProductFinder", + "addition_date": "2026/06/17", + "url": "https://geedo.com/product-finder/", + "instances": [ + "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; GeedoShopProductFinder) Chrome/142.0.0.0 Safari/537.36" + ], + "description": "GeedoShopProductFinder is the automated crawler used by Geedo, a global product-search engine specialized in online retail.", + "tags": [ + "search-engine" + ] + }, + { + "pattern": "WatchForBot", + "addition_date": "2026/07/24", + "url": "https://watchfor.io/bot", + "instances": [ + "Mozilla/5.0 (compatible; WatchForBot/1.0; +https://watchfor.io/bot)" + ], + "description": "WatchFor uptime monitoring agent — scheduled availability, SSL, DNS and performance checks of customer-configured endpoints", + "tags": [ + "monitoring" + ] + }, + { + "pattern": "ListGemBot", + "addition_date": "2026/08/06", + "url": "https://listgem.com/bot", + "instances": [ + "ListGemBot/1.0 (+https://listgem.com/bot)" + ], + "description": "ListGemBot is the link unfurler for ListGem, a structured curation platform. When a user adds a URL to one of their lists, it fetches that single page to read its public metadata for the preview.", + "tags": [ + "social-preview" + ] + }, + { + "pattern": "IsonSearchBot", + "addition_date": "2026/09/29", + "url": "https://search.isonai.com/bot", + "instances": [ + "IsonSearchBot/1.0 (+https://search.isonai.com/bot)", + "IsonSearchBot/2.0 (+https://search.isonai.com/bot)" + ], + "description": "IsonSearchBot is the web crawler of Ison Search, a search engine run from Indonesia. It indexes public pages for search results and cited answers that link back to the source; content is not used to train AI models.", + "tags": [ + "search-engine" + ] + } +] diff --git a/src/GenHTTP.Lambda/Services/Analytics/AnalyticsConcern.cs b/src/GenHTTP.Lambda/Services/Analytics/AnalyticsConcern.cs new file mode 100644 index 0000000..ddabace --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Analytics/AnalyticsConcern.cs @@ -0,0 +1,40 @@ +using GenHTTP.Api.Content; +using GenHTTP.Api.Protocol; + +using GenHTTP.Lambda.Services.Visits; + +using GenHTTP.Modules.DependencyInjection; + +namespace GenHTTP.Lambda.Services.Analytics; + +/// +/// Hands the visit a request was to the owner's analytics, where the owner +/// counts visits. +/// +/// +/// Outside the concern that makes the visit (), +/// so it is on the request by the time the answer passes here - whether the +/// lambda answered or threw. All it does is put it on the queue: what the +/// visit is counted as is worked out, and written, by the job that empties it +/// (). +/// +public sealed class AnalyticsConcern(AnalyticsQueue queue) : IDependentConcern +{ + + public async ValueTask HandleAsync(IHandler content, IRequest request) + { + try + { + return await content.HandleAsync(request); + } + finally + { + // a preview is somebody trying a change, never a visitor + if (request.GetVisit() is { Preview: false, Lambda.AnalyticsSince: not null } visit) + { + queue.Add(visit); + } + } + } + +} diff --git a/src/GenHTTP.Lambda/Services/Analytics/AnalyticsJob.cs b/src/GenHTTP.Lambda/Services/Analytics/AnalyticsJob.cs new file mode 100644 index 0000000..cc639bf --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Analytics/AnalyticsJob.cs @@ -0,0 +1,29 @@ +using GenHTTP.Lambda.Configuration; +using GenHTTP.Lambda.Services.Background; + +namespace GenHTTP.Lambda.Services.Analytics; + +/// +/// Writes down the visits the owners' analytics count, every few minutes. +/// +/// +/// The only writer of the analytics database, and off every request: what +/// the requests queued since the last time is summed into a row per lambda, +/// day and value and written in one transaction. A server that stops loses +/// what was queued since. +/// +public sealed class AnalyticsJob(IAnalyticsService analytics, LambdaOptions options) : IBackgroundJob +{ + + public string Name => "Analytics"; + + public TimeSpan Interval => options.AnalyticsInterval; + + public ValueTask ExecuteAsync(CancellationToken cancellation) + { + analytics.Flush(); + + return ValueTask.CompletedTask; + } + +} diff --git a/src/GenHTTP.Lambda/Services/Analytics/AnalyticsModel.cs b/src/GenHTTP.Lambda/Services/Analytics/AnalyticsModel.cs new file mode 100644 index 0000000..0743cd4 --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Analytics/AnalyticsModel.cs @@ -0,0 +1,168 @@ +namespace GenHTTP.Lambda.Services.Analytics; + +/// +/// What visits are counted by, besides the day: a table each in the analytics +/// database. +/// +public enum AnalyticsDimension +{ + /// The path that was answered, relative to the lambda + Pages, + + /// The host of the page a visitor came from, where it was another site + Referrers, + + /// The country the visitor's address was allocated to, or empty where it is not known + Countries, + + /// desktop, mobile, tablet, bot or other + Devices, + + /// The browser of a person, or the app it was opened in - never of a bot + Browsers, + + /// The operating system of a person, never of a bot + Systems, + + /// + /// A bot: whose it is where its operator confirms it by its address, "unconfirmed" where it only claimed to be one + /// of those, and the kind of crawler the list says otherwise + /// + Crawlers +} + +/// +/// The names the dimensions have in the database and in what is handed out. +/// +public static class AnalyticsDimensions +{ + + public static IReadOnlyList All { get; } = Enum.GetValues(); + + /// + /// The table of a dimension, which is also its name in an answer. + /// + public static string TableOf(AnalyticsDimension dimension) => dimension switch + { + AnalyticsDimension.Pages => "pages", + AnalyticsDimension.Referrers => "referrers", + AnalyticsDimension.Countries => "countries", + AnalyticsDimension.Devices => "devices", + AnalyticsDimension.Browsers => "browsers", + AnalyticsDimension.Systems => "systems", + AnalyticsDimension.Crawlers => "crawlers", + _ => throw new ArgumentOutOfRangeException(nameof(dimension)) + }; + + /// + /// How many different values one lambda may add to a dimension in a day, + /// or nothing where what it can hold is bounded anyway. The rest is + /// counted under . + /// + /// + /// Paths and referrers come from whoever sends a request, and a scanner + /// trying ten thousand paths should not cost ten thousand rows a day. + /// + public static int? MostOf(AnalyticsDimension dimension) => dimension switch + { + AnalyticsDimension.Pages => 200, + AnalyticsDimension.Referrers => 100, + _ => null + }; + + /// + /// What the values past are counted under. + /// + public const string Other = "(other)"; + +} + +/// +/// What one lambda's visitors did over a stretch of days. +/// +/// Everything in the stretch, summed +/// Each day of the stretch, oldest first, days without visits included +/// Each dimension's values, the most viewed first, then the most requested +public sealed record AnalyticsReport( + AnalyticsTotals Totals, + IReadOnlyList Days, + IReadOnlyDictionary> Breakdowns +); + +/// +/// What was counted, summed over some days. +/// +/// Every request answered - pages, the files they load, calls of an API +/// The requests that were somebody opening a page +/// +/// Different visitors, each counted once a day: summed over several days, a visitor who came back on another day is +/// counted again +/// +/// Answered with a server error +/// Answered with a client error, not found included +/// What the responses carried, where that was known +/// The time it took to answer them, all together +public sealed record AnalyticsTotals(long Hits, long Views, long Visitors, long Failed, long Rejected, long Bytes, double Millis) +{ + + public static AnalyticsTotals None { get; } = new(0, 0, 0, 0, 0, 0, 0); + + public AnalyticsTotals Add(AnalyticsTotals other) + => new(Hits + other.Hits, Views + other.Views, Visitors + other.Visitors, Failed + other.Failed, Rejected + other.Rejected, + Bytes + other.Bytes, Millis + other.Millis); + +} + +/// +/// One day of a lambda's visits, in UTC. +/// +public sealed record AnalyticsDay(DateOnly Date, AnalyticsTotals Totals); + +/// +/// One value of a dimension and how often it came up. +/// +/// The path, the host, the country code or the name +/// Requests it was part of +/// Pages opened it was part of +public sealed record AnalyticsValue(string Value, long Hits, long Views); + +/// +/// Whether a lambda's visits are counted, and what its owner may see of them. +/// +/// The lambda +/// When counting started, or nothing while it is off +/// How many days this answer covers, today included +/// The most days the lambda's tier shows +/// How many days are kept, whatever the tier shows +/// The first day of the answer +/// The last day of the answer, today +/// The first day anything was counted on, which may lie before what the tier shows +/// What was counted from to , or nothing while off +/// The same number of days before, where the tier shows those too +/// When counts were last written, or nothing since the server came up +/// How often they are written +/// The operators whose crawlers are confirmed by their addresses, as far as those are known +public sealed record LambdaAnalytics( + string PublicKey, + string Tier, + DateTime? Since, + int Shown, + int Window, + int Kept, + DateOnly From, + DateOnly To, + DateOnly? Earliest, + AnalyticsReport? Report, + AnalyticsTotals? Previous, + DateTime? Updated, + TimeSpan Interval, + IReadOnlyList Confirmed +); + +/// +/// What one writing of the counts came to. +/// +/// The visits taken off the queue +/// Those of lambdas whose owners count them, which were written +/// Visits not queued since the last writing, because the queue was full +public sealed record AnalyticsFlush(int Visits, int Counted, long Dropped); diff --git a/src/GenHTTP.Lambda/Services/Analytics/AnalyticsQueue.cs b/src/GenHTTP.Lambda/Services/Analytics/AnalyticsQueue.cs new file mode 100644 index 0000000..4ac2d0f --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Analytics/AnalyticsQueue.cs @@ -0,0 +1,83 @@ +using System.Collections.Concurrent; + +using GenHTTP.Lambda.Services.Visits; + +namespace GenHTTP.Lambda.Services.Analytics; + +/// +/// The visits waiting to be counted. +/// +/// +/// Filled by every request to a lambda whose owner counts visits, and emptied +/// by the job that writes them down every few minutes - so a request does no +/// more than put its visit on the end, and nothing is written to a database +/// while somebody waits for an answer. +/// +/// Without a lock: adding is a compare and swap in the queue and an increment +/// of the count, and taking walks the queue from the front while requests keep +/// adding at the back, so neither waits for the other. +/// +/// Bounded, because what fills it comes from outside: a flood of requests +/// between two writings would otherwise be held in memory in full. What does +/// not fit is counted as dropped and not counted otherwise; losing visits is +/// acceptable here, as it is when the server restarts with visits not yet +/// written. +/// +public sealed class AnalyticsQueue +{ + + /// + /// How many visits are held at most. + /// + public const int Most = 250_000; + + private readonly ConcurrentQueue _visits = new(); + + private int _count; + + private long _dropped; + + #region Functionality + + /// + /// Puts a visit on the end, or drops it when the queue is full. + /// + public void Add(LambdaVisit visit) + { + if (Interlocked.Increment(ref _count) > Most) + { + Interlocked.Decrement(ref _count); + Interlocked.Increment(ref _dropped); + return; + } + + _visits.Enqueue(visit); + } + + /// + /// Takes every visit that is waiting, and how many were dropped since it + /// was last asked. + /// + /// + /// Takes what was there when it started: a visit added while it walks the + /// queue may make it in or wait for the next time, and never both. + /// + public (List Visits, long Dropped) Take() + { + var waiting = Volatile.Read(ref _count); + + var taken = new List(waiting); + + while (taken.Count < waiting && _visits.TryDequeue(out var visit)) + { + taken.Add(visit); + } + + Interlocked.Add(ref _count, -taken.Count); + + return (taken, Interlocked.Exchange(ref _dropped, 0)); + } + + #endregion + +} diff --git a/src/GenHTTP.Lambda/Services/Analytics/AnalyticsService.cs b/src/GenHTTP.Lambda/Services/Analytics/AnalyticsService.cs new file mode 100644 index 0000000..52539dd --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Analytics/AnalyticsService.cs @@ -0,0 +1,239 @@ +using GenHTTP.Lambda.Configuration; +using GenHTTP.Lambda.Data; +using GenHTTP.Lambda.Data.Entities; +using GenHTTP.Lambda.Services.Meta; +using GenHTTP.Lambda.Services.Settings; + +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.Logging; + +namespace GenHTTP.Lambda.Services.Analytics; + +/// +/// How a lambda's visitors reach it, counted by day for its owner where the +/// owner asked for it. +/// +/// +/// The switch is kept with the lambda in the platform's database, where +/// serving a request reads it from memory (ResolvedLambda.AnalyticsSince); +/// the counts are kept in the analytics database (). +/// A request puts its visit on a queue () and +/// this service writes the queue down when the job asks it to - so the +/// requests never wait for a database, and the database is written once every +/// few minutes rather than once per request. +/// +public sealed class AnalyticsService(IDbContextFactory databases, AnalyticsStore store, AnalyticsQueue queue, DailyVisitors visitors, + VisitorClassifier classifier, CrawlerRanges ranges, ILimitsService limits, LambdaOptions options, ILogger logger) : IAnalyticsService +{ + private const string Missing = "This lambda does not exist (or has been deleted)."; + + /// + /// How many values of a dimension an answer lists at most. + /// + private const int Listed = 50; + + /// + /// Held while visits are written down and while a lambda's counts are + /// deleted, so a writing that started before analytics were switched off + /// cannot put back what switching them off removed. + /// + private readonly Lock _turn = new(); + + private DateTime? _updated; + + private DateOnly _expired; + + #region Owner + + public LambdaAnalytics Get(string privateKey, int? days = null, DateTime? now = null) + { + using var database = databases.CreateDbContext(); + + var lambda = database.Lambdas.AsNoTracking().FirstOrDefault(l => l.PrivateKey == privateKey) + ?? throw LambdaException.NotFound(Missing); + + return Describe(lambda, days, now ?? DateTime.UtcNow); + } + + public LambdaAnalytics Enable(string privateKey) + { + using var database = databases.CreateDbContext(); + + var lambda = RequireEditable(database, privateKey); + + if (lambda.AnalyticsSince == null) + { + // whatever a writing left behind while they were off - or of a + // lambda that had this id before - is not this lambda's + lock (_turn) + { + store.Forget(lambda.Id); + } + + lambda.AnalyticsSince = DateTime.UtcNow; + + database.SaveChanges(); + } + + return Describe(lambda, null, DateTime.UtcNow); + } + + public string Disable(string privateKey) + { + using var database = databases.CreateDbContext(); + + var lambda = RequireEditable(database, privateKey); + + lock (_turn) + { + if (lambda.AnalyticsSince != null) + { + lambda.AnalyticsSince = null; + + database.SaveChanges(); + } + + // also where they were off already: what is deleted is never + // shown, and nothing is left that could be + store.Forget(lambda.Id); + } + + return lambda.PublicKey; + } + + #endregion + + #region Writing + + public AnalyticsFlush Flush(DateTime? now = null) + { + var at = now ?? DateTime.UtcNow; + + var today = DateOnly.FromDateTime(at); + + lock (_turn) + { + var (visits, dropped) = queue.Take(); + + var counted = 0; + + if (visits.Count > 0) + { + var tally = new VisitTally(visitors, classifier); + + var since = Counting(visits.Select(v => v.Lambda.Id).Distinct().ToList()); + + foreach (var visit in visits) + { + // switched off since the visit was queued, or off and on + // again: then it was made before what is counted now + if (since.TryGetValue(visit.Lambda.Id, out var start) && visit.At >= start) + { + tally.Add(visit); + counted++; + } + } + + store.Write(tally, at); + } + + visitors.Prune(today); + + if (_expired != today) + { + var removed = store.Expire(today.AddDays(-(AnalyticsStore.Kept - 1))); + + _expired = today; + + if (removed > 0) + { + logger.LogInformation("Expired {Rows} rows of analytics before {Day}", removed, today.AddDays(-(AnalyticsStore.Kept - 1))); + } + } + + if (dropped > 0) + { + logger.LogWarning("Dropped {Dropped} visits for analytics, the queue holding {Most} at most", dropped, AnalyticsQueue.Most); + } + + _updated = at; + + return new AnalyticsFlush(visits.Count, counted, dropped); + } + } + + /// + /// Since when each of the lambdas has its visits counted, for those that do. + /// + private Dictionary Counting(List lambdas) + { + using var database = databases.CreateDbContext(); + + return database.Lambdas.AsNoTracking() + .Where(l => lambdas.Contains(l.Id) && l.AnalyticsSince != null) + .Select(l => new { l.Id, l.AnalyticsSince }) + .ToDictionary(l => l.Id, l => l.AnalyticsSince!.Value); + } + + #endregion + + #region Helpers + + /// + /// What the owner of a lambda is shown: as many days as were asked for, + /// and never more than its tier shows. + /// + /// + /// The days before are summed for a comparison only where the tier shows + /// those as well - a free lambda's week is not compared with the week + /// before, which it does not show. + /// + private LambdaAnalytics Describe(LambdaEntity lambda, int? days, DateTime now) + { + var window = Math.Clamp(limits.Of(lambda.Tier).AnalyticsDays, 1, AnalyticsStore.Kept); + + var shown = Math.Clamp(days ?? window, 1, window); + + var to = DateOnly.FromDateTime(now); + + var from = to.AddDays(-(shown - 1)); + + var tier = lambda.Tier.ToString(); + + if (lambda.AnalyticsSince == null) + { + return new LambdaAnalytics(lambda.PublicKey, tier, null, shown, window, AnalyticsStore.Kept, from, to, null, null, null, _updated, + options.AnalyticsInterval, ranges.Confirmed); + } + + var report = store.Read(lambda.Id, from, to, Listed); + + var previous = shown * 2 <= window ? store.Sum(lambda.Id, from.AddDays(-shown), from.AddDays(-1)) : null; + + return new LambdaAnalytics(lambda.PublicKey, tier, lambda.AnalyticsSince, shown, window, AnalyticsStore.Kept, from, to, + store.Earliest(lambda.Id), report, previous, _updated, options.AnalyticsInterval, ranges.Confirmed); + } + + /// + /// The lambda behind an editor key, as long as its owner may change it. + /// + /// + /// A demo's key is announced, so holding it says nothing about being the + /// one who decides whether its visitors are counted. + /// + private static LambdaEntity RequireEditable(LambdaDbContext database, string privateKey) + { + var lambda = database.Lambdas.FirstOrDefault(l => l.PrivateKey == privateKey) + ?? throw LambdaException.NotFound(Missing); + + if (lambda.Tier == LambdaTier.Demo) + { + throw LambdaException.Forbidden($"'{lambda.PublicKey}' is a demo and read only: its visitors are not counted. A lambda of your own made from it can count them."); + } + + return lambda; + } + + #endregion + +} diff --git a/src/GenHTTP.Lambda/Services/Analytics/AnalyticsStore.cs b/src/GenHTTP.Lambda/Services/Analytics/AnalyticsStore.cs new file mode 100644 index 0000000..cf54fa6 --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Analytics/AnalyticsStore.cs @@ -0,0 +1,431 @@ +using System.Globalization; +using System.Reflection; + +using GenHTTP.Lambda.Configuration; + +using Microsoft.Data.Sqlite; +using Microsoft.Extensions.Logging; + +namespace GenHTTP.Lambda.Services.Analytics; + +/// +/// The analytics database: what was counted of every lambda whose owner +/// counts visits, a row per lambda, day and value. +/// +/// +/// A SQLite file of its own beside the platform's (), +/// laid out as a star - see the migration for why. Spoken to with plain +/// commands rather than Entity Framework: a writing is a few hundred upserts +/// in one transaction, and a reading a handful of sums over a range of a key, +/// neither of which a model would make clearer. +/// +/// Synchronous, as SQLite is: written by the background job, and read for an +/// owner a lambda's year at most, by its key. +/// +public sealed class AnalyticsStore : IAnalyticsStore +{ + private const string MigrationNamespace = "GenHTTP.Lambda.Services.Analytics.Migrations"; + + /// + /// How many days are kept, today included: a year, whatever a tier shows of it. + /// + public const int Kept = 366; + + private readonly string _connection; + + #region Initialization + + public AnalyticsStore(LambdaOptions options, ILogger logger) + { + Directory.CreateDirectory(options.DataDirectory); + + _connection = $"Data Source={options.AnalyticsFile};Default Timeout=30;Pooling=True"; + + using var connection = Open(); + + Execute(connection, "PRAGMA journal_mode = WAL;"); + + var evolve = new EvolveDb.Evolve(connection, message => logger.LogDebug("{Message}", message)) + { + Locations = [], + EmbeddedResourceAssemblies = [Assembly.GetExecutingAssembly()], + EmbeddedResourceFilters = [MigrationNamespace], + IsEraseDisabled = true, + MetadataTableName = "schema_versions" + }; + + evolve.Migrate(); + } + + #endregion + + #region Writing + + /// + /// Adds what a writing counted to the rows of its days, in one transaction. + /// + /// + /// A dimension with a most per day () + /// takes a new value only while the day has room for one; past that, it + /// is added to . Values the day + /// already has keep being counted under their own. + /// + public void Write(VisitTally tally, DateTime now) + { + if (tally.IsEmpty) + { + return; + } + + using var connection = Open(); + + using var transaction = connection.BeginTransaction(); + + using (var lambdas = Command(connection, "INSERT OR IGNORE INTO lambdas (id, created) VALUES ($lambda, $created)", "$lambda", "$created")) + { + foreach (var lambda in tally.Lambdas) + { + lambdas.Parameters["$lambda"].Value = lambda; + lambdas.Parameters["$created"].Value = now.ToString("O", CultureInfo.InvariantCulture); + lambdas.ExecuteNonQuery(); + } + } + + using (var days = Command(connection, """ + INSERT INTO days (lambda, year, month, day, hits, views, visitors, failed, rejected, bytes, millis) + VALUES ($lambda, $year, $month, $day, $hits, $views, $visitors, $failed, $rejected, $bytes, $millis) + ON CONFLICT (lambda, year, month, day) DO UPDATE SET + hits = hits + excluded.hits, views = views + excluded.views, visitors = visitors + excluded.visitors, + failed = failed + excluded.failed, rejected = rejected + excluded.rejected, + bytes = bytes + excluded.bytes, millis = millis + excluded.millis + """, [.. DayKey, "$hits", "$views", "$visitors", "$failed", "$rejected", "$bytes", "$millis"])) + { + foreach (var (lambda, day, totals) in tally.Days) + { + Day(days, lambda, day); + days.Parameters["$hits"].Value = totals.Hits; + days.Parameters["$views"].Value = totals.Views; + days.Parameters["$visitors"].Value = totals.Visitors; + days.Parameters["$failed"].Value = totals.Failed; + days.Parameters["$rejected"].Value = totals.Rejected; + days.Parameters["$bytes"].Value = totals.Bytes; + days.Parameters["$millis"].Value = totals.Millis; + days.ExecuteNonQuery(); + } + } + + foreach (var group in tally.Cells.GroupBy(c => c.Cell.Dimension)) + { + var table = AnalyticsDimensions.TableOf(group.Key); + + var most = AnalyticsDimensions.MostOf(group.Key); + + using var upsert = Command(connection, $""" + INSERT INTO {table} (lambda, year, month, day, value, hits, views) + VALUES ($lambda, $year, $month, $day, $value, $hits, $views) + ON CONFLICT (lambda, year, month, day, value) DO UPDATE SET + hits = hits + excluded.hits, views = views + excluded.views + """, [.. DayKey, "$value", "$hits", "$views"]); + + using var known = Command(connection, $"SELECT value FROM {table} WHERE lambda = $lambda AND year = $year AND month = $month AND day = $day", DayKey); + + // the busiest first, so where a day runs out of room it is the + // values that came up least that are gathered under other + foreach (var day in group.GroupBy(c => (c.Cell.Lambda, c.Cell.Day))) + { + HashSet? values = null; + + if (most != null) + { + Day(known, day.Key.Lambda, day.Key.Day); + values = Strings(known); + } + + foreach (var (cell, hits, views) in day.OrderByDescending(c => c.Views).ThenByDescending(c => c.Hits)) + { + var value = cell.Value; + + if (values != null && !values.Contains(value)) + { + if (values.Count < most) + { + values.Add(value); + } + else + { + value = AnalyticsDimensions.Other; + } + } + + Day(upsert, cell.Lambda, cell.Day); + upsert.Parameters["$value"].Value = value; + upsert.Parameters["$hits"].Value = hits; + upsert.Parameters["$views"].Value = views; + upsert.ExecuteNonQuery(); + } + } + } + + transaction.Commit(); + } + + /// + /// Removes everything counted of a lambda. + /// + public void Forget(long lambda) + { + using var connection = Open(); + + using var transaction = connection.BeginTransaction(); + + foreach (var table in Tables) + { + using var delete = Command(connection, $"DELETE FROM {table} WHERE lambda = $lambda", "$lambda"); + + delete.Parameters["$lambda"].Value = lambda; + delete.ExecuteNonQuery(); + } + + using (var row = Command(connection, "DELETE FROM lambdas WHERE id = $lambda", "$lambda")) + { + row.Parameters["$lambda"].Value = lambda; + row.ExecuteNonQuery(); + } + + transaction.Commit(); + } + + /// + /// Removes every day before the one given. + /// + /// + /// Lambda by lambda, so every delete is a range of the key rather than a + /// look at every row of a table. + /// + /// How many rows were removed + public int Expire(DateOnly before) + { + using var connection = Open(); + + var lambdas = new List(); + + using (var list = Command(connection, "SELECT id FROM lambdas")) + using (var reader = list.ExecuteReader()) + { + while (reader.Read()) + { + lambdas.Add(reader.GetInt64(0)); + } + } + + var removed = 0; + + using var transaction = connection.BeginTransaction(); + + foreach (var table in Tables) + { + using var delete = Command(connection, $"DELETE FROM {table} WHERE lambda = $lambda AND (year, month, day) < ($year, $month, $day)", DayKey); + + foreach (var lambda in lambdas) + { + Day(delete, lambda, before); + removed += delete.ExecuteNonQuery(); + } + } + + transaction.Commit(); + + return removed; + } + + #endregion + + #region Reading + + /// + /// What was counted of a lambda from one day to another, both included. + /// + /// How many values of a dimension are listed at most, the rest left out + public AnalyticsReport Read(long lambda, DateOnly from, DateOnly to, int most) + { + using var connection = Open(); + + var counted = new Dictionary(); + + using (var days = Ranged(connection, lambda, from, to, """ + SELECT year, month, day, hits, views, visitors, failed, rejected, bytes, millis FROM days + WHERE lambda = $lambda AND (year, month, day) >= ($fromYear, $fromMonth, $fromDay) AND (year, month, day) <= ($toYear, $toMonth, $toDay) + """)) + using (var reader = days.ExecuteReader()) + { + while (reader.Read()) + { + counted[new DateOnly(reader.GetInt32(0), reader.GetInt32(1), reader.GetInt32(2))] = Totals(reader, 3); + } + } + + var series = new List(); + + var total = AnalyticsTotals.None; + + for (var day = from; day <= to; day = day.AddDays(1)) + { + var totals = counted.GetValueOrDefault(day, AnalyticsTotals.None); + + series.Add(new AnalyticsDay(day, totals)); + + total = total.Add(totals); + } + + var breakdowns = new Dictionary>(); + + foreach (var dimension in AnalyticsDimensions.All) + { + using var values = Ranged(connection, lambda, from, to, $""" + SELECT value, SUM(hits) AS hits, SUM(views) AS views FROM {AnalyticsDimensions.TableOf(dimension)} + WHERE lambda = $lambda AND (year, month, day) >= ($fromYear, $fromMonth, $fromDay) AND (year, month, day) <= ($toYear, $toMonth, $toDay) + GROUP BY value ORDER BY views DESC, hits DESC, value LIMIT $most + """); + + values.Parameters.AddWithValue("$most", most); + + using var reader = values.ExecuteReader(); + + var listed = new List(); + + while (reader.Read()) + { + listed.Add(new AnalyticsValue(reader.GetString(0), reader.GetInt64(1), reader.GetInt64(2))); + } + + breakdowns[dimension] = listed; + } + + return new AnalyticsReport(total, series, breakdowns); + } + + /// + /// What was counted of a lambda from one day to another, summed. + /// + public AnalyticsTotals Sum(long lambda, DateOnly from, DateOnly to) + { + using var connection = Open(); + + using var sum = Ranged(connection, lambda, from, to, """ + SELECT IFNULL(SUM(hits), 0), IFNULL(SUM(views), 0), IFNULL(SUM(visitors), 0), IFNULL(SUM(failed), 0), + IFNULL(SUM(rejected), 0), IFNULL(SUM(bytes), 0), IFNULL(SUM(millis), 0) FROM days + WHERE lambda = $lambda AND (year, month, day) >= ($fromYear, $fromMonth, $fromDay) AND (year, month, day) <= ($toYear, $toMonth, $toDay) + """); + + using var reader = sum.ExecuteReader(); + + return reader.Read() ? Totals(reader, 0) : AnalyticsTotals.None; + } + + /// + /// The first day anything was counted of a lambda on, or nothing. + /// + public DateOnly? Earliest(long lambda) + { + using var connection = Open(); + + using var first = Command(connection, "SELECT year, month, day FROM days WHERE lambda = $lambda ORDER BY year, month, day LIMIT 1", "$lambda"); + + first.Parameters["$lambda"].Value = lambda; + + using var reader = first.ExecuteReader(); + + return reader.Read() ? new DateOnly(reader.GetInt32(0), reader.GetInt32(1), reader.GetInt32(2)) : null; + } + + #endregion + + #region Helpers + + private static IEnumerable Tables => AnalyticsDimensions.All.Select(AnalyticsDimensions.TableOf).Prepend("days"); + + private SqliteConnection Open() + { + var connection = new SqliteConnection(_connection); + + connection.Open(); + + return connection; + } + + private static void Execute(SqliteConnection connection, string sql) + { + using var command = connection.CreateCommand(); + + command.CommandText = sql; + command.ExecuteNonQuery(); + } + + /// + /// The parameters that name a day of a lambda. + /// + private static readonly string[] DayKey = ["$lambda", "$year", "$month", "$day"]; + + /// + /// A command with the parameters given, to be set and executed as often as needed. + /// + private static SqliteCommand Command(SqliteConnection connection, string sql, params string[] parameters) + { + var command = connection.CreateCommand(); + + command.CommandText = sql; + + foreach (var name in parameters) + { + command.Parameters.Add(new SqliteParameter(name, null)); + } + + return command; + } + + private static SqliteCommand Ranged(SqliteConnection connection, long lambda, DateOnly from, DateOnly to, string sql) + { + var command = connection.CreateCommand(); + + command.CommandText = sql; + + command.Parameters.AddWithValue("$lambda", lambda); + command.Parameters.AddWithValue("$fromYear", from.Year); + command.Parameters.AddWithValue("$fromMonth", from.Month); + command.Parameters.AddWithValue("$fromDay", from.Day); + command.Parameters.AddWithValue("$toYear", to.Year); + command.Parameters.AddWithValue("$toMonth", to.Month); + command.Parameters.AddWithValue("$toDay", to.Day); + + return command; + } + + private static void Day(SqliteCommand command, long lambda, DateOnly day) + { + command.Parameters["$lambda"].Value = lambda; + command.Parameters["$year"].Value = day.Year; + command.Parameters["$month"].Value = day.Month; + command.Parameters["$day"].Value = day.Day; + } + + private static HashSet Strings(SqliteCommand command) + { + var values = new HashSet(StringComparer.Ordinal); + + using var reader = command.ExecuteReader(); + + while (reader.Read()) + { + values.Add(reader.GetString(0)); + } + + return values; + } + + private static AnalyticsTotals Totals(SqliteDataReader reader, int at) + => new(reader.GetInt64(at), reader.GetInt64(at + 1), reader.GetInt64(at + 2), reader.GetInt64(at + 3), reader.GetInt64(at + 4), + reader.GetInt64(at + 5), reader.GetDouble(at + 6)); + + #endregion + +} diff --git a/src/GenHTTP.Lambda/Services/Analytics/CrawlerJob.cs b/src/GenHTTP.Lambda/Services/Analytics/CrawlerJob.cs new file mode 100644 index 0000000..ff2faac --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Analytics/CrawlerJob.cs @@ -0,0 +1,167 @@ +using GenHTTP.Lambda.Configuration; +using GenHTTP.Lambda.Services.Background; + +using Microsoft.Extensions.Logging; + +namespace GenHTTP.Lambda.Services.Analytics; + +/// +/// Keeps what tells crawlers from people current: the list of crawlers, and +/// the addresses the operators that publish them crawl from. +/// +/// +/// New crawlers appear every month and the operators move their addresses +/// more often than that, so both are fetched again daily by default rather +/// than shipped once. What was fetched is kept on the data volume and read on +/// the way up, like the country table (see GeoJob): a server that comes +/// back while the sources are unreachable knows what it knew yesterday. One +/// that has never reached them has the list of its release and confirms no +/// crawler - every claim is counted as unconfirmed, which is what it is. +/// +public sealed class CrawlerJob(CrawlerList crawlers, CrawlerRanges ranges, LambdaOptions options, ILogger logger) : IBackgroundJob +{ + + /// + /// Where the list of crawlers is published. + /// + private const string ListSource = "https://raw.githubusercontent.com/monperrus/crawler-user-agents/master/crawler-user-agents.json"; + + private const string ListFile = "crawler-user-agents.json"; + + private bool _loaded; + + public string Name => "Crawlers"; + + public TimeSpan Interval => options.CrawlerRefresh; + + public async ValueTask ExecuteAsync(CancellationToken cancellation) + { + if (!options.CrawlerLists) + { + return; + } + + var directory = Path.Combine(options.DataDirectory, "crawlers"); + + Directory.CreateDirectory(directory); + + // whatever is on disk first, so a restart confirms crawlers within + // milliseconds rather than after a dozen downloads + if (!_loaded) + { + _loaded = true; + + LoadList(directory); + + foreach (var claimed in CrawlerRanges.Operators) + { + LoadRanges(directory, claimed); + } + } + + if (await FetchAsync(ListSource, Path.Combine(directory, ListFile), cancellation) && LoadList(directory)) + { + logger.LogInformation("Refreshed the list of {Count:N0} crawlers", crawlers.Count); + } + + foreach (var claimed in CrawlerRanges.Operators) + { + var fetched = false; + + for (var i = 0; i < claimed.Lists.Count; i++) + { + fetched |= await FetchAsync(claimed.Lists[i], FileOf(directory, claimed, i), cancellation); + } + + if (fetched && LoadRanges(directory, claimed)) + { + logger.LogInformation("Refreshed the addresses of the crawlers of {Operator}", claimed.Name); + } + } + } + + /// + /// Fetches a source where the copy on disk is older than the interval. + /// + /// Whether a new copy was written + private async ValueTask FetchAsync(string source, string file, CancellationToken cancellation) + { + // nothing to do if the copy is younger than the interval, which is + // what makes a restart cheap + if (File.Exists(file) && DateTime.UtcNow - File.GetLastWriteTimeUtc(file) < options.CrawlerRefresh) + { + return false; + } + + try + { + using var client = new HttpClient { Timeout = TimeSpan.FromMinutes(2) }; + + client.DefaultRequestHeaders.UserAgent.ParseAdd("GenHTTP-Lambda (+https://genhttp.dev)"); + + var body = await client.GetByteArrayAsync(source, cancellation); + + // written beside and moved, so a download cut off halfway leaves + // yesterday's copy rather than half of today's + var scratch = file + ".writing"; + + File.WriteAllBytes(scratch, body); + + File.Move(scratch, file, true); + + return true; + } + catch (Exception e) when (e is not OperationCanceledException) + { + // one source unreachable leaves what is known of it, which is not + // worth a failed job + logger.LogDebug(e, "Failed to refresh {Source}", source); + return false; + } + } + + private bool LoadList(string directory) + { + var file = Path.Combine(directory, ListFile); + + if (!File.Exists(file)) + { + return false; + } + + using var stream = File.OpenRead(file); + + if (crawlers.Load(stream)) + { + return true; + } + + logger.LogWarning("The list of crawlers in {File} does not read, the one before stays", file); + return false; + } + + private bool LoadRanges(string directory, CrawlerOperator claimed) + { + var files = Enumerable.Range(0, claimed.Lists.Count).Select(i => FileOf(directory, claimed, i)).Where(File.Exists).ToList(); + + if (files.Count == 0) + { + return false; + } + + var streams = files.Select(f => (Stream)File.OpenRead(f)).ToList(); + + try + { + return ranges.Load(claimed.Name, streams); + } + finally + { + streams.ForEach(s => s.Dispose()); + } + } + + private static string FileOf(string directory, CrawlerOperator claimed, int index) + => Path.Combine(directory, $"{claimed.Name.ToLowerInvariant()}-{index}.json"); + +} diff --git a/src/GenHTTP.Lambda/Services/Analytics/CrawlerList.cs b/src/GenHTTP.Lambda/Services/Analytics/CrawlerList.cs new file mode 100644 index 0000000..4d6278d --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Analytics/CrawlerList.cs @@ -0,0 +1,252 @@ +using System.Buffers; +using System.Reflection; +using System.Text; +using System.Text.Json; +using System.Text.RegularExpressions; + +namespace GenHTTP.Lambda.Services.Analytics; + +/// +/// The programs that are not people - search engines, AI crawlers, link +/// previews, monitors, scanners, HTTP libraries - by the agents they send. +/// +/// +/// Not written here: the patterns are those of crawler-user-agents +/// (github.com/monperrus/crawler-user-agents, MIT), a list kept by many people +/// and checked against browsers of people in its own tests, with a kind for +/// every entry. A guess of our own - "bot" anywhere in the agent - was taken +/// in by a phone called Cubot and missed half of what the list knows. +/// +/// The copy of the release is embedded, and the job that keeps the crawlers' +/// addresses current fetches the list again weekly (), +/// so a crawler that appears after a release is known without one. A list +/// that does not read is not taken, and the one before stays. +/// +/// The agents come from whoever sends a request, so nothing here may take +/// longer the more cleverly an agent is written. Nearly every pattern of the +/// list is plain text (1,465 of 1,501 when it was added), and those are looked +/// for all at once in one pass over the agent; text that has to begin the +/// agent is compared with its start; and the few real expressions run on the +/// engine that cannot backtrack. One that engine cannot take is left out +/// rather than run on the other. +/// +public sealed class CrawlerList +{ + private const string Resource = "GenHTTP.Lambda.Resources.Crawlers.crawler-user-agents.json"; + + /// + /// The kinds of the list, the one a crawler is counted under first where + /// it has several - a search engine that also feeds an AI is a search + /// engine to whoever wants to be found. + /// + public static IReadOnlyList Kinds { get; } = + [ + "search-engine", "ai-crawler", "social-preview", "advertising", "seo", "monitoring", "feed-reader", "archiver", + "academic", "scanner", "browser-automation", "http-library" + ]; + + /// + /// What a crawler is counted under whose kinds are none of . + /// + public const string OtherKind = "other"; + + private Snapshot _snapshot; + + #region Get-/Setters + + /// + /// How many patterns are in use. + /// + public int Count => Volatile.Read(ref _snapshot).Patterns.Length; + + /// + /// Changes whenever another list is taken, so what was worked out with + /// the one before can be let go of. + /// + public int Generation => Volatile.Read(ref _snapshot).Generation; + + #endregion + + #region Initialization + + public CrawlerList() + { + using var stream = Assembly.GetExecutingAssembly().GetManifestResourceStream(Resource) + ?? throw new InvalidOperationException($"The embedded crawler list '{Resource}' is missing."); + + _snapshot = Read(stream, 0) ?? throw new InvalidOperationException("The embedded crawler list does not read."); + } + + #endregion + + #region Functionality + + /// + /// Takes a newer list, as published. + /// + /// Whether it was taken - not when it does not read, or holds nothing + public bool Load(Stream json) + { + var current = Volatile.Read(ref _snapshot); + + var loaded = Read(json, current.Generation + 1); + + if (loaded == null) + { + return false; + } + + Volatile.Write(ref _snapshot, loaded); + + return true; + } + + /// + /// The kind of crawler an agent belongs to, or nothing where it is none + /// the list knows. + /// + public string? KindOf(string agent) + { + var snapshot = Volatile.Read(ref _snapshot); + + // most agents are browsers, and need no more than to be found in none + // of them: one pass for all the text, a look at the start, and the few + // expressions + if (!agent.AsSpan().ContainsAny(snapshot.Texts) + && !snapshot.Patterns.Any(p => p.Prefix != null && agent.StartsWith(p.Prefix, StringComparison.Ordinal)) + && !snapshot.Patterns.Any(p => p.Expression != null && p.Expression.IsMatch(agent))) + { + return null; + } + + // the first that matches says what kind it is, as the list orders them + foreach (var pattern in snapshot.Patterns) + { + if (pattern.Matches(agent)) + { + return pattern.Kind; + } + } + + return OtherKind; + } + + private static Snapshot? Read(Stream json, int generation) + { + List patterns = []; + + try + { + using var document = JsonDocument.Parse(json); + + foreach (var entry in document.RootElement.EnumerateArray()) + { + if (!entry.TryGetProperty("pattern", out var source) || source.GetString() is not { Length: > 0 } text) + { + continue; + } + + var tags = entry.TryGetProperty("tags", out var listed) && listed.ValueKind == JsonValueKind.Array + ? listed.EnumerateArray().Select(t => t.GetString()).ToHashSet() + : []; + + var kind = Kinds.FirstOrDefault(tags.Contains) ?? OtherKind; + + if (Literal(text) is { Length: > 0 } literal) + { + patterns.Add(new Pattern(kind, Text: literal)); + } + else if (text[0] == '^' && Literal(text[1..]) is { Length: > 0 } prefix) + { + patterns.Add(new Pattern(kind, Prefix: prefix)); + } + else + { + try + { + patterns.Add(new Pattern(kind, Expression: new Regex(text, Options))); + } + catch (Exception e) when (e is ArgumentException or NotSupportedException) + { + // a construct the engine without backtracking does not + // have; one crawler fewer rather than a pattern that + // could take a request's time + } + } + } + } + catch (JsonException) + { + return null; + } + catch (InvalidOperationException) + { + return null; + } + + if (patterns.Count == 0) + { + return null; + } + + var texts = SearchValues.Create([.. patterns.Where(p => p.Text != null).Select(p => p.Text!).Distinct()], StringComparison.Ordinal); + + return new Snapshot([.. patterns], texts, generation); + } + + /// + /// The text a pattern stands for where it is nothing but text, with the + /// escapes of punctuation taken out - Googlebot\/ is + /// Googlebot/ - or nothing where it is an expression. + /// + private static string? Literal(string pattern) + { + var text = new StringBuilder(pattern.Length); + + for (var i = 0; i < pattern.Length; i++) + { + var c = pattern[i]; + + if (c == '\\') + { + // an escaped letter or digit is a class (\d, \s), anything + // else is the character itself + if (i + 1 < pattern.Length && !char.IsLetterOrDigit(pattern[i + 1])) + { + text.Append(pattern[++i]); + continue; + } + + return null; + } + + if (".^$*+?()[]{}|".Contains(c)) + { + return null; + } + + text.Append(c); + } + + return text.ToString(); + } + + private const RegexOptions Options = RegexOptions.NonBacktracking | RegexOptions.CultureInvariant; + + /// + /// One pattern of the list: text found anywhere, text the agent begins + /// with, or an expression. + /// + private sealed record Pattern(string Kind, string? Text = null, string? Prefix = null, Regex? Expression = null) + { + public bool Matches(string agent) + => Text != null ? agent.Contains(Text, StringComparison.Ordinal) + : Prefix != null ? agent.StartsWith(Prefix, StringComparison.Ordinal) + : Expression!.IsMatch(agent); + } + + private sealed record Snapshot(Pattern[] Patterns, SearchValues Texts, int Generation); + + #endregion + +} diff --git a/src/GenHTTP.Lambda/Services/Analytics/CrawlerRanges.cs b/src/GenHTTP.Lambda/Services/Analytics/CrawlerRanges.cs new file mode 100644 index 0000000..31be6c7 --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Analytics/CrawlerRanges.cs @@ -0,0 +1,256 @@ +using System.Buffers.Binary; +using System.Net; +using System.Net.Sockets; +using System.Text.Json; +using System.Text.RegularExpressions; + +namespace GenHTTP.Lambda.Services.Analytics; + +/// +/// The crawlers that can be told from impostors: those whose operators publish +/// the addresses they crawl from. +/// +/// +/// Anybody can send Googlebot as their agent, and scrapers do, to be let +/// in where Google is. So a visit is only counted as Google's when it came +/// from an address Google publishes for its crawlers; one that claims to be +/// Google from anywhere else is counted as a claim that was not confirmed. +/// Google, Bing, OpenAI, Apple and Perplexity publish theirs, as JSON in the +/// same shape, fetched and kept current by . +/// The others are not named at all: the list says what kind of crawler they +/// are (), which is as much as an agent can be trusted +/// to say. +/// +/// The address checked is the one the connection came from, never the one a +/// forwarding header claims: the header is written by the sender, and would +/// let anybody claim Google's address with Google's name. An installation +/// behind a proxy therefore confirms nothing - its connections come from the +/// proxy - and says so by counting every claim as unconfirmed. +/// +public sealed class CrawlerRanges +{ + + /// + /// The operators whose crawlers are confirmed: how their crawlers name + /// themselves, and where they publish the addresses those crawl from. + /// + /// + /// The names are those each operator documents for the crawlers its lists + /// cover. A list going away, or moving, leaves its operator unconfirmed + /// until this is changed - the job logs a failed fetch. + /// + public static IReadOnlyList Operators { get; } = + [ + new("Google", @"Googlebot|Google-InspectionTool|GoogleOther|Storebot-Google|AdsBot-Google|Mediapartners-Google|APIs-Google|FeedFetcher-Google|Google-Read-Aloud|Google-Site-Verification|Google-CloudVertexBot|Google-Safety", + [ + "https://developers.google.com/static/search/apis/ipranges/googlebot.json", + "https://developers.google.com/static/search/apis/ipranges/special-crawlers.json", + "https://developers.google.com/static/search/apis/ipranges/user-triggered-fetchers.json" + ]), + new("Bing", @"bingbot|BingPreview|msnbot|adidxbot", ["https://www.bing.com/toolbox/bingbot.json"]), + new("OpenAI", @"GPTBot|OAI-SearchBot|ChatGPT-User", + ["https://openai.com/gptbot.json", "https://openai.com/searchbot.json", "https://openai.com/chatgpt-user.json"]), + new("Apple", @"Applebot", ["https://search.developer.apple.com/applebot.json"]), + new("Perplexity", @"PerplexityBot|Perplexity-User", + ["https://www.perplexity.com/perplexitybot.json", "https://www.perplexity.com/perplexity-user.json"]) + ]; + + /// + /// What a claim is counted under that was not confirmed: from another + /// address, from one that is not known, or before any list was read. + /// + public const string Unconfirmed = "unconfirmed"; + + private readonly Dictionary _ranges = []; + + private readonly Lock _lock = new(); + + #region Get-/Setters + + /// + /// The operators whose addresses are known, whose crawlers are confirmed. + /// + public IReadOnlyList Confirmed + { + get + { + lock (_lock) + { + return [.. Operators.Select(o => o.Name).Where(_ranges.ContainsKey)]; + } + } + } + + #endregion + + #region Functionality + + /// + /// The operator whose crawler an agent claims to be, or nothing. + /// + public static CrawlerOperator? Claimed(string agent) => Operators.FirstOrDefault(o => o.Claims.IsMatch(agent)); + + /// + /// What a crawler that claims to be the operator's is counted under: the + /// operator's name where it came from one of its addresses, unconfirmed + /// otherwise. + /// + /// Who asked, as the log names them: an address, or "claimed via peer" + public string Confirm(CrawlerOperator claimed, string? client) + { + var peer = PeerOf(client); + + if (peer == null) + { + return Unconfirmed; + } + + Ranges? ranges; + + lock (_lock) + { + _ranges.TryGetValue(claimed.Name, out ranges); + } + + return ranges != null && ranges.Contain(peer) ? claimed.Name : Unconfirmed; + } + + /// + /// Takes the addresses an operator publishes, from every list it + /// publishes them in. + /// + /// Whether any were read - lists that do not read leave the ones before + public bool Load(string name, IEnumerable lists) + { + var four = new List<(uint Start, uint End)>(); + var six = new List<(UInt128 Start, UInt128 End)>(); + + foreach (var list in lists) + { + try + { + using var document = JsonDocument.Parse(list); + + foreach (var prefix in document.RootElement.GetProperty("prefixes").EnumerateArray()) + { + foreach (var property in prefix.EnumerateObject()) + { + if (property.Value.GetString() is { } text && IPNetwork.TryParse(text, out var network)) + { + Add(network, four, six); + } + } + } + } + catch (Exception e) when (e is JsonException or InvalidOperationException or KeyNotFoundException) + { + // one list of an operator that does not read is not a reason + // to forget the others + } + } + + if (four.Count == 0 && six.Count == 0) + { + return false; + } + + lock (_lock) + { + _ranges[name] = new Ranges([.. four], [.. six]); + } + + return true; + } + + /// + /// The address a connection came from, out of the way the log names a + /// caller - never the address a forwarding header claimed. + /// + private static IPAddress? PeerOf(string? client) + { + if (client == null) + { + return null; + } + + var via = client.LastIndexOf(" via ", StringComparison.Ordinal); + + var peer = via < 0 ? client : client[(via + 5)..]; + + if (!IPAddress.TryParse(peer, out var address)) + { + return null; + } + + return address.IsIPv4MappedToIPv6 ? address.MapToIPv4() : address; + } + + private static void Add(IPNetwork network, List<(uint, uint)> four, List<(UInt128, UInt128)> six) + { + Span bytes = stackalloc byte[16]; + + if (network.BaseAddress.AddressFamily == AddressFamily.InterNetwork) + { + network.BaseAddress.TryWriteBytes(bytes, out _); + + var start = BinaryPrimitives.ReadUInt32BigEndian(bytes); + var size = network.PrefixLength == 0 ? uint.MaxValue : (1u << (32 - network.PrefixLength)) - 1; + + four.Add((start, start + size)); + } + else + { + network.BaseAddress.TryWriteBytes(bytes, out _); + + var start = BinaryPrimitives.ReadUInt128BigEndian(bytes); + var size = network.PrefixLength == 0 ? UInt128.MaxValue : (UInt128.One << (128 - network.PrefixLength)) - 1; + + six.Add((start, start + size)); + } + } + + /// + /// The addresses of one operator. A few thousand at most, so they are + /// walked rather than indexed. + /// + private sealed record Ranges((uint Start, uint End)[] Four, (UInt128 Start, UInt128 End)[] Six) + { + public bool Contain(IPAddress address) + { + Span bytes = stackalloc byte[16]; + + address.TryWriteBytes(bytes, out _); + + if (address.AddressFamily == AddressFamily.InterNetwork) + { + var value = BinaryPrimitives.ReadUInt32BigEndian(bytes); + + return Four.Any(r => value >= r.Start && value <= r.End); + } + + var six = BinaryPrimitives.ReadUInt128BigEndian(bytes); + + return Six.Any(r => six >= r.Start && six <= r.End); + } + } + + #endregion + +} + +/// +/// An operator whose crawlers can be confirmed by their addresses. +/// +/// What its crawlers are counted under, once confirmed +/// How its crawlers name themselves in their agents +/// Where it publishes the addresses they crawl from +public sealed record CrawlerOperator(string Name, string Tokens, IReadOnlyList Lists) +{ + + /// + /// Whether an agent claims to be one of its crawlers - without + /// backtracking, since agents come from anybody. + /// + public Regex Claims { get; } = new(Tokens, RegexOptions.NonBacktracking | RegexOptions.CultureInvariant | RegexOptions.IgnoreCase); + +} diff --git a/src/GenHTTP.Lambda/Services/Analytics/DailyVisitors.cs b/src/GenHTTP.Lambda/Services/Analytics/DailyVisitors.cs new file mode 100644 index 0000000..fc25c92 --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Analytics/DailyVisitors.cs @@ -0,0 +1,101 @@ +using System.Buffers.Binary; +using System.Security.Cryptography; +using System.Text; + +namespace GenHTTP.Lambda.Services.Analytics; + +/// +/// Tells a visitor seen before today from a new one, without remembering who +/// either was. +/// +/// +/// A visitor is their address and their browser, hashed with a salt made at +/// random for the day and the lambda - so the same person is the same hash all +/// day on one lambda, and a different one on another lambda or tomorrow. Only +/// the hashes of today and yesterday are held, in memory, and the salts with +/// them: nothing here is written anywhere, and once a day is over nothing can +/// tell whether a hash was anybody in particular. +/// +/// So what is counted is visitors per day. A server that restarts forgets who +/// it saw, and counts somebody who comes back the same day once more; an +/// installation that keeps no addresses counts no visitors at all. +/// +/// Used by the job that writes visits down, one at a time, never by a request. +/// +public sealed class DailyVisitors +{ + + /// + /// How many visitors of a lambda are told apart in a day. Past that every + /// one counts as new, rather than every hash being held. + /// + public const int Most = 200_000; + + private readonly Dictionary<(long Lambda, DateOnly Day), Seen> _days = []; + + #region Functionality + + /// + /// Whether the visitor is new to the lambda on that day. + /// + /// Whether they are, or nothing where there is no address to tell them by + public bool? IsNew(long lambda, DateOnly day, string? client, string? agent) + { + if (client == null) + { + return null; + } + + if (!_days.TryGetValue((lambda, day), out var seen)) + { + _days[(lambda, day)] = seen = new Seen(RandomNumberGenerator.GetBytes(32)); + } + + var hash = Hash(seen.Salt, client, agent); + + if (seen.Hashes.Count >= Most) + { + return !seen.Hashes.Contains(hash); + } + + return seen.Hashes.Add(hash); + } + + /// + /// Forgets every day before yesterday, with its salts. + /// + /// + /// Yesterday is kept, because a visit made just before midnight may be + /// written down just after it. + /// + public void Prune(DateOnly today) + { + foreach (var key in _days.Keys.Where(k => k.Day < today.AddDays(-1)).ToList()) + { + _days.Remove(key); + } + } + + private static ulong Hash(byte[] salt, string client, string? agent) + { + // only the address that was claimed: the proxy it came through says + // nothing about who it was + var cut = client.IndexOf(' '); + + var address = cut < 0 ? client : client[..cut]; + + Span digest = stackalloc byte[32]; + + HMACSHA256.HashData(salt, Encoding.UTF8.GetBytes(address + "\n" + agent), digest); + + return BinaryPrimitives.ReadUInt64LittleEndian(digest); + } + + private sealed record Seen(byte[] Salt) + { + public HashSet Hashes { get; } = []; + } + + #endregion + +} diff --git a/src/GenHTTP.Lambda/Services/Analytics/IAnalyticsService.cs b/src/GenHTTP.Lambda/Services/Analytics/IAnalyticsService.cs new file mode 100644 index 0000000..c472d8b --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Analytics/IAnalyticsService.cs @@ -0,0 +1,48 @@ +namespace GenHTTP.Lambda.Services.Analytics; + +/// +/// How a lambda's visitors reach it, counted by day for its owner where the +/// owner asked for it. +/// +/// +/// Opt-in: nothing is counted of a lambda until its owner switches analytics +/// on, and switching them off deletes what was counted. A year is kept of +/// every lambda; how much of it the owner is shown depends on the tier +/// (TierLimits.AnalyticsDays), so a lambda that becomes premium shows +/// what was counted before it did. +/// +/// Apart from the figures of the overview and the stats, which are counted in +/// memory since the server came up and are there for every lambda: these are +/// kept, and are about the audience rather than the program - who came, from +/// where, to which pages, on what. +/// +public interface IAnalyticsService +{ + + /// + /// What the lambda behind an editor key has counted, over the last days. + /// + /// How many days, today included - at most what its tier shows, which is also what is shown without one + /// What time it is, which tests move + LambdaAnalytics Get(string privateKey, int? days = null, DateTime? now = null); + + /// + /// Starts counting the visits of a lambda. Nothing happens where they are + /// counted already. + /// + LambdaAnalytics Enable(string privateKey); + + /// + /// Stops counting the visits of a lambda and deletes what was counted. + /// + /// The public key of the lambda + string Disable(string privateKey); + + /// + /// Writes down the visits that are waiting, and removes the days that are + /// no longer kept. Done by a job every few minutes. + /// + /// What time it is, which tests move + AnalyticsFlush Flush(DateTime? now = null); + +} diff --git a/src/GenHTTP.Lambda/Services/Analytics/IAnalyticsStore.cs b/src/GenHTTP.Lambda/Services/Analytics/IAnalyticsStore.cs new file mode 100644 index 0000000..b2dceda --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Analytics/IAnalyticsStore.cs @@ -0,0 +1,14 @@ +namespace GenHTTP.Lambda.Services.Analytics; + +/// +/// Where the analytics of the lambdas are kept, as other services reach it. +/// +public interface IAnalyticsStore +{ + + /// + /// Removes everything counted of a lambda, called when it is deleted. + /// + void Forget(long lambda); + +} diff --git a/src/GenHTTP.Lambda/Services/Analytics/Migrations/V1__Analytics.sql b/src/GenHTTP.Lambda/Services/Analytics/Migrations/V1__Analytics.sql new file mode 100644 index 0000000..51294cd --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Analytics/Migrations/V1__Analytics.sql @@ -0,0 +1,114 @@ +-- The analytics of the lambdas, in a database of their own (analytics.db). +-- +-- A star: the lambdas whose visits are counted, and a table of facts per +-- statistic - the totals of a day, and one for each thing a visit is counted +-- by - each row the counts of one lambda on one day (in UTC), and of one value +-- where there is one. The day is kept as its year, month and day, which lead +-- the key after the lambda, so a lambda's days are read as a range of the key +-- and old ones are removed the same way. +-- +-- Written by a background job every few minutes, adding to the rows of the +-- day, never while a request is served. Kept apart from the platform's +-- database so that writing a batch of counts never holds up what the editor +-- and the API write there, and so that it can be backed up, moved or thrown +-- away on its own: nothing else depends on it. +-- +-- hits are every request answered, views the requests that were somebody +-- opening a page. + +CREATE TABLE lambdas ( + id INTEGER PRIMARY KEY, + created TEXT NOT NULL +); + +CREATE TABLE days ( + lambda INTEGER NOT NULL, + year INTEGER NOT NULL, + month INTEGER NOT NULL, + day INTEGER NOT NULL, + hits INTEGER NOT NULL DEFAULT 0, + views INTEGER NOT NULL DEFAULT 0, + visitors INTEGER NOT NULL DEFAULT 0, + failed INTEGER NOT NULL DEFAULT 0, + rejected INTEGER NOT NULL DEFAULT 0, + bytes INTEGER NOT NULL DEFAULT 0, + millis REAL NOT NULL DEFAULT 0, + PRIMARY KEY (lambda, year, month, day) +) WITHOUT ROWID; + +CREATE TABLE pages ( + lambda INTEGER NOT NULL, + year INTEGER NOT NULL, + month INTEGER NOT NULL, + day INTEGER NOT NULL, + value TEXT NOT NULL, + hits INTEGER NOT NULL DEFAULT 0, + views INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (lambda, year, month, day, value) +) WITHOUT ROWID; + +CREATE TABLE referrers ( + lambda INTEGER NOT NULL, + year INTEGER NOT NULL, + month INTEGER NOT NULL, + day INTEGER NOT NULL, + value TEXT NOT NULL, + hits INTEGER NOT NULL DEFAULT 0, + views INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (lambda, year, month, day, value) +) WITHOUT ROWID; + +CREATE TABLE countries ( + lambda INTEGER NOT NULL, + year INTEGER NOT NULL, + month INTEGER NOT NULL, + day INTEGER NOT NULL, + value TEXT NOT NULL, + hits INTEGER NOT NULL DEFAULT 0, + views INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (lambda, year, month, day, value) +) WITHOUT ROWID; + +CREATE TABLE devices ( + lambda INTEGER NOT NULL, + year INTEGER NOT NULL, + month INTEGER NOT NULL, + day INTEGER NOT NULL, + value TEXT NOT NULL, + hits INTEGER NOT NULL DEFAULT 0, + views INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (lambda, year, month, day, value) +) WITHOUT ROWID; + +CREATE TABLE browsers ( + lambda INTEGER NOT NULL, + year INTEGER NOT NULL, + month INTEGER NOT NULL, + day INTEGER NOT NULL, + value TEXT NOT NULL, + hits INTEGER NOT NULL DEFAULT 0, + views INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (lambda, year, month, day, value) +) WITHOUT ROWID; + +CREATE TABLE systems ( + lambda INTEGER NOT NULL, + year INTEGER NOT NULL, + month INTEGER NOT NULL, + day INTEGER NOT NULL, + value TEXT NOT NULL, + hits INTEGER NOT NULL DEFAULT 0, + views INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (lambda, year, month, day, value) +) WITHOUT ROWID; + +CREATE TABLE crawlers ( + lambda INTEGER NOT NULL, + year INTEGER NOT NULL, + month INTEGER NOT NULL, + day INTEGER NOT NULL, + value TEXT NOT NULL, + hits INTEGER NOT NULL DEFAULT 0, + views INTEGER NOT NULL DEFAULT 0, + PRIMARY KEY (lambda, year, month, day, value) +) WITHOUT ROWID; diff --git a/src/GenHTTP.Lambda/Services/Analytics/VisitTally.cs b/src/GenHTTP.Lambda/Services/Analytics/VisitTally.cs new file mode 100644 index 0000000..6603e12 --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Analytics/VisitTally.cs @@ -0,0 +1,203 @@ +using GenHTTP.Lambda.Services.Visits; + +namespace GenHTTP.Lambda.Services.Analytics; + +/// +/// The visits of one writing, summed into the rows they are written as. +/// +/// +/// A visit becomes one row of totals for its lambda and day and a row in each +/// dimension it says something about, and a writing adds to those rows rather +/// than to a row per visit: thousands of visits to the same page in five +/// minutes are one row to write. What a visit is counted as - the host it was +/// referred from, the device it came on - is worked out here, once, off the +/// requests. +/// +public sealed class VisitTally(DailyVisitors visitors, VisitorClassifier classifier) +{ + private readonly Dictionary<(long Lambda, DateOnly Day), Totals> _days = []; + + private readonly Dictionary _cells = []; + + #region Get-/Setters + + /// + /// The totals of each lambda and day. + /// + public IEnumerable<(long Lambda, DateOnly Day, AnalyticsTotals Totals)> Days + => _days.Select(d => (d.Key.Lambda, d.Key.Day, + new AnalyticsTotals(d.Value.Hits, d.Value.Views, d.Value.Visitors, d.Value.Failed, d.Value.Rejected, + d.Value.Bytes, d.Value.Millis))); + + /// + /// What each value of a dimension came to, per lambda and day. + /// + public IEnumerable<(Cell Cell, long Hits, long Views)> Cells => _cells.Select(c => (c.Key, c.Value.Hits, c.Value.Views)); + + /// + /// The lambdas anything was counted for. + /// + public IEnumerable Lambdas => _days.Keys.Select(k => k.Lambda).Distinct(); + + public bool IsEmpty => _days.Count == 0; + + #endregion + + #region Functionality + + /// + /// Counts a visit. + /// + public void Add(LambdaVisit visit) + { + var lambda = visit.Lambda.Id; + + var day = DateOnly.FromDateTime(visit.At); + + if (!_days.TryGetValue((lambda, day), out var totals)) + { + _days[(lambda, day)] = totals = new Totals(); + } + + var view = visit.Page ? 1 : 0; + + totals.Hits++; + totals.Views += view; + totals.Bytes += visit.Bytes; + + // an upgraded connection returned at the handshake; what follows is a + // socket, and its time says nothing about how fast the lambda answers + if (visit.Status != 101) + { + totals.Millis += visit.Elapsed.TotalMilliseconds; + } + + switch (visit.Status) + { + case >= 500: + totals.Failed++; + break; + case >= 400: + totals.Rejected++; + break; + } + + var agent = classifier.Classify(visit.Agent, visit.Hints, visit.Client); + + // a visitor is a person, counted with the first thing they asked for + // that day, whatever it was - a crawler opens pages, but is nobody + if (agent.Browser != null && visitors.IsNew(lambda, day, visit.Client, visit.Agent) == true) + { + totals.Visitors++; + } + + // what was not there is not a page of the lambda: a scanner's guesses + // are in the totals, and the paths stay the lambda's own + if (visit.Status < 400) + { + Count(lambda, day, AnalyticsDimension.Pages, visit.Path, view); + } + + Count(lambda, day, AnalyticsDimension.Countries, visit.Country ?? string.Empty, view); + + if (ReferrerOf(visit.Referrer, visit.Domain) is { } referrer) + { + Count(lambda, day, AnalyticsDimension.Referrers, referrer, view); + } + + Count(lambda, day, AnalyticsDimension.Devices, agent.Device, view); + + if (agent.Browser != null) + { + Count(lambda, day, AnalyticsDimension.Browsers, agent.Browser, view); + } + + if (agent.System != null) + { + Count(lambda, day, AnalyticsDimension.Systems, agent.System, view); + } + + if (agent.Crawler != null) + { + Count(lambda, day, AnalyticsDimension.Crawlers, agent.Crawler, view); + } + } + + /// + /// The site a visitor came from, by its host, or nothing where they came + /// from the lambda itself or said nothing. + /// + /// + /// Only the host is kept. Most browsers send no more than that to another + /// site anyway - a search engine is the address of the search engine and + /// not what was searched for - and where they do send the whole address, + /// it may carry what was typed into a form or a token of somebody's. + /// The leading www is left off, so a site is one row whichever way it was + /// written. + /// + public static string? ReferrerOf(string? referrer, string? domain) + { + if (string.IsNullOrWhiteSpace(referrer) || !Uri.TryCreate(referrer, UriKind.Absolute, out var address) || address.Host.Length == 0) + { + return null; + } + + var host = Bare(address.Host); + + if (domain != null && host == Bare(domain)) + { + return null; + } + + return host.Length <= 100 ? host : host[..100]; + } + + private static string Bare(string host) + { + host = host.ToLowerInvariant().TrimEnd('.'); + + return host.StartsWith("www.", StringComparison.Ordinal) ? host[4..] : host; + } + + private void Count(long lambda, DateOnly day, AnalyticsDimension dimension, string value, int view) + { + var cell = new Cell(lambda, day, dimension, value); + + if (!_cells.TryGetValue(cell, out var counts)) + { + _cells[cell] = counts = new Counts(); + } + + counts.Hits++; + counts.Views += view; + } + + #endregion + + #region Rows + + /// + /// One value of a dimension on a day of a lambda. + /// + public readonly record struct Cell(long Lambda, DateOnly Day, AnalyticsDimension Dimension, string Value); + + private sealed class Totals + { + internal long Hits; + internal long Views; + internal long Visitors; + internal long Failed; + internal long Rejected; + internal long Bytes; + internal double Millis; + } + + private sealed class Counts + { + internal long Hits; + internal long Views; + } + + #endregion + +} diff --git a/src/GenHTTP.Lambda/Services/Analytics/VisitorClassifier.cs b/src/GenHTTP.Lambda/Services/Analytics/VisitorClassifier.cs new file mode 100644 index 0000000..0c96d4a --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Analytics/VisitorClassifier.cs @@ -0,0 +1,324 @@ +using System.Collections.Concurrent; +using System.Text.RegularExpressions; + +using GenHTTP.Lambda.Services.Visits; + +namespace GenHTTP.Lambda.Services.Analytics; + +/// +/// Whether a visitor was a person or a program, and what they came with. +/// +/// +/// In this order, each step only where the one before said nothing: +/// +/// 1. A crawler that claims to be one of the operators that publish their +/// addresses is counted as theirs where it came from one of them, and as +/// an unconfirmed claim otherwise (). +/// 2. A crawler the list of crawlers knows is counted by its kind +/// (). +/// 3. Anything that does not name itself as a browser does - no agent, an +/// app's own client - is neither: "other", and no visitor. +/// 4. A person, with the browser, system and device the client hints say +/// where a Chromium browser sent them, and its agent's product tokens +/// otherwise. +/// +/// Everything a visitor says about themselves can be made up, so this is what +/// they claim, coarsely, and never more: an owner learns that most of their +/// visitors come on phones, not who anybody is. Values that come from the +/// sender are only ever taken where they are one of a known few, so nobody can +/// write a word of their own into an owner's figures. +/// +/// Worked out when visits are written down, never while a request is served, +/// and remembered per agent: a busy day is a few hundred different agents +/// sent a hundred thousand times. +/// +public sealed class VisitorClassifier(CrawlerList crawlers, CrawlerRanges ranges) +{ + + public const string Desktop = "desktop"; + + public const string Mobile = "mobile"; + + public const string Tablet = "tablet"; + + public const string Bot = "bot"; + + public const string Unknown = "other"; + + /// + /// How many agents are remembered. Past that, it starts over. + /// + private const int MostRemembered = 20_000; + + /// + /// How much of an agent is read. The request line keeps 200 characters of + /// it already; this holds for anything else that calls. + /// + private const int MostRead = 500; + + private readonly ConcurrentDictionary<(string Agent, ClientHints? Hints), Reading> _known = []; + + private int _generation = -1; + + #region Functionality + + /// + /// What a visitor was, from what they sent. + /// + /// Their User-Agent + /// The client hints of a Chromium browser, where it sent them + /// Who asked, as the log names them - what a crawler's claim is checked against + public VisitorAgent Classify(string? agent, ClientHints? hints = null, string? client = null) + { + if (string.IsNullOrWhiteSpace(agent)) + { + return new VisitorAgent(Unknown, null, null, null); + } + + // a newer list of crawlers makes what was read with the one before old + if (Interlocked.Exchange(ref _generation, crawlers.Generation) != crawlers.Generation || _known.Count >= MostRemembered) + { + _known.Clear(); + } + + // cut before it is remembered, so what is kept of an agent is bounded + // however long it was sent + var read = agent.Length <= MostRead ? agent : agent[..MostRead]; + + var reading = _known.GetOrAdd((read, hints), key => Read(key.Agent, key.Hints)); + + // the claim is remembered, its check is not: it is the address that + // decides, and every visit has its own + if (reading.Claimed is { } claimed) + { + return new VisitorAgent(Bot, null, null, ranges.Confirm(claimed, client)); + } + + return reading.Agent!; + } + + private Reading Read(string agent, ClientHints? hints) + { + if (CrawlerRanges.Claimed(agent) is { } claimed) + { + return new Reading(null, claimed); + } + + if (crawlers.KindOf(agent) is { } kind) + { + return new Reading(new VisitorAgent(Bot, null, null, kind), null); + } + + // a browser names itself Mozilla, for reasons older than most of them + if (!agent.StartsWith("Mozilla/", StringComparison.Ordinal) && !agent.StartsWith("Opera/", StringComparison.Ordinal)) + { + return new Reading(new VisitorAgent(Unknown, null, null, null), null); + } + + // the hints of Chromium, only where the agent is Chromium's too: what + // another browser sends as hints is not what it is + var chromium = Has(agent, "Chrome/") || Has(agent, "Chromium/"); + + var trusted = chromium ? hints : null; + + var system = SystemOfHints(trusted?.Platform) ?? SystemOfAgent(agent); + + return new Reading(new VisitorAgent(DeviceOf(agent, trusted, system), BrandOf(trusted?.Brands) ?? BrowserOf(agent), system, null), null); + } + + #endregion + + #region Browsers + + /// + /// Apps that open pages in a browser of their own, which their agents + /// name: a visitor from Instagram is worth telling from one in Safari. + /// + private static readonly (string Token, string Name)[] Apps = + [ + ("Instagram ", "Instagram"), ("FBAN/", "Facebook"), ("FBAV/", "Facebook"), ("FB_IAB/", "Facebook"), + ("musical_ly", "TikTok"), ("BytedanceWebview", "TikTok"), ("MicroMessenger/", "WeChat"), (" Line/", "LINE"), + ("Snapchat", "Snapchat"), ("Pinterest/", "Pinterest"), ("LinkedInApp", "LinkedIn") + ]; + + /// + /// Browsers by the token they add to the agent, the specific before the + /// general: Edge, Opera and Samsung Internet name Chrome as well, and + /// Chrome names Safari. + /// + private static readonly (string Token, string Name)[] Browsers = + [ + ("Edg/", "Edge"), ("EdgA/", "Edge"), ("EdgiOS/", "Edge"), + ("OPR/", "Opera"), ("OPT/", "Opera"), ("Opera", "Opera"), + ("SamsungBrowser/", "Samsung Internet"), + ("YaBrowser/", "Yandex Browser"), + ("Vivaldi/", "Vivaldi"), + ("UCBrowser/", "UC Browser"), + ("DuckDuckGo/", "DuckDuckGo"), + ("Firefox/", "Firefox"), ("FxiOS/", "Firefox"), + ("CriOS/", "Chrome"), ("Chrome/", "Chrome"), ("Chromium/", "Chromium") + ]; + + private static string BrowserOf(string agent) + { + foreach (var (token, name) in Apps) + { + if (Has(agent, token)) + { + return name; + } + } + + foreach (var (token, name) in Browsers) + { + if (Has(agent, token)) + { + return name; + } + } + + // Safari says which version it is; a page shown inside an app on an + // iPhone says Safari, but not that + return Has(agent, "Safari/") && Has(agent, "Version/") ? "Safari" : Unknown; + } + + /// + /// The brands a Chromium browser names in its hints, by the names the + /// agent would give them. + /// + private static readonly (string Brand, string Name)[] Brands = + [ + ("Google Chrome", "Chrome"), ("Microsoft Edge", "Edge"), ("Opera", "Opera"), ("Opera GX", "Opera"), ("Brave", "Brave"), + ("Samsung Internet", "Samsung Internet"), ("YaBrowser", "Yandex Browser"), ("Yandex", "Yandex Browser"), ("Vivaldi", "Vivaldi") + ]; + + /// + /// The quoted brands of Sec-CH-UA: "Google Chrome";v="141", "Not?A_Brand";v="8", "Chromium";v="141". + /// + private static readonly Regex Quoted = new("\"([^\"]{1,40})\"", RegexOptions.NonBacktracking | RegexOptions.CultureInvariant); + + /// + /// The browser the hints name - not Chromium, which every one of them is, + /// and not the made-up brand they add so nobody relies on the order + /// (GREASE, "Not?A_Brand") - or nothing where it is none of the known. + /// + private static string? BrandOf(string? brands) + { + if (brands == null) + { + return null; + } + + foreach (Match match in Quoted.Matches(brands)) + { + var brand = match.Groups[1].Value; + + foreach (var (known, name) in Brands) + { + if (string.Equals(brand, known, StringComparison.OrdinalIgnoreCase)) + { + return name; + } + } + } + + return null; + } + + #endregion + + #region Systems and devices + + private static string? SystemOfHints(string? platform) => platform?.Trim('"', ' ') switch + { + null => null, + "Windows" => "Windows", + "macOS" => "macOS", + "Android" => "Android", + "iOS" => "iOS", + "Chrome OS" or "Chromium OS" => "ChromeOS", + "Linux" => "Linux", + _ => null + }; + + private static string SystemOfAgent(string agent) + { + if (Has(agent, "Windows")) + { + return "Windows"; + } + + if (Has(agent, "iPhone") || Has(agent, "iPad") || Has(agent, "iPod")) + { + return "iOS"; + } + + // before Linux, which every Android agent names as well + if (Has(agent, "Android")) + { + return "Android"; + } + + if (Has(agent, "CrOS")) + { + return "ChromeOS"; + } + + if (Has(agent, "Macintosh") || Has(agent, "Mac OS X")) + { + return "macOS"; + } + + return Has(agent, "Linux") || Has(agent, "X11") ? "Linux" : Unknown; + } + + /// + /// A phone, a tablet or a computer. The hints say whether it is a phone; + /// an Android that is not one is a tablet, as Android says in its agent by + /// leaving out "Mobile". An iPad that asks for the pages of a computer + /// says Macintosh and is counted as one - there is nothing else to tell + /// it by. + /// + private static string DeviceOf(string agent, ClientHints? hints, string system) + { + switch (hints?.Mobile) + { + case "?1": + return Mobile; + case "?0": + return system == "Android" ? Tablet : Desktop; + } + + if (Has(agent, "iPad") || Has(agent, "Tablet") || (Has(agent, "Android") && !Has(agent, "Mobile"))) + { + return Tablet; + } + + // "Mobi" is what Mozilla recommends looking for; the rest are the + // phones that do not say it + return Has(agent, "Mobi") || Has(agent, "iPhone") || Has(agent, "iPod") ? Mobile : Desktop; + } + + private static bool Has(string agent, string token) => agent.Contains(token, StringComparison.Ordinal); + + /// + /// What an agent was read as, apart from the address: a claim to be one of + /// the operators' crawlers is checked per visit. + /// + private sealed record Reading(VisitorAgent? Agent, CrawlerOperator? Claimed); + + #endregion + +} + +/// +/// What a visitor used. +/// +/// desktop, mobile, tablet, bot or other +/// The browser of a person, or nothing for anything else +/// The operating system of a person, or nothing for anything else +/// +/// Whose crawler it is - an operator that was confirmed, or "unconfirmed" - or what kind it is, for a bot; nothing for +/// a person +/// +public sealed record VisitorAgent(string Device, string? Browser, string? System, string? Crawler); diff --git a/src/GenHTTP.Lambda/Services/Diagnostics/Caller.cs b/src/GenHTTP.Lambda/Services/Diagnostics/Caller.cs index 8bbecc6..a2a5e53 100644 --- a/src/GenHTTP.Lambda/Services/Diagnostics/Caller.cs +++ b/src/GenHTTP.Lambda/Services/Diagnostics/Caller.cs @@ -63,16 +63,15 @@ public sealed record CallerInfo(string? Client, string? Agent, string Method, st public static CallerInfo From(IRequest request, StringPool pool, bool addresses, GeoTable? geo = null, GeoPlaces? places = null, string? domain = null) { - string? client = null; + var peer = Format(request.Client.Address); - if (addresses) - { - var peer = Format(request.Client.Address); + var claimed = First(request.Header.Headers.GetEntry("X-Forwarded-For")); - var claimed = First(request.Header.Headers.GetEntry("X-Forwarded-For")); + var address = claimed == null || claimed == peer ? peer : $"{claimed} via {peer}"; - client = claimed == null || claimed == peer ? peer : $"{claimed} via {peer}"; - } + // the country is placed whether or not the address is kept: it names + // nobody, and the owner's analytics count visits by it + var client = addresses ? address : null; return new CallerInfo( pool.Share(client), @@ -81,7 +80,7 @@ public static CallerInfo From(IRequest request, StringPool pool, bool addresses, request.Header.Path.ToString(), // looked up once here rather than per line, and the codes are // already shared by the table that produced them - geo?.CountryOf(client), + geo?.CountryOf(address), // a town and a network, where a database has one. Pooled: a busy // server sees the same few hundred callers over and over pool.Share(Empty(places?.Find(client)?.Describe())), diff --git a/src/GenHTTP.Lambda/Services/Diagnostics/CallerConcern.cs b/src/GenHTTP.Lambda/Services/Diagnostics/CallerConcern.cs index 2c96638..6c67884 100644 --- a/src/GenHTTP.Lambda/Services/Diagnostics/CallerConcern.cs +++ b/src/GenHTTP.Lambda/Services/Diagnostics/CallerConcern.cs @@ -6,6 +6,7 @@ using GenHTTP.Lambda.Configuration; using GenHTTP.Lambda.Services.Hosting; using GenHTTP.Lambda.Services.Protection; +using GenHTTP.Lambda.Services.Visits; using GenHTTP.Modules.DependencyInjection; @@ -91,16 +92,27 @@ private void Record(CallerInfo caller, IRequest request, IResponse? response, Ti return; } + /* + * A request a lambda or a preview answered was measured once, as the + * visit it was (see LambdaVisitConcern), and the line says what the + * lambda's figures say. Only a request to the platform, or one no + * lambda was found for, is measured here. + */ + var visit = request.GetVisit(); + // nothing answered is a not found by the time the client sees it - var status = response != null ? (int)response.Status : 404; + var status = visit?.Status ?? (response != null ? (int)response.Status : 404); + + var bytes = visit?.Bytes ?? (long)(response?.Content?.Length ?? 0); - var bytes = response?.Content?.Length ?? 0; + took = visit?.Elapsed ?? took; var level = status >= 500 ? "error" : status >= 400 ? "warn" : "info"; // resolved by a concern below this one, so by the time the answer is - // on its way back the request knows which lambda it reached - var lambda = request.GetLambda(); + // on its way back the request knows which lambda it reached - even + // where it never got to be a visit, refused by the rate limit + var lambda = visit?.Lambda ?? request.GetLambda(); // a lambda's host is named in front of the path: its paths are its // own, and "/api/v1/logs" there is not the platform's diff --git a/src/GenHTTP.Lambda/Services/Execution/LambdaRoute.cs b/src/GenHTTP.Lambda/Services/Execution/LambdaRoute.cs index 11493d1..7cb81ea 100644 --- a/src/GenHTTP.Lambda/Services/Execution/LambdaRoute.cs +++ b/src/GenHTTP.Lambda/Services/Execution/LambdaRoute.cs @@ -1,11 +1,13 @@ using GenHTTP.Api.Content; using GenHTTP.Lambda.Configuration; +using GenHTTP.Lambda.Services.Analytics; using GenHTTP.Lambda.Services.Deployment; using GenHTTP.Lambda.Services.Diagnostics; using GenHTTP.Lambda.Services.Protection; using GenHTTP.Lambda.Services.Settings; using GenHTTP.Lambda.Services.Telemetry; +using GenHTTP.Lambda.Services.Visits; using GenHTTP.Modules.DependencyInjection; using GenHTTP.Modules.ErrorHandling; @@ -56,12 +58,20 @@ public static IHandler Create(IServiceProvider services, ILambdaLocator locator) .. options.CaptureLambdaOutput ? new IConcernBuilder[] { Dependent.Concern() } : [], - // outside the error handler and the throttle, inside the lookup: so - // it records the answer the visitor actually got, and knows which - // lambda to file it under. Inside the throttle it saw neither the - // timeout nor the rejection - a lambda that timed out on every - // request was recorded as succeeding, slowly. + // the visit the request was, made once: outside the error handler + // and the throttle, inside the lookup - so it is the answer the + // visitor actually got, filed under the lambda that gave it. + // Inside the throttle it saw neither the timeout nor the + // rejection - a lambda that timed out on every request was + // recorded as succeeding, slowly. + Dependent.Concern(), + // what reads the visit, outside what makes it, so it is on the + // request as the answer passes: the figures, and the owner's + // analytics, which only queue it - nothing is written while the + // visitor waits. The request log reads it at the outside of the + // host. Dependent.Concern(), + Dependent.Concern(), new LambdaResolutionConcernBuilder(locator), Dependent.Concern() ], execution); diff --git a/src/GenHTTP.Lambda/Services/Meta/LambdaRemoval.cs b/src/GenHTTP.Lambda/Services/Meta/LambdaRemoval.cs index 4a7ad24..ffb4daf 100644 --- a/src/GenHTTP.Lambda/Services/Meta/LambdaRemoval.cs +++ b/src/GenHTTP.Lambda/Services/Meta/LambdaRemoval.cs @@ -1,5 +1,6 @@ using GenHTTP.Lambda.Data; using GenHTTP.Lambda.Data.Entities; +using GenHTTP.Lambda.Services.Analytics; using GenHTTP.Lambda.Services.Databases; using GenHTTP.Lambda.Services.Deployment; using GenHTTP.Lambda.Services.Hosting; @@ -20,7 +21,7 @@ namespace GenHTTP.Lambda.Services.Meta; /// it, or the sweep finds it abandoned - and both have to leave nothing behind. /// public sealed class LambdaRemoval(IDeploymentService deployments, ITelemetryService telemetry, LambdaHistory history, ISecretVault secrets, - IDatabaseVault databaseVault, IDomainRegistry domains, IStorageService storage) + IDatabaseVault databaseVault, IDomainRegistry domains, IStorageService storage, IAnalyticsStore analytics) { public void Remove(LambdaDbContext database, LambdaEntity lambda) @@ -61,6 +62,9 @@ public void Remove(LambdaDbContext database, LambdaEntity lambda) } storage.Delete(lambda.Id); + + // what its owner counted of its visitors goes with it + analytics.Forget(lambda.Id); } } diff --git a/src/GenHTTP.Lambda/Services/Meta/MetaService.cs b/src/GenHTTP.Lambda/Services/Meta/MetaService.cs index 69628fa..93f801e 100644 --- a/src/GenHTTP.Lambda/Services/Meta/MetaService.cs +++ b/src/GenHTTP.Lambda/Services/Meta/MetaService.cs @@ -317,7 +317,8 @@ public void Delete(string privateKey) var workspace = DataSwitches.IsEnabled(database, lambda.Id, DataKinds.Workspace); - return new ResolvedLambda(lambda.Id, lambda.PublicKey, lambda.Tier, deployment.Version, deployment.Created, workspace); + return new ResolvedLambda(lambda.Id, lambda.PublicKey, lambda.Tier, deployment.Version, deployment.Created, workspace, + AnalyticsSince: lambda.AnalyticsSince); } public IReadOnlyList GetVersions(string privateKey) diff --git a/src/GenHTTP.Lambda/Services/Meta/Model/LambdaModel.cs b/src/GenHTTP.Lambda/Services/Meta/Model/LambdaModel.cs index b0acf33..5bb026d 100644 --- a/src/GenHTTP.Lambda/Services/Meta/Model/LambdaModel.cs +++ b/src/GenHTTP.Lambda/Services/Meta/Model/LambdaModel.cs @@ -99,8 +99,12 @@ public sealed record LambdaActivation(int Version, DateTime Started, string? Ori /// The feature whose preview is being served, or nothing for the lambda itself. A preview runs the feature's files /// against the feature's copy of the data, and is kept out of the lambda's figures. /// +/// +/// Since when its owner has its visits counted, or nothing while analytics are off - read on every request, so +/// switching them on counts from the next one, and switching them off stops at once. +/// public sealed record ResolvedLambda(long Id, string PublicKey, LambdaTier Tier, int ActiveVersion, DateTime DeployedAt, - bool WorkspaceEnabled = true, ResolvedFeature? Feature = null); + bool WorkspaceEnabled = true, ResolvedFeature? Feature = null, DateTime? AnalyticsSince = null); /// /// The preview of a feature a request was addressed to. diff --git a/src/GenHTTP.Lambda/Services/Settings/LimitsService.cs b/src/GenHTTP.Lambda/Services/Settings/LimitsService.cs index 2132349..1496e69 100644 --- a/src/GenHTTP.Lambda/Services/Settings/LimitsService.cs +++ b/src/GenHTTP.Lambda/Services/Settings/LimitsService.cs @@ -3,6 +3,7 @@ using GenHTTP.Lambda.Configuration; using GenHTTP.Lambda.Data; using GenHTTP.Lambda.Data.Entities; +using GenHTTP.Lambda.Services.Analytics; using GenHTTP.Lambda.Services.Meta; using GenHTTP.Lambda.Services.Workspace; @@ -39,6 +40,17 @@ public sealed class LimitsService(IDbContextFactory databases, private ProductLimits? _current; + /// + /// How many days of its analytics a free lambda shows its owner: a week, + /// enough to see whether anybody comes. + /// + private const int FreeAnalyticsDays = 7; + + /// + /// How many a premium lambda shows: the year that is kept of every lambda. + /// + private const int PremiumAnalyticsDays = 365; + #region Reading /// @@ -106,8 +118,8 @@ public long DataOf(LambdaTier tier) /// options say, which is the environment or the built in defaults. /// public static ProductLimits DefaultsOf(LambdaOptions options) => new( - new TierLimits(options.BuildBytes, options.DataBytes, options.MaxVersions, options.MaxFeatures), - new TierLimits(options.PremiumBuildBytes, options.PremiumDataBytes, options.MaxVersions, options.MaxFeatures), + new TierLimits(options.BuildBytes, options.DataBytes, options.MaxVersions, options.MaxFeatures, FreeAnalyticsDays), + new TierLimits(options.PremiumBuildBytes, options.PremiumDataBytes, options.MaxVersions, options.MaxFeatures, PremiumAnalyticsDays), options.DeploymentLifetime, options.Retention, options.MaxShowcaseImageBytes, @@ -179,6 +191,7 @@ public long DataOf(LambdaTier tier) yield return ($"{Prefix}{tier}.data-bytes", Text(values.DataBytes)); yield return ($"{Prefix}{tier}.versions", Text(values.Versions)); yield return ($"{Prefix}{tier}.features", Text(values.Features)); + yield return ($"{Prefix}{tier}.analytics-days", Text(values.AnalyticsDays)); } yield return ($"{Prefix}offline-after-hours", Text(limits.OfflineAfter.TotalHours)); @@ -207,6 +220,11 @@ private static void Validate(ProductLimits limits) throw LambdaException.Invalid($"The data of a lambda needs room for at least one block of {WorkspaceLimits.Block} bytes."); } + if (limits.Free.AnalyticsDays > AnalyticsStore.Kept || limits.Premium.AnalyticsDays > AnalyticsStore.Kept) + { + throw LambdaException.Invalid($"Analytics are kept for {AnalyticsStore.Kept} days, so no tier can show more of them."); + } + var free = limits.Free; var premium = limits.Premium; @@ -215,7 +233,8 @@ private static void Validate(ProductLimits limits) ("build bytes", premium.BuildBytes < free.BuildBytes), ("data bytes", premium.DataBytes < free.DataBytes), ("versions", premium.Versions < free.Versions), - ("features", premium.Features < free.Features) + ("features", premium.Features < free.Features), + ("analytics days", premium.AnalyticsDays < free.AnalyticsDays) }.Where(c => c.Smaller).Select(c => c.Name).ToList(); if (smaller.Count > 0) @@ -253,7 +272,8 @@ private ProductLimits Read() (long)Number(stored, $"{tier}.build-bytes", Before(stored, tier, "code-characters", "asset-bytes", defaults.BuildBytes)), (long)Number(stored, $"{tier}.data-bytes", Before(stored, tier, "workspace-bytes", "database-bytes", defaults.DataBytes)), (int)Number(stored, $"{tier}.versions", defaults.Versions), - (int)Number(stored, $"{tier}.features", defaults.Features) + (int)Number(stored, $"{tier}.features", defaults.Features), + (int)Number(stored, $"{tier}.analytics-days", defaults.AnalyticsDays) ); /// @@ -306,7 +326,11 @@ private static double Number(Dictionary stored, string key, doub /// The room its data may take: its database and its workspace together /// How many of its versions are kept /// How many features it may have open at once -public sealed record TierLimits(long BuildBytes, long DataBytes, int Versions, int Features); +/// +/// How many days of its analytics its owner is shown. A year is counted of every lambda whatever its tier, so a lambda +/// moved to a tier that shows more shows what was counted before as well. +/// +public sealed record TierLimits(long BuildBytes, long DataBytes, int Versions, int Features, int AnalyticsDays); /// /// Every limit of the product. diff --git a/src/GenHTTP.Lambda/Services/Telemetry/LambdaActivityConcern.cs b/src/GenHTTP.Lambda/Services/Telemetry/LambdaActivityConcern.cs index 0a7d8ed..aa93e16 100644 --- a/src/GenHTTP.Lambda/Services/Telemetry/LambdaActivityConcern.cs +++ b/src/GenHTTP.Lambda/Services/Telemetry/LambdaActivityConcern.cs @@ -1,78 +1,40 @@ -using System.Diagnostics; - using GenHTTP.Api.Content; using GenHTTP.Api.Protocol; -using GenHTTP.Lambda.Services.Hosting; -using GenHTTP.Lambda.Services.Protection; +using GenHTTP.Lambda.Services.Visits; using GenHTTP.Modules.DependencyInjection; namespace GenHTTP.Lambda.Services.Telemetry; /// -/// Counts what a single lambda does. +/// Counts what a single lambda does, from the visit each request was. /// /// -/// Sits outside the throttle and the error handler, so what it records is the -/// answer the visitor was given - a timeout is a timeout here and not the -/// success the lambda eventually got round to - and what it times is the whole -/// wait rather than only the part spent running. +/// Outside the concern that makes the visit (), +/// so it finds it on the request as the answer passes - whether the lambda +/// answered or threw - and counts what the visitor was given without measuring +/// anything itself. /// public sealed class LambdaActivityConcern(LambdaTelemetry telemetry) : IDependentConcern { public async ValueTask HandleAsync(IHandler content, IRequest request) { - var lambda = request.GetLambda(); - - // a preview is somebody trying a change: counting it would make the - // lambda look used, and keep a free one online, while nobody visits it - if (lambda == null || lambda.Feature != null) - { - return await content.HandleAsync(request); - } - - // read before the lambda routes it: the target is a pointer that the - // handlers below move along, and afterwards it points at nothing - var path = PathOf(request); - - var domain = request.GetHost()?.Name; - - var started = Stopwatch.GetTimestamp(); - try { - var response = await content.HandleAsync(request); - - var status = response != null ? (int)response.Status : 404; - - telemetry.Record(lambda.Id, lambda.PublicKey, Stopwatch.GetElapsedTime(started), status, - (long)(response?.Content?.Length ?? 0), path, domain); - - return response; + return await content.HandleAsync(request); } - catch (Exception) + finally { - // the error handler above turns this into a response for the - // visitor, but as far as the lambda is concerned it failed - telemetry.Record(lambda.Id, lambda.PublicKey, Stopwatch.GetElapsedTime(started), 500, 0, path, domain); - throw; + // a preview is somebody trying a change: counting it would make + // the lambda look used, and keep a free one online, while nobody + // visits it + if (request.GetVisit() is { Preview: false } visit) + { + telemetry.Record(visit); + } } } - /// - /// What was asked for, relative to the lambda and cut to a length worth - /// showing - the lookup above has already stepped past the key, where - /// there was one. - /// - private static string PathOf(IRequest request) - { - var remaining = request.Header.Target.AsString(decode: false, remainingOnly: true); - - var path = remaining.Length == 0 ? "/" : remaining[0] == '/' ? remaining : "/" + remaining; - - return path.Length <= 100 ? path : string.Concat(path.AsSpan(0, 99), "…"); - } - } diff --git a/src/GenHTTP.Lambda/Services/Telemetry/LambdaTelemetry.cs b/src/GenHTTP.Lambda/Services/Telemetry/LambdaTelemetry.cs index b87f9fd..6f7f289 100644 --- a/src/GenHTTP.Lambda/Services/Telemetry/LambdaTelemetry.cs +++ b/src/GenHTTP.Lambda/Services/Telemetry/LambdaTelemetry.cs @@ -1,5 +1,7 @@ using System.Collections.Concurrent; +using GenHTTP.Lambda.Services.Visits; + namespace GenHTTP.Lambda.Services.Telemetry; /// @@ -55,6 +57,12 @@ public sealed class LambdaTelemetry #region Functionality + /// + /// Records a visit to a lambda, as made it. + /// + public void Record(LambdaVisit visit) + => Record(visit.Lambda.Id, visit.Lambda.PublicKey, visit.Elapsed, visit.Status, visit.Bytes, visit.Path, visit.Domain); + /// /// Records a request a lambda answered. /// diff --git a/src/GenHTTP.Lambda/Services/Visits/LambdaVisit.cs b/src/GenHTTP.Lambda/Services/Visits/LambdaVisit.cs new file mode 100644 index 0000000..99aa2fb --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Visits/LambdaVisit.cs @@ -0,0 +1,106 @@ +using GenHTTP.Api.Protocol; + +using GenHTTP.Lambda.Services.Meta.Model; + +namespace GenHTTP.Lambda.Services.Visits; + +/// +/// One request a lambda answered, as everything that counts or logs requests +/// needs to know it. +/// +/// +/// Made once per request to a lambda or a preview of one, by +/// , and left on the request +/// () for whoever reads it: the figures of +/// the overview and the stats, the owner's analytics and the line the request +/// log writes. So the path is cut, the clock read and the status decided once, +/// rather than by each of them in its own way, and they never disagree about +/// what the visitor was given. A reader that wants something of a request to a +/// lambda takes it from here, and what is missing is added here. +/// +/// The visit says what happened; who is interested in which visits is the +/// reader's to decide - the figures and the analytics leave previews out, the +/// log keeps them. It holds what is cheap to have at hand and nothing worked +/// out from it: what only analytics make of a visit - the host of the +/// referrer, the kind of device - is worked out by them, off the request, when +/// they write it down. +/// +/// The lambda that answered, with the feature whose preview it was, if it was one +/// When the answer was given +/// What was asked for, relative to the lambda and cut to a length worth showing +/// The host it was reached at - its subdomain of the hosting domain, or its own domain +/// The status the visitor was given - a not found where nothing answered, a server error where the lambda threw +/// How long the answer took, waiting for a slot included +/// What the response carried, where that is known +/// Who asked, as the log names them - nothing where the installation does not keep addresses +/// The browser or program that asked, as it called itself +/// The country the address was allocated to, where it is known +/// +/// The page the visitor came from, as the browser said - read only for a lambda whose owner counts visits, since +/// nothing else asks for it +/// +/// +/// Whether this was somebody opening a page - a browser navigating to it, or a program asking for HTML - rather than +/// the scripts, pictures and calls that page goes on to load. Only worked out for a lambda whose owner counts visits. +/// +/// +/// What a Chromium browser says about itself apart from its agent, which it no longer keeps accurate - read only +/// for a lambda whose owner counts visits +/// +public sealed record LambdaVisit( + ResolvedLambda Lambda, + DateTime At, + string Path, + string? Domain, + int Status, + TimeSpan Elapsed, + long Bytes, + string? Client, + string? Agent, + string? Country, + string? Referrer, + bool Page, + ClientHints? Hints = null +) +{ + + /// + /// Whether this was somebody trying a feature at its preview address + /// rather than visiting the lambda. + /// + public bool Preview => Lambda.Feature != null; + +} + +/// +/// The client hints a Chromium browser sends with every request by default: +/// its brands, whether it is on a phone, and its operating system. +/// +/// +/// Chromium froze most of its User-Agent - every Android phone is "Android 10; +/// K" there - and says the rest here instead, so where they are sent they are +/// the better answer. Kept as sent; like the agent, they are written by the +/// sender and read with that in mind. +/// +/// Sec-CH-UA, such as "Google Chrome";v="141", "Not?A_Brand";v="8", "Chromium";v="141" +/// Sec-CH-UA-Mobile, ?1 on a phone +/// Sec-CH-UA-Platform, such as "Android" +public sealed record ClientHints(string? Brands, string? Mobile, string? Platform); + +/// +/// Carries the visit a request was to whatever reads it. +/// +public static class LambdaVisits +{ + private const string Key = "__LAMBDA_VISIT"; + + public static void SetVisit(this IRequest request, LambdaVisit visit) => request.Properties[Key] = visit; + + /// + /// The visit the request was, once a lambda or a preview answered it, or + /// nothing before that and for a request that reached no lambda. + /// + public static LambdaVisit? GetVisit(this IRequest request) + => request.Properties.TryGet(Key, out var visit) ? visit : null; + +} diff --git a/src/GenHTTP.Lambda/Services/Visits/LambdaVisitConcern.cs b/src/GenHTTP.Lambda/Services/Visits/LambdaVisitConcern.cs new file mode 100644 index 0000000..41c0691 --- /dev/null +++ b/src/GenHTTP.Lambda/Services/Visits/LambdaVisitConcern.cs @@ -0,0 +1,150 @@ +using System.Diagnostics; + +using GenHTTP.Api.Content; +using GenHTTP.Api.Protocol; + +using GenHTTP.Lambda.Services.Diagnostics; +using GenHTTP.Lambda.Services.Hosting; +using GenHTTP.Lambda.Services.Protection; + +using GenHTTP.Modules.DependencyInjection; + +namespace GenHTTP.Lambda.Services.Visits; + +/// +/// Makes the visit a request to a lambda was, and leaves it on the request for +/// whatever reads it. +/// +/// +/// It only makes the visit: the concerns outside it read it as the answer +/// passes them on the way back - the figures (LambdaActivityConcern), the +/// owner's analytics (AnalyticsConcern) - and so does the request log at +/// the outside of the host (CallerConcern). None of them measures the +/// request again. +/// +/// Outside the throttle and the error handler, so the visit is the answer the +/// visitor was given - a timeout is a timeout here and not the success the +/// lambda eventually got round to - and its time is the whole wait rather than +/// only the part spent running. Inside the lookup, so it knows which lambda it +/// was. +/// +public sealed class LambdaVisitConcern : IDependentConcern +{ + + public async ValueTask HandleAsync(IHandler content, IRequest request) + { + var lambda = request.GetLambda(); + + if (lambda == null) + { + return await content.HandleAsync(request); + } + + // read before the lambda routes it: the target is a pointer that the + // handlers below move along, and afterwards it points at nothing + var path = PathOf(request); + + var domain = request.GetHost()?.Name; + + /* + * What only the owner's analytics read, and only where they count + * visits - never for a preview. Read before the lambda is asked, + * because the headers of a request are let go of once its body was read. + */ + var counted = lambda.AnalyticsSince != null && lambda.Feature == null; + + var headers = request.Header.Headers; + + var referrer = counted ? headers.GetEntry("Referer") : null; + + var destination = counted ? headers.GetEntry("Sec-Fetch-Dest") : null; + + var hints = counted ? HintsOf(headers) : null; + + var started = Stopwatch.GetTimestamp(); + + IResponse? response = null; + + // what the visitor is told when the lambda throws: the error handler + // above makes a response of it, but as far as the lambda is concerned + // it failed + var status = 500; + + try + { + response = await content.HandleAsync(request); + + // nothing answered is a not found by the time the client sees it + status = response != null ? (int)response.Status : 404; + + return response; + } + finally + { + // the caller is made once for the whole request, its country and + // its agent included, before anything below was asked + var caller = Caller.Ambient; + + request.SetVisit(new LambdaVisit(lambda, DateTime.UtcNow, path, domain, status, Stopwatch.GetElapsedTime(started), + (long)(response?.Content?.Length ?? 0), caller?.Client, caller?.Agent, caller?.Country, + referrer, counted && IsPage(request, status, destination, response), hints)); + } + } + + /// + /// What was asked for, relative to the lambda and cut to a length worth + /// showing - the lookup above has already stepped past the key, where + /// there was one. + /// + private static string PathOf(IRequest request) + { + var remaining = request.Header.Target.AsString(decode: false, remainingOnly: true); + + var path = remaining.Length == 0 ? "/" : remaining[0] == '/' ? remaining : "/" + remaining; + + return path.Length <= 100 ? path : string.Concat(path.AsSpan(0, 99), "…"); + } + + /// + /// The client hints of a Chromium browser, or nothing from anything else. + /// + private static ClientHints? HintsOf(IKeyValueList headers) + { + var brands = headers.GetEntry("Sec-CH-UA"); + var mobile = headers.GetEntry("Sec-CH-UA-Mobile"); + var platform = headers.GetEntry("Sec-CH-UA-Platform"); + + return brands == null && mobile == null && platform == null ? null : new ClientHints(Cut(brands), Cut(mobile), Cut(platform)); + } + + private static string? Cut(string? value) => value == null || value.Length <= 200 ? value : value[..200]; + + /// + /// Whether somebody opened a page, rather than the page loading what it + /// is made of. + /// + /// + /// A browser says so itself: it marks a navigation with + /// Sec-Fetch-Dest: document, and only that - so a page revalidated + /// from its cache (Not Modified) is a page opened again, and a script or + /// a call to an API is not one, whatever it answers with. Programs that + /// do not send the header - crawlers, mostly - opened a page when what + /// they got back was HTML. A redirect is not a page: the page it sends to + /// is counted when it is opened. + /// + private static bool IsPage(IRequest request, int status, string? destination, IResponse? response) + { + if (request.Header.Method != RequestMethod.Get || status is not (>= 200 and < 300 or 304)) + { + return false; + } + + if (destination != null) + { + return string.Equals(destination, "document", StringComparison.OrdinalIgnoreCase); + } + + return status != 304 && response?.Content?.Type is { } type && type.Bytes.Span.StartsWith("text/html"u8); + } + +} diff --git a/tests/GenHTTP.Lambda.Tests/Analytics/AnalyticsTests.cs b/tests/GenHTTP.Lambda.Tests/Analytics/AnalyticsTests.cs new file mode 100644 index 0000000..c06b3f3 --- /dev/null +++ b/tests/GenHTTP.Lambda.Tests/Analytics/AnalyticsTests.cs @@ -0,0 +1,494 @@ +using System.Net; +using System.Text.Json.Nodes; + +using GenHTTP.Lambda.Api.Model; +using GenHTTP.Lambda.Data.Entities; +using GenHTTP.Lambda.Services.Analytics; +using GenHTTP.Lambda.Services.Diagnostics; +using GenHTTP.Lambda.Tests.Infrastructure; + +using GenHTTP.Testing; + +using Microsoft.Extensions.DependencyInjection; + +namespace GenHTTP.Lambda.Tests.Analytics; + +/// +/// What an owner who switched analytics on is shown of their visitors. +/// +[TestClass] +public sealed class AnalyticsTests +{ + + /// + /// A page that answers at every path, as a small website does. + /// + private const string Website = """ + return Content.From(Resource.FromString("Hello").Type(new ContentType("text/html; charset=utf-8"))); + """; + + private const string Chrome = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36"; + + private const string IPhone = "Mozilla/5.0 (iPhone; CPU iPhone OS 18_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.6 Mobile/15E148 Safari/604.1"; + + private const string Googlebot = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"; + + #region Switching + + [TestMethod] + public async Task NothingIsCountedUntilTheOwnerSwitchesAnalyticsOn() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + var lambda = await OnlineAsync(fixture, "quiet"); + + await VisitAsync(fixture, "http://quiet.localhost/"); + + Flush(fixture); + + var before = await ReadAsync(fixture, lambda.PrivateKey); + + Assert.IsFalse(before.Enabled); + Assert.IsNull(before.Totals, "there is nothing to show while it is off"); + Assert.AreEqual(7, before.Window, "and what a free lambda would show is said"); + + await EnableAsync(fixture, lambda.PrivateKey); + + await VisitAsync(fixture, "http://quiet.localhost/"); + + Flush(fixture); + + var after = await ReadAsync(fixture, lambda.PrivateKey); + + Assert.IsTrue(after.Enabled); + Assert.AreEqual(1, after.Totals!.Hits, "the visit before it was switched on is not counted"); + } + + [TestMethod] + public async Task SwitchingAnalyticsOffDeletesWhatWasCounted() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + var lambda = await OnlineAsync(fixture, "forgetful"); + + await EnableAsync(fixture, lambda.PrivateKey); + + await VisitAsync(fixture, "http://forgetful.localhost/"); + + Flush(fixture); + + using (var off = await fixture.SendAsync(HttpMethod.Delete, $"/api/v1/lambdas/{lambda.PrivateKey}/analytics")) + { + Assert.AreEqual(HttpStatusCode.NoContent, off.StatusCode); + } + + // queued while it was on, written after it was switched off + await VisitAsync(fixture, "http://forgetful.localhost/"); + + Flush(fixture); + + await EnableAsync(fixture, lambda.PrivateKey); + + var again = await ReadAsync(fixture, lambda.PrivateKey); + + Assert.AreEqual(0, again.Totals!.Hits, "switched on again, it starts from nothing"); + Assert.IsNull(again.Earliest); + } + + [TestMethod] + public async Task ADemosVisitorsAreNotCounted() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + var lambda = await OnlineAsync(fixture, "showpiece"); + + await fixture.MakeDemoAsync("showpiece"); + + using var refused = await fixture.SendAsync(HttpMethod.Put, $"/api/v1/lambdas/{lambda.PrivateKey}/analytics"); + + Assert.AreEqual(HttpStatusCode.Forbidden, refused.StatusCode); + } + + #endregion + + #region Counting + + [TestMethod] + public async Task AVisitIsCountedByPageReferrerCountryAndDevice() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + Place(fixture, "CH"); + + var lambda = await OnlineAsync(fixture, "shop"); + + await EnableAsync(fixture, lambda.PrivateKey); + + await VisitAsync(fixture, "http://shop.localhost/", Chrome, referrer: "https://www.google.com/", destination: "document"); + await VisitAsync(fixture, "http://shop.localhost/menu", IPhone, referrer: "https://news.example.org/today?id=4", destination: "document"); + await VisitAsync(fixture, "http://shop.localhost/menu", Googlebot); + + Flush(fixture); + + var shown = await ReadAsync(fixture, lambda.PrivateKey); + + Assert.AreEqual(3, shown.Totals!.Hits); + Assert.AreEqual(3, shown.Totals.Views, "a browser navigating and a crawler fetching HTML both opened a page"); + + Assert.AreEqual(2, Views(shown.Pages, "/menu")); + Assert.AreEqual(1, Views(shown.Pages, "/")); + + Assert.AreEqual(1, Views(shown.Referrers, "google.com"), "the host, without its www"); + Assert.AreEqual(1, Views(shown.Referrers, "news.example.org"), "never the rest of the address"); + Assert.HasCount(2, shown.Referrers!); + + Assert.AreEqual(3, Views(shown.Countries, "CH")); + + Assert.AreEqual(1, Views(shown.Devices, "desktop")); + Assert.AreEqual(1, Views(shown.Devices, "mobile")); + Assert.AreEqual(1, Views(shown.Devices, "bot")); + + Assert.AreEqual(1, Views(shown.Browsers, "Chrome")); + Assert.AreEqual(1, Views(shown.Browsers, "Safari")); + Assert.AreEqual(1, Views(shown.Systems, "iOS")); + Assert.IsNull(shown.Browsers!.FirstOrDefault(b => b.Value == "bot"), "a bot is no browser of a person"); + + Assert.AreEqual(1, Views(shown.Crawlers, "unconfirmed"), "Google publishes its addresses, and none are known here"); + + Assert.AreEqual(2, shown.Totals.Visitors, "the crawler opened a page, but is nobody"); + + var today = shown.Daily!.Last(); + + Assert.AreEqual(DateOnly.FromDateTime(DateTime.UtcNow), today.Date); + Assert.AreEqual(3, today.Totals.Views); + } + + [TestMethod] + public async Task ACrawlerIsCountedAsGooglesWhereItCameFromGoogle() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + var lambda = await OnlineAsync(fixture, "indexed"); + + await EnableAsync(fixture, lambda.PrivateKey); + + // the test host's own address, both families, as Google's + fixture.Application.Services.GetRequiredService().Load("Google", [ + new MemoryStream("""{"prefixes": [{"ipv4Prefix": "127.0.0.0/8"}, {"ipv6Prefix": "::1/128"}]}"""u8.ToArray()) + ]); + + await VisitAsync(fixture, "http://indexed.localhost/", Googlebot); + + Flush(fixture); + + var shown = await ReadAsync(fixture, lambda.PrivateKey); + + Assert.AreEqual(1, Views(shown.Crawlers, "Google")); + Assert.Contains("Google", shown.ConfirmedCrawlers, "the page marks whose crawlers are confirmed"); + Assert.AreEqual(0, shown.Totals!.Visitors); + } + + [TestMethod] + public async Task OnlyOpeningAPageIsAView() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + var lambda = await OnlineAsync(fixture, "loader"); + + await EnableAsync(fixture, lambda.PrivateKey); + + await VisitAsync(fixture, "http://loader.localhost/", Chrome, destination: "document"); + + // what that page goes on to load, with the page as its referrer + await VisitAsync(fixture, "http://loader.localhost/app.js", Chrome, referrer: "http://loader.localhost/", destination: "script"); + await VisitAsync(fixture, "http://loader.localhost/api/items", Chrome, referrer: "http://loader.localhost/", destination: "empty"); + + Flush(fixture); + + var shown = await ReadAsync(fixture, lambda.PrivateKey); + + Assert.AreEqual(3, shown.Totals!.Hits); + Assert.AreEqual(1, shown.Totals.Views); + Assert.AreEqual(1, shown.Totals.Visitors, "one visitor, however much their page loaded"); + + Assert.AreEqual(1, shown.Pages!.Single(p => p.Value == "/app.js").Hits, "what was loaded is requested, not viewed"); + Assert.AreEqual(0, Views(shown.Pages, "/app.js")); + Assert.IsEmpty(shown.Referrers!, "the lambda referring to itself is nobody coming from elsewhere"); + } + + [TestMethod] + public async Task AScannerCannotFillADayWithPaths() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + var lambda = await OnlineAsync(fixture, "scanned"); + + await EnableAsync(fixture, lambda.PrivateKey); + + var most = AnalyticsDimensions.MostOf(AnalyticsDimension.Pages)!.Value; + + for (var i = 0; i < most + 10; i++) + { + await VisitAsync(fixture, $"http://scanned.localhost/probe-{i}"); + + // over more than one writing, as a day is + if (i == most / 2) + { + Flush(fixture); + } + } + + Flush(fixture); + + using var response = await fixture.GetAsync($"/api/v1/lambdas/{lambda.PrivateKey}/analytics"); + + var store = fixture.Application.Services.GetRequiredService(); + + var id = fixture.Meta.GetId(lambda.PrivateKey)!.Value; + + var today = DateOnly.FromDateTime(DateTime.UtcNow); + + var pages = store.Read(id, today, today, int.MaxValue).Breakdowns[AnalyticsDimension.Pages]; + + Assert.HasCount(most + 1, pages, "as many paths as a day has room for, and the rest together"); + Assert.AreEqual(10, pages.Single(p => p.Value == AnalyticsDimensions.Other).Hits); + } + + [TestMethod] + public async Task APreviewOfAFeatureIsNotCounted() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + var lambda = await OnlineAsync(fixture, "drafty"); + + await EnableAsync(fixture, lambda.PrivateKey); + + using (var created = await fixture.SendAsync(HttpMethod.Post, $"/api/v1/lambdas/{lambda.PrivateKey}/features", new CreateFeatureRequest("Try", null, null))) + { + var feature = await created.GetContentAsync(); + + using (await fixture.SendAsync(HttpMethod.Post, $"/api/v1/lambdas/{lambda.PrivateKey}/features/{feature.Key}/preview/start")) { } + + using var preview = await fixture.GetAsync(feature.PreviewPath); + + Assert.AreEqual(HttpStatusCode.OK, preview.StatusCode); + } + + Flush(fixture); + + Assert.AreEqual(0, (await ReadAsync(fixture, lambda.PrivateKey)).Totals!.Hits, "somebody trying a change is not a visitor"); + } + + #endregion + + #region What is shown and kept + + [TestMethod] + public async Task AFreeLambdaShowsAWeekAndKeepsTheYearForPremium() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + var lambda = await OnlineAsync(fixture, "growing"); + + await EnableAsync(fixture, lambda.PrivateKey); + + await VisitAsync(fixture, "http://growing.localhost/"); + + Flush(fixture); + + var analytics = fixture.Application.Services.GetRequiredService(); + + var later = DateTime.UtcNow.AddDays(30); + + var free = analytics.Get(lambda.PrivateKey, 365, later); + + Assert.AreEqual(7, free.Shown, "asked for a year, a free lambda shows a week"); + Assert.AreEqual(0, free.Report!.Totals.Hits, "and the visit is older than that"); + Assert.AreEqual(DateOnly.FromDateTime(DateTime.UtcNow), free.Earliest, "but it is kept, and said to be"); + Assert.IsNull(free.Previous, "nor is the week before compared, which a free lambda does not show"); + + fixture.ChangeTier(lambda.PrivateKey, LambdaTier.Premium); + + var premium = analytics.Get(lambda.PrivateKey, 60, later); + + Assert.AreEqual(60, premium.Shown); + Assert.AreEqual(1, premium.Report!.Totals.Hits, "premium shows what was counted before it was premium"); + Assert.IsNotNull(premium.Previous); + } + + [TestMethod] + public async Task AYearIsKeptAndNoMore() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + var lambda = await OnlineAsync(fixture, "aging"); + + fixture.ChangeTier(lambda.PrivateKey, LambdaTier.Premium); + + await EnableAsync(fixture, lambda.PrivateKey); + + await VisitAsync(fixture, "http://aging.localhost/"); + + var analytics = fixture.Application.Services.GetRequiredService(); + + analytics.Flush(); + + Assert.IsNotNull(analytics.Get(lambda.PrivateKey, now: DateTime.UtcNow.AddDays(AnalyticsStore.Kept - 1)).Earliest, "a year later it is still there"); + + analytics.Flush(DateTime.UtcNow.AddDays(AnalyticsStore.Kept + 1)); + + Assert.IsNull(analytics.Get(lambda.PrivateKey).Earliest, "a day past the year it is gone"); + } + + [TestMethod] + public async Task DeletingALambdaDeletesItsAnalytics() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + var lambda = await OnlineAsync(fixture, "leaving"); + + await EnableAsync(fixture, lambda.PrivateKey); + + await VisitAsync(fixture, "http://leaving.localhost/"); + + Flush(fixture); + + var id = fixture.Meta.GetId(lambda.PrivateKey)!.Value; + + var store = fixture.Application.Services.GetRequiredService(); + + Assert.IsNotNull(store.Earliest(id)); + + fixture.Meta.Delete(lambda.PrivateKey); + + Assert.IsNull(store.Earliest(id)); + } + + #endregion + + #region Agents + + [TestMethod] + public async Task AnAgentReadsWhatTheOwnerIsShown() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + var lambda = await OnlineAsync(fixture, "asked"); + + var off = await CallAsync(fixture, new JsonObject { ["privateKey"] = lambda.PrivateKey }); + + Assert.IsFalse(off["enabled"]!.GetValue()); + + var on = await CallAsync(fixture, new JsonObject { ["privateKey"] = lambda.PrivateKey, ["enable"] = true }); + + Assert.IsTrue(on["analytics"]!["enabled"]!.GetValue(), "switched on when the user asked for it"); + + await VisitAsync(fixture, "http://asked.localhost/", Chrome, destination: "document"); + + Flush(fixture); + + var read = await CallAsync(fixture, new JsonObject { ["privateKey"] = lambda.PrivateKey, ["days"] = 3 }); + + Assert.AreEqual(3, read["analytics"]!["days"]!.GetValue()); + Assert.AreEqual(1, read["analytics"]!["totals"]!["views"]!.GetValue()); + } + + #endregion + + #region Helpers + + private static async Task OnlineAsync(LambdaFixture fixture, string key) + { + var lambda = await fixture.CreateLambdaAsync(key); + + await fixture.DeployAsync(lambda.PrivateKey, Website); + + return lambda; + } + + private static async Task EnableAsync(LambdaFixture fixture, string privateKey) + { + using var on = await fixture.SendAsync(HttpMethod.Put, $"/api/v1/lambdas/{privateKey}/analytics"); + + Assert.AreEqual(HttpStatusCode.OK, on.StatusCode, await on.Content.ReadAsStringAsync()); + } + + private static async Task ReadAsync(LambdaFixture fixture, string privateKey) + { + using var read = await fixture.GetAsync($"/api/v1/lambdas/{privateKey}/analytics"); + + Assert.AreEqual(HttpStatusCode.OK, read.StatusCode, await read.Content.ReadAsStringAsync()); + + return await read.GetContentAsync(); + } + + private static async Task VisitAsync(LambdaFixture fixture, string address, string? agent = null, string? referrer = null, string? destination = null) + { + using var request = fixture.Request(address); + + if (agent != null) + { + request.Headers.TryAddWithoutValidation("User-Agent", agent); + } + + if (referrer != null) + { + request.Headers.TryAddWithoutValidation("Referer", referrer); + } + + if (destination != null) + { + request.Headers.TryAddWithoutValidation("Sec-Fetch-Dest", destination); + } + + using var response = await fixture.Host.GetResponseAsync(request); + + Assert.AreEqual(HttpStatusCode.OK, response.StatusCode); + } + + /// + /// What the job does every few minutes. + /// + private static void Flush(LambdaFixture fixture) => fixture.Application.Services.GetRequiredService().Flush(); + + /// + /// Has every address of the test host - loopback, both families - + /// allocated to a country. + /// + private static void Place(LambdaFixture fixture, string country) + => fixture.Application.Services.GetRequiredService().Load([ + $"test|{country}|ipv4|127.0.0.0|16777216|20000101|allocated", + $"test|{country}|ipv6|::1|128|20000101|allocated" + ]); + + private static long Views(IReadOnlyList? values, string value) => values!.SingleOrDefault(v => v.Value == value)?.Views ?? 0; + + private static async Task CallAsync(LambdaFixture fixture, JsonObject arguments) + { + var call = new JsonObject + { + ["jsonrpc"] = "2.0", + ["id"] = 1, + ["method"] = "tools/call", + ["params"] = new JsonObject { ["name"] = "analytics", ["arguments"] = arguments } + }; + + using var request = fixture.Host.GetRequest("/mcp", HttpMethod.Post); + + request.Content = new StringContent(call.ToJsonString(), System.Text.Encoding.UTF8, "application/json"); + request.Headers.Add("Accept", "application/json, text/event-stream"); + + using var response = await fixture.Host.GetResponseAsync(request); + + var body = JsonNode.Parse(await response.Content.ReadAsStringAsync())!; + + var result = body["result"]!; + + Assert.IsFalse(result["isError"]?.GetValue() ?? false, result.ToJsonString()); + + return result["structuredContent"]!.AsObject(); + } + + #endregion + +} diff --git a/tests/GenHTTP.Lambda.Tests/Analytics/VisitorTests.cs b/tests/GenHTTP.Lambda.Tests/Analytics/VisitorTests.cs new file mode 100644 index 0000000..d36fd87 --- /dev/null +++ b/tests/GenHTTP.Lambda.Tests/Analytics/VisitorTests.cs @@ -0,0 +1,269 @@ +using System.Diagnostics; +using System.Text; +using System.Text.Json; + +using GenHTTP.Lambda.Services.Analytics; +using GenHTTP.Lambda.Services.Visits; +using GenHTTP.Lambda.Tests.Infrastructure; + +using Microsoft.Extensions.DependencyInjection; + +namespace GenHTTP.Lambda.Tests.Analytics; + +/// +/// Whether the analytics take a visitor for a person or a program, and what +/// they say a person came with. +/// +[TestClass] +public sealed class VisitorTests +{ + + #region Programs + + [TestMethod] + public async Task EveryCrawlerTheListKnowsIsTakenForABot() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + var classifier = Classifier(fixture); + + using var stream = typeof(Application).Assembly.GetManifestResourceStream("GenHTTP.Lambda.Resources.Crawlers.crawler-user-agents.json")!; + + using var list = JsonDocument.Parse(stream); + + var instances = list.RootElement.EnumerateArray() + .Where(e => e.TryGetProperty("instances", out _)) + .SelectMany(e => e.GetProperty("instances").EnumerateArray().Select(i => i.GetString()!)) + .ToList(); + + Assert.IsGreaterThan(1000, instances.Count, "the list comes with the agents its crawlers really send"); + + var missed = instances.Where(i => classifier.Classify(i).Device != VisitorClassifier.Bot).ToList(); + + Assert.IsEmpty(missed, "taken for people:\n" + string.Join("\n", missed.Take(20))); + } + + [TestMethod] + [DataRow("okhttp/4.12.0")] + [DataRow("curl/8.5.0")] + [DataRow("python-requests/2.32.3")] + [DataRow("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/141.0.0.0 Safari/537.36")] + [DataRow("Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)")] + [DataRow("meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)")] + [DataRow("Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)")] + public async Task AProgramIsNoPerson(string agent) + { + await using var fixture = await LambdaFixture.CreateAsync(); + + var read = Classifier(fixture).Classify(agent); + + Assert.AreEqual(VisitorClassifier.Bot, read.Device); + Assert.IsNull(read.Browser, "a program has no browser to count"); + Assert.IsNotNull(read.Crawler); + } + + [TestMethod] + [DataRow(null)] + [DataRow("")] + [DataRow("MyShop/3.2 CFNetwork/1568.100.1 Darwin/24.0.0")] + public async Task WhatDoesNotNameABrowserIsNeither(string? agent) + { + await using var fixture = await LambdaFixture.CreateAsync(); + + var read = Classifier(fixture).Classify(agent); + + Assert.AreEqual(VisitorClassifier.Unknown, read.Device); + Assert.IsNull(read.Browser); + Assert.IsNull(read.Crawler, "nothing says it is a crawler either"); + } + + #endregion + + #region People + + [TestMethod] + [DataRow("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36", "desktop", "Chrome", "Windows")] + [DataRow("Mozilla/5.0 (iPhone; CPU iPhone OS 18_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.6 Mobile/15E148 Safari/604.1", "mobile", "Safari", "iOS")] + [DataRow("Mozilla/5.0 (iPad; CPU OS 18_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.6 Mobile/15E148 Safari/604.1", "tablet", "Safari", "iOS")] + [DataRow("Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Mobile Safari/537.36", "mobile", "Chrome", "Android")] + [DataRow("Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36", "tablet", "Chrome", "Android")] + [DataRow("Mozilla/5.0 (Linux; Android 13; Cubot X30) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Mobile Safari/537.36", "mobile", "Chrome", "Android")] + [DataRow("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.6 Safari/605.1.15", "desktop", "Safari", "macOS")] + [DataRow("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:143.0) Gecko/20100101 Firefox/143.0", "desktop", "Firefox", "Windows")] + [DataRow("Mozilla/5.0 (iPhone; CPU iPhone OS 18_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) FxiOS/143.0 Mobile/15E148 Safari/605.1.15", "mobile", "Firefox", "iOS")] + [DataRow("Mozilla/5.0 (iPhone; CPU iPhone OS 18_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/141.0.7390.41 Mobile/15E148 Safari/604.1", "mobile", "Chrome", "iOS")] + [DataRow("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36 Edg/141.0.0.0", "desktop", "Edge", "Windows")] + [DataRow("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36 OPR/120.0.0.0", "desktop", "Opera", "Linux")] + [DataRow("Mozilla/5.0 (Linux; Android 14; SAMSUNG SM-S918B) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/28.0 Chrome/130.0.0.0 Mobile Safari/537.36", "mobile", "Samsung Internet", "Android")] + [DataRow("Mozilla/5.0 (X11; CrOS x86_64 14541.0.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36", "desktop", "Chrome", "ChromeOS")] + [DataRow("Mozilla/5.0 (iPhone; CPU iPhone OS 18_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/22G86 Instagram 389.0.0.0 (iPhone15,2; iOS 18_6; de_DE; de; scale=3.00; 1179x2556)", "mobile", "Instagram", "iOS")] + [DataRow("Mozilla/5.0 (Linux; Android 14; Pixel 8 Build/AP2A; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/141.0.0.0 Mobile Safari/537.36 [FB_IAB/FB4A;FBAV/480.0.0.0;]", "mobile", "Facebook", "Android")] + [DataRow("Mozilla/5.0 (iPhone; CPU iPhone OS 18_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148", "mobile", "other", "iOS")] + public async Task APersonIsReadByTheBrowserTheyCameWith(string agent, string device, string browser, string system) + { + await using var fixture = await LambdaFixture.CreateAsync(); + + var read = Classifier(fixture).Classify(agent); + + Assert.AreEqual(device, read.Device); + Assert.AreEqual(browser, read.Browser); + Assert.AreEqual(system, read.System); + Assert.IsNull(read.Crawler); + } + + [TestMethod] + public async Task TheHintsOfChromiumSayWhatItsAgentNoLongerDoes() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + var classifier = Classifier(fixture); + + // Chromium's agent of a desktop is the same on every system; Brave + // does not name itself there at all + const string agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36"; + + var brave = classifier.Classify(agent, new ClientHints("\"Brave\";v=\"141\", \"Not?A_Brand\";v=\"8\", \"Chromium\";v=\"141\"", "?0", "\"macOS\"")); + + Assert.AreEqual("Brave", brave.Browser); + Assert.AreEqual("macOS", brave.System); + Assert.AreEqual(VisitorClassifier.Desktop, brave.Device); + + // a phone that asked for the desktop site still says it is a phone + var phone = classifier.Classify(agent, new ClientHints("\"Google Chrome\";v=\"141\", \"Chromium\";v=\"141\"", "?1", "\"Android\"")); + + Assert.AreEqual(VisitorClassifier.Mobile, phone.Device); + Assert.AreEqual("Android", phone.System); + Assert.AreEqual("Chrome", phone.Browser); + } + + [TestMethod] + public async Task HintsAreTakenOnlyWhereTheyAreChromiumsAndSayWhatIsKnown() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + var classifier = Classifier(fixture); + + // a Firefox sending Chromium's hints is a Firefox + var firefox = classifier.Classify("Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:143.0) Gecko/20100101 Firefox/143.0", + new ClientHints("\"Google Chrome\";v=\"141\"", "?1", "\"Android\"")); + + Assert.AreEqual("Firefox", firefox.Browser); + Assert.AreEqual("Windows", firefox.System); + Assert.AreEqual(VisitorClassifier.Desktop, firefox.Device); + + // a brand or a system nobody knows is not written into anybody's + // figures: the agent says what it is instead + var made = classifier.Classify("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36", + new ClientHints("\"Buy now\";v=\"1\", \"Not_A Brand\";v=\"8\", \"Chromium\";v=\"141\"", "?0", "\"TempleOS\"")); + + Assert.AreEqual("Chrome", made.Browser); + Assert.AreEqual("Windows", made.System); + } + + #endregion + + #region Claims + + [TestMethod] + public async Task ACrawlerIsGooglesOnlyFromAnAddressGooglePublishes() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + var classifier = Classifier(fixture); + + const string googlebot = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"; + + Assert.AreEqual(CrawlerRanges.Unconfirmed, classifier.Classify(googlebot, client: "66.249.66.1").Crawler, + "nothing is confirmed before the addresses are known"); + + Ranges(fixture).Load("Google", [Prefixes("66.249.64.0/19", "2001:4860:4801:10::/64")]); + + Assert.AreEqual("Google", classifier.Classify(googlebot, client: "66.249.66.1").Crawler); + Assert.AreEqual("Google", classifier.Classify(googlebot, client: "2001:4860:4801:10::1").Crawler); + + var impostor = classifier.Classify(googlebot, client: "203.0.113.9"); + + Assert.AreEqual(CrawlerRanges.Unconfirmed, impostor.Crawler, "a scraper calling itself Googlebot"); + Assert.AreEqual(VisitorClassifier.Bot, impostor.Device, "but still no person"); + + Assert.AreEqual(CrawlerRanges.Unconfirmed, classifier.Classify(googlebot, client: "66.249.66.1 via 203.0.113.9").Crawler, + "a forwarding header is written by the sender, and claims Google's address as easily as Google's name"); + + Assert.AreEqual(CrawlerRanges.Unconfirmed, classifier.Classify(googlebot).Crawler, "nor without an address"); + + Assert.Contains("Google", Ranges(fixture).Confirmed); + } + + [TestMethod] + public async Task AListThatDoesNotReadLeavesTheOneBefore() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + var list = fixture.Application.Services.GetRequiredService(); + + var known = list.Count; + + Assert.IsFalse(list.Load(new MemoryStream("Service unavailable"u8.ToArray()))); + Assert.IsFalse(list.Load(new MemoryStream("[]"u8.ToArray()))); + Assert.IsFalse(Ranges(fixture).Load("Bing", [new MemoryStream("{\"oops\": 1}"u8.ToArray())])); + + Assert.AreEqual(known, list.Count); + + // and a newer one is taken, what was read before let go of + var classifier = Classifier(fixture); + + Assert.AreEqual(VisitorClassifier.Unknown, classifier.Classify("FreshBot/1.0").Device); + + Assert.IsTrue(list.Load(new MemoryStream("[{\"pattern\": \"FreshBot\", \"tags\": [\"seo\"]}]"u8.ToArray()))); + + Assert.AreEqual("seo", classifier.Classify("FreshBot/1.0").Crawler); + } + + #endregion + + #region Abuse + + [TestMethod] + public async Task AnAgentMadeToBeSlowIsNot() + { + await using var fixture = await LambdaFixture.CreateAsync(); + + var classifier = Classifier(fixture); + + // warm, so what is measured is the reading and not the first build of + // the expressions + classifier.Classify("Mozilla/5.0 (Windows NT 10.0) Chrome/141.0.0.0"); + + var hostile = "Mozilla/5.0 (" + string.Concat(Enumerable.Repeat("compatible; a", 20_000)) + ")"; + + var clock = Stopwatch.StartNew(); + + for (var i = 0; i < 50; i++) + { + classifier.Classify(hostile + i); + } + + Assert.IsLessThan(5000, clock.ElapsedMilliseconds, "matching takes time in proportion to what is read, and only so much is read"); + } + + #endregion + + #region Helpers + + private static VisitorClassifier Classifier(LambdaFixture fixture) => fixture.Application.Services.GetRequiredService(); + + private static CrawlerRanges Ranges(LambdaFixture fixture) => fixture.Application.Services.GetRequiredService(); + + /// + /// A list of addresses in the shape the operators publish them. + /// + private static MemoryStream Prefixes(params string[] prefixes) + { + var entries = prefixes.Select(p => p.Contains(':') ? $"{{\"ipv6Prefix\": \"{p}\"}}" : $"{{\"ipv4Prefix\": \"{p}\"}}"); + + return new MemoryStream(Encoding.UTF8.GetBytes($"{{\"creationTime\": \"2026-10-12T00:00:00\", \"prefixes\": [{string.Join(',', entries)}]}}")); + } + + #endregion + +} diff --git a/tests/GenHTTP.Lambda.Tests/Api/LimitsTests.cs b/tests/GenHTTP.Lambda.Tests/Api/LimitsTests.cs index 17b5c15..16bbd24 100644 --- a/tests/GenHTTP.Lambda.Tests/Api/LimitsTests.cs +++ b/tests/GenHTTP.Lambda.Tests/Api/LimitsTests.cs @@ -170,6 +170,25 @@ public async Task ThePremiumTierMayNotAllowLessThanTheFreeOne() Assert.AreEqual(current, await ReadAsync(fixture), "and nothing of it was saved"); } + [TestMethod] + public async Task AnalyticsAreShownForAWeekOrAYearAndNeverLongerThanTheyAreKept() + { + await using var fixture = await LambdaFixture.CreateAsync(LambdaFixture.WithPanel); + + var current = await ReadAsync(fixture); + + Assert.AreEqual(7, current.Free.AnalyticsDays); + Assert.AreEqual(365, current.Premium.AnalyticsDays); + + using var refused = await SendAsync(fixture, HttpMethod.Put, current with { Premium = current.Premium with { AnalyticsDays = 400 } }); + + Assert.AreEqual(HttpStatusCode.BadRequest, refused.StatusCode, "a year is all that is kept"); + + await ChangeAsync(fixture, current with { Free = current.Free with { AnalyticsDays = 30 } }); + + Assert.AreEqual(30, (await ReadAsync(fixture)).Free.AnalyticsDays); + } + [TestMethod] public async Task ALimitOfZeroIsRefused() {