Watchtower

Two halves of one thing: a scanner that scores a repository against the Agentic Compatibility Benchmark — how safely an AI agent can work in a codebase — and a dashboard that shows the results for a fleet of them.

• The scanner is this repository's root, and a GitHub composite action: uses: mindvalley/watchtower@v1. Everything below describes it.
• The dashboard is dashboard/, a Node web server you can run on one machine with Docker, or behind whatever hosts things for you. It has its own README.

You can use either without the other. The scanner writes results; the dashboard stores and shows them.

The scanner

Headless by design. The engine never compiles, installs, or runs the code it measures. It reads source, manifests, configuration and git history — on a shallow clone, or on a copy of a folder on your own machine. That is what lets it score a repository it knows nothing about.

What it measures

Seven criteria, each scored out of 5 and coloured red / amber / green:

#	Criterion	Measured by
1	Clear Domain Boundaries	tree-sitter code graph (cycles, fan-out) + change coupling from git history
2	Documented APIs	description coverage over OpenAPI, GraphQL SDL, and in-code API declarations
4	Observable State	logging, tracing and error-tracking, scored declared → configured → exercised
6	Test Coverage	test-to-source breadth + whether CI enforces a coverage floor
7	Deployment Safety	DORA capabilities visible in the repo: progressive delivery, rollback, pipeline safety, independent deployability
8	Codebase Simplicity	cyclomatic complexity density + whether a complexity gate is enforced, and duplication
9	Security Posture	secrets, dependency CVEs, static analysis — each triaged before it is scored

Numbering is not contiguous: two criteria from the original spec were removed.

Scoring principles

• Convention-anchored. A criterion is scored only against a published standard or tool default — McCabe ≤ 10, DORA, Google's coverage tiers. Never an arbitrary number. Where this benchmark chose a convention of its own, it says so.
• Never falsely green. A scanner that cannot run fails the scan; it does not report a clean subject. A measurement that cannot be trusted is withheld and labelled, not estimated.
• Language-agnostic first. Scoring never takes language as an input. Language enters only as a lookup for which files are manifests, source, or tests. Language-specific parsers raise fidelity; they are never a prerequisite.
• One number for the system. Criteria score out of 5; the system composite is their mean rescaled to 100, banded red 0–40, amber 41–70, green 71–100 (the former 2.0 and 3.5 boundaries, rescaled — no new anchor). A Critical finding multiplies the composite by 0.4, so the best a capped system can reach is exactly 40, still red. The colour is derived from that number and nothing else, so a score can never disagree with the badge beside it. Scaling rather than clamping keeps capped systems ordered against each other, which is what says which of them is closest to being fixable.

Configuration

Two ways to run it. Which one you are in decides what you install, what you need credentials for, and where the scores end up.

Local — everything on one machine

You install the scanners, point a config at a folder or a repo, and run the seven scans. Scores land as JSON. Optionally run the board beside them and load the files in. Nothing publishes and nothing needs a credential. This is the default, and it is the right one for a single codebase or for trying it out.

Step by step: guides/local-setup.md.

Remote — scans run themselves

The scanner runs in CI as the composite action, in each repository you want scored. It installs its own scanners at the pinned versions, so nothing is installed by hand. Scores are uploaded as build artifacts, and — if you declare "publish": "ingest" — posted to a dashboard somebody hosts.

  a repository you control
  ┌──────────────────────────────┐
  │ watchtower.config.json       │  which systems, and where each one lives
  │ .github/workflows/scan.yaml  │  uses: mindvalley/watchtower@v1
  └──────────────┬───────────────┘
                 │  the action installs the pinned scanners, scans every
                 │  system in the config, and assembles the scores
                 ▼
         ┌───────────────┐
         │ build artifact│  always
         └───────────────┘
                 │
                 │  and, only if the config says "publish": "ingest"
                 ▼
            POST /ingest ────▶ ┌─────────────────────────┐
                               │ a dashboard you host    │
                               │ web server + Postgres   │
                               └─────────────────────────┘
                                            │
                                            ▼
                                        one board

One dashboard receives from as many of these repositories as you like, across organisations. Each one scans only the systems its own config names.

The allowlist decides which repository may publish which systems, and the audience pins what a scanner's token must be minted for. Until both are set, /ingest is off and returns 503 — so a team can adopt the scanner before anyone has a board, and the scores wait in the artifacts.

Hosting the board: dashboard/README.md. The workflow to copy: Using it, below.

Using it

The engine ships as a composite action so that the code and the scanner versions it is calibrated against travel as one unit. A caller one version behind on any scanner produces numbers that look comparable and are not.

- uses: mindvalley/watchtower@v1
  id: engine
  with:
    config: config/systems.json

- run: node "${{ steps.engine.outputs.engine-path }}/scripts/benchmark/scan-security.js"
  env:
    SYSTEM: my-service
    GH_TOKEN: ${{ steps.token.outputs.token }}
    WATCHTOWER_CONFIG: config/systems.json
    WATCHTOWER_REPORTS: reports

Toolchains are installed from what you declare

Most of what the action installs is language-agnostic and runs on every scan. Two things are not:

Toolchain	Used for	Installed when
Elixir + OTP	Credo (complexity on Elixir source), and the AST helpers the C2 reader uses on Elixir systems	a system declares "stack": "elixir"
Ruby	Rubocop (complexity on Ruby source)	a system declares "stack": "ruby"

Point the config input at your configuration and the action reads it before installing anything. A fleet with no Elixir in it never builds the BEAM toolchain. If the configuration cannot be read, or a system declares no stack, everything is installed — an unreadable input must not quietly become a smaller scan.

There is one case the configuration cannot express. Complexity is measured in every language the repository materially contains, not the one it is declared as, so a repository declared typescript that also holds a few thousand lines of Ruby needs Rubocop. That scan fails and says so rather than scoring the Ruby clean; install-ruby: true (or install-elixir: true) forces the toolchain in without misdeclaring the stack.

Where the engine reads and writes

The caller says what to measure; the engine says how. Four locations come from the environment:

Variable	Holds	Default
WATCHTOWER_CONFIG	the list of systems to scan and where each one lives	watchtower.config.json
WATCHTOWER_REPORTS	where raw scanner output is written	reports
WATCHTOWER_DATA	where assembled scores are read from and written to	data
WATCHTOWER_ALLOWANCES	findings already judged acceptable	watchtower.allowances.json

A missing, malformed, or empty system list stops the run. A scan of nothing reports no findings, which reads exactly like a clean result.

Credentials in the config file are refused outright — it is the one file that gets committed.

What a system points at

Each system declares repo or path. Exactly one; both is refused rather than resolved, because it is two answers to one question.

{
  "systems": {
    "billing": { "repo": "org/billing",  "stack": "elixir" },  // clone from GitHub
    "web":     { "path": "code/web",     "stack": "ts" }  // a folder here
  }
}

A relative path is resolved against the directory holding the config, not the working directory, so the same config means the same thing wherever you run it from. GH_TOKEN is only needed for a repo.

Local targets are what make a scan usable before you push. Run it on the code in front of you rather than on whatever is currently on the remote's default branch.

A local folder is copied before it is read, never scanned in place. The scan writes into the tree it reads — graphify leaves a graphify-out/ directory — and nothing here is going to leave build output inside your working copy. What travels is what git carries: tracked files plus untracked ones no ignore rule covers, uncommitted edits included. node_modules, build directories and anything else ignored stay behind, which is what keeps a local run comparable to a CI one. Symlinks are skipped and counted rather than followed.

Change coupling needs a commit log, and the copy has no .git; it is read from the source folder instead, read-only. A folder that is not a git repository is still scannable — the copy falls back to a fixed list of build directories to skip, and change coupling reports itself unmeasured rather than clean.

The temp copy is always removed afterwards, and only ever the copy.

Where the results go

A configuration that declares "publish": "ingest" posts each system's scores to a dashboard's /ingest. One that declares nothing writes them out and posts nowhere — benchmark.json and a findings-<system>.json per system, under WATCHTOWER_DATA. That is the default, and it is the right one for a run on your own machine: a scanner should not need somewhere to publish before it will tell you what it found.

To see those results on a board, run the dashboard locally and load them:

cd dashboard && npm run load -- path/to/data --config path/to/watchtower.config.json

Allowances

See guides/allowances.md.

Development

npm install
npm test

The suite runs offline and needs nothing but Node and js-yaml. Two tests exercise Elixir AST helpers and are skipped when Elixir is absent.

Status

Used in production by Mindvalley across three organisations. Open-sourcing properly — setup guide, configurable triage filters — is intended but not yet done. The Elixir prerequisite is gone for anyone who does not declare the stack; the two AST helpers that need the runtime at all are still there, and replacing them with tree-sitter would remove it entirely.