Repository navigation
Conversation
…ne, results API, and Postgres persistence Ports the analysis pipeline from an in-progress working state that was previously uncommitted: real analyzer implementations (Strategy pattern), finding normalization/fingerprinting, secure workspace preparation (allowlisted hosts, validated commit/branch, no shell=True), a read-only results API, and the Postgres schema + repository backing it.
Removes the Pylint/Radon/Cppcheck analyzers and the Reviewdog aggregation layer entirely, and replaces them with a purpose-built ESLint (JS/TS) analyzer that parses ESLint's own --format json output directly. Adds a new metrics/ package that computes AnalysisMetrics (issue density, cyclomatic complexity, cognitive complexity via eslint-plugin-sonarjs, code size, unused code) plus per-rule and per-file statistics from real ESLint violation output and an independent workspace LOC scan — stored alongside findings on AnalysisResult (Postgres JSONB, migration-safe) so main-backend/web-interface can display them without recalculating anything themselves. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Extends the engine beyond JS/TS with a PythonAnalyzer composite that runs all three tools concurrently behind the existing Analyzer interface, so AnalyzerFactory and AnalysisOrchestrator treat Python exactly like ESLint and neither needed to change. Each tool's own exit-code convention is handled explicitly rather than assumed — Pylint's bitmask (only bit 32 means the tool itself failed), Bandit's exit 1 for "issues found", and Radon's always-0 with per-file errors embedded in its JSON. All three were verified against the pinned versions rather than taken from documentation. A missing binary, a timeout, or malformed output is reported as that tool's own AnalyzerRunStatus, so one failing tool never discards the other two's results; only an all-three failure fails the job. Metrics (issue density, cyclomatic complexity, maintainability index, Halstead, security counts) are computed once in metrics/python/ and persisted as JSONB. LOC comes from Radon's own raw output rather than a second line counter. Adds a fixture Python repository and 61 tests covering output parsing, severity/category mapping, metric aggregation, failure isolation and the Postgres round-trip, plus a Dockerfile so both toolchains run reproducibly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
First step toward review that can reason about a change in context. Every analyzer currently sees one file at a time, so nothing can answer "who called the function this PR just deleted?" — the question most real review comments depend on. CodeIndexer walks a checked-out workspace and builds a symbol/call graph: classes, functions and methods with their line spans, plus calls, imports and contains edges. symbols_touching_lines() is the bridge from a diff to that graph, and callers_of()/neighborhood() give the blast radius of a change. Deliberately AST-derived rather than model-extracted. On this service's own source that is 159 symbols and 937 edges across 50 files in 0.03s, with no tokens spent and no run-to-run variation — an LLM-built graph would cost money per pull request and can assert edges that don't exist. Call resolution is by unique name only, which resolves 32% of call edges here. The remainder is almost entirely stdlib (len, sum, Path). Where a name genuinely is ambiguous — _relative_path, defined in three analyzer classes — it stays unresolved rather than guessing: a missing edge makes a reviewer say less, a wrong one makes it say something false. Languages are data (Python, JS, TS, TSX), so adding one is a table entry rather than a new code path, matching how AnalyzerFactory handles tools. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- New diffing/ package: fetches the target branch, finds the merge base (deepening 50 -> 250 -> 1000 commits if needed), and diffs against it, so commits merged into the target branch since aren't counted as the PR's work. numstat -z lists files; --unified=0 gives exact changed lines. Lockfiles, build output and minified bundles are left out. - Every finding gets metadata.on_changed_line; changed lines are mapped to the innermost function/method/class, with its known caller count. - Runs alongside the analyzers and never fails the job: problems end as changes.status = "unavailable" with a reason. - AnalysisJob accepts optional targetBranch/title/description; AnalysisResult.changes is stored in a new pull_request_changes column. - docs/agent-architecture.md: the agent design, now with OpenAI as the provider and cost-first model defaults. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- retrieval/: eight read-only, size-limited tools for agents (read_file, get_symbol_source, find_symbol, callers_of, callees_of, list_tests_for, search_code, get_linter_findings), with OpenAI strict-mode tool specs and an executor that turns every error into text the model can fix. - Safety: paths are resolved (following symlinks) and confined to the checkout, .git/ is refused, likely credentials are redacted from everything an agent reads, and search is literal-only via git grep. - The indexer no longer follows symlinks, which could have pulled files from elsewhere on the host into the index. - context/: the bounded bundle the Reviewer starts from — PR intent, diff, full source of edited (not newly added) functions, callers, findings on changed lines, repo map — each section budgeted and announcing what it cut. - Findings are referenced by 10-char fingerprint prefixes to save tokens. - scripts/preview_context_pack.py shows a real PR's pack and its cost without calling any model; OpenAI prices live in agents/runtime/pricing.py. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- agents/runtime: provider-neutral LLMProvider, an OpenAI Responses API adapter (stateless store=False, encrypted reasoning passed back, strict tools, stable prefix for prompt caching), and an agent loop with hard limits on rounds, output tokens, time and dollars. Near any limit the model is forced (tool_choice) to submit what it has. - agents/reviewer: instructions, a strict ReviewerOutput schema delivered via a submit_review tool call, and validation that drops any claim whose quote isn't at the cited location, whose file/line doesn't exist, or that cites a non-existent linter finding. Dropped items are recorded. - review/: decides whether a PR is reviewed, runs the Reviewer, ranks and caps findings at 7. Never raises; failures are recorded with their cost. - The orchestrator now keeps the checkout and index alive for the review and hands the linter result to the caller before the review starts; the consumer saves it first, then the review (agent_reviews table, upserted). - Pricing includes GPT-5.6 cache-write charges; finding refs are accepted with display brackets (found in the first real review). - scripts/review_pr.py runs a real review and prints its exact cost. - Tests run against a scripted fake provider: no API cost. First real review with gpt-5.6-luna: $0.00073 with a warm prompt cache. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- evaluation/: 18 PRs (12 with a planted bug linters can't see, 6 clean including two decoys), each built as a real git repository and run through the real pipeline; only the clone is replaced. - Scoring: recall, precision, false alarms per clean PR, p95 duration, cost per PR, against the targets in the architecture doc. Defects are located by snippet, with alternative locations (e.g. a changed signature or the caller it breaks). - --dry-run exercises everything except the model, for free; real runs stop starting cases once --max-cost is reached. Results are saved to evaluation-results/ (git-ignored) and compared with the previous run. Baseline on gpt-5.6-luna: 12/12 found, every reported issue real, no false alarms on clean PRs, $0.0006 per PR. Two "clean" cases were initially mislabelled; the Reviewer's reports on them were correct, so the cases were fixed. The set is still too easy; harder cases next. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…se 4) - 10 harder cases: bugs among legitimate multi-file changes, two bugs in one PR, a changed contract breaking untouched callers, idempotency, JWT verification, timezones, a threshold on the wrong total, a default flipped inside a large refactor, and larger clean PRs. - Verifier (Agent 2): one fresh-context call per reported issue that tries to refute it. Its verdict is applied under rules in code: only an explicit "drop" removes an issue, a drop citing code that isn't there is rejected, severity can only go down, an unconfirmable issue is dropped but a provider outage keeps issues (unverified). - The first Verifier version cut recall from 0.95 to 0.71: it labelled "was this intended? - no" as refuting, and any refuting check could override a keep. Checks are now an explicit refutes_issue boolean. - Harness: drop reasons recorded per case, runs compared only against the same case set, --no-verifier flag. 28 PRs on gpt-5.6-luna with the Verifier: recall 0.95, precision 0.95, 0.12 false alarms per clean PR, $0.0016 per PR. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…re, phase 5)
- Rules come from .codepulse/rules.yml read from the PR's target branch
(a PR can't weaken the rules it's reviewed against), plus rules stored
per repository (business_rules table): added in the dashboard, or
suggested by the Rule Miner and accepted by a person. One malformed rule
is reported without disabling the rest; yaml.safe_load only.
- The context pack lists applicable rules; a get_rule tool. The Reviewer
records a verdict per rule, and rule violations must cite a real rule
id plus code evidence. A rule is "violated" only when an issue citing it
survives the evidence checks and the Verifier; a rule's severity caps
the issue's.
- Rule Miner (Agent 3): suggests rules from a repository's docs, tests and
constraint statements. Every suggestion's quoted source is verified,
scopes must match real files, ids must be new; stored as "suggested".
- Rules API: list, add, edit/accept/reject, delete, and background mining.
- Files that hold credentials (.env, keys, credential files) are never
shown to any agent: refused by read_file, filtered from search, left out
of diffs and mining. Redaction alone doesn't catch plain values.
- Fixed: applies_to cleanup used lstrip("./"), silently turning
"../secrets/**" into "secrets/**" instead of rejecting it.
- Fixed: a malformed answer on an agent's last round now gets its repair.
- Evaluation: 7 business-rule cases. 35 PRs: 25/25 found, $0.0014 per PR.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Private repositories:
- A clone token per job from integration-service's internal API, passed
to git only through GIT_CONFIG_COUNT/KEY/VALUE as an extraheader scoped
to the provider's host: never on the command line (ps), never in
.git/config, never in os.environ (linters and agent tools can't inherit
it), never logged. Used for the clone, the target-branch fetch and the
Rule Miner's ls-remote. GIT_TERMINAL_PROMPT=0.
- No INTERNAL_SERVICE_TOKEN, an unconnected repository, or an unreachable
integration-service all mean an anonymous clone.
- Tested against a local fake git host: header sent to the right host,
not to another, and absent from .git/ and errors.
Phase 6:
- PUT .../review/findings/{fingerprint}/feedback: useful / not_useful /
wrong, one verdict per user (X-User-Id, set by main-backend from the
verified session). Feedback is attached when a review is read.
- An issue a developer marked wrong isn't reported again on later
reviews (dropped with stage "feedback").
- GET .../review-usage?days=30: reviews, cost, issues, verdicts and the
real-world false-alarm rate.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The evaluation harness printed its results and forgot them, so comparing a prompt change against last week's numbers meant scrolling back through a terminal. --save writes each case's result and review to the database, where they can be queried and compared. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Runs on every branch and pull request: install, then the whole suite,
then build the image.
Two things the workflow has to supply that a developer's machine gets from
.env, which is not in the repository:
* The database settings. Settings deliberately has no defaults for them
— the service refuses to start half-configured — and that check runs at
import time, so the suite needs them even though most tests open no
connection. A throwaway Postgres service container covers the tests
marked `integration`, which store a real result through the real schema
because the JSONB columns are exactly where a silent coercion would
hide.
* ESLint's own toolchain, installed from tools/eslint, since part of the
suite runs the real linter rather than a mock.
No OPENAI_API_KEY is set, deliberately: the agent tests stub the provider,
so a test that ever reached the real API will fail here rather than quietly
spend money on every push.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The dependency layer copied pyproject.toml on its own and then ran an editable install, which setuptools cannot resolve: packages.find looks in src/, and at that point in the build there is no src/ at all. Every build of this image has failed there — "Getting requirements to build editable did not run successfully" — which CI made visible the moment it started building images. A placeholder package satisfies the resolution; COPY . . replaces it with the real source that the editable install already points to, so the dependency layer still only rebuilds when dependencies change. Verified in the built image: the app imports and ESLint resolves to /app/tools/eslint/node_modules/.bin/eslint. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Runs only after CI passes on main, so a red build cannot reach the running platform. It builds the image, pushes it to ECR, and asks the application host to pull and restart just this service. No AWS access key is stored anywhere. GitHub signs a short-lived OIDC token, and the role that trusts it is restricted to this organization's repositories on main — a pull request from a fork cannot assume it, and the credentials expire with the job. The rollout goes through SSM rather than SSH: the host has no SSH port open and no key pair, so there is no long-lived credential to leak or lose. It fails the job when the rollout fails, because a broken deploy that reports green is worse than one that reports red. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ShenethFd
added a commit
that referenced
this pull request
Oct 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.