Skip to content

Help improve the review bot #622

Open

Description

@negz

What problem are you facing?

The review bot from #611 now reviews every PR an org member opens. Its first 15 reviews were rarely wrong (2 of 102 inline comments), but posted too many nits, and every summary repeated the inline comments. Ideas for fixing this come up in review threads and chat, and get lost.

We also can't yet tell whether a change to the skills makes reviews better or worse.

How could Modelplane help solve your problem?

Please add ideas here, one per comment. Say what's wrong and suggest a change, with links to bot comments that show the problem, and open a separate issue for anything bigger than a small change.

Please also react 👍 or 👎 to bot comments, or reply when one is wrong. Those reactions are the labels an eval will need. Each review's footnote records which version of the skills wrote it.

Activity

  1. pinned this issue on Nov 3, 2026
  2. negz commented on Nov 3, 2026

    @negz
    CollaboratorAuthor

    Ideas from an LLM-as-judge pass over the first 15 reviews:

    1. Shrink the summary to the verdict and one sentence of reason. All 15 summaries repeated the inline comments or described the PR (#610, #572). The skill already says not to, so a smaller target may work better than another rule.
    2. Allow at most two nits, and none alongside substantive findings. 28% of comments were nits. #538 posted seven nits around one real finding. #488 questioned the approach, then posted nits the skill says to hold back until the design settles.
    3. Don't base a finding on an unmerged PR, beyond noting the overlap. #574 questioned the core design using details from Propose OCI and PersistentVolumeClaim ModelCache sources #505, an unmerged draft. #607 depended on what InferenceCluster: Let clusters declare their own system node pools, or none at all #550, also unmerged, does.
    4. Check a claimed consequence against today's code. The pass found two wrong comments, and both failed this check. #578 claimed a cost that doesn't exist, and #505 called the doc wrong when the code matched it.
    5. Only raise problems the PR introduced. #570 speculated about a race in code the PR didn't change.
    6. One point per comment. #572 asked the same question in two comments, and #592 put two asks in one.
    7. Explain why style comments don't ask. The skill says to state style points plainly, without saying why. The project has a style, and a PR isn't the place to debate it. A style point phrased as a question invites that debate. With the reason, the bot can tell a project-style point from a design choice, and phrase each the way it should be.
  3. negz commented on Nov 3, 2026

    @negz
    CollaboratorAuthor

    Ideas from comparing the first 15 reviews with 131 maintainer reviews on crossplane/crossplane, from mid-2022 to January 2025, before we reviewed with LLMs:

    Maintainers Bot
    Comment length, median characters of prose 136 442
    Comments of 120 characters or fewer 46% 2%
    Inline comments in a first review of a PR, median 3 7
    Reviews with a summary 44% 100%
    Comments with a suggestion block 18% 0%
    1. Allow one-line comments, and aim for 150-250 characters. That's between the maintainers' median (136) and their 75th percentile (270). Nearly half of maintainer comments are one line, such as "Were these proto changes accidental?"
    2. Post fewer comments. Maintainers post a few in a first review, then fewer in later rounds as the author replies. The bot posts everything at once.
    3. Make the summary optional, or one line.
    4. Use suggestion blocks for small edits. The bot avoids them because applying one makes a commit without a Signed-off-by, which fails DCO, so it describes edits in prose instead.

    The sample is each maintainer's most-commented PRs on a more mature project, so typical reviews are probably lighter.

  4. negz commented on Nov 3, 2026

    @negz
    CollaboratorAuthor

    ^ I had my agent generate these suggestions. I agree with most, but not all. For example I don't mind reviews that contain a lot of nits. I definitely wouldn't want to cap at 2. I also think the fact that human maintainers review in waves is a bit of an anti-pattern - though unavoidable when a PR changes shape significantly in response to review.

  5. changed the title [-]Improve the review bot[/-] [+]Help improve the review bot[/+] on Nov 3, 2026
  6. kaessert commented on Nov 3, 2026

    @kaessert

    I did some review too :)

    Eval of these skills with an agent

    I ran the skills from this PR (head fc77425) through an agent eval to see what an agent actually reads and whether it changes the output.

    Setup

    Two repeats per cell, so these are trends, not significant results.

    The skills work for designs and reviews

    Share of applicable rules followed, two repeats per cell:

    Task GLM without GLM with Sonnet without Sonnet with
    Fix #527 0.84, 0.96 0.89, 0.92 0.94, 0.99 1.00, 0.99
    Design #502 0.74, 0.61 0.83, 0.85 0.75, 0.76 0.91, 0.87
    Review #584 0.50, 0.48 0.86, 0.63 0.64, 0.68 0.90, 0.64*

    * This run didn't load reviewing-pull-requests. It scored the same as the runs without skills.

    • Agents found the right skill on their own in 11 of 12 runs, usually within the first few tool calls.
    • reviewing-pull-requests (0.71 vs 0.43) and design-docs (0.84 vs 0.60) made the biggest difference.
      • Several rules were never followed without the skill and always followed with it: the review summary under 1,000 characters, opening with a hedged mergeability sentence, fixes suggested as questions, and putting the runner-up of a close choice in Alternatives.
      • Reviews written with the skill were 180–580 words, against 840–1,250 without.
    • For the code fix, the code skills changed little: both models followed those rules anyway.

    Read, but not followed

    These design-docs rules were broken in every design run that loaded the skill:

    • gather discarded ideas and objections in Alternatives instead of arguing them inline
    • a worked example for each case the proposal handles, including the awkward one
    • lift the sharp problem statement into the opening of the background
    • give each alternative, including the current system, a fair account

    Reading them isn't enough. They might work better as a short checklist the agent runs over its finished draft, or as a Vale rule where one is possible. Also in this group: "hyperlink named projects" in writing-style.

    Never read

    • writing-style was loaded for design docs, but never for PR descriptions or review comments, though both are in its scope. Naming those explicitly in its description might help.
    • The reference files rarely load: design-docs/references/voice.md in 2 of 4 design runs, de-slop-review.md in 1 of 4. If they matter, a stronger pointer from the SKILL.md, or moving their key rules into it, would help.
    • nix-flake, nix-tools and python-code-factoring/references/example.md were never loaded, but none of these tasks really needed them.

    Not covered

    The judge graded the skills' rules, not correctness: whether the fix is right, or whether the design and review match what the maintainers concluded. One judge pass per run, so verdicts have some noise.

  7. negz commented on Nov 4, 2026

    @negz
    CollaboratorAuthor

    Thanks @kaessert! Right now I intend the skills only to guide review, not implementation. That said it's good to know they help somewhat with implementation, and not surprising given I mostly just copied them from skills I wrote to help with implementation.

    I imagine we need to eval using the same harness and model as the review bot to get the strongest signal, right?

    BTW I edited the comment as the report was repeated twice.

  8. negz commented on Nov 4, 2026

    @negz
    CollaboratorAuthor

    I ran the review bot five times on each of four PRs, to see how much its findings vary between runs. We'd like evals that tell us whether a change to the skills makes the bot catch more or fewer problems. If two runs of the same skills raise different findings, an eval has to separate what the change did from what sampling did. This measures how much sampling does.

    Sampling does a lot, and more on the two large PRs. One run raised 44 to 65% of the distinct findings that five runs raised between them. Each run raised a similar number of findings, but a different set. On #632 every run raised 9 or 10 of the 19, and only 2 came up in every run. On the small PRs, most blocking findings and bugs came up in every run. On the large PRs even those came and went, and three of the four findings the bot called blocking on #632 appeared in only one or two runs.

    PR Lines changed Distinct findings In all 5 runs In 1 run One run's share Blocking findings and bugs One run's share of those
    #625 298 15 5 4 60% 7 91%
    #619 321 22 10 6 65% 4 80%
    #632 1,302 19 2 6 51% 6 63%
    #607 12,125 22 2 10 44% 4 60%

    A run's share is the fraction of the five runs' distinct findings that it raised, averaged over the five runs. Blocking findings are the ones the bot presented as blocking, and bugs are the ones that claim something breaks.

    The opening verdicts were steadier than the findings, but not stable. All five runs agreed on #619 and #607. On #625, four runs asked for a step back on the design and one for another look at part of it. On #632, three asked for a step back and two called it close, and the five led with different concerns.

    So one run of each PR before and after a change isn't enough to show whether the change made the bot catch more. Sampling alone changes which findings a run raises, especially on large PRs. An eval needs several runs of each PR, and should compare how often each finding comes up rather than whether it came up once. I haven't checked whether the findings only one run raised are right, so some of the spread may be noise rather than missed problems.

    The results also suggest a way to improve reviews of large PRs. On #632 and #607, two runs' findings combined would have covered 17 and 18 points more of the blocking findings and bugs than one run, at twice the cost.

    How I ran it:

    The workflow, prompts, scripts, grouped findings and all 20 reviews are in this gist, with steps to run it again.

  9. negz commented on Nov 6, 2026

    @negz
    CollaboratorAuthor

    In #659 the bot reviewed our nascent charter. We should ask it not to review governance document updates - I want to limit those to human review.

    Edit: Done in #661.

  10. added
    g/test-and-ciMake Modelplane changes verifiable across supported environments as its capabilities evolve
    on Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestg/test-and-ciMake Modelplane changes verifiable across supported environments as its capabilities evolve

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions