Open
Labels
enhancementNew feature or requestNew feature or requestg/test-and-ciMake Modelplane changes verifiable across supported environments as its capabilities evolveMake Modelplane changes verifiable across supported environments as its capabilities evolve
Description
What problem are you facing?
The review bot from #611 now reviews every PR an org member opens. Its first 15 reviews were rarely wrong (2 of 102 inline comments), but posted too many nits, and every summary repeated the inline comments. Ideas for fixing this come up in review threads and chat, and get lost.
We also can't yet tell whether a change to the skills makes reviews better or worse.
How could Modelplane help solve your problem?
Please add ideas here, one per comment. Say what's wrong and suggest a change, with links to bot comments that show the problem, and open a separate issue for anything bigger than a small change.
Please also react 👍 or 👎 to bot comments, or reply when one is wrong. Those reactions are the labels an eval will need. Each review's footnote records which version of the skills wrote it.
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or requestg/test-and-ciMake Modelplane changes verifiable across supported environments as its capabilities evolveMake Modelplane changes verifiable across supported environments as its capabilities evolve
Type
Projects
- StatusShow more project fieldsBacklog
Activity
negz commentedon Nov 3, 2026
Ideas from an LLM-as-judge pass over the first 15 reviews:
negz commentedon Nov 3, 2026
Ideas from comparing the first 15 reviews with 131 maintainer reviews on crossplane/crossplane, from mid-2022 to January 2025, before we reviewed with LLMs:
Signed-off-by, which fails DCO, so it describes edits in prose instead.The sample is each maintainer's most-commented PRs on a more mature project, so typical reviews are probably lighter.
negz commentedon Nov 3, 2026
^ I had my agent generate these suggestions. I agree with most, but not all. For example I don't mind reviews that contain a lot of nits. I definitely wouldn't want to cap at 2. I also think the fact that human maintainers review in waves is a bit of an anti-pattern - though unavoidable when a PR changes shape significantly in response to review.
[-]Improve the review bot[/-][+]Help improve the review bot[/+]kaessert commentedon Nov 3, 2026
I did some review too :)
Eval of these skills with an agent
I ran the skills from this PR (head
fc77425) through an agent eval to see what an agent actually reads and whether it changes the output.Setup
AGENTS.md, nothing else.Two repeats per cell, so these are trends, not significant results.
The skills work for designs and reviews
Share of applicable rules followed, two repeats per cell:
* This run didn't load
reviewing-pull-requests. It scored the same as the runs without skills.reviewing-pull-requests(0.71 vs 0.43) anddesign-docs(0.84 vs 0.60) made the biggest difference.Read, but not followed
These
design-docsrules were broken in every design run that loaded the skill:Reading them isn't enough. They might work better as a short checklist the agent runs over its finished draft, or as a Vale rule where one is possible. Also in this group: "hyperlink named projects" in
writing-style.Never read
writing-stylewas loaded for design docs, but never for PR descriptions or review comments, though both are in its scope. Naming those explicitly in its description might help.design-docs/references/voice.mdin 2 of 4 design runs,de-slop-review.mdin 1 of 4. If they matter, a stronger pointer from theSKILL.md, or moving their key rules into it, would help.nix-flake,nix-toolsandpython-code-factoring/references/example.mdwere never loaded, but none of these tasks really needed them.Not covered
The judge graded the skills' rules, not correctness: whether the fix is right, or whether the design and review match what the maintainers concluded. One judge pass per run, so verdicts have some noise.
negz commentedon Nov 4, 2026
Thanks @kaessert! Right now I intend the skills only to guide review, not implementation. That said it's good to know they help somewhat with implementation, and not surprising given I mostly just copied them from skills I wrote to help with implementation.
I imagine we need to eval using the same harness and model as the review bot to get the strongest signal, right?
BTW I edited the comment as the report was repeated twice.
negz commentedon Nov 4, 2026
I ran the review bot five times on each of four PRs, to see how much its findings vary between runs. We'd like evals that tell us whether a change to the skills makes the bot catch more or fewer problems. If two runs of the same skills raise different findings, an eval has to separate what the change did from what sampling did. This measures how much sampling does.
Sampling does a lot, and more on the two large PRs. One run raised 44 to 65% of the distinct findings that five runs raised between them. Each run raised a similar number of findings, but a different set. On #632 every run raised 9 or 10 of the 19, and only 2 came up in every run. On the small PRs, most blocking findings and bugs came up in every run. On the large PRs even those came and went, and three of the four findings the bot called blocking on #632 appeared in only one or two runs.
A run's share is the fraction of the five runs' distinct findings that it raised, averaged over the five runs. Blocking findings are the ones the bot presented as blocking, and bugs are the ones that claim something breaks.
The opening verdicts were steadier than the findings, but not stable. All five runs agreed on #619 and #607. On #625, four runs asked for a step back on the design and one for another look at part of it. On #632, three asked for a step back and two called it close, and the five led with different concerns.
So one run of each PR before and after a change isn't enough to show whether the change made the bot catch more. Sampling alone changes which findings a run raises, especially on large PRs. An eval needs several runs of each PR, and should compare how often each finding comes up rather than whether it came up once. I haven't checked whether the findings only one run raised are right, so some of the spread may be noise rather than missed problems.
The results also suggest a way to improve reviews of large PRs. On #632 and #607, two runs' findings combined would have covered 17 and 18 points more of the blocking findings and bugs than one run, at twice the cost.
How I ran it:
mainat 7f50acb, with Opus 5.5.The workflow, prompts, scripts, grouped findings and all 20 reviews are in this gist, with steps to run it again.
negz commentedon Nov 6, 2026
In #659 the bot reviewed our nascent charter. We should ask it not to review governance document updates - I want to limit those to human review.
Edit: Done in #661.