ci: add informational code-erosion (slop metrics) workflow - #588
Open
jeff-r2026 wants to merge 2 commits into
Open
jeff-r2026 wants to merge 2 commits into
jeff-r2026 wants to merge 2 commits into
Conversation
Report SlopCodeBench verbosity/erosion metrics on every PR using the
official scb-check tool, pinned to 0.2.0 (the first release with
TypeScript support; 0.1.3 is Python-only).
The workflow is informational and never blocks a merge: scb-check's exit
code is swallowed, and the numbers are posted as a deduplicated PR comment
with the run's job summary as a fallback (so fork PRs, whose token is
read-only, still surface the report). Tests are excluded via scb-check.toml
so metrics reflect the product surface.
On TypeScript the ast-grep verbosity rule component is Python-only and
contributes 0, so verbosity reflects clone + wrapper detection only;
erosion is fully faithful. This caveat is documented in the bilingual
docs/ci-code-erosion.{md,zh-CN.md}.
Code Erosion Report (informational)Reported by
Scanned 234 files / 38342 SLOC · high-CC functions 220/2601 * On TypeScript, Rule hits (TS verbosity layer)Structural slop rules ported from SlopCodeBench and run via a standalone
30 distinct lines flagged across |
scb-check only runs its ast-grep rules on Python files, so on this TypeScript repo its verbosity rule component is always 0. This adds a standalone ast-grep pass with a small, hand-ported rule set to fill that gap, reported as a separate "Rule hits (TS verbosity layer)" section in the same non-blocking PR comment. Only purely structural rules are ported. Rules that hinge on truthiness or type semantics (len==0, ==True, redundant template strings) were tried and deliberately dropped: they are false positives in TypeScript, where arr.length>0 is idiomatic and x!==true is not equivalent to x===false (TS has undefined). Ported rules verified against src/ for false positives: unnecessary-else-after-return, empty-catch-block, redundant-ternary-same, if-return-boolean-literal, return-ternary-boolean-literal, duplicated-if-condition, self-assignment. Rules use severity: hint and the scan step has `|| true`, so the layer never blocks CI. Bilingual docs updated with the honest scope: this is an extra signal, not a reproduction of the paper's verbosity number.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Adds an informational CI workflow that reports two "code sloppiness"
metrics on every PR, using the official
scb-checktool (the SlopCodeBenchreference implementation from Measuring the sloppiness of code):
It never blocks a merge — it only posts the numbers for reviewers to eyeball.
Design decisions
scb-check==0.2.0— the first release with TypeScriptsupport (the version the paper pins,
0.1.3, is Python-only). Pinning alsokeeps numbers comparable across runs.
scb-checkexits non-zero whenever it finds any slop(the normal case). The workflow swallows the exit code; the render step and
the whole job stay green under every failure mode.
marker, updated in place on each push), with the job summary as a fallback so
fork PRs (read-only token) still surface the report.
src/, excluding tests viascb-check.toml, so metricsreflect the product surface.
Honest caveat (documented)
On TypeScript,
scb-check's 197 ast-grep verbosity rules are Python-onlyand contribute 0, so
verbosityhere reflects clone + wrapper detectiononly.
erosionis fully faithful (cyclomatic complexity is language-agnostic).Reference bands are Python-calibrated — read them as direction, not verdict.
Documented in the bilingual
docs/ci-code-erosion.{md,zh-CN.md}.Files
.github/workflows/code-erosion.yml.github/scripts/erosion-summary.pyscb-check.tomldocs/ci-code-erosion.md+.zh-CN.md.github/CONTRIBUTING.mdci.ymlis untouched — this runs fully independently.Test plan
Verified locally end to end (scb-check 0.2.0 against this branch's
src/):npx tsc --noEmitpasses (no source touched)python3 -m py_compileall passscb-checkproduces JSON (exit 1 swallowed) → renderer emits marker-bearing Markdown → github-script reads it and finds the markerexit 0on every failure mode: renamed JSON key, empty/corrupt JSON, missing fileRemaining verification that only real CI can exercise (will confirm on this PR's run):
Code Erosionjob runs and is green even though scb-check exits 1Update: independent TS verbosity rule layer (commit 910ed06)
scb-check only runs its ast-grep rules on Python files, so this repo's verbosity
rule component is structurally always 0. Added a standalone
ast-greppass witha small hand-ported, purely-structural rule set, reported as a separate
Rule hits (TS verbosity layer)section in the same non-blocking comment.Deliberately dropped (false positives in TS, verified via PoC):
len==0(
arr.length>0is idiomatic),==true(x!==truehandlesboolean|undefined,not equivalent to
x===false), redundant template strings (multi-line concat).Ported (verified against
src/, no false positives):unnecessary-else-after-return,empty-catch-block,redundant-ternary-same,if-return-boolean-literal,return-ternary-boolean-literal,duplicated-if-condition,self-assignment.Test plan (rule layer)
ast-grep scan→ 30 hits onsrc/(else-after-return 23, empty-catch 6, ternary-same 1), all true positives (sampled)severity: hint+ step has|| true→ scan exits 0 even with matchesRule hits (TS verbosity layer)section, numbers matching local (30 hits)