Skip to content

ci: add informational code-erosion (slop metrics) workflow - #588

Open
jeff-r2026 wants to merge 2 commits into
mainfrom
feature/code-erosion-ci
Open

jeff-r2026 wants to merge 2 commits into
mainfrom
feature/code-erosion-ci

Conversation

@jeff-r2026

@jeff-r2026 jeff-r2026 commented Sep 16, 2026

Copy link
Copy Markdown
Collaborator

What & why

Adds an informational CI workflow that reports two "code sloppiness"
metrics on every PR, using the official
scb-check tool (the SlopCodeBench
reference implementation from Measuring the sloppiness of code):

  • Verbosity — redundant lines (clones + wrappers + ast-grep rule hits) / SLOC
  • Erosion — share of complexity mass held by functions with cyclomatic complexity > 10

It never blocks a merge — it only posts the numbers for reviewers to eyeball.

Design decisions

  • Tool pinned to scb-check==0.2.0 — the first release with TypeScript
    support (the version the paper pins, 0.1.3, is Python-only). Pinning also
    keeps numbers comparable across runs.
  • Never blocks CIscb-check exits non-zero whenever it finds any slop
    (the normal case). The workflow swallows the exit code; the render step and
    the whole job stay green under every failure mode.
  • Report surface — a single deduplicated PR comment (keyed by an HTML
    marker, updated in place on each push), with the job summary as a fallback so
    fork PRs (read-only token) still surface the report.
  • Scope — scans src/, excluding tests via scb-check.toml, so metrics
    reflect the product surface.

Honest caveat (documented)

On TypeScript, scb-check's 197 ast-grep verbosity rules are Python-only
and contribute 0, so verbosity here reflects clone + wrapper detection
only. erosion is fully faithful (cyclomatic complexity is language-agnostic).
Reference bands are Python-calibrated — read them as direction, not verdict.
Documented in the bilingual docs/ci-code-erosion.{md,zh-CN.md}.

Files

File Purpose
.github/workflows/code-erosion.yml The workflow
.github/scripts/erosion-summary.py JSON → Markdown renderer (always exits 0)
scb-check.toml Excludes tests from the scan
docs/ci-code-erosion.md + .zh-CN.md Team-facing explainer + TS caveat
.github/CONTRIBUTING.md One line noting the non-blocking check

ci.yml is untouched — this runs fully independently.

Test plan

Verified locally end to end (scb-check 0.2.0 against this branch's src/):

  • npx tsc --noEmit passes (no source touched)
  • YAML syntax, embedded github-script JS syntax (async-wrapped), and python3 -m py_compile all pass
  • Full 3-step chain: scb-check produces JSON (exit 1 swallowed) → renderer emits marker-bearing Markdown → github-script reads it and finds the marker
  • Renderer stays exit 0 on every failure mode: renamed JSON key, empty/corrupt JSON, missing file
  • Measured baseline: verbosity 0.092 / erosion 0.653 / cog_erosion 0.858 (234 files / 38,337 SLOC, high-CC 219/2598)

Remaining verification that only real CI can exercise (will confirm on this PR's run):

  • The Code Erosion job runs and is green even though scb-check exits 1
  • PR comment appears with the numbers, and updates (not duplicates) on a second push
  • Job summary renders the same table

Update: independent TS verbosity rule layer (commit 910ed06)

scb-check only runs its ast-grep rules on Python files, so this repo's verbosity
rule component is structurally always 0. Added a standalone ast-grep pass with
a small hand-ported, purely-structural rule set, reported as a separate
Rule hits (TS verbosity layer) section in the same non-blocking comment.

Deliberately dropped (false positives in TS, verified via PoC): len==0
(arr.length>0 is idiomatic), ==true (x!==true handles boolean|undefined,
not equivalent to x===false), redundant template strings (multi-line concat).

Ported (verified against src/, no false positives): unnecessary-else-after-return,
empty-catch-block, redundant-ternary-same, if-return-boolean-literal,
return-ternary-boolean-literal, duplicated-if-condition, self-assignment.

Test plan (rule layer)

  • Local: ast-grep scan → 30 hits on src/ (else-after-return 23, empty-catch 6, ternary-same 1), all true positives (sampled)
  • Rules use severity: hint + step has || true → scan exits 0 even with matches
  • Renderer degrades gracefully: missing/empty rule-hits file omits the section, main report unaffected
  • Real CI green with the rule layer added (run 35099854211)
  • PR comment updated in place (not duplicated) with the new Rule hits (TS verbosity layer) section, numbers matching local (30 hits)
  • Bilingual docs updated with honest scope (extra signal, not a reproduction of the paper's verbosity)

Report SlopCodeBench verbosity/erosion metrics on every PR using the
official scb-check tool, pinned to 0.2.0 (the first release with
TypeScript support; 0.1.3 is Python-only).

The workflow is informational and never blocks a merge: scb-check's exit
code is swallowed, and the numbers are posted as a deduplicated PR comment
with the run's job summary as a fallback (so fork PRs, whose token is
read-only, still surface the report). Tests are excluded via scb-check.toml
so metrics reflect the product surface.

On TypeScript the ast-grep verbosity rule component is Python-only and
contributes 0, so verbosity reflects clone + wrapper detection only;
erosion is fully faithful. This caveat is documented in the bilingual
docs/ci-code-erosion.{md,zh-CN.md}.
@github-actions

github-actions Bot commented Sep 16, 2026

Copy link
Copy Markdown

Code Erosion Report (informational)

Reported by scb-check==0.2.0 (SlopCodeBench metrics). This never blocks CI.

Metric Value Human band Agent band Reading
Verbosity* 0.092 0.15 0.33 human
Erosion 0.650 0.31 0.68 agent
Cognitive erosion 0.857

Scanned 234 files / 38342 SLOC · high-CC functions 220/2601

* On TypeScript, verbosity covers clone + wrapper detection only (scb-check's 197 ast-grep rules are Python-only). erosion is faithful. Reference bands are Python-calibrated — read them as direction, not verdict. See docs/ci-code-erosion.md.

Rule hits (TS verbosity layer)

Structural slop rules ported from SlopCodeBench and run via a standalone ast-grep (scb-check only rules Python files). Separate from the verbosity number above; also non-blocking.

Rule Hits
unnecessary-else-after-return 23
empty-catch-block 6
redundant-ternary-same 1

30 distinct lines flagged across src/.

scb-check only runs its ast-grep rules on Python files, so on this
TypeScript repo its verbosity rule component is always 0. This adds a
standalone ast-grep pass with a small, hand-ported rule set to fill that
gap, reported as a separate "Rule hits (TS verbosity layer)" section in
the same non-blocking PR comment.

Only purely structural rules are ported. Rules that hinge on truthiness or
type semantics (len==0, ==True, redundant template strings) were tried and
deliberately dropped: they are false positives in TypeScript, where
arr.length>0 is idiomatic and x!==true is not equivalent to x===false
(TS has undefined). Ported rules verified against src/ for false positives:
unnecessary-else-after-return, empty-catch-block, redundant-ternary-same,
if-return-boolean-literal, return-ternary-boolean-literal,
duplicated-if-condition, self-assignment.

Rules use severity: hint and the scan step has `|| true`, so the layer
never blocks CI. Bilingual docs updated with the honest scope: this is an
extra signal, not a reproduction of the paper's verbosity number.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant