Skip to content

[Utils] Add token-based duplicate code detector with CI gate - #8482

Merged
TomasVotruba merged 1 commit into
mainfrom
tv-phpcpd
Sep 12, 2026
Merged

[Utils] Add token-based duplicate code detector with CI gate#8482
TomasVotruba merged 1 commit into
mainfrom
tv-phpcpd

Conversation

@TomasVotruba

@TomasVotruba TomasVotruba commented Sep 11, 2026

Copy link
Copy Markdown
Member

Adds utils/duplicates, a minimal phpcpd clone, plus a CI gate.

How it works

  • Tokenizes PHP with token_get_all() (no php-parser dependency).
  • Strips whitespace, comments and tags, builds a per-token signature (type byte + CRC32 of the token text).
  • Finds exact duplicate token spans with a Rabin-Karp rolling hash over a --min-tokens window, filtered by --min-lines.

Usage

php utils/duplicates/bin/duplicates.php src rules

Options: --min-lines (default 5), --min-tokens (default 70), --fuzzy. Exit code is 1 when clones are found.

CI

A Duplicated Code workflow runs composer duplicated-code and fails when a copy-pasted block >= 400 tokens is added. 400 is the clean floor on current main; once the rule de-duplication in #8486 lands, this can be tightened to 150.

Covered by CloneDetectorTest.

@TomasVotruba
TomasVotruba force-pushed the tv-phpcpd branch 2 times, most recently from 5d0d8db to a5bcd16 Compare September 11, 2026 18:59
@TomasVotruba TomasVotruba changed the title [Utils] Add token-based duplicate code detector in utils/duplicates [Utils] Add token-based duplicate code detector in utils/duplicates, lower duplicateds under 250 tokens Sep 12, 2026
@TomasVotruba TomasVotruba changed the title [Utils] Add token-based duplicate code detector in utils/duplicates, lower duplicateds under 250 tokens [Utils] Add token-based duplicate code detector in utils/duplicates, lower duplicateds under 210 tokens Sep 12, 2026
@TomasVotruba TomasVotruba changed the title [Utils] Add token-based duplicate code detector in utils/duplicates, lower duplicateds under 210 tokens [Utils] Add token-based duplicate code detector in utils/duplicates, lower duplicateds under 150 tokens Sep 12, 2026
Adds utils/duplicates, a minimal phpcpd clone (token_get_all + Rabin-Karp),
a plain PHP bin script, unit tests, and a Duplicated Code CI job that fails
when a copy-pasted block >= 400 tokens is added.
@TomasVotruba TomasVotruba changed the title [Utils] Add token-based duplicate code detector in utils/duplicates, lower duplicateds under 150 tokens [Utils] Add token-based duplicate code detector with CI gate Sep 12, 2026
@TomasVotruba
TomasVotruba merged commit 9c56f19 into main Sep 12, 2026
45 checks passed
@TomasVotruba
TomasVotruba deleted the tv-phpcpd branch September 12, 2026 09:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant