[uplift] chore: Vitest benchmarks vs. last release and fast-deep-equal

.github/pull_request_template.md

@@ -13,5 +13,5 @@ Please include a summary of the change and which issue is fixed. Please also inc
 
 - [ ] All tests are passing
 - [ ] Type definitions, if updated, pass `types:check` and `pack:check`
-- [ ] Benchmark performance has not significantly decreased
+- [ ] Benchmark performance (`pnpm bench`, CI job summary) has not significantly decreased
 - [ ] Bundle size (`pnpm size`, CI job summary) has not been significantly impacted

.github/workflows/ci.yml

@@ -126,3 +126,25 @@ jobs:
           node -e 'for (const [p, v] of [["react", "18.0.0"], ["react-dom", "18.0.0"], ["preact", "10."]]) { const got = require(`${p}/package.json`).version; console.log(`${p} ${got}`); if (!got.startsWith(v)) throw new Error(`${p} is ${got}, expected ${v}`); }'
       - name: Test
         run: pnpm test --project node
+
+  # Report only: `pnpm bench` adds the benchmark tables to the job summary.
+  # Shared runners are too noisy to gate on the numbers; the job fails only
+  # if the benchmark itself breaks.
+  bench:
+    name: Benchmarks
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read # to check out repo (actions/checkout)
+    steps:
+      - uses: actions/checkout@v7
+        with:
+          persist-credentials: false
+      - uses: pnpm/action-setup@ea17c68df8912ef543352723c149a84f56e3d413 # v6.1.0
+      - uses: actions/setup-node@v7
+        with:
+          node-version-file: '.nvmrc'
+          cache: 'pnpm'
+      - name: Install dependencies
+        run: pnpm install --frozen-lockfile
+      - name: Benchmarks
+        run: pnpm bench

CONTRIBUTING.md

@@ -13,8 +13,8 @@ We encourage pull requests concerning:
 - Integrating updates from `fast-deep-equal` － This, unfortunately, now requires more manual work. Use the comment blocks in `index.js`
   to figure out what to paste and where.
 - Integrating tests from `fast-deep-equal` － Copy its `spec/` test files into `test/fixtures/fast-deep-equal/`
-  and update the version in that directory's README. The `fast-deep-equal` npm package in `devDependencies` is the
-  benchmark baseline.
+  and update the version in that directory's README. The `fast-deep-equal` npm package in `devDependencies` is one of
+  the [benchmark](#benchmarks) baselines.
 - Bugs in this library
 - New tests for React
 - Documentation
@@ -55,7 +55,7 @@ included. For an urgent security fix that's newer than that, add the package to
 ```sh
 # Format, lint, types, all tests, then the packed package
 $ pnpm run check
-$ pnpm run benchmark
+$ pnpm run bench
 ```
 
 ### Testing
@@ -171,6 +171,45 @@ CI runs it against the base commit and adds the table to the job summary. It's
 a report, not a check: if the min+gz size grows significantly, flag it to your
 reviewers so you can discuss whether the addition is worth it.
 
+### Benchmarks
+
+`pnpm bench` runs `bench/compare.bench.mjs` with Vitest's
+[benchmarking](https://vitest.dev/guide/benchmarking) in Node. For each group
+of test fixtures (`generic`, `es6`, `react`, `custom`), it compares:
+
+- `this tree`: `index.js` in your working tree,
+- `published`: the last release (`react-fast-compare-published`, an aliased
+  devDependency), and
+- `fast-deep-equal`: the upstream version we forked from.
+
+A candidate that throws on a group's cases (`fast-deep-equal` on
+`Object.create(null)` objects) is left out of that group, and the test title
+says so.
+
+```sh
+$ pnpm bench
+# One fixture group
+$ pnpm bench -t react
+# In a browser instead of Node (coarser timers)
+$ pnpm exec vitest bench --run --project 'browser (chromium) (bench)'
+```
+
+CI runs it in its own job and adds the tables, with each candidate's
+difference from `published`, to the job summary. It's a report, not a check:
+shared runners are too noisy to gate on. If your change touches the comparison
+code, run it locally before and after, and mention any difference beyond a few
+percent in your PR.
+
+Differences under about 3–5% in a single run are noise. As a calibration:
+`this tree` and `published` run the same code until a source change lands, yet
+they differ by up to ~2% from run to run, more than the ±RME column suggests
+(it only measures variation within a run). Re-run before reading anything into
+a small gap.
+
+After a release, bump `react-fast-compare-published` to it
+(`pnpm add -D react-fast-compare-published@npm:react-fast-compare@<version>`)
+once it's a week old (`minimumReleaseAge`).
+
 ## Before submitting a PR...
 
 ... please make sure that you have done the following:
@@ -179,13 +218,12 @@ reviewers so you can discuss whether the addition is worth it.
 
    ```sh
    $ pnpm run check
-   $ pnpm run benchmark
+   $ pnpm run bench
    ```
 
-2. Confirm we don't have any significant performance regressions (check out `master` for a baseline comparison on _your_ machine).
+2. Confirm we don't have any significant performance regressions (see [benchmarks](#benchmarks)).
 
 3. Confirm you aren't significantly impacting our bundle size (see [size](#size)).
-   - _Org members:_ Update the README's benchmark comparison png using this [internal Google Sheet template](https://docs.google.com/spreadsheets/d/1GuqpO0wgPjQ9usx6sR3t0Y_HTmAdRqjXkSjs3SBsmTc/edit?usp=sharing_eip&ts=5ed1642f).
 
 ## Releasing
 

README.md

@@ -12,10 +12,6 @@ This is a fork of the brilliant
 [fast-deep-equal](https://github.com/epoberezkin/fast-deep-equal) with some
 extra handling for React.
 
-![benchmark chart](https://raw.githubusercontent.com/FormidableLabs/react-fast-compare/master/assets/benchmarking.png 'benchmarking chart')
-
-(Check out the [benchmarking details](#benchmarking-this-library).)
-
 ## Install
 
 ```sh
@@ -74,51 +70,21 @@ be for you!
 
 ## Benchmarking this Library
 
-The absolute values are much less important than the relative differences
-between packages.
-
-Benchmarking source can be found
-[here](https://github.com/FormidableLabs/react-fast-compare/blob/master/benchmark/index.js).
-Each "operation" consists of running all relevant tests. The React benchmark
-uses both the generic tests and the react tests; these runs will be slower
-simply because there are more tests in each operation.
-
-The results below are from a local test on a laptop _(stats last updated 6/2/2020)_:
-
-### Generic Data
-
-```
-react-fast-compare x 177,600 ops/sec ±1.73% (92 runs sampled)
-fast-deep-equal x 184,211 ops/sec ±0.65% (87 runs sampled)
-lodash.isEqual x 39,826 ops/sec ±1.32% (86 runs sampled)
-nano-equal x 176,023 ops/sec ±0.89% (92 runs sampled)
-shallow-equal-fuzzy x 146,355 ops/sec ±0.64% (89 runs sampled)
-  fastest: fast-deep-equal
-```
-
-`react-fast-compare` and `fast-deep-equal` should be the same speed for these
-tests; any difference is just noise. `react-fast-compare` won't be faster than
-`fast-deep-equal`, because it's based on it.
-
-### React and Generic Data
-
-```
-react-fast-compare x 86,392 ops/sec ±0.70% (93 runs sampled)
-fast-deep-equal x 85,567 ops/sec ±0.95% (92 runs sampled)
-lodash.isEqual x 7,369 ops/sec ±1.78% (84 runs sampled)
-  fastest: react-fast-compare,fast-deep-equal
-```
-
-Two of these packages cannot handle comparing React elements, because they
-contain circular reference: `nano-equal` and `shallow-equal-fuzzy`.
-
-### Running Benchmarks
+`react-fast-compare` is based on `fast-deep-equal`, and aims to be as fast on
+generic data while also handling React elements. The benchmark
+([`bench/`](./bench/compare.bench.mjs), Vitest's
+[benchmarking](https://vitest.dev/guide/benchmarking)) compares this code with
+the last published version and with `fast-deep-equal`, over the same test cases
+the tests use:
 
 ```sh
 $ pnpm install
-$ pnpm run benchmark
+$ pnpm bench
 ```
 
+Absolute numbers depend on the machine; compare rows from the same run. CI adds
+the results to the job summary of each run.
+
 ## Differences between this library and `fast-deep-equal`
 
 `react-fast-compare` is based on `fast-deep-equal`, with some additions:

bench/compare.bench.mjs

@@ -0,0 +1,60 @@
+// Self-regression benchmark (`pnpm bench`): this tree vs. the last published
+// version vs. fast-deep-equal, the upstream we forked from. One benchmark per
+// fixture group; each operation compares every case in the group once.
+//
+// A candidate that throws on any case in a group is left out of that group
+// (and named in the test title): timing exceptions would swamp the numbers.
+// Correctness is the tests' job, not this file's.
+import { test } from 'vitest';
+
+import fastDeepEqual from 'fast-deep-equal/es6/react.js';
+import published from 'react-fast-compare-published';
+
+import equal from '../index.js';
+import * as fixtures from '../test/fixtures/index.mjs';
+
+// `published` is an aliased devDependency on the last release; bump it after
+// each release (see CONTRIBUTING).
+const CANDIDATES = {
+  'this tree': equal,
+  published,
+  'fast-deep-equal': fastDeepEqual,
+};
+
+const GROUPS = {
+  generic: fixtures.generic,
+  es6: fixtures.es6,
+  react: fixtures.react,
+  custom: fixtures.custom,
+};
+
+const throws = (fn, cases) =>
+  cases.some(({ value1, value2 }) => {
+    try {
+      fn(value1, value2);
+      return false;
+    } catch {
+      return true;
+    }
+  });
+
+for (const [group, suites] of Object.entries(GROUPS)) {
+  const cases = suites.flatMap((suite) => suite.tests);
+  const candidates = Object.entries(CANDIDATES);
+  const skipped = candidates.filter(([, fn]) => throws(fn, cases));
+  const title = skipped.length
+    ? `${group} (${skipped.map(([name]) => name).join(', ')} skipped: throws)`
+    : group;
+
+  test(title, async ({ bench }) => {
+    await bench.compare(
+      ...candidates
+        .filter((candidate) => !skipped.includes(candidate))
+        .map(([name, fn]) =>
+          bench(name, () => {
+            for (const { value1, value2 } of cases) fn(value1, value2);
+          })
+        )
+    );
+  });
+}

bench/summary-reporter.mjs

@@ -0,0 +1,60 @@
+// Vitest reporter for `pnpm bench`: in GitHub Actions, appends the benchmark
+// results to the job summary as Markdown tables. Elsewhere it does nothing;
+// the default reporter already prints the tables.
+import fs from 'node:fs';
+import { createRequire } from 'node:module';
+
+const BASELINE = 'published';
+const { version } = createRequire(import.meta.url)(
+  'react-fast-compare-published/package.json'
+);
+
+const format = (n, digits = 0) =>
+  n.toLocaleString('en-US', {
+    minimumFractionDigits: digits,
+    maximumFractionDigits: digits,
+  });
+
+const versus = (task, baseline) => {
+  if (!baseline || task === baseline) return '';
+  const diff = (task.throughput.mean / baseline.throughput.mean - 1) * 100;
+  return `${diff >= 0 ? '+' : ''}${diff.toFixed(1)}%`;
+};
+
+const table = (title, { tasks }) => {
+  const baseline = tasks.find((task) => task.name === BASELINE);
+  return [
+    `#### ${title}`,
+    '',
+    `| | ops/s | Mean (µs) | ±RME | Samples | vs. ${BASELINE} |`,
+    '|---|--:|--:|--:|--:|--:|',
+    ...tasks.map(
+      (task) =>
+        `| ${task.name} | ${format(task.throughput.mean)} | ${format(task.latency.mean * 1000, 3)} | ${task.latency.rme.toFixed(2)}% | ${format(task.latency.samplesCount)} | ${versus(task, baseline)} |`
+    ),
+    '',
+  ];
+};
+
+export default class SummaryReporter {
+  sections = [];
+
+  onTestCaseResult(testCase) {
+    const benchmarks = testCase.benchmarks().filter((b) => b.tasks.length);
+    for (const benchmark of benchmarks)
+      this.sections.push(table(testCase.name, benchmark));
+  }
+
+  onTestRunEnd() {
+    const file = process.env.GITHUB_STEP_SUMMARY;
+    if (!file || !this.sections.length) return;
+    const report = [
+      '### Benchmarks',
+      '',
+      `\`${BASELINE}\` is react-fast-compare@${version}. Node ${process.version}; operations per second, higher is better. A report, not a check: CI runners are noisy, so compare locally before drawing conclusions.`,
+      '',
+      ...this.sections.flat(),
+    ].join('\n');
+    fs.appendFileSync(file, report + '\n');
+  }
+}