[sferik]

This adds Duel, a gem I wrote to catch performance regressions in CI. On every pull request it runs the benchmarks in duel/ against master and the branch side by side, and fails the build if the branch is slower or allocates more.

The 20 benchmarks cover what http.rb does with form data on every request that has a body: urlencoding small, large, nested and multibyte params, building multipart bodies with text fields and in-memory files, and reading them back with to_s, in 16 KB chunks the way http.rb streams them, with rewind and with the content length. Every multipart body uses a fixed boundary.

To check that it actually catches things, I added a .dup to DefaultEncoder.escape, and Duel failed all five urlencoding benchmarks. Some allocation-free work in the CompositeIO#read_chunks loop failed the six reading benchmarks at 2.3 to 3.7 times the CPU time. Comparing master with itself passed three times in a row.

You can run it locally with bundle exec rake duel.

Duel requires Ruby 3.4, so the Gemfile only adds it on 3.4 and newer, and the Rake task is skipped when it isn't installed. Everything else still works on older Rubies.

[github-actions]

Duel passed: none of 20 benchmarks slower

• Baseline: 664bf90, where HEAD left origin/master
• Candidate: 66f36ee

20 benchmarks in 12s. A timing is slower when a 99.0% confidence interval for its ratio, the candidate's to the baseline's, is above 1.02.

Across the board

• cpu 0.999 [0.993, 1.002] same (limit 1.0050, 20 benchmarks)

20 benchmarks unchanged

Benchmark	Timings (ratio [interval])
multipart: build a few text fields	cpu 0.998 [0.624, 1.009]
multipart: build 200 text fields	cpu 1.000 [0.991, 1.010]
multipart: build with an array of values	cpu 0.995 [0.990, 1.002]
multipart: build with a file	cpu 0.995 [0.974, 1.003]
multipart: build with 20 files	cpu 0.994 [0.880, 1.009]
multipart: to_s a few text fields	cpu 1.001 [0.989, 1.005]
multipart: to_s 200 text fields	cpu 1.000 [0.994, 1.018]
multipart: to_s a 16 KB file	cpu 0.995 [0.985, 1.011]
multipart: read a 64 KB file in 16 KB chunks	cpu 0.993 [0.980, 1.004]
multipart: read 200 fields in 16 KB chunks	cpu 0.999 [0.938, 1.017]
multipart: read in small chunks	cpu 1.006 [0.991, 1.017]
multipart: rewind 200 fields	cpu 1.004 [0.991, 1.014]
multipart: content length of 200 fields	cpu 1.001 [0.982, 1.020]
multipart: content length with a file	cpu 0.985 [0.912, 1.007]
urlencode: a small form	cpu 0.993 [0.982, 1.006]
urlencode: a large form of 200 fields	cpu 0.988 [0.971, 1.008]
urlencode: nested hashes and arrays	cpu 1.002 [0.995, 1.013]
urlencode: pairs with repeated keys	cpu 0.994 [0.983, 1.011]
urlencode: values with multibyte characters	cpu 0.999 [0.979, 1.016]
urlencode: read a large body	cpu 1.002 [0.984, 1.019]

[sferik]

I updated the Perf workflow here after a review of the same workflow in another project. The Duel job now only reads, and its checkout keeps no credentials. A separate Comment job, which checks nothing out and runs none of the branch's code, posts the report. .duel.rb also watches duel/, .duel.rb and the workflow, so a pull request that changes only the benchmarks is compared too.

[sferik]

I updated this to Duel 0.1.1. The Perf workflow now only comments when Duel fails, posts a new comment instead of editing the old one, and the report is much shorter.

[github-actions]

🟢 Duel: 100% of benchmarks passed

Compared 79752ea with 664bf90 (where HEAD left origin/master).

Details: 20 benchmarks in 17s, 2 unsettled

Benchmark	Verdict	Timings (ratio [interval])
multipart: to_s a 16 KB file	unsettled	cpu 1.006 [0.993, 1.022] undecided
multipart: rewind 200 fields	unsettled	cpu 1.001 [0.984, 1.023] undecided
multipart: build a few text fields	same	cpu 0.995 [0.942, 1.015]
multipart: build 200 text fields	same	cpu 0.996 [0.979, 1.012]
multipart: build with an array of values	same	cpu 0.999 [0.992, 1.011]
multipart: build with a file	same	cpu 0.993 [0.982, 1.010]
multipart: build with 20 files	same	cpu 1.002 [0.975, 1.019]
multipart: to_s a few text fields	same	cpu 0.997 [0.985, 1.018]
multipart: to_s 200 text fields	same	cpu 0.985 [0.975, 0.997]
multipart: read a 64 KB file in 16 KB chunks	same	cpu 1.005 [0.997, 1.016]
multipart: read 200 fields in 16 KB chunks	same	cpu 0.990 [0.970, 1.019]
multipart: read in small chunks	same	cpu 0.991 [0.528, 1.012]
multipart: content length of 200 fields	same	cpu 0.993 [0.955, 1.002]
multipart: content length with a file	same	cpu 0.991 [0.974, 1.008]
urlencode: a small form	same	cpu 0.998 [0.995, 1.004]
urlencode: a large form of 200 fields	same	cpu 0.996 [0.987, 1.009]
urlencode: nested hashes and arrays	same	cpu 1.011 [1.004, 1.020]
urlencode: pairs with repeated keys	same	cpu 0.997 [0.981, 1.009]
urlencode: values with multibyte characters	same	cpu 1.004 [0.982, 1.018]
urlencode: read a large body	same	cpu 0.999 [0.991, 1.019]

An unsettled timing was too close to the limit to settle. It doesn't fail the run, but one that's unsettled in run after run is noisy.

Across the board:

• cpu 0.997 [0.991, 1.004] same (limit 1.0050, 20 benchmarks)

A timing is slower when a 99.0% confidence interval for its ratio, the candidate's to the baseline's, is above 1.02. How Duel decides