Skip to content

Add Duel performance regression guards - #771

Open
sferik wants to merge 4 commits into
ged:masterfrom
sferik:add-duel
Open

sferik wants to merge 4 commits into
ged:masterfrom
sferik:add-duel

Conversation

@sferik

@sferik sferik commented Nov 10, 2026 •

Copy link
Copy Markdown

This adds Duel, a gem I wrote to catch performance regressions in CI. On every pull request it runs the benchmarks in duel/ against master and the branch side by side, in alternating worker processes, and fails the build if the branch is slower or allocates more. It judges CPU time and instruction counts with confidence intervals, and counts allocations exactly. When something regresses, it profiles both versions with StackProf and says where the extra time went.

There are 44 benchmarks, and most of them stay away from the server on purpose, so that what's timed is the driver and not PostgreSQL. Results are fetched once and then read in memory with values, getvalue, each (with string and symbol keys), each_row, tuple, column_values and the field name and type lookups. The type maps decode the same 100-row result through BasicTypeMapForResults in text and binary format and through a TypeMapByColumn, and BasicTypeMapForQueries picks encoders for a mixed set of parameters and feeds embed_params. The coders get benchmarks of their own, 100 fixed values a call: the Integer, Float, Date and both Timestamp text decoders (with the recent work on those in mind), integer and quoted-string arrays, Bytea, the matching encoders and the binary Int8, Float8 and TimestampUtc coders. Then there's escape_string, escape_literal, quote_ident and the bytea round trip, the CopyRow encoder and decoder, and parse_connect_args with a URL, a keyword string and a Hash. The rest do go to a server: exec, exec_params, exec_prepared, 100 rows with and without a result type map, and COPY in and out with coders. They use the duel-postgresql plugin, which starts a PostgreSQL server for each worker on a Unix socket and also judges each benchmark on the queries and rows the server ran.

The tricky part was pg being the plugin's own dependency. duel-postgresql needs a released pg, and it used to require it as .duel.rb loaded, before Duel loaded the version. So every worker was quietly measuring pg 1.7.0 from the bundle instead of either version. I checked this with a probe that wrote out each worker's $LOADED_FEATURES. I fixed that in duel-postgresql 0.1.4, which requires pg only once the worker's version has loaded, so .duel.rb needs no workaround. With it, the probe showed each worker with its own pg.rb and pg_ext.bundle and nothing else. The released pg also broke rake spec once it was in the bundle, because its fat binary gem has a lib/3.4/pg_ext that pg.rb tries first. So Duel lives in an optional duel group in the Gemfile, with the released pg as the source gem, and bundle install and the specs are exactly as they were unless you opt in. With the group installed the specs still pass (1163 examples, 0 failures).

A full run takes 20 to 25 seconds on my laptop, and the Duel job took about two and a half minutes on GitHub Actions, most of it setup and compiling both versions.

The Duel job only reads. The comment on a pull request comes from a second workflow, perf-comment.yml, which runs none of the pull request's code, only duel comment from Duel's gem, and hides Duel's earlier comments as outdated so the newest one is the one in view. If a change is slower on purpose, a duel: accept label lets it through, and adding it runs the Duel job again.

@larskanis

Copy link
Copy Markdown
Collaborator

Thank you for your proposal! It is interesting. Linux is the platform, which I use the most, so that I'm propably aware of performance changes. But not so much on Windows and Macos. Does Duel work there too?

Comment thread .github/workflows/perf-comment.yml Resolved
@sferik

sferik commented Nov 11, 2026 •

Copy link
Copy Markdown
Author

Thank you for your proposal! It is interesting.

Thanks for considering it!

Linux is the platform, which I use the most, so that I'm propably aware of performance changes. But not so much on Windows and Macos. Does Duel work there too?

Duel runs on Linux and macOS. It doesn't run on Windows, because it controls its workers through Unix sockets and process groups. The Perf job here runs on Ubuntu, so it would catch a regression in the shared C code (the decoders, type maps, COPY and so on), but not one that only shows up in a macOS code path, but I could add that to the CI matrix. I’d like to add Windows support to Duel in the future, but that would be a major undertaking.

@sferik
sferik force-pushed the add-duel branch 2 times, most recently from 36f2f83 to e65e47a Compare November 12, 2026 20:35
Duel (https://github.com/sferik/duel-ruby) runs the benchmarks in duel/
on the branch and on master side by side, and fails a pull request that
is slower or allocates more. The 44 benchmarks cover queries over a
socket to a PostgreSQL server of each worker's own (exec, exec_params,
exec_prepared, COPY in and out), reading results in memory (values,
getvalue, each, each_row, tuples, field names), the type maps, the text
and binary coders on their own, escaping, and parsing connection
parameters.

Each version compiles its own extension (c.build). The duel-postgresql
plugin needs a released pg, so Duel is in an optional Gemfile group.

Checked by comparing master with itself three times, and with planted
regressions: the Integer decoder without its fast path failed 6
benchmarks on instructions, and an extra rb_str_dup in the String
decoder failed 13 on allocations.
pg's code paths differ by system, and Linux's was the only one Duel
measured. The Perf workflow runs on Linux, macOS and Windows now, each
saying what it found in a report of its own, and the Perf comment
workflow comments with all three.

On Linux and macOS, it calls Duel 0.2.0's reusable workflow, so the
copy of its steps is gone. It grants contents: write, which GitHub
checks every job of the reusable workflow against, though none of the
jobs that write is turned on here, and the Duel job still only reads.
The reusable workflow's steps are Unix shell's, so the Windows job runs
the same ones itself, in bash, with what pg's build needs there as
source-gem.yml has it (MSYS2's gcc, and PostgreSQL's binaries, for
libpq). Its servers are the runner's own PostgreSQL's, whose initdb
drops the rights of the runner's user, an administrator, as postgres
needs: the binaries' doesn't. The Perf comment workflow calls Duel's
reusable one too.
@sferik

sferik commented Nov 16, 2026

Copy link
Copy Markdown
Author

@larskanis I've updated this to Duel 0.2.0, which I released today. It now measures pg on Linux, macOS and Windows.

On Linux and macOS, the workflow calls Duel's reusable workflow, so the ~100 lines of copied steps are gone.. The Windows job has its own steps, in bash, and sets up the build the same way source-gem.yml does: MSYS2's gcc, and PostgreSQL's binaries for libpq.

Thanks so much for suggesting this improvement to Duel!

Could you please take another look when you have a chance?

@larskanis

Copy link
Copy Markdown
Collaborator

Looks good after a quick check! Thank you very much for all your efforts! The mswin CI failure is due to ruby/setup-ruby#971 and so unrelated.

@cbandy , @ged Are you more familiar with github's permissions and automations than I am? Could you have a look at this PR?

@ged

ged commented Nov 18, 2026

Copy link
Copy Markdown
Owner

This looks awesome. I'll dig into it some more, but from first glance I'm a fan. I have some other projects I could use Duel on as well!

@sferik

sferik commented Nov 18, 2026

Copy link
Copy Markdown
Author

Any and all feedback is welcome. I use Duel on most of my projects but I understand that other projects may have different needs than my own, which is a big part of why I opened this PR. I’m also happy to help setup Duel for some of your other projects.

Duel 0.3.0's reusable workflow runs on Windows (its steps in bash,
with setup-after-ruby for what needs Ruby's devkit), so the Windows
job is a call of it like Linux's and macOS's, in place of a copy of
its steps: setup downloads PostgreSQL's binaries, for libpq, and
names the runner's own postgres, in PGBIN, for the servers
(DUEL_POSTGRESQL_SERVER, as duel-postgresql 0.3.0 names what was
DUEL_POSTGRESQL_BIN, a directory), and setup-after-ruby installs
MSYS2's gcc and puts the devkit's directories, then libpq's, on the
PATH.

Each Duel job grants contents: read alone, now that the jobs that
write for a push are a reusable workflow of Duel's own, which each
platform's push calls, with its report. And the Duel jobs no longer
take accept-label, as Duel reads it from .duel.rb itself. The
Gemfile's optional group asks for Duel 0.3.0 and duel-postgresql
0.3.0.

.duel.rb leaves the build (bundle exec rake compile, which Duel 0.3.0
runs by default for a project with ext/ and a Rakefile) and the accept
label (duel: accept, its default too) to Duel, showing each commented
out, as the .duel.rb duel init writes now does. And the README shows
Duel's badge with the other build badges, linked to the Perf
workflow's runs.
@sferik

sferik commented Nov 20, 2026

Copy link
Copy Markdown
Author

I released Duel 0.3.0 today and updated this PR for it (3cf2c9f).

The biggest change is the Windows job. Duel's reusable workflow now runs on Windows too, so the Windows job is a call of it, like the Linux and macOS ones, instead of about a hundred lines copied from it. Its setup downloads PostgreSQL's binaries (for libpq) and points duel-postgresql at the runner's own postgres, in PGBIN, for each worker's server. Its setup-after-ruby installs MSYS2's gcc and puts the devkit's directories, then libpq's, on the PATH, as source-gem.yml does. duel-postgresql 0.3.0 takes the server's program as DUEL_POSTGRESQL_SERVER, in place of DUEL_POSTGRESQL_BIN, its directory.

Each Duel job now grants only contents: read. Duel 0.3.0 moves the jobs that write for a push (commenting on the commit and marking it passed) into a reusable workflow of their own, and each platform has a call of it that runs only for a push, with its own report. So the jobs that run a branch's code can't write to the repository at all.

The rest is small. The Gemfile's optional duel group asks for Duel 0.3.0 and duel-postgresql 0.3.0. The Duel jobs no longer take accept-label, since Duel reads it from .duel.rb. And .duel.rb leaves the build (bundle exec rake compile, which Duel now runs by default for a project with ext/ and a Rakefile) and the label (duel: accept) to Duel's defaults, commented out. I also added Duel's badge to the README's build badges, linked to the Perf workflow's runs. I'm happy to take it out if you'd rather not have it.

Before pushing, I ran the workflow in a PR in my fork whose base is this PR's base. All three platforms compared pg's 44 benchmarks, and none regressed. The jobs took 3m25s on Linux, 5m13s on macOS and 8m56s on Windows. Windows is about two minutes slower than the old job's 6m40s, because with setup-after-ruby the reusable workflow doesn't cache the bundle, so it installs it each time.

Each worker's PostgreSQL server takes a System V shared memory
segment, and macOS allows 32 in all. With Duel's 12 workers of each
version, 24 servers, and those that build their clusters, are at that
limit, and a server that can't make its segment doesn't start: the
macOS job failed before measuring with "could not create shared memory
segment: No space left on device":
https://github.com/ged/ruby-pg/actions/runs/44777546632/job/136595330712

With 6 workers of each version (DUEL_PROCESSES), the macOS job runs 12
servers, well under the limit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants