Repository navigation
Conversation
|
Thank you for your proposal! It is interesting. Linux is the platform, which I use the most, so that I'm propably aware of performance changes. But not so much on Windows and Macos. Does Duel work there too? |
Thanks for considering it!
Duel runs on Linux and macOS. It doesn't run on Windows, because it controls its workers through Unix sockets and process groups. The Perf job here runs on Ubuntu, so it would catch a regression in the shared C code (the decoders, type maps, COPY and so on), but not one that only shows up in a macOS code path, but I could add that to the CI matrix. I’d like to add Windows support to Duel in the future, but that would be a major undertaking. |
36f2f83 to
e65e47a
Compare
Duel (https://github.com/sferik/duel-ruby) runs the benchmarks in duel/ on the branch and on master side by side, and fails a pull request that is slower or allocates more. The 44 benchmarks cover queries over a socket to a PostgreSQL server of each worker's own (exec, exec_params, exec_prepared, COPY in and out), reading results in memory (values, getvalue, each, each_row, tuples, field names), the type maps, the text and binary coders on their own, escaping, and parsing connection parameters. Each version compiles its own extension (c.build). The duel-postgresql plugin needs a released pg, so Duel is in an optional Gemfile group. Checked by comparing master with itself three times, and with planted regressions: the Integer decoder without its fast path failed 6 benchmarks on instructions, and an extra rb_str_dup in the String decoder failed 13 on allocations.
pg's code paths differ by system, and Linux's was the only one Duel measured. The Perf workflow runs on Linux, macOS and Windows now, each saying what it found in a report of its own, and the Perf comment workflow comments with all three. On Linux and macOS, it calls Duel 0.2.0's reusable workflow, so the copy of its steps is gone. It grants contents: write, which GitHub checks every job of the reusable workflow against, though none of the jobs that write is turned on here, and the Duel job still only reads. The reusable workflow's steps are Unix shell's, so the Windows job runs the same ones itself, in bash, with what pg's build needs there as source-gem.yml has it (MSYS2's gcc, and PostgreSQL's binaries, for libpq). Its servers are the runner's own PostgreSQL's, whose initdb drops the rights of the runner's user, an administrator, as postgres needs: the binaries' doesn't. The Perf comment workflow calls Duel's reusable one too.
|
@larskanis I've updated this to Duel 0.2.0, which I released today. It now measures On Linux and macOS, the workflow calls Duel's reusable workflow, so the ~100 lines of copied steps are gone.. The Windows job has its own steps, in bash, and sets up the build the same way Thanks so much for suggesting this improvement to Duel! Could you please take another look when you have a chance? |
|
Looks good after a quick check! Thank you very much for all your efforts! The @cbandy , @ged Are you more familiar with github's permissions and automations than I am? Could you have a look at this PR? |
|
This looks awesome. I'll dig into it some more, but from first glance I'm a fan. I have some other projects I could use Duel on as well! |
|
Any and all feedback is welcome. I use Duel on most of my projects but I understand that other projects may have different needs than my own, which is a big part of why I opened this PR. I’m also happy to help setup Duel for some of your other projects. |
Duel 0.3.0's reusable workflow runs on Windows (its steps in bash, with setup-after-ruby for what needs Ruby's devkit), so the Windows job is a call of it like Linux's and macOS's, in place of a copy of its steps: setup downloads PostgreSQL's binaries, for libpq, and names the runner's own postgres, in PGBIN, for the servers (DUEL_POSTGRESQL_SERVER, as duel-postgresql 0.3.0 names what was DUEL_POSTGRESQL_BIN, a directory), and setup-after-ruby installs MSYS2's gcc and puts the devkit's directories, then libpq's, on the PATH. Each Duel job grants contents: read alone, now that the jobs that write for a push are a reusable workflow of Duel's own, which each platform's push calls, with its report. And the Duel jobs no longer take accept-label, as Duel reads it from .duel.rb itself. The Gemfile's optional group asks for Duel 0.3.0 and duel-postgresql 0.3.0. .duel.rb leaves the build (bundle exec rake compile, which Duel 0.3.0 runs by default for a project with ext/ and a Rakefile) and the accept label (duel: accept, its default too) to Duel, showing each commented out, as the .duel.rb duel init writes now does. And the README shows Duel's badge with the other build badges, linked to the Perf workflow's runs.
|
I released Duel 0.3.0 today and updated this PR for it (3cf2c9f). The biggest change is the Windows job. Duel's reusable workflow now runs on Windows too, so the Windows job is a call of it, like the Linux and macOS ones, instead of about a hundred lines copied from it. Its Each Duel job now grants only The rest is small. The Gemfile's optional Before pushing, I ran the workflow in a PR in my fork whose base is this PR's base. All three platforms compared pg's 44 benchmarks, and none regressed. The jobs took 3m25s on Linux, 5m13s on macOS and 8m56s on Windows. Windows is about two minutes slower than the old job's 6m40s, because with |
Each worker's PostgreSQL server takes a System V shared memory segment, and macOS allows 32 in all. With Duel's 12 workers of each version, 24 servers, and those that build their clusters, are at that limit, and a server that can't make its segment doesn't start: the macOS job failed before measuring with "could not create shared memory segment: No space left on device": https://github.com/ged/ruby-pg/actions/runs/44777546632/job/136595330712 With 6 workers of each version (DUEL_PROCESSES), the macOS job runs 12 servers, well under the limit.
This adds Duel, a gem I wrote to catch performance regressions in CI. On every pull request it runs the benchmarks in
duel/againstmasterand the branch side by side, in alternating worker processes, and fails the build if the branch is slower or allocates more. It judges CPU time and instruction counts with confidence intervals, and counts allocations exactly. When something regresses, it profiles both versions with StackProf and says where the extra time went.There are 44 benchmarks, and most of them stay away from the server on purpose, so that what's timed is the driver and not PostgreSQL. Results are fetched once and then read in memory with
values,getvalue,each(with string and symbol keys),each_row,tuple,column_valuesand the field name and type lookups. The type maps decode the same 100-row result throughBasicTypeMapForResultsin text and binary format and through aTypeMapByColumn, andBasicTypeMapForQueriespicks encoders for a mixed set of parameters and feedsembed_params. The coders get benchmarks of their own, 100 fixed values a call: the Integer, Float, Date and both Timestamp text decoders (with the recent work on those in mind), integer and quoted-string arrays, Bytea, the matching encoders and the binary Int8, Float8 and TimestampUtc coders. Then there'sescape_string,escape_literal,quote_identand the bytea round trip, the CopyRow encoder and decoder, andparse_connect_argswith a URL, a keyword string and a Hash. The rest do go to a server:exec,exec_params,exec_prepared, 100 rows with and without a result type map, and COPY in and out with coders. They use the duel-postgresql plugin, which starts a PostgreSQL server for each worker on a Unix socket and also judges each benchmark on the queries and rows the server ran.The tricky part was pg being the plugin's own dependency. duel-postgresql needs a released pg, and it used to require it as
.duel.rbloaded, before Duel loaded the version. So every worker was quietly measuring pg 1.7.0 from the bundle instead of either version. I checked this with a probe that wrote out each worker's$LOADED_FEATURES. I fixed that in duel-postgresql 0.1.4, which requires pg only once the worker's version has loaded, so.duel.rbneeds no workaround. With it, the probe showed each worker with its ownpg.rbandpg_ext.bundleand nothing else. The released pg also brokerake speconce it was in the bundle, because its fat binary gem has alib/3.4/pg_extthatpg.rbtries first. So Duel lives in an optionalduelgroup in the Gemfile, with the released pg as the source gem, andbundle installand the specs are exactly as they were unless you opt in. With the group installed the specs still pass (1163 examples, 0 failures).A full run takes 20 to 25 seconds on my laptop, and the Duel job took about two and a half minutes on GitHub Actions, most of it setup and compiling both versions.
The Duel job only reads. The comment on a pull request comes from a second workflow,
perf-comment.yml, which runs none of the pull request's code, onlyduel commentfrom Duel's gem, and hides Duel's earlier comments as outdated so the newest one is the one in view. If a change is slower on purpose, aduel: acceptlabel lets it through, and adding it runs the Duel job again.