Skip to content

test: run test modules headless unless they need a workbench - #1622

Open
joaodinissf wants to merge 1 commit into
masterfrom
headless-harness-step-2
Open

joaodinissf wants to merge 1 commit into
masterfrom
headless-harness-step-2

Conversation

@joaodinissf

@joaodinissf joaodinissf commented Nov 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Test modules now run under Tycho's default headless harness, so a module only starts a workbench if it asks for one.

  • Seven modules need a workbench and opt in. They set tycho.surefire.useUIHarness=true and test.reactorConcurrencyLevel=1, so under -T they take turns, which keeps the no-overlap guarantee master has today: check.core.test, check.ui.test, test.ui.test, checkcfg.core.test, check.test.runtime.tests, sample.helloworld.ui.test, xtext.ui.test.
  • Seven modules run headless, with no lock: test, typesystem.test, xtext.export.test, check.runtime.core.test, xtext.test, xtext.generator.test, xtext.format.test.
  • A new test module runs headless by default and fails loudly if it needs a workbench: PlatformUI.getWorkbench() throws, and the Check UI injector can't be created without an IWorkbench. AGENTS.md explains how to opt in.

Performance

This PR makes builds a little faster than today's master, and gives back part of what one runtime per test module cost.

CI, on Linux (the build step of maven-verify, which runs serially):

Code Runs Build step
Before one runtime per module (Nov 11–13) 12 median about 615 s, range 512–850 s
master today (since Nov 16) 12 median about 735 s, range 658–964 s
This PR 3 525 s, 631 s, 681 s

CI times vary a lot, and 3 runs are few; the 681 s run was on this PR's earlier base.

Local, on macOS with Temurin 21. Load on the machine varied a lot between runs, so only large, consistent differences mean anything.

Build master today This PR
Serial Surefire time, measured back to back 172 s 143 s
Tests only, serial 187–189 s 179 s, 206 s
Full goal list, -T 3C, older base 245–272 s 213–233 s
Full goal list, -T 3C, later bases 240 s, 277 s 268 s, 296 s
  • Later bases: the gap is all in modules this PR doesn't touch, such as compilation and static analysis, under load averages of 55–155. The test modules took the same time or up to 46 s less.
  • The lock: skipping it in the headless modules saves nothing measurable today: 213–233 s against 223–228 s when all 14 queue, on the older base. The build's dependency order runs most headless modules before the workbench ones, which then set the end of the build. They still skip it because they don't need it, so future headless tests can run in parallel.

Validation

  • ./mvnw -U -T 3C clean verify checkstyle:check pmd:pmd pmd:cpd pmd:check pmd:cpd-check spotbugs:check -f ddk-parent/pom.xml --batch-mode --fail-at-end passed locally on this change on an earlier master. This commit sits on a newer master, so CI is its validation.
  • In that local run, all 490 tests matched a run of its base, test by test, with the same statuses and the same 2 skipped, both in check.core.test. No headless test is skipped or aborted.
  • At most 1 workbench JVM ran at a time. Headless JVMs ran alongside it for about 11–20 s per build.
  • CI passes on Linux on this commit, including the headless modules.

Known limits

  • Two runs with headless modules outside the lock can't rule out a rare flake.
  • checkcfg.core.test can't run headless. Besides the injector errors, its validation fails there with a ConcurrentModificationException, deterministically: CheckCfgUtil.getAllLanguages() iterates the resource service provider registry, and during that iteration Xtext removes the entry of a language whose UI injector can't be created without a workbench. With a workbench those injectors are created and nothing is removed, so the exception doesn't occur in this PR, where the module keeps its opt-in.

🤖 Generated with Claude Code

@joaodinissf
joaodinissf added this pull request to stack #1623 November 6, 2026 10:43
@joaodinissf
joaodinissf force-pushed the headless-harness-step-2 branch from b66ed24 to 402b761 Compare November 15, 2026 19:07
Base automatically changed from test-discovery to master November 16, 2026 08:25
@joaodinissf
joaodinissf force-pushed the headless-harness-step-2 branch from 402b761 to a28f0a9 Compare November 16, 2026 08:25
Test modules use Tycho's default headless harness. A module that needs a
workbench opts in by setting tycho.surefire.useUIHarness=true, and fails
loudly otherwise, so a new module does not start a workbench by accident.

Seven modules need one and opt in: check.core.test, check.ui.test,
test.ui.test, checkcfg.core.test, check.test.runtime.tests,
sample.helloworld.ui.test and xtext.ui.test. Only these take the Surefire
reactor lock (test.reactorConcurrencyLevel=1), since a workbench could take
focus from a SWTBot test in another one. The other seven have no workbench
and run in parallel under -T.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@joaodinissf
joaodinissf force-pushed the headless-harness-step-2 branch from a28f0a9 to 203076b Compare November 20, 2026 04:50
@joaodinissf
joaodinissf marked this pull request as ready for review November 20, 2026 04:50
<packaging>eclipse-test-plugin</packaging>

<properties>
<tycho.surefire.useUIHarness>true</tycho.surefire.useUIHarness>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

something is off here, this is a core.test plugin, so it should not need UI. Maybe some test need a bit of rework to make this possible?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, both modules are UI tests in practice. I looked into what it takes to run them headless:

  • Every headless failure has one cause: the Check UI injector cannot be created without a workbench ("No implementation for IWorkbench was bound", requested by Xtext's MarkerResolutionGenerator).
  • check.core.test: 9 classes use the UI injector directly, and AbstractCheckTestCase uses it for 6 more that need a plug-in project in the workspace. Headless, 69 of 72 tests error.
  • checkcfg.core.test: resolving a Check catalog loads the .check resources through the Check UI, so 6 of 11 tests fail headless.

Proposed rework, all in test bundles:

  1. Switch 6 check.core.test classes to the runtime injector. I tried this locally: their 63 tests pass headless.
  2. Move AbstractCheckTestCase, AbstractCheckGenerationTestCase and their 6 subclasses to check.ui.test. check.core.test is then headless and loses the opt-in.
  3. Rename checkcfg.core.test to checkcfg.ui.test, since its tests cover content assist, a workspace project and catalogs contributed by plug-ins.

I would do this in a follow-up PR instead of here. This PR only changes how the harness is configured, while the rework moves about 1100 lines of tests between bundles and deserves its own review. The follow-up would remove both opt-ins. Would that work for you, or do you prefer the rework to land first?

This also explains the ConcurrentModificationException listed under "Known limits": it only happens headless, when the Check UI injector fails while CheckCfgUtil.getAllLanguages() iterates the registry. I will update the description.

Posted by Claude Code on behalf of @joaodinissf.

<packaging>eclipse-test-plugin</packaging>

<properties>
<tycho.surefire.useUIHarness>true</tycho.surefire.useUIHarness>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

something is off here, this is a core.test plugin, so it should not need UI. Maybe some test need a bit of rework to make this possible?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Answered in the thread on check.core.test/pom.xml; the plan there covers this module too (rename to checkcfg.ui.test).

Posted by Claude Code on behalf of @joaodinissf.

@joaodinissf

Copy link
Copy Markdown
Collaborator Author

@rubenporras the follow-up for your two comments is up as a draft: #1640.

  • check.core.test runs headless there, with no workbench opt-in. Six classes move to the runtime injector, and the three workspace-based classes move to check.ui.test.
  • checkcfg.core.test is renamed to checkcfg.ui.test and keeps its opt-in, since all 11 of its tests depend on registrations from the UI bundles.
  • The same 490 tests run with the same results, and the split becomes 8 headless modules and 6 with a workbench.

This differs from my reply above: checkcfg keeps its opt-in after the rename, and fewer classes had to move.

#1640 builds on this PR's commit, so it needs this one merged first; I'll rebase it onto master right after. Could you take another look at this one with that in mind?

Posted by Claude Code on behalf of @joaodinissf.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants