Repository navigation
test: run test modules headless unless they need a workbench - #1622
joaodinissf wants to merge 1 commit into
Conversation
b66ed24 to
402b761
Compare
402b761 to
a28f0a9
Compare
Test modules use Tycho's default headless harness. A module that needs a workbench opts in by setting tycho.surefire.useUIHarness=true, and fails loudly otherwise, so a new module does not start a workbench by accident. Seven modules need one and opt in: check.core.test, check.ui.test, test.ui.test, checkcfg.core.test, check.test.runtime.tests, sample.helloworld.ui.test and xtext.ui.test. Only these take the Surefire reactor lock (test.reactorConcurrencyLevel=1), since a workbench could take focus from a SWTBot test in another one. The other seven have no workbench and run in parallel under -T. Co-Authored-By: Claude Opus 5.5 <[email protected]>
a28f0a9 to
203076b
Compare
| <packaging>eclipse-test-plugin</packaging> | ||
|
|
||
| <properties> | ||
| <tycho.surefire.useUIHarness>true</tycho.surefire.useUIHarness> |
There was a problem hiding this comment.
something is off here, this is a core.test plugin, so it should not need UI. Maybe some test need a bit of rework to make this possible?
There was a problem hiding this comment.
You are right, both modules are UI tests in practice. I looked into what it takes to run them headless:
- Every headless failure has one cause: the Check UI injector cannot be created without a workbench ("No implementation for IWorkbench was bound", requested by Xtext's
MarkerResolutionGenerator). check.core.test: 9 classes use the UI injector directly, andAbstractCheckTestCaseuses it for 6 more that need a plug-in project in the workspace. Headless, 69 of 72 tests error.checkcfg.core.test: resolving a Check catalog loads the.checkresources through the Check UI, so 6 of 11 tests fail headless.
Proposed rework, all in test bundles:
- Switch 6
check.core.testclasses to the runtime injector. I tried this locally: their 63 tests pass headless. - Move
AbstractCheckTestCase,AbstractCheckGenerationTestCaseand their 6 subclasses tocheck.ui.test.check.core.testis then headless and loses the opt-in. - Rename
checkcfg.core.testtocheckcfg.ui.test, since its tests cover content assist, a workspace project and catalogs contributed by plug-ins.
I would do this in a follow-up PR instead of here. This PR only changes how the harness is configured, while the rework moves about 1100 lines of tests between bundles and deserves its own review. The follow-up would remove both opt-ins. Would that work for you, or do you prefer the rework to land first?
This also explains the ConcurrentModificationException listed under "Known limits": it only happens headless, when the Check UI injector fails while CheckCfgUtil.getAllLanguages() iterates the registry. I will update the description.
Posted by Claude Code on behalf of @joaodinissf.
| <packaging>eclipse-test-plugin</packaging> | ||
|
|
||
| <properties> | ||
| <tycho.surefire.useUIHarness>true</tycho.surefire.useUIHarness> |
There was a problem hiding this comment.
something is off here, this is a core.test plugin, so it should not need UI. Maybe some test need a bit of rework to make this possible?
There was a problem hiding this comment.
Answered in the thread on check.core.test/pom.xml; the plan there covers this module too (rename to checkcfg.ui.test).
Posted by Claude Code on behalf of @joaodinissf.
|
@rubenporras the follow-up for your two comments is up as a draft: #1640.
This differs from my reply above: #1640 builds on this PR's commit, so it needs this one merged first; I'll rebase it onto Posted by Claude Code on behalf of @joaodinissf. |
Summary
Test modules now run under Tycho's default headless harness, so a module only starts a workbench if it asks for one.
tycho.surefire.useUIHarness=trueandtest.reactorConcurrencyLevel=1, so under-Tthey take turns, which keeps the no-overlap guaranteemasterhas today:check.core.test,check.ui.test,test.ui.test,checkcfg.core.test,check.test.runtime.tests,sample.helloworld.ui.test,xtext.ui.test.check.core.testandcheckcfg.core.testare on this list because their tests use the Check UI injector or create workspace projects. test: run check.core.test headless and rename checkcfg.core.test to checkcfg.ui.test #1640 makescheck.core.testheadless and renamescheckcfg.core.testtocheckcfg.ui.test.test,typesystem.test,xtext.export.test,check.runtime.core.test,xtext.test,xtext.generator.test,xtext.format.test.PlatformUI.getWorkbench()throws, and the Check UI injector can't be created without anIWorkbench.AGENTS.mdexplains how to opt in.Performance
This PR makes builds a little faster than today's
master, and gives back part of what one runtime per test module cost.CI, on Linux (the build step of
maven-verify, which runs serially):mastertoday (since Nov 16)CI times vary a lot, and 3 runs are few; the 681 s run was on this PR's earlier base.
Local, on macOS with Temurin 21. Load on the machine varied a lot between runs, so only large, consistent differences mean anything.
mastertoday-T 3C, older base-T 3C, later basesValidation
./mvnw -U -T 3C clean verify checkstyle:check pmd:pmd pmd:cpd pmd:check pmd:cpd-check spotbugs:check -f ddk-parent/pom.xml --batch-mode --fail-at-endpassed locally on this change on an earliermaster. This commit sits on a newermaster, so CI is its validation.check.core.test. No headless test is skipped or aborted.Known limits
checkcfg.core.testcan't run headless. Besides the injector errors, its validation fails there with aConcurrentModificationException, deterministically:CheckCfgUtil.getAllLanguages()iterates the resource service provider registry, and during that iteration Xtext removes the entry of a language whose UI injector can't be created without a workbench. With a workbench those injectors are created and nothing is removed, so the exception doesn't occur in this PR, where the module keeps its opt-in.🤖 Generated with Claude Code