Repository navigation
Fail the build when storm-client or storm-server imports relocated packages directly - #9286
dragonknightk wants to merge 2 commits into
Conversation
…ckages directly Closes issue 7641 ## What is the purpose of the change storm-client and storm-server now fail validate when their main code imports one of the relocated packages of storm-shaded-deps. Such an import still compiles if another dependency brings in the original jar, so the module silently depends on an unshaded copy. Six imports already did this: - storm-client Utils used org.apache.commons.io.IOUtils, which reaches storm-client through commons-compress. - storm-server EquivalenceUtils, SchedulerConfigCache, WorkerTokenManager, CgroupManager and Slot used com.google.common, which reaches storm-server only through the optional auto-service dependency. They now use the shaded classes, like the other imports in the same modules. Each module gets its own checkstyle execution with an IllegalImport rule, so the shared storm_checkstyle.xml stays unchanged for other modules. A comment next to the relocations in storm-shaded-deps/pom.xml asks to update both lists together. The two lists differ: - storm-client bans all relocated packages. - commons-io, commons-lang3, commons-collections4, json-smart and snakeyaml are direct dependencies of storm-server, so their relocated packages are allowed there. - storm-server bans com.google.common and com.google.thirdparty, the com.google packages in the shaded jar, so com.google.auto.service stays allowed. Other modules are not checked, because storm-core, storm-webapp and the external modules declare some of the relocated packages themselves. Test code is not checked, because tests use curator-test, which is not shaded. The check covers import statements only, not fully qualified class names in code. ## How was the change tested Without the fixes, storm-client reports the IOUtils import and storm-server reports the five com.google.common imports, and nothing else. A temporary class in storm-server that imports one class from each relocated package fails on the 15 banned packages and passes on the allowed ones and on auto-service. The build with the examples and externals profiles passes validate in every module. UtilsTest passes in storm-client. SlotTest, WorkerTokenTest, EquivalenceUtilsTest, TestCgroups, TestResourceAwareScheduler and MultitenantSchedulerTest pass in storm-server. ## AI Disclosure Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission. Signed-off-by: dragonknightk <[email protected]>
| <execution> | ||
| <!-- Fail the build if main code imports one of the relocated packages of storm-shaded-deps. | ||
| Keep the list in sync with the relocated packages in storm-shaded-deps/pom.xml. --> | ||
| <id>check-unshaded-imports</id> |
There was a problem hiding this comment.
nit: this execution is copied almost verbatim into storm-server; only illegalPkgs differs. You could define the check-unshaded-imports execution once in the root pom's checkstyle pluginManagement, reading illegalPkgs from a property that each module sets. Then the comment in storm-shaded-deps only has to point at one place. Fine as a follow-up.
| <module name="Checker"> | ||
| <module name="TreeWalker"> | ||
| <module name="IllegalImport"> | ||
| <property name="illegalPkgs" value="org.apache.thrift, com.google.common, |
There was a problem hiding this comment.
Worth noting in the description: this also bans direct org.apache.curator imports in main code. That fits the ongoing Curator decoupling, since any new Curator use has to go through the shaded package.
Closes issue 7641 ## What is the purpose of the change storm-client and storm-server now fail validate when their main code imports one of the relocated packages of storm-shaded-deps. Such an import still compiles if another dependency brings in the original jar, so the module silently depends on an unshaded copy. Six imports already did this: - storm-client Utils used org.apache.commons.io.IOUtils, which reaches storm-client through commons-compress. - storm-server EquivalenceUtils, SchedulerConfigCache, WorkerTokenManager, CgroupManager and Slot used com.google.common, which reaches storm-server only through the optional auto-service dependency. They now use the shaded classes, like the other imports in the same modules. The root pom defines the check-unshaded-imports checkstyle execution once, in pluginManagement, with an IllegalImport rule that reads the unshaded.illegalPkgs property. The shared storm_checkstyle.xml stays unchanged. The execution is bound to the phase in the unshaded.check.phase property, which defaults to none. storm-client and storm-server set it to validate and set their own unshaded.illegalPkgs. A comment next to the relocations in storm-shaded-deps/pom.xml points to these two properties. The two lists differ: - storm-client bans all relocated packages. - commons-io, commons-lang3, commons-collections4, json-smart and snakeyaml are direct dependencies of storm-server, so their relocated packages are allowed there. - storm-server bans com.google.common and com.google.thirdparty, the com.google packages in the shaded jar, so com.google.auto.service stays allowed. Both lists ban org.apache.curator, so new Curator code in the main code of storm-client and storm-server has to use the shaded package. This fits the ongoing Curator decoupling. Other modules are not checked, because storm-core, storm-webapp and the external modules declare some of the relocated packages themselves. They inherit the execution with phase none, so it never runs there. Test code is not checked, because tests use curator-test, which is not shaded. The check covers import statements only, not fully qualified class names in code. ## How was the change tested Without the fixes, storm-client reports the IOUtils import and storm-server reports the five com.google.common imports, and nothing else. A temporary class in storm-server that imports one class from each relocated package fails on the 15 banned packages and passes on the allowed ones and on auto-service. Temporary imports of the first, the last and two other banned packages fail validate in storm-client and in storm-server. The effective pom of storm-client and storm-server binds check-unshaded-imports to validate with their own package lists. The build with the examples and externals profiles passes validate in every module and runs check-unshaded-imports only in storm-client and storm-server. UtilsTest passes in storm-client. SlotTest, WorkerTokenTest, EquivalenceUtilsTest, TestCgroups, TestResourceAwareScheduler and MultitenantSchedulerTest pass in storm-server. ## AI Disclosure Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission. Signed-off-by: dragonknightk <[email protected]>
|
@rzo1 Thanks for the review! I took both suggestions in this PR: the check-unshaded-imports execution is now defined once in the root pom's pluginManagement, and storm-client and storm-server only set their own package lists through a property. The description now also notes that main code can no longer import org.apache.curator directly. (。•̀ᴗ-)✧ |
Fixes #7641
What is the purpose of the change
storm-client and storm-server now fail validate when their main code imports one of the relocated packages of storm-shaded-deps. Such an import still compiles if another dependency brings in the original jar, so the module silently depends on an unshaded copy. Six imports already did this:
They now use the shaded classes, like the other imports in the same modules.
The root pom defines the check-unshaded-imports checkstyle execution once, in pluginManagement, with an IllegalImport rule that reads the unshaded.illegalPkgs property. The shared storm_checkstyle.xml stays unchanged. The execution is bound to the phase in the unshaded.check.phase property, which defaults to none. storm-client and storm-server set it to validate and set their own unshaded.illegalPkgs. A comment next to the relocations in storm-shaded-deps/pom.xml points to these two properties.
The two lists differ:
Both lists ban org.apache.curator, so new Curator code in the main code of storm-client and storm-server has to use the shaded package. This fits the ongoing Curator decoupling.
Other modules are not checked, because storm-core, storm-webapp and the external modules declare some of the relocated packages themselves. They inherit the execution with phase none, so it never runs there. Test code is not checked, because tests use curator-test, which is not shaded. The check covers import statements only, not fully qualified class names in code.
How was the change tested
Without the fixes, storm-client reports the IOUtils import and storm-server reports the five com.google.common imports, and nothing else. A temporary class in storm-server that imports one class from each relocated package fails on the 15 banned packages and passes on the allowed ones and on auto-service. Temporary imports of the first, the last and two other banned packages fail validate in storm-client and in storm-server. The effective pom of storm-client and storm-server binds check-unshaded-imports to validate with their own package lists. The build with the examples and externals profiles passes validate in every module and runs check-unshaded-imports only in storm-client and storm-server. UtilsTest passes in storm-client. SlotTest, WorkerTokenTest, EquivalenceUtilsTest, TestCgroups, TestResourceAwareScheduler and MultitenantSchedulerTest pass in storm-server.
AI Disclosure
Human-led contribution. Grok Bot, working for dragonknightk, assisted with this change, including the code, tests, and description. All submitted content was reviewed by a human before submission.
(。•̀ᴗ-)✧