Optimize build by refactoring fragile tests and stripping heavy dependencies#3068
Conversation
| @@ -0,0 +1,28 @@ | |||
| # Build Optimization Plan & Report | |||
There was a problem hiding this comment.
Ignore this file, I'm removing it from the commit imminently.
5e6d0ae to
ede87e2
Compare
|
I am curious if this part is always safe: "Stripped heavy Docker (buildNomulusImage) and 5x frontend (buildConsoleForAll) staging dependencies from the standard build task, ensuring they are only run when explicitly deployed." |
weiminyu
left a comment
There was a problem hiding this comment.
@weiminyu reviewed 8 files and all commit messages, and made 1 comment.
Reviewable status: all files reviewed, 3 unresolved discussions (waiting on CydeWeys).
core/src/test/java/google/registry/server/TestServer.java line 171 at r2 (raw file):
port = urlAddress.getPortOrDefault(DEFAULT_PORT); } return new URL(String.format("http://%s:%d%s", urlAddress.getHost(), port, path));
new approach is new URI(...).toURL()
Code quote:
new URL|
@CydeWeys Yes, it is safe! The heavy Docker and 5x frontend staging dependencies are only required when actually packaging the artifact for deployment or running full E2E staging tests. They are not required for local compilation or running unit/integration tests, which is what developers use the default @weiminyu I have updated |
|
If this is removed here, it won't be invoked until build time. Maybe make Kokoro build this explicitly? Code quote: buildToolImage |
CydeWeys
left a comment
There was a problem hiding this comment.
@CydeWeys made 1 comment.
Reviewable status: 6 of 8 files reviewed, 3 unresolved discussions (waiting on weiminyu).
core/build.gradle line 762 at r3 (raw file):
Previously, weiminyu (Weimin Yu) wrote…
If this is removed here, it won't be invoked until build time.
Maybe make Kokoro build this explicitly?
What's the issue we're trying to solve for here -- a PR contains changes to the tool, and we want Kokoro to build the tool with those changes included prior to calling it?
(But this isn't an issue for local builds?)
CydeWeys
left a comment
There was a problem hiding this comment.
@CydeWeys made 1 comment.
Reviewable status: 6 of 9 files reviewed, 3 unresolved discussions (waiting on weiminyu).
core/build.gradle line 762 at r3 (raw file):
Previously, CydeWeys (Ben McIlwain) wrote…
What's the issue we're trying to solve for here -- a PR contains changes to the tool, and we want Kokoro to build the tool with those changes included prior to calling it?
(But this isn't an issue for local builds?)
PTAL, now with a buildAll target that I will configure Kokoro to use once this is merged.
CydeWeys
left a comment
There was a problem hiding this comment.
@CydeWeys made 1 comment.
Reviewable status: 6 of 9 files reviewed, 3 unresolved discussions (waiting on weiminyu).
core/src/test/java/google/registry/server/TestServer.java line 171 at r2 (raw file):
Previously, weiminyu (Weimin Yu) wrote…
new approach is new URI(...).toURL()
Done.
CydeWeys
left a comment
There was a problem hiding this comment.
@CydeWeys resolved 1 discussion.
Reviewable status: 6 of 9 files reviewed, 2 unresolved discussions (waiting on weiminyu).
weiminyu
left a comment
There was a problem hiding this comment.
@weiminyu reviewed 4 files and all commit messages, and resolved 1 discussion.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on CydeWeys).
weiminyu
left a comment
There was a problem hiding this comment.
@weiminyu resolved 1 discussion.
Reviewable status:complete! all files reviewed, all discussions resolved (waiting on CydeWeys).
9590a39 to
98ad630
Compare
|
PTAL. Didn't make further changes, but did have to rebase a bunch of commits from master. |
This commit dramatically optimizes the local Gradle build time, shaving over 5 minutes off a full build execution: - Instrumented the build to identify fragileTest taking > 3 minutes. - Refactored TestServer.java to dynamically bind to ephemeral port 0, resolving race conditions. - Updated UploadBsaUnavailableDomainsActionTest to use the thread-safe TestServer, allowing it to run in parallel. - Removed outdated exclusions for HostInfoFlowTest and RegistryPipelineWorkerInitializerTest. - Moved these tests to the highly parallelized standardTest suite. - Removed the redundant sqlIntegrationTest execution from the standard test phase. - Stripped heavy Docker (buildNomulusImage) and 5x frontend (buildConsoleForAll) staging dependencies from the standard build task, ensuring they are only run when explicitly deployed.
This commit adds the buildAll task to restore the existence of a target that builds everything, which was unintentionally removed when the default build was stripped down in PR google#3068. It also introduces necessary sequential constraints to the console-webapp build tasks to prevent parallel execution from corrupting the Angular CLI cache. Finally, it addresses paths for the newer Angular esbuild output and hardens the style injection in ConsoleScreenshotTest to prevent fragile test failures.
This commit reverts changes from 5599a0e and most of 5286b1a (PR google#3068) that stripped essential dependencies (buildConsoleForAll, buildNomulusImage, buildToolImage, fragileTest) from the default './gradlew build' target, which broke downstream deployment pipelines. It restores the default build to correctly generate all necessary production artifacts and Docker images. It introduces a new 'fastBuild' target designed explicitly for local developers and CI checks. This lightweight target disables the execution of heavy Docker image builds, Angular compilations, and fragile tests to provide rapid feedback. Sequential execution constraints for parallel Angular builds are maintained to prevent cache corruption. It updates the ':core:generateSqlSchema' task to execute using the 'unittest' environment instead of 'alpha'. The 'alpha' configuration is a private, internal environment config that is not distributed in the open-source repository, which caused the task to fail for public contributors. By switching to 'unittest', the generator can successfully run using the public test configuration. With this fixed, it also includes the newly generated 'db-schema.sql.generated' file, which now correctly tracks the 'FORBID_INSECURE_ALGORITHMS_RFC_9904' feature flag that was recently added. Finally, it unconditionally configures the 'sqlIntegrationTest' task to execute with 'useJUnitPlatform()' and injects the 'junit-platform-suite' dependency. This permanently resolves 'failed to discover tests' exceptions on Kokoro by allowing the modern Jupiter engine to natively discover and run cross-version deployed artifacts compiled with legacy JUnit 4 '@RunWith' wrappers.
This commit reverts changes from 5599a0e and most of 5286b1a (PR google#3068) that stripped essential dependencies (buildConsoleForAll, buildNomulusImage, buildToolImage, fragileTest) from the default './gradlew build' target, which broke downstream deployment pipelines. It restores the default build to correctly generate all necessary production artifacts and Docker images. It introduces a new 'fastBuild' target designed explicitly for local developers and CI checks. This lightweight target disables the execution of heavy Docker image builds, Angular compilations, and fragile tests to provide rapid feedback. Sequential execution constraints for parallel Angular builds are maintained to prevent cache corruption. It updates the ':core:generateSqlSchema' task to execute using the 'unittest' environment instead of 'alpha'. The 'alpha' configuration is a private, internal environment config that is not distributed in the open-source repository, which caused the task to fail for public contributors. By switching to 'unittest', the generator can successfully run using the public test configuration. With this fixed, it also includes the newly generated 'db-schema.sql.generated' file, which now correctly tracks the 'FORBID_INSECURE_ALGORITHMS_RFC_9904' feature flag that was recently added. Finally, it restores the conditional 'useJUnit()' fallback in 'sqlIntegrationTest' for deployed artifacts on Kokoro, preventing classpath collisions and NoSuchMethodErrors between the modern junit-platform-suite engine and the embedded older engine present in deployed 'alldeps' jars.
This commit reverts changes from 5599a0e and most of 5286b1a (PR google#3068) that stripped essential dependencies (buildConsoleForAll, buildNomulusImage, buildToolImage, fragileTest) from the default './gradlew build' target, which broke downstream deployment pipelines. It restores the default build to correctly generate all necessary production artifacts and Docker images. It introduces a new 'fastBuild' target designed explicitly for local developers and CI checks. This lightweight target disables the execution of heavy Docker image builds, Angular compilations, and fragile tests to provide rapid feedback. Sequential execution constraints for parallel Angular builds are maintained to prevent cache corruption. It updates the ':core:generateSqlSchema' task to execute using the 'unittest' environment instead of 'alpha'. The 'alpha' configuration is a private, internal environment config that is not distributed in the open-source repository, which caused the task to fail for public contributors. By switching to 'unittest', the generator can successfully run using the public test configuration. With this fixed, it also includes the newly generated 'db-schema.sql.generated' file, which now correctly tracks the 'FORBID_INSECURE_ALGORITHMS_RFC_9904' feature flag that was recently added. Finally, it implements a split-runner execution strategy for the 'sqlIntegrationTest' task to permanently resolve 'failed to discover tests' and 'NoSuchMethodError' exceptions on Kokoro. Because Kokoro tests cross-version compatibility against both legacy deployed artifacts (compiled with JUnit 4 @RunWith wrappers) and modern artifacts (compiled with JUnit 5 @suite annotations), we cannot statically configure a single test runner. We now dynamically run both the legacy 'useJUnit()' and modern 'useJUnitPlatform()' runners sequentially with 'failOnNoDiscoveredTests' disabled, allowing the appropriate engine to discover and execute the suite without causing classpath collisions.
This commit dramatically optimizes the local Gradle build time, shaving over 5 minutes off a full build execution:
This change is