🐛 Auto-download Go module dependencies for licensing

README.md

@@ -185,8 +185,8 @@ same commands whichever CI system the team chooses.
 | [`cd-get-config`](./guides/reading-project-configuration.md) | Prints a project configuration value. | Either `--key` (`-k`) or `--config-variable` (`-c`) | Developer or CI |
 | [`cd-tag-and-release`](./guides/automating-releases.md) | Runs the release flow, including documentation, licensing summaries, Git tagging, packaging and publication through the selected [language plugin](./continuous_delivery_scripts/plugins). | Required `--release-type` (`-t`); optional `--current-branch` (`-b`) | CI release flow |
 | [`cd-generate-docs`](./guides/generating-code-documentation.md) | Generates code documentation in the configured output directory (or one specified on the command line). | `--output_directory` | Developer preview or CI |
-| [`cd-generate-spdx`](./guides/generating-an-spdx-sbom.md) | Generates SPDX documents and third-party licence summaries, then checks licence compliance when project metadata is available. | Required `--output-dir` (`-o`); optional `--lookup-scancode` | CI audit; developer review also possible |
-| [`cd-check-licence-compliance`](./guides/checking-licence-compliance.md) | Checks project and dependency licences without creating SPDX documents; optionally writes third-party IP summaries. | Optional `--output-dir` (`-o`) for reports, `--lookup-scancode` | CI gate or developer review |
+| [`cd-generate-spdx`](./guides/generating-an-spdx-sbom.md) | Generates SPDX documents and third-party licence summaries, then checks licence compliance when project metadata is available. | Required `--output-dir` (`-o`); optional `--lookup-scancode`, `--skip-dependency-download` | CI audit; developer review also possible |
+| [`cd-check-licence-compliance`](./guides/checking-licence-compliance.md) | Checks project and dependency licences without creating SPDX documents; optionally writes third-party IP summaries. | Optional `--output-dir` (`-o`) for reports, `--lookup-scancode`, `--skip-dependency-download` | CI gate or developer review |
 | [`cd-license-files`](./guides/licence-header-management.md) | Adds or updates source-file licence and copyright headers when the language plugin supports them. | `--verbose` (`-v`) | Developer or CI |
 | [`cd-record-secrets`](./guides/recording-secrets.md) | Records accepted findings in the project's [detect-secrets](https://github.com/Yelp/detect-secrets) registry. | `--registry-file` (`-r`); defaults to the configured registry | Developer (registry maintenance) |
 | [`cd-detect-secrets`](./guides/checking-for-secrets.md) | Checks Git-tracked files against that registry and fails if new secrets are found. | `--registry-file` (`-r`); defaults to the configured registry | CI; developer checks also possible |

continuous_delivery_scripts/check_licence_compliance.py

@@ -90,6 +90,11 @@ def main() -> int:
         action="store_true",
         help="Consult ScanCode LicenseDB for unclassified licences or missing assessment rules.",
     )
+    parser.add_argument(
+        "--skip-dependency-download",
+        action="store_true",
+        help="Skip automatic dependency downloads before licence checks and SPDX dependency analysis.",
+    )
     parser.add_argument("-v", "--verbose", action="count", default=0, help="Increase logging verbosity.")
     args = parser.parse_args()
     set_log_level(args.verbose)
@@ -98,7 +103,7 @@ def main() -> int:
         plugin = get_language_specifics()
         if not plugin.can_get_project_metadata():
             raise ValueError("The selected language plugin cannot provide metadata for licence compliance checks.")
-        project = plugin.get_current_spdx_project()
+        project = plugin.get_current_spdx_project(skip_dependency_download=args.skip_dependency_download)
         if project is None:
             raise ValueError("The selected language plugin did not return a project for licence compliance checks.")
         if args.lookup_scancode:

continuous_delivery_scripts/plugins/GOLAND.MD

@@ -20,14 +20,21 @@ usable `go-licenses` binary on `PATH` before installing
 must be available; an installed binary can also be located under `GOBIN` or
 the first `GOPATH` entry's `bin` directory.
 
+For licence and SPDX analysis, the plugin also runs `go mod download all` for
+each detected Go module by default so callers do not need a separate dependency
+prefetch step. If your environment already downloads dependencies and you want
+to skip that work, use `--skip-dependency-download` on
+`cd-check-licence-compliance` or `cd-generate-spdx`, or set
+`SKIP_DEPENDENCY_DOWNLOAD_FOR_LICENSING = true` in configuration.
+
 Set `PROGRAMMING_LANGUAGE = "golang"`, `PROJECT_NAME`, `PROJECT_ROOT` and
 `SOURCE_DIR` in `[ProjectConfig]`. Put a `go.mod` under the source or project
 root; `go.work` modules and nested Go modules are also considered. Supply
 `FILE_LICENCE_IDENTIFIER` for the project licence and set `PROJECT_UUID`
 and `[spdx]` namespace fields as explained in the
 [SPDX configuration guide](../../DEVELOPMENT.md#spdx-document-identity).
 
-With the project and its Go dependencies available, run:
+With the project available, run:
 
 ```bash
 mkdir -p licensing

continuous_delivery_scripts/plugins/PYTHON.MD

@@ -17,8 +17,10 @@ The plugin uses [pdoc](https://pdoc3.github.io/pdoc/) to generate the code refer
 
 The Python plugin reads metadata from the **installed** project distribution and
 walks its `Requires-Dist` entries to find installed, transitive dependencies.
-Run the audit in the environment containing the dependencies to be released;
-it does not install packages.
+Install the dependencies you intend to audit, then run the audit in that
+environment; the plugin does not install packages itself. This is deliberate:
+Python projects can define dependencies in many different ways, so CDS cannot
+reliably infer and reproduce the intended audit environment automatically.
 
 > **Warning — platform limitation:** The audit only sees dependencies installed
 > for the current platform and Python version. It does **not** inspect

continuous_delivery_scripts/plugins/basic.py

@@ -51,6 +51,6 @@ def can_get_project_metadata(self) -> bool:
         """States whether project metadata can be retrieved."""
         return False
 
-    def get_current_spdx_project(self) -> Optional["SpdxProject"]:
+    def get_current_spdx_project(self, skip_dependency_download: bool = False) -> Optional["SpdxProject"]:
         """Basic."""
         return None

continuous_delivery_scripts/plugins/ci.py

@@ -61,7 +61,7 @@ def can_get_project_metadata(self) -> bool:
         """States whether project metadata can be retrieved."""
         return False
 
-    def get_current_spdx_project(self) -> Optional["SpdxProject"]:
+    def get_current_spdx_project(self, skip_dependency_download: bool = False) -> Optional["SpdxProject"]:
         """Gets current SPDX description."""
         # TODO
         return None

continuous_delivery_scripts/plugins/docker.py

@@ -53,7 +53,7 @@ def can_get_project_metadata(self) -> bool:
         """States whether project metadata can be retrieved."""
         return False
 
-    def get_current_spdx_project(self) -> Optional["SpdxProject"]:
+    def get_current_spdx_project(self, skip_dependency_download: bool = False) -> Optional["SpdxProject"]:
         """Gets current SPDX description."""
         # TODO
         return None

continuous_delivery_scripts/plugins/golang.py

@@ -135,6 +135,26 @@ def _go_module_name(module_dir: Path, env: MutableMapping[str, str]) -> str:
     return str(module["Path"])
 
 
+def _download_go_module_dependencies(module_directories: List[Path], env: MutableMapping[str, str]) -> None:
+    """Download Go dependencies for each detected module.
+
+    `go-licenses` expects the module graph to be available locally. Fetching the
+    dependencies here means callers do not need a separate CI step before
+    running `cd-check-licence-compliance` or `cd-generate-spdx`.
+    """
+    for module_directory in module_directories:
+        logger.info("Downloading Go module dependencies in [%s].", module_directory)
+        check_call(["go", "mod", "download", "all"], cwd=module_directory, env=env)
+
+
+def _should_download_go_module_dependencies() -> bool:
+    """Return whether Go module dependencies should be prefetched automatically."""
+    configured = configuration.get_value_or_default(ConfigurationVariable.SKIP_DEPENDENCY_DOWNLOAD_FOR_LICENSING, False)
+    if isinstance(configured, str):
+        configured = configured.strip().lower() in ("true", "1", "yes", "on")
+    return not bool(configured)
+
+
 def _parse_go_licences(output: str) -> List[PackageMetadata]:
     """Translate go-licenses template output into shared package metadata."""
     packages = []
@@ -226,15 +246,20 @@ def _stage_root_licence_files_for_modules(module_directories: List[Path]) -> Ite
 class GoProjectMetadataFetcher(ProjectMetadataFetcher):
     """Retrieve Go dependency licences for the shared SPDX and compliance report."""
 
-    def __init__(self) -> None:
+    def __init__(self, skip_dependency_download: bool = False) -> None:
         """Initialise with the configured project name."""
         super().__init__(str(configuration.get_value(ConfigurationVariable.PROJECT_NAME)))
+        self._skip_dependency_download = skip_dependency_download
 
     def fetch_project_metadata(self) -> ProjectMetadata:
         """Collect module dependencies using go-licenses without replacing shared policy checks."""
         module_directories = _go_licence_module_directories()
         env = os.environ.copy()
         env[ENVVAR_GO_MOD] = GO_MOD_ON_VALUE
+        if not self._skip_dependency_download and _should_download_go_module_dependencies():
+            _download_go_module_dependencies(module_directories, env)
+        else:
+            logger.info("Skipping Go module dependency downloads before licence analysis by configuration.")
         executable = _ensure_go_tool_installed(
             tool_name="go-licenses",
             version_command=["go-licenses", "report", "--help"],
@@ -462,9 +487,9 @@ def get_secret_registry_exclude_files(self) -> List[str]:
             (r"^\.github[\\/]workflows[\\/].*"),
         ]
 
-    def get_current_spdx_project(self) -> Optional[SpdxProject]:
+    def get_current_spdx_project(self, skip_dependency_download: bool = False) -> Optional[SpdxProject]:
         """Gets current SPDX description."""
-        return SpdxProject(GoProjectMetadataFetcher())
+        return SpdxProject(GoProjectMetadataFetcher(skip_dependency_download=skip_dependency_download))
 
     def should_clean_before_packaging(self) -> bool:
         """States whether the repository must be cleaned before packaging happens."""

continuous_delivery_scripts/plugins/noop.py

@@ -53,7 +53,7 @@ def can_get_project_metadata(self) -> bool:
         """States whether project metadata can be retrieved."""
         return False
 
-    def get_current_spdx_project(self) -> Optional["SpdxProject"]:
+    def get_current_spdx_project(self, skip_dependency_download: bool = False) -> Optional["SpdxProject"]:
         """No Op."""
         return None
 

continuous_delivery_scripts/plugins/python.py

@@ -198,6 +198,6 @@ def should_include_spdx_in_package(self) -> bool:
         # FIXME Comment out SPDX package as no longer working
         return False
 
-    def get_current_spdx_project(self) -> Optional["SpdxProject"]:
+    def get_current_spdx_project(self, skip_dependency_download: bool = False) -> Optional["SpdxProject"]:
         """Gets the current SPDX description."""
         return _get_current_spdx_project()

continuous_delivery_scripts/report_third_party_ip.py

@@ -36,9 +36,11 @@ def generate_spdx_project_reports(project: "SpdxProject", output_directory: Path
     return project
 
 
-def generate_spdx_reports(output_directory: Path, lookup_scancode: bool = False) -> Optional["SpdxProject"]:
+def generate_spdx_reports(
+    output_directory: Path, lookup_scancode: bool = False, skip_dependency_download: bool = False
+) -> Optional["SpdxProject"]:
     """Generates all the SPDX reports for the current project."""
-    project = get_language_specifics().get_current_spdx_project()
+    project = get_language_specifics().get_current_spdx_project(skip_dependency_download=skip_dependency_download)
     if not project:
         return None
     if lookup_scancode:
@@ -66,6 +68,11 @@ def convert_to_path(arg: Any) -> Path:
         action="store_true",
         help="Consult ScanCode LicenseDB for unclassified licences or missing assessment rules.",
     )
+    parser.add_argument(
+        "--skip-dependency-download",
+        action="store_true",
+        help="Skip automatic dependency downloads before licence checks and SPDX dependency analysis.",
+    )
 
     parser.add_argument(
         "-v",
@@ -79,7 +86,11 @@ def convert_to_path(arg: Any) -> Path:
 
     try:
         if get_language_specifics().can_get_project_metadata():
-            project = generate_spdx_reports(args.output_dir, lookup_scancode=args.lookup_scancode)
+            project = generate_spdx_reports(
+                args.output_dir,
+                lookup_scancode=args.lookup_scancode,
+                skip_dependency_download=args.skip_dependency_download,
+            )
             if project:
                 project.check_licence_compliance()
                 if args.lookup_scancode:

continuous_delivery_scripts/utils/configuration.py

@@ -109,6 +109,8 @@ class ConfigurationVariable(enum.Enum):
     """Project-specific records explaining manual reviews of REVIEW assessments."""
     LICENCE_ASSESSMENT_FAIL_ON = 45
     """Optional top-level list of assessment statuses that fail compliance checks."""
+    SKIP_DEPENDENCY_DOWNLOAD_FOR_LICENSING = 46
+    """Skip automatic dependency downloads before licence and SPDX dependency analysis."""
 
     @staticmethod
     def choices() -> List[str]:
@@ -219,6 +221,7 @@ class StaticConfig(GenericConfig):
     LICENCE_ASSESSMENT_FAIL_ON = None
     """When unset, use fail_on from the embedded or project licence assessment rules."""
     GENERATE_LICENSING_SUMMARY_ON_RELEASE = False
+    SKIP_DEPENDENCY_DOWNLOAD_FOR_LICENSING = False
     DEPENDENCY_UPDATE_NEWS_MESSAGE = "Dependency upgrade: {message}"
     DEPENDENCY_UPDATE_NEWS_TYPE = NewsType.bugfix
     DEPENDENCY_UPDATE_BRANCH_PATTERN = r"^\s*[Dd]ependabot\/.+\/(?P<DEPENDENCY>.+)"

continuous_delivery_scripts/utils/language_specifics_base.py

@@ -121,7 +121,7 @@ def release_package_to_repository(self, mode: CommitType, version: str) -> None:
         pass
 
     @abstractmethod
-    def get_current_spdx_project(self) -> Optional["SpdxProject"]:
+    def get_current_spdx_project(self, skip_dependency_download: bool = False) -> Optional["SpdxProject"]:
         """Gets current project SPDX."""
         pass
 

guides/checking-licence-compliance.md

@@ -8,9 +8,9 @@ without generating SPDX documents. The command writes no files by default.
 
 ## Inputs and example
 
-Install the dependencies you intend to audit and select a language plugin that
-provides project metadata. Configure `ACCEPTED_THIRD_PARTY_LICENCES` and any
-documented manual reviews in `[ProjectConfig]` in `pyproject.toml`. Then run:
+Select a language plugin that provides project metadata. Configure
+`ACCEPTED_THIRD_PARTY_LICENCES` and any documented manual reviews in
+`[ProjectConfig]` in `pyproject.toml`. Then run:
 
 ```bash
 cd-check-licence-compliance
@@ -28,6 +28,11 @@ The optional `--output-dir` (`-o`) writes `third_party_IP_report.html`, `.csv`,
 `.txt` and `.json`. It does not create `.spdx` files. The reports are written
 before the compliance check, so they remain available if the check fails.
 
+For Go projects, CDS downloads module dependencies automatically before running
+licence analysis so CI does not need a separate `go mod download all` step.
+Pass `--skip-dependency-download` when your environment already prefetches the
+required dependencies and you want to skip that extra work.
+
 ## Result
 
 The command succeeds when the discovered project and dependency licences meet

guides/generating-an-spdx-sbom.md

@@ -9,7 +9,6 @@ see [TPIP reporting](third-party-ip-reporting.md).
 
 ## Inputs and example
 
-Install the project and its dependencies in the environment being audited.
 Configure `pyproject.toml` with `PROJECT_ROOT`, `SOURCE_DIR`,
 `PROGRAMMING_LANGUAGE`, your licence policy and any values required by the
 selected language plugin. Provide
@@ -27,6 +26,10 @@ the command returns without a report. The selected plugin uses language-specific
 tools to obtain dependency and licence information; consult the
 [plugin documentation](https://github.com/ARMmbed/continuous-delivery-scripts/tree/main/continuous_delivery_scripts/plugins)
 for its prerequisites and metadata support.
+For Go projects, CDS downloads module dependencies automatically before
+running the dependency scan. Pass `--skip-dependency-download` when the
+environment already prefetches those dependencies and you want to skip that
+extra step.
 SPDX tag-value generation also requires an SDK compatible with this project's
 writer. Check the output files before treating an audit as complete. Installed
 packages and platform-specific dependency markers determine what is covered;

guides/third-party-ip-reporting.md

@@ -10,9 +10,9 @@ see [SPDX generation](generating-an-spdx-sbom.md).
 
 ## Inputs and example
 
-Install the dependencies you intend to audit, configure your language plugin
-and set `ACCEPTED_THIRD_PARTY_LICENCES` and any documented manual checks in
-`pyproject.toml`. Choose an existing output directory:
+Configure your language plugin and set `ACCEPTED_THIRD_PARTY_LICENCES` and any
+documented manual checks in `pyproject.toml`. Choose an existing output
+directory:
 
 ```bash
 mkdir -p licensing
@@ -27,6 +27,9 @@ for language-specific prerequisites and support.
 An audit covers the installed environment, so run it for each supported
 platform if dependencies differ by operating system. Review unknown licences
 and packaged notices rather than assuming a missing value means permission.
+For Go projects, CDS downloads module dependencies automatically before the
+scan. Pass `--skip-dependency-download` when those dependencies have already
+been prefetched and you want to skip that extra work.
 
 ## Output
 

news/20261002191514321000000.bugfix

@@ -0,0 +1 @@
+:bug: Automatically run `go mod download all` for detected Go modules before licence and SPDX dependency analysis, with config and command-line opt-outs for CI setups that already prefetch dependencies.