diff --git a/.github/README.md b/.github/README.md new file mode 100644 index 0000000000..0a111b854e --- /dev/null +++ b/.github/README.md @@ -0,0 +1,52 @@ +# CI / Release Workflows + +Reference for the GitHub Actions workflows in [`.github/workflows/`](workflows/), grouped by whether they run automatically or need to be dispatched manually. Branching/release strategy is documented in [`docs/branching-strategy.md`](../docs/branching-strategy.md) — briefly: `develop` (next protocol version) ⊇ `release/XXX` (latest published protocol version, hotfixes) ⊇ `main` (public-facing latest release). Workflow/CI changes should land on `main` first and be propagated up through `release/XXX` into `develop`. + +## Automatic workflows + +These trigger on push/PR/schedule/release — nothing to run by hand. + +| Workflow | Trigger | Purpose | Action | +|---|---|---|---| +| `ci.yml` | Every PR; push to `develop`, `main`, `release/*` | Unit/integration tests, Sonar, Codecov, cross-compile check | [Runs](https://github.com/radixdlt/babylon-node/actions/workflows/ci.yml) | +| `docker.yml` | Every PR; push to `develop`, `main`, `release/*`; GitHub Release published | Builds `.deb`, builds/pushes private multiarch Docker image (PR/branch) or public release image (on Release), Snyk image/project monitoring | [Runs](https://github.com/radixdlt/babylon-node/actions/workflows/docker.yml) | +| `deploy-and-smoke-test.yaml` | Push to `develop` | Deploys to "gilganet" testnet via Jenkins, then runs smoke tests via Jenkins | [Runs](https://github.com/radixdlt/babylon-node/actions/workflows/deploy-and-smoke-test.yaml) | +| `add-artifacts-to-release.yml` | GitHub Release published (also `workflow_dispatch`, see below) | Builds `libcorerust` for 6 targets + the Java distribution zip, attaches both to the Release | [Runs](https://github.com/radixdlt/babylon-node/actions/workflows/add-artifacts-to-release.yml) | + +## Manual workflows + +| Workflow | Trigger | Purpose | Action | +|---|---|---|---| +| `publish-typescript-sdk.yml` | `workflow_dispatch` (input `package_version_number`, required) | Publishes `@radixdlt/babylon-core-api-sdk` to npmjs.org. Deliberately **not** wired to the `release` event — SDK versioning is decoupled from node release tags. | [Runs](https://github.com/radixdlt/babylon-node/actions/workflows/publish-typescript-sdk.yml) | +| `add-artifacts-to-release.yml` | `workflow_dispatch` | Can be run standalone to build the `libcorerust` artifacts (the publish-to-release jobs only run `if: github.event_name == 'release'`, so manual dispatch only exercises the build, not the upload) | [Runs](https://github.com/radixdlt/babylon-node/actions/workflows/add-artifacts-to-release.yml) | + +### Release sequence + +1. A GitHub **Release** is published from `release/XXX` (per the branching strategy, `release/XXX` is merged into `main` as part of this). +2. This auto-fires **`add-artifacts-to-release.yml`**: builds `libcorerust` for macOS/Linux/Windows × x86_64/aarch64, zips and uploads each to the Release, then (gated behind the **`publish-artifacts`** environment) builds and uploads the Java distribution zip. +3. This also auto-fires **`docker.yml`**'s release path: builds+pushes the public `docker.io/radixdlt/babylon-node` image (AMD64 + ARM64), joins them into a multiarch manifest, then runs Snyk container/project monitoring against the release. +4. If the release includes an SDK-relevant Core API change, separately run **`publish-typescript-sdk.yml`** manually with the new `package_version_number`. + +## External dependencies per job + +Shared building blocks used across nearly every workflow (not repeated per-row below): +- **`RDXWorks-actions/*`** — the org's own mirror of common third-party actions (`checkout`, `cache`, `setup-java`, `setup-node`, `codecov-action`, `snyk-actions`, `notify-slack-action`, `action-gh-release`, `configure-aws-credentials`, `aws-secretsmanager-get-secrets`, etc.). +- **`radixdlt/public-iac-resuable-artifacts`** — external repo providing reusable workflows (`docker-build.yml`, `join-docker-images-all-tags.yml`) and composite actions (`fetch-secrets`, `tailnet`, `snyk-container-monitor`). +- **`./.github/actions/fetch-secrets`** — local composite action that assumes an AWS IAM role (`RDXWorks-actions/configure-aws-credentials`) and reads a named secret from Secrets Manager (`RDXWorks-actions/aws-secretsmanager-get-secrets`). This is the core AWS integration point used by several of the workflows. +- **`./.github/actions/setup-env`** — installs the pinned Rust toolchain (must match the corresponding radixdlt-scrypto branch) + JDK 17; no secrets. +- **`./.github/actions/setup-version-properties`** / **`./.github/actions/gradle-task`** — pure git/gradle wrappers computing `VERSION_*` outputs; no secrets. + +| Workflow | Secrets | AWS / Secrets Manager | Self-hosted runner(s) | GH environment | Other external deps | +|---|---|---|---|---|---| +| `ci.yml` | `COMMON_SECRETS_ROLE_ARN`; `AWS_SECRET_NAME_CODECOV`; `GITHUB_TOKEN` | `build` job fetches the Codecov secret via **external** `public-iac-resuable-artifacts/fetch-secrets` using `COMMON_SECRETS_ROLE_ARN`/`AWS_SECRET_NAME_CODECOV`, and fetches the Sonar token via the **local** `fetch-secrets` action from the literal path `github-actions/common/sonar-token` | `selfhosted-ubuntu-22.04-16-cores` (build/sonar, steadystate-integration, targeted-integration) | none | SonarCloud/SonarQube, Codecov.io | +| `docker.yml` | `DOCKERHUB_RELEASER_ROLE`; `AWS_ROLE_NAME_SNYK_SECRET`; `AWS_SECRET_NAME_DOCKERHUB`; `AWS_SECRET_NAME_SNYK`; `SNYK_ORG_ID` | Private-image multiarch join uses hardcoded OIDC role `arn:aws:iam::308190735829:role/gh-common-secrets-read-access` + Secrets Manager path `github-actions/common/dockerhub-credentials`; public/release join uses `DOCKERHUB_RELEASER_ROLE` + path `github-actions/rdxworks/dockerhub-images/release-credentials`; release image builds also pass `DOCKERHUB_RELEASER_ROLE` as `role_to_assume` into the reusable `docker-build.yml`; Snyk monitor jobs fetch via `AWS_ROLE_NAME_SNYK_SECRET`/`AWS_SECRET_NAME_SNYK`(/`AWS_SECRET_NAME_DOCKERHUB`/`SNYK_ORG_ID`) | `selfhosted-ubuntu-22.04-16-cores` (`build_deb`), `selfhosted-ubuntu-22.04-arm` (both ARM image builds) | none | Docker Hub (`private-babylon-node` for PR/branch builds, public `babylon-node` for releases), Snyk (image + project monitoring) | +| `deploy-and-smoke-test.yaml` | `BABYLON_SECRETS_ROLE_ARN`; `SECRETS_ACCOUNT_ID` | Local `fetch-secrets` with `BABYLON_SECRETS_ROLE_ARN` reads Jenkins API token from `github-actions/radixdlt/babylon-node/jenkins-api-token`; Tailscale connect step assumes `arn:aws:iam::${SECRETS_ACCOUNT_ID}:role/gh-common-secrets-read-access` and reads `github-actions/common/tailscale-public-workflows-DpiE80` | none (GitHub-hosted `ubuntu-22.04`; actual deploy/test execution happens on Jenkins, external to Actions) | none | Jenkins (`v2-jobs/job/babylon-deploy-main`, `v2-jobs/job/babylon-testnet-smoke-tests` against gilganet), Tailscale (private tailnet to reach Jenkins) | +| `add-artifacts-to-release.yml` | none | none | none (`ubuntu-22.04`, `macos-14`, `windows-2025`) | **`publish-artifacts`** (distribution-zip job only) | GitHub Releases API (`action-gh-release` asset upload) | +| `publish-typescript-sdk.yml` | `BABYLON_SECRETS_ROLE_ARN` | Local `fetch-secrets` reads npm publishing token from `github-actions/radixdlt/babylon-node/npm-publishing-secret` (yields `NODE_AUTH_TOKEN`) | none | none (relies on `workflow_dispatch` alone for gating) | npmjs.org registry | + +Notes: +- Any workflow gated behind an `environment:` (`publish-artifacts`) requires whatever manual-approval/reviewer rule is configured for that environment in repo settings before the job proceeds. +- `ci.yml` and `docker.yml` both use **two different** `fetch-secrets` implementations: the **external** one (`radixdlt/public-iac-resuable-artifacts/fetch-secrets@main`, used for Snyk/Codecov) and the **local** one (`./.github/actions/fetch-secrets`, used for Sonar/Jenkins/npm). Same AWS pattern (assume role → Secrets Manager `GetSecretValue`), different repos hosting the composite action. +- `build_push_container_private*` jobs in `docker.yml` run on every PR — an ARM build only runs on a PR if it's labeled `ARM-TEST` (cost-saving guard on the self-hosted ARM pool). +- `deploy-and-smoke-test.yaml` and the Jenkins jobs it triggers are the only workflows here where the actual work happens outside GitHub Actions entirely (on Jenkins) — a GitHub Actions success only means the Jenkins job was *triggered*, not that the deploy/smoke-test itself passed (check Jenkins for that). +- `publish-typescript-sdk.yml` has a dead code path referencing `github.event.release.tag_name` for its version — the `release:` trigger that would populate it is commented out, so this branch of the version-detection logic never executes; the `package_version_number` input is always what's used in practice. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 17ed3d7b6a..3979065a61 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -13,69 +13,6 @@ on: - main - release\/* jobs: - phylum-analyze: - if: ${{ github.event.pull_request }} - uses: radixdlt/public-iac-resuable-artifacts/.github/workflows/phylum-analyze.yml@main - permissions: - id-token: write - pull-requests: write - contents: read - deployments: write - secrets: - phylum_api_key: ${{ secrets.PHYLUM_API_KEY }} - with: - phylum_pr_number: ${{ github.event.number }} - phylum_pr_name: ${{ github.head_ref }} - phylum_group_name: Protocol - phylum_project_id: 3f5b2c53-46bd-4f68-b050-5898f929002f - github_repository: ${{ github.repository }} - add_report_comment_to_pull_request: true - snyk-scan-deps-licences: - name: Snyk deps/licences scan - runs-on: ubuntu-latest - permissions: - id-token: write - pull-requests: read - contents: read - deployments: write - steps: - - uses: RDXWorks-actions/checkout@main - - uses: radixdlt/public-iac-resuable-artifacts/fetch-secrets@main - with: - role_name: ${{ secrets.AWS_ROLE_NAME_SNYK_SECRET }} - app_name: 'babylon-node' - step_name: 'snyk-scan-deps-licenses' - secret_prefix: 'SNYK' - secret_name: ${{ secrets.AWS_SECRET_NAME_SNYK }} - parse_json: true - - name: Run Snyk to check for deps vulnerabilities - uses: RDXWorks-actions/snyk-actions/gradle-jdk17@master - with: - args: --all-projects --org=${{ env.SNYK_NETWORK_ORG_ID }} --severity-threshold=critical - snyk-scan-code: - name: Snyk code scan - runs-on: ubuntu-latest - permissions: - id-token: write - pull-requests: read - contents: read - deployments: write - steps: - - uses: RDXWorks-actions/checkout@main - - uses: radixdlt/public-iac-resuable-artifacts/fetch-secrets@main - with: - role_name: ${{ secrets.AWS_ROLE_NAME_SNYK_SECRET }} - app_name: 'babylon-node' - step_name: 'snyk-scan-code' - secret_prefix: 'SNYK' - secret_name: ${{ secrets.AWS_SECRET_NAME_SNYK }} - parse_json: true - - name: Run Snyk to check for code vulnerabilities - uses: RDXWorks-actions/snyk-actions/gradle-jdk17@master - continue-on-error: true - with: - args: --all-projects --org=${{ env.SNYK_NETWORK_ORG_ID }} --severity-threshold=high - command: code test build: name: Unit tests and sonarqube runs-on: selfhosted-ubuntu-22.04-16-cores diff --git a/.github/workflows/phylum-daily-analysis.yaml b/.github/workflows/phylum-daily-analysis.yaml deleted file mode 100644 index 6bc1aa2dc6..0000000000 --- a/.github/workflows/phylum-daily-analysis.yaml +++ /dev/null @@ -1,65 +0,0 @@ -name: Daily Analysis Phylum - -on: - schedule: - # Runs at 14:00 UTC every day - - cron: '0 13 * * *' - -env: - PHYLUM_PROJECT_ID: 3f5b2c53-46bd-4f68-b050-5898f929002f - PHYLUM_GROUP_NAME: Protocol - PHYLUM_NAME: babylon-node -jobs: - analyze_branch_phylum: - name: Analyze dependencies with Phylum - permissions: - contents: read - pull-requests: write - runs-on: ubuntu-latest - strategy: - matrix: - branch: [main, develop, release/babylon, release/anemone, release/bottlenose] - include: - - branch: main - - branch: develop - - branch: release/babylon - - branch: release/anemone - - branch: release/bottlenose - fail-fast: false - steps: - - uses: RDXWorks-actions/checkout@main - with: - ref: ${{ matrix.branch }} - fetch-depth: 0 - - uses: RDXWorks-actions/setup-python@main - with: - python-version: 3.10.6 - - name: Install Phylum - run: | - curl https://sh.phylum.io/ | sh -s -- --yes - # Add the Python user base binary directory to PATH - echo "$HOME/.local/bin" >> $GITHUB_PATH - - name: Run Phylum Analysis - env: - PHYLUM_API_KEY: ${{ secrets.PHYLUM_API_KEY }} - run: | - phylum analyze --quiet --label ${{ matrix.branch }}_branch_daily_schedule > /dev/null 2>&1 || exit_code=$? - if [ $exit_code -eq 100 ]; then - echo "Phylum Analysis returned exit code 100, but continuing."; - echo "phylum_analyze_status=failure" >> $GITHUB_ENV - exit 0; - else - echo "phylum_analyze_status=success" >> $GITHUB_ENV - exit $?; - fi - - name: Analysis Status Failure notification - if: always() - uses: RDXWorks-actions/notify-slack-action@master - with: - status: ${{ env.phylum_analyze_status }} - notify_when: 'failure' - notification_title: ':clock3: Phylum Scheduled Daily Analysis:' - message_format: 'Automatic phylum analysis has found vulnerabilities on ${{ env.PHYLUM_NAME }} in ${{ matrix.branch }} branch:boom:' - footer: "Linked Repository <{repo_url}|{repo}> | " - env: - SLACK_WEBHOOK_URL: ${{ secrets.SLACK_PHYLUM_PROTOCOL_TEAM_WEBHOOK }} \ No newline at end of file diff --git a/.github/workflows/postman.yml b/.github/workflows/postman.yml deleted file mode 100644 index da8ec28792..0000000000 --- a/.github/workflows/postman.yml +++ /dev/null @@ -1,142 +0,0 @@ -name: Sync OpenAPI specs with Postman - -on: - push: - branches: - - develop - workflow_dispatch: - inputs: - network_name: - description: "Testnet to publish collections for" - required: true - default: gilganet - options: - - gilganet - - enkinet - - hammunet - - adapanet - -jobs: - cancel_running_workflows: - name: Cancel running workflows - runs-on: ubuntu-22.04 - steps: - - name: cancel running workflows - uses: RDXWorks-actions/cancel-workflow-action@main - with: - access_token: ${{ github.token }} - sync_collections: - name: Sync Postman collections w/ latest specs - environment: Postman - runs-on: ubuntu-22.04 - permissions: - id-token: write - contents: read - steps: - - name: Checkout core repo - uses: RDXWorks-actions/checkout@main - with: - fetch-depth: 0 - - uses: ./.github/actions/fetch-secrets - with: - role_name: "${{ secrets.BABYLON_SECRETS_ROLE_ARN }}" - app_name: "babylon-node" - step_name: "sync_collections" - secret_prefix: "POSTMAN_API" - secret_name: "github-actions/radixdlt/babylon-node/postman-token" - parse_json: true - - name: Set current date - id: date - run: echo "date=$(date +'%b %d, %H:%M')" >> $GITHUB_OUTPUT - - name: Set git tags - id: gittag - run: echo "gittag=$(git describe --tags --dirty)" >> $GITHUB_OUTPUT - - # ========== Mapping whatever network was selected to postman variables ========== - - name: Map network name to its variables - uses: RDXWorks-actions/variable-mapper@master - with: - key: "${{ github.event.inputs.network_name || 'gilganet' }}" - map: | - { - "gilganet": { - "core_api_collection_name": "Core API (${{steps.gittag.outputs.gittag}}) ${{steps.date.outputs.date}} UTC", - "system_api_collection_name": "System API (${{steps.gittag.outputs.gittag}}) ${{steps.date.outputs.date}} UTC", - "core_api_collection_id": "14449947-8696b2db-9d33-4892-b498-cf342d51d93f", - "system_api_collection_id": "14449947-4ce0b70b-0d0f-48da-aba7-ca7894877c4d" - }, - "enkinet": { - "core_api_collection_name": "Enkinet Core API (${{steps.gittag.outputs.gittag}})", - "system_api_collection_name": "Enkinet System API (${{steps.gittag.outputs.gittag}})", - "core_api_collection_id": "14449947-129acf1b-ad0d-483a-b2a9-b115b357de14", - "system_api_collection_id": "14449947-31bf4098-8854-429a-87e4-b8c52139f7d1" - }, - "hammunet": { - "core_api_collection_name": "Hammunet Core API (${{steps.gittag.outputs.gittag}})", - "system_api_collection_name": "Hammunet System API (${{steps.gittag.outputs.gittag}})", - "core_api_collection_id": "", - "system_api_collection_id": "" - }, - "adapanet": { - "core_api_collection_name": "Adapanet [ALPHANET] Core API (${{steps.gittag.outputs.gittag}})", - "system_api_collection_name": "Adapanet [ALPHANET] System API (${{steps.gittag.outputs.gittag}})", - "core_api_collection_id": "", - "system_api_collection_id": "" - }, - ".*": { - "core_api_collection_name": "Core API (${{steps.gittag.outputs.gittag}}) ${{steps.date.outputs.date}} UTC", - "system_api_collection_name": "System API (${{steps.gittag.outputs.gittag}}) ${{steps.date.outputs.date}} UTC", - "core_api_collection_id": "14449947-8696b2db-9d33-4892-b498-cf342d51d93f", - "system_api_collection_id": "14449947-4ce0b70b-0d0f-48da-aba7-ca7894877c4d" - } - } - - name: Echo environment - run: echo ${{ env.environment }} - - # ==================== Credentials ==================== - - name: Setup Postman credentials - uses: RDXWorks-actions/write-file-action@master - with: - path: .env - contents: POSTMAN_API_KEY=${{ env.POSTMAN_API_TOKEN }} - write-mode: overwrite - - # ==================== Core API Update ==================== - - name: Change OpenApi version in the Core API spec - run: sed -i 's/3.1.0/3.0.0/' core-rust/core-api-server/core-api-schema.yaml - - name: Update Postman's Core API collection from the spec - run: | - npx @apideck/portman@1.18.1 -l core-rust/core-api-server/core-api-schema.yaml \ - --postmanUid ${{env.core_api_collection_id}} --syncPostman true --envFile .env --ignoreCircularRefs true - - name: Download the Core API collection - run: | - curl -X GET -H "X-API-KEY:${{env.POSTMAN_API_TOKEN}}" https://api.getpostman.com/collections/${{env.core_api_collection_id}} > tmp.core.collection.json - - name: Update the collection name w/ timestamp and git tag - run: | - sed -i 's/"name":".*","schema"/"name":"${{env.core_api_collection_name}}","schema"/' tmp.core.collection.json - - name: Set the Core API's {{baseUrl}} to {{coreBaseUrl}} - run: | - sed -i 's/{{baseUrl}}/{{coreBaseUrl}}/g' tmp.core.collection.json - - name: Update the Core API collection - run: | - curl -X PUT -H "X-API-KEY:${{env.POSTMAN_API_TOKEN}}" -H "Content-Type: application/json" \ - https://api.getpostman.com/collections/${{env.core_api_collection_id}} --data "@tmp.core.collection.json" - - # ==================== System API Update ==================== - - name: Update Postman's System API collection from the spec - run: | - npx @apideck/portman@1.18.1 -l core/src/main/java/com/radixdlt/api/system/system-api-schema.yaml \ - --postmanUid ${{env.system_api_collection_id}} --syncPostman true --envFile .env - - name: Download the System API collection - run: | - curl -X GET -H "X-API-KEY:${{env.POSTMAN_API_TOKEN}}" https://api.getpostman.com/collections/${{env.system_api_collection_id}} > tmp.system.collection.json - - name: Update the collection name w/ timestamp and git tag - run: | - sed -i 's/"name":".*","schema"/"name":"${{env.system_api_collection_name}}","schema"/' tmp.system.collection.json - - name: Set the System API's {{baseUrl}} to {{systemBaseUrl}} - run: | - sed -i 's/{{baseUrl}}/{{systemBaseUrl}}/g' tmp.system.collection.json - - name: Update the System API collection - run: | - curl -X PUT -H "X-API-KEY:${{env.POSTMAN_API_TOKEN}}" -H "Content-Type: application/json" \ - https://api.getpostman.com/collections/${{env.system_api_collection_id}} --data "@tmp.system.collection.json" diff --git a/Dockerfile b/Dockerfile index 35fb974c32..df0e2e3a23 100644 --- a/Dockerfile +++ b/Dockerfile @@ -50,13 +50,16 @@ ENV VERSION_LAST_TAG=$VERSION_LAST_TAG RUN apt-get update \ && apt-get install -y --no-install-recommends \ docker.io=20.10.24+dfsg1-1+deb12u1+b6 \ - libssl-dev=3.0.20-1~deb12u1 \ + libssl-dev=3.0.20-1~deb12u2 \ pkg-config=1.8.1-1 \ unzip=6.0-28 \ wget=${WGET_VERSION} \ software-properties-common=0.99.30-4.1~deb12u1 \ && apt-get install -y --no-install-recommends \ openjdk-17-jdk=17.0.19+10-1~deb12u2 \ + openjdk-17-jre=17.0.19+10-1~deb12u2 \ + openjdk-17-jdk-headless=17.0.19+10-1~deb12u2 \ + openjdk-17-jre-headless=17.0.19+10-1~deb12u2 \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* @@ -126,12 +129,12 @@ RUN apt-get update \ ca-certificates \ build-essential=12.9 \ # https://security-tracker.debian.org/tracker/CVE-2023-38545 - curl=7.88.1-10+deb12u14 \ + curl=7.88.1-10+deb12u15 \ g++-aarch64-linux-gnu \ g++-x86-64-linux-gnu \ libc6-dev-arm64-cross=2.36-8cross1 \ libclang-dev=1:14.0-55.7~deb12u1 \ - libssl-dev=3.0.20-1~deb12u1 \ + libssl-dev=3.0.20-1~deb12u2 \ pkg-config=1.8.1-1 \ && rm -rf /var/lib/apt/lists/* @@ -261,7 +264,7 @@ RUN apt-get update -y \ && apt-get -y --no-install-recommends install \ openjdk-17-jre-headless=17.0.19+10-1~deb12u2 \ # https://security-tracker.debian.org/tracker/CVE-2023-38545 - curl=7.88.1-10+deb12u14 \ + curl=7.88.1-10+deb12u15 \ gettext-base=0.21-12 \ daemontools=1:0.76-8.1 \ # https://security-tracker.debian.org/tracker/CVE-2023-4911