Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
138 changes: 138 additions & 0 deletions .depot/workflows/run-spec.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
# Reusable workflow: runs the behavioural spec at `spec-ref` against the calling SDK.
#
# Usage from an SDK repository:
#
# jobs:
# sdk-spec:
# uses: Flagsmith/sdk-spec/.github/workflows/run-spec.yml@v0.3.0
# with:
# spec-ref: v0.3.0
# language: python
name: Run SDK spec

on:
workflow_call:
inputs:
spec-ref:
description: Spec tag (or commit) to run. Keep it in sync with the ref in `uses:`.
type: string
required: true
spec-repository:
description: Repository holding the spec.
type: string
default: Flagsmith/sdk-spec
tags:
description: >-
pytest marker expression selecting scenarios by Gherkin tag, e.g. "core and not realtime".
Empty runs every scenario.
type: string
default: ''
language:
description: Harness to build the SDK image with, i.e. a directory under `harness/` in the spec.
type: string
required: true
dockerfile:
description: Path to the SDK's spec Dockerfile, relative to the SDK repository root.
type: string
default: sdk-spec/Dockerfile
context:
description: Docker build context for the SDK image, relative to the SDK repository root.
type: string
default: .
sdk-repository:
description: SDK repository to test. Defaults to the calling repository.
type: string
default: ''
sdk-ref:
description: SDK ref to test. Defaults to the triggering ref.
type: string
default: ''
flagsmith-image:
description: Override the Flagsmith image the spec is pinned to.
type: string
default: ''
missing-steps:
description: >-
`fail` or `xfail`. With `xfail`, scenarios using a step the SDK does not implement are expected
failures and keep the job green; the steps are reported as warnings.
type: string
default: fail
results-name:
description: Name of the uploaded JUnit results artifact. Must be unique per workflow run.
type: string
default: sdk-spec-results

permissions:
contents: read

jobs:
spec:
name: Spec ${{ inputs.spec-ref }}
runs-on: ubuntu-latest
timeout-minutes: 20
env:
SDK_IMAGE: sdk-spec-sdk:local
FLAGSMITH_IMAGE: ${{ inputs.flagsmith-image }}
steps:
- name: Check out SDK
uses: actions/checkout@v7
with:
repository: ${{ inputs.sdk-repository || github.repository }}
ref: ${{ inputs.sdk-ref }}
path: sdk

- name: Check out spec
uses: actions/checkout@v7
with:
repository: ${{ inputs.spec-repository }}
ref: ${{ inputs.spec-ref }}
path: spec

- name: Build SDK image
working-directory: sdk
env:
DOCKERFILE: ${{ inputs.dockerfile }}
CONTEXT: ${{ inputs.context }}
HARNESS: ${{ github.workspace }}/spec/harness/${{ inputs.language }}
run: |
if [ ! -d "$HARNESS" ]; then
echo "::error::No harness for language '$(basename "$HARNESS")' in the spec"
exit 1
fi
docker build --tag "$SDK_IMAGE" --file "$DOCKERFILE" --build-context harness="$HARNESS" "$CONTEXT"

- name: Start Flagsmith
working-directory: spec
run: |
[ -n "$FLAGSMITH_IMAGE" ] || unset FLAGSMITH_IMAGE
docker compose up --detach --wait --wait-timeout 300

- uses: astral-sh/setup-uv@v10.3.0
with:
working-directory: spec/runner

- name: Run spec
working-directory: spec/runner
env:
FLAGSMITH_URL: http://localhost:8000
FLAGSMITH_SDK_URL: http://flagsmith:8000
SDK_COMMAND: docker run --interactive --rm --network sdk-spec_default ${{ env.SDK_IMAGE }}
TAGS: ${{ inputs.tags }}
MISSING_STEPS: ${{ inputs.missing-steps }}
run: |
uv run pytest -v --junitxml="$GITHUB_WORKSPACE/spec-results.xml" ${TAGS:+-m "$TAGS"}

- name: Show container logs
if: failure()
working-directory: spec
run: docker compose logs

- name: Upload results
if: always()
# Depot CI disables artifact writes in reusable workflows called from another repository.
continue-on-error: true
uses: actions/upload-artifact@v7
with:
name: ${{ inputs.results-name }}
path: spec-results.xml
if-no-files-found: ignore
30 changes: 30 additions & 0 deletions .github/workflows/depot-copy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# Depot CI only calls reusable workflows from `.depot/workflows/`, and does not follow symlinks,
# so `run-spec.yml` is kept there as a copy.
name: Depot copy

on:
pull_request:
paths:
- .github/workflows/run-spec.yml
- .depot/workflows/run-spec.yml
push:
branches: [main]
paths:
- .github/workflows/run-spec.yml
- .depot/workflows/run-spec.yml

permissions:
contents: read

jobs:
check:
name: run-spec.yml in sync
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- name: Compare copies
run: |
diff -u .github/workflows/run-spec.yml .depot/workflows/run-spec.yml \
|| { echo "::error::Copy .github/workflows/run-spec.yml to .depot/workflows/run-spec.yml"; exit 1; }
2 changes: 2 additions & 0 deletions .github/workflows/run-spec.yml
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,8 @@ jobs:

- name: Upload results
if: always()
# Depot CI disables artifact writes in reusable workflows called from another repository.
continue-on-error: true
uses: actions/upload-artifact@v7
with:
name: ${{ inputs.results-name }}
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ executed against every SDK.
- [`harness/`](harness) — one harness per language, running an SDK's steps file
([`HARNESS.md`](HARNESS.md)).
- [`.github/workflows/run-spec.yml`](.github/workflows/run-spec.yml) — reusable workflow SDKs call.
[`.depot/workflows/run-spec.yml`](.depot/workflows/run-spec.yml) is an identical copy for SDKs on
Depot CI, which calls reusable workflows from `.depot/workflows/` only. Change both together.
- [`.github/workflows/compatibility.yml`](.github/workflows/compatibility.yml) — runs the spec
against every onboarded SDK on pull requests and nightly.

Expand Down
Loading