Skip to content

Run the live/keyed test suite in CI on push to main #131

Description

@martinkersner

What

Add a CI lane that runs the live keyed test suite (DATAMAXI_API_KEY=... python -m pytest tests/) on push to main only.

Why

.github/workflows/python-package.yml currently runs on every push but executes only the offline mocked lane (python -m pytest tests/ -m "not integration", keyless). The live lanes — tests/test_integration.py (integration marker) and tests/test_call.py (smoke lane) — hit prod endpoints and are deliberately skipped in CI (smoke skips cleanly without a key; integration is deselected). So real API/SDK regressions against prod are never caught in CI.

Scope / approach

  • New job (or gated step) that sets DATAMAXI_API_KEY from a repo secret and runs python -m pytest tests/ (full suite, incl. integration + smoke).
  • Gate to main only, e.g. if: github.ref == 'refs/heads/main' (or a separate workflow with on: push: branches: [main]) — do not run on every branch push (avoids leaking the key to fork/branch runs and burning prod quota).
  • Requires adding a DATAMAXI_API_KEY GitHub Actions secret (repo settings → Secrets).

Caveat — expect intermittent red

PR #130 removed the flaky-prod xfail markers, so the live lane is no longer masked and can go red intermittently through no code fault:

  • Cold API pods → funding_rate.* and naver.* return 500 "no data found" (NATS-warmed caches empty on cold start).
  • premium() on an empty page raises ValueError('no data found') instead of returning empty (client-side SDK gap; limit=10 reduces but doesn't eliminate the empty-window risk).

Because of this, make the live lane non-blocking for now — a scheduled/nightly run, or continue-on-error: true, rather than a required check — until the empty-result handling is normalized in the SDK. Note the DATAMAXI_TIMEOUT env knob (conftest, default 30s) for slow prod endpoints.

Done when

  • A CI run on push to main executes the full keyed suite against prod using the DATAMAXI_API_KEY secret.
  • The lane is non-blocking (does not fail the required build on transient prod-data flakiness).
  • Branch/fork pushes do not run the keyed lane (secret not exposed).

Related: #130

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions