Skip to content

Improve Northflank diagnostics and support optional NORTHFLANK_TEAM_ID - #353

Open
kisernl wants to merge 15 commits into
masterfrom
devin/northflank-diagnostic
Open

Improve Northflank diagnostics and support optional NORTHFLANK_TEAM_ID#353
kisernl wants to merge 15 commits into
masterfrom
devin/northflank-diagnostic

Conversation

@kisernl

@kisernl kisernl commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

Investigation showed that the benchmark is failing because the configured NORTHFLANK_TOKEN can authenticate to account-level endpoints (/v1/plans and /v1/projects) but gets 401 Unauthorized on every project-level endpoint:

GET /v1/projects/{NORTHFLANK_PROJECT_ID}            -> 401
GET /v1/projects/{NORTHFLANK_PROJECT_ID}/services   -> 401
POST /v1/projects/{NORTHFLANK_PROJECT_ID}/services/deployment -> 401

The token does see one personal project whose ID matches NORTHFLANK_PROJECT_ID, so the project exists and the token can list it, but it cannot read or modify it. Northflank API tokens inherit their permissions from an RBAC role, and the role assigned to this token is missing the project-level permissions the provider needs:

  • Project > Projects > Manage > Read
  • Project > Services > General > Read
  • Project > Services > General > Create (and Update/Delete for destroy)

This PR makes two changes to help surface and fix the issue:

  1. Better diagnostics in benchmark output

    • packages/benchsdk-runner/src/runner.ts prints the provider's errorMessage alongside the error code, so runs now show FAILED — NorthflankApiCallError: Unauthorized instead of just FAILED — NorthflankApiCallError.
    • benchmarks/sandbox/types.ts and benchmarks/sandbox/legacy-results.ts carry that message through to the legacy JSON results.
  2. Support for team-scoped Northflank projects

    • benchmarks/sandbox/providers.ts now passes process.env.NORTHFLANK_TEAM_ID to the Northflank provider.
    • All relevant sandbox workflows load NORTHFLANK_TEAM_ID from the vault.
    • @computesdk/northflank already supported teamId in projectParams/serviceParams; the benchmark was simply not wiring the env var through.

If the project is team-owned, the token must be generated under that team and NORTHFLANK_TEAM_ID must be set in the vault. If the project is personal, the token's RBAC role needs the project/service permissions above. The NorthflankApiCallError: Unauthorized message should now make this obvious in future runs.

Link to Devin session: https://app.devin.ai/sessions/fd1b73e919a34eae8db2f1f1937c3205
Requested by: @kisernl


Open in Devin Review

devin-ai-integration Bot and others added 4 commits August 20, 2026 20:42
…nd legacy results

Co-Authored-By: Noah Kiser <noah@computesdk.com>
Co-Authored-By: Noah Kiser <noah@computesdk.com>
Co-Authored-By: Noah Kiser <noah@computesdk.com>
Co-Authored-By: Noah Kiser <noah@computesdk.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment, CI, and merge conflict monitoring

@open-cla

open-cla Bot commented Aug 20, 2026

Copy link
Copy Markdown

Contributor License Agreement

All contributors are covered by a CLA.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Open in Devin Review

…onse shape

Co-Authored-By: Noah Kiser <noah@computesdk.com>
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration Bot and others added 3 commits August 20, 2026 21:31
… id-token permission

Co-Authored-By: Noah Kiser <noah@computesdk.com>
…-client dependency

Co-Authored-By: Noah Kiser <noah@computesdk.com>
…workflow

Co-Authored-By: Noah Kiser <noah@computesdk.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

The temporary northflank-diagnostic.yml workflow was removed from this PR, so the id-token: write concern no longer applies.

devin-ai-integration Bot and others added 5 commits August 20, 2026 21:57
Co-Authored-By: Noah Kiser <noah@computesdk.com>
Co-Authored-By: Noah Kiser <noah@computesdk.com>
Co-Authored-By: Noah Kiser <noah@computesdk.com>
Co-Authored-By: Noah Kiser <noah@computesdk.com>
Co-Authored-By: Noah Kiser <noah@computesdk.com>
devin-ai-integration[bot]

This comment was marked as resolved.

…eate

Co-Authored-By: Noah Kiser <noah@computesdk.com>
devin-ai-integration[bot]

This comment was marked as resolved.

…gnostic

Co-Authored-By: Noah Kiser <noah@computesdk.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant