Skip to content

Keep basecamp connect running, and starting, through a rate-limited or failing token endpoint - #811

Merged
robzolkos merged 20 commits into
mainfrom
connect-survive-token-throttle
Oct 1, 2026
Merged

robzolkos merged 20 commits into
mainfrom
connect-survive-token-throttle

Conversation

@jeremy

@jeremy jeremy commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

The failure

When the agent token endpoint answers a renewal with a 429 or a 5xx, or doesn't answer at all, basecamp connect stops.

  • Running: the event feed classifies a failure only by the SDK's error type. managerTokens handed it the CLI's own error, so the feed read every non-transport failure as unrecoverable and ended the run.
  • Starting: the first read of who the agent is needs a token. A failed renewal ended the start.

Since #814 the connector is started from the person's Claude session by /basecamp-connect, and nothing restarts it. A connector that exits on a brief rate limit stays down until someone notices, and requests made while it's down are not picked up later.

The fix

One rule for the feed and the start. tokenRetry is the only classifier.

  • The feed reads through feedTokens, which hands it every token failure already classified as the SDK's error type.
  • A starting connector waits by the same rule in awaitConnectToken.
  • The two can't drift apart.

The status decides first. A response that arrived is classified by its status, even if its body was cut off or too large to read.

  • A 401/403 or invalid_client/invalid_grant is a refusal.
  • A 429 or 5xx is retryable.
  • Only a 200 cut short counts as a network failure.

A request that got no response is a network failure, and it's retried. That covers a dropped or refused connection, DNS, a timeout, and TLS or proxy failures. Retrying a misconfigured TLS or proxy setup forever is accepted on purpose: the process stays visibly alive and logs why.

Never retried: an egress-policy block (surfguard.ErrBlocked), or a URL the client can't send.

How long it waits: exactly the wait the server named, never longer than one cap. Otherwise it uses the feed's backoff. Nothing else sets it.

The start waits instead of exiting. It logs each wait once and keeps trying. A SIGTERM, an interrupt or a canceled context ends the wait promptly, even mid-renewal, with the connector's usual exit codes, and a second signal forces the exit. Any other failure at start still reads "Could not read who profile … is". Before waiting, the start checks the instance lock under the agent that connect.json names, taking it and releasing it at once. A second connector for a running agent is therefore refused immediately instead of waiting out a rate limit first. The lock is held for the run only once the credential is proven to be that agent, so a start holding some other agent's credential doesn't block the real connector while it waits.

The skill knows. /basecamp-connect treats the "could not get the agent's token yet" line as a connector that is waiting, not one that failed, and leaves it running.

What stays fatal

A credential refusal exits as a disconnect, in the running feed and at start alike. That means invalid_client, invalid_grant, or a bare 401/403, including one whose body was cut off. Other non-retryable answers keep their current behaviour.

Dropped since earlier revisions

  • The systemd start-limit rationale. Remove the connector's worker side #815 removed connect service, so the CLI no longer writes a systemd unit. The start waiting still matters, more so now: nothing restarts the connector at all.
  • Nothing on the worker side. This PR never touched the dispatcher, the drivers or the outbox, so Remove the connector's worker side #815's removal didn't take anything from it.

Tests

  • connect_token_retry_test.go: the live feed through feedTokens, against a fake token endpoint:
    • 429 with Retry-After in seconds and as a date → throttled
    • 429 without one → throttled for the default hold
    • 5xx, a truncated 503, or a dropped connection → transient
    • invalid_client, a bare 401, or a truncated 401 → unrecoverable, and recognised as a disconnect
  • connect_start_wait_test.go: the start:
    • 429 then 200, and 503 three times then 200 → waits, then comes up
    • a stored 429 hold due 90s out → waits 90s, not the backoff
    • a truncated 401 → exits as a disconnect
    • a truncated 503, a truncated 200, or an oversized 503 page → waited out
    • an untrusted TLS certificate → keeps waiting, logged
    • refusals → no wait
    • cancel, SIGTERM or interrupt during a wait or mid-renewal → ends promptly
    • a second signal → forces the exit
    • a second connector while the first holds the lock → refused at once, with nothing minted
    • a start waiting for its token → holds no lock
    • a 503 or 502 that names a Retry-After → waited exactly that, by the start and the feed
    • a 503 naming Retry-After 99999999 → waited for the cap, by the start and the feed, with the capped wait logged once
    • a closed signal channel → not taken as a stop; no log set → still waits
    • which no-response failures are retried and which are not
  • agent_hold_test: pins the wait a held mint carries.

go test ./internal/auth/... ./internal/commands/... ./internal/connector/... -race -count=2 and lint, compared with a clean origin/main on the same machine, show no new failures.

Side effects

  • The SDK's raw GET path now retries a rate-limited or failing token renewal within one call, the same way it retries a 429 from the API.
  • setup.ErrorText now renders such a failure as HTTP 429 / HTTP 503.

Not addressed

A credential refusal that arrives while the feed sits idle on its websocket. The feed makes no token requests while idle, so nothing notices a refusal until it next needs a token.

@jeremy
jeremy requested a balanced review from Copilot October 1, 2026 00:27
@jeremy

jeremy commented Oct 1, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-01T19:54:39.267534Z ecc90a7 Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions github-actions Bot added commands CLI command implementations tests Tests (unit and e2e) auth OAuth authentication labels Oct 1, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 247204b957

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/auth/revoke.go

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Valid HTTP-date and oversized Retry-After values are discarded, causing premature retries.

Review effort: Balanced
Findings: 1 Medium severity · 2 Low severity

Open (3)
What changed in this PR

Keeps active connectors alive when agent token renewal encounters retryable HTTP failures.

Changes:

  • Translates retryable CLI token errors into SDK errors.
  • Preserves numeric Retry-After values.
  • Adds feed classification tests.

[!TIP]
If you aren't ready for review, convert to a draft PR.
Click "Convert to draft" or run gh pr ready --undo.
Click "Ready for review" or run gh pr ready to reengage.

File Description
internal/​commands/​connect.go Adapts retryable renewal errors for SDK classification.
internal/​commands/​connect_token_retry_test.go Tests transient, throttled, and fatal renewals.
internal/​auth/​revoke.go Carries parsed retry delays through the error chain.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread internal/auth/revoke.go
Comment thread internal/commands/connect_token_retry_test.go Outdated
Comment thread internal/commands/connect_token_retry_test.go Outdated
Copilot AI balanced review requested due to automatic review settings October 1, 2026 00:34
@jeremy

jeremy commented Oct 1, 2026

Copy link
Copy Markdown
Member Author

@codex review

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread internal/auth/revoke.go
Comment thread internal/auth/revoke.go Fixed
Copilot AI previously approved these changes Oct 1, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approved

The retry classification and delay propagation are correct and comprehensively tested.

Review effort: Balanced
Findings: None

Resolved since last review (3)

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 10384de3fa

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot AI balanced review requested due to automatic review settings October 1, 2026 00:40
Copilot AI dismissed their stale review, a newer Copilot review was requested October 1, 2026 00:40
@jeremy

jeremy commented Oct 1, 2026

Copy link
Copy Markdown
Member Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: 5a37fbfd18

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot AI previously approved these changes Oct 1, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approved

The retry classification and delay propagation are coherent and comprehensively tested.

Review effort: Balanced
Findings: None

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread internal/commands/connect_token_retry_test.go Outdated
Copilot AI balanced review requested due to automatic review settings October 1, 2026 00:46
@jeremy

jeremy commented Oct 1, 2026

Copy link
Copy Markdown
Member Author

@codex review

Copilot AI dismissed their stale review, a newer Copilot review was requested October 1, 2026 00:46
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. What shall we delve into next?

Reviewed commit: 20d3240248

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Copilot AI previously approved these changes Oct 1, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approved

The retry classification is correct, preserves fatal credential refusals, and is comprehensively tested.

Review effort: Balanced
Findings: None

@jeremy

jeremy commented Oct 1, 2026

Copy link
Copy Markdown
Member Author

Converged on 20d3240: CI is green (CodeQL and all four race shards included), Copilot approved this head with no suppressed or previously-missed items in its review body, Codex found no issues on this head, cubic's check passed on it, the branch merges cleanly into main, and there are 0 unresolved threads.

All review findings were fixed rather than declined:

  • Retry-After in HTTP-date form, and oversized values, were being dropped. Fixed in 10384de (Codex, Copilot, cubic).
  • CodeQL flagged an unbounded integer conversion. Fixed in 5a37fbf.
  • require was being called from the handler goroutine in a test. Fixed in 20d3240 (cubic).
  • Two test naming and wording nits. Fixed in 10384de (Copilot).

Left for a human decision, as the PR body explains: the connector's startup identity read still exits during a rate limit or hold, and a refusal that arrives while the feed is idle on its websocket is not acted on here. This PR is based on main, not stacked on #787. If #787's heldMint attaches the same Retry-After cause, the feed will wait out a hold exactly. Without it, the feed falls back to its own backoff.

A 503 or a bot user's OAuth refresh could name a Retry-After of years,
and the connector would wait it: a hang, not a wait. Every wait a server
names, whatever the status or token path, now goes through one cap,
auth.MaxServerWait, the bound #787 already puts on a mint hold, so a
held rate limit and every other named wait agree. When the server asked
for longer, the start and the feed say so once: what it asked for, and
what is waited.
Copilot AI balanced review requested due to automatic review settings October 1, 2026 19:08
@jeremy
jeremy force-pushed the connect-survive-token-throttle branch from 796cba0 to cc2f30e Compare October 1, 2026 19:08
@jeremy

jeremy commented Oct 1, 2026

Copy link
Copy Markdown
Member Author

@codex review

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Agent-mint 429 waits still use the old 30-minute cap instead of the intended unified one-hour cap.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)

Comment thread internal/auth/agent_hold.go Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: cc2f30e4a6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/auth/agent_hold.go Outdated
MaxServerWait was an hour, while #787 writes a rate-limit hold under the
thirty-minute bound its Retry-After parser applies: an agent's 429 was
capped at thirty minutes and every other named wait at an hour. The one
cap is now that thirty minutes, and the hold is written under it by
name. A 429 held for the cap keeps what the server asked for, so the
start and the feed say both.
Copilot AI balanced review requested due to automatic review settings October 1, 2026 19:24
@jeremy

jeremy commented Oct 1, 2026

Copy link
Copy Markdown
Member Author

@codex review

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 5 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Re-trigger cubic

Comment thread internal/auth/agent.go

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Oversized numeric Retry-After values are still persisted as one-minute agent holds instead of the unified cap.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
Resolved since last review (1)

Comment thread internal/auth/agent_hold.go

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a4438209e8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/commands/connect.go
Comment thread internal/auth/agent_hold.go
Two parsers read the header: the agent-connect one, which #787's hold
used, read a count too large for an int as no header at all and held for
the one-minute default, while the connector's saturated it to the cap.
The hold was then shorter than the wait the connector reported, and
another process could mint inside it. retryAfter now reads through
retryAfterSeconds and applies its own ceiling, so there is one parser and
the hold and the reported wait agree.
Copilot AI balanced review requested due to automatic review settings October 1, 2026 19:37
@jeremy

jeremy commented Oct 1, 2026

Copy link
Copy Markdown
Member Author

@codex review

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

It changes authentication retries, signal handling, and process locking across startup and runtime paths.

Review effort: Balanced
Findings: None

Resolved since last review (1)

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0921b56798

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/commands/connect_start_wait.go
A bot user's refresh reaches the start wrapped in this package's SDK
error, whose wait is already capped, and the start's log read only that
outer error: it said the cap as if the server had asked for it. Every
SDK error down the chain is read now, so the log says what was asked
and what is waited.
Copilot AI balanced review requested due to automatic review settings October 1, 2026 19:49
@jeremy

jeremy commented Oct 1, 2026

Copy link
Copy Markdown
Member Author

@codex review

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

It combines authentication, retry timing, signal concurrency, and cross-process locking changes that warrant final human validation.

Review effort: Balanced
Findings: None

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. More of your lovely PRs please.

Reviewed commit: ecc90a7e6c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@jeremy

jeremy commented Oct 1, 2026

Copy link
Copy Markdown
Member Author

Review threads: 8 resolved (7 fixed, 1 declined with the reasoning in its thread).
Declined:

  • Wake the start when the credential is replaced: another variant of the wait rule. Waking early would mean watching the credential store; the worst cost is a reconnected agent's connector waiting out at most the 30-minute cap, which it logs, and a restart ends that.

The wait rule, which applies to any further variant: wait exactly what the server named, on any retryable status, never longer than the one cap (auth.MaxServerWait, 30 minutes, the bound #787 writes a rate-limit hold under). Otherwise use the feed's own backoff. Every Retry-After goes through one parser. A wait capped below what the server asked for is logged once, with both durations.

Head ecc90a7:

  • CI is green: all four Race Detection shards, CodeQL and cubic.
  • Codex found no issues on this head. Copilot reported on this head with no findings and nothing suppressed or previously missed.
  • The branch merges cleanly into main, and there are 0 unresolved threads.

@robzolkos
robzolkos merged commit 9a8c607 into main Oct 1, 2026
32 checks passed
@robzolkos
robzolkos deleted the connect-survive-token-throttle branch October 1, 2026 20:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

auth OAuth authentication commands CLI command implementations skills Agent skills tests Tests (unit and e2e)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants