Repository navigation
[Bug][github] GitHub App installation token not refreshed for remote scopes / connection APIs, causing silent "zombie" connection after about 1 hour #8847
Description
Search before asking
- I had searched in the issues and found no similar issues.
What happened
When using **GitHub App authentication (**not PAT) on a GitHub connection, the connection silently becomes non-functional for administrative operations after approximately 1 hour, even when no requests have been made during that period.
Observed symptoms:
- The private GitHub organization no longer appears when trying to add a new scope through the Config UI.
- The API endpoint used to search/list repositories before creating a scope returns empty results, making it impossible to add new scopes.
- No error is logged in the devlake-lake pod — the failure is completely silent.
- Data collection on already-configured scopes continues to work normally during the same period. Blueprints run successfully, data is collected, pipelines complete without error.
- Editing the connection and re-submitting the same GitHub App private key restores the listing functionality for another ~1 hour.
- Restarting the devlake-lake pod (without re-editing the connection) also restores functionality for ~1 hour.
Key discriminating observation:
Once the connection is in the "zombie" state, REST data collection tasks keep succeeding on existing scopes while the remote scopes listing API returns empty. This decoupling strongly indicates that two different HTTP clients with independent token lifecycles are in use: one is being refreshed correctly, the other is not.
What do you expect to happen
The GitHub App installation token should be transparently refreshed for all HTTP clients used by the GitHub plugin — including the ones backing the remote scopes API (org/repo listing) and the Test Connection endpoint — the same way it is already refreshed for REST data collection tasks since #8746.
The connection should remain fully usable indefinitely without requiring a pod restart or a manual re-edit of the private key.
How to reproduce
- Deploy DevLake v1.0.3-beta10 (Helm chart) on Kubernetes.
- Create a GitHub connection using GitHub App authentication (App ID + installation ID + private key). Do not use a PAT.
- Create a project and add at least one scope pointing to a private repository of the installed organization. Verify that data collection works.
- Wait ~1 hour without making any request against the connection.
- In the Config UI, attempt to add a new scope to the project: the organization no longer appears in the repository picker.
- Via the API, attempt to search repositories through the remote scopes endpoint before creating a scope: the response is empty.
- Trigger a blueprint on an existing scope: data collection still succeeds, proving the installation token itself is being refreshed somewhere in the process but not on the admin/listing code path.
- Restart the devlake pod (no connection edit, no DB change): listing works again for another ~1 hour.
Anything else
Root cause hypothesis
This appears to be the same architectural pattern as #8788, but on a different code path.
PR #8746 introduced NewAppInstallationTokenProvider + RefreshRoundTripper and wired them into CreateApiClient so that REST data collection tasks get transparent token refresh. Issue #8788 identified that the GraphQL client in backend/plugins/github_graphql/impl/impl.go was not updated as part of #8746 and still uses oauth2.StaticTokenSource, which freezes the token at task start time.
The present bug strongly suggests that the HTTP client(s) backing the remote scopes / connection admin APIs (org listing, repo search, test connection) suffer from the same issue: they were not recabled to the RefreshRoundTripper machinery in #8746. The installation token captured when the process starts (or when the connection is last edited) is cached in memory and never refreshed, so after the 1-hour GitHub installation token TTL expires, all admin API calls fail silently — likely because GitHub returns 401 Bad credentials and the listing code path interprets the empty/errored response as "no orgs available" without surfacing an error.
This would explain every observed characteristic:
- Why it is silent: the error is swallowed at the listing layer instead of being logged (unlike the GraphQL path in [Bug][github_graphql] GitHub App installation token not refreshed for GraphQL client, panic and pipeline crash on large repos #8788 which panics loudly).
- Why collection still works: the REST path uses the client correctly refreshed by feat: GitHub App token refresh #8746.
- Why a pod restart fixes it temporarily: the in-memory frozen token is replaced by a freshly minted one when the process starts.
- Why re-editing the connection fixes it temporarily: it likely forces a new connection object to be built, including a new client with a fresh token.
- Why other environments with the same version may not see it: pods that are restarted more frequently (rolling deploys, aggressive liveness probes, autoscaling) never reach the 1-hour threshold in steady state.
Version
v1.0.3-beta10
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
fxgagnon commentedon Apr 23, 2026
I've been able to get debug logs from lake pod when it happens.
dosubot commentedon Apr 23, 2026
Hi @fxgagnon! I'm Dosu and I’m helping the incubator-devlake team.
Those debug logs confirm the root cause. The
"[remote_api_helper] Reused api client"message comes fromDsRemoteApiProxyHelper.getApiClient(), which caches API clients in memory keyed byconnection.GetHash()(a combination of connection ID +UpdatedAttimestamp). Once a client is cached, it's reused indefinitely — the GitHub App installation token baked into that client is never refreshed.The flow when the bug triggers:
"Reused api client"), send the expired token to GitHub, get a 401, and the response is interpreted as "no orgs/repos" — hence the empty result with a200status (no error surfaced).The
"Authorization is empty"line in your logs is likely from the DevLake auth middleware checking the incoming HTTP request (user → DevLake), not the outgoing GitHub request — it's unrelated to the token issue itself.The same
RemoteApiHelper.GetApiClient()inremote_api_helper.gohas the identical caching pattern without any token expiry check.Why collection still works: Task-based data collection creates a fresh API client per task execution via
CreateApiClient(taskCtx, connection), so each pipeline run gets a new token.Quickest fix path if you're planning to submit a PR: make
GetHash()return""for GitHub App connections (disabling caching and forcing a fresh client + token per request), or add a TTL/expiry check ingetApiClient()so cached clients are evicted before the token's ~1 hour lifetime. The Zentao plugin already uses the empty-hash approach as a precedent for connections with expiring credentials.To reply, just mention @dosu.
Share context across your team and agents. Try Dosu.
fix: disable API client caching for GitHub App connections (apache#8850)
fix: disable API client caching for GitHub App connections (apache#8850)
Sync upstream Apache DevLake (v1.0.3-beta9 → beta15) (#22)