Account for system prompt growth when sizing compaction - #496
Open
AshishKumar4 wants to merge 1 commit into
Open
AshishKumar4 wants to merge 1 commit into
AshishKumar4 wants to merge 1 commit into
Conversation
Compaction sizing starts from the tokens the provider reported for the last measured step and estimates only the messages added since. That assumed the system prompt was unchanged between turns, but it is rebuilt every turn from the workspace: the gadget list, the connectable vendors, the output formats, and (with #267) the ambient catalogs. Anything that grew there could keep the estimate under the threshold and send a prompt past the model's window. The step's system prompt length is now recorded beside its token total (StoredChatMetadata.measuredSystemPromptChars, cleared wherever totalTokens is) and the next turn adds the growth against the current length. A shrink is not credited: the usage is exact and the ratio a heuristic, so subtracting could under-count and skip a compaction the provider then refuses, with no successful step left to correct the baseline. Over-counting only compacts earlier, and the next measured step re-baselines. A step measured before the length was recorded prices the prompt as unchanged, as before. The estimate moves into estimateContextTokens, which the unit tests cover.
Preview:
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
|
LGTM! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this change?
Compaction sizing starts from the tokens the provider reported for the last measured step and
estimates only the messages added since (
agent.ts, thecontextTokenscomputation). That assumedthe system prompt was unchanged between turns. It is not: the prompt is rebuilt every turn from the
workspace's gadget list, connectable vendors and output formats, and #267 adds the ambient catalogs.
Growth there never entered the estimate, so a chat near its window could skip compaction and send a
prompt the provider rejects. Because a failed request records no usage, the stale baseline then
repeats on the next turn.
The step's system prompt length is now stored beside its token total
(
StoredChatMetadata.measuredSystemPromptChars, written inaddChatMessages, cleared at bothtotalTokensreset sites, stripped from client deliveries inchatMetaForClient). The next turnadds the growth against the current length. A shrink is not credited: the usage is exact and the
four-characters-per-token ratio is a heuristic, so subtracting could under-count; over-counting only
compacts earlier, and the next measured step re-baselines. A step measured before the length was
recorded prices the prompt as unchanged, which is today's behavior.
The estimate moves out of
runAgentintoestimateContextTokensinagent-compaction.ts.Why is this obviously correct and trivially verifiable?
The helper reproduces the previous estimate exactly (same measured gate, same post-measurement
filter including the measured record's tool results) and adds one non-negative term. The unit tests
cover the unmeasured path, the measured path, growth, shrink, and a measurement taken before the
length was recorded. The stored field is written only with
totalTokens, cleared wherevertotalTokensis, and never reaches clients.Run:
pnpm --filter @gadgets/workshop-backend exec vitest run __tests__/agent-compaction.test.tsChecklist
Checking every item does not guarantee acceptance. Maintainers determine whether
a pull request meets the contribution policy.